正在学习

17.1.1 Encompassing Tests under MSE Loss

A simple linear regression of the outcome on the two forecasts can be used to test forecast encompassing under MSE loss:

$y_{t + 1} = β_{1} f_{1 t + 1∣ t} + β_{2} f_{2 t + 1∣ t} + ε_{1 t + 1} . (17.5)$

The first forecast encompasses the second provided that $β_{1} = 1$ and $β_{2} = 0.$ This regression test in levels of the variables provides a useful link to forecast combinations.3 Forecast encompassing occurs when one forecast gets a weight of 1 and the other forecast gets a weight of 0. This could arise, for example, if one forecast was just a noisy proxy (where the noise is independent of the forecast and outcome) of the other forecast. In this situation it is optimal to rely on only a single forecast. For all other situations, it is potentially better to combine the two forecasts.

Chong and Hendry (1986) suggest imposing $β_{1} = 1$ in (17.5) and testing $β_{2} = 0$ after subtracting $f_{1 t + 1∣ t}$ from both sides,

$y_{t + 1} - f_{1 t + 1∣ t} = e_{1 t + 1 t} = β_{2} f_{2 t + 1∣ t} + ε_{2 t + 1∣ t} . (17.6)$

A more common forecast encompassing test is to run the regression

$e_{1 t + 1∣ t} = β (e_{1 t + 1∣ t} - e_{2 t + 1∣ t}) + ε_{3 t + 1∣ t} . (17.7)$

The null hypothesis is $H_{0} : β = 0$ versus the alternative $H_{1} : β > 0$ . The levels specifications in (17.5) and (17.7) are identical under the restriction that $β_{1} + β_{2} = 1$ (which includes the null hypothesis of $β_{1} = 1$ and $β_{2} = 0)$ , but will otherwise differ.

Harvey, Leybourne, and Newbold (1998) propose the out-of-sample test statistic

$T_{P}^{1/2} \frac{c ˉ}{S ^{1/2}}, (17.8)$

where S is a consistent estimator of the long-run variance of $c_{t + 1} = \overset{e}{^}_{1 t + 1} (\overset{e}{^}_{1 t + 1} -$ $\overset{e}{^}_{2 t + 1})$ and $\overset{c}{ˉ} = T_{P}^{- 1} \sum_{t = T_{R}}^{T - 1} (\overset{e}{^}_{1 t + 1} - \overset{e}{^}_{2 t + 1}) \overset{e}{^}_{1 t + 1}$ . As in the previous chapter we assume that the sample T has been split into an estimation sample of length $T_{R}$ and an evaluation sample $T_{P} = T - T_{R}$ . We retain hats on the errors to indicate that they are based on estimated forecasts. For one-step-ahead forecasts with serially uncorrelated and homoskedastic errors $(h = 1)$ , this becomes what Clark and McCracken (2001) refer to as the ENC-T test:

$ENC - T = \frac{( T _{p} - 1 ) ^{1/2} c ˉ}{( T _{p}^{- 1} \sum _{t = T_{R}}^{T - 1} ( c _{t + 1}^{2} - c ˉ ) ^{2} ) ^{1/2}} = \frac{( T _{p} - 1 ) ^{1/2} T _{p}^{- 1} \sum _{t = T_{R}}^{T - 1} ( e ^ _{1 t + 1}^{2} - e ^ _{1 t + 1} e ^ _{2 t + 1} )}{( T _{p}^{- 1} \sum _{t = T_{R}}^{T - 1} ( e ^ _{1 t + 1}^{2} - e ^ _{1 t + 1} e ^ _{2 t + 1} ) ^{2} - c ˉ ^{2} ) ^{1/2}}, (17.9)$

where the denominator is the usual variance estimator for c¯. This is a regression of $c_{t + 1}$ on a constant, scaled by $(T_{p} - 1)^{1/2}$ . Notice that $c_{t + 1}$ is the element being summed in the numerator of the regression coefficient from (17.7) and so ENC-T tests the restriction that $β = 0$ in that regression.

An alternative, regression-based test, labeled the ENC-REG test, takes the form

$ENC - REG = \frac{( T _{\overset{p}{ˉ}} - 1 ) ^{1/2} T _{p}^{- 1} \sum _{t = T_{R}}^{T - 1} ( e ^ _{1 t + 1}^{2} - e ^ _{1 t + 1} e ^ _{2 t + 1} )}{( T _{p}^{- 1} \sum _{t = T_{R}}^{T - 1} ( e ^ _{1 t + 1} - e ^ _{2 t + 1} ) ^{2} ( T _{\overset{p}{ˉ}}^{- 1} \sum _{t = T_{R}}^{T - 1} e ^ _{1 t + 1}^{2} ) - c ˉ ^{2} ) ^{1/2}} . (17.10)$

Clark and McCracken (2001) suggest a different test when forecasts are based on estimates that arise from nested models:

$ENC - NEW = \frac{T _{\overset{p}{^}} c ˉ}{( T _{\overset{p}{^}}^{- 1} \sum _{t = T_{R}}^{T - 1} e ^ _{2 t + 1}^{2} )} . (17.11)$

Apart from scaling by $T_{p}$ rather than $T_{P} - 1$ , the difference is that the usual variance estimator is removed from the denominator of the t-statistic and replaced with the MSE of the encompassed (under the null) forecast. Clark and McCracken suggest that this test will have better properties than the t-test when the parameters of the models generating the forecasts are estimated and the models are nested.

These tests assume that the encompassing forecasts are unbiased. The tests can be modified if this assumption does not hold. For tests based on the regressions (17.5), (17.6), and (17.7), a constant can be added to the regression. For the statistic in (17.8) an adjustment needs to be made to the forecast error $\hat{e}_{1 t + 1}$ , so that it has a mean of 0 (see Marcellino, 2000).

When deriving limit distributions for these test statistics, care needs to be taken with the standard errors since the statistics are functions of objects (data) that were themselves constructed from previously estimated forecasting models. Such estimation errors need not disappear asymptotically, depending on how the forecasts were constructed. We might therefore have to adjust the standard errors or, alternatively, use different distributional approximations for nested forecasting models.

These concerns matter if we are interested in drawing conclusions, from sample statistics on models’ relative forecasting performance, about which model best fits the data. However, if we are interested in testing which forecasting method performed best over a particular sample period, estimation error only matters indirectly in so far as it affects the sampling distribution of the models’ relative losses. As we shall see below, this is the perspective taken by Diebold and Mariano (1995) and Giacomini and White (2006).

Many empirical applications assume that the forecasts are “given,” or taken as primitives, and so regard parameter estimation error as a fundamental feature of the forecast. For example, Chong and Hendry (1986) reference the potential need to adjust for estimated forecasts, but do not account for estimation errors in the forecasts in their analysis. The main application they have in mind is examining forecasts from large complicated models, for which no results are available for accounting for estimation error. Where results are available (e.g., West (2001) for nonnested tests), this approximation is still reasonable if the size of the sample on which the model is estimated is large relative to the evaluation sample, a result that arises because the estimation error becomes negligible in large estimation samples. Harvey, Leybourne, and Newbold (1998) assume that the forecasts are constructed without estimation error, which yields asymptotically normal results with the usual variance–covariance matrix. They suggest using a student-t distribution to construct critical values as is common for nonnormal forecasts and small sample sizes. Size distortions caused by using the asymptotic approximation can be quite substantial in such cases.

When models are not nested, the results of West and McCracken (1998) may apply to the regression tests in (17.5), (17.6), and (17.7) for recursively constructed forecasts that use rolling regressions or a fixed estimation window. Forecast models must satisfy the requirements of West (1996, Theorem 4.1) discussed in chapter 16, as must the underlying data. In addition, the models that generate the two forecasts must not be nested, nor both nest the true model. Under such conditions the results of West and McCracken apply directly to the statistic in (17.8) which, apart from the asymptotically irrelevant correction to the scaling by the sample size, yields a robust t-test on the coefficient from regressing $c_{t + 1}$ on a constant. West (2001) applies these results to this test, showing that when the forecasts are constructed recursively, a correction to the denominator is typically required to achieve the correct size asymptotically.

Under the conditions of West and McCracken (1998), consider forecasts constructed from nonnested regression models,

$γ_{t + 1} = β_{1} x_{1 t} + e_{1 t + 1}, γ_{t + 1} = β_{2} x_{2 t} + e_{2 t + 1},$

so the forecasts are $\hat{β}_{i} x_{i t}$ for $i = 1$ , 2 and the parameters are estimated using either a fixed, rolling or recursively expanding window with the evaluation sample starting at observation $T_{R} + 1$ , so $T_{R}$ observations are available for the initial estimates and $T_{P}$ observations are available for evaluation.

The results of West and McCracken (1998) applied to the problem in West (2001) show that

$T_{p} (\overset{c}{ˉ} - E (c_{t + 1})) \to^{d} N (0, S + π D V_{β} D^{'}),$

where $D = [E (e_{1 t + 1} x_{2 t}), E (e_{2 t + 1} x_{1 t})], e_{i t + 1}$ is the forecast error from the i th model, $π = T_{P} / T_{R}$ , and $V_{β}$ is the variance–covariance matrix of $(\hat{β}_{1}, \hat{β}_{2})^{'}$ This can be made operational by estimating D from the sample analogs $\hat{D} =$ $[T_{\overset{p}{^}}^{- 1} \sum \overset{e}{^}_{2 t + 1} x_{1 t}, T_{\overset{p}{^}}^{- 1} \sum^{^} \overset{e}{^}_{1 t + 1} x_{2 t}]$ , where the sums are over the evaluation sample; $\overset{ˉ}{V_{β}}$ can be estimated in the usual way considering the two univariate forecasting models as a system.

If the evaluation sample is small relative to the estimation sample (say 10% or less) and the forecasts are generated recursively, size distortions are typically not large. However, when this ratio gets closer to 0.5, the effect of estimation error on the size of the test is not negligible.

Monte Carlo simulations in Clark and McCracken (2001, 2005a) and Clark and West (2007) suggest that the tests generally have the right size with reasonable power when the forecast horizon covers only a single period $(h = 1)$ . Conversely, the tests are often oversized for multiperiod forecasts (h > 1). For this case, simulations reported by the same studies have found that bootstrap methods such as the fixedregressor bootstrap seem to work quite well.

When we construct the competing forecasts from nested linear regressions and account for the sampling error that arises from estimating the parameters of the underlying regressions used to construct the forecasts, distributions are no longer asymptotically normal. Clark and McCracken (2001) provide asymptotic distribution theory and tables of critical values for the ENC-T, ENC-REG, and ENC-NEW tests when the parameters of the underlying forecasting models are updated using the recursive scheme throughout the evaluation sample.

练习题

In the simple linear regression for forecast encompassing under MSE loss, what condition must be satisfied for the first forecast to encompass the second forecast?

β_{1} = 0

and

β_{2} = 1

β_{1} = 1

and

β_{2} = 0

β_{1} = 0.5

and

β_{2} = 0.5

β_{1} = - 1

and

β_{2} = 1

What is the null hypothesis in the common forecast encompassing regression test $e_{1 t + 1∣ t} = β (e_{1 t + 1∣ t} - e_{2 t + 1∣ t}) + ε_{3 t + 1∣ t}$ ?

H_{0} : β = 1

H_{0} : β = - 1

H_{0} : β = 0

H_{0} : β > 0

Which of the following are components of the Harvey, Leybourne, and Newbold's out - of - sample test statistic $T_{P}^{1/2} \frac{c ˉ}{S ^{1/2}}$ ?

T_{P}

, the length of the evaluation sample

\overset{c}{ˉ}

, the sample average of a certain error combination

S

, a consistent estimator of the long - run variance of

c_{t + 1}

β_{1}

and

β_{2}

from the simple linear regression for forecast encompassing

\overset{e}{^}_{1}

and

\overset{e}{^}_{2}

, the estimated forecast errors

In the ENC - T test, if the errors are serially uncorrelated and homoskedastic ( $h = 1$ ), the test statistic has a specific form as given by Clark and McCracken.

In the Chong and Hendry's forecast encompassing test, after imposing $β_{1} = 1$ in $y_{t + 1} = β_{1} f_{1 t + 1∣ t} + β_{2} f_{2 t + 1∣ t} + ε_{1 t + 1}$ and subtracting $f_{1 t + 1∣ t}$ from both sides, the resulting regression is $y_{t + 1} - f_{1 t + 1∣ t} = e_{1 t + 1 t} = β_{2} f_{2 t + 1∣ t} + ε_{2 t + 1∣ t}$ . The variable $e_{1 t + 1 t}$ represents the ___.

Explain the significance of the condition $β_{1} = 1$ and $β_{2} = 0$ in the simple linear regression for forecast encompassing under MSE loss.

What is the denominator of the ENC - REG test statistic?

(T_{p}^{- 1} \sum_{t = T_{R}}^{T - 1} (\overset{e}{^}_{1 t + 1} - \overset{e}{^}_{2 t + 1})^{2} (T_{\overset{p}{ˉ}}^{- 1} \sum_{t = T_{R}}^{T - 1} \overset{e}{^}_{1 t + 1}^{2}) - \overset{c}{ˉ}^{2})^{1/2}

(T_{p}^{- 1} \sum_{t = T_{R}}^{T - 1} (c_{t + 1}^{2} - \overset{c}{ˉ})^{2})^{1/2}

(T_{p}^{- 1} \sum_{t = T_{R}}^{T - 1} (\overset{e}{^}_{1 t + 1}^{2} - \overset{e}{^}_{1 t + 1} \overset{e}{^}_{2 t + 1})^{2} - \overset{c}{ˉ}^{2})^{1/2}

S^{1/2}

The ENC - NEW test is used when forecasts are based on estimates from non - nested models.

When the assumption of unbiased encompassing forecasts does not hold, for tests based on regressions (17.5), (17.6), and (17.7), a ___ can be added to the regression.

How do estimation errors affect the derivation of limit distributions for forecast encompassing test statistics?

Which of the following knowledge points are related to the concept of comparing the predictive accuracy of forecasting methods? (Select all that apply)

A. kp_17_1_1_1 (Simple Linear Regression for Forecast Encompassing under MSE Loss)

B. kp_16_6_009 (Outcomes of comparing predictive accuracy of forecasting methods)

C. kp_17_1_1_3 (Common Forecast Encompassing Regression Test)

D. kp_16_6_003 (Comparison of multiple forecasts)

E. kp_17_1_1_7 (ENC - NEW Test by Clark and McCracken)

In the context of forecast encompassing, what does it mean if one forecast encompasses another under MSE loss? (Combining knowledge from kp_17_1_1_1 and kp_17_1_007)

A. The encompassed forecast has a lower MSE than the encompassing forecast.

B. The encompassing forecast has a lower MSE than the encompassed forecast, and the expected loss from the encompassed forecast exceeds that of the encompassing forecast.

C. Both forecasts have the same MSE.

D. The relationship between MSEs of the two forecasts is indeterminate.

Which of the following correctly describes the null hypothesis for the common forecast encompassing regression test? The regression is given by $e_{1 t + 1∣ t} = β (e_{1 t + 1∣ t} - e_{2 t + 1∣ t}) + ε_{3 t + 1∣ t}$ .

H_{0} : β = 1

H_{0} : β = 0

H_{0} : β > 0

H_{0} : β < 0

Which of the following statements are true regarding the ENC-T test by Clark and McCracken? Select all that apply.

A. It is used for one-step-ahead forecasts with serially uncorrelated and homoskedastic errors.

B. The test statistic is given by

T_{P}^{1/2} \frac{c ˉ}{S ^{1/2}}

C. The null hypothesis is

H_{0} : β = 0

in the regression

e_{1 t + 1∣ t} = β (e_{1 t + 1∣ t} - e_{2 t + 1∣ t}) + ε_{3 t + 1∣ t}

D. The ENC-T test is identical to the Harvey, Leybourne, and Newbold's out-of-sample test statistic under certain restrictions.

The Chong and Hendry's forecast encompassing test involves imposing $β_{1} = 1$ in the regression $y_{t + 1} = β_{1} f_{1 t + 1∣ t} + β_{2} f_{2 t + 1∣ t} + ε_{1 t + 1}$ and testing $β_{2} = 0$ .

The ENC-REG test by Clark and McCracken is an alternative, regression-based test that takes the form $ENC - REG = \frac{( T _{\overset{p}{ˉ}} - 1 ) ^{1/2} T _{p}^{- 1} \sum _{t = T_{R}}^{T - 1} ( e ^ _{1 t + 1}^{2} - e ^ _{1 t + 1} e ^ _{2 t + 1} )}{( T _{p}^{- 1} \sum _{t = T_{R}}^{T - 1} ( e ^ _{1 t + 1} - e ^ _{2 t + 1} ) ^{2} ( T _{\overset{p}{ˉ}}^{- 1} \sum _{t = T_{R}}^{T - 1} e ^ _{1 t + 1}^{2} ) - c ˉ ^{2} ) ^{1/2}}$ . The numerator of this test statistic is related to the regression coefficient from the regression ___.

登录后解锁笔记、知识点解析、AI 问答

立即登录

17.1.1 Encompassing Tests under MSE Loss