17.3.1 Conditional Test of Forecasting Performance

17.3.1 Conditional Test of Forecasting Performance

Giacomini and White (2006) introduce conditional tests for predictive accuracy that are conditional on current information, $Z_t$ . For these tests the null hypothesis in (17.20) is altered to

$$\begin{array}{r}E\left[\Delta L_{t+1}({\hat{\beta }}_{1t},{\hat{\beta }}_{2t},y_{t+1})|Z_t\right]=0,\end{array}\text{\hspace{1em}(17.22)}$$

where $Z_t=\{z_1,\dots ,z_t\}$ is the information set available at time t. For a single-period forecast horizon, $h=1$ , the null is that the loss difference is a martingale difference sequence with respect to $Z_t$ . For longer horizons, the null implies that information available at time, t, is not correlated with the difference in the losses.

The hypothesis in (17.22) is interesting from an economic point of view since it allows us to test whether certain forecasting models are better in some economic states than others. For example, Henkel, Martin, and Nardari (2011) find that stock returns are predictable during economic recessions but not during expansions. This could be tested by letting the conditioning information include a recession indicator, $1({\mathrm{NBER}}_t=1)$ , that equals 1 if the NBER views period t as a recession, and otherwise equals 0, although the NBER indicator is not available in real time.

To make the conditional null hypothesis in (17.22) operational, we need to choose a set of test functions, $v_t$ , which are functions of data available at the time the forecast is made, i.e., functions of $Z_t$ . Letting $v_t$ be a $(q\times 1)$ vector, we can test the moment restriction in (17.22) using a standard GMM quadratic form,

$$\mathrm{GW}(T_P)=T_P{\left(T_P^{-1}\sum _{t=T_R}^{T-1}{v}_t\Delta L_{t+1}\right)}^{\prime }{W}^{-1}\left(T_P^{-1}\sum _{t=T_R}^{T-1}{v}_t\Delta L_{t+1}\right),\text{\hspace{1em}(17.23)}$$

where W is the optimal weight matrix for the GMM problem and $\Delta L_{t+1}=$ $L(f_{1t+1|t}({\hat{\beta }}_{1t}),y_{t+1})-L(f_{2t+1|t}({\hat{\beta }}_{2t}),y_{t+1})$ . Under relatively mild and standard conditions, Giacomini and White show that this statistic has a limiting ${\chi }_q^2$ distribution under the null hypothesis.

An interesting aspect of this test is that different choices for the window length in the rolling regressions, $\omega ,$ change what is being tested. The reason is that if we change the window length, we also alter ${\hat{\beta }}_{1t},{\hat{\beta }}_{2t}$ and thus the sequence of forecasts, $\{f_{1t+1|t}({\hat{\beta }}_{1t}),f_{2t+1|t}({\hat{\beta }}_{2t})\}$ , and the null hypothesis. The upshot of this is that with the same models but different window lengths, forecasters might find that the tests yield different results. However, this property is part of the point of undertaking the test for the forecast method (which includes the choice of estimation window) rather than attempting to learn which forecasting model is best when evaluated at the limit of the parameter estimates.

A failure of finding that the test in (17.23) detects superior performance for a model whose parameters are estimated using a rolling window does not imply, of course, that the same model, with parameters estimated on an expanding window, would not have generated better forecasts. Using a rolling window estimator in such situations can worsen the performance of large models with a greater number of estimated parameters and so can impair the test’s ability to identify these models as being superior relative to more parsimonious models with fewer estimated parameters, even if the large model is the best specification. For example, forecasts based on rolling window estimation with 10 years of observations might lead to rejections of the large model, while the large model could be preferred with a rolling estimation window of 20 years of observations.

17.4 COMPARING FORECASTING PERFORMANCE ACROSS NESTED MODELS

When comparing the finite-sample performance of two nested models, estimation error can cause the large model to produce less precise forecasts—generate higher MSE values—than the small model which requires estimation of fewer parameters. The test statistics proposed by McCracken (2000) take this into account. Specifically, the distribution of test statistics that account for estimation error shifts further to the left and in many cases takes on negative values, the greater the number of additional parameters that have to be estimated for the large model.

This property means that a possible outcome of the test of equal predictive accuracy could be to favor a large forecasting model even though this model generates less precise forecasts in a particular finite sample than a smaller model. The logic in such cases is that although the large model underperformed the small model in a finite sample, its performance was not as bad as one would have expected given the additional number of parameters that require estimation by the large model.

From the point of conducting inference about two models, this is a valid point. However, from the perspective of a forecaster who is deciding on which model to use, it seems risky to choose the large model in situations where it is underperforming the smaller model. This holds even if the sample evidence suggests that the larger model eventually will be preferred when enough data are available to estimate its additional parameters with greater precision.

Two approaches have been proposed to address these issues for nested models. One approach suggested by Clark and West (2007) recenters the test statistic in a way that explicitly adjusts the test for the greater effect of parameter estimation error on the large model. The second approach is to directly focus the test on finite-sample performance, as suggested by Giacomini and White (2006). We first explain how recursive parameter estimation induces standard evaluation test statistics to follow nonstandard distributions and next describe these approaches.