17.3 COMPARING FORECASTING METHODS: THE GIACOMINI–WHITE APPROACH

17.3 COMPARING FORECASTING METHODS: THE GIACOMINI–WHITE APPROACH

Giacomini and White (2006) propose a fundamentally different but highly relevant approach to testing between alternative forecasts. They do not rely on asymptotic results that replace parameter estimates by their probability limits—an approach that basically tests which model is better in population. Instead Giacomini and White (2006) retain the effect that estimation errors have on the forecasts and ask whether two forecasting methods produce the same quality of forecasts (according to the chosen loss function) or if instead one method is better. Their test takes as given an observed sequence of forecasts from the two methods that are being compared and assumes that the parameters of the models are estimated using a rolling window of fixed length. This preserves estimation errors and can be viewed as a sequence of observations on the methods’ performance. Tests for equal expected loss as well as tests of orthogonality of forecast errors with respect to all information available when the forecasts were produced can then be performed. The distribution of such tests is approximated under the assumption that the observed sequence of forecasts gets large.

More formally, consider two models that at time t are used to generate one-stepahead forecasts $f_{1t+1|t}$ and $f_{2t+1|t}$ using a fixed window of $\omega$ observations. Each forecast $f_{it+1|t}$ is a function of the data $(z_t,z_{t-1},\dots ,z_{t-\omega +1})$ and parameter estimates ${\hat{\beta }}_{1t},{\hat{\beta }}_{2t}$ and is denoted by $f_{it+1|t}({\hat{\beta }}_{it}),i=1,2$ for short. From these forecasts we can construct losses $L(f_{it+1}({\hat{\beta }}_{it}),y_{t+1})$ for $t=T_R,\dots ,T-1$ . These can be used to compare the models’ finite sample predictive accuracy evaluated at the current parameters, ${\hat{\beta }}_{1t},{\hat{\beta }}_{2t}$ , through the following null:

$$H_0:E\left[L(f_{1t+1\mid t}({\hat{\beta }}_{1t}),y_{t+1})-L(f_{2t+1\mid t}({\hat{\beta }}_{2t}),y_{t+1})\right]=0.\text{\hspace{1em}(17.20)}$$

It is useful to contrast the null hypothesis in (17.20) with the null of equal predictive accuracy at the population parameters, ${\beta }_1^{\ast },{\beta }_2^{\ast }$ , in the analysis of West (1996) and many subsequent papers:

$$\begin{array}{r}{H}_0:E\left[L(f_{1t+1\mid t}({\beta }_1^{\ast }),y_{t+1})-L(f_{2t+1\mid t}({\beta }_2^{\ast }),y_{t+1})\right]=0.\end{array}\text{\hspace{1em}(17.21)}$$

The null in (17.20) tested by Giacomini and White (2006) is fundamentally different from the null in (17.21). The key difference is that the effect of estimation error does not vanish in (17.20), which assumes a fixed estimation window, whereas it does so for (17.21) which evaluates the expected losses at the (probability) limits of the estimators, $\hat{\beta },$ , as the estimation sample gets very large. For example, suppose that the finite-sample bias in the small model due to the omission of relevant predictor variables balances exactly against the reduced effect of estimation error, both measured relative to a larger, unrestricted model. Then, the null hypothesis tested by Giacomini and White should not be rejected. In contrast, the null tested by West should be rejected as the estimation sample expands and the effect of estimation error vanishes.

Conversely, when comparing nested models, (17.20) can set a higher standard relative to tests such as (17.21) since the large model is now required to outperform the small model by a margin big enough to make up for the greater effect that estimation error has on the large model’s forecasting performance.

Giacomini and White (2006) establish that, under a finite estimation window,

$$T_P^{-1/2}\sum _{t=T_R}^{T-1}[\Delta L_{t+1}({\hat{\beta }}_{1t},{\hat{\beta }}_{2t},y_{t+1})-E[\Delta L]{]}^d\mathrm{N}(0,{\tilde{S}}_y(0)),$$

where $\begin{array}{r}\Delta L_{t+1}({\hat{\beta }}_{1t},{\hat{\beta }}_{2t},y_{t+1})=L(f_{1t+1\mid t}({\hat{\beta }}_{1t}),y_{t+1})-L(f_{2t+1\mid t}({\hat{\beta }}_{2t}),y_{t+1})\end{array}$ measures the differential loss, while the variance ${\tilde{S}}_y(0)$ is given by

\begin{array} { r l } & { \tilde { \boldsymbol { \mathrm {S} } } _ { y } ( 0 ) = \displaystyle \operatorname* { l i m } _ {T _ { P } \to \infty } \mathrm { V a r } \Bigg ( T _ { P } ^ { - 1 / 2} \sum _ {t = T _ {R}} }^{T - 1} \Big [ L ( f _ { 1 t + 1 \mid t } ( \hat { \beta } _ { 1 t } ) , y _ { t + 1 } ) - L ( f _ { 2 t + 1 \mid t } ( \hat { \beta } _ { 2 t } ) , y _ { t + 1 } ) } \\ & { ~ - E [ \Delta L _ { t + 1} ] \Big ] \Bigg ) . } \end{array}

Note that this result is obtained in the limit for $T_P\infty$ but without the assumption that $T_R$ expands asymptotically. Hence, Giacomini and White (2006) do not need to make assumptions such as positive-definiteness of  and so their approach allows for both nested and nonnested comparisons. Moreover, a much wider class of forecast methods that do not necessarily fit in the mold of the analysis of West (1996) can be considered, including Bayesian, nonlinear, and nonparametric models, as well as forecasts based on a variety of nonstandard estimators. In each case it is important that the effect of estimation error does not vanish so that ${\tilde{S}}_y(0)$ does not become degenerate.