17.5 COMPARING MANY FORECASTS

17.5 COMPARING MANY FORECASTS

In cases with more than two forecasts to compare, we need to examine a vector of forecasts. Some of the methods discussed above extend to the general case with m forecasts (e.g., encompassing) while others do not (e.g., the Granger–Newbold comparison). This section discusses the effect on the tests of having an arbitrary number of forecasts.

With two forecasts, it is natural to test a null of equal predictive accuracy. In contrast, in the presence of multiple alternative forecasts, it is natural to ask whether there exists any forecast that is superior to the benchmark. This results in a composite hypothesis which is generally more difficult to test than the simple pairwise test of equal predictive accuracy.

17.5.1 Forecast Encompassing

Under MSE loss, forecast encompassing generalizes readily to testing that the first forecast encompasses the remaining forecasts for a general $(m\times 1)$ vector of forecasts. The vector analogs to (17.5), (17.6), and (17.7) become

$$\begin{array}{r}{y}_{t+1}={\beta }_1{\hat{f}}_{1t+1|t}+{\beta }_2^{\prime }{\hat{f}}_{2t+1|t}+{\epsilon }_{1t+1},\end{array}\text{\hspace{1em}(17.33)}$$

$$\begin{array}{r}{y}_{t+1}-{\hat{f}}_{1t+1|t}={\beta }_2^{\prime }{\hat{f}}_{2t+1|t}+{\epsilon }_{2t+1},\end{array}\text{\hspace{1em}(17.34)}$$

$${\hat{e}}_{1t+1|t}={\beta }^{\prime }({\hat{e}}_{1t+1|t}-{\iota }_{m-1}{\hat{e}}_{2t+1|t})+{\epsilon }_{3t+1},\text{\hspace{1em}(17.35)}$$

respectively, where ${\hat{f}}_{2t+1|t}$ and ${\hat{e}}_{2t+1|t}$ are $((m-1)\times 1)$ vectors of the forecasts and forecast errors, respectively, ${\iota }_{m-1}$ is an $((m-1)\times 1)$ vector of 1s and $\beta$ or ${\beta }_2$ are $((m-1)\times 1)$ vectors of regression coefficients. The null hypotheses $H_0:\beta =0$ or $H_0:{\beta }_2=0$ yield tests that the first forecast encompasses the remaining $m-1$ forecasts.

17.5.2 Testing for Superior Predictive Ability

White (2000) studies the case with a vector of forecast errors or, more generally, losses that relate to these errors. However, he examines a very different question, namely how confident we can be that the best forecast, among a set of competing forecasts, is genuinely better than a pre-specified benchmark, given that the best forecast is selected from a potentially large set of models. That the best model has been drawn from a potentially large pool can be very important when assessing its performance: in any given sample, a forecast model may produce a smaller average loss even though in expectation (i.e., across all samples we could have seen) the model would not have been so good. A search across multiple forecast models may result in the recovery of a genuinely good model, but it may also uncover a bad model that just happens to perform well in a given sample. More broadly, this is the issue of skill versus luck, i.e., of having a genuinely superior model versus having an inferior model that just happens to perform well in a given sample.

The Reality Check setup considered by White (2000) closely mimics how many economists go about model building. Many different specifications are typically examined on a given data set. The tests used in model comparisons often ignore the specification search that may have preceded the selection of the prediction models although such models could be survivors from previous tests.

White uses a recursive setup for construction of the forecasts and considers the joint distribution of the performance of the individual forecasting models. With m models, there are m sample estimates of the out-of-sample (average) losses. Let $d_{kt+1|t}({\hat{\beta }}_t)=L(f_{0t+1|t}({\hat{\beta }}_{0t}),y_{t+1})-L(f_{kt+1|t}({\hat{\beta }}_{kt}),y_{t+1})$ be the loss difference in period $t+1$ for the benchmark model (numbered as model 0) measured relative to model $k,$ so that we obtain a zero value if the average loss of model k equals that of the benchmark. Negative values suggest higher loss of model k relative to the benchmark, while positive values indicate that model k beats the benchmark. Moreover, let ${\bar{d}}_k$ denote the sample average of $d_{kt+1|t}$ for $t=T_R,\dots ,T-1$ , and let $\bar{d}=({\bar{d}}_1,\dots ,{\bar{d}}_m{)}^{\prime }$ be an $(m\times 1)$ vector of sample averages computed over some (out-of-sample) evaluation period with $T_p$ observations. Finally, let ${\beta }_k^{\ast }=plim({\hat{\beta }}_{kT})$ be the pseudo-true value for ${\beta }_k$ , given the estimation method. The null hypothesis tested by the Reality Check is that the benchmark model is not inferior to any of the m alternatives:

$$H_0:\underset{k=1,\dots ,m}{\max }E\left[d_{kt+1|t}({\beta }_k^{\ast })\right]\le 0,\text{\hspace{1em}(17.36)}$$

whereas the alternative is that at least one model produces lower expected loss than the benchmark:

$$H_1:\underset{k=1,\dots ,m}{\max }E[d_{kt+1|t}({\beta }^{\ast })]>0.\text{\hspace{1em}(17.37)}$$

If all models perform as well as the benchmark, the $\bar{d}$ vector has zero mean and so the maximum of the vector also has zero mean. By examining the maximum of the vector of losses, the hypothesis reflects the practice that researchers are likely to search for the best model among a large set of competing specifications. The alternative hypothesis is that the best model outperforms the benchmark model, i.e., that there exists a superior model.

Under the assumption that $T_{♭}^{1/2}(\bar{d}-E({\bar{d}}^{\ast }))\Rightarrow \mathrm{N}(0,\Omega )$ as $T_p\to \infty$ , where $\Rightarrow$ denotes convergence in distribution and $d_{t+1}^{\ast }=L(f_{0t+1|t}({\beta }_0^{\ast }),y_{t+1})-$ $L(f_{t+1\mid t}({\beta }^{\ast }),y_{t+1})$ with ${\beta }^{\ast }=plim({\hat{\beta }}_T)$ as $T\to \infty$ , White (2000) establishes conditions under which

$$\underset{k=1,\dots ,m}{\max }{T}_{\hat{P}}^{1/2}({\bar{d}}_k-E[{\bar{d}}_k^{\ast }])⟹\underset{k=1,\dots ,m}{\max }\{U_k\},\text{\hspace{1em}(17.38)}$$

where U is an $(m\times 1)$ vector whose components $U_k$ are distributed as N(0, ). White’s high-level assumption is similar to the assumptions in West (1996) and so rules out a number of interesting cases, including comparisons of nested models.11

Even if we can characterize the limiting distribution of the vector of sample averages, we cannot easily determine the distribution of the maximum of this vector due to the correlation between the models captured by the unknown covariance matrix . From a testing perspective,  is a nuisance parameter that we do not care about other than in so far as it affects inference about the predictive performance. White (2000) addresses this problem by making the high level assumption that the vector of losses is asymptotically normal and by developing a bootstrap approach for drawing the maximum from the distribution in (17.38) and computing the $p\cdot$ -value that the null is true.

To see how the bootstrap works, denote by ${\bar{d}}_k^b$ the average differential loss computed in the bth bootstrap, $b=1,\dots ,B$ and consider the following statistics:

$${\bar{d}}_{\max ,m}=\underset{k=1,\dots ,m}{\max }\{T_p^{1/2}{\bar{d}}_k\},\text{\hspace{1em}(17.39)}$$

$${\bar{d}}_{\max ,m}^b=\underset{k=1,\dots ,m}{\max }\{T_{\gamma }^{1/2}({\bar{d}}_k^b-{\bar{d}}_k)\},b=1,\dots ,B.\text{\hspace{1em}(17.40)}$$

We can then compare the best performance of any model from the actual data in (17.39) to the quantiles from the bootstrap in (17.40) to obtain White’s bootstrap Reality Check $p.$ -value for the null hypothesis in (17.36). By considering the sampling distribution of the maximum value of the relative loss, computed across the m models, we account for the effect of data mining over models.

White suggests using the stationary bootstrap of Politis and Romano (1994) to construct bootstrap samples. The stationary bootstrap resamples blocks of random length from the original data, letting the block length be drawn from a geometric distribution whose mean block length is $1/q$ . Large values of $q$ are appropriate for data with little (time-series) dependence, while smaller values of $q$ can appropriately be used for more strongly dependent data.

Draws from the stationary bootstrap proceed as follows. Define a set of the random time indexes ${\tau }_t,$ , which are numbers between $T_R$ and $T$ . Each bootstrap, b, then generates a sample estimate $\begin{array}{r}{\bar{L}}_k^b={\sum }_{t=T_R}^{T}L(f_{k{\tau }_t-1},y_{{\tau }_t})\end{array}$ , as follows.

1. Set $t=T_R+1$ . Draw ${\tau }_t={\tau }_{T_R}$ at random, independently, and uniformly from $\{T_R+1,\dots ,T\}$

2. Increase t by $1.\mathrm{If}t>T$ , stop. Otherwise, draw a standard uniform random variable, U , independently of all other random variables.

   a) If $U<q$ , draw ${\tau }_t$ at random, independently, and uniformly, from $\{T_R+1,\dots ,T\}$

   b) If $U\ge q$ , expand the block by setting ${\tau }_t={\tau }_{t-1}+1;\mathrm{if}{\tau }_t>T$ , reset ${\tau }_t=T_R+1$

3. Repeat step (2).

This bootstrap is simple to implement because it requires resampling only from the forecast errors (or, more broadly, losses), as opposed to resampling from the original data, $\{y_t,z_{t-1}\}$ , reestimating forecasting models, and generating sequences of forecasts and forecast errors. Hence, no estimation is involved.

Hansen (2005) notes that the bootstrap procedure in White (2000) means that, in practice, the assumption under the null hypothesis is that $E\left[d_k({\beta }^{\ast })\right]=0$ . He shows that if the maximum of $E(\bar{d})$ is negative, then $T_p^{1/2}{\max }_{k=1,\dots ,m}{\bar{d}}_k^p-\infty$ . Thus with probability 1 we get a degenerate distribution when the benchmark model is better than all other models. Hansen’s superior predictive ability (SPA) test modifies the Reality Check by normalizing and recentering the test statistic. First, he proposes to use a studentized test statistic,

$$\mathrm{SPA}=\max \left[\underset{k=1,\dots ,m}{\max }\frac{T_{\hat{P}}^{1/2}{\bar{d}}_k}{{\hat{s}}_k},0\right],\text{\hspace{1em}(17.41)}$$

where ${\hat{s}}_k^2$ is a consistent estimator of $s_k^2=\mathrm{var}(T_p^{1/2}{\bar{d}}_k)$ . Second, Hansen’s SPA test bases the null distribution on

$${\mathrm{N}}_m({\hat{\mu }}^c,\hat{\Omega }),\text{\hspace{1em}(17.42)}$$

$$\begin{array}{rl}\mathrm{where}& {\hat{\mu }}_k^c={\bar{d}}_{k}1\left(T_{\hat{P}}^{1/2}{\bar{d}}_k/{\hat{s}}_k\le -\sqrt{2\log \log (T_{\hat{P}})}\right),\end{array}$$

where 1(·) is an indicator function. By recentering the null distribution for poorly performing models, the distribution becomes sample dependent under the null. The test does not discard poor models (models for which the indicator function equals 1) but reduces their influence through the recentering, while maintaining the influence of the models for which ${\hat{\mu }}_k^c=0$

Hansen presents Monte Carlo simulations that illustrate potentially large power gains from applying these modifications to the Reality Check test statistic. How important these modifications are depends on the particular application at hand and the number of “poor” forecasting models under consideration. Even in the absence of obviously inferior models, it is generally a good idea for many financial and economic time series to use a studentized test as in (17.41).