16.5 EVALUATION OF AGGREGATE VERSUS DISAGGREGATE FORECASTS

16.5 EVALUATION OF AGGREGATE VERSUS DISAGGREGATE FORECASTS

In many situations a multitude of forecasts emerges from individual forecasting models or from a survey. Forecasters then face the choice of either basing their models and inference on the individual forecasts or using some aggregate consensus measure of the forecasts. Often the only reason for restricting the analysis to the aggregate forecasts is that individual forecasts are unavailable since some agencies report only cross-sectional averages of forecasts.

To analyze this situation, suppose that data are available on individual forecasts, denoted $f_{i,t+h|t},$ for $i=1,\dots ,m$ . Individual forecast errors are denoted by $e_{i,t+h|t}$ and these are typically different across forecasters. Individual forecasts can be jointly examined in a panel regression model for $e_{it+h|t}=y_{it+h|t}-f_{it+h|t};$

$$e_{it+h|t}=z_{it}^{\prime }{\beta }_i+u_{it+h},\text{\hspace{1em}(16.21)}$$

where $i=1,\dots ,m$ and $t=1,\dots ,T-h$ . As a starting point, suppose the coefficients are different for each forecaster. Rationality tests under squared error loss involve the null hypothesis $H_0:{\beta }_i=0.$ , for $i=1,\dots ,m$ . A multitude of alternative hypotheses can be tested, reflecting various departures from rationality. For example, a subset of the forecasters might depart from rationality, in which case the alternative hypothesis is that $H_1:{\beta }_i\ne 0,i\in I$ , where I is the subset of irrational forecasters. If all forecasters differ from rationality in the same way, we have ${\beta }_i=\beta \ne 0$ Alternatively, we can consider testing rationality separately for each individual forecaster by studying the ${\beta }_i$ coefficients. Further, we can split tests of biasedness and orthogonality by running a fixed effects regression, where the fixed effects are individual effects.

The regression in (16.21) gives rise to a seemingly unrelated regression (SUR) problem because the error terms, $u_{it+h}.$ , are likely to be correlated due to the fact that $y_{t+h}$ is identical across individuals which induces cross-sectional dependence among the $u_{it+h}$ terms. The individual regressions can be stacked to form a system,

$$\begin{array}{r}{e}_{t+h|t}=\left(\begin{array}{c}{e}_{1t+h|t}\\ e_{2t+h|t}\\ ⋮\\ e_{mt+h|t}\end{array}\right)=\left(\begin{array}{cccc}{z}_{1t}& 0& \cdots & 0\\ 0& z_{2t}& 0& 0\\ ⋮& 0& \ddots & ⋮\\ 0& \cdots & 0& z_{mt}\end{array}\right)\left(\begin{array}{c}{\beta }_1\\ {\beta }_2\\ ⋮\\ {\beta }_m\end{array}\right)+\left(\begin{array}{c}{u}_{1t+h}\\ u_{2t+h}\\ ⋮\\ u_{mt+h}\end{array}\right)\\ =Z_t\beta +u_{t+h},\end{array}\text{\hspace{1em}(16.22)}$$

where $u_{t+h}=[u_{1t+h},\dots ,u_{mt+h}{]}^{\prime }$ . Under the assumption of no serial correlation in the forecast errors beyond that accounted for in $z_{it}.$ , the SUR estimator for $\beta$ is $\begin{array}{r}{\hat{\mathbf{\beta }}}_{\mathrm{SUR}}={\left({\sum }_t{Z}_t{\mathbf{\Omega }}_c{Z}_t^{\prime }\right)}^{-1}\left({\sum }_t{Z}_t{\mathbf{\Omega }}_c{\mathbf{e}}_{t+h}\right)\end{array}$ , where ${\Omega }_c=E\left[u_{t+h}{u}_{t+h}^{\prime }\right]$ . In practice, ${\Omega }_c$ can be estimated directly as the covariance matrix of the regression residuals, initially setting the variance to the identity matrix. Some authors have proposed more elaborate schemes based on assumptions on the form of the covariance matrix. For example, the regression residuals might have a common component due to variations in ${\mathbf{e}}_{t+h|t}$ that arise from variations in the common component, $y_{t+h}$ . This suggests modeling the residuals as $u_{it+h}=c_{t+h}+v_{it+h}$ and hence $E\left[u_{it+h}{u}_{jt+h}\right]=$ $E\left[c_{t+h}^2\right]+E\left[v_{it+h}{v}_{jt+h}\right]$ . The second term can be set to 0 for $i\ne j$ and ${\sigma }_i^2$ for $i=j$ for each $i=1,\dots ,m$ , so ${\Omega }_c$ takes the form

$${\Omega }_c={\sigma }_c^2{u}^{\prime }+\left(\begin{array}{cccc}{\sigma }_1^2& 0& \cdots & 0\\ 0& {\sigma }_2^2& 0& 0\\ ⋮& 0& \ddots & ⋮\\ 0& \cdots & 0& {\sigma }_m^2\end{array}\right),\text{\hspace{1em}(16.23)}$$

where ι is an $(m\times 1)$ vector of 1s. Estimation of this matrix can proceed by first obtaining the OLS residuals from each individual regression and then estimating the restricted parameters in (16.23) from the OLS residuals.

This analysis can be extended in yet another direction by stacking forecast errors from different horizons for each individual and each time period. This extension is examined in Davies and Lahiri (1995). Further assumptions can be made on the structure of the variance–covariance matrix and exploited to obtain better estimates.

In many cases, data on individual forecasts are not reported, and sample averages ${\bar{f}}_t=m^{-1}{\dot{\sum }}_{i=1}^m{f}_{it}$ are the only data available for forecast evaluation. Averaging cross-sectionally in (16.21), we obtain

$$\begin{array}{l}{\bar{e}}_{t+h\mid t}=\frac{1}{m}\sum _{i=1}^m{e}_{it+h\mid t}=\frac{1}{m}\sum _{i=1}^m{z}_{it}^{\prime }{\beta }_i+\frac{1}{m}\sum _{i=1}^m{u}_{it+h}\\ =\frac{1}{m}\sum _{i=1}^m{z}_{it}^{\prime }{\beta }_i+{\bar{u}}_{t+h}.\end{array}\text{\hspace{1em}(16.24)}$$

If there is heterogeneity across the ${\beta }_i$ coefficients, the weighted estimate in (16.24) will differ from the regression on the averaged data,

$${\bar{e}}_{t+h|t}=\left(\frac{1}{m}\sum _{i=1}^m{z}_{it}^{\prime }\right)\beta +{\tilde{u}}_{t+h}.\text{\hspace{1em}(16.25)}$$

Under the null of forecast rationality $({\beta }_1=\cdot \cdot \cdot ={\beta }_m=0)$ this does not cause any problem. However, unless all the deviations from rationality are in the same direction, it is unlikely that results from regressions (16.24) and (16.25) would be identical.

Interpreting results established for the average, as with the individual results, takes care. First, a failure to reject the null of forecast rationality clearly does not imply that each of the individual forecasts are rational. The averaged regression parameters in (16.24) and (16.25) will be consistent for a weighted average of the individual parameters, and hence this weighted average could be 0 even when its components are not. Researchers understand this and consider the average forecast to be an estimate of the consensus forecast and hence the test is really that the consensus forecast is rational, overcoming this issue.

One can also compare the aggregate and individual results. Rejecting the hypothesized forecast rationality for individual forecasts but not for the aggregate forecast might indicate a lack of power in the aggregated data, or simply suggest that the average coefficient is small even though individual coefficients can stray away from 0. Rejecting for the average forecast but not for individual forecasts could also be due to differences in the power of the tests. If there is a large amount of individual heterogeneity in the forecasts, so each of the $u_{it}$ terms vary a great deal, the power of the individual tests could be low. However, this variation could average out, so the power could be greater for tests based on the average data.