14.2.3 Forecast Combination Puzzle

14.2.3 Forecast Combination Puzzle

Empirical studies often find that simple equal-weighted forecast combinations perform well compared with more sophisticated combination schemes that rely on estimated combination weights. This finding is known as the forecast combination puzzle. To quote Smith and Wallis (2009), “Why is it that, in comparisons of combinations of point forecasts based on mean-squared forecast errors. . . , a simple average with equal weights, often outperforms more complicated weighting schemes.” Since the optimal combination scheme of Bates and Granger (1969) amounts to restricted least squares under MSE loss, it is indeed puzzling that such a venerated method tends to provide combination weights that are inferior to simply averaging the forecasts.

Although the least squares combination weights of Bates and Granger (1969) are optimal in population, estimation errors from constructing the weights from data mean that the optimal loss is not achieved in finite samples. Consider combinations of unbiased forecasts under MSE loss. The population loss from optimal combination weights can be expected to be smaller than the loss from using equal weights, i.e., $({\iota }_m^{\prime }{\Sigma }_e^{-1}{\iota }_m{)}^{-1}<m^{-2}({\iota }^{\prime }{\Sigma }_e\iota )$ . However, it is not necessarily the case that $({\iota }_m^{\prime }{\hat{\Sigma }}_e{\iota }_m{)}^{-1}<$ $m^{-2}({\iota }^{\prime }{\Sigma }_e\iota )$ . Errors introduced by estimation of the combination weights could overwhelm any gains from setting the weights to their optimal values rather than using equal weights. Explanations of the forecast combination puzzle based on estimation error must therefore show that (a) estimation error is large and/or (b) the gains from setting the combination weights to their optimal values are small relative to using equal weights.

Consider showing that estimation error is large. In sufficiently large samples OLS estimation error should be of the order $(m-1)/T$ . Unless equal weights are close to being optimal, estimation error is unlikely to be the full story, however, at least when $m/T$ is small. In many situations, $m/T$ will be quite small, although in cases with survey data, m can be large relative to T and so estimation error could be important.

Of course if the optimal population weights are sufficiently close to equal weights, estimation error could still dominate the small gains from using estimated optimal weights; see Smith and Wallis (2009) for some Monte Carlo evidence.

In practice we might expect that poor forecasts get weeded out and so the forecasts included in most combinations have similar forecast error variances, leading to a nearly constant diagonal of ${\Sigma }_e$ . In this situation, large differences across correlations would be required to cause deviations from equal weights. Typically the greatest gains from forecast combination arise when correlations are negative since forecasts can then be combined in a way that offsets individual errors. However, a large and unpredictable component of the outcome that is outside and therefore common across all the forecasts, pushes correlations towards positive numbers. Small differences between forecast error variances and positive correlations limit the possibility of large gains from using optimal combination weights rather than equal weights.

Explanations that aim to solve the forecast combination puzzle by means of large estimation errors instead require model misspecification or more complicated datagenerating processes than is assumed when estimating the combination weights. One type of model misspecification occurs when the data-generating process changes over time in a way that affects the optimal weights. In this situation, full-sample estimates of the optimal weights will be biased with a bias that grows larger, the bigger and more recent the change in the data-generating process. For example, consider a process for which the constant term shifts discretely to a higher value at one point in the sample. If we use data from before the parameter shift to construct the forecast, the estimated parameters will be a weighted average of the current parameters and the parameters prior to the shift. Equal-weighted forecasts require no estimation and so the shift has no effect on the simple averaged forecast; see Hendry and Clements (2004) for Monte Carlo evidence on the importance of this effect.