14.4 MODEL COMBINATION

14.4 MODEL COMBINATION

When the data underlying the individual forecasts are observed, we might consider constructing forecasts from many different models and averaging over the resulting forecasts. With the exception of the first step of constructing the individual forecasts, this setup is identical to the forecast combination problem discussed so far. Indeed, if we ignore how the forecasts were constructed, all the methods discussed in section 14.1 are available for combination of models. However, in model combination it is explicitly understood that the models depend on similar data and that such data are available for constructing the model weights. Treating the individual forecasts as if they are fixed therefore does not make sense when considering their sampling properties. Moreover, ignoring information about how the model forecasts were constructed might underutilize the available data. Model combination seeks to address this issue.

For linear combinations the model average forecast is

$$f^c=\sum _{i=1}^m{\omega }_i{f}_i,$$

where the individual forecasts $f_i$ depend on some underlying data, $z_t$ . We typically assume that the weights sum to 1 and often restrict them to be nonnegative.

One approach to constructing model averages uses weights that depend on the AIC. Hjort and Claeskens (2003) show that the resulting weights can be approximated by

$${\omega }_i\approx \frac{\exp (-0.5{\mathrm{AIC}}_i)}{{\sum }_{i=1}^m\exp (-0.5{\mathrm{AIC}}_i)},\text{\hspace{1em}(14.24)}$$

so that models with the smallest value of AIC, and thus the best fit relative to the penalty factor, are assigned the greatest weight in the combination. This method does not require that the models in the combination are linear regressions; see chapter 6 on the generality of the AIC criterion. Basing the weights on the AIC means that the underlying data matter both through the fit of the forecasting models and through differences in the penalty applied to forecasting models of different dimensions. Other information criteria such as the BIC can also be used as we discuss below.

Hansen (2007, 2008a) suggest combining linear regression models with weights constructed to minimize the Mallows $C_p$ criterion. His setup considers m covariates that have a natural ordering from 1 to $m{.}^9$ We can then construct sets of covariates $z_i$ that contain all of the z-variables from 1 to $i\le m$ , so $z_i$ is $i\times 1$ . This method requires a nested setup for the models and ignores the possibility that variables 1 and 3 are included but variable 2 is not. Moreover, the method is not invariant to the ordering of the variables. Different orderings, even when based on the same method, might lead to different results.

The approach works as follows. For each model, project y onto $z_i,$ yielding forecast errors $e_i$ . Placing these forecast errors in a $(T\times m)$ matrix, $e,$ the weights are selected as

$${\hat{\omega }}_H=\arg \underset{\omega }{\min }{\omega }^{\prime }{e}^{\prime }e\omega +2{\hat{\sigma }}^2{k}^{\prime }\omega ,$$

$$\mathrm{subject}\mathrm{to}{\omega }^{\prime }{\iota }_m=1,{\omega }_i\ge 0\mathrm{for}i=1,\dots ,m.$$

Here ω are the weights, ${\hat{\sigma }}^2$ is an estimate of the variance of the residual for the true model—in Hansen (2007) this is a model with an infinite number of regressors— and k is an $(m\times 1)$ vector with elements $k_i$ equal to the number of regressors in the i th regression. In practice, Hansen suggests estimating ${\hat{\sigma }}^2$ from a model that uses all m regressors. The objective function used is similar to that of Bates and Granger (1969) evaluated at the estimated variance–covariance matrix, with the addition of a term that penalizes the models differently for having different sizes through the vector k. The method is developed for outcomes and regressors that are i.i.d., but the method can be employed more generally although without the properties established in Hansen (2007).