6.1 TRADE-OFFS IN MODEL SELECTION

6.1 TRADE-OFFS IN MODEL SELECTION

Model selection involves nontrivial issues because improvements in model fit could either reflect a model’s genuine ability to produce better forecasts or, alternatively, be due to the model’s tendency to “overfit,” i.e., provide a better in-sample fit without offering genuine improvements in predictive accuracy. This can be seen clearly in the case of linear regression as the next example shows.

Example 6.1.1 (Risk for the multivariate regression model). For the model in Example 3.2.3, the full sample estimate $\begin{array}{r}{\hat{\beta }}_T={\left({\sum }_{t=0}^{T-1}{x}_t{x}_t^{\prime }\right)}^{-1}\left({\sum }_{t=0}^{T-1}{x}_t{y}_{t+1}\right)\end{array}$ leads to a risk estimate

$$T^{-1}\sum _{t=0}^{T-1}(y_{t+1}-{\hat{\beta }}_T^{\prime }{x}_t{)}^2=T^{-1}\sum _{t=0}^{T-1}{\epsilon }_{t+1}^2-({\hat{\beta }}_T-\beta {)}^{\prime }\left(T^{-1}\sum _{t=0}^{T-1}{x}_t{x}_t^{\prime }\right)({\hat{\beta }}_T-\beta ).$$

The first term has mean ${\sigma }^2{}_{;}$ ; for the second term (estimation-related risk) we have

\begin{array} { r l } & { E [ ( \widehat { \beta } _ { T} - \beta ) ^ { \prime} \Bigg ( T ^ { - 1} \underset { t = 0 } { \overset { T - 1 } { \sum } } x _ { t} x _ { t} ^ { \prime} \Bigg ) ( \widehat { \beta } _ {T} - \beta ) ] } \\ & { \quad = T ^ { - 1} E [ ( \underset { t = 0 } { \overset { T ^ { - 1} } { \sum ^ { n} } } x _ { t} \varepsilon _ {t + 1} ) ^ { \prime} ( \underset { t = 0 } { \overset { T ^ { - 1} } { \sum } } x _ {t} x _ {t} ^ { \prime} ) ^ { - 1} ( \underset { t = 1 } { \overset { T ^ { - 1} } { \sum } } x _ {t} \varepsilon _ {t + 1} ) ] } \\ & { \quad = T ^ { - 1} E [ ( \underset { t = 0 } { \overset { T ^ { - 1} } { \sum ^ { n} } } x _ {t} x _ {t} ^ { \prime} ) ^ { - 1} E [ ( \underset { t = 1 } { \overset { T ^ { - 1} } { \sum ^ { n} } } x _ {t} \varepsilon _{t + 1} ) ( \underset { t = 1 } { \overset { T ^ { - 1} } { \sum ^ { n} } } x _ {t} \varepsilon _ {t + 1} ) ^ { \prime} \bigg| x _ {1}, \ldots , x _ {T} ] ] } \\ & { \quad = T ^ { - 1} E [ ( \underset { t = 0 } { \overset { T ^ { - 1} } { \sum ^ { n} } } x _ {t} x _ {t} ^ { \prime} ) ^ { - 1} \sigma ^ { 2} ( \underset { t = 0 } { \overset { T ^ { - 1} } { \sum ^ { n} } } x _ {t} x _ {t} ^ { \prime} ) ] } \\ & \quad = \pi ^ { 2} T ^ { - 1} \delta [ ( \underset { t = 0 } { \overset {T ^ { - 1} } { \sum ^ { n} } } x _ {t} x _ {t} ^ { \prime} ) ^ { - 1} \sigma ^ { 2} ( \underset { t = 0 } \overset \end{array}

So the risk here is ${\sigma }^2(1-T^{-1}k)$ . Instead of increasing the expected loss by this term as in (3.19), the in-sample loss estimate subtracts this term from ${\sigma }^2$ . Hence, the downward bias in the expected loss estimate can be large in small samples and tends to be greater for more complex models that include more regressors (large $K)$ . Even $so,$ estimation effects disappear asymptotically.

In this example the inclusion of additional variables may decrease ${\sigma }^2$ , in which case the additional variables help to improve the model. Alternatively, they might reduce only the second (variance) component—the effect is negative because of the added dimensions over which the in-sample risk is minimized as a result of the inclusion of extra variables. Improvement in the in-sample fit does not tell us which of these effects is occurring.

Next, consider a situation with two models, $M_1=f_1(z_t,{\beta }_1)$ and $M_2=f_2(z_t,{\beta }_2)$ ， where the second model nests the first.3 For each model the coefficient estimates are selected such that, for the sample $t=0,\dots ,T$

$${\hat{\beta }}_i=\arg \underset{{\beta }_i}{\min }{T}^{-1}\sum _{t=0}^{T-1}L(f_i(z_t,{\beta }_i),y_{t+1}).\text{\hspace{1em}(6.1)}$$

Because $f_2(z_t,{\beta }_2)$ nests $f_1(z_t,{\beta }_1)$ , it follows from the properties of minimization that, for a given data sample,

$$T^{-1}\sum _{t=0}^{T-1}L(f_2(z_t,{\hat{\beta }}_2),y_{t+1})\le T^{-1}\sum _{t=0}^{T-1}L(f_1(z_t,{\hat{\beta }}_1),y_{t+1}).\text{\hspace{1em}(6.2)}$$

$$\begin{array}{rl}& M_1:y_{t+1}={\beta }_1^{\prime }{x}_{1t}+{\epsilon }_{1t+1},\\ & \\ & M_2:y_{t+1}={\beta }_1^{\prime }{x}_{1t}+{\beta }_2^{\prime }{x}_{2t}+{\epsilon }_{2t+1},\end{array}$$

Hence the larger model $\left(M_2\right)$ always provides at least as good a fit as the smaller model $\left(M_1\right)$ and in most cases will provide a strictly better in-sample fit. The key here is that (6.2) compares the in-sample performance of the two models so that the same sample $t=0,\dots ,T$ is used for parameter estimation and model evaluation. The well-known result that the regression $R^2$ is (weakly) increasing as additional variables are added to a set of existing regressors is a special case of this result. Indeed, linear regression models with $T$ predictors yield a perfect fit in a sample with $T$ observations even when the predictors are independent of the outcome, assuming that the predictors are not perfectly collinear. Thus the in-sample ranking in (6.2) is always true even if in fact the population expected loss under the first (small) model is smaller than the expected loss under the second (large) model, when evaluated at the respective probability limits of the parameters, ${\beta }_i^{\ast }$ . Hence, (6.2) could hold simultaneously with

$$\mathcal{E}\left[\mathcal{L}(f_1(z_t,{\beta }_1^{\ast }),y_{t+1})\right]<\mathcal{E}\left[\mathcal{L}(f_2(z_t,{\beta }_2^{\ast }),y_{t+1})\right].\text{\hspace{1em}(6.3)}$$

A superior in-sample fit does not by itself lead to the conclusion that a particular forecasting model necessarily produces better forecasts. This special result for nested models gives intuition for the general case: more highly parameterized (or complex) models often perform well in comparisons of in-sample fit even when they produce poor forecasts compared with smaller models that, when applied to new data, may generate a smaller expected loss.

Selection of forecasting models involves a trade-off between the additional estimation error associated with larger models versus the potential for greater model specification error that comes with using more parsimonious models that exclude relevant predictors. This trade-off is very clear for the linear regression model under MSE loss. We saw in Example 3.2.6 that the estimation error effect can be broken into two components: a variance term disappearing at rate $T$ and a bias term which may be of the same order if the coefficients of the omitted variables are close to 0. Alternatively, the bias term could be of a higher order if the model is grossly misspecified. Using the notation and definitions of section 3.2.1, we can write this estimation error term as

$$R_{2k}(\beta )=k{\sigma }^2+T[(\beta -E({\hat{\beta }}_k|Z_{-kT}){)}^{\prime }E[x_T{x}_T^{\prime }|Z_{-kT}](\beta -E({\hat{\beta }}_k|Z_{-kT}))],$$

where k is the number of parameters estimated for the kth model.

Let the model be $y_{t+1}=x_t^{\prime }\beta +{\epsilon }_{t+1}$ , where some of the elements of $\beta$ are ${}^{0,}$ ${\mathbf{x}}_t^{\prime }{\mathbf{x}}_t^{\prime }$ is of full rank, and dim $(\beta )\ll T$ . Different submodels, $y_{t+1}=x_t^{\prime }{\beta }_k+{\epsilon }_{kt+1},$ arise by setting elements of $\beta$ to 0 by omitting the corresponding covariate(s) from the forecasting model. For any of these submodels, the estimation-related risk term, $R_{2k}(\beta )$ , obtained by omitting the common first-order risk component, ${\sigma }^2=E\left[{\epsilon }^{\prime }\epsilon /T\right]$ , is

$$R_{2k}(\beta )=k{\sigma }^2+(\beta -{\beta }_k{)}^{\prime }\sum _{t=0}^{T-1}{x}_t{x}_t^{\prime }(\beta -{\beta }_k).\text{\hspace{1em}(6.4)}$$

The first term in (6.4) reflects estimation error and is easy to estimate. This increases, the larger the dimension of the model, as measured by $n_k$ . The second term is due to model specification error and is difficult to obtain as it depends on the true model through $\beta$ which is never known in practice—if it were, there would be no model selection problem. This term will generally decrease as additional variables are included in the model.

The trade-off identified by (6.4) depends on both the true model and on the number of predictors included in the largest model, K . When the dimension of $x_t$ is small, the estimation error term cannot get too large. Many of the traditional model selection methods were developed for this situation. With large-dimensional models (big $n_k)$ , the estimation error could potentially become very large. For example, socalled kitchen sink regressions that include all possible predictors often produce very poor forecasts, as $n_k{\sigma }^2$ is large even though they protect against the misspecification error reflected in the second term in (6.4).

Sparse models are models for which the number of true nonzero elements of $\beta$ is small relative to the dimension of $x_t$ . The misspecification term for such models is likely to be small relative to the estimation error associated with large models and so methods that remove many covariates are preferable in this situation. Conversely, in situations with many covariates where many elements of $\beta$ are expected to be nonzero, i.e., dim $(\beta )<T$ but growing with $T_{:}$ , the misspecification term might be large for all parsimonious models. In this case alternative methods that consider many covariates might be more useful although model selection methods are also available for this situation. Chapter 10 on factor models is particular relevant for this case.

Approaches to model section are affected by the size of the set of models to be considered. The number of potential models can quickly become very large, large enough to be $\mathbf{a}$ difficulty even with modern computing power. For example consider different forecast models corresponding to different linear regressions involving subsets of variables selected from the list $\{x_1,\dots ,x_K\}$ . To conduct a comprehensive search over all possible linear models with K potential predictor variables, $\{x_1,\dots ,x_K\}$ means considering $2^K$ possible model specifications. For example, with two possible predictors, $\{x_1,x_2\}$ , there are four possible constellations, {0, 0}, {1, 0}, {0, 1}, and {1, 1}, where a 0 indicates that the variable is excluded from the model, while a 1 indicates that it is included. Since the parameters of each model have to be estimated, this can limit comprehensive specification searches to relatively small sets of predictor variables. For example, with K = 20, the number of possible models exceeds one million. For this reason, a global search across model specifications is employed only for relatively low-dimensional model sets.