6.6 HARD VERSUS SOFT THRESHOLDS: BAGGING

6.6 HARD VERSUS SOFT THRESHOLDS: BAGGING

By design, model selection provides hard thresholds for constructing a model in the sense that variables are either included ${(}^{\ast }{\mathrm{in}}^{†})$ or excluded (“out”). This makes sense if one is absolutely certain that the ${\beta }^{\circ }s$ are 0 or far from 0 so that each variable should be in or out of the model. Better forecasting models might be obtained if variables whose inclusion is uncertain are allowed to have some limited influence. One such approach with this property is shrinkage estimation, which is discussed in chapter 4. A second approach is to combine forecasting models rather than select a single specification. In this case the models can be combined in a way that assigns large weight to models that include variables known to be valuable and assigns lower weight to models using variables whose inclusion is more uncertain. In this sense all variables can be included with weights reflecting how useful we find them to be. Model combination is covered in chapter 14. A third approach is Bagging, which can be viewed as an extension of many of the methods discussed in this chapter.

Bagging, short for bootstrap aggregating, is a general method for estimating a predictor. Introduced by Breiman (1996), the method is included here since the main application of Bagging in the forecasting literature has been to variable selection in prediction models. The method modifies the hard thresholds traditionally used in model selection—corresponding to 0–1 variable selection indicators—to soft thresholds which allow the variable selection indicators to take on a continuum of values. The approach involves three steps for obtaining the estimators $\hat{\beta }$ from the data $\{y_{t+1},z_t{\widehat{\}}}_{t=1}^{\hat{T}-1}$ :

1. Draw resamples of the data $\{y_{t+1},x_t{\}}_{t=1}^{T-1}$ to obtain a bootstrap sample $\{y_{t+1}^b,x_t^b{\}}_{t=1}^{T-1}$ of the same length as the original sample.

2. Compute an estimator of ${\beta }_j,{\hat{\beta }}_j^b$ , based on the bth resample of the data, $b=1,\dots ,B$

3. Compute the final estimator, ${\bar{\mathbf{\beta }}}_j$ , as either the mean or some other function of the resampled estimates $\{{\hat{\beta }}_j^b{\}}_{b=1}^B$

Steps (1) and (3) can be tailored to reflect the presumed data-generating process as well as the targeted properties of the final estimator. For example, in Breiman’s original formulation, the data are i.i.d. and hence an i.i.d. (empirical) bootstrap is chosen. If the data are dependent, bootstrap methods that retain the dependence are required. For example, Inoue and Kilian (2008) use a block bootstrap that allows for dependent data. Other methods could also be employed, resulting in different estimators and hence different risk functions.

Example 6.6.1 (Bagging used in model selection). Bagging could be used to determine inclusion of variables to get a predictor:

$$f_{t+1|t}=B^{-1}\sum _{b=1}^B\left({\hat{\beta }}_{0k}^b+\sum _{k=1}^K{\hat{\beta }}_k^{b}1\left(|t_{{\hat{\beta }}_k}^b|>\bar{t}\right)\right)x_{kt},\text{\hspace{1em}(6.32)}$$

where the indicator variable from the bth bootstrap, $\nabla .(|t_{{\hat{\beta }}_k}^b|>\bar{t})$ , equals 1 if the absolute value of the kth t-statistic in the regression model, $t_{{\hat{\beta }}_k}^b$ , exceeds the threshold, t.¯

Model selection based on hard thresholding is one area where Bagging can be expected to lead to better results; see, e.g., Bühlmann and Yu (2003) in the context of i.i.d. data. Although hard thresholds are used to select a model in any given resample, Bagging translates this into a soft threshold as a result of choosing different models across different resamples in the second step and then averaging across these choices in the third step.

6.6.1 Generalized Shrinkage Methods

Stock and Watson (2012) show that many of the above methods can be viewed as generalized shrinkage methods in the context of a linear regression model with $n_k$ predictors:

$$Y_{t+1}={\beta }^{\prime }{X}_t+{\epsilon }_{t+1},t=0,\dots ,T-1,\text{\hspace{1em}(6.33)}$$

where the regressors $X_t$ have been orthonormalized, i.e., normalized to have zero mean and unit variance and be uncorrelated so that $X_t^{\prime }{X}_t/T=I_{n_k}$ . Stock and Watson show that, asymptotically and under assumptions that do not require the

predictors to be strictly exogenous, forecasts based on pre-tests, Bagging, and normal Bayes methods can be written in a generalized shrinkage form

$$f_{T+1|T}=\sum _{i=1}^{n_k}\psi (\kappa t_i){\hat{\beta }}_i{X}_{iT}+o_{\hat{P}}(1),$$

where $\begin{array}{r}{\hat{\beta }}_i=T^{-1}{\sum }_{t=0}^T{X}_{it-1}{Y}_t\end{array}$ is the least squares estimator of ${\beta }_i,t_i=\sqrt{T}{\hat{\beta }}_i/{\hat{\sigma }}_e$ and $\begin{array}{r}{\hat{\sigma }}_e^2={\sum }_{t=0}^{T-1}(Y_{t+1}-{\hat{\beta }}^{\prime }{X}_t{)}^2/(T-n_k)\end{array}$ . The scalar κ depends on the forecasting method. For pre-testing methods, the shrinkage function takes the form $\psi (t_i)=$ $1(|\kappa t_i|>c)$ , where c is a threshold. For example, the AIC asymptotically sets $c=\sqrt{2}.$