4.2.3 Estimation Based on Penalized Loss Functions

4.2.3 Estimation Based on Penalized Loss Functions

The need to estimate the parameters of the forecasting model adds to the risk of the associated forecast. Under squared error loss, parameter estimation error usually adds a term of order $T^{-1}$ . Although this term is asymptotically irrelevant, in practice it can still be a significant addition to risk. This is particularly true in practical forecast problems with monthly, quarterly, or even annual data where the sample size is not all that large and estimation errors can be important.

An alternative to choosing $\hat{\mathbf{\beta }}$ to minimize the in-sample loss is to use a method that penalizes the in-sample average loss or constrains the dimension of the forecasting model. In this case we choose $\hat{\mathbf{\beta }}$ as the minimizer of the Lagrangian

$$\hat{\beta }=\arg \underset{\beta }{\min }{T}^{-1}\sum _{t=0}^{T-1}L(f(z_t,\beta ),y_{t+1})+\lambda k(\beta ),\text{\hspace{1em}(4.10)}$$

with $\lambda \ge 0$ . We can view (4.10) as a penalized loss function with a penalty term $\lambda k(\beta )$ Applying this method to squared error loss and linear models, we have

$$\hat{\beta }=\arg \underset{\beta }{\min }{T}^{-1}\sum _{t=0}^{T-1}(y_{t+1}-{\beta }^{\prime }{z}_t{)}^2+\lambda k(\beta ),$$

where variation in the choice of the function $k(\beta )$ results in different procedures. Ridge regression sets $\begin{array}{r}k(\beta )={\sum }_{i=1}^k{\beta }_i^2\end{array}$ , while the extended Ridge method uses $k(\beta )=$ ${\beta }^{\prime }A\beta$ for some matrix A. Ridge regression was originally introduced to deal with multicollinearity in linear regression. Even though multicollinearity is not a problem for forecasting, the Ridge estimator turns out to be a shrinkage estimator which can be useful in forecasting.

The Lasso (least absolute shrinkage and selection operator) method introduced by Tibshirani (1996) sets $\begin{array}{r}k(\beta )={\sum }_{i=1}^k\left|{\beta }_i\right|\end{array}$ and also yields a shrinkage estimator. This method is often used for variable selection (see chapter 6) and so is useful in cases with large sets of possible predictors, few of which have sufficient predictive power to be included in the model. Bai and Ng (2008) consider Lasso methods to predict US inflation, while Elliott, Gargano, and Timmermann (2013) provide an application of both the Lasso and Ridge methods to predictability of US stock market returns.

Another, more recent, approach is the elastic net estimator proposed by Zou and Hastie (2005). This uses a penalty term $\begin{array}{r}k(\beta )={\lambda }_1{\sum }_{i=1}^k|{\beta }_i|+{\lambda }_2{\sum }_{i=1}^k{\beta }_i^2\end{array}$ , for ${\lambda }_1,{\lambda }_2\ge 0$ which is a convex combination of the Lasso and Ridge penalty terms. By using such a combination of penalty terms, the hope is to avoid the tendency of Lasso to perform poorly when many predictors within a group have similar predictive power and Lasso tends to select an arbitrary predictor from the group. Bai and Ng (2008) use this approach to construct an inflation forecast from more than 100 variables.

A variation on these methods, due to Breiman (1995), is the Garrote, which chooses the parameters to solve

$$\hat{\mathbf{\beta }}=\arg \underset{\mathbf{\beta }}{\min }{\mathbf{T}}^{-1}\sum _{t=0}^{T-1}({\mathbf{y}}_{t+1}-\sum _{i=1}^k{w}_i{\hat{\mathbf{\beta }}}_i{z}_{it}{)}^2,$$

$$\mathrm{subject}\mathrm{to}{w}_i>0\mathrm{for}\mathrm{all}i,\sum _{i=1}^k{w}_i<K,$$

where ${\hat{\mathbf{\beta }}}_i$ are least squares estimates. This method has not, however, gained much use in the economic forecast literature.

Constrained optimization methods are examples of shrinkage estimators which shrink the coefficients of the unconstrained optimization methods to a particular point, usually 0. This can be seen in the following examples.

Example 4.2.6. Let $y_t\sim iid(\mu ,{\sigma }^2)$ and consider forecasting the mean of ${\gamma }_t$ . We can use the in-sample estimator with the lowest MSE subject to the constraint that ${\mu }^2\le K$ which prevents $\mu$ from becoming too large. This produces a penalized loss function of the form

$$\underset{\mu }{\min }\left(T^{-1}\sum _{t=1}^T(y_t-\mu {)}^2+\lambda {\mu }^2\right),\text{\hspace{1em}(4.11)}$$

where $\lambda \ge 0$ . The solution to (4.11) yields the estimator

$$\hat{\mu }=\left(\frac{T}{T+\lambda }\right)\bar{y},$$

so the estimator shrinks the sample mean towards 0.

A second example is the generalized Ridge estimator.

Example 4.2.7. Consider observations {Y, X}, where Y is $T\times 1$ and X is $T\times k.$ . Let $k(\beta )={\beta }^{\prime }A\beta$ , where A is such that $A+X^{\prime }X$ is positive definite. Then the solution to

$$\hat{\mathbf{\beta }}=\arg \underset{\mathbf{\beta }}{\min }\left[(Y-X\mathbf{\beta }{)}^{\prime }(Y-X\mathbf{\beta })+\lambda {\mathbf{\beta }}^{\prime }A\mathbf{\beta }\right]$$

is $\hat{\beta }=(X^{\prime }X+\lambda A{)}^{-1}{X}^{\prime }Y$ which is the generalized Ridge estimator; the simple Ridge estimator sets $A=I_k$