6.5 LASSO MODEL SELECTION

6.5 LASSO MODEL SELECTION

The Lasso—least absolute shrinkage and selection operator—introduced by Tibshirani (1996) as a shrinkage estimator for least squares regression and briefly discussed in chapter 4, turns out also to have useful model selection properties. The Lasso method estimates linear regression coefficients by minimizing the sum of least squares residuals subject to a penalty function,

$$T^{-1}\sum _{t=0}^{T-1}(y_{t+1}-{\beta }^{\prime }{x}_t{)}^2+\lambda \sum _{i=1}^{n_k}|{\beta }_i|.\text{\hspace{1em}(6.25)}$$

Here $\lambda >0$ is a tuning parameter that captures the (relative) weight on the penalty function and has the effect of shrinking the parameter estimates towards 0. Since all nonzero parameter values get penalized by the same amount—λ is a scalar as opposed to a vector—it is assumed that all variances of the predictors, $x_t,$ have been scaled to equal 1. There is generally no closed-form solution for minimizing the object in (6.25) and so computational methods are required.

The use of a squared objective function penalized by the absolute value of the parameters means that there are many corner solutions for which the parameter estimates ${\hat{\beta }}_i$ are 0. This motivates thinking of Lasso as a model selection tool: simply choose the model that remains after removing the x-variables that correspond to ${\beta }^{\prime \prime }s$ whose estimates are equal to 0.

To see why corner solutions arise under the Lasso approach, we follow Tibshirani (1996) and consider the linear regression model in matrix form $Y=X\beta +\epsilon$ and suppose that the regressors have been orthonormalized so $X^{\prime }X=I$ and ${\hat{\beta }}^{\mathrm{ols}}=X^{\prime }Y.$ The least squares forecasts are $\hat{Y}=X{\hat{\beta }}^{\mathrm{ols}}$ and hence (in matrix form)

$${\mathcal{T}}^{-1}(Y-X\beta {)}^{\prime }(Y-X\beta )+\lambda \sum _{i=1}^{n_k}|{\beta }_i|\text{\hspace{1em}(6.26)}$$

$$=T^{-1}(Y-\hat{Y}+\hat{Y}-X\beta {)}^{\prime }(Y-\hat{Y}+\hat{Y}-X\beta )+\lambda \sum _{i=1}^{n_k}|{\beta }_i|\text{\hspace{1em}(6.27)}$$

$$=T^{-1}(Y-\hat{Y}{)}^{\prime }(Y-\hat{Y})+(\hat{Y}-X\beta {)}^{\prime }(\hat{Y}-X\beta )+\lambda \sum _{i=1}^{n_k}|{\beta }_i|,\text{\hspace{1em}(6.28)}$$

where the cross product drops out since $X^{\prime }X=I$ and

$$(\hat{Y}-X\beta {)}^{\prime }(Y-\hat{Y})=({\hat{\beta }}^{\mathrm{ols}}-\beta {)}^{\prime }{X}^{\prime }(Y-\hat{Y})=({\hat{\beta }}^{\mathrm{ols}}-\beta {)}^{\prime }(X^{\prime }Y-X^{\prime }X{X}^{\prime }Y)=0.$$

The first term in (6.28) does not depend on the choice of $\beta ,$ and the second term is

$$\begin{array}{l}(\hat{Y}-X\beta {)}^{\prime }(\hat{Y}-X\beta )=(X{\hat{\beta }}^{\mathrm{ols}}-X\beta {)}^{\prime }(X{\hat{\beta }}^{\mathrm{ols}}-X\beta )\\ =({\hat{\beta }}^{\mathrm{ols}}-\beta {)}^{\prime }({\hat{\beta }}^{\mathrm{ols}}-\beta )\\ =\sum _{i=1}^{n_k}({\hat{\beta }}_i^{\mathrm{ols}}-{\beta }_i{)}^2.\end{array}$$

Minimizing (6.28) over $\beta$ is therefore the same as minimizing variable by variable, i.e., for each i find

$${\hat{\beta }}_{i,\mathrm{lasso}}=\arg \underset{{\mathbf{\beta }}_i}{\min }\left[({\hat{\beta }}_i^{\mathrm{ols}}-{\beta }_i{)}^2+\lambda |{\mathbf{\beta }}_i|\right].\text{\hspace{1em}(6.29)}$$

The first-order conditions to this problem give the result $-({\hat{\beta }}_i^{\mathrm{ols}}-{\beta }_i)+$ $\left(T\lambda /2\right)\mathrm{sign}({\beta }_i)=0$ , or, after rearranging, ${\beta }_i={\hat{\beta }}_i^{\mathrm{ols}}-(T\lambda /2)\mathrm{sign}({\beta }_i)$ . This implies that if $|{\hat{\beta }}_i^{\mathrm{ols}}|<T\lambda /2$ , then ${\beta }_i$ must be 0. To see this, suppose ${\beta }_i>0$ so that ${\beta }_i={\hat{\beta }}_i^{\mathrm{ols}}-T\lambda /2<0$ because we assumed that $|{\hat{\beta }}_i^{\mathrm{ols}}|<T\lambda /2$ . Hence ${\beta }_i$ cannot be positive. But if ${\dot{\beta }}_i<0,{\beta }_i={\hat{\beta }}_i^{\mathrm{ols}}+T\lambda /2>0$ through the same assumption, which also cannot hold.

Figure 6.1: Model selection under Lasso.

Conversely, if ${\hat{\beta }}_i^{\mathrm{ols}}>T\lambda /2$ , then ${\beta }_i={\hat{\beta }}_i^{\mathrm{ols}}-\left(T\lambda /2\right)\mathrm{sign}({\beta }_i)>0$ and if ${\hat{\beta }}_i^{\mathrm{ols}}<$ $-T\lambda /2$ then ${\beta }_i={\hat{\beta }}_i^{\mathrm{ols}}-\mathrm{sign}({\beta }_i)T\lambda /2<0,$ . These results can be summarized as follows:

$${\hat{\beta }}_{\mathrm{lasso},i}=\mathrm{sign}({\hat{\beta }}_i^{\mathrm{ols}})(|{\hat{\beta }}_i^{\mathrm{ols}}|-T\lambda /2)1(|{\hat{\beta }}_i^{\mathrm{ols}}|>T\lambda /2).$$

Thus, for $|{\hat{\beta }}_i^{\mathrm{ols}}|<T\lambda /2$ the coefficient estimate is set to 0. The larger is λ, the stronger the penalty and the more coefficients will be set to 0. See Tibshirani (1996) for further analysis and details.

We can also visualize the effect by considering the problem in two dimensions (i.e., only two covariates). To see this, consider the model

$$y_i={\beta }_1{x}_{1i}+{\beta }_2{x}_{2i}+{\epsilon }_i,$$

so we have a bivariate problem and can draw it in $({\beta }_1,{\beta }_2)$ space. Assuming quadratic objectives, the objective function has contours $(y_i-{\beta }_1{x}_{1i}-{\beta }_2{x}_{2i}{)}^2=z$ that are ellipses in the $({\beta }_1,{\beta }_2)$ space. Figure 6.1 visualizes this effect.

The LASSO constraints are now of the form $|{\beta }_1|+|{\beta }_2|\le s$ , i.e.,

$$\begin{array}{r}{\beta }_1+{\beta }_2\le s,{\beta }_1>0,{\beta }_2>0,\\ \\ -{\beta }_1+{\beta }_2\le s,{\beta }_1<0,{\beta }_2>0,\\ {\beta }_1-{\beta }_2\le s,{\beta }_1>0,{\beta }_2<0,\\ \\ -{\beta }_1-{\beta }_2\le s,{\beta }_1<0,{\beta }_2<0,\end{array}$$

which creates a “diamond” around 0 with coordinates at $(0,s),(0,-s),(s,0)$ ， and $(-s,0)$ . Figure 6.1 shows this graphically, where the diamond is the preceding set of constraints and the ellipse is drawn for that value of z that has the constraints binding. The chance that the ellipse intersects the constraints at a corner is high—as pictured in figure 6.1, we have ${\beta }_1=0$ at the constraint.

In practical applications a choice of λ must be made. Choosing $\lambda =0$ delivers least squares estimates which means that the largest model almost always gets chosen. The larger the choice of $\lambda$ , the more likely it is that individual ${\beta }^{\prime \prime }\mathsf{s}$ are set to 0 and that a more parsimonious model is chosen. Unsurprisingly, the actual choice of $\lambda -$ or ${\lambda }_T$ , since it is generally recommended to let it be a function of the sample size, T—impacts the properties of the method. Typical results provide rate conditions on ${\lambda }_T$ rather than pinning down a specific value and are developed in fixed-regressor settings with i.i.d. data. For example, Belloni and Chernozhukov (2011) suggest setting ${\lambda }_T=2c\sigma \sqrt{T}{\Phi }^{-1}(1-\alpha /2n_k)$ for α near (but below) 1 and $c>1$ , where (·) is the normal c.d.f., and ${\sigma }^2$ is the variance of the residuals from the true model.

The main difficulty in estimating the Lasso coefficients is that the objective function is nondifferentiable, so standard hill-climbing optimizers will not be useful for solving the minimization problem in (6.25). However, in recent years this estimation problem has been a subject of intense research and there now exist many methods that work well even when the dimension of X is very large.

Having selected a model by Lasso, a post-Lasso step can be added by estimating the parameter of the model using ordinary least squares or similar methods.

The Lasso approach has been extended to a wider group of penalized least squares estimators whose coefficients are estimated by minimizing

$$T^{-1}\sum _{t=0}^{T-1}(y_{t+1}-{\beta }^{\prime }{z}_t{)}^2+\sum _{i=1}^k{p}_{\lambda }(|{\beta }_i|),$$

where $P\lambda$ is a penalty function. Bridge estimators set $p_{\lambda }(|{\beta }_i|)=|{\beta }_i{|}^{\gamma }$ , where $\gamma \ge 0$ As a special case, if $\gamma =1$ , Lasso is obtained, while $\gamma =2$ yields the Ridge estimator. For $\gamma \le 1$ this estimator retains the property that some of the estimates for $\beta$ will be 0 for large enough values of the penalties; see Fan and Li (2001) for theoretical results and Fan, Lv, and Qi (2011) for a review.