5.2 RIDGE AND SHRINKAGE ESTIMATORS

5.2 RIDGE AND SHRINKAGE ESTIMATORS

Under MSE loss, the Bayesian point forecast will be the average of the conditional mean given the parameters, weighted by the posterior distribution of the parameters. Letting $\mu (z,\theta )$ be the model for the conditional mean of $y$ given $z,$ the Bayesian point forecast under MSE loss is therefore

$$f(z)={\int }_{\theta }\mu (z,\theta )\pi (\theta |z)d\theta ,\text{\hspace{1em}(5.8)}$$

where $\pi (\theta |z)$ is the posterior distribution of $\stackrel{\rightharpoonup }{\theta }$ given $z.$ To derive (5.8), start with

$$\begin{array}{r}L(f(z),y)=(y-f(z){)}^2=(y-\mu (z,\theta ){)}^2+(\mu (z,\theta )-f(z){)}^2\\ +2(y-\mu (z,\theta ))\left(\mu (z,\theta )-f(z)\right).\end{array}$$

The first term affects the risk regardless of our choice of forecast $f(z)$ and so we can ignore this term when choosing $f(z)$ to minimize risk. Given the definition of $\mu (z,\theta )$ , the expected value of the third term equals 0. Bayes risk is then a constant plus the second term,

$$\begin{array}{c}r(\pi ,f)={\int }_z{\int }_{\theta }{\int }_y{\left(\mu (z,\theta )-f(z)\right)}^2{\hat{\mathcal{P}}}_Y(y|z,\theta ){\hat{\mathcal{P}}}_Z(z|\theta )\pi (\theta )d\theta dydz\\ \\ ={\int }_z{\int }_{\theta }{\left(\mu (z,\theta )-f(z)\right)}^2{\hat{\mathcal{P}}}_Z(z|\theta )\pi (\theta )d\theta dz,\end{array}\text{\hspace{1em}(5.9)}$$

where we used that the squared loss term is independent of $y.$ . Assuming that the distributions allow us to differentiate under the integral, the first-order

condition yields

$$2{\int }_z{\int }_{\theta }\left(\mu (z,\theta )-f(z)\right)p_Z(z|\theta )\pi (\theta )d\theta dz=0.$$

Rearranging, we have, conditional on $z,$

$$\begin{array}{c}f(z)={\int }_{\theta }\mu (z,\theta )\frac{p_Z(z|\theta )\pi (\theta )}{\int p_Z(z|\theta )\pi (\theta )d\theta }d\theta \\ ={\int }_{\theta }\mu (z,\theta )\pi (\theta |z)d\theta ,\end{array}$$

which is (5.8). Hence, under MSE loss we can combine the model for the conditional mean with the posterior to compute the Bayesian point forecast.

For the linear prediction model we have $\mu (z,\theta )={\beta }^{\prime }z$ Note that $\begin{array}{r}{\int }_{\beta }({\beta }^{\prime }z)\pi (\beta |z)d\beta ={\tilde{\beta }}^{\prime }\dot{z}\end{array}$ , where $\tilde{\beta }=E_{\pi }[\beta |z]$ , so $\tilde{\mathbf{\beta }}$ is the mean of the posterior distribution for $\beta .$ . It follows that the Bayesian point forecast simply uses the posterior mean in place of the unknown $\beta .$ . The following example illustrates this result for a general regression with known variances and shrinkage.

Example 5.2.1 (Shrinkage estimator for multivariate regression model). Let $y\sim$ N(Xβ, V) and consider the conjugate normal prior, $\beta \sim \mathrm{N}({\beta }_0,{\Sigma }_0)$ . Suppose that V is known and the marginal distribution for X does not involve $\beta .$ . Then the posterior distribution for $\beta$ is $\beta |\bar{\mathcal{Y}},X\sim \mathrm{N}(\tilde{\beta },\Sigma )$ , where $\Sigma =(X^{\prime }{V}^{-1}X+{\Sigma }_0^{-1})$ and

$$\tilde{\beta }=(X^{\prime }{V}^{-1}X+{\Sigma }_0^{-1}{)}^{-1}{X}^{\prime }{V}^{-1}X\hat{\beta }+(X^{\prime }{V}^{-1}X+{\Sigma }_0^{-1}{)}^{-1}{\Sigma }_0^{-1}{\beta }_0.\text{\hspace{1em}(5.10)}$$

The posterior mean is a general shrinkage estimator where the weights depend on the prior and the (generalized) least squares estimator, $\hat{\mathbf{\beta }}=(X^{\prime }{V}^{-1}X{)}^{-1}{X}^{\prime }{V}^{-1}\mathbf{y},$ is shrunk towards the mean of the prior, ${\beta }_0$

Interesting special cases arise directly from this formulation. Setting $V={\sigma }^{2}I$ and ${\Sigma }_0={\tau }^{2}I$ yields the posterior mean

$${\tilde{\beta }}_{\mathrm{Ridge}}=(X^{\prime }X+\lambda I{)}^{-1}(X^{\prime }X)\hat{\beta }+(X^{\prime }X+\lambda I{)}^{-1}\lambda {\beta }_0,\text{\hspace{1em}(5.11)}$$

where $\lambda ={\sigma }^2/{\tau }^2$ . This is the standard Ridge estimator discussed in the previous chapter when, as is customary in Ridge regression, we set ${\beta }_0=0$ . Less informative priors correspond to larger values for ${\tau }^2$ and thus smaller λ which results in less shrinkage towards the prior. The generalized Ridge estimator of the previous chapter arises when we set $V={\sigma }^{2}I$ and use a prior ${\tau }^2{\Sigma }_0$ with ${\beta }_0=0$ . This Bayesian motivation for the Ridge estimator can guide the choice of the shrinkage parameter λ in empirical applications.

Another popular special case is to use Zellner’s g -prior. Under this approach consider setting the prior variance ${\Sigma }_0=g{\sigma }^2(X^{\prime }X)\overline{{}^{-1}}$ . When $V={\sigma }^{2}I$ , using this prior in equation (5.10) yields

$${\tilde{\beta }}_g=\frac{1}{1+g}\left(g\hat{\beta }+{\beta }_0\right).\text{\hspace{1em}(5.12)}$$

For example, setting $g=1$ gives an estimator that is an average of the least squares estimator and the prior mean. Rewriting (5.12) as ${\hat{\beta }}_g=\hat{\beta }+(1+g{)}^{-1}({\beta }_0-\hat{\beta })$ , it readily follows that the larger the value of g , the more diffuse the prior is.

When both the mean and variance of the normal model, $\theta =(\beta ,V)$ , are unknown, it is common to write the prior as $\pi (\theta )=\pi (\beta |V)\pi (V)$ , where $\pi (\beta |V)$ is the Gaussian distribution as in the above example and $\pi (V)$ is an inverse Wishart distribution; see Zellner (1971).