3.3 BAYESIAN APPROACH

3.3 BAYESIAN APPROACH

The Bayesian approach starts with the idea of averaging risk over all possible models, using a weighting function or prior, $\pi (\theta )$ , for the parameters. This leads to Bayes risk which is defined by

$$r(\pi ,f)={\int }_{\theta }R(\theta ,f)\pi (\theta )d\theta ,\text{\hspace{1em}(3.24)}$$

where $R(\theta ,f)$ is given by (3.15). Any forecast, $f(z)$ , that minimizes Bayes risk is a Bayes decision rule. Notice that this concept of risk is identical to the weighted average risk of the previous section.

To construct a Bayes decision rule (i.e., a forecast), we need a weighting scheme or prior, $\pi (\theta )$ , over the parameters of the model. This prior tells us which parameter values are likely and which are not. As in the classical approach, a model for the outcome variable, $p_Y(y|z)$ , is also required. Finally, we also need a model for the data used to construct the forecast $p_Z(z|\theta )$ . Given the prior and the model for the data, we can calculate the posterior density for the parameters

$$\pi (\theta |z)=p_Z(z|\theta )\pi (\theta )/m(z),\text{\hspace{1em}(3.25)}$$

where $\begin{array}{r}m(z)={\int }_{\theta }{p}_Z(z|\theta )\pi (\theta )d\theta \end{array}$ is the marginal likelihood of the data, z. Using (3.15) and the posterior density in (3.25), Bayes risk can then be computed as

$$r(\pi ,f)={\int }_z\left({\int }_{\theta }\left\{{\int }_{y}L(f(z),y{)}_{PY}(y|z,\theta )dy\right\}\pi (\theta |z)d\theta \right)m(z)dz.\text{\hspace{1em}(3.26)}$$

Conditional on $Z=z,$ we can choose the forecasting rule that, for each $z,$ minimizes the conditional risk

$${\int }_{\theta }\left\{{\int }_{y}L(f(z),y)p_Y(y|z,\theta )dy\right\}\pi (\theta |z)d\theta .$$

To see why this delivers a forecast that minimizes Bayes risk, consider the expected loss conditional on $Z=z.A$ Bayesian approach would be to choose $f(z)$ to minimize this for any $Z=z$ . Since this rule minimizes the conditional expected loss, it also minimizes the unconditional expected loss, i.e., it minimizes Bayes risk and is hence a Bayes decision rule.

Example 3.3.1 (Bayes risk under MSE loss, continued). From (3.4) we get the Bayes risk

$$\begin{array}{l}r(\pi ,f)={\int }_z\left({\int }_{\theta }\mathrm{Var}(Y|\theta ,z)\pi (\theta |z)d\theta \right)m(z)dz\\ +{\int }_z\left({\int }_{\theta }{\left[E(Y|\theta ,z)-f(z)\right]}^2\pi (\theta |z)d\theta \right)m(z)dz.\end{array}$$

The first term does not depend on the forecast. Minimizing the second term conditional on $z,$ and so ignoring the outer integral, yields the first-order condition

$$\begin{array}{c}{\int }_{\theta }E(Y|\theta ,z)\pi (\theta |z)d\theta =f(z){\int }_{\theta }\pi (\theta |z)d\theta \\ =f(z).\end{array}$$

The optimal forecast is the conditional mean using the posterior, $\pi (\theta |z)$ , as weights. When $y_{t+1}$ given $x_t$ is normal with unknown mean ${\theta }^{\prime }{x}_T$ and known variance ${\sigma }^2{}_{;}$ and the posterior distribution for $\theta isN(\tilde{\theta },\mathcal{V})$ , the optimal forecast reduces to $f(z)=$ $\begin{array}{r}\int {\theta }^{\prime }{x}_T\pi (\theta |z)d\theta ={\tilde{\theta }}^{\prime }{x}_T\end{array}$ , i.e., we replace the unknown parameters with their posterior mean.

Example 3.3.2 (Bayes risk under Linex loss). Again assume that $z=\{y_t{\}}_{t=1}^T$ and $y_t\sim$ ind $\mathrm{N}(\mu ,{\sigma }^2)$ , with $\theta =[\mu ,{\sigma }^2]$ . For simplicity, also assume that ${\sigma }^2$ is known. Suppose the prior, $\pi (\mu )$ , is such that $\mu \sim \mathrm{N}({\mu }_0,{\tau }^2)$ with ${\tau }^2$ known. The posterior distribution is then

$$\mu |z\sim \mathrm{N}\left(\frac{T{\tau }^2}{T{\tau }^2+{\sigma }^2}{\bar{y}}_T+\frac{{\sigma }^2}{T{\tau }^2+{\sigma }^2}{\mu }_0,\frac{{\tau }^2{\sigma }^2}{T{\tau }^2+{\sigma }^2}\right),\text{\hspace{1em}(3.27)}$$

where ${\bar{y}}_T=T^{-1}{\sum }_{t=1}^T{y}_t$ . For notational convenience, write this posterior as $\mu |z\sim$ $\mathrm{N}(\tilde{\mu },{\tilde{\sigma }}^2)$ . To derive the optimal forecast we need to integrate out the unknown parameter, $\mu .$ . Note that from (3.9) and (3.16),

$$\begin{array}{l}f(z)=a^{-1}\ln {\int }_{\theta }{M}_Y(a)\pi (\theta |z)d\theta \\ =a^{-1}\ln {\int }_{\mu }\exp \left[a\mu +a^2{\sigma }^2/2\right]\frac{1}{\sqrt{2\pi }\tilde{\sigma }}\exp \left[-\frac{1}{2{\tilde{\sigma }}^2}(\mu -\tilde{\mu }{)}^2\right]d\mu .\end{array}$$

Combining the exponential components and completing the square yields the optimal rule

$$\begin{array}{l}f(z)=a^{-1}\ln \left(\exp \left[\frac{1}{2}{a}^2({\sigma }^2+{\tilde{\sigma }}^2)+a\tilde{\mu }\right]\right)\\ \\ \end{array}$$

Plugging the results for $\tilde{\mu }$ and ${\tilde{\sigma }}^2$ into (3.27) yields the rule

$$f(z)=\frac{T{\tau }^2}{T{\tau }^2+{\sigma }^2}\bar{z}+\frac{{\sigma }^2}{T{\tau }^2+{\sigma }^2}{\mu }_0+\frac{a{\sigma }^2}{2}\left(1+\frac{{\tau }^2}{T{\tau }^2+{\sigma }^2}\right).\text{\hspace{1em}(3.28)}$$

When ${\tau }^2\to \infty$ , we move towards a flat prior and

$$f(z)=\bar{z}+\frac{a{\sigma }^2}{2}\left(1+\frac{1}{T}\right).$$

Notice the close relationship between the classical plug-in method that minimizes average risk over all relevant models and the more formal Bayesian approach which chooses the forecast rule to directly minimize the weighted average risk. In the example with MSE loss the forms of the models are the same, so differences arise only through the estimation of θ. In the Linex example there is also a difference in the coefficient multiplying the variance term.