5.1 BAYES RISK

5.1 BAYES RISK

Chapter 3 introduced Bayes risk as the weighted average of risk with weights chosen as the priors over the possible parameter values π (θ ):

$$r(\pi ,f)={\int }_{\theta }R(\theta ,f)\pi (\theta )d\theta .\text{\hspace{1em}(5.1)}$$

Any forecast $f(z)$ that minimizes Bayes risk is a Bayes decision rule.

Using the notation for risk introduced in chapter 3, we have

$$R(\theta ,f)={\int }_z{\int }_{y}L(f(z),y{)}_{PY}(y|z,\theta {)}_{PZ}(z|\theta )dydz,\text{\hspace{1em}(5.2)}$$

and so Bayes risk can be written as (see (3.26))

$$r(\pi ,f)={\int }_z\left({\int }_{\theta }\left\{{\int }_{y}L(f(z),y{)}_{PY}(y|z,\theta )dy\right\}\pi (\theta |z)d\theta \right)m(z)dz.\text{\hspace{1em}(5.3)}$$

Here we constructed the posterior for the parameters using Bayes’ rule $\pi (\theta |z)=$ $pz(z|\theta )\pi (\theta )/m(z)$ , where $\begin{array}{r}m(z)=\int pz(z|\theta )\pi (\theta )d\theta \end{array}$ is the marginal likelihood. This can be viewed as a prior predictive distribution, i.e., a predictive distribution that does not depend on the observed data, z.

Minimizing (5.3) over forecast methods $f(z)$ conditional on $Z=z$ is the same as minimizing

$$\begin{array}{rl}{\int }_{\theta }\{{\int }_{y}L(f(z),y)\{p_Y(y|z,\theta )dy\}\pi (\theta |z)d\theta & \\ & ={\int }_{y}L(f(z),y)({\int }_{\theta }{p}_Y(y|z,\theta )\pi (\theta |z)d\theta )dy\\ & ={\int }_{y}L(f(z),y)p_Y^P(y|z)dy,\end{array}$$

where the minimization is over $f(z)$ and we used that the loss function does not depend on θ. Here $\begin{array}{r}{p}_Y^P(y|z)={\int }_{\theta }{p}_Y(y|z,\theta )\pi (\theta |z)d\theta \end{array}$ is the (posterior) predictive density for Y. The optimal Bayesian point forecast is the forecast that minimizes expected loss with respect to this posterior predictive density.

Obtaining the posterior density of the parameters, $\pi (\theta |z)$ , involves the usual Bayesian posterior density estimation problem for the model parameters. Hence, Bayesian forecasting can utilize a set of well-established methods. Many expositions cover this general problem and explain how to go about constructing the posterior density; see, e.g., Geweke (2005). The next example illustrates how the Bayesian predictive density can be derived in a simple case.

Example 5.1.1 (Computing the predictive density for an i.i.d. Gaussian series). Let $y_i\sim$ ind $\mathrm{N}(\mu ,{\sigma }^2)$ with ${\sigma }^2$ known, and suppose the prior on $\mu$ is $\mathrm{N}({\mu }_0,{\tau }^2)$

Using observations $z=(y_1,\dots ,y_T)$ to construct the forecast, the posterior density for $\mu$ is

$$\mu |(y_1,\dots ,y_T)\sim \mathbf{N}\left(\frac{T{\tau }^2}{T{\tau }^2+{\sigma }^2}{\bar{y}}_T+\frac{{\sigma }^2}{T{\tau }^2+{\sigma }^2}{\mu }_0,\frac{{\tau }^2{\sigma }^2}{T{\tau }^2+{\sigma }^2}\right)\equiv \mathbf{N}(\tilde{\mu },{\tilde{\sigma }}^2),$$

where ${\bar{y}}_T=T^{-1}{\sum }_{t=1}^T{y}_t$ is the sample mean. The predictive density for $y_{T+1}givenz$ then follows from

$$p_{y_{T+1}}^p(y_{T+1}|z)=\int f(y_{T+1}|\mu ,{\sigma }^2)\pi (\mu |z,{\mu }_0,{\tau }^2,{\sigma }^2)d\mu ,$$

which results in

$$y_{T+1}|z\sim \mathrm{N}(\tilde{\mu },{\sigma }^2+{\tilde{\sigma }}^2),\text{\hspace{1em}(5.4)}$$

where the variance ${\sigma }^2+{\tilde{\sigma }}^2={\sigma }^2(1+{\tau }^2/(T{\tau }^2+{\sigma }^2))$ . This distribution is the Bayesian density forecast $P_y^P(y|z)$

Just as in the classical case, for some problems the predictive density may be considered a more useful summary of the forecast than a point forecast. In many practical forecast situations this is what most Bayesians would consider to be the objective of a forecasting problem because of the generality of the solution.

Once we have the predictive density, $p_Y^P(y|z)$ , the best forecast can be constructed using the equivalent to the first step of the two-step classical approach, but with the predictive density in place of the unknown conditional density. For example, under MSE loss, the best estimator is the conditional mean of $y$ given z. The Bayesian equivalent is the mean of the predictive density $p_Y^P(y|z)$ . Similarly, for the lin-lin loss function we could report the relevant quantile of the predictive density.

Example 5.1.2 (Forecasting the mean of an i.i.d. Gaussian series under MSE loss). Under MSE loss and using the setup in Example 5.1.1, the forecast becomes

$$f(z)=E\left[y_{T+1}|z\right]=\tilde{\mu }=\frac{T{\tau }^2}{T{\tau }^2+{\sigma }^2}{\bar{y}}_T+\frac{{\sigma }^2}{T{\tau }^2+{\sigma }^2}{\mu }_0.$$

This is a shrinkage estimator that shrinks the sample mean, ${\bar{y}}_T$ , towards the prior mean, ${\mu }_0,$ with weights that depend on the information in the prior $({\tau }^2)$ and the sample $({\sigma }^2,T)$ . In large samples the weight on the sample mean gets closer to 1. A similar result occurs for a more diffuse prior, i.e., a larger value of ${\tau }^2$

Example 5.1.3 (Forecasting the mean of an i.i.d. Gaussian series under Linex loss). Under the Linex loss function, the optimal forecast given the conditional density is the mean plus $\left(a/2\right)$ times the variance. Using the predictive distribution in $(5.4)$ , this is

$$f(z)=\tilde{\mu }+\frac{a}{2}\left({\sigma }^2+{\tilde{\sigma }}^2\right),$$

which is the Bayesian point forecast.

Both Examples 5.1.2 and 5.1.3 use the same predictive density to construct point forecasts under different loss functions. Provision of the predictive density would therefore have been sufficient for forecasters in each of these cases.

Finally, when specifying the likelihood function used to construct the predictive density we required a model for both y and z, the choice of which will influence the results. This assumption of a known model is also implicit when we use the same θ for both the model and the true joint density of the data. Extension of Bayesian methods to allow for an unknown model often involves averaging over different models with weights reflecting the posterior likelihood that a particular model is “true.” We examine such approaches in chapter 14.

In addition to providing the predictive density or Bayesian point forecasts, we can also examine the expected loss through the risk function. Expressions for these objects can be obtained by plugging the optimal forecast, $f(z)$ , into either Bayes risk (equations (5.1) and (5.3)) or the risk function (5.2) which depends on θ . First, consider the risk function, $R(\theta ,f(z))$ . Denoting the optimal forecast by $f^{\ast }(z)$ , the risk becomes

$$R^{\ast }(\theta ,f)={\int }_z{\int }_{y}L(f^{\ast }(z),y)p_Y(y|z,\theta )p_Z(z|\theta )dydz.\text{\hspace{1em}(5.5)}$$

Given the function $f^{\ast }(z)$ , this reduces to a function of θ —just as in the classical case. For a given forecast, $f^{\ast }(z)$ , the risk function can be examined for various values of $\theta$ to consider the associated risk.

Example 5.1.4 (Forecasting the mean of an i.i.d. Gaussian series under MSE loss, continued). For this example, $f^{\ast }(z)=\tilde{\mu }$ , so we need to evaluate

$$R^{\ast }(\mu ,f^{\ast })={\int }_z{\int }_y(y-\tilde{\mu }{)}^2{p}_Y(y|z,\theta )p_Z(z|\theta )dydz.$$

We know that the conditional density is $y_{T+1}|z,\theta \sim \mathrm{N}(\mu ,{\sigma }^2)$ . Integrating over this, we obtain $\begin{array}{r}{\int }_{\nu }(y-\tilde{\mu }{)}^2{p}_Y(y|z,\theta )dy={\sigma }^2+(\mu -\tilde{\mu }{)}^2\end{array}$ , i.e., the variance plus the squared bias. Here $\tilde{\mu }$ is a function of $z$ since it depends on ${\bar{y}}_T$ . Integrating over $z,$ we have

$$\begin{array}{c}{R}^{\ast }(\mu ,f)={\int }_z\left({\sigma }^2+(\mu -\tilde{\mu }{)}^2\right)p_Z(z|\theta )dz\\ ={\sigma }^2(1+T^{-1}{a}^2)+(\mu -{\mu }_0{)}^2(1-a{)}^2,\end{array}$$

where $a=T{\tau }^2/(T{\tau }^2+{\sigma }^2)$ . This risk function shows the trade-off between using an informative prior versus using the sample mean. The sample mean is obtained from by letting ${\tau }^2\to \infty ,$ , so $a\to 1$ , corresponding to a flat prior. From, we have $R(\mu ,{\bar{y}}_T)={\sigma }^2(1+T^{-1})$ . Since $a<1$ , the prior reduces the variance component but adds a squared bias component $(\mu -{\mu }_0{)}^2(\bar{1}-a{)}^2$ . When the prior mean, ${\mu }_0,$ is sufficiently close to the true value, $\mu ,$ , this will reduce risk. However, for more distant values the squared bias term can be greater than the estimation risk from using the least squares forecast.

For the simplest of forecast situations, this example again shows the lack of a single approach that is optimal across all models. Different choices of prior result in different estimators and different risk functions, but none is uniformly better than the others.

To reduce risk from a function of θ to a single number, and hence a measure that allows us to rank different methods, we need to weight all the possible values for θ. This requires a distribution over θ. Using the prior distribution, $\pi (\theta )$ , we have

$$r^{\ast }(\pi ,f^{\ast })={\int }_{\theta }{R}^{\ast }(\theta ,f^{\ast })\pi (\theta )d\theta .$$

Since $f^{\ast }(z)$ was chosen to minimize this weighted average risk, clearly the Bayes approach both minimizes the risk for this weighting and provides a constructive method for deriving the best forecasting rule. Indeed, under any weighting over the parameters, the Bayes’ rule constructed for priors that are identical to that weighting provides an optimal forecast method. Classical methods can also be optimal if they are equivalent to some Bayes’ rule.

Example 5.1.5 (Forecasting the mean of an i.i.d. Gaussian series under MSE loss, continued). Using the prior density $\pi (\mu )\sim \mathrm{N}({\mu }_0,{\tau }^2)$ , we have

$$\begin{array}{c}{r}^{\ast }(\pi ,f^{\ast })={\int }_{\mu }\left({\sigma }^2(1+T^{-1}{a}^2)+(\mu -{\mu }_0{)}^2(1-a{)}^2\right)\pi (\mu )d\mu \\ \\ ={\sigma }^2(1+T^{-1}{a}^2)+(1-a{)}^2{\tau }^2.\end{array}$$

For any choice of ${\tau }^2,r^{\ast }(\pi ,f^{\ast })\le r(\pi ,\bar{y})={\sigma }^2(1+T^{-1})$ . Equality holds as ${\tau }^2\to \infty$