3.2.2 Risk and the Information Set

3.2.2 Risk and the Information Set

The way risk is defined in equation (3.15) means that its precise form depends on the specification of the information set. How the information set is specified is thus important for risk, just as it is in the evaluation and comparison of forecasts, as we shall see in the third part of the book.

The specification of the information set is particularly important to the notion of a trade-off between bias and variance that is key to understanding many forecasting problems. We next use the special case with a linear forecasting model and MSE loss to illustrate this point. We assume that not all possible predictor variables have been included in the model.

Suppose there are K potential predictor variables and let $Z_T$ be the random variable that captures current and past values of all the K potential predictor variables, including past and present values of the outcome, $y.$ Similarly, let $Z_{kT}$ be the random variable generated by a subset of $k<K$ predictors. Finally, define $Z_{-kT}$ as the random variable generated by the subset of omitted variables, so $Z_{-kT}\cup Z_{kT}=$ $Z_T$ . Note that different expressions are obtained in equation (3.15), depending on whether we integrate over all of $Z_T$ or over only a subset of it.

Example 3.2.6 (Risk for subset regressions). Consider the same problem as in Example 3.2.3. However, suppose that the forecasting model includes only $k<K$ regressors, including a constant unless all variables (including y) have mean 0.

The least squares estimators for β can be written as ${\hat{\mathbf{\beta }}}_{kT}$ , where ${\hat{\beta }}_{kT}$ has the same dimension as $\beta$ with 0s in place of estimates for the omitted variables and the least squares estimators from a regression $of{y}_{t+1}$ on the included regressors in the remaining places partitioned according to $x_t.$ . Accounting for estimation, the forecast error is given by

$$\begin{array}{rl}& y_{T+1}-{\hat{\beta }}_k^{\prime }{x}_T={\epsilon }_{T+1}+(\beta -{\hat{\beta }}_k{)}^{\prime }{x}_T\\ & ={\epsilon }_{T+1}+(\beta -E[{\hat{\beta }}_k|Z_{-kT}]{)}^{\prime }{x}_T+(E[{\hat{\beta }}_k|Z_{-kT}]-{\hat{\beta }}_k{)}^{\prime }{x}_T,\end{array}$$

where $E[{\hat{\beta }}_k|Z_{-kT}]$ is the expected value of ${\hat{\beta }}_k$ conditional on the omitted regressors. This is the standard omitted variable bias formula, where the mean of the ordinary least squares (OLS) estimates is a function of the omitted variables.

Integrating the squared forecast error only over $(y_{T+1},Z_{kT})$ conditional on $Z_{-kT}$ the expected loss becomes

$$\begin{array}{rl}& E\left[{\left({\epsilon }_{T+1}+(\beta -E({\hat{\beta }}_k|Z_{-kT}){)}^{\prime }{x}_T+(E({\hat{\beta }}_k|Z_{-kT})-{\hat{\beta }}_k{)}^{\prime }{x}_T\right)}^2|Z_{-kT}\right]\\ & ={\sigma }^2+E\left[({\hat{\beta }}_k-E({\hat{\beta }}_k|Z_{-kT}){)}^{\prime }{x}_T{x}_T^{\prime }({\hat{\beta }}_k-E({\hat{\beta }}_k|Z_{-kT}))|Z_{-kT}\right]\\ & +\left[(\beta -E({\hat{\beta }}_k|Z_{-kT}){)}^{\prime }E[x_T{x}_T^{\prime }|Z_{-kT}](\beta -E({\hat{\beta }}_k|Z_{-kT}))\right].\end{array}\text{\hspace{1em}(3.21)}$$

The second term is a variance term related to estimation error. Using the same math as in Example 3.2.3, this second term is approximately equal to ${\sigma }^{2}k/T$ . The third term is a bias term.

We refer to the second and third terms in (3.21) (the terms inside the large square brackets) collectively as the “estimation effect terms.” Notice that nothing (apart from collecting more data and hence further enlarging $Z_T)$ can be done to improve on the first term, although this avenue for improving forecasts should never be neglected.

This formulation of the problem results in the usual bias–variance trade-off that is common in calculations of the MSE. When forecasts are built from linear regressions and a constant is included, one can consider the notion of a biased forecast only when conditioning on potential omitted predictors. Nearly all of the choices that arise in the construction of forecasting models, regardless of the loss function, revolve around the cost and benefit from adding predictor variables. Under MSE loss we see that there is a trade-off between the variance (the second term in (3.21)) and the squared bias (the third term).

Adding variables increases the loss through the variance term, which increases by ${\sigma }^2/T$ . However, the potential benefit is that the bias term $(\beta -E({\hat{\beta }}_k|Z_{-kT}))$ could well be smaller as a result of the estimation of an extra parameter of $\hat{\mathbf{\beta }}$ .

The variance term disappears at rate $T$ and so is of a smaller order (as a function of the sample size) than the first term in (3.21). For egregiously misspecified regressions, the bias term could be of the same order as the first term, and hence for large enough biases this term can dominate the variance term by an order of magnitude. Note, however, that sensibly applied statistical tests can usually be used to rule out such situations. Recall that under standard assumptions (and certainly for our i.i.d. examples) the power of hypothesis tests for the significance of the√ regression coefficients grows at rate $\sqrt{T}$ . This means that statistical tests for model misspecification have good power in situations where the bias term is of a greater order than the variance term. When statistical tests find it difficult to distinguish between models, the bias–variance trade-off becomes more complicated.

Notice that the risk in (3.21) is a function of $Z_{-kT}$ . Alternatively, we could have considered integrating over all of $Z_T$ , to obtain a different expression for the risk, as we show in the next example.

Example 3.2.7 (Risk for subset regression, continued). Consider the same problem as in Example 3.2.6. We can write the full regression as

$$\begin{array}{rl}& y_{t+1}={\beta }^{\prime }{x}_t+{\epsilon }_{t+1}\\ & =(\beta -{\beta }_k^{\ast }+{\beta }_k^{\ast }{)}^{\prime }{x}_t+{\epsilon }_{t+1}\\ & ={\beta }_k^{\ast \prime }{x}_t+u_{t+1},\end{array}$$

where $u_{t+1}=(\beta -{\beta }_k^{\ast }{)}^{\prime }{x}_t+{\epsilon }_{t+1}$ and ${\beta }_k^{\ast }$ minimizes $E[(y_{t+1}-{\beta }_k^{\prime }{x}_T{)}^2]$ , where the expectation is computed over all random variables. The variance of $u_{t+1}$ is ${\sigma }_u^2=$ ${\sigma }^2+(\beta -{\beta }_k^{\ast }{)}^{\prime }{\Sigma }_X(\beta -{\beta }_k^{\ast })\Sigma {\sigma }^2$ , where ${\Sigma }_X=E\left[x_t{x}_t^{\prime }\right]$ . Now the regression to be run is seen to be a “short” regression with a larger variance. It follows directly from the result in Example 3.2.3 that the risk is

$$R(\theta ,{\hat{\beta }}_k^{\prime }{x}_T)=\left((\beta -{\beta }_k^{\ast }{)}^{\prime }{\Sigma }_X(\beta -{\beta }_k^{\ast })\right)(1+k/T)\text{\hspace{1em}(3.22)}$$

$$={\sigma }_u^2(1+k/T).\text{\hspace{1em}(3.23)}$$

Hence the scaling variable ${\sigma }_u^2$ is larger than for the expression in Example 3.2.6.

The expression in (3.23) is the average of (3.21), averaged over the omitted variables, $Z_{-kT}$ . In this version where we integrate out the omitted variables, the “estimation effect terms” are convoluted with the unpredictable component. The first approach is often more useful for comparing models. For methods where the variance term vanishes asymptotically as the sample size becomes large, often the result amounts to a comparison of $\dot{{\sigma }^2}$ versus ${\sigma }_u^2.$ . This holds, for example, for many approaches to out-of-sample forecast evaluation, covered in the third part of the book.