3.2.1 Loss-Based versus Two-Step Approaches

3.2.1 Loss-Based versus Two-Step Approaches

The classical (or frequentist) approach to construction of forecasting models falls into two broad areas: either working directly with the sample analog of the loss function or using a two-step approach that first finds the form of the optimal forecast, then uses plug-in estimates for the unknown parameters. This section examines general issues that arise under these approaches. Much of the remainder of the book examines forecast construction in the context of more specific forecasting problems.

The sample analog approach requires finding the sample analog to (3.15) and constructing a forecast model $f(z,\beta )$ , where $\beta$ are unknown parameters with values that depend on ${\theta }_{;}$ , although as we discuss in chapters 6 and 11 the forecast model is typically also unknown. If $f(z,\beta )$ is identical to the optimal forecast $(i.e.,f(z,\beta )$ equals the conditional mean of $Y$ under MSE loss), then the forecast model is correctly specified. In practice, this should be considered an unlikely event. More likely, $f(z,\beta )$ is an approximation to the correct specification.

The two-step approach approximates the relevant feature of the conditional distribution of the predicted variable—e.g., under MSE loss this feature is the conditional mean of $Y$ given $Z=z-\mathrm{by}$ a forecast model $f(z,\beta )$ . Again, if this model is identical to the relevant feature of $p_Y(y|z)$ , then the model is correctly specified, although more likely $f(z,\beta )$ is an approximation. In the second step, an estimate for $\beta ,$ , denoted $\hat{\mathbf{\beta }},$ is used to generate a forecast $f(z,\hat{\mathbf{\beta }})$ . The two-step approach therefore requires that the relevant feature of $p_Y(y|z)$ can be determined whereas the sample analog approach can always be used. Nevertheless, the two-step approach is more ubiquitous in practice. This is largely because the majority of forecasting studies is undertaken under MSE loss, reducing the forecast analysis to attempts at estimating the conditional mean of the outcome.

Example 3.2.1 (Linex loss, continued). Let $Z_T=\{y_t{\}}_{t=1}^T$ , so the data only comprise past values of the dependent variable. $If$ we assume that $y_t\sim$ ind $\mathrm{N}(\mu ,{\sigma }^2)$ , then

$$M_{y|Z}=\exp \left[a\mu +a^2{\sigma }^2/2\right],\text{\hspace{1em}(3.16)}$$

and we have $f^{\star }\left(z\right)=a^{-1}\lceil a\mu +a^2{\sigma }^2/2\rceil =\mu +a{\sigma }^2/2.$ . A plug-in estimator might use the forecast

$$f_{T+1|T}^{\star }\left(Z_T\right)={\bar{y}}_T+a{\hat{\sigma }}^2/2,\text{\hspace{1em}(3.17)}$$

where ${\bar{y}}_T=T^{-1}{\sum }_{t=1}^T{y}_t$ and ${\hat{\sigma }}^2$ is some estimator of the variance, $i.e.$ $(T-$ $\begin{array}{r}1{)}^{-1}{\sum }_{t=1}^T(y_t-{\bar{y}}_T{)}^2\end{array}$

Since we are estimating a feature of a conditional distribution such as its mean, it follows that point forecasting is really an estimation problem. Most of the results on risk and forecast optimality are analogous to those in parameter estimation theory. For the remainder of this section we go through some of these basic results, adapted to the forecasting problem.

First, consider the case where the forecast model is correctly specified, so $f(z,\beta )$ is equal to the correct feature of $p_Y(y|z)$ . The analog procedure or the second step of the plug-in approach requires that $\beta$ be estimated from data, so the actual forecast model is not $f(z,\beta )$ but $f(z,\hat{\mathbf{\beta }})$ . Calculation of the risk for a forecasting method involves taking the expectation over z and so gets affected by estimation of the unknown parameters. Because the additional randomness induced by parameter estimation increases the risk, the estimated forecast model will never obtain the optimality properties discussed in the previous section.

Example 3.2.2 (MSE loss given the observed history of the outcome). Consider forecasting ${\gamma }_{T+1}$ under MSE loss using $Z_T=\{y_t{\}}_{t=1}^T$ , where $y_t\sim$ ind $\mathrm{N}(\mu ,{\sigma }^2)$ so that the data comprise past observations drawn independently from the same distribution. The population optimal forecast is $\mu$ . This produces a risk of $R(\theta ,\mu )=E(y_{T+1}-$ $\mu {)}^2={\bar{\sigma }}^2$ . When $\mu$ is unknown, we might instead use the sample mean ${\bar{y}}_T$ computed over $T$ observations as our forecast, where ${\bar{y}}_T\sim (\mu ,T^{-1}{\sigma }^2)$ . The risk of this forecast is

$$\begin{array}{c}R(\theta ,{\bar{y}}_T)=E_Y(y_{T+1}-\mu {)}^2+E_Z({\bar{y}}_T-\mu {)}^2\\ \\ ={\sigma }^2(1+T^{-1}).\end{array}$$

This is strictly greater than the risk of the population forecast, ${\sigma }^2$

The risk measure in (3.15) typically differs from the conditional expected loss measure in (3.2) since $z_t$ varies with t. We illustrate this point in the next example.

Example 3.2.3 (MSE loss for linear forecasting model). Consider a linear forecasting model $y_{t+1}={\beta }^{\prime }{x}_t+{\epsilon }_{t+1}$ , where ${\epsilon }_{t+1}\sim iid(0,{\sigma }^2)$ is independent of $Z_T=$ $(x_0^{\prime },.....,x_T^{\prime }{)}^{\prime }$ . Under squared error loss the expected loss is

$$E[({\epsilon }_{T+1}^2+({\hat{\beta }}_T-\beta {)}^{\prime }{x}_T{x}_T^{\prime }({\hat{\beta }}_T-\beta ))].$$

Conditional on the data, $Z_{T:d}$ , and a forecast $f_{T+1|T}={\hat{\beta }}_T^{\prime }{x}_T$ , for the least squares estimator $\begin{array}{r}{\hat{\mathbf{\beta }}}_T={\left({\mathbf{T}}^{-1}{\sum }_{t=0}^{T-1}{x}_t{x}_t^{\prime }\right)}^{-1}\left({\mathbf{T}}^{-1}{\sum }_{t=0}^{T-1}{x}_t{y}_{t+1}\right)\end{array}$ this becomes

$$\begin{array}{rlr}& & E_Y[(y_{T+1}-{\hat{\beta }}_T^{\prime }{x}_T{)}^2|Z_T]\approx E[({\epsilon }_{T+1}^2+({\hat{\beta }}_T-\beta {)}^{\prime }{x}_T{x}_T^{\prime }({\hat{\beta }}_T-\beta ))|Z_T]\\ & & \\ & & ={\sigma }^2(1+T^{-1}{x}_T^{\prime }(T^{-1}\sum _{t=0}^{T-1}{x}_t{x}_t^{\prime }{)}^{-1}{x}_T),\end{array}\text{\hspace{1em}(3.18)}$$

which depends on all values of $x$ up to time $T.$ The unconditional expected loss, obtained by integrating over all $z_T$-values, becomes

$$\begin{array}{l}R(\theta ,{\hat{\beta }}_T^{\prime }{X}_T)=E\left[{\sigma }^2(1+T^{-1}{x}_T^{\prime }(T^{-1}\sum _{t=0}^{T-1}{x}_t{x}_t^{\prime }{)}^{-1}{x}_T)\right]\\ \approx {\sigma }^2(1+T^{-1}K),\end{array}\text{\hspace{1em}(3.19)}$$

where $K=\dim (x_t)$

Risk functions that assume a correct specification, $f(z,\beta )$ , and a known value of $\theta$ (and hence $\beta )$ can be considered infeasible lower bounds on risk or “limiting” approximations to risk in large samples. At the same time when considering the optimality of a forecast procedure, parameter estimation error should be taken into account. A direct consequence of this is that even in the simplest forecast environments—and even with a correct model specification—there is no single optimal procedure that minimizes risk for all possible values of $\theta .$ Instead, the risk functions of different procedures will typically cross for different regions of $\theta .$ . The next example illustrates this point.

Example 3.2.4 (MSE loss for shrinkage estimator). Consider the forecasting problem in Example 3.2.2. Rather than use the sample mean estimator, ${\bar{y}}_T$ we could consider the shrinkage estimator $\gamma {\bar{y}}_T$ where $0<\gamma <1$ . In this case the risk is

$$\begin{array}{rl}& R(\theta ,\gamma {\bar{y}}_T)=E_{Z,Y}(y_{T+1}-\gamma {\bar{y}}_T{)}^2\\ & =E_Y(y_{T+1}-\mu {)}^2+{\gamma }^2{E}_Z({\bar{y}}_T-\mu {)}^2+(1-\gamma {)}^2{\mu }^2\\ & ={\sigma }^2(1+{\gamma }^2{T}^{-1})+(1-\gamma {)}^2{\mu }^2.\end{array}\text{\hspace{1em}(3.20)}$$

For $\mu$ in the range $\pm \sqrt{T^{-1}{\sigma }^2(1+\gamma )/(1-\gamma )}$ the forecast $\gamma {\bar{y}}_T$ has lower risk than the sample mean ${\bar{y}}_T;$ outside this range ${\bar{y}}_T$ has lower risk.

The results above were all predicated on the forecast model being correctly specified up to a set of unknown parameters, $\beta .$ However, as discussed in the previous subsection, we may restrict our search to a subset of possible models that does not include the optimal model. For example, a forecaster with MSE loss might restrict models to linear projections even though the conditional mean is not linear in the conditioning variables. In this case, optimality is established with reference to this limited set of models.

Standard notions of risk for estimators extend to forecasts, including issues related to optimality of estimation methods. In parallel to estimation methods, a forecast method belongs to a family of optimal procedures if there exists some region of the parameter space, $\theta$ , for which no other forecast method has lower risk. It should be clear from the above examples that this concept of optimality generally defines a set of forecast methods as opposed to a single forecast method. Hence it is generally not possible to claim that there is a single optimal forecast method for any problem, unless $\theta$ is restricted to a point, in which case it must be known.

These considerations also raise the possibility that some forecast procedures are dominated by other forecasts for every value of $\theta$ and so are inferior and should never be used. This notion corresponds to the idea of inadmissible estimators.

Example 3.2.5 (MSE loss for shrinkage estimator, continued). Allowing the shrinkage factor $\gamma$ in Example 3.2.4 to take on any value in a set such as the real line, R, generates a family of estimators which we denote $\{\gamma {\bar{y}}_T\}$ . For $|\gamma |>1$ , it follows from (3.20) that there is no range of $\mu$ for which $\gamma {\bar{y}}_T$ dominates the sample mean ${\hat{y}}_T$ , since the associated risk is strictly larger. Such estimators should thus never be employed. However, there is no such ranking when $0<\gamma \le 1$ , in which case stronger shrinkage leads to lower risk for $\mu$ near 0.