15.3.1 Efficiency Properties under Squared Error Loss

15.3.1 Efficiency Properties under Squared Error Loss

Concrete and well-known results are available under squared error or MSE loss. Specifically, when the loss function is quadratic in the forecast error, $L(f,Y)=$ $L(e)=e^2$ , the first-order condition for an optimal forecast in (15.6) greatly simplifies:

$$\frac{\partial E\left[(y_{t+h}-f_{t+h|t}{)}^2|Z_t\right]}{\partial f_{t+h|t}}=-2E\left[e_{t+h|t}|Z_t\right]=0,\text{\hspace{1em}(15.10)}$$

or, equivalently,

$$E\left[e_{t+h\mid t}{v}_t\right]=0\text{\hspace{1em}(15.11)}$$

for any $v_t=g(z_t)$ , where g is an arbitrary measurable function of $z_t$

Several testable implications follow from this result. These are just different manifestations of the same first-order condition (15.10). Specifically, the following properties hold:4

1. Forecasts are unbiased, $i.e.,$ the forecast error ${\mathbf{e}}_{t+h|t}$ has zero mean, both conditionally and unconditionally:

$$E\left[e_{t+h|t}\right]=E\left[e_{t+h|t}|Z_t\right]=0.\text{\hspace{1em}(15.12)}$$

2. The h-period forecast errors $\left(e_{t+h|t}\right)$ are uncorrelated with information available at the time the forecast was computed $(Z_t)$ . In particular, the h-period forecast error is at most serially correlated of order $h-1$

$$\mathrm{Cov}(e_{t+h|t},e_{t-j|t-h-j})=0\mathrm{for}\mathrm{all}j\ge 0.\text{\hspace{1em}(15.13)}$$

Hence, ${\mathbf{e}}_{t+h|t}$ follows an $\mathrm{MA}(h-1)$ process. In the special case with a single-period horizon, $h=1$ , forecast errors, ${\mathbf{e}}_{t+1|t},$ are serially uncorrelated:

$$E\left[e_{t+1|t}{e}_{t|t-1}\right]=0.$$

3. The variance of the forecast error $\left(e_{t+h|t}\right)$ is a nondecreasing function of the forecast horizon, h:

$$\mathrm{Var}(e_{t+h\mid t})\ge \mathrm{Var}(e_{t+h\mid t+1})\mathrm{for}\mathrm{all}h\ge 1.\text{\hspace{1em}(15.14)}$$

It is instructive to illustrate these results using simple time-series representations as we do in the next examples.

Example 15.3.1 (Efficiency properties for a covariance stationary process). Suppose that Y is a covariance stationary process and, without loss of generality, assume it has zero unconditional mean. The Wold representation theorem then establishes that Y can be represented as a linear combination of serially uncorrelated white noise terms (see chapter 7):

$$Y_t=\sum _{i=0}^{\infty }{\theta }_i{\epsilon }_{t-i},\text{\hspace{1em}(15.15)}$$

where, from chapter 7, ${\epsilon }_t=Y_t-\mathrm{Proj}(Y_t|y_{t-1},y_{t-2},...)$ is the serially uncorrelated projection error5 at time t.

Assuming that $Z_t=\{{\epsilon }_t,{\epsilon }_{t-1},...\}$ and the moving average parameters $\{{\theta }_0,{\theta }_1,\dots \}$ are known, the h-period forecast becomes

$$f_{t+h|t}=\sum _{i=0}^{\infty }{\theta }_{h+i}{\epsilon }_{t-i}.\text{\hspace{1em}(15.16)}$$

This is an optimal forecast provided that ${\epsilon }_t$ is Gaussian or can be viewed as an optimal linear forecast given past values of the process $\{y_{t-1},y_{t-2},...\}$ . The associated forecast error is

$$e_{t+h\mid t}=y_{t+h}-f_{t+h\mid t}=\sum _{i=0}^{h-1}{\theta }_i{\epsilon }_{t+h-i}.\text{\hspace{1em}(15.17)}$$

Using that ${\epsilon }_t$ is serially uncorrelated mean-zero white noise, it follows that

$$E\left[e_{t+h\mid t}\right]=0,$$

$$\mathrm{Var}(e_{t+h\mid t})={\sigma }^2\sum _{i=0}^{h-1}{\theta }_i^2,\text{\hspace{1em}(15.18)}$$

$$\begin{array}{r}\mathrm{Cov}(e_{t+h\mid t},e_{t+h-j\mid t-j})=\{\begin{array}{ll}{\sigma }^2{\sum }_{i=j}^{h-1}{\theta }_i{\theta }_{i-j}& for0<j<h,\\ 0& forj\ge h.\end{array}\end{array}$$

Our second example helps clarify the properties established in the first example.

Example 15.3.2 (Efficiency properties for AR(1) process). Consider the stationary first-order autoregressive process,

$$y_t=\varphi y_{t-1}+{\epsilon }_t,{\epsilon }_t\sim ind\mathbf{N}(0,{\sigma }^2),|\varphi |<1,\text{\hspace{1em}(15.19)}$$

where the information set is $Z_t=(y_t,y_{t-1},...)$ , so that

$$y_{t+h}={\varphi }^h{y}_t+\sum _{i=1}^h{\varphi }^{h-i}{\epsilon }_{t+i}.\text{\hspace{1em}(15.20)}$$

Using that $E\left[{\epsilon }_{t+i}|Z_t\right]=0.$ for all $i\ge 1_{\mathrm{i}}$ , it follows from (15.20) that

$$f_{t+h|t}={\varphi }^h{y}_t,$$

$$e_{t+h|t}=\sum _{i=1}^h{\varphi }^{h-i}{\epsilon }_{t+i}.$$

It is easily seen that

$$E\left[e_{t+h}|Z_t\right]\equiv E\left[e_{t+h|t}\right]=E\left[\sum _{i=1}^h{\varphi }^{h-i}{\epsilon }_{t+i}\right]=0.$$

Moreover, Cov $(e_{t+1|t},e_{t|t-1})=0$ , while for $h>j\ge 1$

$$\mathrm{Cov}(e_{t+h\mid t},e_{t+h-j\mid t-j})=\frac{{\sigma }^2{\varphi }^j(1-{\varphi }^{2(h-j)})}{1-{\varphi }^2},$$

which is increasing in h and shows that forecast errors will be serially correlated at horizons longer than one period. Finally, note from (15.20) that

$$\mathrm{Var}(e_{t+h|t})=\frac{{\sigma }^2(1-{\varphi }^{2h})}{1-{\varphi }^2},$$

which is a nondecreasing function of h, verifying the third property that the variance of the forecast error increases weakly in the forecast horizon.

The first two optimality properties, (15.12) and (15.13), are commonly tested by regressing forecast errors on elements in the forecaster’s information set, $v_t=g(z_t);$

$$e_{t+h|t}={\beta }^{\prime }{v}_t+u_{t+h}.\text{\hspace{1em}(15.21)}$$

Tests of the null of forecast efficiency are of the form

$$\begin{array}{r}{H}_0:\beta =0,\\ \\ \mathrm{versus}{H}_1:\beta \ne 0.\end{array}$$

Tests for unbiasedness and orthogonality simply correspond to different choices for the test function, $v_t$ . Tests for unconditional unbiasedness set $v_t$ equal to a constant:

$$e_{t+h|t}={\beta }_0+u_{t+h}.\text{\hspace{1em}(15.22)}$$

In a scatterplot of the forecasts against the outcome variable, this represents the case where the point described by the average of the forecast and outcome lies on the $45^{\circ }$ line (see figure 15.2). A rejection of ${\beta }_0=0$ suggests that the forecast is a biased estimator of the outcome and thus explains why this procedure is called an unbiasedness test. Unbiasedness is also commonly viewed as a joint test of ${\beta }_0=0,{\beta }_1=1$ in the Mincer–Zarnowitz regression (Mincer and Zarnowitz, 1969) that adds the forecast, $f_{t+h|t}$ , to both sides of (15.22),

$$y_{t+h}={\beta }_0+{\beta }_1{f}_{t+h|t}+u_{t+h},\text{\hspace{1em}(15.23)}$$

and thus corresponds to setting $v_t=(1,f_{t+h|t})$ in (15.21).

As an empirical illustration, returning to the Greenbook forecasts, we use the Mincer–Zarnowitz test to check whether the forecasts are unbiased, adopting an F -test to test that ${\beta }_0=0,{\beta }_1=1$ . The null of unbiased forecasts is strongly rejected for the GDP forecasts, but not for the inflation rate. From the individual t-tests applied separately to ${\beta }_0$ and ${\beta }_1$ , it is clear that the intercept is nonnegative for the GDP series. As an alternative test, we check whether the forecast errors are serially uncorrelated. With autocorrelations around 0.5, we can strongly reject the null for both series, suggesting that the forecasts could be improved.

Extending the test function $v_t$ in (15.21) to contain other information allows us to test whether the forecast errors are orthogonal to such information. For example, setting $v_t=(e_{t\mid t-h},e_{t-1\mid t-h-1},...)$ , gives a test for weak forecast rationality and so examines whether there is serial correlation in forecast errors over and above what would be expected due to any data overlaps resulting from the use of horizons exceeding a single period. As shown in (15.18), even optimal multistep forecasts may be serially correlated due to overlaps in forecast errors. Irrespective of this, the general rule is that rationality tests should use information dated at the point where the forecast was generated. Doing so always results in a valid specification of the rationality test.

Suppose the null of forecast rationality is rejected. This indicates that the forecaster failed to use all information in the test efficiently which implies that the forecast can be improved. This is perhaps most easily seen in the case of MSE loss and the regression,

$$y_{t+h}=\alpha +\beta f_{t+h|t}+\lambda v_t+{\epsilon }_{t+h}.\text{\hspace{1em}(15.24)}$$

Suppose we reject the null that $\alpha =0,\beta =1,\lambda =0$ , so the forecast is biased. A biased-adjusted forecast can then be computed as

$${\hat{\mathbf{f}}}_{t+h|t}^{\mathrm{adj}}=\hat{\mathbf{\alpha }}+\hat{\beta }{\hat{\mathbf{f}}}_{t+h|t}+\hat{\lambda }{\mathbf{v}}_t,\text{\hspace{1em}(15.25)}$$

where “hats” indicate least squares estimates. The benefit of this approach is that it is a simple strategy for producing potentially improved forecasts. The disadvantage is that it is a mechanical adjustment that provides no explanation for the biases in the original model and does not attempt to produce a better forecasting model. Moreover, due to estimation error in $\hat{\alpha },\hat{\beta },\hat{\lambda }$ , there is no guarantee that in practice this procedure will yield a more attractive risk function than the original (biased) forecast.

It is less common to test the third property, (15.14), that the variance of the forecast error should be weakly increasing in the forecast horizon, in part because such a test requires that forecasts are available at different horizons. This type of data is most common for economic surveys. For such situations, Patton and Timmermann (2012) propose a procedure that exploits the full information in the term structure of mean squared forecast errors, i.e., the full set of MSE values recorded for different horizons. Suppose MSE values corresponding to a set of forecast horizons of increasing length, $h=h_1,h_2,\dots ,h_H$ are available and denote the corresponding MSE population values by ${\mathbf{\mu }}^e={\left[{\mu }_1^e,\dots ,{\mu }_H^e\right]}^{\prime }$ , where ${\mu }_j^e\equiv \mathcal{E}\left[e_{t+h_j|t}^2\right]$ . Defining the associated MSE differentials as ${\Delta }_j^e\equiv {\mu }_j-{\mu }_{j-1}=E[e_{t+h_i|t}^2]-E[e_{t+h_{i-1}|t}^2].$ it follows from (15.14) that the expected value of the squared forecast errors under squared error loss is weakly increasing in the forecast horizon:

$${\Delta }_j^e\ge 0\mathrm{for}j=2,\dots ,H.\text{\hspace{1em}(15.26)}$$

This weak monotonicity property can be tested through the null hypothesis:

$$H_0:{\Delta }^e\ge 0\mathrm{versus}{H}_1:{\Delta }^e\ngeqq 0,\text{\hspace{1em}(15.27)}$$

where the $((H-1)\times 1)$ vector of MSE differentials is given by ${\Delta }^e\equiv {\left[{\Delta }_2^e,\dots ,{\Delta }_H^e\right]}^{\prime }$ The test uses the sample analogs to the unknown MSE differentials, ${\hat{\Delta }}_j^e={\hat{\mu }}_j-{\hat{\mu }}_{j-1}$ for $\begin{array}{r}{\hat{\mu }}_j\equiv (1/T){\sum }_{t=1}^T{e}_{t+h_i|t}^2.\end{array}$ Standard critical values are not available for the test. However, results by Wolak (1987) show that these can be obtained under the null as a weighted sum of chi-squared variables, $\begin{array}{r}{\sum }_{i=0}^{H-1}\omega (H-1,i){\chi }^2(i)\end{array}$ , where $\omega (H-1,i)$ are the weights and ${\chi }^2\overline{(i)}$ is a chi-squared variable with i degrees of freedom. The weights correspond to the probability that the vector $Z\sim \mathrm{N}(0,\Sigma )$ has i positive elements, where  is the long-run covariance matrix of the estimated parameter vector, ${\hat{\Delta }}^e$ . This can be computed either via simulation or in closed form; see Patton and Timmermann (2012) for details. Alternatively, the bootstrap methods of White (2000) or Hansen (2005) can be adopted; such methods can be useful particularly when the number of forecast horizons is large.

Figure 15.6 provides an empirical illustration of the bound applied to quarterly forecasts of the unemployment rate and real GDP growth obtained from the Survey of Professional Forecasters. We use the mean forecasts over the sample 1968Q4 to 2009Q4. As can be readily seen, the mean squared errors are monotonically increasing in the forecast horizon for the unemployment rate forecasts, though not, at the longest four-quarter horizon, for the GDP growth.

Figure 15.6: Mean squared forecast error as a function of the forecast horizon for the Survey of Professional Forecasters’ mean forecasts.

Bounds such as (15.26) hold when the forecasting model is evaluated at the population values of the parameters. In the presence of parameter estimation error, the monotonicity properties could break down as shown by Hoque, Magnus, and Pesaran (1988) and Magnus and Pesaran (1989). A term structure of MSE values that is not weakly increasing therefore need not be indicative of inefficient forecasts and could be caused by estimation errors. In the reply to the discussion of their paper, Patton and Timmermann (2012) conduct Monte Carlo simulations for an AR(1) model and find that for estimation sample sizes of 100 or more all of the monotonicity and bounds results in Patton and Timmermann (2012) hold in the presence of parameter estimation error. They find that estimation error can overturn the bounds only in very small sample sizes such as 25 observations. In fact, tests based on inequality restrictions appear to be less sensitive to parameter estimation error than orthogonality tests based on equality restrictions.