20.5 FORECAST EVALUATION

20.5 FORECAST EVALUATION

For univariate models the in-sample mean squared forecast error will converge to the variance of the unpredictable component for models that are correctly specified or close to being correctly specified $(e.g.$ , models that impose a unit root when $\rho$ is local to unity). This is unsurprising and follows directly from the estimation error being of smaller order than variation due to the unpredictable error component.

Things get more complicated out-of-sample, although only a small body of research extends the out-of-sample results of chapters 16 and 17 to nonstationary models. We briefly discuss the main issues in a simple model to clarify the implications of using persistent data with a unit root or near unit root. A key issue with persistent data is that out-of-sample averages of squared errors typically no longer converge to the expected loss when forecasting a single outcome, so conventional estimates of performance are not necessarily indicative of the expected performance. At long horizons, sample averages of MSE values diverge as the sample size increases. These results have implications for commonly reported performance measures such as the MSE, ratios of MSE values, or $R^2$ -values.

20.5.1 Out-of-Sample Evaluation for Univariate Models

To establish results for out-of-sample forecast evaluation, consider the simplest version of (20.2), where $d_t=1$ and $c(L)=1$ so the model is an $\mathrm{AR}(1)$ with a mean shift. We also set the initial condition to 0. The sample split point is $T_1=[T\times r]$ so the first $T_1$ observations are used for the first estimation period and a recursively expanding estimation window is subsequently used up to $T+h$

First, let h be fixed. This is a reasonable approach if we are approximating forecast horizons that are short relative to the sample size. As above,

$$y_{t+h}-y_t=({\rho }^h-1)(y_t-\varphi )+\sum _{i=1}^h{\rho }^{h-i}{\epsilon }_{t+i},$$

and so the infeasible forecast that uses the true values of $\rho$ and $\varphi$ produces an MSE of

$$\begin{array}{l}\mathrm{MSE}(\rho ,\varphi )=\frac{1}{T-T_1-h}\sum _{t=T_1+1}^{T-h}{\left(y_{t+h}-y_t-({\rho }^h-1)(y_t-\varphi )\right)}^2\\ =\frac{1}{T-T_1-h}\sum _{t=T_1+1}^{T-h}{u}_{t+h}^2,\end{array}$$

where $\begin{array}{r}{u}_{t+h}={\sum }_{i=1}^h{\rho }^{h-i}{\epsilon }_{t+i}\end{array}$ . When $u_{t+h}$ displays limited dependence, we expect that $\mathrm{MSE}(\rho ,\varphi )PE[u_{t+h}^2]$ under sufficient regularity conditions.

The MSE value associated with the random walk (rw) forecast that sets $f_{t+h|t}=y_t$ is

$$\begin{array}{l}\mathrm{MSE}(\mathrm{rw})=\frac{1}{T-T_1-h}\sum _{t=1+1}^{T-h}(y_{t+h}-y_t{)}^2\\ =\frac{1}{T-T_1-h}{\sum }_{t=1+1}^{T-h}(u_{t+h}+({\rho }^h-1)(y_t-\varphi ){)}^2\\ =\frac{1}{T-T_1-h}{\sum }_{t=1+1}^{T-h}(u_{t+h}^2+T^{-1}[T({\rho }^h-1)(T^{-1/2}(y_t-\varphi )){]}^2\\ +2T({\rho }^h-1)T^{-1}{u}_{t+h}(y_t-\varphi )).\end{array}$$

The second term disappears at rate $T$ and the third term disappears even faster. Local misspecification of the model results in a term of order $T^{-1}$ as in the regular case. Here this term has a random limit.

When $\rho$ is estimated, the leading term remains the unpredictable component and again there will be a random term of order $T^{-1}$ that enters the limit expression. The out-of-sample MSE now becomes

$$\begin{array}{l}\mathrm{MSE}(\hat{\rho },\hat{\varphi })=\frac{1}{T-T_1-h}\sum _{t=T_1+1}^{T-h}{\left(y_{t+h}-({\hat{\rho }}^h-1)(y_t-\hat{\varphi })\right)}^2\\ =\frac{1}{T-T_1-h}\sum _{t=T_1+1}^{T-h}{\left(u_{t+h}+({\rho }^h-1)(y_t-\varphi )-({\hat{\rho }}^h-1)(y_t-\hat{\varphi })\right)}^2\\ =\frac{1}{T-T_1-h}\sum _{t=T_1+1}^{T-h}[u_{t+h}-T^{-1/2}[T({\hat{\rho }}^h-{\rho }^h)T^{-1/2}(y_t-\varphi )\\ {-T({\hat{\rho }}^h-1)T^{-1/2}(\hat{\varphi }-\varphi )]]}^2.\end{array}$$

Again the leading term asymptotically will be the limit of the MSE for the known model. There is also a term disappearing at rate $T$ which equals the limit of

$$\frac{1}{T-T_1-h}\sum _{t=T_1+1}^{T-h}{\left[T({\hat{\rho }}^h-{\rho }^h)T^{-1/2}(y_t-\varphi )-T({\hat{\rho }}^h-1)T^{-1/2}(\hat{\varphi }-\varphi )\right]}^2.$$

The limit of this term will depend on $\gamma ;$ recall that $\rho =1-\gamma /T$ . Analytic expressions that are functions of Brownian motions can be obtained but are sufficiently complicated to be of limited practical value.

Apart from the case with a known model, the limits of the $O_p(T)$ terms in the MSE expressions depend on $r,$ the choice of the proportion for the sample split.

In this sense, the expected value of the first-order risk term from a single h-stepahead forecast differs from the average of the out-of-sample h-step-ahead squared forecast errors. Average MSE values therefore may not be a good guide to estimating the true risk.

We can consider the same approaches when h is large relative to the sample size, so that $h=\left[T\lambda \right]$ grows at the same rate as the sample for the asymptotic results. Let the forecast error be $y_{t+h}-f_{t+h|t}$ . Then the statistic of interest in the out-of-sample experiment is

$$\mathrm{MSE}=\frac{1}{T-T_1-h}\sum _{t=T_1+1}^{T-h}{\left(y_{t+h}-f_{t+h|t}\right)}^2.$$

To analyze this statistic and its large sample behavior, first assume that the model is known, so $y_{t+h}-f_{t+h|t}=y_{t+h}-{\rho }^h{y}_t$ . Then we have, for known ${\rho }_{;}$ ,

$$\begin{array}{l}{T}^{-1}\mathrm{MSE}(\rho )=\frac{1}{T-T_1-h}\frac{1}{T}\sum _{t=T_1+1}^{T-h}(y_{t+h}-{\rho }^h{y}_t{)}^2\\ =\frac{1}{1-[Tr]/T-[T\lambda ]/T}\frac{1}{T}\sum _{t=[Tr]+1}^{T-[T\lambda ]}{\left(T^{-1/2}{y}_{t+[T\lambda ]}-{\rho }^h{T}^{-1/2}{y}_t\right)}^2\\ \Rightarrow \frac{1}{1-\lambda -r}{\int }_r^{1-\lambda }{\left(M(s+\lambda )-e^{-\gamma \lambda }M(s)\right)}^{2}ds.\end{array}$$

This is just the average of the limits of the unpredictable components. It depends on the split point (r ), the forecast horizon (λ), as well as the degree of persistence, $\gamma$

If instead we impose a unit root on the forecasting model and use $f_{t+h|t}=y_t$ , we obtain

$$\begin{array}{rl}{T}^{-1}\hat{\mathbf{M}}\hat{\mathbf{S}}[\mathbf{w}]=& \frac{1}{T-T-\hslash }\frac{1}{T}\frac{{\Gamma }_{\mathrm{st}}^{-1}}{\underset{{\mathrm{LF}}_{\mathrm{st}}^{-1}+1/T}{\Vert {\mathbf{F}}_{\mathrm{st}}^1\Vert }}\frac{{\Gamma }_{\mathrm{st}}^{-1}}{T}(\mathrm{sgn}-\gamma {\mathrm{s}}^{\prime }{)}^{♯}\\ & =\frac{1}{1-|{\mathbf{F}}_{\mathrm{st}}/T|T^{\prime }-|\Delta |/T^{\prime }|T^{\prime }}\frac{1}{T}\\ & \times \sum _{u={\mathrm{i}}^{|\mathbf{F}|+1/T^{\prime }|}+1}^{\frac{T-|\Delta |}{T}}({\Gamma }^{-1/2}{y}_{\mathrm{st}(\Omega )}-{\rho }^{\mathrm{i}}{T}^{-1/2}{y}_{\mathrm{st}}+({\varphi }^{\mathrm{s}}-1)\mathrm{id}\{\xi \}{)}^{♯}\\ & =\frac{1}{1-\lambda }\sum _{u={\mathrm{i}}^{|\mathbf{F}|+1/T^{\prime }|}}{\int }_0^{-2}(\Delta (\theta +\Delta )-e^{-\mathrm{i}{\tau }^{\prime }/\lambda }M\{\xi \}{)}^2+(e^{-\gamma \top }-1{)}^{♯}d\{\xi {\}}^{♯}d\mathcal{L}\\ & =\frac{1}{1-\frac{1}{\lambda }-{\mathrm{i}}^{\prime }}{\int }_0^{-\frac{1}{\lambda }}{\left(M(\delta +3)-e^{-\mathrm{i}\tau M\{\xi \}}\right)}^{2}d\Delta \\ & \end{array}$$

This depends on the same three parameters r, λ, and $\gamma$ .

Again, we can compute the means of these limit expressions. These depend on r , the choice of sample split point, unless there is an exact unit root $(\gamma =0)$ in which case the mean is $\lambda$ for all values of $r.$ The means of these expressions are typically not the same as the means of the expressions for the same h-step-ahead forecast in a single outcome.