16.4 OUT-OF-SAMPLE ASYMPTOTICS FOR RATIONALITY TESTS

16.4 OUT-OF-SAMPLE ASYMPTOTICS FOR RATIONALITY TESTS

West and McCracken (1998) develop asymptotic results that facilitate inference on regressions of the form

$$e_{t+h|t}({\hat{\mathbf{\beta }}}_t)=\gamma v_t({\hat{\mathbf{\beta }}}_t)+u_{t+h},\text{\hspace{1em}(16.17)}$$

where both the right- and left-hand-side variables can depend on the sequence of parameter estimates from a forecasting model, ${\hat{\mathbf{\beta }}}_t$ . Examples of such regressions include many of the bias and orthogonality regressions discussed in chapter 15. For example, $e_{t+h|t}({\hat{\mathbf{\beta }}}_t)$ could be the first derivative of the loss function evaluated each period, where the forecasts depend on estimated parameters, ${\hat{\mathbf{\beta }}}_t$ . In such a regression $v_t({\hat{\beta }}_t)$ could be the lagged forecast errors (such as in a weak exogeneity test) which would also depend on the estimated model parameters. The setup in (16.17) is general and allows either the right- or left-hand-side variable to be independent of ${\hat{\mathbf{\beta }}}_t$ . For in-sample regressions where the parameters, ${\hat{\mathbf{\beta }}}_t^{}$ , are estimated once over the entire sample we know that standard methods apply when $v_t$ is independent of $\beta$ . In such cases corrections must typically be made for the standard errors when the regressor, $v_t(\beta )$ , is a function of estimated parameters; this is known as the generated regressor problem; see Pagan (1984).

West and McCracken (1998) extend the standard results to the forecasting problem by explicitly considering how the forecasts were generated and how this could impact inference on $\hat{\gamma }$ from the regression (16.17). We will use the notation $e_{t+h|t}$ for the dependent variable since in most applications the dependent variable is the forecast error. However, the definition of $e_{t+h|t}$ can be extended to include other functions of the forecast errors. West and McCracken (1998, Theorem 4.1) gives results for the case with general regression functions and multistep forecasts. We list the essential assumptions of this result below.

1. The variables $e_{t+h|t}(\beta )$ and $v_t(\beta )$ are measurable and twice differentiable in a neighborhood around ${\beta }^{\ast }$ . Moreover, $E\left[e_{t+h|t}({\beta }^{\ast })v_t({\beta }^{\ast })\right]=0;$ $E\left[e_{t+h|t}({\beta }^{\ast })\partial e_{t+h|t}({\beta }^{\ast })/\partial {\beta }^{\ast }\right]=0;E\left[e_{t+h|t}({\beta }^{\ast })\partial v_t({\beta }^{\ast })/\partial {\beta }^{\ast }\right]=0;$ and $E\left[v_t(\beta )v_t(\beta {)}^{\prime }\right]$ has full rank k.

2. The estimate ${\hat{\mathbf{\beta }}}_t$ takes the form of a moment-type estimator of full rank: ${\hat{\beta }}_t-{\beta }^{\ast }=B(t)G(t)$ , where $B(t)$ is $k\times q,G(t)$ is $q\times 1$ with $B(t{)}^{\mathrm{as}}B$ , B of rank $k;$ under recursive estimation $G(t)=t^{-1}\sum g_s({\beta }^{\ast })$ with $E\left[g_s({\beta }^{\ast })\right]=0$

3. The second derivative $\left|{\partial }^2{e}_{t+h|t}(\beta )/\partial \beta \partial {\beta }^{\prime }\right|$ is bounded.

$$w_t=((\partial {\dot{e}}_{t+h|t}(\beta )/\partial \beta {)}^{\prime },\mathrm{vec}(\dot{\partial }{v}_t(\beta )/\partial \beta {)}^{\prime },e_{t+h|t}(\beta {)}^{\prime },v_t(\beta {)}^{\prime },g_t^{\prime }{)}^{\prime }$$

Moreover, $\begin{array}{r}{S}_c(0)={\sum }_{i=-\infty }^{\infty }\Gamma (j)\end{array}$ is positive definite, where

$\Gamma (j)=E\left[c_{t+h|t}({\beta }^{\ast })c_{t-j+h|t-j}^{{}^{\prime }}({\beta }^{\ast }{)}^{\prime }\right]$ and $c_{t+h|t}(\beta )=e_{t+h|t}(\beta )v_t(\beta )$

5. We have $T_P,T_R\infty$ and $T_P/T_R\to \pi$ , where $\left(\mathtt{a}\right)0\le \pi \le \infty$ under an expanding estimation window; $(♭)0\le \pi <\infty$ for rolling and fixed windows.

Under these assumptions, West and McCracken show that

$$T^{1/2}\hat{\gamma }{\to }^d\mathrm{N}(0,V),$$

where

$$V=E[X_t{X}_t^{\prime }]\Omega E[X_t{X}_t^{\prime }],$$

$$\Omega =S_c(0)+{\lambda }_{cg}(FB{S}_{cg}^{\prime }(0)+S_{cg}(0)B^{\prime }{F}^{\prime })+{\lambda }_{gg}F{V}_{\beta }{F}^{\prime }.$$

This result is quite general, which explains the cumbersome notation. The theorem uses the notation

$$\begin{array}{c}F=E\partial c_{t+h\mid t}(\beta )/\partial \beta ,\\ \\ {\Gamma }_{cg}(j)=E\left[c_{t+h\mid t}({\beta }^{\ast })g_{t-j}^{\prime }\right],\end{array}$$

and so

$$S_{cg}(0)=\sum _{j=-\infty }^{\infty }{\Gamma }_{cg}(j),{\Gamma }_{gg}(j)=E\left[g_t{g}_{t-j}^{\prime }\right],$$

$$S_{gg}(0)=\sum _{j=-\infty }^{\infty }{\Gamma }_{gg}(j),V_{\beta }=B{S}_{gg}(0)B^{\prime }.$$

The definition of $\left({\lambda }_{cg},{\lambda }_{gg}\right)$ depends on the method used for constructing the forecasts:

$$\begin{array}{rlrc}& \frac{\mathrm{Estimation}\mathrm{window}}{\mathrm{Expanding}\mathrm{window}}& 1-\ln (1+\pi )/\pi & 2(1-\ln (1+\pi )/\pi )\\ & \mathrm{Rolling}\mathrm{window}\left(\pi \le 1\right)& \pi /2& \pi -{\pi }^2/3\\ & \mathrm{Fixed}\mathrm{window}& 0& \pi \end{array}$$

Inference on regressions of the form (16.17) commonly assumes that the coefficient estimates $\hat{\gamma }$ are asymptotically normal and use robust standard error methods to construct hypothesis tests. This is a valid approach provided that $\Omega =S_y(0)-\mathrm{a}$ situation that only applies to special cases. More generally, the variance–covariance matrix must be corrected according to (16.19).

The leading case where one can proceed with the usual standard errors is when the forecasts are generated from the recursive estimation scheme and the residuals of orthogonality regressions such as (16.17) are conditionally homoskedastic. Assuming an expanding estimation window, it can be established that $S_c(0)=-FB{S}_{cg}^{\prime }(0)=$ $-S_{cg}(0)B^{\prime }{F}^{\prime }=F{V}_{\gamma }{F}^{\prime }$ , and the second and third terms of (16.19) sum to 0 since ${\lambda }_{cg}={\lambda }_{gg}/2$ . For the rolling window estimation scheme, these terms sum to $-{\pi }^2/3$ and so the variance–covariance matrix is $(1-{\pi }^2/3)S_c(0)$ . For this case, conventional tests based on $S_c(0)$ will use too large a variance–covariance matrix and hence will be undersized. The reverse happens for a fixed window for this model, resulting in oversized tests. It is also true for all models that if $\pi =T_P/T_R=0$ , so that asymptotically the proportion of the sample used for forecast evaluation $\left(T_P\right)$ is overwhelmed by the sample used to construct the initial model estimates $\left(T_R\right)$ , conventional methods for inference apply since the effect of estimation error vanishes.

As for the results in West (1996), upon which these results build, there is a large number of assumptions that limit the applicability of the results in (16.18) and (16.19). The relevant functions of the parameters of the forecast model must be twice differentiable at $\beta .$ . This rules out some loss functions although for orthogonality regressions under MSE loss, the left-hand side of (16.17) becomes the forecast error which is linear in the forecast and hence this condition restricts only the form of the forecasting model itself. The second assumption is the same as in West (1996) and restricts attention to forecasting models of the least squares type (including GMM) but rules out many shrinkage estimators as well as Bayesian, semiparametric, and nonparametric estimators. The third and fourth assumptions require moment and dependence conditions on the data which rule out many nonstationarities in the data.

Assumption (1) requires that the population value for $\gamma$ is 0 which explains the lack of centering of $\hat{\gamma }$ in the theorem. This means that the result is constructed under only the null hypothesis that $\gamma =0$ and that the instruments $v_t$ are not useful in predicting $e_{t+h|t}$ . This is the standard form of the null hypothesis in orthogonality tests. Results are not available under the alternative hypothesis.

Very few results are available for more complicated loss functions and alternative forecasting techniques. A common approach is to simply ignore variation due to how the forecast is constructed. However, the theoretical results above show that this is not necessarily an appropriate course of action. Nonetheless, the results in West and McCracken (1998)) allow us to construct confidence intervals for commonly used out-of-sample estimates of average loss.