16.3 CONDUCTING INFERENCE ON THE OUT-OF-SAMPLE AVERAGE LOSS

16.3 CONDUCTING INFERENCE ON THE OUT-OF-SAMPLE AVERAGE LOSS

The simulated (or “pseudo”) out-of-sample (OoS) approach described in the previous section seeks to mimic the real-time updating scheme underlying most forecasts. The basic idea is to split the data into an initial estimation sample (the in-sample period) and a subsequent evaluation sample (the out-of-sample period). Forecasts are based on parameter estimates that use data only up to the date when the forecast is computed. As the sample expands, the model parameters get updated, resulting in a sequence of forecasts. Comparing realized values to these forecasts, we can compute the average loss.

West (1996) provides the main set of results that establishes conditions under which the estimated loss is both consistent and asymptotically normal under expanding (recursive), rolling, and fixed estimation windows. The assumptions involve conventional restrictions on the data-generating process, but impose further limitations on the form of the forecasting model. The analysis assumes that the initial sample for estimating ${\hat{\mathbf{\beta }}}_t$ has $T_R$ observations, and that this allows the construction of $T_P=T-T_R-h$ out-of-sample forecast errors, where $h$ is the forecast horizon. If the full sample is used for parameter estimation, each forecast is computed using ${\hat{\mathbf{\beta }}}_T$

Let $l_{t+h|t}(\beta )=L(f(z_t,\beta ),y_{t+h})$ be a vector of losses whose individual elements are denoted $l_{t+h|t,i}(\beta )$ . The observed losses depend on the parameter estimates, ${\hat{\mathbf{\beta }}}_t,$ and are denoted $l_{t+h|t}({\hat{\beta }}_t)$ . The analysis requires that there exists a pseudo-true vector, ${\beta }^{\ast }$ , for the model used to generate the forecasts; ${\beta }^{\ast }$ is the limiting value of ${\hat{\mathbf{\beta }}}_t$ as the sample size gets very large. Define $l_{t+h\mid t}({\beta }^{\ast })$ as the loss function evaluated at this pseudo-true value, while $F=E\lceil \partial l_{t+h|t}(\beta )/\partial \beta \rceil$ is the derivative evaluated at ${\beta }^{\ast }$ Also, let ${\eta }_{t+h}$ be the orthogonality conditions used to estimate the model parameters so that ${\eta }_{t+h}({\beta }^{\ast })$ is a zero-mean process. The following assumptions summarize the technical conditions needed for West’s result (West, 1996, Theorem 4.1).

1. The function $l_{t+h|t}(\beta )$ is twice differentiable in a neighborhood around ${\beta }^{\ast }$

2. The estimate ${\hat{\mathbf{\beta }}}_t$ can be written as a moment-type estimator of full rank. Specifically, ${\hat{\beta }}_t-{\beta }^{\ast }=B(t)H(t)$ , where $B(t)$ is $k\times q,H(t)$ is $q\times 1$ with $B(t{)}^{\mathrm{as}}B$ , for B of rank k; $\begin{array}{r}H(t)=(t-h{)}^{-1}{\sum }_{s=1}^{t-h}{\eta }_{s+h}({\beta }^{\ast })\end{array}$ ; and $E\left[{\eta }_s\left({\beta }^{\ast }\right)\right]=0$

3. The second derivative, $|{\partial }^2{l}_{t+h\mid t}(\beta )/\partial \beta \partial {\beta }^{\prime }|$ , is bounded.

4. The derivative, $\partial l_{t+h|t}(\beta )/\partial \beta .$ , when evaluated at the pseudo-true value ${\beta }^{\ast }$ satisfies certain mixing, moment, and stationarity conditions. Moreover, letting $\Gamma (j)=E[(l_{t+h\mid t}({\beta }^{\ast })-E[l_{t+h\mid t}({\beta }^{\ast })])(l_{t-j+h\mid t-j}({\beta }^{\ast }{)}^{\prime }$ $-E\left[l_{t-j+h|t-j}({\beta }^{\ast }{)}^{\prime }\right])]$ and $\begin{array}{r}{S}_y(0)={\sum }_{j=-\infty }^{\infty }\Gamma (j),S_y(0)\end{array}$ is assumed to be positive definite.

5. We have $T_R,T_P\infty$ and $T_P/T_R\to \pi$ . Define $\Pi =1-\ln (1+\pi )/\pi$ if $0<\pi <\infty$ and = 0 if $\pi =0$ and $\Pi =1\mathrm{if}\pi =\infty$

Under assumptions (1)–(5), West (1996) establishes that

$$T_P^{-1/2}(\sum _{t=T_R}^{T-h}{l}_{t+h|t}({\hat{\beta }}_t)-E[l_{t+h}({\beta }^{\ast })]{)}^d\mathrm{N}(0,\Omega ),$$

where, in general,

$$\Omega =S_y(0)+\Pi (FB{S}_{yh}^{\prime }(0)+S_{yh}(0)B^{\prime }{F}^{\prime })+2\Pi F{V}_{\beta }{F}^{\prime },\text{\hspace{1em}(16.14)}$$

and $\begin{array}{r}{S}_{yh}(0)={\sum }_{j=-\infty }^{\infty }E[{\eta }_t{\eta }_{t-j}^{\prime }]\end{array}$ , and $V_{\beta }$ is the limiting variance–covariance matrix of the estimators, i.e., asymptotically $T^{1/2}(\hat{\beta }-\beta ){\to }^{d}N(0,V_{\beta })$ . If either $\pi =0$ or $F=0$ , then $\Omega =S_{\nu }(0)$

The result establishes that asymptotically, we can conduct inference about forecast accuracy using standard normal distributions provided that the correct (asymptotically valid) estimate of is used. From (16.14), consists of three components: $S_y(0)$ is the long-run variance of the loss under known model parameters, the third term reflects how parameter estimation affects the variance of the loss, while the second term reflects the covariance between the first and third terms. Since the second and third terms reflect estimation error, they will depend on the underlying estimation scheme—fixed, rolling, or expanding—via .

To understand the result, it is useful to go through the heuristics of the proof. The proof works by expanding the average OoS loss around the pseudo-true parameter value, ${\beta }^{\ast }$ , at each data point. Assumption (1) in West’s theorem allows a mean value theorem expansion around ${\beta }^{\ast }$ so that, for some intermediate value, $\tilde{\mathbf{\beta }}$ ,

$$\begin{array}{rl}& l_{t+h\mid t}({\hat{\mathbf{\beta }}}_t)=l_{t+h\mid t}({\mathbf{\beta }}^{\ast })+{\frac{\partial l_{t+h\mid t}(\mathbf{\beta })}{\partial \mathbf{\beta }}}_{{\mathbf{\beta }}^{\ast }}^{\prime }({\hat{\mathbf{\beta }}}_t-{\mathbf{\beta }}^{\ast })\\ & +\frac{1}{2}({\hat{\mathbf{\beta }}}_t-{\mathbf{\beta }}^{\ast }{)}^{\prime }({\frac{{\partial }^2{l}_{t+h\mid t}(\mathbf{\beta })}{\partial \mathbf{\beta }\partial {\mathbf{\beta }}^{\prime }}}_{\tilde{\mathbf{\beta }}})({\hat{\mathbf{\beta }}}_t-{\mathbf{\beta }}^{\ast }).\end{array}\text{\hspace{1em}(16.15)}$$

The first term in (16.15) is the contribution to the average loss evaluated at the pseudo-true value, i.e., the average loss when the parameters are known. Under West’s conditions, this converges to its expected value and contributes the term $S_y(0)$ to the covariance matrix. This is similar to the usual covariance for the mean of a stationary series. The second term in (16.15) involves the term $({\hat{\mathbf{\beta }}}_t-{\mathbf{\beta }}^{\ast })$ and need not be identically equal to 0 since parameter estimates are periodically updated.4 In some cases this term is not only nonzero but does not vanish asymptotically when scaled by $T^{1/2}$ , resulting in the additional term in the variance in (16.14). The third term in (16.15) involves a quadratic in $({\hat{\mathbf{\beta }}}_t-{\mathbf{\beta }}^{\ast })$ and also does not disappear asymptotically.

In cases where the second and third terms in equal $0$ , estimation error does not contribute to the variance of the average out-of-sample loss and so the result captures only first-order effects, while ignoring terms of order $T^{-1}$ such as parameter estimation error. In such cases, in-sample and out-of-sample average loss have identical means and approximately identical variances, which makes it difficult to determine what is a better way to evaluate predictive performance in general.

One case of particular interest arises when the same loss function (e.g., squared error loss) is used for parameter estimation as well as forecast evaluation. In this case the extra terms in West’s variance drop out and so the expression simplifies a great deal.

When West’s theorem applies, $\hat{R}(f{)}^{p}E[L_{t+h}]$ , and so the result establishes that the estimated average loss is a consistent estimator for the actual average loss.

West’s result deals with inference about the expected loss evaluated at the population value of the model parameters, ${\beta }^{\ast }$ , i.e., $E\left[L_{t+h|t}({\beta }^{\ast })\right]$ . Complications arise because the inference cannot be based on the unobserved population value of the parameters but instead is based on finite sample information for which parameter estimation error can play an important role.

West’s result does not cover inference about the finite-sample predictive ability of a model which is related to the model’s expected loss evaluated at the current parameter values, $E\left[L_{t+h|t}({\hat{\mathbf{\beta }}}_t)\right]$ . This case is considered by Giacomini and White (2006).

West (1996, table 2) presents Monte Carlo results for a dynamic bivariate model with exogenous regressors. For his design $\Omega \ne S_c(0)$ , and so the variance–covariance matrix should account for parameter estimation error in the construction of the forecasts. Across a range of cases West shows that without this correction, the size of the test can be far greater than its nominal size, e.g., with rejection rates of 20–50% instead of 5%. With the correction for estimation error, size is well controlled unless the number of observations in the initial regression model is small. Specifically, the test is undersized if $T_p$ is small and oversized if $T_p$ is relatively large.