6.4 CROSS VALIDATION

6.4 CROSS VALIDATION

Cross validation is often used for model selection in conjunction with series estimators (see chapter 11) but is also applicable to model selection more generally. The idea of cross validation is to avoid overfitting a model by removing the correlation that causes the estimated in-sample loss to be “small” due to the use of the same observations for both parameter estimation and model evaluation purposes. In contrast to out-of-sample model selection methods discussed in chapter 16, cross validation makes use of the entire data set for both estimation and evaluation. The method averages over all possible combinations (or possibly a subset of these) of estimation and evaluation samples obtainable from a given data set.

The most common cross-validation approach is the ${\mathsf{\Pi }}^{\mathrm{ec}}\mathrm{leave}-T_v-{\mathrm{out}}^{3\gamma }$ crossvalidation estimator developed for independent data. This method holds out $T_v=$ $2v+1$ observations for model evaluation, with the remaining observations used for parameter estimation. The loss is calculated solely for each evaluation sample and gets repeated for all possible sets of $T_v$ data points used for model evaluation. In situations where the data are i.i.d., this procedure results in a very large number of possible evaluation sets and can be computationally very costly. For example, there are $T!/((T-T_v)!T_v!)$ possible ways to split $T$ observations into different evaluation $(T_v)$ and estimation $(T-T_v)$ samples. For serially dependent time series the holdout samples are often constrained to be contiguous observations, which makes the estimation of average loss far simpler since there are only $T-T_v$ ways to split the sample. For each observation, t, the loss is evaluated on the evaluation sample $[t-v;t+v]$ and averaged across all values of $t,$ omitting ends of the sample where there is no data after removing observations. The cross-validated estimator of MSE loss is

$${\mathrm{CV}}_v=\frac{1}{T-2v-1}\sum _{t=v+1}^{T-v}\{\frac{1}{2v+1}\sum _{s=t-v}^{t+v}(y_{s+1}-f(z_s,{\hat{\beta }}_{\{t-v:t+v\}}){)}^2\},\text{\hspace{1em}(6.18)}$$

where ${\hat{\beta }}_{\{a:b\}}$ is the estimator for $\hat{\beta }$ that uses all of the sample apart from the observations from a to b with $0<a<b<T$ . Model selection proceeds by choosing the model, ${\hat{M}}_k$ , that minimizes CV across all models $M_k\in {\mathcal{M}}_K$

When data are serially independent, the usefulness of the approach comes from the fact that the parameter estimates will be independent of the evaluation sample. This breaks the connection that leads to overfitting the objective function. In most forecasting situations it is not reasonable to assume serial independence of the forecast errors, perhaps because some of the models are incorrectly specified. In addition, $z_t$ may include lags of ${\gamma }_t$ which would induce correlation between the estimation and evaluation samples. This necessitates removing enough observations around each estimation date to ensure that the correlation between the estimation and evaluation sample is close to 0. Burman, Chow, and Nolan (1994) and

Racine (2000) propose methods to accomplish this. We briefly describe Racine’s ${\ell }^{\ast }v.$ block cross-validation approach.

Racine (2000) proposes to omit ${\ell }^{\ast }$ observations on each side of the validation sample in order to counter any dependence between the estimation and evaluation steps. As with the ${\omega }^{\mathrm{cc}}\mathbf{leave}-n_v-{\mathbf{out}}^{\mathfrak{n}}$ cross-validation method, a window of v observations is used before and after time t, resulting in a validation sample of $2v+1$ observations to average over for each t. This means that the forecast $f(z_t)$ is based on all observations apart from a block of observations running from $\begin{array}{r}s=(t-{\ell }^{\ast }-v)\tan s=(t+{\ell }^{\ast }+v)\end{array}$ . Under MSE loss, the cross-validated average loss statistic is

$$\begin{array}{rl}& \mathrm{CV}=\frac{1}{T-2v-1-2{\ell }^{\ast }}\\ & \times \sum _{t={\ell }^{\ast }+v+1}^{T-v-{\ell }^{\ast }}\frac{1}{2v+1}\sum _{s=t-v}^{t+v}(y_{s+1}-f(z_s,{\hat{\beta }}_{\{t-{\ell }^{\ast }-v:t+{\ell }^{\ast }+v\}}){)}^2.\end{array}\text{\hspace{1em}(6.19)}$$

The method requires choosing two parameters, ${\ell }^{\ast }$ and v. One rule suggested by Racine is to set ${\ell }^{\ast }=0.25T$ and use an evaluation sample equal to the integer part of $T^{1/2}$ .

Example 6.4.1 (MSE loss for sample estimate of the mean). Suppose the sample mean, ${\bar{y}}_T=T^{-1}{\sum }_{t=1}^T{y}_t$ is used to forecast an i.i.d. time series, ${\gamma }_t,$ in a sample with T observations. If we use the same data sample to estimate the sample mean and evaluate the average MSE loss, we have

$$\begin{array}{c}{T}^{-1}\sum _{t=1}^T(y_t-{\bar{y}}_T{)}^2=T^{-1}\sum _{t=1}^T((y_t-\mu )-({\bar{y}}_T-\mu ){)}^2\\ \\ =T^{-1}\sum _{t=1}^T{\epsilon }_t^2+({\bar{y}}_T-\mu {)}^2-2({\bar{y}}_T-\mu )T^{-1}\sum _{t=1}^T(y_t-\mu )\\ \\ =T^{-1}\sum _{t=1}^T{\epsilon }_t^2-({\bar{y}}_T-\mu {)}^2.\end{array}\text{\hspace{1em}(6.20)}$$

Hence the squared in-sample estimation error, $({\bar{y}}_T-\mu {)}^2$ , gets subtracted from, rather than added to, the average in-sample loss. This happens because of the cross-product term which is expected to equal 0 in the out-of-sample period but is equal to $-({\bar{y}}_T-$ $\mu {)}^2$ . Cross validation attempts to remove this cross-product term by breaking the correlation between the forecast error and the estimation error induced in the insample period. It does so by separating observations used to estimate the parameters of the prediction model (here the sample mean) from observations used to evaluate the average loss.

In a setting with independently and identically distributed data, the classic crossvalidation method is to “leave-one-out.” Although this method is not particularly relevant for forecasting, it is helpful for illustrating how the method can be used to remove the cross-product term.

Example 6.4.2 (Cross validation for sample mean). For each point in time, t, suppose the sample mean ${\bar{\mathbf{y}}}_{\{-t\}}$ is used as the estimate, where $\begin{array}{r}{\bar{y}}_{\{-t\}}=(T-1{)}^{-1}{\sum }_{s=1,s\ne t}^T{y}_s\end{array}$ $i.e.,$ , we leave out observation ${\gamma }_t$ when estimating the sample mean at time t. This term, by virtue of omitting ${\gamma }_{t.}$ , is now independent of ${\dot{y}}_t$ . Specifically, we have

$$T^{-1}\sum _{t=1}^T(y_t-{\bar{y}}_{\{-t\}}{)}^2=T^{-1}\sum _{t=1}^T{\epsilon }_t^2+\sum _{t=1}^T({\bar{y}}_{\{-t\}}-\mu {)}^2-2T^{-1}\sum _{t=1}^T({\bar{y}}_{\{-t\}}-\mu ){\epsilon }_t.\text{\hspace{1em}(6.21)}$$

From the independence assumption, the cross-product term now has zero expectation and tends to be “small.” The cost of “leaving-one-out” is that the second term in (6.21) differs from the value we would use in the forecast, ${\bar{y}}_t$ . This results in a slight difference in the expected loss, or, asymptotically, a different second-order term in the expected average loss. To see this, note that if we rearrange (6.21) and ignore terms that are 0 in expectation, we have

$$E\left[T^{-1}\sum _{t=1}^T(y_t-{\bar{y}}_{\{-t\}}{)}^2\right]=T^{-1}\sum _{t=1}^{T}E\left[{\epsilon }_t^2\right]+\sum _{t=1}^{T}E\left[({\bar{y}}_{\{-t\}}-\mu {)}^2\right].\text{\hspace{1em}(6.22)}$$

Now,

$$E\left[({\bar{y}}_{\{-t\}}-\mu {)}^2\right]=(T-1{)}^{-2}E\left[\sum _{\begin{array}{c}s=1,s\ne t\end{array}}^T{\epsilon }_s^2\right]=\frac{{\sigma }^2}{T-1},\text{\hspace{1em}(6.23)}$$

and so

$$E[({\bar{y}}_{\{-t\}}-\mu {)}^2]={\sigma }^2\left(1+\frac{1}{T-1}\right).\text{\hspace{1em}(6.24)}$$

If instead we used the usual forecast, ${\bar{y}}_t.$ , the expected average loss would be ${\sigma }^2(\stackrel{\cdot }{1}+T^{-1})$ , the difference being due to our use of one less observation. This bias increases with the size of the prediction model; for example in a linear regression with $n_k$ regressors the bias of the leave-one-out loss would be ${\sigma }^2(T-n_k{)}^{-1}$ instead o $\bar{\tau }{\sigma }^2(T-\bar{1}{)}^{-1}$ . The difference between the estimated loss and the true loss is, however, of a lower order than that for in-sample evaluation.