17.8 CHOICE OF SAMPLE SPLIT

17.8 CHOICE OF SAMPLE SPLIT

White’s Reality Check bootstrap approach can be used to address the effect on inference of searching for a superior model across many different model specifications. Another dimension along which forecasters could have searched for superior performance is the sample split, i.e., the decision how to split a sample with $T=T_R+T_P$ observations into an initial estimation and model selection period that uses the first $T_R$ observations, and a forecast evaluation period that uses the remaining $T_p$ observations. Instead of studying the specific starting date for the forecast evaluation period, $T_R$ , we can equivalently consider $\lambda =T_R/T\in (0,1)$ which is the fraction of the sample used for initial estimation. To obtain well-behaved test statistics, it is common to assume that the choice of sample split $\lambda$ is limited to an interval $\lambda \in [\underline{\lambda },\bar{\lambda }]$ , typically leaving out 10–15% of the sample on each side. This ensures sufficient data for initial model estimation and for subsequent forecast evaluation.

Hansen and Timmermann (2012) study the size of tests such as (17.28) when the $p\cdot$ -value is chosen as the smallest value associated with all possible split points, i.e.,

$$p_{\min }=\underset{\lambda \in [\underline{\lambda },\bar{\lambda }]}{\min }{P}_{\lambda }^{-1}\left(\frac{D_T(\lambda )}{{\hat{\sigma }}_{\epsilon }^2}\right),\text{\hspace{1em}(17.46)}$$

where $D_T(\lambda )$ is defined in (17.25), ${\hat{\sigma }}_{ϵ}^2$ is a consistent estimate of the variance of the forecast errors, $P_{\lambda }^{-1}$ is the inverse of the c.d.f. of the test statistic $D_T(\lambda )/{\hat{\sigma }}_{\epsilon }^2$ . This is a quantile function and so gives the p-value of the test for a given value of λ. A single randomly selected $p.$ -value will follow a uniform distribution but this is no longer the case when we consider the smallest $p\cdot$ value selected from a potentially large number of test statistics. Overlaps in the estimation and evaluation windows used by the tests, $t=T_R,T_R+1,\dots ,T$ , introduce further complications since the individual $p\cdot$ -values will be strongly correlated. Thus, if the reported $p\cdot$ -value is the smallest value chosen from many possible sample splits, conventional inference will no longer be valid.

Hansen and Timmermann show that conventional tests of the predictive accuracy of nested models that account for nonstandard limiting distributions but ignore the effect of sample split mining can be greatly oversized. For example, with one additional predictor variable in the large model, a test with a nominal size of 5% can actually reject close to 15% of the time as a result of such sample split mining. Inflation in the size of the test grows even larger when the number of additional predictor variables in the large forecasting model is increased.

As a constructive way to account for the effect of sample split mining on inference, Hansen and Timmermann (2012) suggest an adjusted minimum $p\cdot$ -value test corresponding to the statistic in (17.46). The test is based on the $P_{\min }$ statistic whose critical values can be derived using Monte Carlo simulations of the asymptotic distribution. Hansen and Timmermann compute adjusted $\mathbf{p}$ -values from the quantiles of the sorted distribution of $P_{\min }$ -values simulated from a large number of sample paths.

For example, for a test with a critical level of 5% and one additional predictor variable, the $P_{\min }$ -value adjusted for sample split mining in the interval $\lambda \in [0.1,0.9]$ would have to fall below 1.3% for there to be significant evidence that the larger model dominates the small model. This result is intuitive: if only a single $p\cdot$ -value had been inspected, there would be no distortion to inference, and we could simply use the critical values reported by McCracken (2007). On the other hand, if the reported p-value is pre-selected from a large set of candidate p-values, then we require stronger evidence—a larger test statistic and a smaller p-value—to be confident in the strength of the evidence.

A similar message comes out of Rossi and Inoue (2012) who consider test statistics computed either from averages or from the maximum of test statistics across different sample splits in a manner analogous to Hansen and Timmermann (2012). Rossi and Inoue also provide sample split mining-robust test statistics and show through Monte Carlo simulations that conventional tests that ignore such mining can lead to large size distortions.