6.2 SEQUENTIAL HYPOTHESIS TESTING

6.2 SEQUENTIAL HYPOTHESIS TESTING

Sequential hypothesis testing is perhaps the most natural approach to building a forecasting model and describes how—at least informally—many researchers construct their model. As a model selection technique the method chooses the “best” submodel from a larger set of available models through a sequence of specification tests that attempt to identify the relevant parts of a model and exclude the remainder. One strategy is to remove variables (or terms) from the model found not to be useful when tested against a smaller model that omits such variables. A great variety of methods have been proposed to this end. For example, t-tests, F -tests, or p-values can be employed to exclude redundant terms. Different orders of the sequence in which variables are tested—forward stepwise and backward stepwise being the most common—can also be considered. Finally, different cross-checks can be employed on the omitted terms in a way that mixes forward and backward stepwise methods.

The two main approaches in the sequential testing literature are general-to-specific and specific-to-general methods. General-to-specific methods include all potential variables in the initial model and then remove variables deemed not to be useful through a sequence of tests. Specific-to-general methods instead begin with a small baseline model, comprising, e.g., the main variables thought to forecast well or simply a constant, and then add further variables if they appear to improve the prediction model. The final selected model will generally depend on the sequence of tests.

Example 6.2.1 (General-to-specific modeling approach). As an illustration of the general-to-specific approach, consider a linear forecasting model with K potential predictor variables $\{x_1,\dots ,x_K\}$ . Suppose least squares is initially used to estimate the “kitchen sink” model with all predictors included:

$$y_{t+1}={\beta }_0+{\beta }_1{x}_{1t}+{\beta }_2{x}_{2t}+\cdot \cdot \cdot +{\beta }_{K-1}{x}_{K-1,t}+{\beta }_K{x}_{Kt}+{\epsilon }_{t+1}.\text{\hspace{1em}(6.5)}$$

We can adopt a unidirectional backward stepwise approach that ranks the predictor variables in terms of the t-statistic of their coefficients in (6.5) or (equivalently) by the p-value of these coefficient estimates. Suppose the smallest absolute value of the t-statistic among all variables falls below some threshold, t, such as $t=2.$

$$t_{\min }=\underset{k=1,\dots ,K}{\min }|t_{{\hat{\beta }}_k}|<\underline{t}.\text{\hspace{1em}(6.6)}$$

Then the variable with the smallest t-statistic (or largest p-value) gets eliminated from the model. Assuming for simplicity that $x_K$ is the eliminated variable, the trimmed model with the remaining $K-1$ variables is next reestimated:

$$y_{t+1}={\beta }_0+{\beta }_1{x}_{1t}+{\beta }_2{x}_{2t}+\cdot \cdot \cdot +{\beta }_{K-1}{x}_{K-1,t}+{ϵ}_{t+1}.\text{\hspace{1em}(6.7)}$$

Once again, we compute the t-statistics for all remaining regressors in this model, check whether mi $\begin{array}{r}{1}_{k=1,\dots ,K-1}\{|t_{{\hat{\beta }}_k}|\}<\underline{t},\end{array}$ and drop the variable with the smallest t-statistic if this condition holds. This procedure is repeated until the t-statistics of all variables in the model exceed t or until some maximum number of iterations, R, is reached.

For the backward stepwise (general-to-specific) approach, the forecast takes the form (assuming that a constant is always included)

$$f_{t+1|t}={\hat{\beta }}_{0k}+\sum _{k=1}^K{\hat{\beta }}_k{1}_k(d(|t_{{\hat{\beta }}_j}^r|),j\in [1,\dots ,K],r\le R)x_{kt},\text{\hspace{1em}(6.8)}$$

where $t_{{\hat{\beta }}_k}^r$ is the t-statistic for ${\beta }_k$ in the r th variable selection round, $r=1,\dots ,R$ , and $k=1,\dots ,K$ . Here $d_k(|t_{{\hat{\beta }}_j}^r|)$ determines whether the kth variable gets eliminated in any of the R steps. The indicator variable $1_k(d_k(|t_{{\hat{\beta }}_i}^r|),j\in [1,\dots ,K],r\le R)$ equals 1 if the kth variable is included in the final model. Whether a variable gets excluded depends not only on its own t-statistic but also on the t-statistics of all other presently included variables. This means that $1(d_k(|t_{{\hat{\beta }}_i}^r|),j\in [1,\dots ,K],r\le R)$ depends on the entire sequence of (absolute values of) t-statistics not only for the kth variable itself but also for all other variables. For example, a variable may not be excluded in a given round even if its test statistic falls below the threshold, provided that the test statistic of another contemporaneously included variable is even lower. This path dependence makes it difficult to establish analytical results for sequential model selection methods.

The so-called “LSE” general-to-specific approach starts with a general model and adds tests for model “congruence” to the sequence of tests. This can include tests for normality, autocorrelation, and conditional heteroskedasticity in the model residuals, but may also involve (out-of-sample) model stability tests; see the discussion in Hoover and Perez (1999). Insignificant predictors are kept in a model if their omission causes the model to fail one or more diagnostic tests. The aim is to ensure that the model residuals are martingale difference sequences with respect to the underlying set of variables considered in the search. A second aim is to obtain a model that is not encompassed by any competing specification, in the sense that no other model adds predictive content beyond the selected model. Hendry (1995) provides a detailed discussion of these issues and references to the literature.

The specific-to-general approach often begins from a simple model that only includes an intercept,

$$y_{t+1}={\beta }_0+{\epsilon }_{t+1}.$$

Under the unidirectional forward stepwise approach, each of the K univariate models is next considered separately:

$$y_{t+1}={\beta }_{0k}+{\beta }_k{x}_{kt}+{\epsilon }_{kt+1},k=1,\dots ,K.\text{\hspace{1em}(6.9)}$$

Suppose that the highest of these t-statistics exceeds some threshold value $\bar{t}:$

$$t_{\max }=\underset{k=1,\dots ,K}{\max }|t_{{\hat{\beta }}_k}|>\overline{t}.\text{\hspace{1em}(6.10)}$$

Then the variable associated with this t-statistic is included in the model. Next, regressors from the remaining pool are added, one by one, to this univariate model. A new regressor is included provided that its t-statistic exceeds t¯, once added to the prediction model. The process continues until the t-statistics of all of the remaining excluded variables fall below the cutoff value, t¯, or until a maximum number of iterations is exceeded.

Forecasts from the forward stepwise approach takes the same form as in (6.8), the only difference being how the inclusion indicator $1_k(d_k(|t_{{\hat{\beta }}_j}^r|),j\in [1,\dots ,K]$ $r\le R)$ is constructed.

Several variations of sequential tests exist. Blocks of variables instead of individual variables can be considered at each step, using F -tests instead of t-tests to evaluate statistical significance. Such tests might be relevant in the presence of strongly correlated regressors. A bidirectional mixture of forward and backward steps can also be employed. Correlation among regressors mean that the inclusion of a new variable in a forward step could lead some previously significant variables to become insignificant in the extended model. These variables could be eliminated in a backward step. Convergence of such procedures is not guaranteed, however.

The main benefit of the sequential approach is its intuitive appeal: its simplicity and the fact that it is computationally simple to implement—particularly if variables are considered for inclusion or exclusion one at a time. However, the approach does not undertake a comprehensive search across all possible models, its outcome can be path dependent, and so the method suffers from the drawback that there is no guarantee that it finds the globally optimal model, i.e., the best possible model within the feasible model set.

Properties of sequential variable selection approaches have been addressed in many studies. Hoover and Perez (1999) undertake a thorough simulation study of the general-to-specific approach. They find that this approach frequently ends up with overparameterized models that include variables that are spuriously significant in the sense that they are not included in the true model. Insignificant variables are sometimes included to allow the models to pass the residual specification tests. Interestingly, their simulations suggest that these issues can be addressed in part by setting the bar higher for inclusion of variables, using smaller nominal sizes (or higher critical values) in the inclusion tests.