6.10 CONCLUSION

6.10 CONCLUSION

When applied to forecasting problems, model selection procedures are best thought of as more complicated estimators rather than as a separate step in the construction of the prediction model. Model selection simply increases the set of functions over which the search for the best forecasting model is conducted.

This view contrasts with how model selection methods are often portrayed in the forecasting literature. In many instances a particular set of models might be chosen as providing the “best” forecasts, and model selection is introduced as if it has no effect on the original analysis. Such an approach has its roots in consistent model selection, the notion that the model selection step can be ignored since asymptotically the correct model is chosen with probability 1. This argument can be very misleading, however. First, it relies on a pointwise argument and does not hold uniformly, i.e., for values of the unknown “true” parameters that are close to the dominant model. Second, such results rely on the—frequently invalid—assumption that a correct true or pseudo-true model is included in the set over which the model specification search is conducted.

The effects of model selection on the risk function are typically of the same order as that of parameter estimation. Risk functions, though, are generally very hard to examine analytically. Viewed as estimators, risk functions are often very complicated functions of the underlying data, and generally nonsmooth due to the presence of data-dependent indicator functions.

We use Monte Carlo methods to examine the general effect of model selection on the risk function. When one model is clearly superior it will nearly always be selected and hence the risk function for the model selection method is equivalent to that for the superior model as if this model were known to be valid. This explains the popularity of model selection methods in applied work. However, there is no free lunch—when a single model is not obviously superior to all other models, model selection methods result in a risk function that is a probability weighted average across all models. In such cases, there is typically a region of the parameter space where the risk associated with the forecasts based on model selection procedures is greater than the risk of any of the models near the optimal model. Such a situation is often highly empirically relevant since statistical techniques for model selection are employed precisely because models are hard to tell apart, and not because one model is obviously superior.

6.11 APPENDIX: DERIVATION OF INFORMATION CRITERIA

This appendix provides a derivation of the Schwarz and Akaike information criteria.