5.5 CONCLUSION

5.5 CONCLUSION

The Bayesian approach to forecasting is becoming increasingly popular in applied forecasting. Bayesian forecasting models build directly on standard Bayesian foundations for model construction. In particular, when the model is known up to a finite set of unknown parameters, the posterior distribution for the predicted outcome given the data is easily derived from the posterior distribution of the model parameters. As with the construction of posterior distributions for parameters, only the simplest situations lead to closed-form solutions for the posterior distribution of the outcome given the observed data. This means that numerical methods are at the forefront of most Bayesian applications to forecasting. Further details of these methods are provided in chapter 9.

Model Selection

Only a very optimistic forecaster trusts that his model is correctly specified andincludes all variables that are relevant for forecasting. This underscores the idea that a set of models, rather than just a single model, might reasonably be considered in constructing a forecast of a particular outcome. Models could differ by their dynamic specification given a set of predictor variables, in which case model selection simplifies to lag length selection. Models could use different subsets from a long list of potential predictor variables, in which case model selection involves choosing which covariates to include. Alternatively, models could vary in their choice of the functional form mapping predictor variables to the outcome, as in the choice between a linear and nonlinear model, or the selection of the order of approximation in a sieve estimator. With many possible forecasting models to be considered, it is natural to ask whether a single “best” model can be identified. Model selection, or subset regression methods, attempt to choose such a “best” model, where “best” typically depends on the forecaster’s loss function, the underlying data-generating process, the forecaster’s information set and the forecast horizon.

To help analyze the problem, suppose there is a finite set of forecasting models, ${\mathcal{M}}_K$ , with each model denoted $M_k$ for k = 1, . . . , K . Each model is used to generate a forecast, $f_k(z_t,{\beta }_k)$ . The model selection problem is related to the classical estimation problem analyzed in chapter 4, the difference being that the minimization of expected loss now involves searching over ${\mathcal{M}}_K$ to find the best forecasting model. Hence, the problem of searching for the best forecast model is extended beyond merely estimating the parameters of a single model to searching over a restricted set of functions as well.1

The list of candidate models could be nested, but does not have to be. If all models are special cases of a “super model” that nests everything else, the distinction between model selection and parameter estimation becomes somewhat blurred. In this case, individual models arise as zero-coefficient constraints on the nesting model’s parameters.

Model selection methods can be split into two broad sets: those that use the full data sample to choose the forecasting model (labeled “in-sample” methods) and those that use a “holdout” sample to choose between models (“out-of-sample”

methods).2 This chapter focuses on in-sample methods as opposed to out-of-sample methods. We make this distinction here mainly because the methods and motivation for the two methods differ, so it is easier to analyze them separately. Out-of-sample methods are examined in chapter 16.

Popular in-sample model selection methods include information criteria (IC), sequential hypothesis testing, and various forms of cross validation. For largedimensional models the Lasso and variations on this method have gained widespread use. How well the approaches apply to a given situation depends on whether (a) there are few or very many potential predictor variables—in the latter case, methods relying on exhaustive searches will not be feasible; (b) there are few or very many useful predictors among these potential predictor variables. A model is considered sparse if only a few predictors are useful, in which case methods that asymptotically overfit can provide poor forecasts. Leeb and Pötscher (2005) provide a general discussion of issues related to model selection and inference.

Section 6.1 briefly discusses the trade-offs faced by different model selection approaches. Next, we cover specific model selection methods such as sequential hypothesis testing (section 6.2), information criteria (section 6.3), cross validation (section 6.4), the Lasso method (section 6.5), and various hard and soft threshold methods such as Bagging (section 6.6). Section 6.7 provides an empirical application to predictability of stock market returns and section 6.8 covers properties of model selection methods. Monte Carlo methods are used to analyze the risk of a variety of methods in section 6.9, before we conclude in section 6.10. Section 6.11 contains technical derivations used throughout the chapter.