6.3 INFORMATION CRITERIA

6.3 INFORMATION CRITERIA

Information criteria (IC) choose models by trading off model fit against a penalty for model complexity as measured by the number of free parameters that have to be estimated for the models. Several information criteria have been suggested in the literature, the two most popular of which are the Bayes information criterion (Schwarz BIC, SBIC, or BIC) proposed by Schwarz (1978) and the Akaike information criterion (AIC) due to Akaike (1974). Less often used methods include the Mallows $C_p$ criterion due to Mallows (1973) and the Hannan–Quinn criterion due to Hannan and Quinn (1979). Finally there is a large literature on variations of these methods and on other forms of information criteria. This plethora of criteria arises because of differences in the properties possessed by the information criteria, as we discuss below. BIC, AIC, and the Mallows $C_p$ all attempt to adjust a minimization criterion for the effect of parameter estimation, which tends to make more highly parameterized models appear better in-sample than they really are. The extent to which parameter estimation improves the in-sample fit depends on the true model, so differences across information criteria hinge on how to practically get around our fundamental lack of knowledge of the “true” forecast model.

Information criteria employ different strategies to trade-off fit against parsimony. The Bayesian information criterion selects the model with the highest posterior probability given the data. To choose a single model from the candidate set, the models are ranked according to their posterior probabilities and the model with the highest posterior probability is chosen. The Akaike information criterion seeks to minimize the (Kullback–Leibler) distance between the density of a candidate model and the density of the true (unknown) model.

Consider a set of K parametric models, ${\mathcal{M}}_k$ , where each model $M_k\in {\mathcal{M}}_K$ $k=1,\dots ,K$ requires estimating $n_k$ parameters, ${\beta }_k$ . The BIC and AIC choose the model (indexed by k) to minimize expressions of the form

$$\begin{array}{r}\mathrm{IC}(k)=-2T^{-1}\ln p({\hat{\beta }}_k|z)+h(n_k)g(T),\end{array}\text{\hspace{1em}(6.11)}$$

over all models, $k=1,\dots ,K$ , where $p({\hat{\beta }}_k|z)$ is the likelihood of the data evaluated at the parameter estimates, $h(n_k)$ is an increasing function of the model size, $n_k$ , and $g(T)$ is a decreasing function of the sample size, T . The second term, $h(n_k)g(T)$ penalizes for the estimation of additional parameters (larger $n_k)$ , particularly in small sample sizes.

Information criteria differ over their choice of $h(n_k)g(T)$ . For BIC and AIC the penalties take the form

$$\mathrm{BIC}:\frac{n_k\ln (T)}{T},\text{\hspace{1em}(6.12)}$$

$$\mathrm{AIC}:\frac{2n_k}{T}.\text{\hspace{1em}(6.13)}$$

Hence, $h(n_k)=n_k$ for both criteria, while $g(T)=\ln (T)/T$ for the BIC and $g(T)=$ $2/T$ for the AIC.6

The following example illustrates how the information criteria can be applied to the univariate linear prediction model.

Example 6.3.1 (AIC and BIC for linear regressions). Consider a set of linear prediction models that differ in their chosen predictor variables, $x_{kt}.$

$$y_{t+1}={\beta }_k^{\prime }{x}_{kt}+{\epsilon }_{kt+1},{\epsilon }_{kt+1}\sim \mathrm{N}(0,{\sigma }_k^2).$$

Let ${\hat{\sigma }}_k^2$ be the MLE estimator of the variance from a regression with $n_k$ regressors. Using the MLE estimates for the kth model, we get (for $Z_T=(y_{t+1},x_{kt}),t=0,\dots ,T-1)$

$$\begin{array}{l}\ln p_Z(Z_T)=-\frac{T}{2}\ln (2\pi {\hat{\sigma }}_k^2)-\frac{1}{2{\hat{\sigma }}_k^2}\sum _{t=0}^{T-1}{\hat{\epsilon }}_{kt+1}^2\\ \\ =-\frac{T}{2}\ln (2\pi {\hat{\sigma }}_k^2)-\frac{T{\hat{\sigma }}_k^2}{2{\hat{\sigma }}_k^2}\\ \\ =-\frac{T}{2}\ln (2\pi {\hat{\sigma }}_k^2)-\frac{T}{2},\end{array}$$

and so

$$-2T^{-1}\ln \hat{p}({\hat{\beta }}_k|Z_T)=\ln {\hat{\sigma }}_k^2+C,$$

where C is a constant that does not vary across models. Hence, for linear regressions with a univariate outcome variable, the AIC takes the form

$$\mathrm{AIC}(k)=\ln {\hat{\sigma }}_k^2+\frac{2n_k}{T}\text{\hspace{1em}(6.14)}$$

and the BIC takes the form

$$\mathrm{BIC}=\ln {\hat{\sigma }}_k^2+\frac{n_k\ln (T)}{T}.\text{\hspace{1em}(6.15)}$$

The Mallows $C_p$ criterion proposed by Mallows (1973) is a third criterion that is often used to select a submodel from a set of linear regressions. Consider a model with $n_k$ regressors and associated sum of squared residuals ${\mathrm{SSR}}_k,$ and let ${\hat{\sigma }}^2$ denote the estimated residual variance for the “super” model that includes all possible $\left(n_K\right)$ regressors. Then the Mallows $C_p$ criterion takes the form

$$C_p(k)=\frac{{\mathrm{SSR}}_k}{{\hat{\sigma }}^2}-T+2n_k.\text{\hspace{1em}(6.16)}$$

The preferred model is that with the smallest $C_p$ -value among all possible models. The $C_P$ criterion is an estimator of the expected in-sample MSE applied to a situation where relevant variables have been omitted.

Model selection in linear regressions based on the Mallows $C_p$ is asymptotically equivalent to model selection over the same set of models based on AIC. To see this, multiply (6.14) by $T$ and subtract the common term ln $({\hat{\sigma }}^2)$ to see that the AIC is equivalent to model selection based on T ln $\left({\hat{\sigma }}_k^2/{\hat{\sigma }}^2\right)+2n_k$ . Applying a first-order Taylor expansion of the natural logarithm around the ratio of the variances set equal to 1, we have

$$\begin{array}{rl}& T\left(\mathrm{AIC}\right)-\ln ({\hat{\sigma }}^2)\approx T\left(\ln (1)+\left(\frac{{\hat{\sigma }}_k^2}{{\hat{\sigma }}^2}-1\right)\right)+2n_k\\ & =\frac{T{\hat{\sigma }}_k^2}{{\hat{\sigma }}^2}-T+2n_k,\end{array}$$

which is the Mallows $C_p$ when ${\hat{\sigma }}_k^2$ is the MLE estimator. In most cases approximation errors from the Taylor expansion disappear in large samples.

To determine the order of a linear autoregressive model, Hannan and Quinn (1979) suggested the model selection criterion

$$\mathrm{HQIC}(k)=\ln {\hat{\sigma }}_k^2+\frac{2k\ln (\ln T)}{T}.\text{\hspace{1em}(6.17)}$$

Their criterion is based on both ensuring consistent estimation of the order of the lag length, k, as well as avoiding overfitting the regression in as far as possible. The penalty term in (6.17) generally falls between that of the AIC and BIC.

The BIC is also often used to choose the lag length in autoregressive or ARMA models; see, e.g., Marcellino, Stock and Watson (2006). Frequently it is imposed that there are no ${\mathrm{\Delta }}^{\mathrm{e}}\mathrm{gap}{\mathrm{s}}^{\prime }$ in the lag orders—for example excluding a model that includes lags 1, 3, and 7—which greatly simplifies the search over model specifications.