6.11.2 Akaike Information Criterion

6.11.2 Akaike Information Criterion

An alternative approach to model selection is to consider a measure of the distance between the true model and the models in some approximating class. One such distance measure is the Kullback–Leibler distance or entropy. For a true density $p_0(y,x,\beta )$ and an approximating density $p(\mathbf{y},\mathbf{x},{\beta }_k)$ this is defined as

$$\begin{array}{rl}& \mathrm{KL}(p_0,p)=E_{p_0}\left[\ln \frac{p_0}{p}\right]\\ & =E_{p_0}\ln p_0-E_{p_0}\ln p,\end{array}\text{\hspace{1em}(6.53)}$$

where $E_{p_0}$ computes the expected value under the true probability. Since the first term is constant across models and depends only on the true model, minimizing the distance between the true and approximating models in (6.53) is equivalent to making the second term as large as possible. The distance can be shown to be nonnegative and is 0 only when $p=p_0$ , almost surely. Unfortunately the true density is unknown so we need to find a way of making this idea operational without reducing the problem a great deal.

One approach is to think of all models as being misspecified in the sense that they are underparameterized relative to some unknown true model with parameters, ${\beta }_K$ For each model, ${\beta }_k$ is then a subset of ${\beta }_K,\mathsf{i}.e.$ , model $M_k$ has fewer parameters than $M_K,n_k<n_K$ . Consider maximum likelihood estimates of ${\beta }_k$ , stacked into a $(K\times 1)$ vector ${\hat{\beta }}_k$ with 0s where a variable is omitted and 1s elsewhere. Once again, we can approximate the log density through a second-order expansion around ${\beta }_K$ evaluated at the MLE values:

$$\begin{array}{r}\ln p({\beta }_K)\approx \ln p({\hat{\beta }}_k)+\frac{1}{2}({\beta }_K-{\hat{\beta }}_k{)}^{\prime }(TI({\hat{\beta }}_k))({\beta }_K-{\hat{\beta }}_k).\end{array}$$

Multiplying by $-2.$

$$-2\ln p({\hat{\beta }}_k)\approx -2\ln p({\beta }_K)+({\beta }_K-{\hat{\beta }}_k{)}^{\prime }(TI({\hat{\beta }}_k))({\beta }_K-{\hat{\beta }}_k).$$

Ignoring terms that are independent of ${\hat{\beta }}_k$ , the approximation of $-2E_{p_0}$ ln $\mathbf{p}$ based on the MLE values becomes

$$-2E_{{\hat{p}}_0}\ln p_k\approx E_{{\hat{p}}_0}({\beta }_K-{\hat{\beta }}_k{)}^{\prime }(TI({\beta }_k))({\beta }_K-{\hat{\beta }}_k).\text{\hspace{1em}(6.54)}$$

Without loss of generality, assume that the coefficients in ${\hat{\beta }}_k$ appear in the first k rows of ${\beta }_K$ . Partition ${\beta }_K=({\beta }_{1k}^{\prime },{\beta }_{2k}^{\prime })$ (along with the information matrix) corresponding to the dimension of the estimated parameters, and partition the information matrix accordingly. We then have17

$$\begin{array}{rl}& E_{p_0}({\beta }_K-{\hat{\beta }}_k{)}^{\prime }T\times \mathrm{I}({\beta }_k)({\beta }_K-{\hat{\beta }}_k)=E_{p_0}({\beta }_{1K}-{\hat{\beta }}_{1k}{)}^{\prime }T{\mathrm{I}}_{11}({\hat{\beta }}_k)({\beta }_{1K}-{\hat{\beta }}_{1k})\\ & +E_{p_0}{\beta }_{2K}^{\prime }T{\mathrm{I}}_{22}({\hat{\beta }}_k){\beta }_{2K}.\end{array}\text{\hspace{1em}(6.55)}$$

This expression breaks the problem into two pieces, namely the estimated part, which introduces a variance term, and a squared bias term involving the parameters of the true model that are omitted. Dealing with the variance term is relatively simple. This term, scaled appropriately by the information matrix, resembles a Wald statistic:

$$T({\beta }_{1K}-{\hat{\beta }}_{1k}{)}^{\prime }(T{I}_{11}({\hat{\beta }}_k))({\beta }_{1K}-{\hat{\beta }}_{1k})\sim {\chi }_{n_k}^2.\text{\hspace{1em}(6.56)}$$

The expected value of this term is approximately $n_k/T$

The squared bias term is simply a nonrandom number that depends on the size of the omitted coefficients, ${\beta }_{2K}$ . Akaike’s approach is to note that this term resembles a likelihood ratio test −2 ln $(\hat{p}({\hat{\beta }}_k|Z)-\stackrel{\sim }{P}({\hat{\beta }}_K|Z))$ testing that the omitted terms are 0 under the alternative, where $p({\hat{\beta }}_K|Z)$ is the likelihood based on the full model and so is invariant across different models, k. This test resembles estimating ${\beta }_{2K}$ with zero restrictions and so converges to a noncentral ${\chi }^2$ -variable. In sufficiently large samples a test statistic that these parameters equal 0 is distributed as ${\chi }_{n_K-n_k}^2(\delta )$ , where δ is the noncentrality parameter ${\beta }_{2K}^{\prime }T{I}_{22}({\beta }_k){\beta }_{2K}$ . Now $E_{{\hat{p}}_0}(-2\ln (\hat{p}({\hat{\beta }}_k|Z)-\hat{p}({\hat{\beta }}_K|Z)))=\delta +(n_K-n_k)$ and hence the noncentrality parameter $\delta =E_{p_0}(-2\ln (\hat{p}({\hat{\beta }}_k|Z)-\hat{p}({\hat{\beta }}_K|Z)))-(n_K-n_k)$ . Combining these results, we obtain

$$\begin{array}{rl}& -2E_{p_0}\ln p\approx E_{p_0}({\beta }_K-{\hat{\beta }}_k{)}^{\prime }(TI({\hat{\beta }}_k))({\beta }_K-{\hat{\beta }}_k)\\ & \approx n_k+E_{p_0}(-2\ln p({\hat{\beta }}_k|Z)-p({\hat{\beta }}_K|Z))-(n_K-n_k)\\ & =E_{p_0}(-2\ln p({\hat{\beta }}_k|Z))+2n_k-(\ln p({\hat{\beta }}_K|Z)+n_K).\end{array}\text{\hspace{1em}(6.57)}$$

When minimizing this criterion, ln $p({\hat{\beta }}_K|Z)+K$ can be ignored since it is the same for each model, and so the AIC seeks to minimize

$$\begin{array}{r}\mathrm{AIC}(k)=E_{p_0}(-2\ln p({\hat{\beta }}_k|Z))+2n_k.\end{array}\text{\hspace{1em}(6.58)}$$

The preferred model is that which minimizes the distance between the true density and the model-implied density and thus has the smallest value of AIC(k) across all models under consideration.

Forecast Methods