10.2.2 Principal Components Estimation (Large N)

10.2.2 Principal Components Estimation (Large N)

Maximum likelihood estimation is not feasible when N is large. In such cases a nonparametric principal components approach can be used instead. Under the assumption in the static representation (10.12) that $F_t^P$ and ${\epsilon }_{xt}$ are uncorrelated at all leads and lags, we can express the covariance matrix of the observed variables, $X_t,$ as the weighted sum of the covariance matrix of the common factors, ${\Sigma }_{FF}=E[(F_t^{\mathcal{P}}-$ ${\bar{F}}^{\mathcal{P}})(F_t^{\breve{\mathcal{P}}}-{\bar{F}}^{\bar{\mathcal{P}}}{)}^{\prime }]$ , where ${\bar{F}}^{†}$ is the time-series average of $F_t^P$ , and the covariance matrix of the idiosyncratic terms, ${\Sigma }_{\epsilon \epsilon }=E\left[{\epsilon }_t{\epsilon }_t^{\prime }\right]$ :

$${\Sigma }_{XX}=\Lambda {\Sigma }_{FF}{\Lambda }^{\prime }+{\Sigma }_{\epsilon \epsilon }.\text{\hspace{1em}(10.15)}$$

Given this expression, estimation proceeds by minimizing a nonlinear least squares objective that treats both - and $F_t^{\hat{P}}$ , for $t=1,\dots ,T$ , as unknown fixed parameters:

$$\underset{F_1^p,\dots ,F_T^p,\Lambda }{\min }(NT{)}^{-1}\sum _{t=1}^T(X_t-\Lambda F_t^p{)}^{\prime }(X_t-\Lambda F_t^p),\text{\hspace{1em}(10.16)}$$

subject to the normalizing constraint that ${\Lambda }^{\prime }\Lambda =I_r$ . Given an estimate of the factor loadings, $\hat{\Lambda }$ , it follows from (10.12) that an estimator of the factors can be obtained from a cross-sectional regression of $X_t$ on $\hat{\Lambda }$ :

$${\hat{F}}_t^p=({\hat{\Lambda }}^{\prime }\hat{\Lambda }{)}^{-1}{\hat{\Lambda }}^{\prime }{X}_t={\hat{\Lambda }}^{\prime }{X}_t.\text{\hspace{1em}(10.17)}$$

Substituting this expression back into the objective function (10.16) and minimizing the resulting expression is equivalent to maximizing ${\Lambda }^{\prime }{\hat{\Sigma }}_{XX}\Lambda$ subject to the constraint ${\Lambda }^{\prime }\Lambda =I_r$ . Setting $\hat{\Lambda }$ equal to the r eigenvectors of $\begin{array}{r}{\hat{\Sigma }}_{XX}={\dot{T}}^{-1}{\sum }_{t=1}^T{X}_t{X}_t^{\prime }\end{array}$ corresponding to the first r eigenvalues of this matrix provides the solution to this problem. This in turn means that the common factors (10.17) take the form ${\hat{F}}_t^p\stackrel{\cdot }{=}{\hat{\Lambda }}^{\prime }{X}_t,i.e.$ , they are the largest r principal components of $X_t.\mathrm{By}$ construction these principal components will be mutually orthogonal.

If ${\epsilon }_{xt}=({\epsilon }_{x1t}\dots {\epsilon }_{xNt}{)}^{\prime }$ are independently and identically distributed (i.i.d.) and Gaussian with homogenous variances, $\mathrm{N}(0,{\sigma }^2{I}_N)$ , the principal components estimator and the maximum likelihood estimator are identical. On the other hand, if ${\epsilon }_{xt}=({\epsilon }_{x1t}\dots {\epsilon }_{xNt}{)}^{\prime }\sim$ ind $\mathrm{N}(0,{\Sigma }_{\epsilon \epsilon })$ with known covariance matrix, a more efficient estimator could be obtained by solving the nonlinear least squares problem

$$\underset{F_1^p,\dots ,F_T^p,\Lambda }{\min }{T}^{-1}\sum _{t=1}^T(X_t-\Lambda F_t^p{)}^{\prime }{\Sigma }_{\epsilon \epsilon }^{-1}(X_t-\Lambda F_t^p).\text{\hspace{1em}(10.18)}$$

In practice ${\Sigma }_{\epsilon \epsilon }$ is unknown and so solving this problem is infeasible. Weighted principal components estimators that are feasible have been proposed by Boivin and Ng (2006). Boivin and Ng suggest estimating the diagonal elements of ${\Sigma }_{\epsilon \epsilon }$ using the residuals from a first-stage regression of $X_t$ on the factors obtained through (unweighted) principal components. Other refinements of the simple principal components problem in (10.16) entail accounting for serial correlation in ${\epsilon }_{xt};$ see Stock and Watson (2005).

While principal components methods do not rely on any particular model for the data-generating process, it has been argued by authors such as Tipping and Bishop (1999) that this method can be given a Gaussian latent variable interpretation and as such is related to more structured factor models. Moreover, in a setting where the predicted variable is included among the variables from which the principal components are extracted, it is easily demonstrated that principal components and factor augmented forecasts become equivalent.

More informally, in situations where a few unobserved factors account for the common variation among the predictors while the remaining idiosyncratic variation in the variables is only weakly correlated, the principal components can be expected to provide a good approximation of the information contained in “most” of the predictors. This information can then be topped up with variablespecific information from a few select predictors, such as autoregressive lags, in the prediction model.