11.2.4 Projection Pursuit Regression

11.2.4 Projection Pursuit Regression

When the dimension of the predictor variables becomes moderately large, the set of approximating models also can become very large. This is a problem known as the curse of dimensionality. Choosing between models in this situation becomes problematic, as does implementation of many of the procedures. Projection pursuit regression, introduced by Friedman and Stuetzle (1981), is a simplified approach that aims to capture the gains from allowing for the estimation of a flexible function without encountering the problems that arise from the curse of dimensionality. The approximation they consider takes the form

$$y_{t+1}={\beta }_0+\sum _{i=1}^m{g}_i({\beta }_i^{\prime }{z}_t)+u_{t+1},\text{\hspace{1em}(11.11)}$$

where $g_i$ is a nonlinear function of the linear index, ${\beta }_i^{\prime }{z}_t$ , with different values of the “weights” $"{}^{\prime }{b}_i$ for each term. Typically the predicted variable $\{{\mathbf{y}}_{t+1}\}$ is first demeaned, giving an estimate for ${\beta }_0$ . An iterative approach is used to building the model, starting with one term and adding terms until the gain from additional terms becomes smaller than a user-defined threshold for the minimum required improvement in the insample MSE.

To construct the first term, an initial guess at ${\beta }_1$ is made and $w_t={\beta }_1^{\prime }{z}_t$ is constructed using this guess. Next, a nonparametric model is fitted to explain $(y_{t+1}-$ ${\hat{\beta }}_0)$ by $w_t$ . Given this model, ${\beta }_1$ is estimated by minimizing the sum of squared residuals $\begin{array}{r}{\sum }_{t=1}^{T-1}{\left(y_{t+1}-{\hat{\beta }}_0-{\hat{g}}_1({\beta }_1^{\prime }{z}_t)\right)}^2\end{array}$ . The procedure is then repeated to obtain the second term replacing $y_{t+1}-{\hat{\beta }}_0$ with $y_{t+1}-{\hat{\beta }}_0-{\hat{g}}_1({\hat{\beta }}_1^{\prime }{z}_t)$ . Computational gains arise because the nonparametric procedure is a function only of a univariate series regardless of the dimension of $z_t$ . As with most of these ad hoc procedures, a wide range of variations exist and the actual nonparametric methods used to construct estimates of $\hat{g}$ vary from application to application.

Figure 11.1: Kernel forecasts of inflation (top) and stock returns (bottom) for three different choices of bandwidth.