11.2.2 Splines

11.2.2 Splines

Splines provide a method for effectively partitioning the space $\{y,z\}$ so that different models are estimated on separate partitions in such a way that the models remain continuous over the full range of z. For example, consider a forecasting regression that allows the model to be one linear regression for $z<z^{\ast }$ and another for $z>z^{\ast }$ such that the curves meet at $z^{\ast }$ . In practice, often a cubic spline is used so that the model is a polynomial in z up to $z^3$ . The partition points are known as knots.

Consider the problem of predicting $y_{t+h}$ with a univariate predictor, $z_t$ . The simplest cubic spline would involve estimating the model,

$$y_{t+1}={\beta }_0+{\beta }_1{z}_t+{\beta }_2{z}_t^2+{\beta }_3{z}_t^3+\sum _{i=1}^m{\beta }_{3+i}(z_t-c_i{)}^{3}1(z_t\ge c_i)+u_{t+1},\text{\hspace{1em}(11.7)}$$

where the $m$ knots are values of $c_i,i=1,\dots ,m$ in the support of the predictor variable. Application of the cubic spline method requires choosing the location of the knots, and hence selecting m. A standard approach is to simply choose them as evenly spaced percentiles of z with m dependent on the sample size; larger values for m result in a less smooth model and also require a greater number of parameters to be estimated. A common choice for a sample size of 100 is to use knots at the 20th, 40th, 60th, and 80th percentiles. Given such knots, estimation of (11.7) can proceed simply by OLS.

Two issues are often considered in spline regression of simple forms such as (11.7). First, near the boundaries of the support for the observed predictors, the fitted model can give very erratic results since the data are not influential in fitting the curve near these points. Hence, the cubic terms can result in large slopes at these points. To overcome this, a standard procedure is to place knots at the ends of the support and use linear functions for the partition between these and the closest knots. It is not clear that this is so helpful since it involves generating forecasts for situations that have hardly ever been previously observed. In such situations a forecaster probably does not want to rely on extrapolating from a model whose main property is that it is picking up nonlinearities and using them to extrapolate over unobserved data.

The second issue that arises is that if the coefficients ${\beta }_4,\dots ,{\beta }_{3+m}$ in (11.7) are vastly different from each other, then the fitted model can vary quite a bit. To counter this, differences in the parameter estimates can be penalized, e.g., by minimizing a weighted sum of squared errors and the penalty term. Penalties usually are of the form $\lambda \int [\tilde{g}(x){]}^{2}dx$ , where $\tilde{g}$ is the derivative or second derivative of the estimated function. For example, the second derivative of the cubic spline in (11.7) is $\begin{array}{r}2{\beta }_2+6{\beta }_{3}z+6{\sum }_{i=1}^m{\beta }_{3+i}(z-c_i)1(z\ge c_i)\end{array}$ . Integrating the square of this over $z_t$ from the lower to the upper points of the support of the data results in a penalty $\lambda {\beta }^{\prime }D\beta$ . The estimation problem now is in the form of a generalized Ridge regression which has a simple closed-form solution (see chapter 6).

A final difficulty in using spline models is that the model gets very complicated as the dimension of z increases. Even with only two predictors, we could include not only polynomials in both predictors but also their cross products. This also complicates the choice of the knot positions since we now need to partition a larger space. One approach is to partition along each of the z-variables independent of the other predictors; however this introduces a large number of terms and hence more $\beta$ coefficients to estimate.