11.2.3 Artificial Neural Networks

11.2.3 Artificial Neural Networks

Artificial neural network (ANN) models are the most frequently used sieve method in economic forecasting. The term ANN refers not to a single choice for $g$ but to a class of models. Their common thread is that the $g$ functions are chosen such that their output is confined to the range [0, 1].

The most common ANN method is to use what is known as a feedforward ANN with a single hidden layer. The notion of hidden layers refers to the possibility that rather than letting $g$ directly be a function of $z_t$ , we can let it be a function of another sigmoid $k(\cdot )$ which in turn is itself a function of $z_t$ , so $g(z_t)=g(k(z_t))$ . The feedforward property is that the output prediction does not feed back into g .

It is common to let the $g$ functions be logistic, i.e., for $z_t={\gamma }_{0j}+{\sum }_{i=1}^k{\gamma }_{ij}{z}_{it}$

$$g(z_t,{\gamma }_j)=\frac{1}{1+\exp \{-z_t\}}.\text{\hspace{1em}(11.8)}$$

Different sigmoid functions than (11.8) can be used, for example, the Gaussian sigmoid sets,

$$g(z_t,{\gamma }_j)=(2\pi {)}^{-1/2}{\int }_{-\infty }^{z_t}\exp (-w^2/2)dw.$$

The on/off (Heaviside function) switch sets

$$g(z_t,{\gamma }_j)=1(z_t>0).$$

Nonsigmoid functions can also be employed. For example the Ridgelet ANN typically sets $g(z_t,{\gamma }_j)=\tilde{g}({\tilde{z}}_t)$ , where $\tilde{g}({\tilde{z}}_t)$ is a so-called Ridge function and ${\tilde{z}}_t={\gamma }_{j+1}^{-1}{z}_t$ where ${\gamma }_{j+1}$ is the standard deviation of $z_t$ and so is a scaled version of $z_t$ . One example is to set $g({\tilde{z}}_t)$ equal to the $(k-1)$ th derivative of a smooth density function such as the normal distribution, so $g({\tilde{z}}_t)=(d^{k-1}/d{\tilde{u}}^{k-1})e^{-{\tilde{z}}_t^2/2}$ . See Chen, Racine, and Swanson (2001) for properties and an application to inflation forecasting.

The popularity of ANN models derives in large part from their very general approximating properties. Hornik, Stinchcombe, and White (1989) show that even with one hidden layer and an arbitrary choice of $g$ , with enough terms included, ANN models are able to approximate any Borel-measurable function with very good accuracy. Hence, these models have global approximation properties unlike, for example, Taylor expansions that approximate functions at a single point. As forecasting models we might therefore expect ANN models to work well when we do not know the shape of the nonlinear function at all points in the data.

The ability of ANN models to approximate unknown nonlinear functions was established for a wide variety of choices for the functional form $g_{:}$ , so this result gives no indications on how to choose these functions. Similarly, the choice between single and multiple layer models is not well guided by theory. Again it is reasonable to add a linear model in addition to the terms that pick up any nonlinear components (Kuan and White, 1994). Not surprisingly, an extraordinarily large set of variations on these methods are used in practice, with little theory to guide any choice between them. White (1996) discusses choices related to expected smoothness of the underlying function.

Applications of ANN models require specifying the values of the $\beta$ and ${\gamma }_j$ coefficients in (11.4). For given values of ${\gamma }_j$ , the regression becomes linear and hence OLS could be applied. The difficulty lies in estimation of the parameters inside the $g$ functions. This clearly complicates estimation, especially when large numbers of terms are included.

We also need to select $J_T$ when applying the ANN models. Given values for ${\gamma }_j$ (however estimated) the model is a linear regression and the techniques for model selection in chapter 6 apply. Information criteria and cross-validation methods can therefore be used.

Finally, a note on terminology. It should be clear from the above discussion that, approximation results not withstanding, ANN models are simply a family of nonlinear regression models that relate the predicted variable, $y_{t+1},$ to available data, $z_t$ . However, many papers adopt different jargon. The $g$ functions are known as squashing functions since they keep to the range [0, 1] regardless of the support of the data. Estimation is known as learning. Once parameterized, the functions themselves are often referred to as nodes.