7.1.1 Covariance Stationarity

7.1.1 Covariance Stationarity

Consider a time series, or stochastic process, of infinite length, $\{y_t{\}}_{t=-\infty }^{\infty }$ , observed at discrete points in time, t. The mean of ${\gamma }_t$ , denoted by ${\mu }_t=E[y_t{]}_{:}$ , is assumed to be a deterministic process. A deterministic process is perfectly predictable infinitely far into the future. Examples include a constant term, a linear time trend, or even a sinusoid with known periodicity. Of special interest is the case where ${\mu }_t$ is the same for all values of t:

$${\mu }_t=E\left[y_t\right]=\mu \mathrm{for}\mathrm{all}t.$$

In what follows we simplify matters by assuming that $\mu =0$ . This assumption is without loss of generality since we can just subtract the common (constant) mean from the process if it were not true and study the demeaned series.

For any integer, $j_{:}$ , define the autocovariance of the zero-mean process as the unconditional expectation

$$\begin{array}{r}\gamma (j,t)\equiv E[y_t{y}_{t-j}].\end{array}$$

Further, assume that this does not depend on t, but only on the distance, j , i.e., $\gamma (j,t)=\gamma (j)$ for all t. Under these conditions the process $\{y_t{\}}_{t=-\infty }^{\infty }$ is said to

be covariance stationary—sometimes called wide sense stationary or second-order stationary.

Covariance stationary processes can be built from white noise. A white noise process has zero unconditional mean, constant variance, and zero autocovariance at all leads and lags:

Definition 7.1. A stochastic process, ${\epsilon }_t,$ is called white noise $if$ it has zero mean, constant unconditional variance, and is serially uncorrelated:

$$\begin{array}{c}E\left[{\epsilon }_t\right]=0,\\ \\ \mathrm{Var}({\epsilon }_t)={\sigma }^2,\\ E\left[{\epsilon }_t{\epsilon }_s\right]=0,forallt\ne s.\end{array}$$

For such processes we write ${\epsilon }_t\sim \mathrm{WN}(0,{\sigma }^2)$ . A result known as the Wold representation theorem establishes that any covariance stationary process can be written as an infinite sum of current and past white noise terms—also known as an infinite-order moving average $\mathrm{MA}(\infty )$ —with weights, ${\theta }_i$ , that are independent of $t:$

Theorem 7.2 (Wold representation theorem). Any covariance stationary stochastic process $\{{\mathbf{y}}_t\}$ can be represented as a linear combination of serially uncorrelated white noise terms ${\epsilon }_{t-j}\in Z_t,j=0,...,$ ∞ and a linearly deterministic component, ${\mu }_t:$

$$y_t=\sum _{j=0}^{\infty }{\theta }_j{\epsilon }_{t-j}+{\mu }_t,$$

where $\{{\theta }_j\}$ are independent of time and ${\sum }_{j=0}^{\infty }{\theta }_j^2<\infty$

We sketch a short proof of the representation of the stochastic component in the Wold representation theorem that follows Sargent (1987). Let ${\epsilon }_t$ be projection errors from a regression of ${\gamma }_t$ on its own past values:

$${\epsilon }_t=y_t-\mathrm{Proj}[y_t|y_{t-1},y_{t-2},...].\text{\hspace{1em}(7.1)}$$

Here $\mathrm{Proj}[y_t|y_{t-1},y_{t-2},\dots ]$ is the projection value from a linear least squares regression of ${\gamma }_t$ on past values of $y:$ :

$$\mathrm{Proj}[y_t|y_{t-1},y_{t-2},\dots ]=\sum _{i=1}^{\infty }{\beta }_i{y}_{t-i},\text{\hspace{1em}(7.2)}$$

and ${\beta }_i$ are projection coefficients. From (7.1), ${\epsilon }_t$ is a linear combination of current and past values of $\mathbf{y}$ constructed in such a way that ${\epsilon }_t$ is orthogonal to all past values of the series $\{y_{t-1},y_{t-2},...\}$

Previous projection errors ${\epsilon }_{t-s}\left(s>0\right)$ can similarly be written as a linear function of past y-values:

$${\epsilon }_{t-s}=y_{t-s}-\mathrm{Proj}[y_{t-s}|y_{t-s-1},y_{t-s-2},...].$$

These errors are again linear functions of $y_{t-j},y_{t-j-1},...$ , and so it follows from the definition of ${\dot{\epsilon }}_t$ in (7.1), that $E\left[{\epsilon }_t{\epsilon }_{t-s}\right]=0$ for all $s\ne 0,\mathrm{i}.e.,{\epsilon }_t$ is serially uncorrelated.

Projecting the current value, ${\gamma }_t$ , on $\{{\epsilon }_t,{\epsilon }_{t-1},\dots ,{\epsilon }_{t-m}\},m>0$ we get the population forecast

$${\hat{y}}_t^m=\sum _{j=0}^m{\theta }_j{\epsilon }_{t-j},\text{\hspace{1em}(7.3)}$$

where ${\theta }_j=E[y_t{\epsilon }_{t-j}]/E[{\epsilon }_t^2]$ because the ε-values are orthogonal to each other. Moreover, the weights ${\theta }_j$ will be constant because of the assumed covariance stationarity of $y_t.$ . From (7.1), we have $E\left[{\epsilon }_t^2\right]=E\left[y_t{\epsilon }_t\right]-$ $E\left[{\epsilon }_t\mathrm{Proj}[y_t|y_{t-1},y_{t-2},\dots ,y_{t-m}]\right]=E\left[y_t{\epsilon }_t\right]={\sigma }^2$ because ${\epsilon }_t$ is orthogonal to all past values, $y_{t-1},\dots ,y_{t-m}.$ . It follows that ${\theta }_0=E\left[y_t{\epsilon }_t\right]/E\left[{\epsilon }_t^2\right]=1\mathrm{in}\left(7.3\right)$ and so the mean squared error is given by

$$\begin{array}{l}E\left[\right(y_t-\sum _{j=0}^m{\theta }_j{\epsilon }_{t-j}{)}^2]=E[y_t^2]-2\sum _{j=0}^m{\theta }_{j}E[y_t{\epsilon }_{t-j}]+E[\sum _{j=0}^m{\theta }_j^2{\epsilon }_{t-j}^2]\\ =E\left[y_t^2\right]-2{\sigma }^2\sum _{j=0}^m{\theta }_j^2+\sum _{j=0}^m{\theta }_j^2{\sigma }^2\\ =E\left[y_t^2\right]-{\sigma }^2\sum _{j=0}^m{\theta }_j^2\ge 0.\end{array}$$

It follows that

$${\sigma }^2\sum _{j=0}^m{\theta }_j^2\le E[y_t^2].$$

Letting $m\infty ,$ we have ${\sum }_{j=0}^{\infty }{\theta }_j^2<E[y_t^2]<\infty$ , and hence the forecast based on the infinite-order projection in $(7.\dot{2})$ converges in mean square.

Note that any nonzero mean is captured by the deterministic component, ${\mu }_t$ From the definition of white noise it follows that ${\epsilon }_t$ is not predictable using linear models of past data. Usually ${\theta }_0=1$ is taken as a normalization. The fact that the moving average coefficients ${\theta }_i$ are time invariant will be important for estimating forecasting models. If the parameters changed over time there would be less reason to use past data to forecast the future. The result that ${\sum }_{j=0}^{\infty }{\theta }_j^2<\infty$ , tells us that the sum of squared moving average parameters ${\theta }_j^2$ converges and in practice implies that shocks in the distant past have a limited effect on the current value.

Example 7.1.1 (MA representation of random walk model). The random walk model has

$$\begin{array}{l}{y}_t=y_{t-1}+{\epsilon }_t\\ =y_0+\sum _{j=1}^t{\epsilon }_{t-j},\end{array}$$

where ${\epsilon }_t$ is white noise and so ${\theta }_j=1$ for all j , $E\left[y_t\right]=y_0$ and $\mathrm{Var}(y_t)=t{\sigma }^2$ . Since the variance is not constant and depends on time, $t,$ this process does not satisfy the assumptions for stationarity.

Example 7.1.2 (Break model). A second example is a model that has a break in the weight on the past innovation:

$$\begin{array}{r}{y}_t=\{\begin{array}{ll}{\epsilon }_t+{\theta }_1{\epsilon }_{t-1}& fort\le t_0,\\ {\epsilon }_t+{\theta }_2{\epsilon }_{t-1}& fort>t_0,\end{array}\end{array}$$

where ${\theta }_1\ne {\theta }_2$ and ${\epsilon }_t$ is white noise. The break in the weight on ${\epsilon }_{t-1}$ occurs at time $t_0$ and so covariances depend on t and the assumption of time-invariant weights is violated.

The Wold representation theorem shows that the moving average representation holds apart from a possible deterministic term, ${\mu }_t$ . Conversely, the $\mathrm{MA}(\infty )$ process captures stochastic movements in ${\gamma }_t$

The Wold representation theorem makes clear the sense in which an MA model captures the entire set of linear models for covariance stationary processes. A practical concern is that the moving average order is potentially infinite. The construction of ${\epsilon }_t$ as ${\epsilon }_t=y_t-\mathrm{Proj}(y_t|y_{t-1},y_{t-2},...)$ , where Proj(·) is the linear projection operator, also means that we potentially need data going back to the infinite past. However, as we show below, even a stationary first-order AR model can be inverted to obtain an MA model of infinite order. Hence, a popular modeling strategy is to approximate the general moving average representation implied by the Wold representation with an $\mathrm{ARMA}(p,q)$ model where both $\mathbf{p}$ and q are of low order. We next turn to such models.