18.3 TESTS BASED ON THE PROBABILITY INTEGRAL TRANSFORM

18.3 TESTS BASED ON THE PROBABILITY INTEGRAL TRANSFORM

The probability integral transform2 (PIT) of a continuous cumulative density function $P_Y(y_{t+1}|Z_t)$ evaluated at some outcome, $y_{t+1}$ , is defined as $U=$ $P_Y(y_{t+1}|z_1,\dots ,z_t)$ . Given a density forecast ${\hat{p}}_{Yt}(y_{t+1})$ and an outcome $y_{t+1}$ , we can compute $u_{t+1}$ as the probability of observing a value less than or equal to $y_{t+1}$ . If the density forecast comes from a parametric model, this number can be calculated directly. If the density estimate is provided as a set of simulated pairs $\{y_i|z,{\hat{p}}_{Yi}(y|z)\}$ , $i=1,\dots ,m$ for a range of y-values, as we discuss in chapter 13, the PIT value can

be estimated as

$${\hat{P}}_{Yt}(Y<y_{t+1})=m^{-1}\sum _{i=1}^m{\hat{p}}_{Yt}(y_i)1(y_i\le y_{t+1})d{y}_i,\text{\hspace{1em}(18.14)}$$

where $d{y}_i$ is the difference in the values of ${\gamma }_i$ .

The PIT has the useful property that if $Y$ is truly distributed as $P_Y(y|z)$ , then U is uniformly distributed on [0, 1]. To see this, notice that $P\left[U\le u\right]=P\left[P_Y(Y)\le u\right]=$ $P[Y\le \dot{P_Y^{-1}}(u)]=P_Y[P_Y^{-1}(u)]=u,$ and $P[U\le u]$ is the uniform density. Since $P_Y(y)$ has support on [0,1], it follows that U ∼ Uniform[0, 1]. The key assumption here is that the density used to compute the PIT is identical to that of the actual outcome.

Moreover, provided that the sequence of conditional densities is correctly specified each period, a sequence of PIT values, $\{{\hat{u}}_{t+1}\},t=T_R,\dots ,T-1$ , generated from a sample of density forecasts, ${\hat{p}}_{Yt}(y_{t+1})$ , and outcomes, $\{y_{t+1}\},t=T_R,...,T-1$ will also be Uniform [0, 1] and mutually independent: ${\hat{u}}_{t+1}\sim i.i.d.\mathrm{Uniform}[0,1]$ For this to hold requires that the conditional density is correctly specified at each point in time; misspecified densities lead to violations of this property. For example, if the densities do not condition correctly on prior information, values of ${\hat{u}}_t$ can be serially correlated.

Example 18.3.1 (PIT score for GARCH(1, 1) process). Suppose that the true density model is a GARCH(1, 1),

$$\begin{array}{rlr}& & y_{t+1}={\mu }_{t+1\mid t}+{\sigma }_{t+1\mid t}{\eta }_{t+1},{\eta }_{t+1}\sim ind\mathrm{N}(0,1),\\ & & {\sigma }_t^2=\omega +{\alpha }_1{\sigma }_{t\mid t-1}{\eta }_t^2+{\beta }_1{\sigma }_{t-1}^2.\end{array}$$

If a forecaster uses a homoskedastic density model of the form

$$y_{t+1}\sim \mathrm{N}({\mu }_{t+1|t},{\sigma }^2),$$

then the PIT becomes

$$\Phi \left(\frac{{\mu }_{t+1|t}+{\sigma }_{t+1|t}{\epsilon }_{t+1}-{\mu }_{t+1|t}}{\sigma }\right)=\Phi \left(\frac{{\sigma }_{t+1|t}}{\sigma }{\eta }_{t+1}\right)\ne \Phi ({\epsilon }_{t+1}),$$

where  is the standard Gaussian c.d.f. Specifically, at times when ${\sigma }_t$ is higher than $\sigma ,$ the average volatility, there is a higher than predicted chance of observing large forecast errors of either sign, and so this misspecification will show up in the form of higher probabilities of very large or very small PIT values at such times.

These results suggest a simple way of testing whether the density forecasts are correctly specified. Given a sequence of PIT values, $\{{\hat{u}}_{t+1}\},t=T_R,\dots ,T-1$ , we can examine whether they are drawn from a uniform distribution.

As an empirical illustration, we use our daily data on S&P500 stock returns and generate one-step-ahead out-of-sample density forecasts from a GARCH(1,1) model with standard normal innovations and computed PIT values. A plot of these values for the last year of the sample preceding 03/31/2015, along with their squares, are shown in figure 18.1. There is no noticeable pattern in these.

PIT values

Squared PIT values

Figure 18.1: Time-series plot of probability integral transform (PIT) values in 2010 from a GARCH(1, 1) model fitted to daily US stock market returns. The top window plots the PIT values, while the bottom window shows the squared PIT values from one-step-ahead density forecasts.

Figure 18.2 shows a histogram of the PIT values using 20 bins, each of which should cover 5%. There is a slight overrepresentation of values near 0.5, suggesting that the GARCH(1,1) model misses the many days with very small return outcomes. Conversely, the model seems to perform reasonably well in the tails of the return distribution, perhaps with a slight tendency to overestimate the probability of large positive outcomes. Of course, the histogram shown here does not reveal what happens in the extreme tails.

To more formally test the hypothesis of correct specification, we can use a Kolmogorov–Smirnov or Cramer–von Mises test. For the Kolmogorov–Smirnov (KS) test, we construct the cumulative density function for U as ${\hat{P}}_T(u)=$ $\begin{array}{r}{T}_P^{-1}{\sum }_{t=T_R}^{T-1}1({\hat{u}}_{t+1}<u)\end{array}$ as a function of u. The cumulative density function of a uniform variable is simply u on [0, 1], so the KS test for this problem is

$$\mathrm{KS}=\underset{u}{\sup }\left|{\hat{P}}_T(u)-u\right|.\text{\hspace{1em}(18.15)}$$

When the predictive distribution $p_{Yt}(y_{t+1})$ is known, the limit distribution for the KS test statistic in (18.15) is $\begin{array}{r}{\sup }_{s\in [0,1]}|W(s)-sW(1)|\end{array}$ , where $W(s)$ is a standard

Figure 18.2: Histogram of one-step-ahead probability integral transform (PIT) values from a GARCH(1, 1) model with recursively estimated parameter values.

Brownian motion process. When parameters of the forecast distribution are estimated, the asymptotic distribution of the KS statistic must account for parameter estimation errors, leading to different critical values; see Durbin (1973). Bai (2003) shows that when the density estimate comes from a distribution known up to a finite set of parameters that must be estimated, the limiting distribution of the Kolmogorov test statistic depends on the true parameters and the true underlying distribution which are of course unknown. This makes it difficult to approximate percentiles of the limit distribution. Bai’s analysis is in-sample, whereas Corradi and Swanson (2006c) address the effect of parameter estimation error when such tests are applied to out-of-sample forecasts.

For the Cramer–von Mises test statistic we compute

$$\mathrm{CvM}={\int }_0^1{\left[{\hat{P}}_T(\hat{u})-u\right]}^{2}du.\text{\hspace{1em}(18.16)}$$

The integral in (18.16) can be approximated numerically by using a fine mesh for $u\in [0,1]$ . Alternatively, given a sample of $T_P$ density forecasts, the expression

$$\mathrm{CvM}=\sum _{t=T_R}^{T-1}{\left({\hat{p}}_{Yt}(y_{t+1})-\frac{t-0.5}{T_P}\right)}^2+\frac{1}{12T_P}\text{\hspace{1em}(18.17)}$$

TABLE 18.2:
Tests of the i.i.d. and zero-mean property of standardized forecast errors from a GARCH(1,1) model with Gaussian shocks estimated on daily US stock market returns.

can be employed. The limit distribution for the CvM statistic is $\begin{array}{r}{\int }_0^1(W(s)-\end{array}$ $sW(1){)}^{2}ds$ , which is the integral of a Brownian bridge. The critical value for a test with a size of 5% is approximately 0.461, with larger values leading to rejections.

Berkowitz (2001) suggests transforming the PIT values through the inverse of the standard Gaussian $c.d.f.,{\mathrm{\Phi }}^{-1}$ . Defining

$${\tilde{u}}_{t+1}={\Phi }^{-1}\left({\int }_{-\infty }^{y_{t+1}}{\hat{p}}_{Yt}(v)dv\right)={\Phi }^{-1}({\hat{u}}_{t+1}),\text{\hspace{1em}(18.18)}$$

and noting that ${\hat{u}}_{t+1}\sim i.i.d$ . Uniform(0, 1), it follows that ${\tilde{u}}_{t+1}\sim$ ind N(0, 1). This suggests simple likelihood ratio tests for correct specification of the distribution forecast. For example, one can run a simple unconstrained regression of ${\tilde{u}}_t$ on a constant and its lagged value,

$${\tilde{u}}_t=\mu +\rho {\tilde{u}}_{t-1}+{\epsilon }_t,{\epsilon }_t\sim (0,{\sigma }^2),\text{\hspace{1em}(18.19)}$$

and use conventional likelihood ratio tests to see whether the mean and variance of ${\tilde{u}}_t$ are 0 and 1, respectively:

$${\mathrm{LR}}_{0,1}=-2\left(\ell (0,1,\hat{\rho })-\ell (\hat{\mu },\hat{\sigma },\hat{\rho })\right)\sim {\chi }_2^2,\text{\hspace{1em}(18.20)}$$

where $\ell (\mu ,\sigma ,\rho )$ denotes the likelihood function in (18.19); ${\tilde{u}}_t$ should also be independent of its lagged values which suggests using the test statistic

$${\mathrm{LR}}_{\rho }=-2\left(\ell (0,1,0)-\ell (\hat{\mu },\hat{\sigma },\hat{\rho })\right)\sim {\chi }_3^2.\text{\hspace{1em}(18.21)}$$

Even if the density model passes these tests, it is possible that the distribution forecasts ignore time-varying volatility. This can be detected by considering persistence in the squared values of ${\tilde{u}}_t$ :

$${\tilde{u}}_t^2=\sigma +\rho {\tilde{u}}_{t-1}^2+v_t,$$

where, under the null of a correctly specified density, ${\sigma }^2=1$ and $\rho =0$ . These regression tests are convenient and easy to conduct, but do not, of course, change the size or power of the original tests for uniformity.

As an empirical illustration of these tests, table 18.2 applies simple regression tests to GARCH(1,1) forecasts of daily US stock returns. We use 2000–2009 as an initial estimation period and 2010–2015 as the forecast evaluation period and assume a constant daily mean return. Tests of zero mean and no serial correlation in ${\tilde{u}}_{t+1}$ are not rejected, nor does there appear to be evidence of serial dependence in the squared value \widehat { u } _ { t + 1 } ^ { 2 } . ^

Constructing a correctly specified density forecast may be too tall an order; after all, correctly specifying the conditional mean is a hard problem. We might therefore expect that tests based on the assumption of a correct density forecast would mostly reject. However, tests such as (18.15) and (18.16) are notorious for having low power unless the sample size is very large, so a failure to reject the null hypothesis may not be very informative. Another problem, detailed in examples by Hamill (2001) and Gneiting and Raftery (2007) is that it is entirely possible that forecast distributions that are not particularly informative about the outcome can look reasonable in tests based on the PIT. We illustrate this by returning to the example from Gneiting and Raftery (2007).

Example 18.3.2 (Comparing Gaussian density forecasts, continued). Consider the two density forecasts from Example 18.2.1, $p_{Y1t+1}\sim \mathrm{N}(0,2)$ and $p_{Y2t+1t}\sim \mathrm{N}({\mu }_t,1)$ Both the density forecasts $p_{Y1t+1}$ and $PY2t+1t$ yield probability integral transforms that are uniformly distributed, despite the second forecast clearly being a better one. Unconditionally, $y_{t+1}\sim \mathrm{N}(0,2)$ , so the first density forecast, $\begin{array}{r}\hat{P}Y1t+1;\end{array}$ is correctly specified and results in a uniform distribution for $\begin{array}{r}{u}_{1t+1}={\int }_{-\infty }^{y_{t+1}}{\dot{p}}_{Y1t+1}(y)dy\end{array}$ . However, conditional on observing μt, $y_{t+1}\sim \mathrm{N}(0,1)$ , so $PY2t+1t$ is also correctly specified and $\begin{array}{r}{u}_{2t+1}={\int }_{-\infty }^{y_{t+1}}{p}_{Y2t+1t}(y|{\mu }_t)dy\end{array}$ is also uniformly distributed.

These examples show that methods based on the PIT may not be able to distinguish between density forecasts that are correctly specified, but use different conditioning variables. Passing the test of uniformity does not imply that conditioning information has been utilized well, or even at all. For this reason, Hamill (2001) regards the PIT uniformity property as a necessary but not sufficient condition for a good forecast density.