18.4 EVALUATION OF MULTICATEGORY FORECASTS

18.4 EVALUATION OF MULTICATEGORY FORECASTS

Pesaran and Timmermann (2009) consider a range of dependency tests for multicategory variables that directly apply to predictability tests for such variables. Suppose that the outcome variable falls in $m_y$ categories, $y_{it+1},i=1,2,\dots ,m_y,$ while the forecast falls in $m_f$ categories, $f_{jt+1|t},j=1,2,...,m_f$ . For serially independent outcomes, a test of independence between $y_{t+1}$ and $f_{t+1|t}$ can be based on the canonical correlation coefficients between $y_{t+1}$ and $f_{t+1|t}$ . For serially dependent outcomes, Pesaran and Timmermann (2009) show that a multicategory predictability test can be constructed using canonical correlations of suitably filtered versions of $y_{t+1}$ and $f_{t+1|t}$ after accounting for the effect of lagged values of $(y_{t+1}^{\prime },f_{t+1|t}^{\prime }{)}^{\prime }$ Specifically, they propose trace and maximum canonical correlation tests that use dynamically augmented reduced rank regression and are simple to compute. The tests hold under general conditions and require only that outcomes and forecasts are generated by ergodic, finite-order Markov processes.

To see how the test works, let $Y=(y_2,y_3,...,y_T{)}^{\prime }$ and $F=$ $(f_{2|1},f_{3|2},\dots ,f_{T|T-1}{)}^{\prime }\mathrm{be}(T-1)\times (m_y-1)$ and $\left(\left(T-1\right)\times \left(m_f-1\right)\right)$ observation matrices tracking the qualitative indicators, respectively. Also, let $M_{\iota }=I_{T-1}-$ ${\iota }_{T-1}({\iota }_{T-1}^{\prime }{\iota }_{T-1}{)}^{-1}{\iota }_{T-1}^{\prime },$ where ${\iota }_{T-1}$ is a $((T-1)\times 1)$ vector of 1s. Finally, define $S_{yf}=\widetilde{S_{f\nu }^{{}^{\prime }}}=(T-1{)}^{-1}{Y}^{\prime }{M}_{t}F,S_{yy}=(T-1{)}^{-1}{Y}^{\prime }{M}_{t}Y,S_{ff}=(T-1{)}^{-1}{F}^{\prime }{M}_{t}F$

In the absence of serial dependencies in the outcome, a trace test of independence between the forecast and the outcome has an asymptotic ${\chi }^2$ distribution, i.e.,

$$(T-1)\times \mathrm{trace}(S_{yy}^{-1}{S}_{yf}{S}_{ff}^{-1}{S}_{fy})\stackrel{a}{\sim }{\chi }_{(m_y-1)\left(m_f-1\right)}^2.\text{\hspace{1em}(18.22)}$$

In the presence of serial dependencies, a test of predictability $(i.e.$ , independence between $y_{t+1}$ and $f_{t+1|t})$ can be based on the maximum or the average of the canonical correlation coefficients of Y and F after filtering both variables for the effects of lagged values. For example, in the presence of first-order dependencies we first compute the eigenvalues of $S_w=S_{yy,w}^{-1}{S}_{yf,w}{S}_{ff,w}^{-1}{S}_{fy,w}$ , where

$$\begin{array}{rl}& S_{yy,w}=(T-1{)}^{-1}{Y}^{\prime }{M}_{w}Y,\\ & S_{ff,w}=(T-1{)}^{-1}{F}^{\prime }{M}_{w}F,\\ & S_{fy,w}=(T-1{)}^{-1}{F}^{\prime }{M}_{w}Y,\\ & M_w=I_{T-1}-W{\left(W^{\prime }W\right)}^{-1}{W}^{\prime },\end{array}$$

with $W=(\iota ,F_{-1},Y_{-1});F_{-1}$ and $Y_{-1}$ are $((T-1)\times (m_f-1))$ and $((T-1)\times$ $(m_{\gamma }-1))$ matrices containing the lagged values $f_{t\mid t-1}$ and $y_t.$ , respectively.

Pesaran and Timmermann show that the trace test based on $S_w$ is equivalent to testing $\Pi =0$ in the dynamically augmented reduced rank regression,

$$Y=F{\Pi }^{\prime }+WB+U,\text{\hspace{1em}(18.23)}$$

where U is a $((T-1)\times (m_y-1))$ ) matrix of serially uncorrelated errors and W controls for the effects of lagged variables. Pesaran and Timmermann (2009, Theorem 1) show that, asymptotically as $T\to \infty$ ,

$$(T-1)\times \mathrm{trace}(S_{yy,w}^{-1}{S}_{yf,w}{S}_{ff,w}^{-1}{S}_{fy,w})\stackrel{a}{\sim }{\chi }_{(m_y-1)(m_f-1)}^2,\text{\hspace{1em}(18.24)}$$

where $S_{yy},S_{yf,w}.$ , and $S_{ff,w}$ are defined above.

This intuitive result shows that provided the trace test is conducted on the residuals from the dynamically augmented regressions so that any dynamics is filtered out, the standard results for the trace canonical correlation test in the static case, (18.22), go through.

In the special case with only two outcomes for the dependent variable, Pesaran and Timmermann (1992) derive a simple test statistic. For example, suppose we are interested in predicting the sign of $y_{t+1}$ by means of the sign of a forecast $f_{t+1|t}$ . Let P be the probability of a correctly predicted sign, while $P_y$ and $P_f$ are the probabilities of observing positive values of the outcome and forecast, respectively. Further, let

$$\hat{P}=(T-1{)}^{-1}\sum _{t=1}^{T-1}{y}_{t+1}{f}_{t+1|t}>0,$$

$${\hat{P}}_y=(T-1{)}^{-1}\sum _{t=1}^{T-1}{y}_{t+1}>0,$$

$${\hat{P}}_f=(T-1{)}^{-1}\sum _{t=1}^{T-1}{f}_{t+1\mid t}>0$$

be the corresponding sample estimates of these probabilities. Under the null that the signs of the outcome and the forecast are independently distributed, $P=P_{\ast }\equiv$ $P_f{P}_y+(1-P_f)(1-P_y)$ . A test can therefore be based on the difference $(P-P^{\ast })$ Pesaran and Timmermann (1992) show that, asymptotically as $T\to \infty$

$$S_T=\frac{\hat{P}-{\hat{P}}_{\ast }}{\sqrt{\widehat{\mathrm{var}}(\hat{P})-\widehat{\mathrm{var}}({\hat{P}}_{\ast })}}\stackrel{a}{\sim }\mathrm{N}(0,1),$$

where

$$\begin{array}{rl}& {\hat{P}}_{\ast }={\hat{P}}_y{\hat{P}}_f+(1-{\hat{P}}_y)(1-{\hat{P}}_f),\\ & \widehat{\mathrm{var}}(\hat{P})=(T-1{)}^{-1}{\hat{P}}_{\ast }(1-{\hat{P}}_{\ast }),\\ & \widehat{\mathrm{var}}({\hat{P}}_{\ast })=(T-1{)}^{-1}(2{\hat{P}}_y-1{)}^2{\hat{P}}_f(1-{\hat{P}}_f)+(T-1{)}^{-1}(2{\hat{P}}_f-1{)}^2{\hat{P}}_y(1-{\hat{P}}_y)\\ & +4(T-1{)}^{-2}{\hat{P}}_y{\hat{P}}_f(1-{\hat{P}}_y)(1-{\hat{P}}_f).\end{array}$$

This test is very simple to compute and has found use in studies of market timing of stock returns as well as in tests of whether survey participants can forecast if their firms’ prices will go up or go down. The test does not account for serial dependence in the outcomes or forecasts, however.