18.2.4 Receiver Operator Characteristic (ROC) curve

18.2.4 Receiver Operator Characteristic (ROC) curve

The Receiver Operator Characteristic (ROC) curve provides a popular way to evaluate binary forecasts. The unusual name arises from the method’s origination in the analysis of radio signals. The method has since become a common tool in medical diagnosis and weather forecasting. Consider a density forecast method that yields $p(z)$ , and recall that for any cutoff, c(z), the forecast is generated as $f(z)=1\left(\hat{p}(z)>c(z)\right)$ as shown in chapter 12. For any density forecast $p(z)$ we can therefore compute the following table of outcomes.

Here TP is the number of true positives, i.e., the number of forecasts of $Y=1$ for which the method correctly predicts the outcome; TN is the number of true negatives, $i.e.,$ correct forecasts of $Y=-1$ . FN and FP are the number of false negatives and false positives, respectively. Since the forecast depends on $c(z)$ , numbers in this table are a function of $c(z)$ ·

Using this table we can construct estimates of different features of the forecasting method. First, consider the ratio of true positives to the actual number of times the outcome is indeed positive, $TP/(\mathrm{TP}+\mathrm{FN})$ . This is known as the true positive rate (or sensitivity) and is a sample estimate of the population object $P[F=1|Y=1]$ . We can also consider the false positive rate $FP/(\mathrm{FP}+\mathrm{TN})$ (often defined as 1 minus the sensitivity) which is a sample estimate of the population object $E[F=1|Y=-1]$ Naturally, we would like the true positive rate to be 1 and the false positive rate to be 0. Without a perfect classifier or forecasting method, increasing the cutoff, $c(z)$ results in fewer positive forecasts and more negative forecasts. Suppose we set the cutoff $c(z)=0;$ , so that all forecasts are equal to 1. Then both the true positive rate and the false positive rate are equal to 1, since $\mathrm{FN}=\mathrm{TN}=0$ . At the other extreme, suppose we set $c(z)=1$ so all forecasts are equal to −1. Then both the true positive rate and the false negative rate are 0, since $\mathrm{TP}=\mathrm{FP}=0$ . Between these extreme choices for $c(z)$ there is a positive relation between these two measures: increasing $c(z)$ reduces the number of positive forecasts, reducing both the true positive and false positive rates. Tracing the curve that relates the two measures as a function of $c(z)$ , where moves to the right decrease $c\left(z\right)$ , produces the ROC curve. The closer this curve is to being a step function that immediately shifts to 1 as $c(z)$ moves away from 1, the better. A common measure reported for the ROC curve is the area underneath the curve.

As derived, the estimated ROC curve is a step function. However, both parametric and nonparametric methods have been considered in the statistical literature for generating smooth ROC curves and estimates of the area under the curve. Higher ROC curves are generally taken to be better estimators of the forecast distribution because they classify outcomes more correctly across different cutoffs, $c(z)$ . The curves can be viewed as sequences of (correlated) averages over the data, and hence should be accompanied by standard errors. Further, since the forecasts are themselves a function of estimated parameters, estimation error should be taken into account. In practice, bootstrap methods are popular for inference.

Empirically, the ROC curve has been used by Jordà and Taylor (2012) to examine carry trades set up to exploit interest rate differentials between pairs of currencies.