Receiver Operating Characteristic Curves with an Indeterminacy Zone

Abstract

This work extends Receiver Operating Characteristic (ROC) curve to the situation where some cases, falling in an intermediate “indeterminacy zone” of the predictor, are not classified. It addresses two challenges: definition of sensitivity and specificity bounds for this case; and summarization of the large number of possibilities arising from different choices of indeterminacy zones.

Graphical Abstract (Optional)

The grayROC extends Receiver Operating Characteristc (ROC) visualization to classifiers that allow for some cases, falling in an indeterminacy zone, to remain unclassified.

1. Introduction

Receiver Operating Characteristic (ROC) curves help with the visual assessment of the performance of classifiers. Fawcett (2006) reviews the field and points out that “ROC graphs are commonly used in medical decision making, and in recent years have been used increasingly in machine learning and data mining research”.

I consider here the basic case of binary classification using a continuous score, such as a classification probability, or a quantitative biomarker. Traditionally, classification is simply implemented by a cutoff dichotomizing the score. In more recent applications, classification may includes an intermediate area of indeterminacy, which I will call gray zone.

For a famous example, Parker et al. (2009) present the PAM50 risk predictor of breast cancers, which provides a continuous risk score. In clinical applications, this score is most often split into three categories: low, intermediate and high. Women in the low and high categories are directed to specific clinical strategies. Women in the intermediate category are considered on a case by case basis by their clinicians. From an algorithmic standpoint, the intermediate group is not classified. Similarly, machine learning algorithms for classification of pathology and radiology images may allow for certain areas to be routed to further human examination. In these cases indeterminacy helps with practical implementation, by handling safe cases algorithmically and complex ones by human intervention.

Here I describe an algorithm for visualizing bounds on sensitivity/specificity pairs, for short grayROC, to assess the performance range of classifiers allowing for a region of indeterminacy, or gray zone. I try to address two challenges. The first is the definition of sensitivity and specificity bound when there is indeterminacy. The second is the visual summarization of the large number of possibilities arising from different choices of gray zones.

2. Algorithm

Consider a validation study of n labeled subjects, with scores x_i, i = 1,…,n. Without loss, let the first n₀ subjects (0 < n₀ < n) have label 0 and the remaining n₁ have label 1. Also, low levels of the score are taken to predict class 0. The proportion of 1’s in the target population is π, and may differ from the validation study proportion n₁/n, for example if the design of the validation study is a case-control.

A gray zone is defined by the interval (c_L, c_U). The extremes are the lower and upper cutoff. Cases with score below c_L are classified as 0’s. Cases above c_U are classified as 1’s. The rest remain unclassified.

Users of the grayROC need to specify a maximum tolerated percentage of unclassified cases, γ, based on the tradeoffs present in the practical application at hand. Let g_j be the number of class j points falling in the gray zone. A gray zone (c_L, c_U) satisfies the γ-constraint if the proportion of cases in the gray zone is less than γ, that is if (g₀ + g₁)/n < γ. A gray zone (c_L, c_U) satisfies the target population γ-constraint if ((1 − π)g₀ + πg₁)/n < γ.

The grayROC algorithm is a model-free visualization. The basic building blocks are bounds on the cumulative frequencies associated with a given gray zone (c_L, c_U).

First, the most favorable bound on these frequencies is calculated assuming perfect discrimination within the gray zone. Imagine an oracle would take care of the points in the gray zone on behalf of the classifier, by moving them to the extremes of the gray zone so that they can be classified correctly. Formally, define the starred scores as follows:

$$\begin{array}{cccc}\hfill \text{if}\hfill & \hfill x_i\notin (c_L,c_U)\hfill & \hfill \text{then}\hfill & \hfill x_i^{\ast }=x_i\hfill \\ \hfill \text{if}\hfill & \hfill i\le n_0,x_i\in (c_L,c_U)\hfill & \hfill \text{then}\hfill & \hfill x_i^{\ast }=c_L\hfill \\ \hfill \text{if}\hfill & \hfill i>n_0,x_i\in (c_L,c_U)\hfill & \hfill \text{then}\hfill & \hfill x_i^{\ast }=c_U.\hfill \end{array}$$

Let I_A be the indicator of the set A, and define the cumulative frequencies:

$$F_0^{\ast }(c_L,c_U)=\sum _{i=1}^{n_0}{I}_{x_i^{\ast }<(c_L+c_U)/2}.$$ (1)

$$F_1^{\ast }(c_L,c_U)=\sum _{i=n_0+1}^n{I}_{x_i^{\ast }<(c_L+c_U)/2}.$$ (2)

Conversely, the least favorable frequencies are constructed considering the worst case scenario for the points within the gray zone. Imagine now that a saboteur may be in charge of the points in the gray zone, by moving them to extremes of the gray zone, so that they are all classified incorrectly. This would result in the ”daggered” scores, defined as:

$$\begin{array}{cccc}\hfill \text{if}\hfill & \hfill x_i\notin (c_L,c_U)\hfill & \hfill \text{then}\hfill & \hfill x_i^{†}=x_i\hfill \\ \hfill \text{if}\hfill & \hfill i\le n_0,x_i\in (c_L,c_U)\hfill & \hfill \text{then}\hfill & \hfill x_i^{†}=c_U\hfill \\ \hfill \text{if}\hfill & \hfill i>n_0,x_i\in (c_L,c_U)\hfill & \hfill \text{then}\hfill & \hfill x_i^{†}=c_L.\hfill \end{array}$$

Now define the cumulative frequencies:

$$F_0^{†}(c_L,c_U)=\sum _{i=1}^{n_0}{I}_{x_i^{\ast }<(c_L+c_U)/2}$$ (3)

$$F_1^{†}(c_L,c_U)=\sum _{i=n_0+1}^n{I}_{x_i^{†}<(c_L+c_U)/2}.$$ (4)

We can form a large number of starred and daggered pairs of cumulative frequencies satisfying the γ-constraint. The gray-ROC algorithm simplifies the visualization of these pairs by grouping them, and selecting a single higher and lower limit within each group, as follows.

Consider the r observed unique ranked values of the biomarker x₍₁₎,…x_(r). These points will constitute the set of possible values for the extremes (c_L, c_U) of the gray zone. Now define the midpoints between two consecutive values as c_j = (x(r−1) + x_(r))/2 for j = 2, …, r. For each c_j, consider the set of (c_L, c_U) pairs built by first adding the two neighboring observed points on either side, then the next two and so forth. This process continues as long as the gray zone satisfies the γ-constraint. If one of the extremes of the distribution is reached, the process continues on the other side. Among the resulting intervals, the grayROC chooses the ”best” for visualization, defined as follows. For each (c_L, c_U), it eliminates the cases in the gray zone and then computes the area under the ROC curve (AUC, Bradley (1997)) using the classified cases only. The (c_L, c_U) pair maximizing the AUC so defined is $(c_L^{\ast }(c_j),c_U^{\ast }(c_j))$. The generating c_j is not necessarily the midpoint of this interval, but will be contained in it. If multiple gray zones are tied in this maximization, the algorithm minimizes gray zone width among optima. In this way, gray zones are not used in regions where discrimination is not helped by not classifying cases.

Then, the upper limits are defined by the set of points

$$\left(1-F_1^{\ast }(c_L^{\ast }(c_j),c_U^{\ast }(c_j)),1-F_0^{\ast }(c_L^{\ast }(c_j),c_U^{\ast }(c_j))\right)$$ (5)

as c_j varies. Conversely, the lower limits are defined by the set of points

$$\left(1-F_1^{†}(c_L^{\ast }(c_j),c_U^{\ast }(c_j)),1-F_0^{†}(c_L^{\ast }(c_j),c_U^{\ast }(c_j))\right).$$ (6)

for j = 2,…, r. To implement, define the degenerate gray zones (x_(i),x_(i)) and (x_(i),x_(i+1)) as the empty set.

Fix y to be either 0 or 1. The sequences defined by $F_y^{\ast }(c_L^{\ast }(c_j),c_U^{\ast }(c_j))$ and $F_y^{†}(c_L^{\ast }(c_j),c_U^{\ast }(c_j))$ as j varies in 2, …, r do not necessarily define proper cumulative distributions, as they would in a standard ROC analysis. Rather the intent is to provide bounds to the sensitivity / specificity pairs available over a range of possible gray area strategies.

Starred and daggered curves are calculated using both classified and unclassified samples. The exclusion of the unclassified samples only affects the calculation of $(c_L^{\ast }(c_j),c_U^{\ast }(c_j))$.

In summary, the algorithm’s steps to produce the data needed for plotting a grayROC graph are as follows:

Algorithm 1:

The grayROC procedure for computation of upper and lower limits in expressions (5) and (6).

I explored an alternative implementation where the lower and upper limit of the gray area are used in turn to index the AUC optimization, instead of the midpoints. Upper and lower limits can produce markedly different results. Bounds are less stable than the midpoints when sample sizes are small. Nonetheless, this strategy provides a different view of the overlap in the tails, and may turn out to be useful in some applications.

3. Illustration

To illustrate the application and interpretation of the gray-ROC, I consider a gene expression biomarker for the prediction of suboptimal (class 0) versus optimal (class 1) surgical debulking in ovarian cancer patients. Data are available from the CuratedOvarianData Bioconductor package by Ganzfried et al. (2013). Clinical and biological background can be found in Riester et al. (2014). The specific biomarker presented here reflects the transcriptional level of the gene ZNF544, as measured using an Agilent microarray by Yoshihara et al. (2012).

Figure 1 shows the observed biomarker levels by class. Higher level of expression are generally associated with optimal debulking (class 1). Figure 1 also illustrates the type of hypothetical scenarios that enter as building block in the construction of the grayROC, to visually represent the definitions of x* and x^†.

Fig. 1.

Dotplots of biomarker levels by class as observed (top) and in the hypothetical scenarios used in the construction of the grayROC plot. The gray zone is (2.8,3.5). In the “oracle” scenario, class 1 points in the gray zone are moved to the upper limit 3.5 while the class 0 points in the gray zone are moved to 2.8. The reverse is true in the “saboteur” scenario.

Each of hypothetical scenarios in Figure 1 enter the optimization used to find the $c_U^{\ast }(c_j)$’s. These in turn are used to form the starred and daggered sensitvity and specificity bounds. Figure 2 shows segments connecting starred and daggered points corresponding to the two bounds associated with the same c_j. These can be used to explore potential gray area strategies. Say one is interested in a classifier with approximately 80% specificity and 70% sensitivity. ZNF544 does not reach this performance. The upper points inform us that if one were allowed to pass 20% of suitably chosen observations to the oracle, than ZNF544 could reach close to the desired sensitivity/specificity trade-off. It also informs us that if the same observations were passed to the saboteur, the sensitivity and specificity would drop close to the diagonal line of no discrimination.

Fig. 2.

grayROC displays at maximum tolerated percentage of unclassified cases, γ, of .2. The top panel shows segments connecting starred and daggered points corresponding to the same c_j. The segments collapse to a point when the optimal gray area for the corresponding c_j is empty. The bottom panel shows, in addition, the area between the two curves defined by connecting the starred and daggered points. The thinner line corresponds to the standard ROC curve.

Figure 2 also shows, in the bottom panel, the region defined by the starred points as the upper limit, and by the daggered points as the lower limit. Points within the region are not easily interpretable in terms of the optimization of the previous section. The shading is purely a visual aid.

Figure 3 shows grayROC visualizations corresponding to four additional choices of γ.

Fig. 3.

grayROC displays for ZNF544 at maximum tolerated percentage of unclassified cases, γ, of .1 (top left) .15 (top right) .25 (bottom left) and .30 (bottom right.) The thinner line corresponds to the standard ROC curve.

Figure 2 also illustrates that the region defined by the upper and lower limits in the grayROC algorithm is not necessarily convex.

If γ = 0 the grayROC region collapses to the standard ROC line, also drawn in Figures 2 and 3.

In regions where the two class-specific distributions have little overlap, say left of 2, there can be little or no advantage in allowing for a gray zone. Conversely, where the density of biomarker points in the two classes is similar, a gray zone has the potential to improve the practical implementation of the biomarker. Figure 4 depicts this trade-off by elucidating where in the biomarker range the gray area is useful. Only in a narrow range of values does the grayROC algorithm needs to make full use of the 20% of data points allowed for the gray zone (top panel).

Fig. 4.

Proportion of points falling in the gray zone (left) and width of the gray zone in the biomarker scale (right) as a function of c_j at γ = .2.

Lastly, Figure 5 shows grayROCs for four additional genes, chosen in part to illustrate less common features. Regions can be disjoint, when stretches of non-empty gray areas are followed by stretches of empty gray areas. Often this is associated with lack of monotonicity in the likelihood ratio of the two conditional biomarker distributions.

Fig. 5.

grayROC displays at maximum tolerated percentage of unclassified cases, γ, of .2 for the four genes indicated at the top of each panel. The thinner line corresponds to the standard ROC curve.

ZNF487 exemplifies a biomarker with relatively good discrimination. The upper bounds indicates that correct reclassification of as few as 20% of cases could lead to high discrimination. This reclassification could be achieved by biomarkers that prove effective in the gray zone for ZNF487. The lower bound indicates that, if unclassified observations are handled poorly, the performance suffers, but discrimination remains above chance by a clear margin even with a gray area of 20%.

4. Discussion

I am not aware of a good visualization approach to examine classification algorithms that allow for an area of indeterminacy. I hope the grayROC will prove of practical help.

A grayROC visualization depends on the specification of the proportion γ of cases falling in the indeterminacy zone. The grayROC is, by design, sensitive to γ. Also, the influence of γ will differ in each dataset. In general, a plausible choice of γ may reflect the trade-offs inherent to the practical implementation of the algorithm. A grayROC can help users quantify and communicate the consequences of adopting a specific γ.

A full decision analytic approach (Raiffa and Schleifer (1961)) for selecting upper and lower thresholds (and thus γ) is feasible if one is able to quantify the utility associated with classifications, as well as the utilities following assignment to the gray zone. While the grayROC is not a method for optimally selecting γ, it can assist if the decision can only be approached informally. For example, if indeterminate cases need to examined by a costly human reader for accurate classification, different grayROC plots at varying γ can be used to informally evaluate the trade-off between added accuracy and added cost.

The grayROC is helpful when all cases have a known binary label but some are not classified. This differs from multi-class ROC analysis (e.g. Hand and Till (2001)), where the number of labels is greater than two. It also differs from semisupervised analyses (Chapelle et al. (2006)), where some cases are not labeled. Lastly, it differs from systems where binary labels and/or classifications are replaced by fuzzy set memberships.

Evangelista et al. (2005) consider ensemble methods for classification. They visualize properties of the ensembles using a single ROC curve based on aggregating multiple classifiers through fuzzy logic operators including T-conorms and T-norms (Jang et al. (1997)). They term this approach ”fuzzy ROC”. In related work Castanho et al. (2007) generalize traditional ROC analysis to evaluate a single fuzzy-rule-based system, not necessarily arising through ensemble learning.

There are many valid alternatives to ROC curves for investigating and visualizing the properties of a threshold-based classifier. These include Total Operating Characteristic (Pontius and Si (2014)), Decision Curve Analysis (Vickers and Elkin (2006)) and Detection Error Tradeoff, which plots the false rejection rate versus the false acceptance rate (Martin et al. (1997)). I hope that the ideas illustrated here may be helpful in generalizing these methods to classifiers with indeterminacy zones.

The grayROC is not a visualization of uncertainty about the ROC curve in the standard statistical sense. Both the upper and lower bound are themselves point estimates, and their variability could be address by simple resampling approaches. Yet visualizing both the set and uncertainty about the set boundaries could be challenging. Also, γ is expressed in terms of the (potentially rescaled) proportion of cases in the validation study, without consideration for uncertainty.

The oracle and saboteur scenarios are extreme. Variants of this algorithm could be constructed by further specifying bounds on the proportion of cases that could be correctly classify by a human if left in the gray area. Then instead of moving all the gray area points to extremes, these known proportions could be used to move only some of the points and achieve less extreme bounds. These classification proportion could potentially depend on the biomarker region.

From a statistical perspective, indeterminacy can also help characterize regions of the score with poor discriminatory ability. Thus, compared to fully deterministic approaches, allowing for indeterminacy may lead to a different evaluation of classifiers and different approaches to biomarker discovery.

Research Highlights (Required).

• The grayROC extends the ROC to classifiers that allow for cases, falling in an indeterminacy zone, to not be unclassified.

• It is based on bounds on sensitivity and specificity derived accounting for the indeterminacy.

• A simple optimization allows to focus on visualization of most interesting cases, avoiding indeterminacy when not needed.