On Optimal Biomarker Cutoffs Accounting for Misclassification Costs in Diagnostic Trilemmas with Applications to Pancreatic Cancer

Abstract

Pancreatic ductal adenocarcinoma (PDAC) is the most deadly cancer and currently there is strong clinical interest in novel biomarkers that contribute to its early detection. Assessing appropriately the accuracy of such biomarkers is a crucial issue and often one needs to take into account that many assays include biospecimens of individuals coming from three groups: healthy, chronic pancreatitis, and PDAC. The ROC surface is an appropriate tool for assessing the overall accuracy of a marker employed under such trichotomous settings. A decision/classification rule is often based on the so-called Youden index and its three-dimensional generalization. However, both the clinical and the statistical literature have not paid the necessary attention to the underlying false classification (FC) rates that are of equal or even greater importance. In this paper we provide a framework to make inferences around all classification rates as well as comparisons. We explore the trinormal model, flexible models based on power transformations, and robust non-parametric alternatives. We provide a full framework for the construction of confidence intervals, regions and spaces for joint inferences or for clinically meaningful points of interest. We further discuss the implications of costs related to different false classifications. We evaluate our approaches through extensive simulations and illustrate them using data from a recent PDAC study conducted at the MD Anderson Cancer Center.

1. Introduction

Pancreatic Ductal Adenocarcinoma (PDAC) is the most lethal cancer among organ site with an overall five-year survival rate of approximately 8% [1]. It is the third leading cause of cancer-related mortality in both men and women in the United States, and is projected to become the second most common cause of cancer-related deaths by 2030 [2]. Screening of the general population for PDAC is not considered cost-effective due to the low incidence of PDAC and the high cost of follow-up procedures following a positive diagnosis. However, it is considered cost-effective for certain higher-risk groups and its early detection is critical in improving survival [3]. Two of the guidelines given in [4] strongly suggest screening for 1) patients that are first-degree relatives of patients with pancreas cancer with at least 2 affected genetically related relatives, 2) patients with genetic syndromes associated with an increased risk of pancreas cancer, including all patients with Peutz-Jeghers syndrome, hereditary pancreatitis, patients with CDKN2A gene mutation, and patients with 1 or more first-degree relatives with pancreas cancer with Lynch syndrome, and mutations in BRCA1, BRCA2, PALB2, and ATM genes.

Even though CA19–9 has shown promising potential as a diagnostic biomarker, its utility is limited in detecting early-stage PDAC. Consequently, there is strong clinical interest in discovering new non-invasive blood-based biomarkers for the early detection of PDAC. Related biomarker studies consider assays that often include individuals coming from three groups: healthy (controls), chronic pancreatitis, and individuals with PDAC. Such a trichotomous nature of the true disease status needs to be taken into account when evaluating candidate biomarkers. Apart from assessing the accuracy of a candidate biomarker in terms of correctly classifying individuals among all three groups, it is of equal or even greater importance, to also make inferences around the false classification rates when a decision rule needs to be established.

Under a two-class case setting, where the disease status is defined by the ‘healthy’ and the ‘diseased’ class, the ROC curve is typically employed for assessing the accuracy of a continuous biomarker. If we denote with Y₁ the scores of the healthy and with Y₂ the scores of the diseased, the ROC visualizes all possible pairs of sensitivity, $sens(c)=P\left(Y_2\ge c\right)$, and specificity $spec(c)=P\left(Y_1<c\right)$, for a given cutoff c. By scanning c on the real line we obtain infinite pairs of (spec(c), sens(c)), where −∞ < c < ∞. The ROC is simply the plot of sens against 1 − spec. An overview of ROC curves is given in [5]. It is easy to see that the ROC curve in the two-class case captures all information in terms of both correct and false classification. Note that the false positive rate is $FPR=P\left(Y_1\ge c\right)=1-P\left(Y_1<c\right)=1-spec(c)$. Note further that the false negative rate is $FNR=P\left(Y_2<c\right)=1-P\left(Y_2\ge c\right)=1-sens(c)$. The standard ROC curve is constructed by plotting the true positive rate (i.e. the sensitivity or TPR) on the y-axis and the FPR along the x-axis. Thus, it is trivial to see that this plot is simply an equivalent visualization of all true and false classification rates. That is, using the traditional ROC curve, one could derive a visualization of the tradeoff of any pair of TPR, TNR, FPR, FNR. Hence, the two-class ROC curve provides inherently all information in terms of classification rates. One way of deriving an optimal cutoff point is the maximization of the Youden index [6].

Under a trichotomous setting we have three groups for discrimination, whose scores are denoted by Y₁, Y₂, and Y₃, and are generated from distributions $F{C}_1(\cdot )$, $F{C}_2(\cdot )$, and $F{C}_3(\cdot )$, respectively. The ROC analogous in this case is the ROC surface ([7],[8], [9]). For an overview see [10]. Under such a setting we have three TC rates and two ordered cutoffs c₁ < c₂. The formers are defined by $T{C}_1=P\left(Y_1<c_1\right)$, $T{C}_2=P\left(c_1\le Y_2<c_1\right)$, and $T{C}_3=P\left(Y_3\ge c_2\right)$. After scanning both c₁ and c₂ on the real line we obtain infinite triplets of $\left(T{C}_1\left(c_1\right),T{C}_2\left(c_1,c_2\right),T{C}_3\left(c_2\right)\right)$. By plotting TC₁, TC₃ and TC₂ on the x–, y–, and z–axis respectively, we obtain the ROC surface that is naturally bounded by the unit cube. However, such a plot does not contain all information with respect to all possible classification rates. A generalization of the Youden index in the three class case has been studied before (see [11]–[20]). This involves maximizing the sum of the three true class rates with respect to the two ordered cutoffs. As a result, one gets the optimal pairs of cutoffs that correspond to a true class triplet (TC₁, TC₂, TC₃). However, this triplet (TC₁, TC₂, TC₃) does not reveal the false classification rate of classifying to group 2 an individual that truly belongs to group 1. In particular, there are six such FC rates in total, and knowledge of a (TC₁, TC₂, TC₃) triplet (or the ROC surface) does not imply knowledge of all underlying false classification rates, as opposed to the two-class case. Even though it is known that there are 6 FC rates, inferences for these FC rates are not currently available in the literature. This is an important scientific gap, since different costs may be associated with each FC rate. When assessing the accuracy of a biomarker in a trichotomous setting, it is important to know that a potentially poor performance in terms of a particular TC_i, is because our decisions tend to bleed falsely towards group j and not group $k,i,j,k=1,2,3,i\ne j\ne k$. Furthermore, it is important to have the inferential framework to construct confidence intervals for all true and false classification rates and to be able to compare FC rates with one another. Such comparisons are not trivial, as the estimated FC rates for a given biomarker are correlated. In this paper, we fill in this literature gap.

The paper is organized as follows. In section 2, we investigate the estimation of the two cutpoints in the general setting utilizing class prevalences and misclassification costs. In section 3, we consider statistical inference under the normality assumption. We provide an overall 100(1 − α)% confidence hyperspace for all 6 FC rates. We then provide confidence regions for a given pair of false classification rates that can be visualized by practitioners in their effort to assess the strengths and weaknesses of a biomarker in terms of classification. We provide a general framework for testing hypotheses concerning a set of linear combinations of FC rates. In particular, we provide an appropriate test to compare the equality of any two FC rates. In section 4, we consider the same points by employing more flexible models based on power transformations. In section 5, we present a non-parametric alternative that addresses the aforementioned points under a more robust framework. In section 6, we evaluate our approaches with an extensive simulation study. In section 7, we apply our approaches on a real data set that refers to PDAC’s early detection. We end with a discussion.

2. The General Framework

We assume the biomarker score, Y_i, of an individual in the i-th group to be a continuous random variable with cdf F_i, survival function $S_i=1-F_i$ and pdf f_i. The classification fractions that are obtained using the two cutoffs, c₁ and c₂ are given by:

$$C{F}_{j\mid i}=P\left(c_{j-1}<Y_i<c_j\right)=S_i\left(c_{j-1}\right)-S_i\left(c_j\right),$$ (1)

where $c_0=-\infty$ and $c_3=+\infty$. When j = i, we obtain the true classification fraction $T{C}_i=C{F}_{i\mid i},i=1,2,3$. Otherwise we obtain the false classification rates, that is $F{C}_{j\mid i}=C{F}_{j\mid i}$, $i=1,2,3$, $j=1,2,3,j\ne i$.

Optimal cutoffs are obtained by considering class prevalences (when available) of the target population and (agreedupon) costs (see also [16]–[19]). Here, we expand on such a framework and provide further insight and technical details regarding the estimation of the cutoffs as well as the involved inferences around the FC rates. Let π_i be the prevalence of disease status i in the target population under consideration. Denote the misclassification cost of classifying an individual in group i as being in group j by $C_{j\mid i},j\ne i$. Without loss of generality (concerning the optimal cutoffs) we assume here a zero cost for correct classification and zero cost for administering the test itself.

Let $t_1=F{C}_{2\mid 1}=S_1\left(c_1\right)-S_1\left(c_2\right)$ and $t_2=F{C}_{3\mid 1}=S_1\left(c_2\right)$ be the two misclassification rates of subjects in the 1st disease stage. We easily obtain $c_1=S_1^{-1}\left(t_1+t_2\right)$ and $c_2=S_1^{-1}\left(t_2\right)$. Thus, the remaining four FC rates can be written as

$$\begin{array}{cc}F{C}_{1\mid 2}=1-RO{C}_2\left(t_1+t_2\right)& F{C}_{3\mid 2}=RO{C}_2\left(t_2\right)\\ F{C}_{1\mid 3}=1-RO{C}_3\left(t_1+t_2\right)& F{C}_{2\mid 3}=RO{C}_3\left(t_1+t_2\right)-RO{C}_3\left(t_2\right),\end{array}$$ (2)

where $RO{C}_i(t)=S_i\left(S_1^{-1}(t)\right)$. The expected overall cost at t₁ and t₂ is then equal to

$$\begin{array}{l}B\left(t_1,t_2\right)={\pi }_1\left[C_{2\mid 1}{t}_1+C_{3\mid 1}{t}_2\right]\\ +{\pi }_2\left[C_{1\mid 2}\left(1-RO{C}_2\left(t_1+t_2\right)\right)+C_{3\mid 2}RO{C}_2\left(t_2\right)\right]\\ +{\pi }_3\left[C_{1\mid 3}\left(1-RO{C}_3\left(t_1+t_2\right)\right)+C_{2\mid 3}\left(RO{C}_3\left(t_1+t_2\right)-RO{C}_3\left(t_2\right)\right)\right].\end{array}$$ (3)

By minimizing (3) one obtains the optimal cutoffs.

Since the derivative of $RO{C}_i(t)$ equals to

$$RO{C}_i^{\prime }(t)=\frac{f_i\left(S_1^{-1}(t)\right)}{f_1\left(S_1^{-1}(t)\right)}=r_i\left(S_1^{-1}(t)\right),$$

we obtain

$$\begin{gathered} \frac{dB\left(t_1,t_2\right)}{d{t}_1}={\pi }_1{C}_{2\mid 1}-{\pi }_2{C}_{1\mid 2}{r}_2\left(S_1^{-1}\left(t_1+t_2\right)\right)-{\pi }_3\left(C_{1\mid 3}-C_{2\mid 3}\right)r_3\left(S_1^{-1}\left(t_1+t_2\right)\right) \\ \frac{dB\left(t_1,t_2\right)}{d{t}_2}={\pi }_1{C}_{3\mid 1}+{\pi }_2\left[C_{3\mid 2}{r}_2\left(S_1^{-1}\left(t_2\right)\right)-C_{1\mid 2}{r}_2\left(S_1^{-1}\left(t_1+t_2\right)\right)\right] \\ -{\pi }_3\left[\left(C_{1\mid 3}-C_{2\mid 3}\right)r_3\left(S_1^{-1}\left(t_1+t_2\right)\right)+r_3\left(S_1^{-1}\left(t_2\right)\right)\right] \end{gathered}$$ (4)

By setting the derivatives to 0, we obtain the following estimating equations for the two cutoffs $c_1=S_1^{-1}\left(t_1+t_2\right)$ and $c_2=S_1^{-1}\left(t_2\right)$:

$$\begin{gathered} {\pi }_3\left(C_{1\mid 3}-C_{2\mid 3}\right)r_3\left(c_1\right)={\pi }_1{C}_{2\mid 1}-{\pi }_2{C}_{1\mid 2}{r}_2\left(c_1\right) \\ {\pi }_3{C}_{2\mid 3}{r}_3\left(c_2\right)={\pi }_1\left(C_{3\mid 1}-C_{2\mid 1}\right)+{\pi }_2{C}_{3\mid 2}{r}_2\left(c_2\right) \end{gathered}$$ (5)

The above equations do not, in general, have unique solutions. When the solutions exist, they simply yield potential local minima of the cost function. Hence, in general one should, in addition, investigate the actual cost function. It may even be the case that the optimal rule is to not perform any testing at all. Unique solutions that yield a global minimum occur in some important situations that are discussed below. The estimating equations simplify when one considers equal overclassification costs for group 1, i.e. $C_{2\mid 1}=C_{3\mid 1}$, and equal underclassifications costs for group 3, i.e. $C_{1\mid 3}=C_{2\mid 3}$. We then solve

$$\begin{gathered} r_2\left(c_1\right)=\frac{{\pi }_1{C}_{2\mid 1}}{{\pi }_2{C}_{1\mid 2}} \\ \frac{r_3\left(c_2\right)}{r_2\left(c_2\right)}=\frac{f_3\left(c_2\right)}{f_2\left(c_2\right)}=\frac{{\pi }_2{C}_{3\mid 2}}{{\pi }_3{C}_{2\mid 3}}. \end{gathered}$$ (6)

Youden-based cutoffs result by further assumming

$$\begin{gathered} \frac{C_{2\mid 1}}{C_{1\mid 2}}=\frac{{\pi }_2}{{\pi }_1} \\ \frac{C_{3\mid 2}}{C_{2\mid 3}}=\frac{{\pi }_3}{{\pi }_2}, \end{gathered}$$ (7)

in which case one simply solves $r_2\left(c_1\right)=1$ and $\frac{f_3\left(c_2\right)}{f_2\left(c_2\right)}=1$.

Equations in (6) and (7) imply strong assumptions that are in play when operating under the classical Youden index approach. In practice, it is very likely that these assumptions do not hold. Thus, there is a need of accommodating the implied different costs and prevalences.

When considering misclassification costs, it is most often reasonable to assume that $C_{1\mid 3}>C_{2\mid 3}$ and $C_{3\mid 1}>C_{2\mid 1}$. By setting ${\rho }_1=\frac{{\pi }_1}{{\pi }_3}$ and ${\rho }_2=\frac{{\pi }_2}{{\pi }_3}$ the estimating equations in (5) become

$$\begin{gathered} r_3\left(c_1\right)+{\rho }_2\frac{C_{1\mid 2}}{C_{1\mid 3}-C_{2\mid 3}}{r}_2\left(c_1\right)={\rho }_1\frac{C_{2\mid 1}}{C_{1\mid 3}-C_{2\mid 3}} \\ {r}_{3:2}\left(c_2\right)={\rho }_1\frac{\left(C_{3\mid 1}-C_{2\mid 1}\right)}{C_{2\mid 3}}\frac{1}{r_2\left(c_2\right)}+{\rho }_2\frac{C_{3\mid 2}}{C_{2\mid 3}}, \end{gathered}$$ (8)

where $r_{3:2}(y)=\frac{f_3(y)}{f_2(y)}$. The estimating equations in (8) yield a unique solution for the cutoffs, when the three distributions are likelihood ratio ordered, that is when r₂(y) and r_3:2(y) are increasing functions, and the likelihood ratios range from 0 to ∞. This is the case when one considers 3 Normal distributions with equal variances and increasing means or 3 Gamma distributions with equal shape parameters and increasing means.

2.1. Normality Assumption

It can be shown that, if one assumes Normal distributions for the three groups, $N\left({\mu }_i,{\sigma }_i^2\right)$, $i=1,2,3$, with ${\mu }_1<{\mu }_2<{\mu }_3$, and further considers equal overclassification costs for group 1, that is, C_1|3 = C_2|3, equal underclassifications costs for group 3, that is, C_1|3 = C_2|3, and unequal variances, one solves a pair of quadratic equations yielding the following roots for c₁ and c₂.

$$\begin{gathered} c_1=\frac{\left({\mu }_2{\sigma }_1^2-{\mu }_1{\sigma }_2^2\right)\pm {\sigma }_1{\sigma }_2\sqrt{{\left({\mu }_1-{\mu }_2\right)}^2+\left({\sigma }_1^2-{\sigma }_2^2\right)log\left(\frac{{\sigma }_1^2}{{\sigma }_2^2}\frac{{\pi }_2^2}{{\pi }_1^2}\frac{C_{1\mid 2}^2}{C_{2\mid 1}^2}\right)}}{{\sigma }_1^2-{\sigma }_2^2} \\ {c}_2=\frac{\left({\mu }_3{\sigma }_2^2-{\mu }_2{\sigma }_3^2\right)\pm {\sigma }_2{\sigma }_3\sqrt{{\left({\mu }_2-{\mu }_3\right)}^2+\left({\sigma }_2^2-{\sigma }_3^2\right)log\left(\frac{{\sigma }_2^2}{{\sigma }_3^2}\frac{{\pi }_3^2}{{\pi }_2^2}\frac{C_{2\mid 3}^2}{C_{3\mid 2}^2}\right)}}{{\sigma }_2^2-{\sigma }_3^2}. \end{gathered}$$ (9)

The roots are real under the conditions

$$\begin{gathered} {\left({\mu }_2-{\mu }_1\right)}^2>\left({\sigma }_2^2-{\sigma }_1^2\right)log\left(\frac{{\sigma }_1^2}{{\sigma }_2^2}\frac{{\pi }_2^2}{{\pi }_1^2}\frac{C_{1\mid 2}^2}{C_{2\mid 1}^2}\right) \\ {\left({\mu }_3-{\mu }_2\right)}^2>\left({\sigma }_3^2-{\sigma }_2^2\right)log\left(\frac{{\sigma }_2^2}{{\sigma }_3^2}\frac{{\pi }_3^2}{{\pi }_2^2}\frac{C_{2\mid 3}^2}{C_{3\mid 2}^2}\right), \end{gathered}$$ (10)

in which case it can be shown that a local minimum of the expected cost is achieved at

$$\begin{gathered} c_1=\frac{\left({\mu }_2{\sigma }_1^2-{\mu }_1{\sigma }_2^2\right)-{\sigma }_1{\sigma }_2\sqrt{{\left({\mu }_1-{\mu }_2\right)}^2+\left({\sigma }_1^2-{\sigma }_2^2\right)log\left(\frac{{\sigma }_1^2}{{\sigma }_2^2}\frac{{\pi }_2^2}{{\pi }_1^2}\frac{C_{1\mid 2}^2}{C_{2\mid 1}^2}\right)}}{{\sigma }_1^2-{\sigma }_2^2} \\ {c}_2=\frac{\left({\mu }_3{\sigma }_2^2-{\mu }_2{\sigma }_3^2\right)-{\sigma }_2{\sigma }_3\sqrt{{\left({\mu }_2-{\mu }_3\right)}^2+\left({\sigma }_2^2-{\sigma }_3^2\right)log\left(\frac{{\sigma }_2^2}{{\sigma }_3^2}\frac{{\pi }_3^2}{{\pi }_2^2}\frac{C_{2\mid 3}^2}{C_{3\mid 2}^2}\right)}}{{\sigma }_2^2-{\sigma }_3^2}. \end{gathered}$$ (11)

At this point one should investigate the behavior of the expected cost function.

When one considers Youden based estimation of the cutoffs, the conditions in (10) are satisfied and a global minimum of the expected cost function (under the assumptions $C_{2\mid 1}=C_{3\mid 1}$, $C_{1\mid 3}=C_{2\mid 3}$, and (7)) is achieved at

$$\begin{gathered} c_1^{\ast }=\frac{\left({\mu }_2{\sigma }_1^2-{\mu }_1{\sigma }_2^2\right)-{\sigma }_1{\sigma }_2\sqrt{{\left({\mu }_1-{\mu }_2\right)}^2+\left({\sigma }_1^2-{\sigma }_2^2\right)log\left(\frac{{\sigma }_1^2}{{\sigma }_2^2}\right)}}{{\sigma }_1^2-{\sigma }_2^2} \\ {c}_2^{\ast }=\frac{\left({\mu }_3{\sigma }_2^2-{\mu }_2{\sigma }_3^2\right)-{\sigma }_2{\sigma }_3\sqrt{{\left({\mu }_2-{\mu }_3\right)}^2+\left({\sigma }_2^2-{\sigma }_3^2\right)log\left(\frac{{\sigma }_2^2}{{\sigma }_3^2}\right)}}{{\sigma }_2^2-{\sigma }_3^2}. \end{gathered}$$ (12)

When the variances are assumed equal (=σ²) then the three distributions are likelihood ratio ordered. According to (8), the equations to be solved to obtain the cutoffs, c₁ and c₂, can be succinctly rewritten by first setting

$$\begin{gathered} k_2=e^{-\frac{{\mu }_2^2-{\mu }_1^2}{2{\sigma }^2}},u_2=e^{\frac{{\mu }_2-{\mu }_1}{{\sigma }^2}}\left(\Rightarrow r_2(y)=k_2{u}_2^y\right) \\ {k}_3=e^{-\frac{{\mu }_3^2-{\mu }_1^2}{2{\sigma }^2}},u_3=e^{\frac{{\mu }_3-{\mu }_1}{{\sigma }^2}}\left(\Rightarrow r_3(y)=k_3{u}_3^y\right). \end{gathered}$$

After some algebra, it can be shown that one needs to solve numerically for c₁ and c₂ the following

$$\begin{gathered} k_3{u}_3^{c_1}+\left(\frac{{\rho }_2{C}_{1\mid 2}}{C_{1\mid 3}-C_{2\mid 3}}{k}_2\right)u_2^{c_1}={\rho }_1\frac{C_{2\mid 1}}{C_{1\mid 3}-C_{2\mid 3}} \\ {k}_3{u}_3^{c_2}-\left(\frac{{\rho }_2{C}_{3\mid 2}}{C_{2\mid 3}}{k}_2\right)u_2^{c_2}={\rho }_1\frac{C_{3\mid 1}-C_{2\mid 1}}{C_{2\mid 3}}. \end{gathered}$$ (13)

2.2. Numerical Examples

2.2.1. Unequal variances example:

To illustrate the theory and equations mentioned above we first consider a setting where $Y_1\sim N\left(1,1^2\right)$, $Y_2\sim N\left(3,{1.5}^2\right)$, and $Y_3\sim N\left(5.5,2^2\right)$. We assume the following costs in some arbitrary units: $C_{2\mid 1}=10,000$, $C_{3\mid 1}=50,000$, $C_{1\mid 2}=20,000$, $C_{3\mid 2}=40,000$, $C_{1\mid 3}=100,000$, $C_{2\mid 3}=60,000$. We further set the underlying prevalences π₁, π₂, and π₃ equal to 1/5, 2/5, and 2/5 respectively. The classical Youden-index based cutoffs are $c_1^{\ast }=2.0870$ and $c_2^{\ast }=4.4038$. There are the cutoffs obtained if we are willing to assume that $\frac{{\pi }_2^2}{{\pi }_1^2}\frac{C_{1\mid 2}^2}{C_{2\mid 1}^2}=1$ and $\frac{{\pi }_3^2}{{\pi }_2^2}\frac{C_{2\mid 3}^2}{C_{3\mid 2}^2}=1$. However, these assumptions in our example are obviously violated. Under our configuration the cutoffs can be derived by solving the equations in (5). The derived cutoffs are 0.5733 and 3.9425 (see also Figure 1). For the same example we further illustrate how the underlying costs may have a severe impact on the decision making process. We note that in an extreme situation where C_3|2 = 40, 000, 000 and all else remains the same, the obtained cutoffs are 0.5733 and 9.1775 which implies that only 3.3% of the third (aggressive) group is correctly classified as such and 96.6% of the individuals of group 3 are misclassified as group 2. The reason is that a potential misclassification to group 3 of a subject that truly belongs in group 2 is extremely costly relative to the other misclassification costs. The corresponding cost function in visualized in Figure 2. It is obvious that for more extreme values of some of the underlying costs, the derived cutoffs may imply that no testing is necessary either under the notion that all individuals are driven to be classified in group 3, or that all individuals are to be classified as healthy (group 1).

Figure 1.

Configuration of the discussed numerical example where $Y_1\sim N\left(1,1^2\right)$, $Y_2\sim N\left(3,{1.5}^2\right)$, $Y_1\sim N\left(5.5,2^2\right)$. We assume that $C_{2\mid 1}=10,000$, $C_{3\mid 1}=50,000$, $C_{1\mid 2}=20,000$, $C_{3\mid 2}=40,000$, $C_{1\mid 3}=100,000$, $C_{2\mid 3}=60,000$ and further set the underlying prevalences π₁, π₂, and π₃ equal to 1/5, 2/5, and 2/5 respectively. The derived cutoffs using the traditional Youden index are $c_1^{\ast }=2.0870$ and $c_2^{\ast }=4.4038$. The proposed cutoffs after solving the corresponding estimating equations that account for the costs and the prevalances are 0.5733 and 3.9425.

Figure 2.

Visualization of the cost function that corresponds to the unequal variances example discussed above. We visualize the same cost function from two different angles (left and right panel) for convenience. The global maximum is achieved at t₁ = 0.6636 and t₂ = 0.0016. These correspond to the pair of cutoffs: 0.5733 and 3.9425. The profile of the cost function with respect to t₁ and t₂ is given in Figure 3.

Note also that, in this example, the approximate cost at the optimal cutoffs is 11348 units, whereas the classical/naive Youden based cutoffs (assuming the conditions given in (7)) correspond to an approximate cost of 25919 units (which is 2.59 times higher). In other words, we achieve a cost reduction of 56.2% ((25919–11348)/25919 = 0.5622) when taking into account the assumed costs and prevalences.

2.2.2. Equal variances example:

To illustrate another example that refers to an equal variances setting, we consider $Y_1\sim N(1,1)$, $Y_2\sim N(3,1)$, $Y_3\sim N(5.5,1)$. We assume the same costs as before in some arbitrary units: $C_{2\mid 1}=10,000$, $C_{3\mid 1}=50,000$, $C_{1\mid 2}=20,000$, $C_{3\mid 2}=40,000$, $C_{1\mid 3}=100,000$, $C_{2\mid 3}=60,000$. Utilizing equations in (13) we obtain the values 1.3065 and 4.3668 for the first and second cutoff respectively. These are exactly the same values that we would get, had we used equations in (8) or equations in (5).

2.3. Remark and comments:

In Figure 1 we illustrate graphically one example with some hypothesized costs and prevalences. This plot highlights the different cutoffs of our approach in contrast to the classical Youden index. Note that the effect of the prevalence (or costs) in a given application can be dominated by costs (or prevalences) that are extremely imbalanced. For example, if the costs of miss-classifying an individual from groups 2 or 3 to group 1 is extremely high, then c₁ will be driven towards the right tail of the density of Y₁. The reason is that it is extremely costly to have an individual misclassified to group 1. This will drive the decision process to prefer miss-classifying individuals from group 1 to group 2 or 3. On the other extreme, if miss-classifying an individual from group 1 or 2 to group 3 is extremely high, then this will drive the second cutoff to be on the left tail of the density of Y₃. The logic is analogous. In the general case, the prevalence kicks in as is shown by equation (5). It reflects a weight that affects the classification rule since if, for example, it is very small for the first group (and at the same time it is very costly to be misclassified to group 1), then there is an even stronger “push” of the first cutoff to the right of the density of Y₁. The reason is that apart from being very costly to be misclassified to group 1, it is also only a few individuals that will be impacted (due to the low prevalence of the first group) and thus this will drive the process to select an even larger c₁. The rational is analogous for other scenarios and the process boils down to the magnitude of the imbalanced costs and prevalences.

3. Inference under the Normality assumption

Under the normality assumption we assume that the biomarker scores of group 1, group 2, and group 3 are generated separately from three normal distributions i.e. $Y_1\sim N\left({\mu }_1,{\sigma }_1^2\right)$, $Y_2\sim N\left({\mu }_2,{\sigma }_2^2\right)$, $Y_3\sim N\left({\mu }_3,{\sigma }_3^2\right)$. The true class fractions are given by:

$$T{C}_1=\text{Φ}\left(\frac{c_1-{\mu }_1}{{\sigma }_1}\right),T{C}_2=\text{Φ}\left(\frac{c_2-{\mu }_2}{{\sigma }_2}\right)-\text{Φ}\left(\frac{c_1-{\mu }_1}{{\sigma }_1}\right),T{C}_3=1-\text{Φ}\left(\frac{c_2-{\mu }_2}{{\sigma }_2}\right)$$ (14)

and the triplet (TC₁, TC₂, TC₃) defines the ROC surface. The corresponding parametric generalized Youden index is:

$$J_3\left(c_1,c_2\right)=\frac{1}{2}\left\{\text{Φ}\left(\frac{c_1-{\mu }_1}{{\sigma }_1}\right)-\text{Φ}\left(\frac{c_1-{\mu }_2}{{\sigma }_2}\right)+\text{Φ}\left(\frac{c_2-{\mu }_2}{{\sigma }_2}\right)-\text{Φ}\left(\frac{c_2-{\mu }_3}{{\sigma }_3}\right)\right\}.$$ (15)

It is common to select cutoff points by maximizing J₃. However, if the class prevalences and misclassification costs are available, then one ought to select the cutoff points by minimizing the expected overall cost. In general, after solving the equations in (8) one obtains cutoffs that correspond to possible local minima of the expected cost function. Further investigation of the expected cost function may result in ‘degenerate’ rules being optimal (such as the diagnostic test classifying everyone as belonging to group 3). As discussed in the previous section, it can be shown that the Youden-based cutoffs minimize the expected overall cost (and thus are optimal) under the assumptions $C_{2\mid 1}=C_{3\mid 1},C_{2\mid 3}=C_{1\mid 3}$, and the conditions in (7). The Youden-based cutoffs are given in (12).

The corresponding estimates ${\widehat{c}}_1^{\ast },{\widehat{c}}_2^{\ast }$ are then obtained by substituting in Equations (12) the maximum likelihood estimates of ${\mu }_1,{\mu }_2,{\mu }_3,{\sigma }_1,{\sigma }_2,{\sigma }_3$. The associated estimated optimal triplet of true class fractions on the ROC surface is defined by $(\widehat{T{C}_1}\left({\widehat{c}}_1^{\ast }\right),\widehat{T{C}_2}\left({\widehat{c}}_1^{\ast },{\widehat{c}}_2^{\ast }\right),\widehat{T{C}_3}\left({\widehat{c}}_2^{\ast }\right)$ where $\widehat{T{C}_1}(\cdot ),\widehat{T{C}_2}(\cdot ),\widehat{T{C}_3}(\cdot )$ are the maximum likelihood estimates of $T{C}_1(\cdot ),T{C}_2(\cdot ),T{C}_3(\cdot )$. Natural interest lies in knowing how a biomarker performs with respect to the underlying false classification rates which are defined by:

$$F{C}_{2\mid 1}=P(\text{classify as}2\text{while true class is}1)=P\left(c_1<Y_1<c_2\right)=\text{Φ}\left(\frac{c_2-{\mu }_1}{{\sigma }_1}\right)-\text{Φ}\left(\frac{c_1-{\mu }_1}{{\sigma }_1}\right)$$

$$F{C}_{3\mid 1}=P(\text{classify as}3\text{while true class is}1)=P\left(Y_1>c_2\right)=1-\text{Φ}\left(\frac{c_2-{\mu }_1}{{\sigma }_1}\right)$$

$$F{C}_{1\mid 2}=P(\text{classify as}1\text{while true class is}2)=P\left(Y_2<c_1\right)=\text{Φ}\left(\frac{c_1-{\mu }_2}{{\sigma }_2}\right)$$

$$F{C}_{3\mid 2}=P(\text{classify as}3\text{while true class is}2)=P\left(Y_2>c_2\right)=1-\text{Φ}\left(\frac{c_2-{\mu }_2}{{\sigma }_2}\right)$$

$$F{C}_{1\mid 3}=P(\text{classify as}1\text{while true class is}3)=P\left(Y_3<c_1\right)=\text{Φ}\left(\frac{c_1-{\mu }_3}{{\sigma }_3}\right)$$

$$F{C}_{2\mid 3}=P(\text{classify as}2\text{while true class is}3)=P\left(c_1<Y_3<c_2\right)=\text{Φ}\left(\frac{c_2-{\mu }_3}{{\sigma }_3}\right)-\text{Φ}\left(\frac{c_1-{\mu }_3}{{\sigma }_3}\right).$$

Note that $F{C}_{1\mid 2}$ depends only on (${\mu }_1,{\sigma }_1,{\mu }_2,{\sigma }_2$) and $F{C}_{3\mid 2}$ depends only on (${\mu }_2,{\sigma }_2,{\mu }_3,{\sigma }_3$), while the remaining FC rates depend on the full parameter vector denoted by $\theta ={\left({\mu }_1,{\sigma }_1,{\mu }_2,{\sigma }_2,{\mu }_3,{\sigma }_3\right)}^T$.

Let $\mathbf{F}{\mathbf{C}}_{.\mid i}$ denote the 2 × 1 vector containing the two FC probabilities corresponding to true disease status i, i.e. $\mathbf{F}{\mathbf{C}}_{.\mid i}^T=\left[F{C}_{j\mid i}F{C}_{j^{\prime }\mid i}\right]$, with $j,j^{\prime }\ne i,j<j^{\prime }$. Denote the full 6 × 1 false classification probability vector by FC, where $\mathbf{F}{\mathbf{C}}^T=\left[\mathbf{F}{\mathbf{C}}_{.\mid 1}^T\mathbf{F}{\mathbf{C}}_{.\mid 2}^T\mathbf{F}{\mathbf{C}}_{.\mid 3}^T\right]$. The full 6 × 6 variance-covariance for the maximum likelihood estimate of θ, let $\widehat{\theta }$, can be derived in closed form. We denote it with Σ, with $\widehat{\mathbf{\text{Σ}}}={\mathbf{\text{Σ}}}_{\theta =\widehat{\theta }}$ being its estimate.

Note that all FC rates depend either partially or fully on θ and hence their estimates exhibit covariability. To ensure that the estimates lie in the ROC space, we first transform the false classification rates on the real line through the probit transformation, ${\text{Φ}}^{-1}(\cdot )$, and then back-transform them in the ROC space through $\text{Φ}(\cdot )$. The 6 × 6 estimated covariance matrix of our estimates of the transformed FC rates, ${\text{Φ}}^{-1}(\widehat{\mathbf{F}\mathbf{C}})$ is denoted by $\widehat{\mathbf{V}}$ and is obtained through the delta method. Its diagonal elements are given by

$$\widehat{Var}\left({\text{Φ}}^{-1}\left({\widehat{FC}}_{j\mid i}\right)\right)={\left[\frac{\partial {\text{Φ}}^{-1}\left(F{C}_{j\mid i}\right)}{\partial \theta }\right]}_{\theta =\widehat{\theta }}^T\widehat{\text{Σ}}{\left[\frac{\partial {\text{Φ}}^{-1}\left(F{C}_{j\mid i}\right)}{\partial \theta }\right]}_{\theta =\widehat{\theta }},$$ (16)

where $i=1,2,3,j=1,2,3$, and $j\ne i$. Its off-diagonal elements are

$$\widehat{Cov}\left({\text{Φ}}^{-1}\left({\widehat{FC}}_{j\mid i}\right),{\text{Φ}}^{-1}\left({\widehat{FC}}_{j^{\prime }\mid i^{\prime }}\right)\right)={\left[\frac{\partial {\text{Φ}}^{-1}\left(F{C}_{j\mid i}\right)}{\partial \theta }\right]}_{\theta =\widehat{\theta }}^T\widehat{\text{Σ}}{\left[\frac{\partial {\text{Φ}}^{-1}\left(F{C}_{j^{\prime }\mid i^{\prime }}\right)}{\partial \theta }\right]}_{\theta =\widehat{\theta }}$$ (17)

where $i=1,2,3$, $i^{\prime }=1,2,3$, $j=1,2,3$, $j^{\prime }=1,2,3$ with $(j,i)\ne \left(j^{\prime },i^{\prime }\right)$, $j\ne i$, and $j^{\prime }\ne i$. Note that in the above formula we allow i to be equal to i′ and that may be actually of clinical interest, since it would refer to making inferences by comparing two different classification rates conditioning on the true class. Namely, such conditioning would serve the purpose of exploring the direction towards which a given class state is misclassified. A joint 100(1 − α)% confidence hyper-space for all probit transformed classification rates is:

$$\left\{\mathbf{\text{x}}\ni {\left(\mathbf{\text{x}}-{\text{Φ}}^{-1}(\widehat{\mathbf{F}\mathbf{C}})\right)}^T{\widehat{\mathbf{V}}}^{-1}\left(\mathbf{\text{x}}-{\text{Φ}}^{-1}(\widehat{\mathbf{F}\mathbf{C}})\right)<{\chi }_{6;\alpha }^2\right\}$$ (18)

where ${\chi }_{\nu ;\alpha }^2$ denotes the (1 − α)-th percentile of a Chi-square distribution with ν degrees of freedom.

Particular interest lies in deriving confidence regions for the following pairs of FC rates: $\left(F{C}_{2\mid 1},F{C}_{3\mid 1}\right)$, $\left(F{C}_{1\mid 2},F{C}_{3\mid 2}\right)$ and $\left(F{C}_{1\mid 3},F{C}_{2\mid 3}\right)$. A joint confidence region for the probits in the 2 × 1 vector ${\text{Φ}}^{-1}\left(\mathbf{F}{\mathbf{C}}_{.\mid i}\right)$ can be constructed and graphed by the ellipse defined by:

$$\left\{\mathbf{\text{x}}\ni {\left(\mathbf{\text{x}}-{\text{Φ}}^{-1}\left({\widehat{\mathbf{F}\mathbf{C}}}_{.\mid i}\right)\right)}^T{\widehat{\mathbf{V}}}_i^{-1}\left(\mathbf{\text{x}}-{\text{Φ}}^{-1}\left({\widehat{\mathbf{F}\mathbf{C}}}_{.\mid i}\right)\right)<{\chi }_{2;\alpha }^2\right\},$$ (19)

where the 2 × 2 matrix ${\widehat{\mathbf{\text{V}}}}_i$ corresponds to class i and is extracted from the full 6 × 6 covariance matrix $\widehat{\mathbf{\text{V}}}$. The 100(1 − α)% confidence region for the two actual FC rates is found by back-transforming that ellipse into the ROC space using Φ(·) on its coordinates. Due to the irregular shape of these proposed confidence regions, we will refer to them as egg-shaped confidence regions (see application).

We consider the general linear hypothesis $H_0:\mathbf{R}\times \mathbf{F}\mathbf{C}=\mathbf{r}$, where the q × 6 matrix R has rank q ≤ 6, and r is a q × 1 vector. For example, by taking

$$\mathbf{R}=\left[\begin{array}{cccccc}1& -1& 0& 0& 0& 0\\ 0& 0& 1& -1& 0& 0\\ 0& 0& 0& 0& 1& -1\end{array}\right]$$

and r = 0 one tests the null hypothesis that states that misclassification probabilities are equal within each group. By taking

$$\mathbf{R}=\left[\begin{array}{c}\begin{array}{cccccc}0& 0& 1& -1& 0& 0\end{array}\end{array}\right]$$

and r = 0 one tests the null hypothesis that states that the probability of under-classification equals the probability of over-classification for group 2. By taking

$$\mathbf{R}=\left[\begin{array}{cccccc}{\pi }_1{C}_{2\mid 1}& {\pi }_1{C}_{3\mid 1}& 0& 0& 0& 0\\ 0& 0& {\pi }_2{C}_{1\mid 2}& {\pi }_2{C}_{3\mid 2}& 0& 0\\ 0& 0& 0& 0& {\pi }_3{C}_{1\mid 3}& {\pi }_3{C}_{2\mid 3}\end{array}\right]$$

one tests the null hypothesis that the expected costs incurred for each group separately equals some pre-specified values in the 3 × 1 vector r. By taking

$$\mathbf{R}=\left[\begin{array}{cccccc}{\pi }_1& {\pi }_1& {\pi }_2& {\pi }_2& {\pi }_3& {\pi }_3\end{array}\right]$$

one tests the null hypothesis that the overall probability of disagreement with the gold standard in the target population is equal to a pre-specified value r.

To test the general linear hypothesis we can employ the Wald test, that is, for a given significance level α, we reject the null hypothesis if

$${(\mathbf{R}\times \widehat{\mathbf{F}\mathbf{C}}-\mathbf{r})}^T{\left[\mathbf{R}{\widehat{\mathbf{V}}}^{\ast }{\mathbf{R}}^T\right]}^{-1}(\mathbf{R}\times \widehat{\mathbf{F}\mathbf{C}}-\mathbf{r})>{\chi }_{q;\alpha }^2,$$ (20)

where ${\widehat{\mathbf{V}}}^{\ast }$ equals the estimated 6 × 6 covariance matrix of the estimated untransformed FC rates which can be obtained by the delta method from $\widehat{\mathbf{\text{V}}}$, that is

$$\widehat{Var}\left({\widehat{FC}}_{j\mid i}\right)\approx \frac{\partial {\widehat{FC}}_{j\mid i}}{\partial {\text{Φ}}^{-1}\left({\widehat{FC}}_{j\mid i}\right)}\times \widehat{Var}\left({\text{Φ}}^{-1}\left({\widehat{FC}}_{j\mid i}\right)\right)$$ (21)

and similarly for $\widehat{Cov}\left({\widehat{FC}}_{j\mid i},{\widehat{FC}}_{j^{\prime }\mid i}\right)$.

A potentially interesting clinical question within a given disease group is whether one FC rate is significantly higher than the other. In particular, we need to make inference towards the direction of our falsely classified individuals that belong to a certain true disease status. For this purpose, we test the null hypothesis: $H_0:F{C}_{j\mid i}=F{C}_{j^{\prime }\mid i}$, where $j\ne j^{\prime }$ versus either the two-sided alternative $H_1^{\ast }:F{C}_{j\mid i}\ne F{C}_{j^{\prime }\mid i}$ or, if warranted, a one-sided alternative. For the two sided alternative we can use the Wald test in (20). For this particular hypothesis we prefer to conduct asymptotic Z tests in the probit scale. The probit scale has been recommended before in ROC settings (see Bantis and Feng (2016, 2018)). Hence, we use the test statistic:

$$Z^{\ast }=\frac{{\text{Φ}}^{-1}\left({\widehat{FC}}_{j\mid i}\right)-{\text{Φ}}^{-1}\left({\widehat{FC}}_{j^{\prime }\mid i}\right)}{\sqrt{\widehat{Var}\left({\text{Φ}}^{-1}\left({\widehat{FC}}_{j\mid i}\right)\right)+\widehat{Var}\left({\text{Φ}}^{-1}\left({\widehat{FC}}_{j^{\prime }\mid i}\right)\right)-2\widehat{Cov}\left({\text{Φ}}^{-1}\left({\widehat{FC}}_{j\mid i}\right),{\text{Φ}}^{-1}\left({\widehat{FC}}_{j^{\prime }\mid i}\right)\right)}}$$ (22)

that asymptotically follows a N(0, 1) under the null hypothesis. The components involved in Z^∗ have already been derived. The expression above implies that for the marginal 95% confidence interval for each $F{C}_{j\mid i}$ we use:

$$\left[\text{Φ}\left({\text{Φ}}^{-1}\left({\widehat{FC}}_{j\mid i}\right)-1.96\sqrt{\widehat{Var}\left({\text{Φ}}^{-1}\left({\widehat{FC}}_{j\mid i}\right)\right)}\right),\text{Φ}\left({\text{Φ}}^{-1}\left({\widehat{FC}}_{j\mid i}\right)-1.96\sqrt{\widehat{Var}\left({\text{Φ}}^{-1}\left({\widehat{FC}}_{j\mid i}\right)\right)}\right)\right].$$ (23)

Such a strategy regarding transforming using ${\text{Φ}}^{-1}(\cdot )$ and back-transforming the endpoints of the confidence intervals using $\text{Φ}(\cdot )$ has been also discussed in Bantis and Feng (2016) and is preferred over the traditional marginal confidence intervals that operate directly on the true or false class rates.

We note that our methods can be trivially extended to the case of known prevalences, using equations in (11). However, a global minimum of the expected cost function may not be obtained in some scenarios and additional restrictions are imposed on the parameter space that are not accounted for in this paper. These issues will be investigated by the authors in the future.

Finally, we note here that when one assumes a proper Normal biomarker, that is ${\sigma }_1={\sigma }_2={\sigma }_3$, maximizing the Youden index yields $c_1^{\ast }=\frac{{\mu }_1+{\mu }_2}{2}$ and $c_2^{\ast }=\frac{{\mu }_2+{\mu }_3}{2}$. Our methods extend easily to this case by applying the delta method using the estimated parameter vector $\widehat{\theta }={\left({\widehat{\mu }}_1,{\widehat{\mu }}_2,{\widehat{\mu }}_3,\widehat{\sigma }\right)}^T$, where $\widehat{\sigma }$ is the usual pooled sample standard deviation in the classical one-way Anova. Furthermore, in the case of a proper Normal biomarker, cost-adjusted and prevelance-adjusted cutoffs are easily obtained as unique solutions of the equations in (13), which are solved numerically. The methods described in this section can then be extended by appealing to the delta method for implicitly defined random variables, as discussed in [21].

4. Box-Cox approach

Even though the normality assumption allows closed-form expressions for the cutoffs and the subsequent inferences, it is most often not justified by the data at hand. However, it might be the case that a monotone transformation can yield measurements that are approximately normal. The so called Box-Cox transformation [22] has been used in the ROC literature before ([23]–[27]). It is defined by:

$$Y_{1^{(\lambda )}}=\{\begin{array}{cc}\frac{Y_1^{\lambda }-1}{\lambda },& \lambda \ne 0\\ \log \left(Y_1\right),& \lambda =0\end{array}$$

and similarly for Y_2^(λ) and Y_3^(λ). The corresponding log-likelihood function to be maximized is:

$$\begin{gathered} l(\gamma )\propto -n_{1}log\left({\sigma }_{1^{(\lambda )}}\right)\frac{1}{2{\left({\sigma }_{1^{(\lambda )}}\right)}^2}\sum _{i=1}^{n_1}{\left(Y_{{1i}^{(\lambda )}}-{\mu }_{1^{(\lambda )}}\right)}^2+(\lambda -1)\sum _{i=1}^{n_1}log\left(Y_{1i}\right) \\ -n_{2}log\left({\sigma }_{1^{(\lambda )}}\right)\frac{1}{2{\left({\sigma }_{2^{(\lambda )}}\right)}^2}\sum _{i=1}^{n_2}{\left(Y_{{2i}^{(\lambda )}}-{\mu }_{2^{(\lambda )}}\right)}^2+(\lambda -1)\sum _{i=1}^{n_2}log\left(Y_{2i}\right) \\ -n_{3}log\left({\sigma }_{3^{(\lambda )}}\right)\frac{1}{2{\left({\sigma }_{3^{(\lambda )}}\right)}^2}\sum _{i=1}^{n_3}{\left(Y_{3i^{(\lambda )}}-{\mu }_{3^{(\lambda )}}\right)}^2+(\lambda -1)\sum _{i=1}^{n_3}log\left(Y_{3i}\right) \end{gathered}$$ (24)

where $\gamma =\left({\mu }_{1^{(\lambda )}},{\sigma }_{1^{(\lambda )}},{\mu }_{2^{(\lambda )}},{\sigma }_{2^{(\lambda )}},{\mu }_{3^{(\lambda )}},{\sigma }_{3^{(\lambda )}},\lambda \right)$. For a given λ, the estimates of all parameters in γ can be derived in closed-form. An estimate of λ is derived by maximizing the underlying profile likelihood (for the computational details see Bantis et al. 2014). The 7 × 7 observed information matrix I is of the form:

$$I=-\left(\begin{array}{ccccccc}\frac{{\partial }^{2}l(\gamma )}{\partial {\mu }_{1^{(\lambda )}}^2}& 0& 0& 0& 0& 0& \frac{\partial l(\gamma )}{\partial {\mu }_{1^{(\lambda )}}\partial \lambda }\\ 0& \frac{{\partial }^{2}l(\gamma )}{\partial {\sigma }_{1^{(\lambda )}}^2}& 0& 0& 0& 0& \frac{\partial l(\gamma )}{\partial {\sigma }_{1^{(\lambda )}}\partial \lambda }\\ 0& 0& \frac{{\partial }^{2}l(\gamma )}{\partial {\mu }_{2^{(\lambda )}}^2}& 0& 0& 0& \frac{\partial l(\gamma )}{\partial {\mu }_{2^{(\lambda )}}\partial \lambda }\\ 0& 0& 0& \frac{{\partial }^{2}l(\gamma )}{\partial {\sigma }_{2^{(\lambda )}}^2}& 0& 0& \frac{\partial l(\gamma )}{\partial {\sigma }_{2^{(\lambda )}}\partial \lambda }\\ 0& 0& 0& 0& \frac{{\partial }^{2}l(\gamma )}{\partial {\sigma }_{3^{(\lambda )}}^2}& 0& \frac{\partial l(\gamma )}{\partial {\sigma }_{3^{(\lambda )}}\partial \lambda }\\ 0& 0& 0& 0& 0& \frac{{\partial }^{2}l(\gamma )}{\partial {\sigma }_{3^{(\lambda )}}^2}& \frac{\partial l(\gamma )}{\partial {\sigma }_{3^{(\lambda )}}\partial \lambda }\\ \frac{\partial l(\gamma )}{\partial {\mu }_{1^{(\lambda )}}\partial \lambda }& \frac{\partial l(\gamma )}{\partial {\sigma }_{1^{(\lambda )}}\partial \lambda }& \frac{\partial l(\gamma )}{\partial {\mu }_{2^{(\lambda )}}\partial \lambda }& \frac{\partial l(\gamma )}{\partial {\sigma }_{2^{(\lambda )}}\partial \lambda }& \frac{\partial l(\gamma )}{\partial {\mu }_{3^{(\lambda )}}\partial \lambda }& \frac{\partial l(\gamma )}{\partial {\sigma }_{3^{(\lambda )}}\partial \lambda }& \frac{{\partial }^{2}l(\gamma )}{\partial {\lambda }^2}\end{array}\right)$$

By extracting the upper left (6 × 6) part of $I^{-1}(\widehat{\gamma })$, denoted by $\widehat{\mathbf{\text{Σ}}}$ we can obtain an estimate of the variance covariance matrix of the vector ${\left({\mu }_{1^{(\lambda )}},{\sigma }_{1^{(\lambda )}},{\mu }_{2^{(\lambda )}},{\sigma }_{2^{(\lambda )}},{\mu }_{3^{(\lambda )}},{\sigma }_{3^{(\lambda )}}\right)}^T$. Then, statistical inference proceeds as in the previous section.

5. Non-Parametric Kernel-Based approach

Even though the Box-Cox approach provides a more flexible parametric alternative compared to the restrictive normality assumption, it might be the case that it fails to transform the data to approximate normality. In those cases, a non-parametric approach would be preferable. Here we explore a kernel-based approach (see [28] for an overview). A kernel density estimate for the $i=1,2,\dots ,n_1$ measurements that correspond to the first group, i.e. Y_1i, is given by:

$${\widehat{f}}_1(y)=\frac{1}{n_1{h}_1}\sum _{i=1}^{n_1}K\left(\frac{y-Y_{1i}}{h_1}\right),$$ (25)

and the expressions are similar for ${\widehat{f}}_1$ and f₂ that correspond to group 2 and 3. We employ the normal kernel i.e. $K(t)=\frac{1}{\sqrt{2\pi }}{e}^{-\frac{t^2}{2}}$, and $h_1,h_2,h_3$ are the bandwidths for group 1, 2 and 3 respectively. These bandwidths can be found using Silverman’s rule and are given by $h_i=0.9min\left(sd\left(y_i\right),iqr\left(y_i\right)/1.34\right)n_i^{-0.2}$, where sd and iqr refer to the standard deviation and inter-quartile range, respectively. Silverman [28] presents bandwidths that are optimal in terms of the asymptotic integrated mean squared error (AMISE), when normal kernels are employed.

• Step 1: Apply the Box-Cox transformation using Y₁, Y₂, and Y₃ to derive the estimate of the common transformation parameter λ, and hence the Box-Cox transformed scores $Y_1^{(\lambda )}$, $Y_2^{(\lambda )}$ and $Y_3^{(\lambda )}$.

• Step 2: Based on $Y_1^{(\lambda )}$, $Y_2^{(\lambda )}$, and $Y_3^{(\lambda )}$ employ the kernel density estimate to estimate f₁, f₂ and f₃.

• Step 3: Based on the kernel estimates of f₁, f₂, f₃, calculate the kernel-based estimates of all 6 FC rates, i.e.: ${\widehat{FC}}_{2\mid 1}$, ${\widehat{FC}}_{3\mid 1}$, ${\widehat{FC}}_{1\mid 2}$, ${\widehat{FC}}_{3\mid 2}$, ${\widehat{FC}}_{1\mid 3}$, ${\widehat{FC}}_{2\mid 3}$.

• Step 4: Sample with replacement separately from each Y₁, Y₂, and Y₃ and obtain the bootstrap based sample for the current b–th bootstrap iteration, denoted by Y_1(b), Y_2(b), and Y_3(b).

• Step 5: Apply the Box-Cox transformation using Y_1(b), Y_2(b), and Y_3(b), in order to derive the current common transformation parameter ${\widehat{\lambda }}_{(b)}$, and hence the Box-Cox transformed bootstrap sample, denoted by $Y_{1(b)}^{\left({\widehat{\lambda }}_{(b)}\right)}$, $Y_{2(b)}^{\left({\widehat{\lambda }}_{(b)}\right)}$, and $Y_{3(b)}^{\left({\widehat{\lambda }}_{(b)}\right)}$.

• Step 6: Based on $Y_{1(b)}^{\left({\widehat{\lambda }}_{(b)}\right)}$, $Y_{2(b)}^{\left({\widehat{\lambda }}_{(b)}\right)}$, and $Y_{3(b)}^{\left({\widehat{\lambda }}_{(b)}\right)}$, apply the kernel density estimate (the bandwidth calculation is based on the current samples) to obtain estimates for the densities f_1(b), f_2(b), and f_3(b).

• Step 7: Using the density estimates of the previous step calculate all 6 FC rates for the current bootstrap sample: ${\widehat{FC}}_{2\mid 1(b)}$, ${\widehat{FC}}_{3\mid 1(b)}$, ${\widehat{FC}}_{1\mid 2(b)}$, ${\widehat{FC}}_{3\mid 2(b)}$, ${\widehat{FC}}_{1\mid 3(b)}$, ${\widehat{FC}}_{2\mid 3(b)}$.

• Step 8: Repeat steps 4–7 B times, and based on the B bootstrap estimates of all 6 FC rates, calculate the 6 × 6 empirical covariance matrix that corresponds to the ${\text{Φ}}^{-1}(\cdot )$ transformed FC rates.

In practice, we employ B = 400 bootstrap samples. Statistical inference then proceeds similarly to section 3.

Based on the kernel estimates in Step 3 and the estimated variance covariance matrix in Step 8, ${\widehat{V}}_{\left(boots,{\text{Φ}}^{-1}(\cdot )\right)}$, we can proceed with the construction of the elliptical confidence region given in (19) and back-transform its coordinates using Φ(·). To do so, we need to employ the 2 × 2 kernel version of V₂ which can be simply extracted from our full 6 × 6 bootstrap based ${\widehat{V}}_{\left(boots,{\text{Φ}}^{-1}(\cdot )\right)}$. In order to compare two FC rates we employ Z^∗. All expressions involved in its denominator can be extracted from ${\widehat{V}}_{\left({\mathit{\text{boots}},\text{Φ}}^{-1}(\cdot )\right)}$, while the numerator expressions are simply the ${\text{Φ}}^{-1}(\cdot )$ transformations of the estimates of Step 3. Note that both ${\widehat{V}}_{(boots)}$ and ${\widehat{V}}_{\left(\mathit{\text{boots}},{\text{Φ}}^{-1}(\cdot )\right)}$ take into account the variability of the estimated Box-Cox related transformation parameter. For the kernel-based marginal confidence intervals we can be based on (23) where all estimates are kernel-based and all variance estimates are derived through the bootstrap described in this section.

6. Simulation Studies

Our simulation studies consist of two parts: (a) investigation of the coverage and width (or areas) of our confidence intervals (or regions) for all true and FC rates under different parametric scenarios with all proposed approaches, and (b): investigation of the size and power of our proposed statistic Z^∗ under various scenarios.

6.1. Confidence intervals for all classification rates

We consider generating data based on three settings: i) all three groups are generated from three different normal distributions, ii) all three groups are generated from three different log-normal distributions, iii) all three groups are generated from three different gamma distributions, and iv) groups 1 and 2 are generated from two normal distributions, while group 3 is generated from a two-component normal mixture that exhibits bi-modality. All parameters utilized for all aforementioned models are given in Table 1 of the Web Appendix. For configurations i) and ii) we have considered different volumes under the ROC surface (VUS values) equal to 0.5 and 0.7. The sample sizes we consider for each group are 50, 100, and 200. The parameter values of the distributions for each scenario are given in Tables 1 and 2 of the Web Appendix.

We observe that when the data are generated from normal distributions (configuration (i)), the delta-based approach that relies on normality performs best, as expected. The coverage is close to the targeted one (i.e. 95%) in all cases, yielding systematically smaller widths (or areas) for the proposed marginal CIs and joint egg-shaped confidence regions compared to the Box-Cox based approach or the Kernel-based method. We observe that the Box-Cox approach yields slightly inflated widths (or confidence areas), because in takes into account the variability of an extra parameter that refers to the estimated λ. Even though the Kernel-based method provides nice coverage properties as well, these come with the cost of even higher widths and areas. See Tables 3 of the Web Appendix for all numerical results.

For configuration (ii) and (iii) where the data are generated from log-normal or gamma distributions, we observe that the Box-Cox performs best. Note that the gamma model lies outside the Box-Cox family and our results indicate that the Box-Cox based approach is flexible enough to capture such violations of normality. It outperforms the non-parametric kernel-based (KernelBC) approach in terms of widths/areas. The flexibility of the Box-Cox approach has been acknowledged before under different settings (see [26] [25]). See Table 4 of the Web Appendix for the corresponding numerical results.

For configuration (iii) we explore a setting that provides a severe violation of normality with the third group exhibiting bimodality. In this setting, we observe the Box-Cox based approach has a poor performance for several classification rates, indicating its limited use for more complex distributions (Table 2 of the Web Appendix). For this scenario, we observe that the kernel-based approach performs best, illustrating its robustness (Table 4 of the Web Appendix).

6.2. Power and size simulation

In the second part of our simulations, we explore the size and the power of Z^∗ for testing the null hypothesis $F{C}_{1\mid 2}=F{C}_{3\mid 2}$. We focus on this null hypothesis as it is often of clinical interest to know whether there misdiagnosis of the benign stage to either group 1 or the aggressive group 3. We consider this setting as more common compared to misclassifying individuals from group 1 to group 3 and vice-versa, even though our proposed statistic accommodates any comparison. To explore the size, we consider sample sizes of 30, 50, 100, and 200. The data are generated based on configurations (i), (ii), and (iii) discussed in the previous subsection so that $\left|F{C}_{1\mid 2}-F{C}_{3\mid 2}\right|=0.00,0.05,0.15$ and 0.20. The distributions considered to generate the data are given in Table 2 of the Web Appendix. All power-related numerical results of the simulations are reported in Table 5 of the Web Appendix.

Our findings are analogous to those presented in the previous subsection and are summarized in Figure 4. We note that the plots included in Figure 4 are comparable column-wise, as different operating points are considered for each distributional scenario. This is because we are interested in fixing specific values for $\left|F{C}_{1\mid 2}-F_{3\mid 2}\right|$. When the score measurements are generated based on normal models, then our delta-based approach performs best, as expected in terms of power. For the gamma distributed scores, the Box-Cox based approach achieves higher power compared to the Kernel, and is fairly robust in such situations even though the gamma distribution lies outside the Box-Cox family. However, for more complex models (such as the one utilized in configuration (iii)) the Box-Cox collapses and does not attain nice size properties. The Kernel is the most robust approach yielding satisfactory size and power in all scenarios.

Figure 4.

Power attained by Z^∗ when testing the hypothesis $\left|F{C}_{1\mid 2}-F{C}_{3\mid 2}\right|=0$. The difference $\left|F{C}_{1\mid 2}-F{C}_{3\mid 2}\right|$ is set to be 0, 0.05, 0.10, 0.15 and 0.20. Each panel refers to a specific simulation scenario that involves data generated either by normal distributions, log-normal distributions, gamma distributions, and a bimodal mixture. The sample sizes considered are (50, 50, 50), (100, 100, 100) and (200, 200, 200).

As requested by a referee, we also considered simulations that assume normality when we generate data from the scenarios that involve gamma distributions, lognormal distributions, and the bimodal mixture. For the gamma and the lognormal related scenarios, the assumption of normality yields a size that is far from the targeted 0.05 level and generally > 0.90 in all cases. For the bimodal mixture scenario, wrongly assuming normality yields sizes of 0.133, 0.242, and 0.442 for sample sizes (50,50,50), (100,100,100) and (200,200,200) respectively (see also Table 5 of the Web Appendix).

In summary, we recommend the following strategy for a given application. Test for normality for each one of the three groups under study. If normality is justified, then proceed with the delta-based approach. If normality is not justified, then proceed with testing the Box-Cox transformed scores for normality, and if successful, the Box-Cox based approach is recommended. In all other situations the kernel-based approach is preferred.

7. Pancreatic Cancer Application

CA19–9 is currently in clinical use as a pancreatic ductal adenocarcinoma (PDAC) biomarker. However, its performance is limited in detecting early-stage disease. Therefore, there is clinical interest in identifying new promising markers towards this direction. A study presented in [29] investigates a set of protein-based markers that could potentially outperform CA19.9. The authors carry out sequential validations starting with 17 protein biomarker candidates to determine which markers can improve detection of early-stage disease. Candidate biomarkers are subjected to enzyme-linked immunosorbent assay based sequential validation using independent multiple sample cohorts consisting of PDAC cases (n₃ = 87), benign pancreatic disease (n₂ = 93), and healthy controls (n₁ = 169). A marker considered in this study is the insulin like growth factor binding protein 2, or simply IGFBP2 [29]. It is a protein-coding gene. The authors of this study provided a pairwise analysis.

Here, we focus on evaluating IGFBP2 in simultaneously discriminating all three groups under study; group 1: healthy (controls), group 2: chronic pancreatitis, and group 3: PDAC. We investigate the biomarker’s properties at a Youden-optimized pair of cutoffs and assess any potential inflated FC rates. For this marker, even after the Box-Cox transformation, normality is violated. We use the Anderson Darling test for normality (see [30] and [31]). The Anderson-Darling related p-values are 0.0031, 0.0005, and 0.0062 for the Box-Cox transformed groups 1, 2, and 3, respectively. Therefore, we consider our kernel approach in this application. All estimated classification rates along with their marginal confidence intervals are given in Table 1. The corresponding box-plots for all three groups and the three empirically estimated cumulative distribution functions are presented in Figure 1 of the Web Appendix. Both the ROC surface based on IGFBP2 and its performance at the Youden-based optimal operating point are illustrated in Figure 5. On the top panels (panels (a) and (b)) of Figure 5 we visualize the ROC surface along with the estimated triplet of (TC₁, TC₂, TC₃) and its 95% confidence space that corresponds to the optimized cutoffs (${\widehat{c}}_1^{\ast }=0.3282$ and ${\widehat{c}}_2^{\ast }=1.1987$). At the bottom left panel (panel (c)) we visualize the three 95% confidence regions for: i) ($F{C}_{2\mid 1}$, $F{C}_{3\mid 1}$) in blue color, ii) ($F{C}_{1\mid 2}$, $F{C}_{3\mid 2}$) in green color, and iii) ($F{C}_{1\mid 3}$, $F{C}_{2\mid 3}$) in red color. For a simultaneous visualization we propose the plot given in panel (d) of Figure 5 which we call ‘clock-plot’. The clock-plot provides an immediate visual representation of all TC and FC class rates along with their marginal 95% confidence intervals that are derived as described in Section 5 since the proposed kernel-based strategy is utilized for this data set. The light-blue arms (thick lines) represent the TC rates. The more extended they are, the better the biomarker’s performance at the estimated cutoffs is. For a perfect marker, these arms extend to the perimeter of the largest circle with radius equal to 1. To the left and right of each light-blue (TC) arm, we visualize the corresponding ‘bleeding’. That is, we depict the corresponding FCs that relate to that particular TC rate. This means that if we add the length of a TC arm to the sum of the lengths of the two arms on its left and right side, then the result is always 1. Note that an FC rate is allowed to exceed any TC rate. Their trade-off is a function of the location of the cutoffs.

Table 1.

Application results table: Estimated true and false class rates for biomarker IGFBP2 regarding the pancreatic cancer data set. The optimized cutoffs are 0.3282 and 1.1987

Classification Rate	Kernel based point estimates	Kernel based 95% confidence interval
TCI	0.6820	0.5211–0.8143
TC 2	0.4318	0.2030–0.6870
TC 3	0.3466	0.1778–0.5537
$F{C}_{2\mid 1}$	0.2765	0.1416–0.4549
$F{C}_{3\mid 1}$	0.0414	0.0034–0.2217
$F{C}_{1\mid 2}$	0.4054	0.2588 – 0.5667
$F{C}_{3\mid 2}$	0.1628	0.0588 – 0.3443
$F{C}_{1\mid 3}$	0.2932	0.1614–0.4603
$F{C}_{2\mid 3}$	0.3602	0.1825–0.5754

Figure 5.

Panel (a): 95% confidence egg-shaped space for the triplet (TC₁, TC₂, TC₃) (left). Panel (b): Corresponding ROC surface for IGFBP2 (right). Panel (c): 95% egg shaped confidence regions for: i) $\left(F{C}_{2\mid 1},F{C}_{3\mid 1}\right)$ in blue color, ii) $\left(F{C}_{1\mid 2},F{C}_{3\mid 2}\right)$ in green color, and iii) $\left(F{C}_{1\mid 3},F{C}_{2\mid 3}\right)$ in red color. Panel (d): (clock-plot) a visualization of all TC and FC rates along with their marginal confidence intervals visualized with red stars alongside each clock arm.

In the clock-plot we can visually explore an interesting behavior concerning the misclassification rates for subjects that suffer from pancreatitis (group 2). ${\widehat{FC}}_{3\mid 2}$ is lower than 0.2 while ${\widehat{FC}}_{1\mid 2}$ is close to 0.4. That means, that given that an individual belongs to group 2 of chronic pancreatitis, it is twice more likely to be misclassified as healthy than having PDAC. We also see that TC₂ is approximately 0.4, and approximately equal to $F{C}_{1\mid 2}$. Hence, without making any formal arguments yet, we can easily observe that the reason TC₂ is not high enough, is because of false decisions ‘bleeding’ to the healthy group as opposed to the PDAC group. That is, based on these cutoffs, it is more likely to misclassify an individual who has chronic pancreatitis as healthy rather as a PDAC patient. A natural question is whether this difference in the two FC is statistically significant. More formally, we are interested in testing the null hypothesis that $H_0:F{C}_{1\mid 2}=F{C}_{3\mid 2}$. Based on Z^∗ the corresponding p−value is 0.0366, and thus we reject this null hypothesis at a nominal level of 5%.

Related to TC₁, which is estimated to be 0.6804, we note the following. Healthy subjects tend to be misclassified as having chronic-pancreatitis (group 2) (${\widehat{FC}}_{2\mid 1}=0.2766$) rather than having PDAC (group 3) (${\widehat{FC}}_{2\mid 1}=0.0430$). If we focus on PDAC patients, we observe an estimated TC₃ equal 0.3466 and false decisions are almost equally likely with their difference not being statistically significant (p – value=0.6012 based on our proposed Z^∗ statistic).

For the results discussed so far we have assumed equal costs and prevalences. For illustration purposes we also show how these results would change under a configuration where the misclassification costs and the underlying prevalences are available. Let us assume the following costs (that essentially can be considered as relevant costs in some units): $C_{2\mid 1}=10000$, $C_{3\mid 1}=20000$, $C_{1\mid 2}=15000$, $C_{3\mid 2}=2000$, $C_{1\mid 3}=7000$, $C_{2\mid 3}=6000$ with prevalences π₁ = 0.56, π₂ = 0.26, π₃ = 0.18. In this case equations in (5) yield two roots for the first cutoff, those being −0.5254 and 1.6413, and two roots for the second cutoff, those being −0.8633 and 3.4213 (see Figure 6). This illustrates the necessity of exploring the underlying cost function based on which we observe that the global minimum is at c₁ = 1.6413 and c₂ = 3.4213. This global minimum is illustrated in Figure 7 and the profiles of the cost function with respect to t₁ and t₂ are presented in Figure 8. Then, the true class rates are $\left(T{C}_1,T{C}_2,T{C}_3\right)=(0.7658,0.4117,0.2855)$ and the corresponding FC rates are $\left(F{C}_{21},F{C}_{31},F{C}_{12},F{C}_{32},F{C}_{13},F{C}_{23}\right)=(0.2321,0.0021,0.4904,0.0979,0.3645,0.3500)$.

Figure 6.

Visualization of equations in (5) for the pancreatic cancer data after emplying the hypothesized costs and prevalences. These are: $C_{2\mid 1}=10000$, $C_{3\mid 1}=20000$, $C_{1\mid 2}=15000$, $C_{3\mid 2}=2000$, $C_{1\mid 3}=7000$, $C_{2\mid 3}=6000$ with prevalences π₁ = 0.56, π₂ = 0.26, π₃ = 0.18. We observe that there are two roots for each of the equations. This implies the necessity of exploring the underlying cost function which is illustrated in Figure 7

Figure 7.

Visualization of the cost function that corresponds to the hypothesized costs and prevalences for the discussed pancreatic cancer data set. We further illustrate the profile of this function with respect to t₁ and t₂ in Figure 8

Figure 8.

Illustrating the profile of the cost function that refers to the hypothesized costs and prevalences for the pancreatic cancer data set with respect to t₁ and t₂.

8. Discussion

There is strong clinical interest related to new biomarkers that could contribute to improved diagnosis of early stage PDAC and pancreatitis. A traditional marker currently used in clinical practice is CA19.9. However, it exhibits limited utility since its accuracy varies with disease stage [32]. This implies the need of exploration of new markers that will be able to contribute to that direction. Common statistical techniques fail to reveal the strengths and shortcomings of new biomarker candidates, as these are based on overall accuracy measures such as the AUC (or VUS in the three-class case) which cannot provide limited insights under a clinical decision making framework. False class rates play a crucial role in understanding the behavior of a biomarker. Knowing the misclassification rates and being able to make inferences around them provides a cornerstone in assessing clinical decision-making guidelines. Such a need is clearly reflected through the implied costs, as it is rarely the case that misclassifying towards any direction has the same impact. This heavily depends on the invasiveness of the implied work-up, the stage of the disease, the cost of the work-up both in resources as well as in health-related issues/side effects. In this paper, we provide a complete framework for evaluating all classification rates under a three-class setting that corresponds to the Youden-based optimized cutoffs. In addition, our methods can take into account underlying costs and prevalences, provided there are some additional restrictions in the parameter space (other than mean ordering are satisfied). We note that it is challenging to know the underlying costs for a given setting as these are affected by the cost of the test, the cost of the follow up and/or how invasive it is, the stage of the disease and how lethal it is and other factors.

By accommodating the underlying costs and prevalences, we discuss a generalization of the commonly used Youden index, illustrating that all three groups contribute to estimating both underlying cutoffs. This is not the case for the 3-class Youden index discussed in [11]. Even though other alternatives to the Youden index have been studied in the literature on the basis of avoiding the under-utilization of samples (see [14], [33], and [34] among others), here we illustrate that this under-utilization (of the Youden index) is simply a special case of a more general framework, that in fact allows all three groups to contribute to the estimation of both underlying cutoffs. We study parametric, flexible parametric and non-parametric kernel-based techniques. Current literature referring to the statistical evaluation of biomarkers in such settings revolves around the true classification rates. This only provides an incomplete picture of the operating characteristics of a biomarker. The work presented in this paper provides a full framework that bridges this important literature gap.

Figure 3.

The profile of the cost function visualized in Figure 2 is provided with respect to t₁ and t₂. Both these profile functions refer to the unequal variances numerical example that we discuss above (section 2.1.1).