Two-step error-controlling classifiers with application to cost-effective disease diagnosis

Abstract

Accurate classifiers that utilize novel biomarkers and readily available predictors significantly enhance decision-making in various clinical scenarios, such as assessing the need for biopsies in cancer diagnosis. When classification performance is limited, a decision framework can be applied to rule in or rule out invasive diagnostic procedures while incorporating a neutral zone for indeterminate classifications. Building on this framework, we propose a new family of two-step classifiers that selectively employ costly biomarker testing for a targeted subset of individuals undergoing multiple evaluations. The optimal solution expands upon the Neyman-Pearson Lemma, highlighting a vital trade-off between the costs of expensive biomarker measurements and improving classification performance while minimizing uncertainty in the decision process. We demonstrate the practical utility of our approach through a biomarker study focused on prostate cancer diagnosis.

1. Introduction

Using classifiers built upon emerging biomarkers and existing clinical predictors, medical tests can improve clinical decisions with better stratification. For example, in the setting of prostate cancer diagnosis, patients are recommended for invasive biopsy if they are identified as being at high risk of having clinically significant cancer and otherwise are recommended to forgo biopsy. However, clinical classifiers are often imperfect, rendering two types of misclassification errors with different clinical consequences. The false negatives may result in skipping biopsies for patients harboring clinically significant diseases. The false positives will lead to overtesting healthy individuals with invasive and unnecessary procedures. Therefore, decision-making must control the false negative rate (FNR) and the false positive rate (FPR) at clinically acceptable levels.

Recently, a new family of classifiers has emerged, known as classification with confidence (Lei, 2014), neutral zone classifiers (Jeske and Smith, 2018; Jeske et al., 2020), or set-valued classifiers with bounded error rates (Sadinle et al., 2019). We refer to this family of models as error-controlling classifiers, which aim to control the FNR and the FPR simultaneously at predefined levels, denoted as ${\alpha }_1$ and ${\alpha }_0$, respectively. Due to their typically suboptimal performance, these classifiers achieve this goal by confidently classifying clear-cut cases while leaving ambiguous cases in a “neutral zone” without making a decision. The proportion of unclassified cases is called the neutral zone rate (NZR), a key metric of the classifier’s performance. A better classifier should have a smaller NZR for a given pair of ${\alpha }_1$ and ${\alpha }_0$, leaving fewer cases unclassified. For example, Tosoian et al. (2023) constructed an error-controlling classifier for prostate cancer diagnosis by establishing low-risk and high-risk cutoffs for a risk score called MyProstateScore (MPS) at ${\alpha }_1$ = 0.1 and ${\alpha }_0$ = 0.2, respectively. They suggested that individuals with MPS above the high-risk cutoff (or below the low-risk cutoff) could confidently proceed to (or forgo) biopsy.

Emerging biomarkers are opening up new opportunities for more accurate disease detection. For instance, Tosoian et al. (2024) demonstrated that the Prostate Cancer Prevention Trial risk score (PCPTrs), a widely used model based on commonly available clinical variables, achieves an 80% FPR at a 5% FNR for detecting clinically significant prostate cancer. However, when novel biomarkers are incorporated, the FPR is reduced to 63%, while maintaining the same 5% FNR. This improvement is promising but introduces additional biomarker testing, increasing the diagnostic burden and the associated costs. Such development raises important questions: Is it always necessary to conduct every available test for each individual? How can we construct diagnostic strategies to optimize performance while controlling errors and reducing cost and test burden?

To address these questions, we propose a new family of two-step error-controlling classifiers (TECs), formally described in Section 2. The two-step approach begins with a base model (e.g., PCPTrs) that classifies straightforward cases using readily available clinical data. For cases in the neutral zone, additional and potentially more costly information, such as genetic biomarkers from urine samples, is incorporated in an augmented model in the second step to refine the classification. By strategically allocating the error budget (i.e., ${\alpha }_1$ and ${\alpha }_0$) between the two steps, this approach ensures efficient use of resources while maintaining performance and controlling errors. Although this paper focuses on two-step classifiers, the methodology is flexible and can be extended to multi-step classifiers for even more complex diagnostic scenarios.

Cost-effective strategies for sequential medical testing have been previously studied. For example, Jeske et al. (2020) proposed a two-stage classifier that balances the overall error rate against the cost of expensive biomarkers, while other approaches optimize weighted overall error rates or the Generalized Youden Index (Ahmed et al., 2011). Our work extends the existing sequential testing literature in two key aspects. First, we explicitly control both the FPR and the FNR in the constraints along with the objective functions, allowing alignment with application-specific clinical priorities. Second, rather than enforcing a deterministic binary decision at the final stage, our approach permits a nonzero NZR at the last step and incorporates it directly into the dual objective functions.

The paper is organized as follows. In Section 2, we formally introduce the TEC framework. Section 3 discusses key properties of the TEC. The estimation and inference procedures are outlined in Section 4, followed by simulation studies in Section 5. We apply the TEC to the motivating example of prostate cancer diagnosis in Section 6. Finally, we conclude with a discussion in Section 7.

2. Framework

2.1. Review of Error-controlling Framework

Let $D\in (0,1)$ be the binary disease status and $h(\mathbf{X})$ the classifier, a function of the covariates $\mathbf{X}$. Specifically, $h(\mathbf{X})\to (0,N,1)$. $h(\mathbf{X})=1\text{or}h(\mathbf{X})=0$ indicates a definitive classification that recommends aggressive or conservative medical intervention based on $\mathbf{X}$, respectively. A particular output in this framework is $h(\mathbf{X})=N$, meaning no definitive classification or specific recommendation is given based on $\mathbf{X}$. The goal of such classifiers is to find $h(\mathbf{X})$ to minimize $P\{h(\mathbf{X})=N\}$, the NZR, under the error constraints:

$$\begin{gathered} \text{minimize}P\{h(\mathbf{X})=N\}≔NZR \\ \text{subject to}P\{h(\mathbf{X})=0\mid D=1\}\le {\alpha }_1\text{and}P\{h(\mathbf{X})=1\mid D=0\}\le {\alpha }_0. \end{gathered}$$

We set ${\alpha }_1$ and ${\alpha }_0$ to be small values to reflect the low tolerance levels of making false negative and false positive errors, respectively, for high-stakes decision-making. That is, we consider the scenario that $P\{h(\mathbf{X})=N\}>0$ given prespecified small values of ${\alpha }_1$ and ${\alpha }_0$.

For the optimality in achieving the goal of minimizing NZR, Lei (2014) gave the following lemma.

Lemma 1. With $P\{h(\mathbf{X})=0\mid D=1\}\le {\alpha }_1$ and $P\{h(\mathbf{X})=1\mid D=0\}\le {\alpha }_0$, the solution to $\arg {\min }_{h(\mathbf{X})}P\{h(\mathbf{X})=N\}$ is

$$h(\mathbf{X})=\{\begin{array}{cc}1,\hfill & if\eta (\mathbf{X})\ge t_0({\alpha }_0)\hfill \\ N,\hfill & if{t}_1({\alpha }_1)\le \eta (\mathbf{X})<t_0({\alpha }_0)\hfill \\ 0,\hfill & if\eta (\mathbf{X})<t_1({\alpha }_1)\hfill \end{array}\phantom{\}},$$

where $\eta (\mathbf{X})=P(D=1\mid X);t_0({\alpha }_0)$ and $t_1({\alpha }_1)$ are chosen such that $P\{\eta (\mathbf{X})\ge t_0({\alpha }_0)\mid D=0\}={\alpha }_0$ and $P\{\eta (\mathbf{X})<t_1({\alpha }_1)\mid D=1\}={\alpha }_1$.

This lemma can be seen as an extension of the result on the binary classification problem with a single error constraint, such as maximizing true positive rate (TPR) under a specific FPR constraint (McIntosh and Pepe, 2002). These optimality results, including those introduced in the later sections, can be seen as extensions of the Neyman-Pearson Lemma to the settings in classification.

In practice, the true conditional probability or the true risk $\eta (\mathbf{X})$ is unknown. We can use an estimated risk or a univariate risk score $R(\mathrm{X})$ with larger values indicative of $D=1$. To construct a $h(\mathrm{X})$, we may choose $t_k({\alpha }_k)$, for $k\in (0,1)$, based on the ${\alpha }_k$ quantiles of $R(\mathrm{X})$ among $D=k$ individuals. This construction procedure is used twice in the two-step extension described in the next subsection.

The minimization of NZR as the objective may be less apparent than standard problems, such as maximizing the percentage of unnecessary biopsy avoided (1-FPR) at a maximally tolerated percentage of true aggressive cancers missed (FNR). We note that under the error constraints on FPR and FNR, the classifier minimizing the NZR is also the classifier maximizing the correct classification rate (CCR), $P\{h(\mathbf{X})=D=1$ or $h(\mathbf{X})=D=0\}$. For example, the ideal classifier in a prostate cancer diagnosis setting only correctly recommends invasive biopsy for all individuals with high-grade cancer and correctly avoids any unnecessary biopsy. Among classifiers that yield specific low rates of missing aggressive cancer and unnecessary biopsies, a better classifier should make a larger proportion of correct recommendations or leave a smaller proportion of unclassified. This interpretation is summarized as Lemma 2.

Lemma 2. With $P\{h(\mathbf{X})=0\mid D=1\}\le {\alpha }_1$ and $P\{h(\mathbf{X})=1\mid D=0\}\le {\alpha }_0$,

$$\text{arg}\underset{h(\mathbf{X})}{\text{min}}P\{h(\mathbf{X})=N\}=\text{arg}\underset{h(\mathbf{X})}{\max }P\{h(\mathbf{X})=D\}.$$

This equivalence is evident from the following equation:

$$1-P\{h(\mathbf{X})=N\}=P\{h(\mathbf{X})=D\}+P\{h(\mathbf{X})=0\mid D=1\}P(D=1)+P\{h(\mathbf{X})=1\mid D=0\}P(D=0).$$

To conclude this subsection and familiarize readers with the error-controlling framework and the goal of minimizing NZR, we visualize the error-controlling classifier in the standard receiver operating characteristic (ROC) space as in Figure 1. Suppose we set ${\alpha }_0$ = 0.2 and ${\alpha }_1$ = 0.1, represented by a cross at (0.2, 0.9) in the ROC space. The ROC curve of a risk score, $R(\mathrm{X})$, is far below (0.2, 0.9), but we may choose two cutoffs, $t_0$ and $t_1$, to meet the error tolerance levels. The two cutoffs are determined by ${\alpha }_0$ and ${\alpha }_1$ respectively, which result in corresponding ${\beta }_0$ (i.e., FNR at $t_0$) and ${\beta }_1$ (i.e., FPR at $t_1$), respectively. The NZR is a weighted sum of the length of the two colored dashed lines in the ROC plot:

$$\begin{gathered} P\{h(\mathbf{X})=N\}=1-P\{R(\mathbf{X})<t_1\}-P\{R(\mathbf{X})>t_0\} \\ =P(D=1)\underset{\text{red line}}{\underbrace{\{1-{\alpha }_1-(1-{\beta }_0)\}}}+P(D=0)\underset{\text{blue line}}{\underbrace{\{{\beta }_1-{\alpha }_0\}}}. \end{gathered}$$

Figure 1:

Visualization of an error-controlling classifier

Notice that if the ROC curve of a risk score dominates another ROC curve, the superiority of this risk score is reflected by a smaller NZR due to shorter lengths of the two colored lines.

2.2. Two-step Error-controlling Classifiers (TEC)

In this extension, we consider a situation where $\mathbf{X}$ is partitioned into two parts $\mathbf{X}=({\mathbf{X}}_1,{\mathbf{X}}_2)$ with ${\mathbf{X}}_1$ being the baseline and routinely collected predictors (e.g., age) and ${\mathbf{X}}_2$ being the novel and expensive biomarkers. To stratify patients, we have a baseline risk score $R({\mathbf{X}}_1)$ as a function of ${\mathbf{X}}_1$ and an augmented risk score $R(\mathbf{X})$ including both ${\mathbf{X}}_1$ and ${\mathbf{X}}_2$. In the motivated example of prostate cancer diagnosis: $R({\mathbf{X}}_1)$ is the PCPTrs using routinely available clinical information, and we may only collect ${\mathbf{X}}_2$, additional novel markers, to form an augmented $R(\mathbf{X})$ in a subset of individuals for cost-effective diagnostic strategies. Within the error-controlling framework, one strategy is to construct a family of classifiers in two steps as follows, which is referred to as the Two-step Error-controlling Classifier (TEC).

The first step is to construct an error-controlling classifier based on $R({\mathbf{X}}_1)$ with a set of ${\alpha }_{(1)1}$ and ${\alpha }_{(1)0}$. The neutral zone from the first step is $N_{(1)}≔\{{\mathbf{x}}_1:t_{(1)1}({\alpha }_{(1)1})\le R({\mathbf{x}}_1)<t_{(1)0}({\alpha }_{(1)0})\}$. In the second step, for these individuals in $N_{(1)}$, ${\mathbf{X}}_2$ is collected, and another error-controlling classifier is constructed with $R(\mathbf{X})$ given a set of ${\alpha }_{(2)1}$ and ${\alpha }_{(2)0}$. The final neutral zone after the two steps is $N_{(2)}=\{\mathbf{x}:t_{(2)1}({\alpha }_{(2)1})<R(\mathbf{x})<t_{(2)0}({\alpha }_{(2)0}),{\mathbf{x}}_1\in N_{(1)}\}$. That is, the TEC takes the following form:

$$h(\mathbf{X})=\{\begin{array}{cc}1,\hfill & \text{if}R({\mathbf{X}}_1)\ge t_{(1)0}({\alpha }_{(1)0})\text{or}\{R({\mathbf{X}}_1)\ge t_{(1)1}({\alpha }_{(1)1})\text{and}R(\mathbf{X})\ge t_{(2)0}({\alpha }_{(2)0})\}\hfill \\ N,\hfill & \text{if}{t}_{(1)1}({\alpha }_{(1)1})\le R({\mathbf{X}}_1)<t_{(1)0}({\alpha }_{(1)0})\text{and}{t}_{(2)1}({\alpha }_{(2)1})\le R(\mathbf{X})<t_{(2)0}({\alpha }_{(2)0})\hfill \\ 0,\hfill & \text{if}R({\mathbf{X}}_1)<t_{(1)1}({\alpha }_{(1)1})\text{or}\{R({\mathbf{X}}_1)<t_{(1)0}({\alpha }_{(1)0})\text{and}R(\mathbf{X})<t_{(2)1}({\alpha }_{(2)1})\}\hfill \end{array}\phantom{\}}.$$

Within the error-controlling framework, we need to ensure that overall ${\alpha }_1$ and ${\alpha }_0$ are maintained after the two steps. To achieve this, we propose to control how much of the error budget is spent in each step. Specifically, for class $k\in (0,1)$, we choose ${\alpha }_{(1)k}$ and ${\alpha }_{(2)k}$ as

$$\begin{gathered} {\alpha }_{(1)k}=s\times {\alpha }_k, \\ {\alpha }_{(2)k}=(1-s)\times {\alpha }_k∕P({\mathbf{X}}_1\in N_{(1)}\mid D=k), \end{gathered}$$

where $s$ controls how much the error budget is spent in the first step. Note that we consider $s$ to be the same for both errors, and one may generalize it to class-specific $s_k$. This error budgeting scheme is analogous to alpha allocation in clinical trials with hypothesis testing on co-primary endpoints. In Section 4, we show that an estimation procedure based on this error budgeting scheme achieves the overall error control (Theorem 2).

We may pre-specify $s$ or determine $s$ based on the observed data. One may argue that we should spend less on the error budget in the first step (i.e., $s<0.5$ < 0.5) when the information from ${\mathbf{X}}_2$ may potentially avoid errors. In other words, the error-tolerance levels in the first step should be much more stringent if the definite classification is made without ${\mathbf{X}}_2$. Hence, $s$ may be chosen a priori based on this interpretation. Alternatively, we may choose $s$ based on a trade-off between the operating characteristics as described in the next section.

With this budgeting scheme indexed by $s$, the four cutoffs are

$$\begin{gathered} t_{(1)1}=F_{(1)}^{-1}({\alpha }_{(1)1}), \\ {t}_{(1)0}=G_{(1)}^{-1}(1-{\alpha }_{(1)0}), \\ {t}_{(2)1}=F_{(2)}^{-1}({\alpha }_{(2)1}), \\ {t}_{(2)0}=G_{(2)}^{-1}(1-{\alpha }_{(2)0}), \end{gathered}$$

where $F_{(1)}$, $G_{(1)}$ are the cumulative distribution functions (CDFs) of $R({\mathbf{X}}_1)$ given $D$ = 1, 0, respectively; $F_{(2)}$, $G_{(2)}$ are the CDFs of $R(\mathbf{X})$ given ${\mathbf{X}}_1\in N_{(1)}$ and $D$ = 1, 0, respectively. Hereinafter, we assume that $R({\mathbf{X}}_1)$ and $R(\mathbf{X})$ are continuous, so the CDFs are strictly increasing and the cutoffs are unique. We also denote $f_{(1)}$, $g_{(1)}$, $f_{(2)}$ and $g_{(2)}$ as the corresponding density functions. When the CDFs are unknown, we describe the corresponding finite sample algorithm for error-controlling in Section 4 after discussing the operating characteristics and properties in the next section.

3. Properties of the TEC

3.1. Optimality

The operating characteristics of the TEC have two parameters: (1) the NZR after the first step, $NZ{R}_{(1)}≔P({\mathbf{X}}_1\in N_{(1)})$; (2) the NZR after the second step, $NZ{R}_{(2)}≔P(\mathbf{X}\in N_{(2)})$. Both $NZ{R}_{(1)}$ and $NZ{R}_{(2)}$ are the minimization goals of the TEC. $NZ{R}_{(1)}$ is also the proportion of individuals in need of measuring ${\mathbf{X}}_2$, and we would like to minimize this proportion to reduce cost and test burden; $NZ{R}_{(2)}$ represents the overall performance after the two-step, as in the regular error-controlling classifiers.

The following theorem can be viewed as an extension of Lemma 1 and is proved in Appendix A.

Theorem 1. Given ${\alpha }_0$, ${\alpha }_1$ and $s$, the TEC constructed based on the true risk models $\eta ({\mathbf{X}}_1)=P(D=1\mid {\mathbf{X}}_1)$ and $\eta (\mathbf{X})=P(D=1\mid \mathbf{X})$ minimizes $NZ{R}_{(1)}$ and $NZ{R}_{(2)}$.

Theorem 1 has a practical implication: one should not discard the information of ${\mathbf{X}}_1$ in the second step. That is, we should use the augmented risk score $R(\mathbf{X})$ instead of $R({\mathbf{X}}_2)$ in the second step. On the contrary, in some literature on sequential testing (e.g., Ahmed et al., 2011), the subsequent test ignores the existing baseline information, which is likely sub-optimal although it avoids additional modeling efforts of constructing $R(\mathbf{X})$. We illustrate this implication of Theorem 1 using the data example in Section 6.

3.2. Trade-off between $NZ{R}_{(1)}$ and $NZ{R}_{(2)}$

For the goal of minimizing both $NZ{R}_{(1)}$ and $NZ{R}_{(2)}$, there is a trade-off that is driven by $s$. We demonstrate this $s$-driven trade-off using a numerical example. In this numerical example, the data are generated as follows $\mathbf{X}=(X_1,X_2)\sim N(0,\Sigma )$, ${\Sigma }_{11}={\Sigma }_{22}=1$, ${\Sigma }_{12}={\Sigma }_{21}=0.5$, $logit\{P(D)\}=-1+X_1+X_2$; we aim to construct a TEC with $R(\mathbf{X})=X_1$ and $R(\mathbf{X})=X_1+X_2$ at ${\alpha }_1$ = 0.1 and ${\alpha }_0$ = 0.2. The smaller value of ${\alpha }_1$ corresponds to less tolerance for making false negative errors. The trade-off in TEC is visualized in Figure 2. In Figure 2(a), we plot $NZ{R}_{(1)}(s)$ and $NZ{R}_{(2)}(s)$ as functions of $s$ to highlight the s-driven nature of the trade-off.

Figure 2:

An example to demonstrate the trade-off

The two horizontal lines in Figure 2(a) represent the NZR of the two single-step error-controlling classifiers based on $R({\mathbf{X}}_1)$ and $R(\mathbf{X})$, denoted as $NZ{R}_{(1)}$ and $NZ{R}_{(2)}$ respectively. On one extreme, when no error budget is spent in the first step (i.e., $s$ = 0) and ${\alpha }_{(2)k}={\alpha }_k$, the TEC reduces to the single-step classifier based on $R(\mathbf{X})$ with $NZ{R}_{(1)}(0)$ = 1 and $NZ{R}_{(2)}(0)=NZ{R}_2$. On the other extreme, when all the error budget is spent in the first step (i.e., $s$ = 1) and ${\alpha }_{(1)k}={\alpha }_k$, the TEC reduces to the single-step classifier based on $R({\mathbf{X}}_1)$ with $NZ{R}_{(1)}(1)=NZ{R}_2(1)=NZ{R}_1$. The behavior of $NZ{R}_{(1)}$ as a function of $s$ can be summarized in the following lemma.

Lemma 3. (Part A) $NZ{R}_{(1)}(s)$ is a decreasing function of $s$ with $NZ{R}_{(1)}(0)=1$ and $NZ{R}_{(1)}(1)=NZ{R}_1$; (Part B) if the density ratio, $\frac{f_{(1)}(t)}{g_{(1)}(t)}$, is increasing, $NZ{R}_{(1)}(s)$ decreases at decreasing rates.

In Appendix B, we prove Lemma 3 for the behavior of $NZ{R}_{(1)}(s)$ by differentiating $NZ{R}_{(1)}(s)$ with respect to $s$ once and twice, which will result in expressions involving density functions of $R({\mathbf{X}}_1)$ among $D$ = 1 or $D$ = 0 (i.e., $f_{(1)}$ or $g_{(1)}$) respectively). The increasing density ratio, equivalently concavity of the ROC curve, is relatively mild, assuming the higher values of $R({\mathbf{X}}_1)$ are indicative of disease.

To study the behavior of $NZ{R}_{(2)}(s)$, we note that $NZ{R}_{(2)}(0)=NZ{R}_2\le NZ{R}_1=NZ{R}_{(2)}(1)$. From Figure 2(a), we conjecture that $NZ{R}_{(2)}(s)$ is increasing in $s$ with increasing rate under some conditions. However, the required conditions are not straightforward and may not be interpretable. We could formally analyze the behavior of $NZ{R}_{(2)}(s)$ by differentiation. The first derivative of $NZ{R}_{(2)}(s)$ has many terms, which involve the joint densities of $R({\mathbf{X}}_1)$ and $R(\mathbf{X})$ among $D$ = 1 or 0 in addition to the marginal densities. The second derivative is only more complicated.

Through simulations in Appendix C, the minimal conditions should be weaker than the requirement of using true risk models for constructing the TEC. The key condition seems to lay in the nested structure of $R({\mathbf{X}}_1)$ and $R(\mathbf{X})$, and the latter utilizes the additional information from ${\mathbf{X}}_2$ on top of ${\mathbf{X}}_1$. Such nested structure is also recommended by the implication of Theorem 1 thus the increasing behavior is typical. In practice, we may empirically examine the behavior by plotting the estimated $NZ{R}_{(2)}(s)$ against $s$.

The behaviors of $NZ{R}_{(1)}(s)$ and $NZ{R}_{(2)}(s)$ may suggest a potential cost-effective trade-off for some small $s$. As seen in Figure 2(a), at $s$ = 0.4, the $NZ{R}_{(2)}(0.4)$ has minimally increased from $NZ{R}_2$ but $NZ{R}_{(1)}(0.4)$ decreases from 1 to about 0.6, which suggests that we save about 40% cost of measuring ${\mathbf{X}}_2$ but minimally impact the performance of the TEC as measured in $NZ{R}_{(2)}$. The real data example in Section 6 also demonstrates the cost-effective opportunity in the trade-off.

To emphasize the trade-off of the two minimization goals, we can visualize the TEC by plotting $NZ{R}_{(1)}$ against $NZ{R}_{(2)}$ as in Figure 2(b). The bottom right segments of the curve suggest that if we have a budget of measuring ${\mathbf{X}}_2$ (i.e., $NZ{R}_{(1)}$), the TEC can minimally sacrifice its performance (i.e., $NZ{R}_{(2)}$) and save some costs of collecting ${\mathbf{X}}_2$ (i.e., $1-NZ{R}_{(1)}$). For interested readers, we note a corresponding optimality result as the parameter s becomes implicit in Figure 2(b):

Corollary 1.

Given ${\alpha }_0$, ${\alpha }_1$ and $NZ{R}_{(1)}$, the TEC constructed based on $\eta ({\mathbf{X}}_1)=P(D=1\mid {\mathbf{X}}_1)$ and $\eta (\mathbf{X})=P(D=1\mid \mathbf{X})$ minimizes $NZ{R}_{(2)}$.

In Appendix D, we provide a proof of Corollary 1 as a consequence of Theorem 1 and Lemma 3.A. Due to Corollary 1, the curve of TEC based on the true risk models in Figure 2(b) becomes the Pareto curve in the view of multi-objective optimization (Boyd and Vandenberghe, 2004, Chapter 4). The optimality of the Pareto curve indicates that a TEC based on $\mathbf{X}$ to achieve small $\{NZ{R}_{(1)}(s),NZ{R}_{(2)}(s)\}$ on the bottom-left side of the curve is unachievable and any TEC based on $\mathbf{X}$ with $\{NZ{R}_{(1)}(s),NZ{R}_{(2)}(s)\}$ on the top-right side of the curve is suboptimal.

In practice, a TEC is derived based on working models for $R({\mathbf{X}}_1)$ and $R(\mathbf{X})$. The optimality results imply that model misspecification may lead to a loss of performance of the resulting classifier. Consequently, a key step is to appropriately incorporate ${\mathbf{X}}_2$ when constructing a well-calibrated estimate of $R(\mathbf{X})$. Standard model diagnostic procedures may be employed to improve the aptness of a parametric regression model, or alternatively, flexible nonlinear approaches, such as random forest or neural network, may be used to enhance the fit of a working model.

The cost-effective trade-off is further governed by the incremental value of ${\mathbf{X}}_2$ beyond $R({\mathbf{X}}_1)$ in the effect of $R(\mathbf{X})$. Within the TEC framework, this incremental value is quantified by the difference in NZRs between the two single-step classifiers, $NZ{R}_1-NZ{R}_2$. In Appendix E, we present a numerical example in which ${\mathbf{X}}_2$ is associated with $Y$ only weakly, yielding limited improvement in decisiveness. We therefore recommend computing both $NZ{R}_1$ and $NZ{R}_2$ first to assess the potential maximal gain from measuring ${\mathbf{X}}_2$ before deriving the TEC.

4. Estimation and Inference

To avoid overfitting of the classifiers, we assume $R({\mathbf{X}}_1)$ and $R(\mathbf{X})$ are estimated from a separate data set as in the motivating example. Thus, we focus on identifying the four cutoffs in the TEC and the inference of $NZ{R}_{(1)}$ and $NZ{R}_{(2)}$ when the dataset consists of $n$ independent and identically distributed (i.i.d.) random vectors $𝒟={\{R({\mathbf{X}}_{1i},R({\mathbf{X}}_{\mathbf{i}},D_i)\}}_{i=1,\dots ,n}$. In the context of error-controlling classifiers, it is known as sample-splitting-based robust error control in a general framework of conformal prediction (Lei, 2014; Lei et al., 2018; Angelopoulos and Bates, 2023).

4.1. Estimation

Given the risk scores, the four cutoffs (i.e., $t_{(j)k}$ for $j$ = 1, 2 and $k\in \{0,1\}$) are the population quantiles dependent on ${\alpha }_{(j)k}$ and apparent estimators for these cutoffs are the sample quantiles, based on the empirical CDF or some other variations of sample quantiles (Hyndman and Fan, 1996).

In the context of the TEC where the error control is a focus, following Lei (2014), we add extra care and consider the following estimation procedure (Algorithm 1) to identify the four cutoffs, ${\widehat{t}}_{(j)kn}$, for the TEC from a dataset with a finite sample size of $n$. Algorithm 1 ensures that as a target user for the TEC, a new patient with the risk scores $R({\mathbf{X}}_1^0)$ and $R({\mathbf{X}}^0)$ receives a classification $h({\mathbf{X}}^0)$ and the errors of $h({\mathbf{X}}^0)$ are controlled on average. This is the idea of coverage guarantee in the conformal prediction and we refer to Angelopoulos and Bates (2023) for an introduction to this line of research.

Formally, the property of the TEC based on Algorithm 1 is characterized by the following theorem:

Algorithm 1 Estimating t(j)k

$$\begin{array}{cc}\hfill & 1:\mathbf{for}k=0,1\mathbf{do}\hfill \\ \hfill & 2:\text{Total budget of erroe:}{B}_{kn}=\lfloor {\sum }_{i=1}^n{I}_{[D_i=k]}\times {\alpha }_k\rfloor \hfill \\ \hfill & 3:\text{The budget for the}{1}^{th}\text{step:}{B}_{(1)kn}=\lfloor B_{kn}\times s\rfloor \hfill \\ \hfill & 4:\text{The budget for the}{2}^{nd}\text{step:}{B}_{(2)kn}=B_{kn}-B_{(1)kn}\hfill \\ \hfill & 5:\text{Let}\{R_{(1)k}\}\text{be the ranked sample of}R({\mathbf{X}}_{1i})\text{among}{D}_i=k.\hfill \\ \hfill & 6:\mathbf{end}\mathbf{for}\hfill \\ \hfill & 7:{\widehat{t}}_{(1)1n}\text{is the}{B}_{(1)1n}^{th}\text{smallest of}\{R_{(1)1}\};{\widehat{t}}_{(1)0n}\text{is the}{B}_{(1)0n}^{th}\text{largest of}\{R_{(1)0}\}.\hfill \\ \hfill & 8:\mathbf{for}k=0,1\mathbf{do}\hfill \\ \hfill & 9:\text{Let}\{R_{(2)k}\}\text{be the ranked sample of}R({\mathbf{X}}_{\mathbf{i}})\text{among}{D}_i=k\text{and}R({\mathbf{X}}_{1i})\in ({\widehat{t}}_{(1)1n},{\widehat{t}}_{(0)1n})\hfill \\ \hfill & 10:\mathbf{end}\mathbf{for}\hfill \\ \hfill & 11:{\widehat{t}}_{(2)1n}\text{is the}{B}_{(2)1n}^{th}\text{smallest of}\{R_{(2)1}\};{\widehat{t}}_{(2)0n}\text{is the}{B}_{(2)0n}^{th}\text{largest of}\{R_{(2)0}\}.\hfill \end{array}$$

Theorem 2. Suppose ${\{R({\mathbf{X}}_{1i},R({\mathbf{X}}_{\mathbf{i}},D_i)\}}_{i=1,\dots ,n}$ and a new data point $(R({\mathbf{X}}_1^0),R({\mathbf{X}}^0),D^0)$ are $(n+1)$ i.i.d. data points. Construct the TEC from Algorithm 1. $h({\mathbf{X}}^0)$ is the classification from the TEC based on $\{R({\mathbf{X}}_1^0),R({\mathbf{X}}^0)\}$. Then, for $k\in \{0,1\}$,

$${\alpha }_k-\frac{1}{n_k}\le P\{h({\mathbf{X}}^0)\ne k\mid D^0=k\}\le {\alpha }_k,$$

where $n_k={\sum }_{i=1}^n{I}_{[D_i=k]}$.

See Appendix F for a proof of Theorem 2. In words, due to the floor function $\lfloor \cdot \rfloor$ applied to the budgets and asymmetry between $k$ = 1 and $k$ = 0 in selecting cutoffs from ranked samples, the cutoffs identified from Algorithm 1 tend to be conservative with some small sample sizes which guarantee the errors are bounded below the prespecified levels on average. In addition to this desired error-controlling property, Algorithm 1 conceptually highlights the idea of error budget spending by splitting the total budget of $B_{kn}$ into $B_{(1)kn}$ and $B_{(2)kn}$.

Alternatively, the apparent approach of estimating the cutoffs is based on sample quantiles, which has two disadvantages. First, the cutoffs based on empirical CDF (or other variants such as the ones implemented in $\text{quantile}()$ in the R package) as the sample quantiles for $F_{(1)}^{-1}({\alpha }_{(1)1}),G_{(1)}^{-1}(1-{\alpha }_{(1)0})$, $F_{(2)}^{-1}({\alpha }_{(2)1})$, $G_{(2)}^{-1}(1-{\alpha }_{(2)0})$ may result in FNR and FPR not being controlled on average. Note that for the first step, ${\widehat{F}}_{(1)}^{-1}({\alpha }_{(1)1})$ controls FNR at ${\alpha }_{(1)1}$ but ${\widehat{G}}_{(1)}^{-1}(1-{\alpha }_{(1)0})$ cannot control FPR at $1-{\alpha }_{(1)0}$ where $\widehat{F}$ and $\widehat{G}$ are the empirical CDF of $R_{(1)1}$ and $R_{(1)0}$ respectively. For the second step, because $F_{(2)}$ and $G_{(2)}$ are the CDFs conditional on ${\mathbf{X}}_1\in N_{(1)}$ (thus ${\alpha }_{(2)k}$ depends on $NZ{R}_{(1)}$), the sample-quantile-based cutoffs of the second step depend on the estimated neutral zone from the first step ${\widehat{N}}_{(1)}$ and its rate (i.e., ${\widehat{NZR}}_{(1)}$). In contrast, Algorithm 1 circumvents estimating $NZ{R}_{(1)}$ by directly dealing with the budget and bounds the errors without being overly conservative. In Section 5, we demonstrate the differences between the apparent approach and Algorithm 1 in a simulation study.

4.2. Inference

With ${\widehat{t}}_{(j)kn}$ obtained from the above algorithm, the natural estimator for $(NZ{R}_{(1)},NZ{R}_{(2)})$ is the empirical proportions:

$$\begin{gathered} {\widehat{NZR}}_{(1)}=\frac{1}{n}\sum _{i=1}^{n}I[{\widehat{t}}_{(1)1n}<R({\mathbf{X}}_{1i})<{\widehat{t}}_{(1)0n}], \\ {\widehat{NZR}}_{(2)}=\frac{1}{n}\sum _{i=1}^{n}I[{\widehat{t}}_{(1)1n}<R({\mathbf{X}}_{1i})<{\widehat{t}}_{(1)0n},{\widehat{t}}_{(2)1n}<R({\mathbf{X}}_{\mathbf{i}})<{\widehat{t}}_{(2)0n}]. \end{gathered}$$

The asymptotic normality of ${\widehat{NZR}}_{(1)}$ and ${\widehat{NZR}}_{(2)}$ can be established via the standard estimating equation methodology. Due to the estimation of $t_{(j)k}$, the asymptotic variances of $NZ{R}_{(1)}$ and $NZ{R}_{(2)}$ include terms involving density functions of $R({\mathbf{X}}_1)$ and $R(\mathbf{X})$. These density functions are typically unknown in practice, and density estimations are often challenging, especially in the tails of distributions. This problem is analogous to the asymptotic variance of TPR at an estimated cutoff corresponding to a fixed FPR in a standard ROC analysis. Like the standard ROC analysis, we deploy a resampling method to approximate the variances (Zhou and Qin, 2005).

If $s$ is selected graphically to balance the trade-off, we suggest only considering $s_l$ in a coarse grid of size $L$; for example, consider $s_l\in \mathbf{S}=(0.1,0.2,\dots ,0.9)$ with $L$ = 9. The reasons for this suggestion are twofold. First, in each finite sample, ${\widehat{NZR}}_{(1)}$ and ${\widehat{NZR}}_{(2)}$ are not monotone in $s$ even if the average of these quantities from the repeats of many finite samples is monotone (see Figure 6 and Figure 7 in Appendix C). Considering a continuous $s$ or having a fine grid shows these jumps increase the chance of seeking a trade-off between $NZ{R}_{(1)}$ and $NZ{R}_{(2)}$ only specific to a particular dataset. Second, due to the dependence on resampling for the pointwise confidence interval, the typical numerical approximation of a simultaneous confidence band based on a fine grid can be computationally intensive.

With the restriction on a small number of discrete points $\mathbf{S}$, we consider a resampling-based method, described as Algorithm 2 in Appendix G, to approximate confidence intervals or bands of ${\widehat{NZR}}_{(1)}$ and ${\widehat{NZR}}_{(2)}$. Montiel Olea and Plagborg-Møller (2019) provided a justification for the confidence band approximation. Note that to account for the variability from estimating the cutoffs, Algorithm 1 is applied to each resampled data set. In Section 5, we use simulation to demonstrate the finite sample coverage.

5. Simulation

We conduct a simulation study to demonstrate the algorithms to construct and evaluate the TEC. The binary disease status $D_i$ is generated with probability $p_i=expit(-1+{\mathbf{X}}_i\beta )$ where $\beta =(1,1)$, ${\mathbf{X}}_i\sim N(0,\Sigma )$, ${\Sigma }_{11}={\Sigma }_{22}=1$ and ${\Sigma }_{12}={\Sigma }_{21}=0.5$. We consider the risk scores $R({\mathbf{X}}_1)=X_1$, $R(\mathbf{X})=X_1+X_2$, and the two error constraints ${\alpha }_1$, ${\alpha }_0$. We generate 5000 random data sets for a sample size ($n$) from 400 to 8000.

To illustrate Algorithm 1 for controlling FPR at 0.1 and FNR at 0.2, we additionally generate large data sets for a sample size of 100, 000 for error rate evaluation given the identified cutoffs. The average of FNR and FPR over 5000 data sets at $s$ = 0.2 are summarized in Figure 3. As expected from Theorem 2, FNR and FPR are conservatively controlled below the desired levels by Algorithm 1 in finite sample sizes, and as sample sizes increase, FNR and FPR approach ${\alpha }_1$ and ${\alpha }_0$, respectively. For comparison, we use the sample quantile estimates ${\widehat{F}}_{(1)}^{-1}({\alpha }_{(1)1})$, ${\widehat{G}}_{(1)}^{-1}(1-{\alpha }_{(1)0})$, ${\widehat{F}}_{(2)}^{-1}({\alpha }_{(2)1})$, ${\widehat{G}}_{(2)}^{-1}(1-{\alpha }_{(2)0})$ for the cutoffs. We consider the default quantile estimator in R and the inverse of the empirical CDF. As these quantile estimators are not designed for the TEC, the error rates are not controlled at finite samples, although the excess errors become negligible in a large sample. Algorithm 1 circumvents estimating $N_{(1)}$ and $NZ{R}_{(1)}$ for identifying the cutoffs, and the resulting FNR and FPR are closer to ${\alpha }_1$ and ${\alpha }_0$ at different sample sizes, compared with the sample-quantile-based cutoffs.

Figure 3:

Algorithm 1 controls FNR or FPR at ${\alpha }_1$ and ${\alpha }_0$ in finite samples, respectively

We investigate the validity of the resampling-based procedure described as Algorithm 2. For each sample size, the true values of $NZ{R}_{(k)}(s)$ are approximated as the average of ${\widehat{NZR}}_{(k)}(s)$ over 5000 simulation runs. The coverage probabilities are the proportions of the intervals in each run containing the true values. We use 1000 bootstrap samples.

Figure 4 summarizes the results of the coverage of the resampling-based 95% confidence interval of ${\widehat{NZR}}_{(k)}(s)$ at each $s\in (0.1,\dots ,0.9)$ and the joint 95% confidence bands across $s$. The coverage probabilities of $NZ{R}_{(k)}(s)$ are close to the nominal 95% except for $NZ{R}_{(1)}(0.1)$ at $n$ = 400. With the small amount of error budget spent in the first step, the cutoffs are on the extreme tails of $R({\mathbf{X}}_1)$ and $NZ{R}_{(1)}(0.1)$ is closer to the boundary $NZ{R}_{(1)}(0)=1$ = 1. The coverage is closer to the nominal for larger sample sizes (n=600, 1000 or above). Overall, the confidence bands are conservative for the sample size commonly encountered in practice. Only for a very large sample size (n=8000), the coverage probabilities of confidence bands are about nominal.

Figure 4:

Resampling-based confidence interval/band for $NZ{R}_{(1)}(s)$ and $NZ{R}_{(2)}(s)$

6. Data Example

We applied the proposed TEC procedure to the PCA3 Evaluation Trial from the National Cancer Institute Early Detection Research Network (NCI-EDRN) (Wei et al., 2014). The study consists of 743 participants entering the study with signs of an elevated risk of prostate cancer. All participants subsequently received a diagnostic biopsy. The binary outcome $D$ is high-grade cancer in the biopsy, i.e., $D$ = 1 if the Gleason score is greater than 6. Of 743 participants, 151 were diagnosed with high-grade prostate cancer. In this setting, due to the invasive nature of the prostate cancer diagnostic biopsy, novel biomarker tests are needed to decide whether a diagnostic biopsy should be recommended. Following Tosoian et al. (2023), we consider ${\alpha }_1$ = 0.1 for ruling out a biopsy so that no more than 10% of high-grade prostate cancer is missed; we set ${\alpha }_0$ = 0.2 for ruling in a biopsy so that no more than 20% of unnecessary biopsies are recommended. Given ${\alpha }_1$ = 0.1 and ${\alpha }_0$ = 0.2, we seek a classifier that makes as many correct recommendations as possible, leaving the fewest individuals without a recommendation. The competing classifiers are based on three risk scores described in Tomlins et al. (2016): $R({\mathbf{X}}_1)$ is the Prostate Cancer Prevention Trial risk score (Thompson et al., 2006), with ${\mathbf{X}}_1$ consisting of routinely collected covariates including serum prostate-specific antigen (PSA), age, digital rectal examination status, African American race, and history of previous negative biopsy. $R({\mathbf{X}}_2)$ is the MPS score, where ${\mathbf{X}}_2$ consists of serum PSA and two gene expression signatures in a participant’s urine sample, namely TMPRSS2-ERG and PCA3. $\mathbf{X}$ is a set of covariates that are the union of variables in ${\mathbf{X}}_1$ and ${\mathbf{X}}_2$, and $R(\mathbf{X})$ is obtained by fitting a logistic regression model (Tomlins et al., 2016).

We want to utilize $R({\mathbf{X}}_1)$ in the first step to triage patients for biopsy decisions, and in the second step, using more expensive biomarkers ${\mathbf{X}}_2$ on those patients for whom a decision could not be reached in the first step. To illustrate the practical implication of Theorem 1, we also consider $R({\mathbf{X}}_2)$ in the second step. Since $R({\mathbf{X}}_2)$ uses only a subset of $\mathbf{X}$, we expect the TEC with $R(\mathbf{X})$ in the second step to be superior to that with $R({\mathbf{X}}_2)$ in the second step.

The trade-off between $NZ{R}_{(1)}$ and $NZ{R}_{(2)}$ is visualized in Figure 5. As expected from Theorem 1, the $NZ{R}_{(2)}(s)$ from the TEC with $R(\mathbf{X})$ in the second step is smaller than the $NZ{R}_{(2)}(s)$ from the TEC with $R({\mathbf{X}}_2)$ in the second step. At $s$ = 0.4, the trade-off between $NZ{R}_{(1)}(s)$ and $NZ{R}_{(2)}(s)$ seems attractive: a minimal increase in $NZ{R}_{(2)}(s)$ exchanges for a meaningful reduction in $NZ{R}_{(1)}(s)$. We further summarize the related metrics with their 95% CI for the two TECs at $s$ = 0.4 in Table 1. For comparison, we also include the single-step error-controlling classifiers using $R({\mathbf{X}}_1)$, $R(\mathbf{X})$, and $R({\mathbf{X}}_2)$. The cutoffs for all the classifiers are listed in Appendix H.

Figure 5:

The trade-off between $NZ{R}_{(1)}$ and $NZ{R}_{(2)}$ in the Example. At $s$ = 0.4, the TEC with $R(\mathbf{X})$ in the second step results in a $NZ{R}_{(1)}$ of 0.74, a $NZ{R}_{(2)}$ of 0.39, represented by the solid triangle and the solid circle, respectively

Table 1:

Summaries of different error-controlling classifiers at $s$ = 0.4

		$FNR$	$FPR$	$NZ{R}_{(1)}$	$NZ{R}_{(2)}$	$CCR$
single-step	$R({\mathbf{X}}_1)$	0.1	0.2	NA	0.49(0.42,0.56)	0.33(0.22,0.43)
	$R(\mathbf{X})$	0.1	0.2	NA	0.38(0.31,0.46)	0.44(0.34,0.54)
	$R({\mathbf{X}}_2)$	0.1	0.2	NA	0.44(0.37,0.51)	0.38(0.29,0.47)
two-step ($s$ = 0.4)	$R({\mathbf{X}}_1)$ then $R(\mathbf{X})$	0.1	0.2	0.74(0.69,0.78)	0.39(0.33,0.46)	0.43(0.35,0.5)
	$R({\mathbf{X}}_1)$ then $R({\mathbf{X}}_2)$	0.1	0.2	0.74(0.69,0.78)	0.42(0.36,0.49)	0.40(0.32,0.47)

^*For the single-step classifiers, $NZ{R}_{(1)}$ is not applicable (NA); the overall $NZR$ is displayed in the $NZ{R}_{(2)}$ column

The single-step classifier based on $R(\mathbf{X})$ has the lowest overall $NZR$ of 0.38 and the highest CCR of 0.44 among all the competing classifiers, as expected. That is, with $FNR$ and $FPR$ controlled at 0.1 and 0.2, respectively, the single-step classifier based on $R(\mathbf{X})$ can make correct biopsy recommendations for 44% of individuals while leaving 38% unclassified. At $s$ = 0.4, the TEC with $R(\mathbf{X})$ in the second step, $R({\mathbf{X}}_1)$ then $R(\mathbf{X})$, results in a $NZ{R}_{(1)}$ of 0.74, a $NZ{R}_{(2)}$ of 0.39, and a $CCR$ of 0.43. This suggests that, in comparison with the existing single-step classifier, using the proposed TEC may save 26% of the cost of measuring the expensive biomarkers while maintaining comparable performance in $NZR$ and $CCR$ at the desired level for $FNR$ and $FPR$.

7. Discussion

We consider constructing a classifier for medical decision-making under an error-controlling framework. Unlike the common binary classification problem, our method features a neutral zone for uncertain or borderline cases. Leaving patients in the neutral zone without a definite recommendation from a classifier may seem unsatisfactory. However, it is a proactive way to keep humans in the loop for decision-making and encourage further information collection, particularly for these ambiguous cases. By allowing the classifier to abstain from making recommendation when the confidence level is insufficient, we may improve overall decision-making accuracy. The idea has been naturally deployed in clinical research but has not been formally adapted for multi-step classification in the statistical literature. We outline a procedure for more rigorous practical implementation and show that the cost-effective trade-off between $NZ{R}_{(1)}$ and $NZ{R}_{(2)}$ makes this two-step extension particularly attractive.

The diagnostic literature has explored an alternative framework involving a neutral zone. Pepe et al. (2008) considered constructing a classifier with a neutral zone using absolute risks at low and high cutoffs. They hypothesized that a cutoff at 20% absolute risk for high-grade prostate cancer could guide biopsy recommendations, while a risk at 2% absolute risk would be low enough to forgo a biopsy. This neutral zone approach can be visualized through the predictiveness curve, which plots the estimated absolute risks against their percentiles. However, the approach relies on a well-calibrated risk prediction model, which may not always be available. Furthermore, determining the neutral zone cutoffs on the absolute risk scale requires a cost-benefit justification or assigned costs that are not straightforward in a single-step scenario (Amirou et al., 2014; Dubos et al., 2016), and it is additionally challenging to extend cost assignment to a multi-step scenario. An alternative family of classifiers based on these assigned costs is often called classification with rejection options (Yuan and Wegkamp, 2010). Such classifiers do not necessarily control the error rate at prespecified levels (Lei, 2014). Consequently, while classifiers with rejection options can be useful in certain contexts, they may not be suitable when precise error control is required.

Outside of the error-controlling framework, alternative strategies for sequential medical testing pursue different optimization objectives. For instance, Jeske et al. (2020) proposed a two-stage classifier designed to balance the overall error rate against the cost associated with expensive biomarkers. Another approach considers a weighted overall error rate or Generalized Youden Index as the primary optimization metric (Ahmed et al., 2011). A key distinction is that while these sequential strategies yield a definitive classification, their resulting FPR and FNR are not explicitly controlled. Nevertheless, the trade-off dynamics characterized in the TEC may be applicable to these alternative sequential testing approaches, an area that merits further research.

A. Proof of Theorem 1

A simple proof of Theorem 1 builds on Lemma 1. We leverage Lemma 1 in each of the two steps to establish the desired result. In the first step, given ${\alpha }_{(1)1}$ and ${\alpha }_{(1)0},\eta ({\mathbf{X}}_1)=P(D=1\mid {\mathbf{X}}_1)$ minimizes $NZ{R}_{(1)}$. In the second step, we may apply Lemma 1 again among the sub-population in $N_{(1)}$ to conclude that $P(D=1\mid {\mathbf{X}}_1\in N_{(1)})$ minimizes $P(t_{(2)1}\le \eta (\mathbf{X})<t_{(2)1}\mid {\mathbf{X}}_1\in N_{(1)})$ given ${\alpha }_{(2)1}$ and ${\alpha }_{(2)0}$. We reach the conclusion by noting that $\eta (\mathbf{X})=P(D=1\mid \mathbf{X},{\mathbf{X}}_1\in N_{(1)})$ and $NZ{R}_{(2)}=P(t_{(2)1}\le \eta (\mathbf{X})<t_{(2)1}\mid {\mathbf{X}}_1\in N(1))\times NZ{R}_{(1)}$.

B. Proof of Lemma 3

We want to verify the signs of the first and the second derivatives of $NZ{R}_1(s)$. That is, $NZ{R}_1^{\prime }(s)<0$ and $NZ{R}_1^{″}(s)>0$. First note that $t_{(1)0}(s)=G_{(1)}^{-1}(1-s{\alpha }_0)$ and $t_{(1)1}(s)=F_{(1)}^{-1}(1-s{\alpha }_1)$. Differentiate $t_{(1)0}(s)$ and $t_{(1)1}(s)$:

$$\begin{gathered} t_{(1)0}^{\prime }(s)=\frac{1}{g_{(1)}(t_{(1)0})}\times (-{\alpha }_0)<0, \\ {t}_{(1)1}^{\prime }(s)=\frac{1}{f_{(1)}(t_{(1)1})}\times ({\alpha }_1)>0. \end{gathered}$$

We note $NZ{R}_1^{\prime }(s)<0$ by plugging in $t_{(1)0}^{\prime }(s)$ and $t_{(1)1}^{\prime }(s)$:

$$\begin{gathered} {NZR}_{(1)}^{\prime }(s)=h_{(1)}(t_{(1)0})t_{(1)0}^{\prime }-h_{(1)}(t_{(1)1})t_{(1)1}^{\prime }+0 \\ =h_{(1)}(t_{(1)0})\frac{-{\alpha }_0}{g_{(1)}(t_{(1)0})}-h_{(1)}(t_{(1)1})\frac{{\alpha }_1}{f_{(1)}(t_{(1)1})} \\ =-\{{\alpha }_0{\pi }_1\frac{f_{(1)}}{g_{(1)}}(t_{(1)0})+{\alpha }_0{\pi }_0+{\alpha }_1{\pi }_1{[\frac{f_{(1)}}{g_{(1)}}(t_{(1)1})]}^{-1}+{\alpha }_1{\pi }_1\}<0 \end{gathered}$$

where ${\pi }_k=P(D=k)$ for $k\in \{0,1\};h_{(1)}(t_{(1)0})=f_{(1)}{(}_{t(1)0}){\pi }_1+g_{(1)}(t_{(1)0}){\pi }_0$ is the marginal density of $R({\mathbf{X}}_1)$; and $\frac{f_{(1)}}{g_{(1)}}$ is the density ratio. If we assume a increasing density ratio (i.e., ${[\frac{f_{(1)}}{g_{(1)}}]}^{\prime }(t)>0$), we can verify that $NZ{R}_{(1)}^{″}(s)>0$:

$${NZR}_{(1)}^{″}(s)=-\left[{\alpha }_0{\pi }_1{[\frac{f_{(1)}}{g_{(1)}}]}^{\prime }(t_{(1)0})\times t_{(1)0}^{\prime }-{\alpha }_1{\pi }_1{\{[\frac{f_{(1)}}{g_{(1)}}](t_{(1)1})\}}^{-2}\times {[\frac{f_{(1)}}{g_{(1)}}]}^{\prime }(t_{(1)1})\times t_{(1)1}^{\prime }\right]>0$$ (1)

C. Behavior of $NZ{R}_{(2)}(s)$

We provide a simulation to demonstrate the heuristic argument given in the main text.

Using simulation, we demonstrate $NZ{R}_{(2)}(s)$ tends to be an increasing function if $R(\mathbf{X})$ includes all the information of ${\mathbf{X}}_1$.

The binary disease status $D_i$ is generated with probability $p_i=expit(-1+{\mathbf{X}}_i\beta )$ where $\beta =(1,1)$, ${\mathbf{X}}_i\sim N(0,\Sigma )$, ${\Sigma }_{11}={\Sigma }_{22}=1$ and ${\Sigma }_{12}={\Sigma }_{21}=0.5$. Let the two error constraints ${\alpha }_1$ = 0.1, ${\alpha }_0$ = 0.2. We consider $R({\mathbf{X}}_1)=X_1$ in the first step, and the true model $\eta (\mathbf{X})=expit(-1+X_1+X_2)$ and two wrong models in the second step. Model $R(\mathbf{X})$ misspecifies the functional form of $X_2$ to be $exp(X_2)$ in a logistic regression model. Model $R({\mathbf{X}}_2)$ omits $X_1$. In each simulated replicate, we fit the logistic regression models with a sample size of 1000 and evaluate the $NZ{R}_{(2)}(s)$ with a new data set with the same sample size. On average, $R(\mathbf{X})=expit(-2.56+1.06X_1+1.32exp(X_2))$ and $R({\mathbf{X}}_1)=expit(-0.90+2.23X_2)$.

The average of $NZ{R}_{(2)}(s)$ over 1000 simulation replicates is summarized in Figure 6. Note that $NZ{R}_{(2)}(s)$ from both $\eta (\mathbf{X})$ and $R(\mathbf{X})$ are increasing functions of $s$ at increasing rates, suggesting that the true risk model is not a necessary condition. $NZ{R}_{(2)}(s)$ from $R({\mathbf{X}}_1)$ is not a increasing function. This observation hints that some notions of nested information structure may play a role. One may make a heuristic argument for this condition: the increasing s results in informative ${\mathbf{X}}_2$ from a larger proportion of patients being incorporated in the TEC, hence a better performance of the TEC and a smaller $NZ{R}_{(2)}(s)$.

Figure 6:

Behavior of $NZ{R}_{(2)}(s)$

We also present $NZ{R}_{(2)}(s)$ based on $R(\mathbf{X})$ from one replicate in Figure 7. Note that on average, $NZ{R}_{(2)}(s)$ estimated from a finite sample is an increasing function as seen in Figure 6 but one realization of empirical ${\widehat{NZR}}_{(2)}(s)$ may display some random variability which seems not increasing in some local points (e.g., between $s$ = 0.1 and $s$ = 0.3 in Figure 7). Therefore, to avoid confusion in practice, we suggest only plot ${\widehat{NZR}}_{(2)}(s)$ in a coarse grid such as ($0,0.1,\dots ,0.9,1$) instead of ($0,0.01,\dots ,0.99,1$).

Figure 7:

Random variability in behavior of $NZ{R}_{(2)}(s)$

D. Proof of Corollary 1

Lemma 3.A shows a one-to-one correspondence between $NZ{R}_{(1)}$ and $s$ for any $R({\mathbf{X}}_1)$ used in the first step. If we choose $\eta ({\mathbf{X}}_1)$ for the first step, there is a fixed s for a given $NZ{R}_{(1)}$. With the fixed $s$, Theorem 1 says that the TEC based on $\eta ({\mathbf{X}}_1)$ and $\eta (\mathbf{X})$ minimizes $NZ{R}_{(2)}$.

E. Additional numerical examples on the trade-off

To complement the analysis in the main text, we present additional numerical examples demonstrating the impact of the incremental predictive value of ${\mathbf{X}}_2$. The numerical example described in Section 3.2 represents a scenario where the novel biomarker provides a strong incremental value. For comparison, we generate a scenario characterized by weak incremental value, defined as follows: $\mathbf{X}=(X_1,X_2)\sim N(0,\Sigma )$ with ${\Sigma }_{11}={\Sigma }_{22}=1$ and ${\Sigma }_{12}={\Sigma }_{21}=0.5$. The true risk is given by $logit\{P(D)\}=-1+1.5X_1+0.3X_2$. We construct a TEC using the risk scores $R({\mathbf{X}}_1)=1.5X_1$ and $R(\mathbf{X})=1.5X_1+0.3X_2$, with error control parameters set to ${\alpha }_1$ = 0.1 and ${\alpha }_0$ = 0.2.

Table 2:

Summaries for novel biomarkers with strong and week incremental values.

Scenario	$NZ{R}_1$	$NZ{R}_2$	$NZ{R}_1-NZ{R}_2$	$NZ{R}_{(1)}(s=0.4)$	$NZ{R}_{(2)}(s=0.4)$
strong	0.326	0.226	0.100	0.615	0.234
weak	0.251	0.241	0.010	0.542	0.241

The comparative results are summarized in Table 2. For the novel biomarker with a strong incremental value, the full incremental value, namely, $NZ{R}_1-NZ{R}_2$, is 0.1. Reducing the proportion of unclassified individuals by this magnitude justifies the adoption of the two-step extension. Specifically, at $s$ = 0.4, the method achieves a NZR of 0.234 (down from 0.326). This improvement is clinically meaningful, while avoiding the cost of the novel biomarker $X_2$ for $(1-0.615)\times 100\%$ = 38.5% of the population.

Conversely, in the scenario of weak incremental value, measuring $X_2$ for 100% of the population yields a maximal NZR reduction of only 0.010 (from 0.251 to 0.241). This marginal gain suggests that the additional complexity and cost of the second step are unwarranted. In such cases, a simpler error-controlling classifier based solely on $R({\mathbf{X}}_1)$ is preferable, as it avoids the costs associated with $X_2$ entirely without significantly sacrificing classification performance.

This comparison highlights that the cost-effective trade-off offered by the TEC is built on the existence of a meaningful incremental value of $R(\mathbf{X})$ over $R({\mathbf{X}}_1)$.

F. Proof of Theorem 2

We present the proof for $k$ = 1 (i.e., controlling FNR). The proof for $k$ = 0 is similar. We expand the notations for the rank samples: $\{R_{(1)1}:r_{(1)1}[1]<r_{(1)1}[2]<,\dots ,<r_{(1)1}[n_1]\}$ and $\{R_{(2)1}:r_{(2)1}[1]<r_{(2)1}[2]<,\dots ,\}$. Because of $(R({\mathbf{X}}_{1i})$, ${R({\mathbf{X}}_{\mathbf{i}},D_i)}_{i=1,\dots ,n}$ and $(R({\mathbf{X}}_1^0),R({\mathbf{X}}^0),D^0)$ are i.i.d., we have

$$\begin{gathered} P(R({\mathbf{X}}_1^0\le r_{(1)1}[z]\mid D^0=1)=\frac{z}{n_1}, \\ P(R({\mathbf{X}}^0)<r_{(2)1}[z],R({\mathbf{X}}_1^0)\in ({\widehat{t}}_{(1)1n},{\widehat{t}}_{(1)0n})\mid D^0=1)=\frac{z}{n_1}, \end{gathered}$$

for any integer $z$ within the ranges. This is known as exchangeability in the conformal prediction literature.

Note that the event $\{h({\mathbf{X}}^0)=0\}$ consists of two mutually exclusive events: $\{R({\mathbf{X}}_1^0)\le {\widehat{t}}_{(1)1n}\}$ and $\{R({\mathbf{X}}^0)\le {\widehat{t}}_{(2)1n},R({\mathbf{X}}_1^0)\in ({\widehat{t}}_{(1)1n},{\widehat{t}}_{(1)0n})\}$, thus we have:

$$\begin{gathered} P(h({\mathbf{X}}^0)=0\mid D^0=1)=P(R({\mathbf{X}}_1^0)\le {\widehat{t}}_{(1)1n}\mid D^0=1)+P(R({\mathbf{X}}^0)\le {\widehat{t}}_{(2)1n},R({\mathbf{X}}_1^0)\in ({\widehat{t}}_{(1)1n},{\widehat{t}}_{(1)0n})\mid D^0=1) \\ =P(R({\mathbf{X}}_1^0)\le r_{(1)1}[B_{(1)1n}]\mid D^0=1)+P(R({\mathbf{X}}^0)\le r_{(2)1}[B_{(2)1n}],R({\mathbf{X}}_1^0)\in ({\widehat{t}}_{(1)1n},{\widehat{t}}_{(1)0n})\mid D^0=1) \\ =\frac{B_{(1)1n}}{n_1}+\frac{B_{(2)1n}}{n_1}(\text{due to exchangeability}) \\ =\frac{B_{1n}}{n_1} \\ =\frac{\lfloor n_1\times {\alpha }_1\rfloor }{n_1}\le {\alpha }_1 \end{gathered}$$

For the lower bound of $P(h({\mathbf{X}}_0)=0\mid D^0=1)$, we utilize the relationship between the floor function $\lfloor \cdot \rfloor$ and the ceiling function $\lceil \cdot \rceil :\lfloor r\rfloor =\lceil r\rceil -1$ for any positive number $r$. Thus, $\frac{\lfloor n_1\times {\alpha }_1\rfloor }{n_1}=\frac{\lceil n_1\times {\alpha }_1\rceil -1}{n_1}\ge {\alpha }_1-\frac{1}{n_1}$ is a lower bound.

G. Resampling-based inference

Algorithm 2 Resampling-based confidence interval/band for ${\widehat{NZR}}_{(k)}(s_l)$

$$\begin{array}{cc}\hfill & 1:\text{Create}B\text{bootstrapped data sets of the data set}𝒟\text{by resampling with replacement:}{𝒟}^{(1)},\dots ,{𝒟}^{(B)}\hfill \\ \hfill & 2:\mathbf{for}b=1,\dots ,B\mathbf{do}\hfill \\ \hfill & 3:\mathbf{for}l=1,\dots ,L\mathbf{do}\hfill \\ \hfill & 4:\text{Given}{s}_l,\text{identify the cutoffs by applying Algorithm 1 on}{𝒟}^{(b)}\hfill \\ \hfill & 5:\text{Compute}{\widehat{NZR}}_{(k)}^{(b)}(s_l)\text{with the cutoffs and}{𝒟}^{(b)}\hfill \\ \hfill & 6:\mathbf{end}\mathbf{for}\hfill \\ \hfill & 7:\mathbf{for}\mathbf{for}\hfill \\ \hfill & 8:\mathbf{for}l=1,\dots ,L\mathbf{do}\hfill \\ \hfill & \text{Compute the sample standard deviation}{\widehat{\sigma }}_{(k)}\text{of}{\widehat{NZR}}_{(k)}^{(1)}(s_l),\dots ,{\widehat{NZR}}_{(k)}^{(B)}(s_l)\hfill \\ \hfill & 9:\mathbf{end}\mathbf{for}\hfill \\ \hfill & 10:\mathrm{A}1-\alpha \text{pointwise confidence interval of}{\widehat{NZR}}_{(k)}(s_l)\text{is}\hfill \\ \hfill & \{{\widehat{NZR}}_{(k)}(s_l)-z_{1-\alpha ∕2}{\widehat{\sigma }}_{(k)}(s_l),{\widehat{NZR}}_{(k)}(s_l)+z_{\alpha ∕2}{\widehat{\sigma }}_{(k)}(s_l)\text{is}\hfill \\ \hfill & 11:\mathbf{for}b=1,\dots ,B\mathbf{do}\hfill \\ \hfill & m^{(b)}=ma{x}_{l=1,\dots ,L}\frac{\mid {\widehat{NZR}}_{(k)}^{(b)}(s_l)-{\widehat{NZR}}_{(k)}(s_l)\mid }{{\widehat{\sigma }}_{(k)}(s_l}\hfill \\ \hfill & 12:\mathbf{end}\mathbf{for}\hfill \\ \hfill & 13:\text{Let}c\text{be}1-\alpha \text{sample quantile of}{m}^{(1)},\dots ,m^{(B)}\hfill \\ \hfill & 14:\mathrm{A}1-\alpha \text{joint simulataneous confidence band of}{\widehat{NZR}}_{(k)}(s_l),\text{for}l=1,\dots ,L,\text{is}\hfill \\ \hfill & \bigcup _{l=1}^L\{{\widehat{NZR}}_{(k)}(s_l)-c{\widehat{\sigma }}_{(k)}(s_l),{\widehat{NZR}}_{(k)}(s_l)+c{\widehat{\sigma }}_{(k)}(s_l)\}\hfill \end{array}$$

H. Cutoffs in the data example

Table 3:

Cutoffs for the single-step classifiers and the two-step classifiers (at $s$ = 0.4)

		$t_1$	$t_0$	$t_{(1)1}$	$t_{(1)0}$	$t_{(2)1}$	$t_{(2)0}$
single-step	$R({\mathbf{X}}_1)$	0.140	0.214
	$R(\mathbf{X})$	0.099	0.259
	$R({\mathbf{X}}_2)$	0.100	0.276
two-step ($s$ = 0.4)	$R({\mathbf{X}}_1)$ then $R(\mathbf{X})$			0.132	0.319	0.085	0.274
	$R({\mathbf{X}}_1)$ then $R({\mathbf{X}}_2)$			0.132	0.319	0.090	0.311

The risk scores are described in Tomlins et al. (2016)