The Undertesting Bias in Clinical Decision Making: The Role of Test Dichotomization

Abstract

Threshold models, formalized by Pauker and Kassirer, guide clinical decisions about testing and treatment by partitioning pretest probabilities into three action zones using two threshold pretest probabilities: the test threshold and the test–treatment threshold. This derivation implicitly assumes that diagnostic tests possess fixed sensitivity and specificity, effectively treating them as binary predictors. However, many diagnostic tests yield continuous or ordinal outputs, in which case the optimal sensitivity–specificity pair varies with the pretest probability. I demonstrate that abstracting from this dependency leads to an undertesting bias: the testing threshold is inflated and/or the test–treatment threshold is deflated, resulting in a narrower than optimal testing window. This bias systematically undervalues continuous diagnostic tests and leads to their underuse. Clinical decision-making models and guidelines should therefore recognize that optimal test score cutoffs depend on pretest probabilities to avoid this systematic underuse of diagnostic testing.

Introduction

Threshold models are widely used to analyze clinical decisions about whether to treat, test, or withhold intervention under diagnostic uncertainty.^1–4 Pauker and Kassirer ¹ formalized this approach by explicitly deriving a “testing threshold” (denoted $T_t$ ) and a “test–treatment threshold” (denoted $T_{trx}>T_t$ ), which partition pretest probabilities (ie, the clinician’s assessment of the patient having the disease prior to conducting the diagnostic test) into 3 regions. When the pretest probability is below the testing threshold $T_t$ , it is optimal to withhold both testing and treatment, and when it is above the test–treatment threshold $T_{trx}$ , it is optimal to treat without testing. In the intermediate range where the pretest probability is between these two values, it is optimal to conduct a diagnostic test to decide whether to treat the patient. Figure 1 depicts these thresholds.

Figure 1.

Pretest probability action thresholds. The pretest disease probability, $p$ , is on the horizontal axis.

This model has been very influential, both in the theoretical and applied literatures on medical decision making, and it continues to be treated as one of the major advances in the field in recent discussions of decision threshold models.^4,5 Variants and extensions of it continue to be developed and applied, for example, in work that introduces treatment thresholds into diagnostic test evaluation or uses threshold analysis to structure decision tables,^6,7 as well as in recent extensions that incorporate therapeutic risk. ⁸

Despite this interest, one simplifying assumption in this approach has not yet received much attention. Specifically, Pauker and Kassirer assume that the diagnostic test possesses a fixed sensitivity and fixed specificity, which is equivalent to it being a binary predictor,^{9 –11} that is, one yielding a positive or negative result, as in the case of simple SARS-CoV-2 rapid antigen tests. ¹² However, numerous laboratory assays, imaging modalities, multivariable risk scores, and signal-to-cutoff–based assays yield continuous or ordinal outputs, such as biomarker concentrations; imaging severity scores; predicted risk percentages; or the cycle threshold value ( $Ct$ ) in COVID polymerase chain reaction tests, rather than dichotomous positive/negative results.^{13 –15} The binary predictor assumption is thus a form of dichotomization—a process explicitly cautioned against in this journal ¹⁶ —which can inadvertently lead to misleading conclusions.

Here, I demonstrate that when the diagnostic test is continuous, employing Pauker and Kassirer’s simplifying assumption systematically leads to an undertesting bias, that is, an inefficient tendency to underutilize diagnostic tests. More precisely, a clinician who adopts Pauker and Kassirer’s thresholds will use a narrower-than-optimal testing range. The rationale is that, compared with a clinician forced to use a test fixed at a particular sensitivity–specificity setting, a clinician who can set sensitivity and specificity anywhere on the test’s ROC curve can extract more value from the test: for patients with a high pretest probability of disease, they can increase sensitivity while decreasing specificity; for patients with a low pretest probability, they can do the reverse—making testing more worthwhile over a broader range of pretest probability values (and therefore for more patients). This result is formalized in an appendix in the supplementary material (henceforth ”Appendix”), because the rationale behind it is intuitive, as explained in further detail next.

When the test is continuous, its sensitivity and specificity can be adjusted. In fact, the efficient decision frontiers associated with continuous tests are described by what are often called receiver operating characteristic (ROC) curves,^{17 –19} which have been studied and used frequently in the medical literature.^{13 –15,20,21} This causes a third threshold, which is absent in Pauker and Kassirer’s model, to emerge: a test score, as opposed to a threshold probability, determining when the clinician will treat the patient upon administering the diagnostic test. ⁱ This threshold, which I call the “treat–withhold threshold” and denote $s_{tw}$ , is distinct from the test–treat threshold. The test–treat threshold is a pretest probability that determines whether a diagnostic test should be conducted or whether the patient should be treated without further testing. In contrast, the treat–withhold threshold is a test score that determines whether the patient should be treated in light of all information available to the clinician and is therefore often also called the “test score cutoff.” When a clinician uses the diagnostic test optimally, this threshold is chosen in a utility-maximizing manner, and its value depends on test characteristics as well as the pretest probability; the derivation of this threshold within a model where the clinician chooses between the three options listed in Figure 1 is described in the Appendix. I also developed a simple program that calculates the optimal values of all three thresholds as well as optimal sensitivity–specificity pairs as a function of pretest probability for equal-variance binormal tests. ²⁴ That program can be used to generate examples, in addition to those presented below, to illustrate the points made here.

Table 1 provides a summary of the 3 thresholds for clarity.

Table 1.

Description of the 3 Thresholds.

Threshold	Notation	Threshold Type	Decision Guided
Test	$T_t$	Pretest probability	Test versus withhold treatment without testing
Test–treat	$T_{trx}$	Pretest probability	Test versus treat without testing
Treat–withhold	$s_{tw}$	Test score	Treat versus withhold based on the test

The treat–withhold threshold has been studied in the medical literature, but in simpler models.^5,6,20,25 Although these models derive the optimal treat–withhold threshold as I do here, they focus on the clinician’s binary decision of whether to treat a patient in light of all available information (e.g., a test result), as opposed to the Pauker and Kassirer framework, in which the clinician’s antecedent decision of whether to conduct a test is also studied. ⁱⁱ However, because Pauker and Kassirer consider a binary predictor, the treat–withhold threshold is absent in their more complicated decision problem: there, the clinician receives a test result that is only either positive or negative and decides to treat the patient only when it is positive. ¹ Thus, models that study the treat–withhold threshold do not consider the clinician’s initial decision problem, and models that incorporate this initial decision problem do not study the treat–withhold threshold. However, the economic framework I recently proposed ²⁶ can be used to combine the initial testing decision and the optimal threshold adjustment as explained here.

Quite importantly, and as noted in the medical decision-making literature, the treat–withhold threshold is a function of the pretest probability.^{2,3,20,27 –29} This is because a high pretest probability combined with a moderate test score can yield the same posttest probability of disease as a low pretest probability combined with a high test score. Therefore, the optimal treat–withhold threshold must adjust to reflect the information already available to the clinician. Consistent with this understanding, both empirical observations and normative analyses indicate that clinicians routinely apply different decision thresholds to the same continuous test across clinical contexts.^2,4,5,29 Reviews and methodological papers explicitly argue that the optimal treat–withhold threshold depends on disease prevalence and other factors, leading to different recommended cutoffs in screening, diagnostic, and confirmatory settings.^6,7,15,20

Theoretical and empirical analyses thus suggest that the treat–withhold threshold is a function of the pretest probability. When one assumes that the diagnostic test has a fixed specificity and sensitivity, one removes this dependency. This, in turn, artificially deflates the value of a test, by assuming that it is incapable of being optimally adjusted in light of other information that the clinician has.

To illustrate, consider an equal variance binormal diagnostic test. The test scores, $s$ , are normally distributed both when the patient has the disease and when he does not, but with different means. If the mean of the disease-distribution is larger than the mean of the no-disease-distribution, it follows that larger test scores are more indicative of the disease. Thus, after a test is conducted, the clinician may choose a cutoff test score, $\hat{s}$ , and recommend treatment only when $s>\hat{s}$ . This test score can be adjusted optimally based on the clinician’s pretest probability estimate or it may be fixed. In the latter case, the diagnostic test has a fixed sensitivity and specificity, and in the former its specificity and sensitivity optimally adjust to the pretest probability.

Figure 2a and b compare the optimal sensitivity and specificity of a test ⁱⁱⁱ against the specificity–sensitivity pair $\left(0.95,0.90\right)$ it achieves through a fixed cutoff score ( $\widehat{s}\approx 1.645$ ). This fixed sensitivity–specificity pair is the one used in an example considered by Pauker and Kassirer. The dashed line indicates the pretest probability ( $p_{\hat{s}}\approx 0.072$ ) for which $\hat{s}$ is also the optimal cutoff score, that is, $\hat{s}=s_{tw}\left(p_{\hat{s}}\right)$ . In other words, the fixed test characteristics assumed in calculations coincide with the optimal test characteristics only when the pretest probability happens to equal $p_{\widehat{s}}$ ; for other pretest values, these test characteristics diverge. ^iv Figure 2a and b illustrate that this divergence grows as the pretest probability moves away from $p_{\widehat{s}}$ .

Figure 2.

(a, b) The optimal sensitivity ( $sen{s}^{\text{*}}\left(p\right)$ ) and specificity ( $spe{c}^{\text{*}}\left(p\right)$ ) as a function of the pretest probability $p$ are plotted against the fixed sensitivity and specificity assumed (0.9 and 0.95). The optimal values are obtained when the diagnostic test is the unique equal variance binormal test capable of generating the assumed fixed test characteristics in an example in Pauker and Kassirer. ¹ The parameters are as assumed in the same example: the benefit of correct treatment is 19 times the cost of administering the test, and the cost of treating a patient without a disease is 2.5 times the cost of conducting a test.

More generally, clinicians who reflect carefully on their pretest assessments can use diagnostic tests in different ways depending on patient characteristics. This is exemplified through the sensitivity–specificity variations that can be optimally adopted by clinicians in Figures 2a and b. Relying on rigid rules of thumb when interpreting diagnostic tests across diverse cases can therefore undermine the clinical process. In fact, assuming one must use a fixed test cutoff across all cases reduces the perceived value of the test. Therefore, the difference between the actual value of the test and the perceived value of the test under the assumption that it must be used with a fixed cutoff is increasing in the gap between the pretest probability and $p_{\widehat{s}}$ . This is illustrated in Figure 3.

Figure 3.

The difference (denoted $\mathrm{\Delta }\left(p\right)$ ) in the utility from using the test when its cutoff is chosen optimally versus when it is set to achieve sensitivity 0.90 and specificity 0.95. The optimal values are obtained using the same assumptions as in Figures 2a and b and are a function of the pretest probability $p$ on the horizontal axis.

This exercise illustrates how using a fixed sensitivity–specificity pair for calculation purposes leads to a low estimated value of using a diagnostic test and a narrower than optimal range between the test and test–treat thresholds. This naturally leads clinicians to use a test less often than is optimal, regardless of which fixed sensitivity–specificity pair is chosen for the calculation. This is illustrated in Figure 4a and b, which depict the optimal test and test–treat thresholds (the dashed curves) versus the Pauker–Kassirer thresholds. The latter are obtained by choosing a particular specificity–sensitivity pair associated with the diagnostic test, and the specificity is varied along the horizontal axis.

Figure 4.

(a, b) Optimal thresholds (dashed) are plotted against the Pauker–Kassirer thresholds (solid) obtained by assuming the diagnostic test specificity equals the level on the horizontal axis.

These observations demonstrate that using a fixed specificity–sensitivity pair as in Pauker and Kassirer ¹ will yield a larger test threshold (i.e., $T_t$ ) and/or a smaller test–treat threshold (i.e., $T_{trx}$ ) than is optimal, which implies a narrower than optimal testing window.

This undertesting bias is general and not limited to the particular examples provided here. It follows whenever the diagnostic test is continuous. Neither the specifics of the diagnostic test nor the precise choice of the fixed specificity–sensitivity pair affect this conclusion. The rationale, as explained here, is that assuming the continuous diagnostic test must be used with a fixed sensitivity–specificity pair amounts to undervaluing it and thus leads to its underuse. Because the formalization of these claims provides little further insight, it is relegated to the Appendix.

The bias identified here can be corrected by anticipating the dependency of the diagnostic test’s sensitivity and specificity on the clinician’s pretest assessments. In practice, this insight can be leveraged to develop decision support tools for personalized medicine that integrate individualized pretest probability estimates with optimally adjusted test thresholds.