Diagnostic accuracy in the presence of a rare outcome

Di Santo and colleagues describe improved diagnostic accuracy for eligibility for radial artery access using a smartphone application compared with the modified Allen test.¹ Specifically, the authors found superior overall diagnostic accuracy (91.8% v. 81.7%), driven by an increase in specificity (93.0% v. 82.8%).

Overall diagnostic accuracy (i.e., the proportion of individuals correctly classified) is advantageous because it is an easily calculable and interpretable measure of diagnostic performance. However, overall diagnostic accuracy is affected by disease prevalence. Holding sensitivity and specificity constant, a test’s overall diagnostic accuracy increases as disease prevalence decreases (assuming the test’s specificity exceeds its sensitivity).

In the study by Di Santo and colleagues, only 6 of 219 (0.3%) participants in the smartphone application group experienced the outcome. The limitations with overall diagnostic accuracy in this setting become apparent when you consider an alternative test where, instead of using the smartphone application, every individual is classified as “test negative.” Such a scenario would classify all 219 participants as test negative, resulting in 213 true negatives, 6 false negatives and no participants coded as “test positive.” This hypothetical test would have improved diagnostic performance compared with the smartphone application based on the two metrics described in the abstract by Di Santo and colleagues, with overall diagnostic accuracy of 97.3% and specificity of 100%.

Other measures of diagnostic accuracy exist that are not as susceptible to disease prevalence as overall diagnostic accuracy, such as diagnostic odds ratio (OR) and Youden’s index.² These measures describe the improved diagnostic performance associated with the smartphone application compared with the modified Allen test, without being as influenced by the low event rate (diagnostic OR 13.2 v. 1.6; Youden’s index 0.43 v. 0.08).