Reference test bias in diagnostic test evaluation – A problem for epidemiologists, too

Epidemiologists depend on accurate assessment of disease states for almost all aspects of their work, whether for research or practice. Most epidemiologists are aware that diagnostic tests are fallible. They have developed and applied sophisticated methods to address the measurement error of outcomes. Epidemiologists are comfortable with sensitivity and specificity, the parameters used to express the accuracy of diagnostic and screening tests. However, few “traditional epidemiologists” have contributed to diagnostic test evaluation methods, despite the centrality of diagnostic tests to their work. Instead, diagnostic test evaluation has been the purview of “clinical epidemiologists” and a few biostatisticians.

Diagnostic test evaluation is subject to numerous potential biases.¹ Fundamentally, evaluation of a new diagnostic test requires comparison to a reference (“gold”) standard, which is usually assumed to discriminate disease and non-disease states perfectly. Unfortunately, few, if any, reference standards are perfect. The resulting reference test bias is one of the most significant, pervasive, and challenging forms of bias.^2,3

The impact of reference test bias can be described with a simple example. Consider a reference test with sensitivity=0.85 and specificity=0.90 and an improved, new test with a true, but unknown sensitivity=0.90 and specificity=0.95. For simplicity, we assume conditional independence of these tests. In a study sample with a prevalence of 0.1, the measured sensitivity and specificity of the new test would be 0.46 and 0.93, respectively. In study sample with a prevalence of 0.5 (i.e. cases and non-cases selected independently), the estimates of sensitivity and specificity change markedly to 0.81 and 0.83, respectively. The dependence of the sensitivity and specificity estimates on prevalence is comparable to the variation of positive and negative predictive values with prevalence. False positives by the reference test increase as prevalence decreases; reference test false negatives increase as prevalence increases. The imperfect classification of disease by the reference standard leads to apparent misclassification by the new test and biased sensitivity and specificity estimates.

Reference test bias is particularly problematic with new tests that are better than the reference standard, as in the example above. The belief that a new test is inherently better has led to intuitive “solutions” to account for reference test bias. Unfortunately, as with other areas of epidemiology, intuition is often a poor statistician. In the 1990’s, microbiologists adopted an intuitively appealing procedure referred to as discrepant analysis for the evaluation of new nucleic acid amplification tests, such as polymerase chain reaction, for chlamydial infection and other infectious diseases.^4-6 The microbiologists recognized that these new tests represented major advances biologically over culture techniques, which were known to have limited sensitivity.⁷ To address the concern of reference test bias using culture as the reference standard, the microbiologists chose to conduct additional testing on specimens with discordant results (i.e. positive by new test, but negative by reference standard). This “resolution of discrepancy” was often performed with tests mechanistically similar to the new test under evaluation. After resolving the discordant specimens, sensitivity and specificity were recalculated. Although intuitively appealing, this procedure was inherently biased with substantial overestimation of test performance, even under ideal conditions.^8-10

The history of discrepant analysis reveals the fundamental challenges of communication between laboratory scientists and statistical methodologists. After the FDA allowed discrepant analysis for clearance of a few tests, Hadgu^8,9 and others^10,11 demonstrated the inherent bias in discrepant analysis. These reports led to vigorous debate between the microbiologists and methodologists in letters and commentaries.^7,10,12-14 At times, the misunderstanding between the “scientists” and the “statisticians” was remarkable. The laboratory scientists recognized that the limits of detection of the new nucleic acid amplification tests under evaluation were markedly better than previous culture-based tests.⁷ Thus, discrepant analysis was employed to account for what they considered a biological fact. Assumptions and bias were secondary considerations, as was evident at a diagnostic test evaluation workshop in 1999 at the Centers for Disease Control and Prevention. After carefully explaining the statistical assumptions of various approaches for reference test bias, a prominent laboratory scientist raised his hand and proclaimed, “I work in a laboratory. I make no assumptions. I simply let the data speak for itself.” This fundamentally different perspective was extremely difficult to overcome. Fortunately, the FDA recognized the inherent issues with discrepant analysis and limited its use.

In this issue of EPIDEMIOLOGY, Hadgu and colleagues address a different intuitive yet biased diagnostic test evaluation procedure, the patient infected status algorithm (PISA).¹⁵ Over the past decade, PISA has replaced discrepant analysis as a common procedure for evaluating diagnostic and screening tests for many infectious diseases.^16-18 Unfortunately, this procedure also yields biased results.

PISA uses a combination of tests to create a composite reference standard.^16-18 Sometimes two tests are used with specimens taken from multiple anatomical sites. Hadgu, et al. clearly show that this approach can be associated with significant bias in the simplest conditions (assuming conditional independence between tests) or with more complex conditions (assuming conditional dependence).¹⁵ Given this bias, PISA should not be an acceptable procedure for FDA clearance.

The bias of simple, intuitive approaches to diagnostic test evaluation necessitates the application of more sophisticated statistical approaches. Several alternative approaches have been developed.^19-23 Latent class analysis is a probabilistic approach which recognizes that the true disease state is an unknown, underlying latent variable. In its simplest form, conditional independence is assumed,²¹ but more advanced applications can incorporate conditional dependence.²² Bayesian approaches, including Bayesian latent class models, have also been developed.²³ Generally, these statistically intensive approaches are important steps forward, but unfortunately, none are “ideal.” For example, latent class approaches have been criticized because the latent variable is unknown and unspecified and may not reflect the clinical entity under study.³These approaches may also be limited by feasibility, such as the number of tests required for analysis.²⁰

The past two decades of controversy and progress in diagnostic test evaluation have largely gone unnoticed by traditional epidemiologists. A quick PubMed search reveals zero relevant diagnostic test evaluation articles in the American Journal of Epidemiology and five articles in EPIDEMIOLOGY since 1990. In contrast, the Journal of Clinical Epidemiology has 15 relevant articles since 2006. Should traditional epidemiologists be concerned about diagnostic test evaluation methods? I would have to say yes. Given that the outcomes used in epidemiological studies are determined by diagnostic tests, epidemiologists must have a deep understanding of the validity of these data. Epidemiologists have a long history of evaluating bias and developing appropriate methods to account for bias. If epidemiology’s methodologists gave attention to diagnostic test evaluation, I am convinced that new and important insights would result. Better evaluation of diagnostic tests would improve the quality of epidemiological studies and simultaneously lead to improved clinical outcomes and public health.