Evaluation of Tuberculosis Diagnostics: Establishing an Evidence Base Around the Public Health Impact

Abstract

The limitations of existing tuberculosis diagnostic tools are significantly hampering tuberculosis control efforts, most noticeably in areas with high prevalence of human immunodeficiency virus (HIV) infection and antituberculosis drug resistance. However, renewed global interest in tuberculosis research has begun to bear fruit, with several new diagnostic technologies progressing through the development pipeline. There are significant challenges in building a sound evidence base to inform public health policies because most diagnostic research focuses on the accuracy of individual tests, with often significant limitations in the design, conduct, and reporting of diagnostic accuracy studies. Diagnostic accuracy studies may not be appropriate to guide public health policies, and clinical trials may increasingly be required to determine the incremental value and cost-effectiveness of new tools. The urgent need for new diagnostics should not distract from pursuing rigorous scientific evaluation focused on public health impact.

Global control of the tuberculosis epidemic is a public health priority [1, 2]. The targets for reduction in tuberculosis prevalence and mortality linked to the Millennium Development Goals and enshrined in the STOP TB Global Plan 2006–2015 will not be achieved with current interventions [3, 4]. There is an acute need for improved tuberculosis diagnostics as one critical component of the public health response to the tuberculosis epidemic.

The rapid growth of the human immunodeficiency virus (HIV) epidemic and the emergence of antituberculosis drug resistance have highlighted the major deficiencies in current diagnostic technologies both for pathogen detection and for diagnosis of drug resistance [5]. In most high-burden countries, sputum smear microscopy remains the principal tool for diagnosing active disease; however, operationally, its sensitivity for pulmonary tuberculosis can be as low as 20% [6, 7]. Sputum culture and drug susceptibility testing are available in certain settings, but their impact is limited by the long duration and complexity of the laboratory processes [8]. Additional challenges are faced in developing diagnostics for extrapulmonary tuberculosis, pediatric tuberculosis, and latent tuberculosis infection [9–11].

The STOP TB Global Plan 2006–2015 included the target that, “by 2010, simple, robust, affordable technologies for use at peripheral levels of the health system will enable rapid, sensitive detection of active tuberculosis at the first point of care” [4, p. 24]. Although this has not been achieved, there have been developments in the tuberculosis diagnostic field, and promising technologies have entered the clinical sphere [6, 12–15]. Most promising has been the Xpert MTB/RIF system, an automated molecular test that simultaneously detects Mycobacterium tuberculosis and mutations associated with rifampicin resistance [16, 17]. It is hoped that the renewed global focus on tuberculosis will in the next few years lead to the further proliferation of diagnostic technologies in parallel with advances in therapeutics and vaccines.

It is the responsibility of the global scientific community to correctly evaluate these new technologies so that proven effective and cost-effective diagnostics can be adopted, thus generating the greatest public health impact. The importance of diagnostic research in the overall tuberculosis research agenda has been highlighted by many different groups [2, 15, 18–22]. However, huge gaps in funding for tuberculosis research and tuberculosis control remain [1, 2, 23]; this should force us to rethink how diagnostic research can be most effectively targeted and rationalized to inform public health policies.

This article focuses on the framework for evaluation of new diagnostics: at the outset, we look at the potential benefits of new diagnostics, and then we discuss different methodologies to evaluate diagnostic performance with a view to their ultimate implementation. Our focus throughout is on diagnostic tests for detection of active tuberculosis disease and/or drug resistance in high-burden countries.

POTENTIAL IMPACT OF NEW TUBERCULOSIS DIAGNOSTICS

It has been hypothesized that a test more sensitive than sputum microscopy for tuberculosis would be the diagnostic intervention that would alleviate the greatest burden of infectious disease in developing countries [24]. More specifically, one mathematical model of the global tuberculosis epidemic suggested that a new rapid diagnostic test with 100% sensitivity, 100% specificity, and 100% access could prevent 625000 deaths annually (equivalent to 36% of all tuberculosis-related deaths) [25]. Other models have derived fairly consistent estimates of mortality reductions of 17%–23% from a more sensitive rapid tuberculosis diagnostic, despite exploring different epidemics [26–28]. In one model, the estimated benefit in terms of mortality from a new diagnostic test was equivalent in magnitude to that expected from a novel vaccine or an optimized 2-month treatment regimen for active disease [26]. This highlights 2 important points: (1) no single intervention will have the impact required to meet tuberculosis control targets; thus, scaled-up investment in research and implementation of diagnostics, drugs, and vaccines will be required; and (2) because new diagnostics could have an equivalent impact to new drugs or vaccines, evaluation of diagnostics should be as rigorous as evaluation of drugs and vaccines.

EXISTING FRAMEWORK FOR TUBERCULOSIS DIAGNOSTIC RESEARCH AND DEVELOPMENT

The fact that sputum smear microscopy remains the cornerstone of tuberculosis diagnosis in most high-burden countries is testament to the relative paucity of research and development in the diagnostic arena and the failure to translate research findings into policy. In medicine broadly, diagnostic research tends to be performed in stepwise fashion, with basic science leading to laboratory-based performance evaluation and then to clinical studies (Figure 1) [29]. This structure inherently tends to exclude the perspectives of end users in the conception and development of diagnostics, although more recently in the tuberculosis field, organizations have assisted this process by defining the ideal specifications for a point-of-care test [30].

Figure 1.

Stepwise approach to evaluation of diagnostic technologies.

In the tuberculosis field, the process of diagnostic development has rarely gone beyond diagnostic accuracy studies to assess the impact in clinical practice on clinical decision making, patient outcomes, and health system costs [13, 31, 32]. This is in part explained by the fact that the regulatory framework for in vitro diagnostic devices usually does not require evidence beyond performance data. Diagnostic accuracy studies are an important part of the evaluation process. However, there is much potential for bias in such studies, and diagnostic accuracy might vary widely between different clinical settings and populations [33–36].

In the field of diagnostic accuracy research, there have been certain key initiatives aimed at improving and standardizing research methodologies and reporting: the guidelines for diagnostic evaluation produced by the TDR Diagnostics Evaluation Expert Panel (DEEP) [37], the Quality Assessment of Diagnostic Accuracy Studies (QUADAS) tool [38], and the Standards for the Reporting of Diagnostic Accuracy Studies (STARD) initiative [39, 40]. The DEEP guidelines outline best practice in the design and conduct of diagnostic evaluations, with focus on performance characteristics and operational feasibility. QUADAS is a quality assessment tool to be used specifically for the assessment of diagnostic accuracy studies included in systematic reviews. The tool consists of 14 items (Figure 2); the majority involve sources of bias, with a few relating to variability and quality of reporting. The objective of the STARD initiative is to improve the quality of reporting of diagnostic accuracy studies. The 25-item checklist (Figure 3) allows the reader to judge the potential for bias (internal validity) and the generalizability and applicability (external validity) of the study.

Figure 2.

The Quality Assessment of Diagnostic Accuracy Studies (QUADAS) tool.

Figure 3.

Standards for the Reporting of Diagnostic Accuracy Studies (STARD) checklist.

A systematic review that used both QUADAS and STARD criteria to assess tuberculosis diagnostic accuracy studies published during 2004–2006 showed significant deficiencies in methodology and reporting of studies [41]. Unfortunately, more widespread use of the STARD system has not been apparent in recent years. As a further example, of the 10 published studies evaluating the diagnostic accuracy of the Genotype MTBDRplus assay (published during 2007–2010) [42–51], only one manuscript explicitly mentions STARD [51]. Additional efforts are required by researchers, research funders, journal editors, and policy makers to encourage the use of these tools, with the aim of improving the quality and validity of this element of the evidence base.

THE NEED FOR HIGH-QUALITY EVIDENCE TO INFORM PUBLIC HEALTH POLICIES

Public health policies and guidelines are now usually informed by a systematic approach to judging the relevant evidence. In the tuberculosis field, the World Health Organization (WHO) convenes expert groups to assess the available evidence for a specific intervention (eg, diagnostic test), and this group then presents their findings to the WHO Strategic and Technical Advisory Group for Tuberculosis (STAG-TB) for consideration and endorsement. The system to assess the evidence now adopted by many organizations, including WHO, is the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) system, which incorporates judgments on the quality of evidence (high, moderate, low, or very low) and on the strength of any recommendation (initially categorized as strong or weak; now incorporates “conditional,” whereby national programs should consider implementation based on their own situation) [52, 53].

The GRADE system is based around the concept of patient-important outcomes, and as such, evidence from diagnostic interventions creates additional challenges. Studies using indirect outcomes (eg, diagnostic accuracy studies) will usually provide lower-quality evidence because of the uncertainty about outcomes important to patients and the potential for bias [54]. It is important to be clear that the rating of low quality in this context does not necessarily imply that studies were conducted poorly, but that data from the study are not optimal for deriving public health recommendations.

GOING BEYOND DIAGNOSTIC ACCURACY STUDIES—THE NEED FOR IMPACT DATA

In the STOP TB New Diagnostics Working Group blueprint for the evaluation of diagnostics, the next step after diagnostic accuracy studies are demonstration studies, which include patient outcomes (Figure 4) [55]. These demonstration studies are designed to assess the scaled-up test performance and to determine patient-level outcomes. This is the stage of the evaluation process that should start to inform policy. It is stated in this document that patient-important outcomes should be assessed (eg, time to initiation of treatment, time to smear and/or culture conversion, and treatment outcome) and that “these impact-related data should be compared to historical data recorded prior to implementation of the new test in routine clinical practice” [55, p. 62]. This use of historical data is problematic as a method of assessing any health care intervention and would not generally be accepted by regulatory bodies in the field of drugs or vaccines [56]. It is difficult to be sure that any comparison is fair; there are potential sources of bias, and consequently, the risk is that the value of the intervention can be exaggerated.

Figure 4.

The pathway for evaluation of new diagnostics (from the STOP TB New Diagnostic Working Group).

Two organizations that have been instrumental in driving forward development and evaluation of diagnostic technologies for tuberculosis are the Foundation for Innovative New Diagnostics and the WHO TDR program (Special Programme for Research and Training in Tropical Diseases). Demonstration studies are key elements of their tuberculosis projects, which aim to determine the feasibility, impact, and cost-effectiveness of the diagnostic test under evaluation. The evidence from these studies is a key element assessed by the expert groups and reported to STAG-TB. If we take the example of the Genotype MTBDRplus assay, preliminary data regarding patient-important outcomes from the South African demonstration projects seemed relatively disappointing because the median turnaround times did not meet their predefined objective of 7 days; of the patients with multidrug-resistant tuberculosis who were identified, only 28% were started on appropriate therapy on the basis of the test result (42% had therapy delayed until results of conventional drug susceptibility testing were available) [57]. Although these results were based only on preliminary data analysis and are understandable during implementation of a new technology, there has, to our knowledge, been no further published evidence from high-burden settings on patient-important outcomes. However, the test has been introduced into routine practice in some countries, and its use is now being scaled up [58].

It is generally considered that the optimal methodology for assessing the clinical impact of any intervention, including diagnostics, is the randomized controlled trial (RCT) [59–61]. This is the methodology least prone to bias in estimating the benefits and risks of any intervention. Data from RCTs can additionally be used to perform economic evaluation, a step of major importance for policy makers. The relative shortage of RCTs in diagnostic research, in contrast to therapeutic and vaccine research, is likely to be explained by a combination of factors: lack of emphasis on this level of evidence by manufacturers and regulatory authorities, limited funding and poor coordination of diagnostic research, and logistical and ethical challenges. There are features specific to diagnostic trials that complicate trial design and implementation. In a tuberculosis diagnostic study, the population of interest might be persons with suspected pulmonary tuberculosis (eg, individuals with cough). Inevitably, the majority of participants will not have tuberculosis; thus, the potential effect size on the total cohort resulting from improved diagnosis is relatively small. However, we have to include the entire cohort in a trial if we want to capture comprehensive outcome data (to balance benefits and harms).

To reveal the value of well-designed RCTs in diagnostic research, it is worthwhile to stop studying tuberculosis and consider malaria, another global health priority. Malaria rapid diagnostic tests (RDTs) have been shown to have good diagnostic accuracy [62], and mathematical models have suggested that implementation of RDTs could lead to significant public health benefits in settings where malaria is endemic [63]. Trials were designed to assess the performance of the tests in a field setting and to measure the impact on health care providers, therapeutic decisions, and patient outcomes [64–67]. Three of these trials showed that, despite good diagnostic accuracy, there was no reduction in incorrect antimalarial treatment with the use of RDTs [64–66]; of more concern, one trial even showed a significant reduction in correct antimalarial treatment [66]. These trials have provided vital information for the further development and implementation of RDTs. The results of these trials highlight the fact that a diagnostic test is only ever a vehicle to guide therapies; it is never of therapeutic benefit, and it is the treatment decision that will impact on patient outcomes.

CONCEPTUALIZING CLINICAL TRIALS OF TUBERCULOSIS DIAGNOSTICS

The first step in any trial is to determine the hypothesis that is to be tested because this will inform the trial design. It is important to consider the likely position of the new test in the diagnostic process. In the case of a test for active pulmonary tuberculosis, we need to decide how the test will be introduced in the existing diagnostic structure, which includes sputum microscopy, sputum culture, drug-susceptibility testing, and chest radiography. It could be proposed as a replacement for ≥1 of these tests, as an addition to these tests, or as a means of triage, for example, to target sputum culture and/or drug-susceptibility testing. This decision is in turn likely to depend on the proposed benefits of the new test (eg, whether it is more rapid, more sensitive, more specific, less technical, safer, or less expensive). Furthermore, we need to consider the outcomes of interest, whether related to benefit or harm; these may be appropriate or inappropriate commencement of tuberculosis treatment, outcomes during treatment (smear or culture conversion), final treatment outcomes (cure or completion), and mortality.

One possible reason to explain the lack of RCTs in diagnostic research is the perception that diagnostic tests carry minimal or no risk. Although the test is unlikely to harm the patient, the consequences of the test (eg, the therapeutic decision) may confer harm, as shown in the example of RDTs of malaria. What risks might we expect in a trial of a tuberculosis diagnostic? Consider a hypothetical trial comparing clinical outcomes between a rapid molecular tuberculosis test and the standard-of-care diagnostic pathway (Figure 5). At a basic level, this trial will tell us whether the benefits from earlier correct diagnosis or exclusion of tuberculosis outweigh the risks from incorrect classification of disease (false-negative or false-positive results). The benefits would seem to be self-evident but need to be quantified. The risks are more complicated and will be context specific. False-negative diagnoses will result in appropriate treatment being withheld, with potential for poorer outcomes. False-positive diagnoses also carry risk, however, because alternative diagnoses may not be considered and, therefore, not treated, and patients may be exposed to potentially toxic therapy. For diagnosis of drug resistance, the risks from incorrect classification are even more complicated. False-negative results of genotypic testing may lead to inappropriate treatment with first-line regimens, with consequent adverse outcomes, including amplification of drug resistance. False-positive results may lead to inappropriate treatment with multidrug-resistant tuberculosis regimens, with lower efficacy against sensitive strains and with risks of severe toxicity.

Figure 5.

Potential impact of false-positive and false-negative tuberculosis diagnoses in a hypothetical trial comparing a rapid molecular test to tuberculosis culture.

These examples highlight another challenge with tuberculosis diagnostic research (and common to much diagnostic research), which is the lack of a perfect gold standard with which to compare new tests. If our new test is potentially more sensitive than the existing test (as might be the case with molecular tests, compared with sputum culture), this will affect any analysis. The lack of a gold standard often requires a construct gold standard that comprises information from the reference test with additional clinical information and follow-up information [68]. Of further concern, discrepancies between phenotypic and genotypic drug-susceptibility results can be extremely difficult to interpret, and it is not always clear which is the more reliable measure of drug resistance [69]. In many ways, these issues reinforce the need for well-designed clinical trials because thorough interpretation of the tests may only be possible with meticulously collected baseline and follow-up clinical data.

PRACTICAL TRIAL DESIGNS

If the outcomes of interest are individual-level outcomes (eg, treatment initiation and mortality), a clinical trial with individual randomization would be the logical and statistically most efficient design. However, because there will be information regarding the diagnostic performance from the laboratory-based evaluation, the question arises, if the test is shown to have comparable accuracy to an existing test but has other advantages (ie, more rapid and/or less invasive), is it ethical to conduct an RCT with individual randomization? Critical to this decision is whether there is equipoise regarding the clinical outcome. Equipoise with regard to clinical outcomes of a diagnostic strategy arises, for example, when the consequences of misdiagnosis are severe (eg, HIV-infected patients who receive a misdiagnosis of tuberculosis who are dying of another HIV-related illness) or when failure to diagnose does not lead to mistreatment or poorer outcomes (eg, patients prescribed tuberculosis treatment regardless of the test result).

Individual randomization may, however, present considerable logistical challenges in certain health care settings, and for this reason, cluster randomized designs may be considered with health care units (eg, hospitals, clinics, and mobile teams) as clusters. Cluster randomized designs are increasingly used in public health research. The principal reasons for considering such a design are as follows: if the intervention is to be delivered to groups rather than individuals, if the outcome is to be measured at a population level, or to avoid contamination by individuals in the same community who are randomized to different trial arms [70]. However, there is also an acceptance that cluster randomization may also be appropriate in settings where it offers greater logistical convenience, compared with an individually randomized trial, although cluster RCTs generally require larger sample sizes and have added challenges in design, analysis, and ethics [70–72].

A further modification of the cluster randomized design is the phased implementation or stepped-wedge design [70, 73]. The key features of this design are that all clusters receive the intervention by the end of the trial, and the order in which the clusters receive the intervention is decided at random. This is particularly appropriate when there is preexisting evidence that the intervention may have a beneficial effect and when assigning clusters to the control arm for the duration of the trial might be ethically unacceptable. This might be particularly suited to evaluation of certain diagnostic technologies, for which there is evidence from initial diagnostic accuracy studies that suggests beneficial effect.

If randomization is not deemed to be appropriate or feasible, alternative prospective trial designs, often termed quasi-experimental designs, may still be able to generate evidence on the effectiveness of diagnostics [74]. An example would be the pre- and postimplementation study in which outcomes are measured during a pre-intervention phase and subsequently during a postintervention phase. Although the lack of randomization threatens the internal validity (no firm conclusion can be made with regard to the effect of the intervention unless the effect size is large), there may conversely be a gain in external validity (improved generalizability of findings if fewer patients are excluded than in conventional RCTs).

Retrospective studies may be the only methodology to obtain outcome data in circumstances in which a diagnostic is widely implemented on the basis of performance characteristics. Such pre- and postimplementation analyses have been used in high-resource settings to estimate the impact of molecular resistance testing on detection and treatment of multidrug-resistant tuberculosis [75, 76].

Whether a clinical trial is justified in the evaluation of diagnostics will ultimately depend on the balance between the benefit to be gained by accurately establishing the impact of a new tool and the costs of running a large clinical trial and potentially delaying full-scale implementation of an effective intervention. These decisions are not straightforward, and collaboration between scientists and policy makers is vital to determine when diagnostic trials are necessary.

CONCLUSIONS

Recent developments in tuberculosis diagnostics have led to much optimism, but we still lack the tools that meet the needs of patients in high-burden countries. The next 10–20 years will hopefully see further developments in diagnostic technology. We need to ensure that the framework for evaluating diagnostic tools is best suited to ensuring that the tools with the greatest public health impact and cost-effectiveness are implemented and that those with minimal impact are developed further or are discarded. Diagnostic accuracy studies are an important early step in the evaluation process but do not produce sufficient evidence to inform public health policies. Well-designed prospective studies (including RCTs) should be integrated in the research pathway to provide reliable information on therapeutic impact, patient outcomes, and cost-effectiveness. This new era of tuberculosis diagnostics should be accompanied by a new era for diagnostic research focused clearly on the evaluation of public health impact.