Promise versus Reality: Optimism Bias in Package Inserts for Tuberculosis Diagnostics

Abstract

Laboratorians and clinicians often rely on package inserts of diagnostic tests to assess their accuracy. We compared test accuracy for tuberculosis diagnostics reported in 19 package inserts against estimates in published meta-analyses and found that package inserts generally report overoptimistic accuracy estimates. However, package inserts of most tests approved by the U.S. Food and Drug Administration (FDA) or endorsed by the World Health Organization provide more realistic estimates that agree with meta-analyses.

TEXT

Most laboratory professionals anticipate that test performance when applied to patient care may be less impressive than what is reported in package inserts by test manufacturers. However, this gap between promise and reality has not been examined in a systematic way.

One approach for studying this gap is to compare sensitivity and specificity estimates in package inserts with pooled estimates from published systematic reviews and meta-analyses, which often include studies from diverse settings and may provide a more realistic assessment of test accuracy.

We conducted such a comparison using commercial tuberculosis (TB) diagnostics as a case study. The TB diagnostic pipeline has rapidly expanded, and a large number of meta-analyses have been published on various TB tests (24). Several tests are now endorsed by the World Health Organization (WHO). At the same time, there are examples of suboptimal and inaccurate TB diagnostic tests, and their use has been discouraged by the WHO (32). In fact, in vitro tests are often poorly regulated in many countries (13).

We searched PubMed, the Cochrane Library, and the Evidence-Based TB Diagnosis website (www.tbevidence.org) for systematic reviews on the accuracy of diagnostics for TB published through March 2012 (search terms are available from the authors upon request). We excluded meta-analyses that reported only performance characteristics other than sensitivity and specificity (e.g., reproducibility, likelihood ratios), as they were not comparable to information provided in package inserts.

We searched company websites for package inserts and contacted test manufacturers if package inserts were not available online. Diagnostic tests were not considered if they were not commercially available. Diagnostic tests for latent TB were not considered, because no good reference standard is available to determine accuracy.

A total of 19 TB tests were included in the final analysis, because they met our eligibility criteria and package inserts as well as meta-analyses where available. These included gamma interferon (IFN-γ) release assays (IGRAs) for active TB, antibody-based serological tests, antigen detection tests, nucleic acid amplification tests (NAAT), and culture-based tests.

As seen in Table 1, the quality of information provided in package inserts varies widely. Eighteen out of 19 package inserts in total overestimated the test accuracy. In particular, technologies that were neither recommended by the WHO nor approved by the U.S. Food and Drug Administration (FDA) reported higher accuracy (i.e., serological tests for active TB, IGRAs for active TB, bacteriophage-based tests for active TB and drug-susceptibility testing, and urine lipoarabinomannan [LAM] antigen assays) (30, 32, 34). Claims made in package inserts ranged on average from 20 to 30% higher for sensitivity estimates than meta-analysis (comparing a range of findings in meta-analyses to estimates in package inserts). A direct comparison of absolute estimates of test accuracy was not feasible for most nonapproved tests, because the systematic reviews were unable to compute pooled sensitivity and specificity due to heterogeneity across studies.

Table 1.

Comparison of test accuracy in package inserts versus meta-analyses^a

Indication	Specimen	Test, yr of package insert publication (if reported)	WHO endorsed	FDA approved	Package insert		No. of samples included	Comment on package insert	Meta-analysis		No. of studies/no. of participants included (reference)	Comment on meta-analysis
					Sensitivity (%)	Specificity (%)			Sensitivity (%)	Specificity (%)
IFN-γ release assays
Active PTB	Blood	TB-SPOT.TB assay (Oxford Immunotec, Abingdon, United Kingdom), 2010	No	No	96	97	189 for sensitivity, 311 for specificity	Retrospective analysis of samples	92	59	27/3,886 (26)	Sensitivity reported for culture-confirmed cases; specificity reported for TB suspects, which may in part explain the reduced estimate compared to the PI
	Blood	QuantiFERON-TB Gold In-Tube (QFT-GIT, Cellestis, Chadstone, Australia), 2006	No	No	89	99	54 for sensitivity, 581 for specificity	Prospective analysis; unpublished; not stratified by clinically relevant subgroups for active TB (i.e., HIV+, TB prevalence)	81	79	27/3,886 (26)	Sensitivity reported for culture-confirmed cases; specificity reported for TB suspects, which may in part explain the reduced estimate compared to the PI
Active PTB	Blood	TB-SPOT	No	No	96	97	189 for sensitivity, 311 for specificity	Retrospective analysis of samples	83 (68 HIV+, 88 HIV−)	61 (52 HIV+)	27/1,434 (19)	Stratified by HIV; sensitivity reported for culture-confirmed cases; specificity reported for TB suspects
	Blood	QFT-IT	No	No	89	99	54 for sensitivity, 581 for specificity	Prospective analysis; unpublished; not stratified by clinically relevant subgroups for active TB (i.e., HIV+, TB prevalence)	69 (65 HIV+, 84 HIV−)	52 (50 HIV+)	27/1,434 (19)	Stratified by HIV; sensitivity reported for culture-confirmed cases; specificity reported for TB suspects.
Active PTB and EPTB	Blood	TB-SPOT	No	No	96	97	189 for sensitivity, 311 for specificity	Retrospective analysis of samples	88–91	86–88	2 MAs: 16/1,658; 36/1,825 for sensitivity and 7/767 for specificity (6, 9)	Specificity reported in healthy controls; specificity lower if assessed in TB suspects
	Blood	QFT-IT	No	No	89	99	54 for sensitivity, 581 for specificity	Prospective analysis; unpublished; not stratified by clinically relevant subgroups for active TB (i.e., HIV+, TB prevalence)	79–81	93–99	2 MAs: 16/1,658; 36/1,825 for sensitivity and 7/767 for specificity (6, 9)	Specificity reported in healthy controls; specificity lower if assessed in TB suspects
Antigen-based tests
Active PTB; HIV+	Urine	LAM-ELISA (Chemogen, Clearview, Now Alere TB-LAM ELISA, Waltham, MA), 2011	No	No	73–81 (for HIV+ only)	70–88	469	2 unpublished, prospective studies in HIV+ patients); varying gold standard; specificity in 96 healthy controls (93–98%)	47–51	94–96	2 MAs: 5/645; 7/2,583 (11, 21)	Evaluated precommercial and commercial tests; concerns about methodology in majority of studies; specificity mostly tested on TB suspects but healthy controls included too; sensitivity approaches no. in package inserts in patients with advanced HIV
Active PTB; HIV−	Urine	LAM-ELISA (Chemogen, Clearview), 2008	No	No	Estimates not reported for HIV−	Estimates not reported for HIV−	No samples from HIV− patients included	2 unpublished, prospective studies in HIV+ patients); varying gold standard; specificity in 96 healthy controls (93–98%)	14	97	2 MAs: 5/645; 7/2,583 (11, 21)	Evaluated precommercial and commercial tests; concerns about methodology in majority of studies; specificity mostly tested on TB suspects but healthy controls included too; sensitivity approaches no. in package inserts in patients with advanced HIV
Serological antibody detection assays
Active S+ PTB	Blood	anda-TB IgG (anda Biologicals, Strasbourg, France)	WHO recommended against	No	48–100	71–100	Not reported	PI reports several published studies but does not provide further details on studies (i.e., SS, study design, gold standard, characteristics of controls)	76	92	11/1,570 (28)	All studies in the literature with serious methodological problems and concerns about study population not being representative
Active S− PTB	Blood	anda IgG	WHO recommended against	No	48–100	71–100	Not reported	PI reports several published studies but does not provide further details on studies (i.e., SS, study design, gold standard, characteristics of controls)	59	91	11/1,570 (28)	All studies in the literature with serious methodological problems and concerns about study population not being representative
Active EPTB	Blood	anda IgG	WHO recommended against	No	48–100	71–100	Not reported	PI reports several published studies but does not provide further details on studies (i.e., SS, study design, gold standard, characteristics of controls)	81	85	11/1,570 (28)	All studies in the literature with serious methodological problems and concerns about study population not being representative
Active PTB	Blood	Commercially available ELISA +ICT (MycoDot, Mossman Blackstone, MA; Mycobacterium tuberculosis IgG, IBL, Hamburg, Germany), 2011; ActiveTBDetect (InBios International, Seattle WA), 2008; SEVA (Mahatma Gandhi Institute of Medical Sciences, India; Pathozyme, Omega Diagnostics, Alva, Scotland), 2009; Hexagon (Human Gesellschaft Biochemica und Diagnostica, Wiesbaden, Germany), 2011; Serocheck-MTB (Zephyr, Biomedicals, Goa, India)	WHO recommended against	No	70–100 (all tests but anda)	90–100 (all tests but anda)	No. often not reported or <50	Many tests are no longer commercially available; reported information in PIs very limited: often only small, unpublished, retrospective analyses using stored samples; positive exception is Mycodot with large prospective studies	60–88	50–98	2 MAs: 54/3,696; 8/mean SS per study 250, total no. not available (10, 28)	All studies with methodological problems as described above
Active EPTB	Blood	Commercially available ELISA +ICT (MycoDot, Mossman Blackstone, MA; Mycobacterium tuberculosis IgG, IBL, Hamburg, Germany), 2011; ActiveTBDetect (InBios International, Seattle WA), 2008; SEVA (Mahatma Gandhi Institute of Medical Sciences, India; Pathozyme, Omega Diagnostics, Alva, Scotland), 2009; Hexagon (Human Gesellschaft Biochemica und Diagnostica, Wiesbaden, Germany), 2011; Serocheck-MTB (Zephyr, Biomedicals, Goa, India)	WHO recommended against	No	48–100	71–100	No. often not reported or <50	Only anda test remains commercially available	43	93	4/604 (10)	All studies of moderate quality
Bacteriophage-based tests
Active PTB	Sput direct	FASTPlaque-TB (Biotec, Kentford, United Kingdom), 2004	No	No	73–82 (S−, 49–67; S+, 87)	98–99 (S−, 98–100; S+, 83–88)	>2,000	2 published, prospective studies	21–94 (S−,13–78; S+ 75–87)	83–100 (S−, 89–99; S+, 60–88)	13/5,820 (14)	Data in meta-analysis not pooled due to heterogeneity
Active PTB; R RIF	Sput S+	FASTPlaque RIF, FASTPlaque Response (Biotec, Kentford, United Kingdom), 2005	No	No	96–100 (only S+ cases)	98–100	374	PI only available for FASTPlaque Response; 2 published studies; 17–27% with uninterpretable results	96	95	31/3,085 (22)	Combines older and newer tests; 3–16% uninterpretable results
Nucleic acid-based test MTB detection
Active PTB	Sput direct	Xpert MTB/Rif (Cepheid, Sunnyvale, CA), 2011	Yes	No	92 (S+, 98; S−, 73)	99	1335	No information on sample collection, controls, other; Seminal papers cited	90 (S+, 99; S−, 75)	98 (S+, 98; S−, 98)	18/10,224 (5)	Study performed simple pooling of sensitivity and specificity
Active PTB	Sput direct	Amplified MTD (Gen-Probe, San Diego, CA), 2001	No	Yes	86 (S+, 97; S−, 72)	99 (S+, 100; S−, 99)	206	Prospective, unpublished data from 7 study sites	88 (S+, 97–100; S−, 70–76)	96 (S+, 96–98; S−, 95–97)	3 MAs: 25/mean SS per study 362; 14/median SS 410; 40/mean SS 715 (10, 12, 16)	Results more consistent for specificity; BD Probe Tec data combined older and newer versions
	Sput direct	Probe Tec ET (BD, Franklin Lakes, NJ), 2010	No	Yes	91 (S+, 99; S−, 75)	97	986	Prospective, unpublished data from 2 study sites	86–88 (S+, 98; S−, 71)	98–99 (S+, 89; S−, 97)	3 MAs: 3/213 mean SS per study; 12/median SS 410; 9/mean SS 715 (10, 12, 16)	Results more consistent for specificity; BD Probe Tec data combined older and newer versions
Nucleic-acid based test resistance detection
Active PTB; R RIF	Sput direct	Xpert MTB/Rif	Yes	No	97	98	567	No further information on sample collection, controls; seminal papers cited	94	97	18/10,224 (5)	Study performed simple pooling of sensitivity and specificity
Active PTB; R RIF	Sput direct and indirect	GenoType MTBDRplus (Hain Lifescience GmbH, Nehren, Germany), 2009	Yes	No	PI without any information about test accuracy	PI without any information about test accuracy	PI without any information about test accuracy	PI without any information about test accuracy	98–99	99	2 MAs: 5/767; 4/931 (2, 17)	One meta-analysis analyzed studies that tested only directly on smear-positive sputum; the other evaluated both direct and indirect testing
Active PTB; R INH	Sput direct and indirect	GenoType MTBDRplus	Yes	No	PI without any information about test accuracy	PI without any information about test accuracy	PI without any information about test accuracy	PI without any information about test accuracy	89–96	99–100	2 MAs: 5/767; 5/981 (2, 17)	One meta-analysis analyzed studies that tested only directly on smear-positive sputum; the other evaluated both direct and indirect testing
Active PTB; R RIF	Sput direct and indirect	Inno LIPA RIF/TB (Innogenetics NV, Gent, Belgium), 2011	Yes	No	99	100	289	Retrospective, unpublished; only done indirectly on culture	80–100	92–100	15/1,738 (23)	Discrepancy in sensitivity estimates compared to PI likely in parts due to the inclusion of studies that tested directly on sputum in meta-analysis
Microscopic observation drug susceptibility testing
Active PTB	Sput direct	TB MODS kit (Hardy Diagnostics, Santa Maria, CA), 2011	Yes	No	98 for MTB detection (R 100% for RIF, 97% for INH)	100 for MTB detection	No data	PI refers to one major published study without further elaborating on study findings	92	96	14/3,731 for sensitivity; 12/7,226 for specificity (15)
Active PTB; R RIF	Sput direct	MODS	Yes	No	98 for MTB detection (R 100% for RIF, 97% for INH)	100 for MTB detection	No data	PI refers to one major published study without further elaborating on study findings	96–98	96–99	2 MAs: 6/1187 and 9/1474 (2, 20)	Sensitivity, specificity for INH varies slightly at different cutoff (0.1 or 0.4 μg/ml)
Active PTB; R INH	Sput direct	MODS	Yes	No	98 for MTB detection (R 100% for RIF, 97% for INH)	100 for MTB detection	No data	PI refers to one major published study without further elaborating on study findings	92	96	2 MAs: 6/1187 and 9/1474 (2, 20)	Sensitivity, specificity for INH varies slightly at different cutoff (0.1 or 0.4 μg/ml)

^aPI, package insert; R, resistance; RIF, rifampin; INH, isoniazid; PTB, pulmonary TB; EPTB, extrapulmonary TB; S⁺, smear positive; S⁻, smear negative; SS, sample size; MAs, meta-analyses; MTB, Mycobacterium tuberculosis.

In contrast, there is a better match between sensitivity and specificity estimates in package inserts and meta-analyses for tests that are approved by the FDA (i.e., Gen-Probe amplified MTD) or endorsed by the WHO (line probe assays, Xpert MTB/RIF, and MODS) (4, 31, 33, 34). For the WHO-endorsed tests that allowed a comparison of test accuracy between meta-analyses and package inserts (i.e., all except for the line probe assays), the package inserts overestimated sensitivity and specificity by at most 5%. This was also true for FDA-approved tests. IGRAs are FDA approved for latent TB but not active TB. The sensitivity for IGRAs for active TB was overestimated in package inserts by up to 20%. The comparison of the specificity of IGRAs was limited by the fact that meta-analyses of active TB often used TB suspects as controls, while package inserts reported specificity for latent TB infection among healthy, low-risk populations (6, 9, 19, 26).

We also found that test accuracy estimates in package inserts are often derived from unpublished, in-house, case-control studies with small numbers of specimens. Confirmed TB cases and healthy controls are often used, which can introduce significant selection (spectrum) bias (25). The data in the package inserts often are not stratified based on important predictors of performance, including prevalence of TB or HIV and adults versus children, which may contribute to the overestimation of accuracy (3, 8, 19). In contrast, meta-analyses were often based on a fairly large number of studies that used cross-sectional or prospective designs and often were conducted in clinical settings with TB suspects that had a confirmed alternative final diagnosis serving as controls. Results were often stratified based on clinically relevant subgroups.

In general, involvement of industry and test developers in diagnostic evaluations has been associated with an overestimation of test accuracy (1). With TB tests, this has been documented with bacteriophage-based tests and urine lipoarabinomannan assays (11, 14, 21, 22). Users in real-world clinical settings may lack the same degree of expertise and skill as test developers. Also, quality control and assurance in routine clinical and laboratory settings may not match that of the industry. While data included in package inserts are almost always funded by industry, a proportion of studies included in the meta-analyses also are industry supported or conducted by test developers. This may then spuriously narrow the gap between package insert and meta-analyses estimates.

Our study has limitations. We were unable to compute numeric differences in the estimates of meta-analyses versus package inserts because pooling of data was often not possible due to heterogeneity between studies and the presence of several meta-analyses that included partially overlapping studies. We acknowledge that real-world performance of tests, especially when tests are scaled up in public health programs, may be worse than those reported in research studies, including meta-analyses (27). Thus, the real gap between package insert estimates and real-world performance may be even wider than what we document here. Pragmatic trials and implementation research are needed to overcome this problem (18). We also acknowledge that tests that measure the immune response to TB (i.e., serology, IGRAs) rather than products of Mycobacterium tuberculosis (i.e., Xpert MTB/RIF) might be more prone to variability in the results; however, this underlines the fact that accuracy data should always be stratified based on clinically relevant subgroups (i.e., HIV positive).

In summary, this case study of TB diagnostics suggests that package inserts often report overoptimistic estimates of test accuracy, especially if the products are not FDA approved (provided that approval was solicited) or WHO endorsed. These data provide some reassurance that independent review by credible agencies such as the FDA and WHO may serve as a yardstick for judging new TB technologies. However, not all TB tests are reviewed by the FDA or WHO, and most developing countries have weak regulatory systems for diagnostics. It is important that these countries create systems for in-country validation of all TB tests, guided by their national TB programs. Also, an expansion of the WHO prequalification of diagnostic programs to TB diagnostics will help countries procure quality-assured TB tests.

To overcome the problem of optimism bias, studies evaluating diagnostics under routine clinical and programmatic conditions, independent of industry sponsorship or test developers, are needed, as they provide more useful and realistic evidence to guide laboratorians, clinicians, and decision-makers. Furthermore, studies must go beyond accuracy and assess clinical impact of tests on decision-making and patient outcomes and collect operational and cost-effectiveness data in programmatic settings (7, 18).