Diagnostic Accuracy of In-House PCR for Pulmonary Tuberculosis in Smear-Positive Patients: Meta-Analysis and Metaregression

Abstract

In-house PCR (hPCR) could speed differential diagnosis between tuberculosis (TB) and nontuberculous mycobacterial disease in patients with positive smears and pulmonary infiltrates, but its reported accuracy fluctuates across studies. We conducted a systematic review and meta-analysis of hPCR sensitivity and specificity for smear-positive TB diagnosis, using culture as the reference standard. After searching English language studies in MEDLINE and EMBASE, we estimated cumulative accuracy by means of summary receiver operating characteristic analysis. The possible influence of hPCR procedures and study methodological features on accuracy was explored by univariate metaregression, followed by multivariate adjustment of items selected as significant. Thirty-five articles (1991 to 2006) met the inclusion criteria. The pooled estimates of the diagnostic odds ratio, sensitivity, and specificity (random-effect model) were, respectively, 60 (confidence interval [CI], 29 to 123), 0.96 (CI, 0.95 to 0.97), and 0.81 (CI, 0.78 to 0.84), but significant variations (mainly in specificity) limit their clinical applicability. The quality of the reference test, the detection method, and real-time PCR use explained some of the observed heterogeneity. Probably due to the limited study power of our meta-analysis and to the wide differences in both laboratory techniques and methodological quality, only real-time PCR also displayed a positive impact on accuracy in the multivariate model. Currently, hPCR can be confidently used to exclude TB in smear-positive patients, but its low specificity could lead to erroneous initiation of therapy, isolation, and contact investigation. As the inclusion of samples from treated patients could have artificially reduced specificity, future studies should report mycobacterial-culture results for each TB and non-TB sample analyzed.

With the continuing expansion of human immunodeficiency virus and other immunosuppressive conditions, tuberculosis (TB), as well as infections caused by nontuberculous mycobacteria (NTM), have increased in many parts of the world. In addition, the spread of tumor necrosis factor alpha-blocking agents has led to high rates of development of both infections among patients affected by inflammatory and autoimmune diseases. Similarities in clinical and radiographic features, particularly in cases of fibrocavitary disease, make differential diagnosis between TB and NTM infection a difficult task. In order to avoid the risk of progressive disease, the initiation of empirical antituberculous therapy has been suggested for patients with both positive acid-fast bacillus smear microscopy (AFB) and nucleic acid amplification tests (NAATs) pending culture results (25).

The diagnostic accuracy of NAATs for AFB-positive samples, however, is still unclear. In a previous meta-analysis of commercially based NAATs, we found that their sensitivities and specificities for AFB-positive samples ranged, among different kits, from 0.96 to 0.98 and from 0.71 to 0.96 and that they were considerably influenced by some characteristics of the primary studies. We concluded that, based on published evidence, the clinical use of commercial NAATs should be limited to the exclusion of TB diagnosis in AFB-positive patients with suspected NTM infection (24).

Among NAATs, in-house PCR (hPCR) appeared first in the diagnostic mycobacterial arena. Despite a plethora of studies published over almost 20 years, considerable variations in hPCR sensitivity and specificity can be observed among different reports (37). Such variations can be due to several factors. In AFB-positive samples containing more than 10,000 Mycobacterium tuberculosis bacilli/ml, sampling and detection errors are infrequent, and the most common cause of impaired hPCR sensitivity is the presence of inhibitors of enzymatic amplification in the specimen. On the other hand, lack of specificity mainly arises from carryover of target DNA (i) from samples containing a heavy M. tuberculosis load, (ii) from laboratory machinery and work surfaces contaminated by amplicons, or (iii) from medical equipment contaminated by M. tuberculosis organisms (42). It should be noted that the presence of antituberculous drugs in samples from treated patients may hamper M. tuberculosis growth and produce pseudo-false positives when hPCR results are compared with those of culture.

A number of different procedures have been developed to overcome these problems, making hPCR not just one technique, but a method encompassing different techniques, each of which is potentially able to modify a test's diagnostic yield (6).

In this meta-analysis, we systematically reviewed studies of hPCR, focusing on its performance for AFB-positive respiratory samples, an issue that was not previously investigated in diagnostic meta-analyses of NAAT accuracy for TB (22, 45). We also analyzed the possible impact of both hPCR testing procedures and primary study methodological characteristics on reported estimates of diagnostic accuracy.

MATERIALS AND METHODS

We searched MEDLINE up to 13 June 2008 and EMBASE up to 1 March 2005, using a search strategy designed to identify studies evaluating hPCR use for pulmonary-TB diagnosis (see the supplemental material). We screened the titles and abstracts of the identified citations and scrutinized the references listed in the retrieved articles, considering any citation that did not obviously fail the inclusion criteria.

After a preliminary analysis of a sample of articles, we considered eligible for inclusion in our meta-analysis studies that (i) examined hPCR diagnostic performance for AFB-positive respiratory samples (<5% nonrespiratory samples was tolerated), (ii) used M. tuberculosis culture of the same sample as a reference standard for the diagnosis of pulmonary TB, (iii) reported primary data sufficient for separately calculating both sensitivity and specificity for AFB-positive samples, and (iv) were written in English.

Reasons for article exclusion were (i) reporting sensitivity and specificity “revised” by means of discrepant analysis as the only study results (in the case of studies in which the samples were retested on the basis of discrepant analysis, only the initial “unrevised” results were considered), (ii) application of hPCR assays for determining drug resistance, (iii) possible duplicate publication (when an author or a research group published more than one study, the existence of overlapping study populations was ascertained by checking sample recruitment sites and/or periods or, if these were not available, contacting the authors for clarification; if this was not provided, only the article reporting on the largest number of samples was included), and (iv) application of hPCR to gastric aspirates (>5% of the total study sample).

Two investigators (S.G. and M.R.) independently evaluated the studies for inclusion and abstracted relevant data. Disagreements were reconciled by consensus.

Data extraction and quality assessment.

The data abstracted were descriptive items (author name, journal, and publication year), sensitivity and specificity estimates, techniques and procedures used for hPCR, culture and AFB staining, and study methodological characteristics.

According to established methodological standards for the evaluation of diagnostic-test studies (8, 57), we considered three aspects of study quality: the patient spectrum, the technical quality of the reference test, and use of blinding. A spectrum of consecutively enrolled patients with suspected pulmonary TB or NTM disease for whom the clinician required the laboratory to perform AFB staining and mycobacterial culture was considered representative of that encountered in clinical practice. Seven items (population of recruitment, method of sample selection, data collection modality, clinical and demographic characteristics, culture results in non-TB patients, and inclusion of patients on treatment) were examined to define the adequacy of the patient spectrum. The employment of at least two different culture media was considered a more reliable reference test. Finally, since knowledge of the results of the reference standard can influence the reading of the test under evaluation (and vice versa), particularly if they are obtained at different times, we evaluated the application of any type of blinding.

Statistical analysis.

For each study, we classified hPCR results as true positives (TP), false negatives (FN), false positives (FP), and true negatives (TN) as determined by comparison with M. tuberculosis culture results. Then, we calculated the TP rate (TP rate = TP/[TP + FN] = sensitivity), the FP rate (FP rate = FP/[FP + TN] = 1 − specificity), and the diagnostic odds ratio (DOR), i.e., the ratio of the odds of a positive hPCR result among M. tuberculosis culture-positive samples to the odds among M. tuberculosis culture-negative samples (DOR = odds_{TP rate}/odds_{FP rate}). The potential problems in odds calculations associated with sensitivities and/or specificities of 100% were solved by adding 0.5 to zero values (30). The pooled estimates of sensitivity, specificity, and DOR were calculated by applying the DerSimonian-Laird random-effect model, which accounts for both within-study variability (random error) and between-study variability (heterogeneity) (MetaDisc software, version 1.4).

The cumulative accuracy was estimated by means of a summary receiver operating characteristic (SROC) curve (for details, see Littemberg and Moses [35]), a regression line that summarizes the results of individual studies and depicts the trade-off between sensitivity and specificity when the test threshold varies across studies. The area under the curve (AUC) was used as measure of the overall accuracy of the test (an AUC value of 100% indicates a perfect test, while an AUC value of 50% means that the test does not have discriminating ability).

To delineate the impact of study characteristics on study estimates of diagnostic accuracy, we performed a univariate metaregression followed by a multivariable adjustment for only those items selected as significant in the univariate analysis (the P value for entry was fixed at <0.05). The regression models were an extension of the SROC curve: the DOR was used as the dependent variable, and study characteristics were added as covariates to the model, while the S parameter accounted for threshold variations across studies (52) (“Metareg” in Stata 8). The variables entered in the univariate metaregression are listed in Table 1 and Table 2. Furthermore, we analyzed the effect of the sample volume added to the hPCR mixture and of the proportion of culture-positive samples. The latter was utilized as a proxy for the pulmonary-TB prevalence, since it is known that sensitivity and specificity vary with disease prevalence when an imperfect reference test is used (9, 10). The numeric variables were included in the model after logarithmic transformation. The studies were weighted by the inverse of variance of the DOR to allow for the precision with which each study measured the DOR.

TABLE 1.

Testing procedures for hPCR

Characteristic	No. of studies (%)c
Amplification target
IS6110	24 (69)
Othera	10 (29)
Unreported	1 (3)
DNA purification method
Lysis only (physical methods)	18 (51)
Chemical methods	17 (49)
Amplification technique
Conventional	24 (69)
Nested or heminested	8 (23)
Real time	3 (9)
Use of positive control
Yes	23 (66)
No or unreported	12 (34)
Use of negative control
Yes	27 (77)
No or unreported	8 (23)
Use of internal control
Yes	12 (34)
No or unreportedb	23 (66)
Use of dUTP-UNG
Yes	7 (20)
No or unreported	28 (80)
Amplicon detection method
Gel electrophoresis + UV	19 (54)
Use of any hybridization method	16 (46)

^aOther amplification targets: 16S rRNA, the genes coding for the 32-kDa and 38-kDa proteins, dnaJ, hsp65, IS986, internal transcribed spacer region between 16S rRNA and 23S rRNA, region 650-900, rpoB, and 23S rRNA. Two further studies using IS6110 plus either 38 kDa or MPB642 were classified as using IS6110.

^bA study applying the internal control to 8/50 samples was classified as not using the internal control.

^cThe sum of the percentages is not 100% due to approximation.

TABLE 2.

Methodological characteristics of primary studies

Characteristic	No. of studies (%)c
Type of respiratory specimen
Sputum	16 (46)
Bronchial secretions	2 (6)
Mixed respiratory secretions	17 (49)
AFB method
Fluorescence	11 (31)
Carbolfuchsin	17 (49)
Unreported	7 (20)
Quality of reference test
At least 2 culture media	16 (46)
One culture media	13 (37)
Unreported	6 (17)
Population of recruitment
M. tuberculosis culture or suspected PTBa	17 (49)
Otherb	3 (9)
Suspected PTB or drug monitoring	3 (9)
Unreported	12 (35)
On anti-TB treatment
No	3 (9)
Yes	12 (35)
Unreported	20 (57)
Method of sample selection
Consecutive or random selection	6 (17)
Consecutive/case control	1 (3)
Case control	4 (11)
Unreported	25 (71)
Data collection modality
Prospective	4 (11)
Retrospective	1 (3)
Unreported	30 (86)
Independence of observation
Any blinding	9 (26)
Unreported	26 (74)
Clinical/demographic characteristics
Reported	2 (6)
Unreported	33 (94)
Culture results in the control group
Reported	9 (26)
Unreported	26 (74)

^aPTB, pulmonary TB.

^bPatients (pts) with fever and chronic cough, HIV-positive pts with pulmonary infiltrates; pts with AFB-positive bronchoalveolar lavage fluid.

^cThe sum of percentages is not 100% due to approximation.

Publication bias is the tendency on the part of investigators to publish (and of reviewers to accept for publication) articles with more optimistic results and may lead to inflated estimates of diagnostic accuracy in meta-analyses. We assessed the possibility of publication bias among the included studies by evaluating a funnel plot for asymmetry, Begg's adjusted rank correlation test, and Egger's regression asymmetry test (“Metabias” in Stata 8).

Finally, we applied Bayes' theorem to define the changes in the probability of pulmonary TB determined by the use of hPCR.

RESULTS

Study description and synthesis.

The study selection process, depicted in Fig. 1, led to the inclusion of 35 journal articles published between 1991 and 2006 (1, 2, 4, 5, 12-15, 17, 19-21, 23, 27, 29, 31, 32, 34, 36, 38, 41, 43, 44, 46-49, 51, 53-56, 58-60). Since three articles analyzed different hPCR techniques for the same samples, we included only the results of the technique that obtained the best overall accuracy (12, 31, 58).

FIG. 1.

Study selection process.

Finally, 2,152 AFB-positive samples were analyzed: 1,578 yielded a positive culture for M. tuberculosis, and 574 were M. tuberculosis culture negative. The median number of samples per study was 35 (interquartile range [IQR], 20 to 67), with a median pulmonary-TB prevalence of 0.77 (IQR, 0.63 to 0.88). As shown by the SROC curve (Fig. 2), the cumulative accuracy of hPCR for AFB-positive samples was quite elevated (AUC, 0.96; DOR, 60; confidence interval [CI], 29 to 123), mainly due to high pooled sensitivity (0.96; 95% CI, 0.95 to 0.97), while the pooled specificity was lower and extremely variable (0.81; 95% CI, 0.78 to 0.84) (Fig. 3 shows forest plots). Only one study displayed a markedly low sensitivity (0.20), attributed by the authors to the low number of IS6110 sequences among M. tuberculosis isolates in India (18, 27). However, a chi-square test demonstrated a marked heterogeneity among sensitivity, specificity, and DOR values (P < 0.001 for all), which limits the clinical utility of the pooled estimates obtained through this meta-analysis.

FIG. 2.

SROC plot for hPCR performed on AFB-positive respiratory samples. The curve is the regression line that summarizes the overall diagnostic accuracy and the trade-off between sensitivity and specificity of the test. Each circle represents a study accuracy value. Circle sizes are proportional to study sizes. Studies applying conventional PCR, black circles; studies applying nested PCR, gray circles; studies applying real-time PCR, white circles.

FIG. 3.

Individual study estimates of sensitivity and specificity of hPCR for the diagnosis of pulmonary TB with AFB-positive samples. The squares are single estimates, and the error bars represent 95% CIs. Square sizes are proportional to study sizes.

As expected, the primary studies differed greatly in terms of the techniques (or combinations of techniques) used to set up hPCR assays (Table 1). Overall, the repetitive sequence IS6110 and conventional gel electrophoresis were the most frequently used amplification target and detection technique, respectively. Template DNA volumes were generally much smaller than in commercially based amplification assays (median, 5 μl; IQR, 5 to 10), a factor related to higher inhibition rates, due to the lower dilution of inhibitory substances potentially present in the sample.

The analysis of methodological characteristics (Table 2) demonstrated that most articles did not comply with the published guidelines for conducting diagnostic-test studies, and nonreporting of items was common. In regard to M. tuberculosis culture, we found that 17% of the primary studies did not provide any description of the reference test used to assess pulmonary-TB diagnosis, while 37% applied only one culture medium. Although more than half of the studies reported the enrollment of patients with suspected pulmonary TB, they often included samples from patients on antituberculous treatment. The clinical spectrum of both pulmonary TB and comparative groups was rarely illustrated, and only nine primary studies applied either single or double blinding for test interpretation.

Effects of study characteristics on hPCR diagnostic accuracy.

Heterogeneity is a common finding in diagnostic meta-analyses and may result from a threshold effect or from differences either in testing procedures or in study methodological characteristics. In our meta-analysis, the value of the S parameter in the SROC analysis (−0.19; standard error, 0.15) indicates that modifications of cutoff values across different studies influenced their results. The random-effect univariate metaregression identified three further potential confounders: the quality of the reference test, application of real-time PCR, and the amplicon detection method. None of the other covariates reached statistical significance. The results are reported as relative DOR, which indicates the diagnostic performance of hPCR in studies sharing each of the above-mentioned characteristics relative to its performance in studies lacking the same characteristics. The studies using at least two M. tuberculosis culture media yielded DOR values approximately six times higher than those using one culture medium (relative DOR, 5.99; IC, 1.19 to 30.27; P = 0.031). The relative DOR of studies applying real-time PCR was about 16 times higher than that of the studies using conventional PCR (relative DOR, 16.44; IC 1.82 to 148.41; P = 0.013). The employment of any hybridization method for amplicon detection was also associated with better accuracy values with respect to studies using UV transillumination of electrophoresis gels (relative DOR, 5.58; IC, 1.40 to 22.20; P = 0.014). The multivariate model including all these variables confirmed the association of real-time PCR use with the DOR, while the quality of the reference test and the amplicon detection method no longer had an impact on accuracy (Table 3).

TABLE 3.

Effects of the study characteristics on estimates of DOR as determined by multiple regression analysis^a

Variable	Relative DOR (95% CI)	P
At least two culture media used	2.44 (0.51-11.59)	0.26
Any hybridization method	3.49 (0.16-14.59)	0.09
Nested PCR	0.72 (0.14-3.63)	0.69
Real-time PCR	8.85 (1.82-74.44)	0.04
Threshold (S)		0.93
Intercept		0.01

^aThe coding used in multiple regression analysis was as follows: for M. tuberculosis culture, at least two media used, 1; no information on culture media used, 1 (not reported in the table); one culture medium used, 0; for hPCR type, nested, 1; real-time PCR, 1; “conventional” PCR, 0; for amplicon detection method, any hybridization method, 1; gel electrophoresis plus UV, 0.

Evaluation of both the Egger's (regression coefficient, 0.23; P = 0.72) and the Begg's (P = 0.43) tests did not show evidence of publication bias. Furthermore, the visual inspection of the funnel plot did not reveal the presence of asymmetry (Fig. 4).

FIG. 4.

Funnel plot with pseudo-95% confidence limits. lnDOR, logarithm of DOR. Each circle represents a study in the meta-analysis, while the line in the center represents the summary value of lnDOR.

Posttest probability of pulmonary TB.

The changes in pulmonary-TB likelihood after hPCR performance are depicted, according to all pretest probabilities, in Fig. 5. The top curve portrays the positive predictive values, i.e., the probabilities of pulmonary TB after obtaining a positive hPCR result; the bottom curve represents the inverse of the negative predictive values, i.e., the probabilities of pulmonary TB after a negative hPCR result. For example, using hPCR on an AFB-positive sample collected from a patient for whom previous diagnostic information (history taking, clinical examination, imaging, etc.) indicated a probability of pulmonary TB of about 30%, a negative test result would reduce the likelihood of pulmonary TB to about 2%, while a positive result would increase it to about 70%.

FIG. 5.

Predictive values for pulmonary TB after hPCR was carried out on AFB-positive samples. PPV, positive predictive values; 1 − NPV, 1 − negative predictive values. For details, see the text.

DISCUSSION

Our study is an up-to-date meta-analysis of the diagnostic value of hPCR for AFB-positive patients clinically suspected of having pulmonary TB. Adopting M. tuberculosis culture as reference standard, we estimated the sensitivity and specificity rates at approximately 96% and 81%, although their substantial between-study variability limits their value for clinical purposes. A univariate metaregression analysis identified three study issues—the quality of the reference test, the detection method, and use of real-time PCR—that account for some of the observed heterogeneity. However, only the application of real-time PCR displayed an impact on accuracy in both the univariate and the multivariate models, probably through an increase in test specificity (see Table S1 and Figure S3 in the supplemental material). Real-time PCR is a novel technology that uses built-in automated thermocyclers and fluorimeters that detect fluorescence emitted during the hybridization reaction at the end of any PCR cycle. In addition to its speed in giving results, real-time PCR minimizes the risk of carryover contamination, since both reaction and detection occur in a single tube that remains sealed during the whole PCR run (16). This may lower the false-positive rate, a key factor in determining DOR variability in our review (Fig. 3).

The physical containment of amplification products appeared to be more effective than their enzymatic digestion by means of uracil DNA N-glycosylase (use of dUTP-UNG) (P > 0.83) (data not shown). Unfortunately, our review included only three studies employing real-time technology, and none of them incorporated an amplicons sterilization step, preventing us from analyzing the combined effects of the two methods for limiting the chances of contamination.

The quality of the reference test for pulmonary-TB diagnosis and the use of any hybridization method (instead of agarose gel electrophoresis and ethidium bromide DNA staining) did not demonstrated a significant impact on accuracy when analyzed in the multivariate model, although they are also expected to positively influence specificity (16, 24, 40). None of the remaining variables showed a significant effect in the current regression models, even those found to be associated with accuracy in previous meta-analyses. The reasons for these discrepancies are in part related to the performance characteristics of hPCR with AFB-positive samples: the reported positive effect of multicopy target IS6110 was undetectable because it is due to an increase in sensitivity values. Also, part of the positive results obtained with nested PCR could be explained by higher contamination rates (reamplification of the amplicons during nested PCR requires the opening of vials) rather than by better sensitivity for smear-negative TB samples (3, 22, 39). The uneven yield of M. tuberculosis culture in different bronchial specimens and their variable proportions in different studies could account for discordant results regarding the type of specimen (7, 24, 33, 45), and only future investigations of the diagnostic performance of both hPCR and M. tuberculosis culture with each type of respiratory specimen might help to clarify this issue. Finally, neither study blinding nor other study design features showed an impact on accuracy. This was probably due to the observed high proportions of unreported items, which could conceal either true methodological flaws or poor reporting of a methodologically sound study, able to influence accuracy in opposite directions.

With respect to the diagnostic value of hPCR in the evaluation of patients with suspected pulmonary TB, we observed that, because of its very high sensitivity with AFB-positive samples, the test can be confidently used to “rule out” pulmonary TB in AFB-positive patients (Fig. 5). Thus, particularly in settings where opportunistic infections are a concern, a negative-inhibitor-free hPCR in patients with AFB-positive smears and suggestive clinical and radiographic findings should direct suspicion toward an NTM lung disease (11). The more limited gain in likelihood of pulmonary TB after a positive result seems to reduce its application as a confirmatory test in these cases. This is particularly relevant, since the “clinical suspicion,” when analyzed as a diagnostic test for TB, demonstrates a low positive predictive value, as well (26). The tendency of both clinical suspicion and hPCR to overestimate the probability of pulmonary TB might lead to erroneous initiation of therapy, isolation, and contact investigation. However, the high false-positive rates of a number of studies included in our meta-analysis were probably related to the enrollment of samples from patients under treatment for TB. We hypothesized that some retrospective collections included samples from treated patients, probably submitted to the laboratory for monitoring contagiousness, among M. tuberculosis culture-negative samples. Five studies (14%) applied discrepant analysis, a statistical ploy that, by attempting to correct the errors hidden among conflicting results of hPCR and culture, could lead to an overestimation of hPCR accuracy (28). We decided to include only “uncorrected” results and to analyze the possible effect of the presence of samples obtained from treated patients, but the unavailability of treatment data from 57% of the studies prevented us from drawing conclusions by means of metaregression.

This review has some potential limitations. Our accuracy estimates were affected by the modest quality of the included articles, in part due to the fact that the vast majority of the studies were performed from a laboratory perspective. Forty-nine percent of them aimed at developing new procedures for optimizing the different steps of the hPCR assay, while none of the remaining studies, designed to determine test performance, provided evidence about diagnostic thinking, therapeutic choice, or patient outcome (50). The main drawback of our meta-analysis was the recruitment of an inadequate sample population in many primary studies, which produced high percentages of pseudo-false-positive results, thus affecting the generalizability of our review.

The metaregression analysis could explain only a small part of the observed heterogeneity in accuracy estimates. The great differences in laboratory techniques and in methodological quality across studies may account for this. Because our analysis unit was the single study, our sample size was only 35 (despite the inclusion of 2,152 AFB-positive specimens), resulting in limited power to detect significant effects of variables. Furthermore, the impacts of several design features on the magnitude of the DOR could not be adequately examined because of incomplete information.

In conclusion, contrary to the increasing importance of NAATs for differential diagnosis of patients with suspected mycobacterial infection, this meta-analysis demonstrated that the scientific background for the use of hPCR on AFB-positive patients is quite poor and that the clinical application of this test should be limited to the exclusion of the diagnosis of TB. Although the high sensitivity prevents possible delay or missing of a diagnosis, thus avoiding the subsequent risk for progressive disease, the observed low specificity could result in unnecessary exposure of patients to potentially toxic and expensive therapy and in unnecessary steps to minimize transmission. Real-time platforms, combining amplification and detection in a single run, seem to offer advantages over conventional assays. The employment of hPCR should be evaluated in well-designed clinical trials, with particular regard to the enrollment of an adequate study population. Mycobacterial-culture results for each sample/patient analyzed, either in the TB or in the non-TB group, should be specified in these reports.

hPCR, relatively inexpensive compared to commercial kits, could contribute to the improvement of TB management, particularly in low-resource settings.