Molecular Tools for Diagnosis of Visceral Leishmaniasis: Systematic Review and Meta-Analysis of Diagnostic Test Accuracy

Abstract

Molecular methods have been proposed as highly sensitive tools for the detection of Leishmania parasites in visceral leishmaniasis (VL) patients. Here, we evaluate the diagnostic accuracy of these tools in a meta-analysis of the published literature. The selection criteria were original studies that evaluate the sensitivities and specificities of molecular tests for diagnosis of VL, adequate classification of study participants, and the absolute numbers of true positives and negatives derivable from the data presented. Forty studies met the selection criteria, including PCR, real-time PCR, nucleic acid sequence-based amplification (NASBA), and loop-mediated isothermal amplification (LAMP). The sensitivities of the individual studies ranged from 29 to 100%, and the specificities ranged from 25 to 100%. The pooled sensitivity of PCR in whole blood was 93.1% (95% confidence interval [CI], 90.0 to 95.2), and the specificity was 95.6% (95% CI, 87.0 to 98.6). The specificity was significantly lower in consecutive studies, at 63.3% (95% CI, 53.9 to 71.8), due either to true-positive patients not being identified by parasitological methods or to the number of asymptomatic carriers in areas of endemicity. PCR for patients with HIV-VL coinfection showed high diagnostic accuracy in buffy coat and bone marrow, ranging from 93.1 to 96.9%. Molecular tools are highly sensitive assays for Leishmania detection and may contribute as an additional test in the algorithm, together with a clear clinical case definition. We observed wide variety in reference standards and study designs and now recommend consecutively designed studies.

INTRODUCTION

Visceral leishmaniasis (VL), or kala-azar, is a vector-borne disease that is caused by the protozoan parasites Leishmania donovani and L. infantum. The disease is transmitted to humans by the bite of infected phlebotomine sandflies. VL is a neglected tropical disease with an estimated 200,000 to 400,000 new cases and 20,000 to 40,000 deaths annually (1). More than 90% of all VL cases occur in Bangladesh, Brazil, India, Ethiopia, Sudan, and South Sudan. The clinical picture consists of fever, weight loss, fatigue, and general weakness; patients may present with enlarged lymph nodes, hepatomegaly, and splenomegaly. As VL is a fatal condition when left untreated and treatments have high toxicity, a diagnostic test that is both highly sensitive and specific is required. The gold standard for diagnosing VL is mainly the demonstration of parasites by microscopic examination of cultures of splenic aspirates. Splenic aspirates can be associated with hemorrhage, and the process should be carried out only with access to surgical facilities. For this reason bone marrow and lymph node aspirates are commonly taken for parasitological diagnosis. The specificity of these methods is high, but the sensitivity varies depending on the type of specimen, i.e., approximately 93 to 99% for the spleen, 53 to 86% for bone marrow, and 53 to 65% for lymph (2). In addition, the sensitivity and specificity of parasitological testing also depend on the ability of the technician reading the preparations.

Several serological tests are available, among which the direct agglutination test (DAT) and rK39 antigen-based test are used in clinical practice. Two meta-analyses that compared the DAT and the rK39 strip test found similar sensitivities of 94 to 95% and specificities of 86 to 91% (3, 4). One of the major drawbacks of serological tests is the fact that they cannot be used to detect relapse cases because antibodies remain present long after clinical cure (5, 6) and many people from areas of endemicity also have antibody titers due to high exposure or asymptomatic infections. Therefore, a strict clinical criterion of suspicion, including prolonged fever (more than 2 weeks), splenomegaly, and weight loss, must be adhered to.

Molecular-based methods have been proposed as highly sensitive tools for parasite detection. Many PCR-based assays have been developed, using several target sequences and different clinical specimens. Other molecular tools are nucleic acid sequence-based amplification (NASBA) and loop-mediated isothermal amplification (LAMP). Here, we performed a systematic review of the diagnostic accuracy of the nucleic acid-based methods available for VL diagnosis.

MATERIALS AND METHODS

Study scope and definition of the gold standard.

We assessed the accuracy of molecular methods for diagnosing primary VL and thus excluded patients with treatment failure (i.e., no initial cure or reappearance of clinical symptoms). The gold standard was defined as a compatible clinical picture in combination with the demonstration of parasites and/or positive serology. Clinical signs were a persistent systemic infection, including prolonged fever (≥2 weeks); general weakness and weight loss; and presence of hepatomegaly, splenomegaly, or enlarged lymph nodes. Demonstration of parasites was defined as positive in microscopic analysis of direct smears of splenic, bone marrow, or lymph aspirates and/or culture of blood, splenic, or bone marrow aspirates. A positive treatment response without confirmation by other tests was not included in the gold standard.

Search for and inclusion of studies.

We performed an electronic search in the MEDLINE and Embase databases (details of the electronic search are available on request) on 25 June 2012, and all articles before that date were searched. The inclusion criteria were (i) use of a molecular technique for diagnosing visceral leishmaniasis, (ii) use of human clinical samples, (iii) comparison with the reference standard according to our definition of the gold standard, and (iv) the ability to complete a 2-by-2 contingency table from the results. Articles were initially screened by title and abstract. After exclusion of ineligible articles, we read the full text of the remaining articles and applied the inclusion criteria again. Study inclusion was performed independently by C. M. de Ruiter and C. van der Veer. In case of disagreement, E. R. Adams was referred to for a final decision. The reference lists of included studies were checked for additional studies. Unpublished data were sought by checking conference reports and contacting experts.

Data extraction.

From each study, the following items were extracted: (i) the molecular test type; (ii) the readout method of the test; (iii) the sample type; (iv) the target gene of the test; (v) the infecting species of Leishmania; (vi) the country where patients originated; (vii) the gold standard used; (viii) the study design, which was called “consecutive” for equally suspected patients or “case-control” if known cases and controls (healthy controls and patients with other known infections) were selected; and (ix) the numbers of true positives, true negatives, false positives, and false negatives.

When a study reported a combination of early-phase results and a clinical evaluation, we extracted the results of the clinical evaluation. When there was a case mix of people with clinical symptoms and asymptomatic persons, we excluded the asymptomatic persons. When it was known that a person had suffered a previous VL infection (i.e., relapse cases), the person was excluded. When the study population consisted of HIV as well as non-HIV patients, data were extracted for these groups separately. When a case-control study reported healthy controls from areas of nonendemicity, as well as from areas of endemicity, and/or a control group consisting of people with other diseases, only the results for the healthy controls from areas of endemicity and/or the other-disease control group were reported, as they were considered a more accurate reflection of specificity.

We used the Quadas-2 (quality assessment of diagnostic-accuracy studies) tool for quality assessment of the included studies (7). In the assessment of reference standard bias, studies using microscopy and/or culture on spleen tissue or a composite reference standard (microscopy and/or culture and/or serology) were judged as having low risk of bias, while studies using solely microscopy and/or culture on bone marrow and/or lymph tissue, or solely serology, were judged as having high risk of bias due to limited accuracy (either sensitivity or specificity) in the case of stand-alone use (8). Serological tests in a composite standard were defined as rapid diagnostic tests: rK39, DAT, and enzyme-linked immunosorbent assay (ELISA) based on rK39.

Data extraction and quality assessment were done independently by C. M. de Ruiter and C. van der Veer. Any discrepancies were resolved by consulting E. R. Adams and M. M. G. Leeflang. For 10% of the included studies, data extraction was also done by S. Deborggraeve to check for errors in the data extraction process.

Statistical analysis and meta-analysis.

For all studies, estimates of sensitivity, specificity, and 95% confidence intervals were expressed in forest plots in Review Manager version 5.1. We used the bivariate logitnormal random-effects meta-analysis model to obtain a summary estimate of sensitivity and specificity and to create summary receiver operating characteristic (SROC) curves. The bivariate method models the logits of sensitivity and specificity in one model and allows correlation between the two. The logit is the natural logarithm of sensitivity (or specificity) divided by 1 minus sensitivity (or specificity). Studies were included in the meta-analysis if at least four studies evaluated the same molecular method in the same sample type.

As diagnostic test accuracy data are expected to be heterogeneous and sensitivity and specificity are expected to correlate with each other, no methods to test for heterogeneity are available. We therefore investigated sources of heterogeneity by adding the following covariates to the model: (i) the target gene, (ii) the infecting species, (iii) the study design, (iv) the gold standard, and (v) the readout method of the test. A covariate was assumed to have a significant effect on the estimates of sensitivity and specificity and thus to explain some of the heterogeneity in the sample if the P value was 0.05. We used xtmelogit in STATA version 10 for the analyses.

RESULTS

Flow of included studies.

The electronic search yielded 1,256 results, 90 of which were taken forward to read the full text (Fig. 1). Articles were excluded at this stage due to (i) absence of a (patient) control group, (ii) reporting a case series, (iii) reporting molecular studies for other purposes, (iv) use of an inappropriate gold standard (e.g., blood culture), (v) reporting diagnostic tools other than molecular techniques, (vi) absence of the use of patient material, (vii) reporting a review study, or (viii) inability to complete a 2-by-2 contingency table. A total of 40 articles were included in the systematic review. Data from 36 articles were included in the meta-analysis. Data from 4 articles could not be included in the meta-analysis because they did not fit the respective subgroups for molecular method and sample type. Data extracted by a third author as a quality assurance procedure was in agreement with the data extracted by the primary authors.

FIG 1.

Flowchart of included studies.

Study characteristics.

The index tests assessed were PCR (n = 28), real-time PCR (n = 2), NASBA (n = 3), and LAMP (n = 2). Studies of HIV-VL-coinfected patients assessed PCR (n = 10) and real-time PCR (n = 1). Studies extracted DNA from buffy coat (n = 16), whole blood (n = 13), or bone marrow (n = 13). Other sample types used were serum (n = 3), oral fluid (n = 2), blood on filter paper (n = 2), bone marrow on filter paper (n = 1), and urine (n = 1).

Diagnostic accuracy of molecular tests and analysis of heterogeneity.

The 40 articles included 59 separate studies (i.e., more than one index test evaluated per article) for which 2-by-2 contingency tables could be completed. Sensitivities ranged from 29% to 100%, and specificities ranged from 25% to 100% (Table 1) (9–48). The pooled estimates for the sensitivity and specificity of PCR are shown in Table 2. The summary ROC curves of PCR in blood and bone marrow samples are shown in Fig. 2. There was no difference in accuracy (sensitivity and specificity) between the various readout methods for PCR (gel electrophoresis, hybridization, oligochromatography, and real time), allowing the results to be pooled in the analysis. There was no difference in accuracy between whole-blood, buffy coat, and bone marrow samples. When the results of the case-control studies were compared with those of the consecutive studies, no difference for sensitivity was found, but the specificity was significantly lower in the consecutive studies (Table 2).

TABLE 1.

Forest plot of statistical analysis in Review Manager 5.1^a

^aTP, true positives; FP, false positives; FN, false negatives; TN, true negatives; BM, bone marrow; OC, oligochromatography; NA, not applicable; SSU, small subunit; ITS, internal transcribed spacer; AOE, area of endemicity.

^bStandard A, microscopy and/or culture; standard B, microscopy and/or culture and/or serology; standard C, serology.

TABLE 2.

Summary estimates for PCR^a

Test and sample type	No. of studies	Pooled sensitivity (95% CIc)	Pooled specificity (95% CI)
Case-control and consecutive studies combined
PCR bloodb	19	93.1 (90.0–95.2)	95.6 (87.0–98.6)
PCR bone marrow	8	95.3 (91.0–97.6)	92.6 (59.3–99.1)
Consecutive studies
PCR bloodb	4	92.3 (88.4–949)	63.3 (53.9–71.8)
PCR bone marrow	5	95.8 (80.0–98.6)	76.4 (46.3–92.4)
HIV-VL coinfection and case-control and consecutive studies combined
PCR buffy coat cells	4	93.1 (83.3–97.3)	96.9 (58.2–99.9)
PCR bone marrow	5	96.6 (59.2–99.8)	96.6 (80.7–99.5)

^aStatistical analysis in STATA version 10, xtmelogit.

^bWhole-blood and buffy coat samples combined.

^cCI, confidence interval.

FIG 2.

SROC curves for PCR in blood and bone marrow samples. Statistical analysis was done in STATA version 10 and xtmelogit. The open symbols represent the results from individual studies. The solid circle and square represent the summary estimates for PCR in blood and bone marrow, respectively. The circled regions around the solid circle and square represent the 95% confidence interval regions around the summary estimate. The confidence intervals for PCR on bone marrow (squares) are larger than those for PCR on blood (circles).

Further analysis was confined to the blood samples, as other subgroups were too small to perform an analysis of heterogeneity. There were no differences in accuracy between subgroups for the infecting species (L. donovani in Asia, L. donovani in East Africa, and L. infantum in Europe and the Mediterranean area). There were no comparable studies for L. infantum in South America for this analysis. There was also no difference between the target genes amplified (18S ribosomal DNA [rDNA], kinetoplast DNA [kDNA], or miniexon). Furthermore, there was no difference in accuracy between studies that used microscopy and/or culture as a reference standard and studies that used a composite reference standard.

Quality assessment of study reports.

The results of quality assessment with the Quadas-2 tool are summarized in terms of risk of bias and concerns regarding applicability (i.e., the extent to which studies are applicable to the research question) (Fig. 3). Of the 40 studies included, 24 were case-control studies, which resulted in a large proportion of studies having high risk of bias and high concern regarding applicability in the domain of patient selection. The conduct and interpretation of the index test did not raise problems, and there were no concerns regarding applicability in all the studies for this domain. A reference standard with high or unclear risk of bias was used in approximately half of the studies (18/40). However, the target condition, as defined by the reference standard, was applicable to our review in all the studies. In 24 out of 40 studies, there were no details about the interval between the index test and the reference standard. In most case-control studies (17/24), the controls did not receive a reference standard or did not receive the same reference standard as the cases (6/7).

FIG 3.

Quadas-2 results showing risk of bias and applicability concerns.

DISCUSSION

Molecular tests have been proposed as sensitive tools for diagnosing VL. In this systematic review we analyzed and summarized data from diagnostic-accuracy studies on molecular tests in primary VL infection. From the available literature, we were able to assess the accuracy of PCR tests in blood and bone marrow samples.

The summary estimates for sensitivity and specificity for both blood and bone marrow samples (general population) were high, and there was no statistically significant difference between the accuracy in whole blood, buffy coat, or bone marrow samples. This means that a simple blood sample would suffice instead of taking more invasive bone marrow samples. There was, however, a significant difference in accuracy between subgroups for case-control and consecutive studies. The consecutive studies still show high summary sensitivities of 92.3% for blood and 95.8% for bone marrow, but specificities are significantly lower at 63% and 76.4%, respectively.

From a methodological point of view, this is to be expected, because the controls in case-control studies are often healthy persons, whereas controls in consecutive studies are in fact suspected patients. The high number of positive PCR results in suspects with a negative reference standard may be explained by a proportion of the false positives being true VL cases when we take into consideration that the gold standard for VL is imperfect and that the sensitivity of PCR is superior to the gold standard (12). In addition, a high proportion of L. donovani carriers in areas of endemicity who might develop prolonged fever and splenomegaly for other reasons may show a false-positive reaction in a VL PCR. This may imply that PCR positivity is a marker of infection rather than of disease (10). There is heterogeneity in the gold standards of the consecutive studies. Of four consecutive studies assessing PCR in blood, one used microscopy on bone marrow, followed by spleen microscopy if negative, and three used a composite reference standard (microscopy and/or serology). Of the five consecutive studies using PCR in bone marrow, the gold standard was microscopy on bone marrow aspirate (n = 3), microscopy on bone marrow and/or spleen aspirate (n = 1), and a composite standard (n = 1). When we consider microscopy performed on bone marrow an imperfect reference standard, this reference test bias could contribute to the low specificity we observed for PCR on bone marrow samples. Our findings on consecutively designed studies are based on a limited number of patients (381 suspected patients for PCR on blood and 570 patients for PCR on bone marrow samples), who were tested with imperfect reference standards. In addition, here we have shown that molecular assays can identify parasites, not only in bone marrow, but also in whole blood and buffy coat, with similar levels of diagnostic accuracy, offering a much safer method of specimen collection than spleen or bone marrow biopsies.

Consecutive studies better reflect the diagnostic situation and are thus of higher methodological quality than case-control studies. We therefore recommend that future diagnostic-accuracy studies use a consecutive design to determine whether our findings about specificity are reproducible and to make valid estimates.

This meta-analysis shows that the molecular methods are very sensitive tools for the detection of Leishmania parasites in blood samples and have the potential to contribute in the diagnostic algorithm of primary VL. Given the low specificity observed in consecutive studies in areas of VL endemicity, the results have to be interpreted in combination with a clear clinical case definition and, potentially, the results of serology tests. Further, when making decisions about the role of PCR in a diagnostic algorithm, we must take into account that the tools are currently restricted to well-equipped laboratories.

As VL diagnostics are known to behave differently in HIV patients, many recent diagnostic-accuracy studies of VL focus on patients with HIV-VL coinfection. In this patient subgroup, we found high pooled estimates for sensitivity, as well as specificity (>98%), though due to limited data, it was not possible to provide summary estimates for case-control studies and consecutive studies separately, and data on whole-blood samples were too few to calculate a summary estimate for this sample type. Although diagnostic accuracy may have been overestimated in case-control studies, considering the limitations of current diagnostic methods in HIV patients, the results for PCR are still promising. Molecular tests are potentially important in this group of patients because of a lower antibody response in HIV-infected patients, which lowers the sensitivity of serological tests (49). We recommend future consecutive studies to further clarify the accuracy of PCR and define its position in the diagnostic pathway in this subgroup of patients, which may be different from that for immunocompetent individuals.

Limitations.

Many studies in our meta-analysis suffer from poor quality. Sixty percent of the included studies had a case-control design; this design is reputed to introduce selection bias, as the cases are confirmed patients and the controls are healthy persons or persons with other confirmed diseases (50). Ideally, this review would be performed only with consecutive studies; however, due to their limited number, this was not possible. For studies focusing on PCR on blood samples, the percentage of case-control studies was 79%, leading to a high bias in patient selection and introducing a considerable constraint on translating the results into practice, as case-control studies are not representative of a true diagnostic setting. As the gold standard for VL is not perfect, reference test bias is an important issue in diagnostic-accuracy studies. In the case of VL, there is a risk of underestimating the specificity of a new test when comparing it to current methods that have low sensitivity and high specificity, such as microscopy of bone marrow or lymph (8). We considered microscopy and/or culture of splenic aspirates and the use of a composite reference (microscopy and/or culture and/or serology) standards with low potential for bias. Quadas assessment showed that 45% of the studies used an inappropriate gold standard when judged against this definition. The analysis of heterogeneity did not show a significant difference between studies that used microscopy and studies that used a composite reference standard (microscopy and/or culture and/or serology), but this was done for all studies combined, and subgroups were too small to allow a comparison between case-control and consecutive studies. It was not possible to analyze the different sample types used for microscopy when comparing the reference tests in the analysis of heterogeneity because of lack of data. Also, conclusions can be made only for conventional PCR followed by electrophoresis in agarose gels, since this was the only method for which the number of evaluation studies was sufficiently high.

Another limitation in comparing diagnostic-accuracy studies for molecular tools is a lack of standardization. This is clearly evidenced by the high number of different protocols presented in Table 1. Protocols vary among laboratories, meaning that various steps involved in performing molecular tests, like the pretreatment of samples, the DNA extraction process, and the use of in-house primer sets, differ between sites. It is therefore important that authors include detailed information in reports on evaluation studies, e.g., the type and volume of sample used for extracting the DNA, the type and volume of solution used for eluting the DNA, and the volume of the DNA extract used in the molecular test. Our study may also suffer from publication bias where only favorable results for molecular diagnostics have been published.

Incomplete reporting of studies was an additional problem encountered during the phases of study selection, data extraction, and quality assessment of the included studies. We recommend the use of the STARD guidelines for reporting future diagnostic-accuracy studies (http://www.stard-statement.org/).

Conclusion.

PCR on blood and bone marrow samples is a highly sensitive method for detecting Leishmania. Our results show that, in immunocompetent persons, the sensitivities for blood and bone marrow are almost equal. The specificity of molecular tests is high in case-control study designs but significantly lower in consecutive studies. PCR positivity should always be interpreted in combination with a standardized clinical case definition and the results of other diagnostic tests.

Molecular tests may have a specific value in diagnosing VL in HIV-infected patients, but large-scale consecutive studies are needed before any recommendation can be made.

Finally, this review highlights the need for further consecutive study designs, standardization of molecular test protocols, and improved reporting of diagnostic-accuracy studies.