Leishmania Infection: Laboratory Diagnosing in the Absence of a “Gold Standard”

Abstract

There is no gold standard for diagnosing leishmaniases. Our aim was to assess the operative validity of tests used in detecting Leishmania infection using samples from experimental infections, a reliable equivalent to the classic definition of gold standard. Without statistical differences, the highest sensitivity was achieved by protein A (ProtA), immunoglobulin (Ig)G2, indirect fluorescenece antibody test (IFAT), lymphocyte proliferation assay, quantitative real-time polymerase chain reaction of bone marrow (qPCR-BM), qPCR-Blood, and IgG; and the highest specificity by IgG1, IgM, IgA, qPCR-Blood, IgG, IgG2, and qPCR-BM. Maximum positive predictive value was obtained simultaneously by IgG2, qPCR-Blood, and IgG; and maximum negative predictive value by qPCR-BM. Best positive and negative likelihood ratios were obtained by IgG2. The test having the greatest, statistically significant, area under the receiver operating characteristics curve was IgG2 enzyme-linked immunosorbent assay (ELISA). Thus, according to the gold standard used, IFAT and qPCR are far from fulfilling the requirements to be considered gold standards, and the test showing the highest potential to detect Leishmania infection is Leishmania-specific ELISA IgG2.

Introduction

Leishmaniases are endemic in 88 countries on four continents. In terms of global disease burden, leishmaniases are the third most important vector-borne disease, after malaria and lymphatic filariasis, causing 2.4 million disability-adjusted life years lost and 59,000 deaths in 2001.¹ The causative agents are parasitic protozoa of the genus Leishmania. Visceral leishmaniasis is caused by species of the Leishmania donovani complex: L. donovani in the Palearctic and L. infantum (= L. chagasi²) in both the Palearctic and the Neotropical ecozones. L. infantum is the etiological agent of a widespread serious zoonotic disease that affects both humans and dogs. Dogs are considered the primary peridomestic reservoir host^3,4 and controlling the dog infection rate reduces the incidence in humans.⁵ Sand flies from the genus Phlebotomus spp. in the Palearctic and Lutzomyia spp. in the Nearctic are the proven vectors of human and canine leishmaniases (CaL).

Prevalence and incidence of the canine infection are important parameters to determine the risk and the ways to control this reemergent zoonosis. The estimation of these parameters depends on the reliable identification of infected dogs.⁶ Parasitological examination is a very specific method but lacks sensitivity.⁷ Serological methods are more sensitive but may be less specific and the choice of the cut-off value may not be obvious.⁶ Moreover, seroprevalence underestimates the true prevalence of infection.⁸ Cellular immune response, as measured by lymphoblastogenesis, seems to be specially suited to detect current or previous infection in clinically healthy dogs.^9,10 However, there may be no obvious cut-off. PCR has been confirmed to be a very sensitive and specific technique.^11–13 However, whether PCR is sensitive enough to detect all infected individuals is controversial due to conflicting results and the difficulties to choose the tissue to be sampled.^11,14

The “gold standard” is the method, or composite of methods, giving results that are regarded as unequivocal classifications. Its use is a necessary prerequisite to examine the diagnostic utility of any test. The only true gold standard for classifying an animal as infected is the isolation of infectious agents or unequivocal histopathological criteria. Unfortunately, there is not such a gold standard for Leishmania infection. A way to validate diagnostic tests when lacking a classic gold standard is to take advantage of experimental infections. The strength of this method is that it not only measures the ability of the assay to detect infection, but also allows the kinetics of infection to be followed. In this setting everything is known about every sample collected and about the individual it is collected from. The aim of the present study is to assess the operative validity of several tests used in detection of Leishmania infection using samples from experimental infections, a reliable equivalent to the classic definition of gold standard.

Material and Methods

Samples and subjects.

Six healthy, 9-month-old, intact, female beagle dogs were used. Dogs were housed in indoor kennels (Isoquimen S.L., St. Feliu de Codines, Spain) with windows covered with delthamethrin-sprayed, double anti-mosquito-nets, and according to the Guiding Principles for the Care and Use of Animals, following the guidelines of the institution's ethics committee.

L. infantum strain MCAN/ES/92/BCN-83/MON-1 obtained from a dog that acquired the infection naturally and had not received treatment was passaged through hamsters. Parasites were then grown in Schneider's insect medium (Sigma, St. Louis, MO), supplemented with 20% fetal calf serum (Gibco, Paisley, Scotland) and 25 µg · mL⁻¹ gentamicin (Sigma) at 26°C. Parasites in stationary phase were washed and resuspended in phosphate buffered saline (PBS) at 5 × 10⁷ promastigotes per mL. Finally 1 mL was injected by the intravenous route to every dog.¹⁵

Samples were collected at monthly intervals beginning 3 months before the infection. One year after experimental infection, the dogs were euthanized and infection verified in all dogs at necropsy. Details of clinicopathological, parasitological, and immunological evolution of the infection have been described elsewhere.¹⁶

Enzyme-linked immunosorbent assay (ELISA).

ELISA was performed as described.^17–20 Polyclonal anti-dog immunoglobulin (Ig) G, IgG1, IgG2, IgA, IgM (Bethyl Laboratories, Montgomery, TX) and Protein A (ProtA) (ImmunoPure^® Recomb^® Protein A, Pierce Rockford, IL), which reacts with the Fc-region of Ig from several species including the dog,²¹ all conjugated to horseradish peroxidase and were used individually as secondary antibodies. The reaction was expressed in ELISA units (EU) in relation to a known positive serum, used as a calibrator and arbitrarily set at 100 EU. This calibrator serum, always the same, was included in all plates, and plates with an interassay variation > 10% were discarded. Sera from 32 healthy dogs living in a nonendemic region for CaL and proven to be parasitologically and serologically L. infantum-negative—tested by visualization of L. infantum organisms on bone marrow smears, by Leishmania DNA in blood and bone marrow, and by detection of specific antibodies—were used to set up a standard cut-off, established at mean + 2 SD.²²

Indirect fluorescenece antibody test (IFAT).

The antigen was prepared from promastigotes of the L. infantum zymodeme MON-1 international reference strain MHOM/FR/78/LEM-75, and anti-Leishmania antibodies were detected using rabbit anti-dog IgG conjugated to fluorescein isothiocyanate (Sigma-Aldrich, Inc., St. Louis, MO). Samples were classified positive if promastigote cytoplasmatic or membrane fluorescence was observed at a serum dilution of 1:80 or higher.²³

Quantitative real-time polymerase chain reaction (qPCR) Real-Time PCR.

qPCR was carried out as described.^16,24,25 Briefly, 0.5 mL of peripheral whole blood (qPCR-Blood) or 0.1 mL of bone marrow aspirate (qPCR-BM) were used. DNA was extracted and diluted in milliQ water (1/10 for bone marrow aspirate and 1/5 for blood) and used for the qPCR. Primers (5′-AACTTTCTGGTCCTCCGGGTAG-3′ and 5′-ACCCCCAGTTTCCCGCC-3′) and the TaqMan MGB probe (5′-AAAAATGGGTGCAGAAAT-3′) were designed to target the kinetoplast minicircle DNA of L. infantum. The eukaryotic 18S RNA Pre-Developed TaqMan assay reagents (Applied Biosystems, Carlsbad, CA) were used as an internal reference of canine genomic DNA amplification to normalize L. infantum amplification for differences in DNA content or the presence of inhibitors. Amplification was performed in triplicate in 25 μL reaction mixture (TaqMan Universal qPCR Master Mix; Applied Biosystems). Thermal cycling profile was 50°C for 2 minutes, 95°C for 10 minutes, 40 cycles at 95°C for 15 seconds, and 60°C for 60 seconds. Quantitative analysis of L. infantum DNA amplification was performed by the comparative threshold cycle (Ct) method (2^−ΔΔCt)²⁶ and using a spiked sample with a known number of parasites/well as calibrator, allowing determination of the number of parasites in any qPCR sample, independently of the amount of DNA added or the presence of inhibitors.

Lymphocyte proliferation assay (LPA).

Peripheral blood mononuclear cells (PBMC) were isolated from 6 mL of heparinised venous blood samples using Ficoll–Hypaque (Histopaque 1077; Sigma, St. Louis, MO).²⁷ Viable PBMC were resuspended at 2 × 10⁶ cells per mL in RPMI-1640 medium (Gibco, Paisley, UK) supplemented with 10% volume/volume heat-inactivated fetal calf serum, 10 mM N-2-hydroxyethylpiperazine-N0-2-ethanesulfonic acid (Hepes buffer), penicillin (100 U/mL) and streptomycin (100 µg/mL). PBMC were cultured in triplicate in flat-bottomed 96-well-microtitre plates at a density of 2 × 10⁵ cells per well in the absence or presence of leishmanial soluble antigen (LSA) or mitogen. Each well contained 100 µL of cell suspension plus 100 µL of either 20 µg/mL LSA, 10 µg/mL phytohemagglutinin (PHA), or medium alone. Cultures were incubated at 37°C in 5% CO₂ for 3 days (control medium alone and PHA) or 6 days (control medium alone and LSA) and pulsed during the last 18 hours of culture with 10 μM of 5-bromodeoxyuridine (BrdU). Cell proliferation was assessed using a non-radioactive ELISA technique, according to the manufacturer's instructions (Cellproliferation ELISA, BrdU colorimetric; Boehringer-Mannheim, Germany).¹⁶

PBMCs from 20 healthy dogs that were Leishmania-negative—tested by visualization of L. infantum organisms on bone marrow smears, by Leishmania DNA in blood and bone marrow, by detection of specific antibodies, and by Leishmanin Skin Test—were cultured in triplicate under the same conditions described above to establish a standard cut-off.¹⁶ This was the mean difference between LSA wells minus medium alone wells plus 2 SD.²² Stimulation with PHA was used as a positive control. PHA proliferation values between 1.12 and 1.55 optical densities (95% confidence interval) for the mean difference between PHA and medium alone were considered positive.¹⁶

Data analysis.

Statistical analysis was performed using JMP version 7.0.1 (SAS Institute, Cary, NC). Receiver operating characteristics (ROC) curve analysis was performed using MedCalc 9.5.1 (MedCalc Software, Mariakerke, Belgium). Confidence intervals (CI) for sensitivity, specificity, and overall accuracy were produced with the Wilson score method²⁸ to avoid exceeding the lower and upper limits acceptable for CI.²⁹ Positive and negative likelihood ratios CI were calculated with the method described by Simel et al.³⁰ A significance statistical level α = 0.05 was used, employing a rough false discovery rate (rFDR) to correct for multiple simultaneous comparisons.³¹

Results

Time to detect infection.

The mean time to detect infection differed significantly among tests (analysis of variance; P < 0.0006), ranging from a mean of 53 days for LPA to a mean of 273 days for IgA. Post-hoc multiple comparison analysis (Tukey-Kramer Honestly Significant Difference Test; P < 0.05) showed that two groups of means existed differing between but not within them. The first group, with the fastest tests to detect infection, contained LPA, IgG, IgG2, qPCR-Blood, ProtA, IgG1, and qPCR-BM; and the second group encompassed IgG, IgG2, qPCR-Blood, ProtA, IgG1, qPCR-BM, IFAT, IgM, and IgA (Table 1).

Table 1.

Descriptive statistics and post hoc Tukey-Kramer HSD analysis for time to detect infection, expressed in days

	LPA	IgG	IgG2	qPCR-Blood	ProtA	IgG1	qPCR-BM	IFAT	IgM	IgA
Mean	53	117	119	132	168	180	207	247	250	273
SD	35	62	62	31	76	143	128	45	101	89
95% CI	16–90	52–182	54–184	99–165	88–248	29–330	73–340	200–294	144–356	179–367

Groups of means that are not statistically different from each other are indicated by a solid grey horizontal bar.

Measures of performance.

Measures of performance for the different tests evaluated are shown in Table 2. Sensitivity, specificity, and overall accuracy differed significantly among tests (Cochran's Q; P < 0.0001 for all three tests). The highest sensitivity was achieved by ProtA with 0.72, although multiple simultaneous comparisons showed that it did not differ statistically from IgG2, IFAT, LPA, qPCR-BM, qPCR-Blood, and IgG (rFDR; P > 0.0256). The highest specificity was achieved simultaneously by IgG1, IgM, IgA, qPCR-Blood, and IgG with 0.99, although they did not differ statistically from IgG2 and qPCR-BM (rFDR; P > 0.0256). The highest overall accuracy was obtained by qPCR-BM with 0.78, though multiple simultaneous comparisons showed that it did not differ statistically from IgG2, ProtA, qPCR-Blood, IgG, and IFAT (rFDR; P > 0.0256).

Table 2.

Measures of diagnostic performance

	IFAT	ProtA	IgG2	IgG1	IgG	IgA	IgM	LPA	qPCR-Blood	qPCR-BM
Sensitivity	0.63	0.72	0.65	0.34	0.47	0.3	0.23	0.6	0.5	0.58
95% CI	0.43, 0.79	0.57, 0.83	0.52, 0.77	0.19, 0.53	0.32, 0.63	0.15, 0.5	0.1, 0.46	0.49, 0.71	0.36, 0.64	0.34, 0.79
Specificity	0.82	0.78	0.99	0.99	0.98	0.99	0.99	0.56	0.99	0.98
95% CI	0.56, 0.94	0.52, 0.92	0.88, 1	0.88, 1	0.86, 1	0.88, 1	0.88, 1	0.35, 0.75	0.88, 1	0.83, 1
Accuracy	0.68	0.73	0.76	0.54	0.62	0.52	0.47	0.59	0.65	0.78
95% CI	0.56, 0.8	0.63, 0.83	0.68, 0.84	0.45, 0.63	0.53, 0.71	0.43, 0.61	0.38, 0.57	0.5, 0.68	0.56, 0.73	0.67, 0.89
PPV	0.89	0.91	0.99	0.98	0.99	0.98	0.97	0.75	0.99	0.97
95% CI	0.78, 1.01	0.84, 0.99	0.96, 1.02	0.93, 1.03	0.95, 1.02	0.92, 1.04	0.9, 1.05	0.64, 0.85	0.96, 1.02	0.88, 1.06
NPV	0.48	0.45	0.57	0.41	0.42	0.4	0.38	0.39	0.46	0.7
95% CI	0.3, 0.66	0.28, 0.63	0.45, 0.69	0.31, 0.51	0.31, 0.54	0.3, 0.5	0.28, 0.47	0.26, 0.53	0.35, 0.57	0.55, 0.85
LR+	3.54	3.23	48.24	24.82	29.43	22.01	17.33	1.36	37	29
95% CI	1.23, 10.17	1.34, 7.77	3.06, 760.5	1.55, 396.34	1.86, 464.62	1.37, 352.65	1.07, 279.83	0.9, 2.04	2.34, 585.33	1.83, 460.12
LR−	0.46	0.36	0.35	0.67	0.53	0.71	0.78	0.72	0.51	0.43
95% CI	0.29, 0.72	0.23, 0.58	0.26, 0.48	0.57, 0.79	0.43, 0.66	0.61, 0.83	0.68, 0.88	0.48, 1.07	0.41, 0.63	0.27, 0.68
DOR	7.78	8.85	136.71	36.85	55.12	30.91	22.32	1.9	73	67.67
95% CI	4.03, 15.01	3.64, 21.55	66.6, 280.64	17.94, 75.68	26.85, 113.1	15.02, 63.6	10.8, 46.14	0.33, 11.01	35.71, 149.2	31.18, 146.8
Youden's	0.45	0.49	0.64	0.32	0.46	0.28	0.22	0.16	0.49	0.56

Maximum positive predictive value (PPV) was obtained simultaneously by IgG2, qPCR-Blood, and IgG with 0.99 although all tests evaluated were above 0.9, except IFAT and LPA. Maximum negative predictive value (NPV) was achieved by qPCR-BM with 0.70, followed by IgG2 with 0.57; all the other tests were below 0.50.

The best positive likelihood ratio (LR+) was obtained by IgG2 with 48.24, followed by qPCR-Blood with 37.00; all other tests were also above 10.00 except IFAT, ProtA, and LPA. The best negative likelihood ratio (LR−) was achieved by IgG2 with 0.35, but all tests showed ratios greater than 0.1. The highest diagnostic odds ratio (DOR) was achieved by IgG2 with 136.71; all other DOR were below 100. Finally, IgG2 showed the highest Youden's index with 0.64.

ROC curve analysis.

The area under the ROC curve ranged from 0.96 for IgG2 to an area of 0.56 for LPA. The area was different from 0.5 for all tests except for LPA (P = 0.2587). Multiple pairwise comparisons showed four groups of areas differing between but not within them (rFDR; P > 0.0256). The first group was composed solely of IgG2; thus, its area was the greatest among all. The second group contained IgG, IFAT, ProtA, IgA, and qPCR-BM. The third group encompassed IFAT, ProtA, IgA, and qPCR-BM. Finally the fourth group embraced qPCR-BM, qPCR-Blood, IgG1, IgM, and LPA (Table 3).

Table 3.

ROC curve analysis

	LPA	IgM	IgG1	qPCR-Blood	qPCR-BM	IgA	qProtA	IFAT	IgG	IgG2
Area ROC curve	0.56	0.66	0.73	0.75	0.79	0.81	0.86	0.885	0.89	0,96
Standard error	0.06	0.05	0.05	0.05	0.07	0.04	0.04	0.043	0.03	0.02
95% CI	0.46, 0.65	0.57, 0.75	0.64, 0.81	0.66, 0.83	0.65, 0.90	0.72, 0.87	0.76, 0.93	0.77, 0.95	0.82, 0.94	0.90, 0.99
Significance level	0.2587	0.0017	0.0001	0.0001	0.0001	0.0001	0.0001	0,00	0.0001	0.0001

Groups of means that are not statistically different from each other are indicated by a solid grey horizontal bar.

Discussion

A plethora of tests are available to improve diagnostic decision-making in Leishmania infection. A gold standard criterion must be present which allows discrimination of patients into two groups: one with infection and the other without infection. Ideally there should be no other difference between patients that may influence the test result.³² To the authors’ knowledge this is the first study using a real gold standard, experimental infection, to assess the accuracy of tests used to diagnose Leishmania infection.

The time a test takes to detect infection is especially important for establishing prompt treatment. In this sense the better tests are LPA, IgG, IgG2, qPCR-Blood, ProtA, IgG1, and qPCR-BM, without statistical differences. However, LPA is useless from the point of view of diagnosing infection because, as explained below, its area under the ROC curve does not differ from 0.5.

The usual measures used to compare the performance of diagnostic tests are sensitivity, specificity, and, to a less extent, accuracy. Sensitivities using our gold standard model range from 0.72 for ProtA to 0.23 for IgM, and the statistical analysis includes the best performing group ProtA, IgG2, IFAT, LPA, qPCR-BM, qPCR-Blood, and IgG. Values are lower than those found in published reports, but sensitivities can only be compared when the same gold standard is used. Furthermore, when using our gold standard, the two most usual surrogates for gold standard in leishmaniasis, IFAT and qPCR, exhibit sensitivities at most equal to some ELISA-based tests. Specificities of the tests used in the present study are high, not different from 1 (95% CI) except for IFAT, ProtA, and LPA. Finally, accuracy ranges from 0.78 for qPCR-BM to 0.47 for IgM, with the most performing group composed of qPCR-BM, IgG2, ProtA, qPCR-Blood, IgG, and IFAT.

ROC curves are particularly useful for comparing the performance of tests. If one test shows a significantly larger area, then it is a better test for evaluating the outcome; if there is no significant difference, then the test does not differ in its ability to predict the outcome.³³ Our results show that the test having the greatest, statistically significant, area under the ROC curve is Leishmania-specific IgG2 ELISA, so this test must be considered the better test for evaluating Leishmania infection. On the other hand, LPA must be considered a toss-up because its area does not differ from 0.5. According to the areas under the ROC curve, IgG2 can be classified as showing high accuracy; IgG, IFAT, ProtA, IgA, qPCR-BM, qPCR-Blood, and IgG1 as showing moderate accuracy; IgM, low accuracy; and LPA useless.³⁴ The Youden's index is another commonly used single summary measure of overall diagnostic effectiveness that also combines sensitivity and specificity. This index, related to the ROC curve, shows again that the best test is IgG2 and the worst LPA.

Sensitivity and specificity of a test are the first-and-mandatory criterion for judging its usefulness, and of interest to public-health policymakers; however, these values offer poor translational information, not being enough to make decisions on a patient. Stated alternatively, the sensitivity and specificity—or its area under the ROC curve or Youden's index—of a test are population measures that look backward at results gathered over time,³⁵ and looking at them it is impossible to conclude whether a patient has a certain condition. The clinician is not interested in population measures, but he faces the question: What is the probability of disease in individuals with a positive or negative test? In this sense, PPV and NPV are usually of keen interest because they address this question providing clinically relevant basis on which to compare the performance of the tests. The highest PPV are obtained by IgG2, qPCR-Blood, and IgG, all of them with 0.99, thus a positive in any one of these tests indicates almost certain infection. On the other hand, all evaluated tests showed lower NPV, qPCR-BM being the best with 0.70. Unfortunately, PPV and NPV are not stable characteristics of diagnostic tests as they are dependent on the prevalence of disease. Thus, the generalizability of PPV and NPV from one population to another is limited because of this dependence, even if test accuracy within the diseased and the non-diseased subgroups is constant across populations.

Another tool has been proposed with some practical advantages for the assessment and use of diagnostic tests: the likelihood ratio (LR). In LR sensitivity and specificity are usefully combined, but the LR goes beyond in the sense that LR deals precisely with comparing odds. Thus, the LR tells how many times more—or less—frequent is the result of a test in patients with the disease, compared with people without the disease. Furthermore, LR does not depend mathematically on prevalence, thus being more portable. In fact, the LR quantifies the increase in knowledge of the presence of the disease through the application of the diagnostic test. In our case, the test with the best LR+ and LR− is IgG2 with 48.24 and 0.35, respectively. By convention,^32,36 marked changes in prior disease probability can be assumed in LR+ exceeding 10.0 and LR− below 0.1; and 2.0 and 0.5 have been suggested as the minimally useful values for LR+ and LR−, respectively. Thus, all tests evaluated are more or less useful according to their LR+ except LPA; and regarding LR−, IgG2, ProtA, qPCR-BM, and IFAT are the only useful tests. A unique advantage of LR is that using the Bayes’ theorem they adapt the sensitivity and specificity to individual patients. Thus, while the prevalence—pretest probability—of Leishmania infection in ecoregions around the Mediterranean basin is 67%,⁸ a positive result in IgG2 test increases this pretest probability by 32.99% giving a posttest probability of 99.99%; in turn, a negative result decreases the pretest probability by 25.26%, giving a posttest probability of 41.74%.^36,37 In contrast, for example, a positive or negative IFAT implies a change of only 21% and 18%, respectively. Finally, the greater DOR, a single metric to combine LR+ and LR−, is obtained by an IgG2 that almost doubles the second one.

To sum up, LPA is very useful for studying cellular mediated immune response,¹⁶ but it is useless from a diagnostic point of view. The commonly used surrogate of gold standard for Leishmania infection IFAT³⁸ is far from satisfying this criterion, and must be considered only a second choice for detecting Leishmania infection. Maybe the best scenario for qPCR is in determining the evolution of parasite load in the evaluation of drugs or vaccines²⁴ or as an adjunct to serological test. Among ELISA tests, IgM performs poorly and its role in leishmaniasis remains to be elucidated; IgA is useful as a means of supporting the clinical assessment during treatment in clinical practice²⁰ and in relation to the dissemination of the parasite,³⁹ but only shows moderate accuracy as a diagnostic tool; finally, IgG isotypes are those showing the greatest interest in diagnosing Leishmania infection, especially IgG2.

In conclusion, in clinical practice the aim is to have a test to accurately identify the patient who has a certain condition. According to our gold standard model, experimental infection, the best suited test to detect individuals harboring Leishmania infection and to provide clinically useful information is ELISA based on polyclonal IgG2.