Detection of Echinococcus multilocularis by MC-PCR: evaluation of diagnostic sensitivity and specificity without gold standard

Abstract

Introduction

A semi-automated magnetic capture probe-based DNA extraction and real-time PCR method (MC-PCR), allowing for a more efficient large-scale surveillance of Echinococcus multilocularis occurrence, has been developed. The test sensitivity has previously been evaluated using the sedimentation and counting technique (SCT) as a gold standard. However, as the sensitivity of the SCT is not 1, test characteristics of the MC-PCR was also evaluated using latent class analysis, a methodology not requiring a gold standard.

Materials and methods

Test results, MC-PCR and SCT, from a previous evaluation of the MC-PCR using 177 foxes shot in the spring (n=108) and autumn 2012 (n=69) in high prevalence areas in Switzerland were used. Latent class analysis was used to estimate the test characteristics of the MC-PCR. Although it is not the primary aim of this study, estimates of the test characteristics of the SCT were also obtained.

Results and discussion

This study showed that the sensitivity of the MC-PCR was 0.88 [95% posterior credible interval (PCI) 0.80–0.93], which was not significantly different than the SCT, 0.83 (95% PCI 0.76–0.88), which is currently considered as the gold standard. The specificity of both tests was high, 0.98 (95% PCI 0.94–0.99) for the MC-PCR and 0.99 (95% PCI 0.99–1) for the SCT. In a previous study, using fox scats from a low prevalence area, the specificity of the MC-PCR was higher, 0.999% (95% PCI 0.997–1). One reason for the lower estimate of the specificity in this study could be that the MC-PCR detects DNA from infected but non-infectious rodents eaten by foxes. When using MC-PCR in low prevalence areas or areas free from the parasite, a positive result in the MC-PCR should be regarded as a true positive.

Conclusion

The sensitivity of the MC-PCR (0.88) was comparable to the sensitivity of SCT (0.83).

Alveolar echinococcosis is caused by infection with the larval stage of Echinococcus multilocularis mainly in rodent intermediate hosts and also in a variety of aberrant hosts including humans. The adult intestinal tapeworms are found in canids. Alveolar echinococcosis is a silently progressing disease mainly involving the liver, with high mortality if untreated, and it is considered to be among the most serious parasitic zoonotic diseases in humans in the Northern hemisphere (1, 2). In Europe, the red fox (Vulpes vulpes) is the main definitive host of E. multilocularis (3), and voles (e.g. Microtus, Arvicola and Myodes spp.) are the key intermediate hosts (4). Research during the last two decades has shown that the geographical distribution of the parasite in Europe is expanding (5). Although reliable data on the geographical distribution are present, further harmonization of monitoring activities is needed to allow for detailed epidemiological analysis at supranational level (3). However, surveillance for the parasite is expensive and there is a need for a more cost-effective approach to determine the prevalence of the parasite as well as its geographical distribution.

A semi-automated magnetic capture probe-based DNA extraction and real-time PCR test (MC-PCR) has therefore been developed in Sweden. Estimation of the test's characteristics is a challenge when no true gold standard exists, this is especially true when the test used as gold standard does not have a very high sensitivity. The specificity of a test (i.e. the proportion of truly negative samples that are correctly identified as such) can however be evaluated on samples from a negative populations or from populations with a very low prevalence, as done for the MC-PCR in Isaksson et al. (6). The analytical sensitivity – or detection limit – (i.e. the lowest concentration of the substance of interest that the test can detect) can be evaluated on spiked samples, which has been done for the MC-PCR (6, 7). However, the diagnostic sensitivity (i.e. the proportion of truly positive samples that are correctly identified as positive) must be evaluated on samples from naturally infected individuals and preferable on samples from the population where the test is intended to be used (8). Therefore, the number of E. multilocularis eggs that are expected to be present in faecal samples and intestines, or worms expected to be present in intestines from naturally infected individuals needs to be taken into account when estimating the test characteristics, as done, for example, by Deplazes et al. (9) for E. multilocularis in foxes.

Because the national prevalence of E. multilocularis in Sweden is very low, approximately 0.1%, it was not possible to obtain enough positive samples from foxes in Sweden to evaluate the diagnostic sensitivity of the MC-PCR. It was therefore evaluated on 177 samples from naturally infected foxes from a high prevalence area in Switzerland. Using the SCT as the gold standard test, the sensitivity of the MC-PCR was estimated to be 0.88 (95% CI 0.798–0.939) (6). However, as the sensitivity of the SCT is not 1 (10) the estimated sensitivity of MC-PCR is likely biased.

The present study aimed at estimating the sensitivity and specificity of MC-PCR by means of latent class analysis, as this method does not require the definition of a gold standard. Latent class analysis hypothesizes the existence of one or more unobserved (i.e. latent) categorical variables to explain the relationships among a set of observed categorical variables. In the medical diagnosis context, the observed variables are signs, symptoms, or test results (usually dichotomized into a binary classification such as positive and negative), while the latent variable is true status on the disease (11). As a secondary output, the test characteristics of SCT were also estimated.

Materials and methods

Collection of samples

Faecal samples were collected as described elsewhere (6). In brief, a total of 177 foxes shot by hunters during the official hunting season in January/February (n=108) (data set A) and in October/November 2012 (n=69) (data set B) in high prevalence areas in Eastern-Switzerland were used in the evaluation (Table 1).

Table 1.

Results of the analysis with SCT and MC-PCR on data set A (n=108) and data set B (n=69)

	SCT+	SCT−	Total
Data set A
MC-PCR+	50	7	57
MC-PCR−	5	46	51
Total	55	53	108
Data set B
MC-PCR+	32	11	43
MC-PCR−	6	20	26
Total	38	31	69

SCT

The foxes were necropsied at the Institute of Parasitology, University of Zurich, and tested with the SCT (3, 12). Worms were visualized by microscopy and counted. If the sample had more than 100 worms, all the worms were collected and an aliquot was counted to estimate the worm burden of each fox. The SCT is considered to be the gold standard for the diagnosis of E. multilocularis at necropsy, and this is the only method enabling quantitative estimates of the worm burdens.

MC-PCR

After completion of the necropsy, a faecal sample was collected from the rectum of each fox. Three grams from each sample were sent to the Department of Virology, Immunobiology and Parasitology, National Veterinary Institute, Uppsala, Sweden. All samples were stored at −80°C for at least 5 days before being analysed with the MC-PCR. Three millilitres of faecal material was homogenized in 12 ml of buffer using zirconia beads to get the target DNA from the E. multilocularis eggs in solution. The homogenate was centrifuged and the supernatant was transferred to a new tube. After removal of naturally occurring biotin by the use of streptavidin sepharose, a biotinylated DNA hybridization probe was added and a denaturation step facilitated hybridization of the probe to the target DNA to create a probe/target complex. Addition of streptavidin-coated paramagnetic beads covalently bound the biotinylated probe/target complex. The beads were pelletized using a strong magnet between washes and were resuspended in buffer. After a denaturation step to free the target DNA from the capture probe, the beads were pelletized one last time before the supernatant containing the target in solution was transferred to a new tube and used for real-time PCR.

Data analysis

The performance of the two diagnostics tests was assessed by a Bayesian version of latent class analysis, as proposed by Branscum et al. (13). This approach allows for the estimation of the sensitivity and the specificity of the two tests when the true infection status of the tested subjects is unknown. The model assumes that 1) the two tests are conditionally independent given the true (but unknown) infection status, that is, the sensitivity and specificity of one test are independent of the outcome of the other test when used to test the same individuals; 2) the test subjects are divided into two or more groups where the proportion of truly infected test subjects differ; and 3) the test properties are constant throughout these groups.

In a Bayesian analysis, all parameters are given as distributions. Thus, for the test properties and the prevalence within the sub-populations, prior distributions must be specified, reflecting the relevant information about the parameters before the onset of the study. This allows for the inclusion of previous knowledge about the parameters under investigation in the analysis. Prior distributions of the sensitivity and specificity of SCT and specificity of PCR were modelled as Beta(a,b) distributions, whose specific shape parameters a and b were derived based on the most likely value (mode) and the nth percentile of the values found in the literature or suggested by experts, as reported in Table 2. Prior information of the remaining parameters (i.e. sensitivity of MC-PCR and the prevalence in the two sub-populations) was uncertain and it was thus modelled using the Beta (1, 1) distribution, which is uniform for the interval between zero and one (i.e. uninformative priors).

Table 2.

Distributions for prior information of known variables (Se=sensitivity, Sp=specificity)

Parameter	Most likely value (mode)	Percentile	Percentile's value	Beta distribution	Reference
SpSCT	0.9999	1	0.99	Beta(5836, 1.584)	(15)
SeSCT	0.8360	10	0.76	Beta(45.24, 9.68)	(10, 16)
SpMC-PCR	0.9900	5	0.95	Beta(88.28, 1.882)	(6)

The model was implemented in OpenBUGS 3.2.3, which uses a Markov Chain Monte Carlo (MCMC) sampling algorithm to obtain a Monte Carlo (MC) sample from the posterior distribution. For the analysis, the first 5,000 Monte Carlo samples were discarded as a burn-in, and the successive 150,000 iterations were used for the posterior inference. Potential autocorrelation was removed by storing one Monte Carlo sample every 50 iterations. Convergence of the MCMC chains was assessed both by visual inspection of the time-series plots and by computing the Gelman–Rubin convergence diagnostic plots using three MCMC chains with different starting values. Posterior inference was performed by calculating the median and the 95% posterior credible intervals (PCI) of the sensitivity and specificity of the two tests as well as the proportion of infected animals in the two sub-populations. To compare the different parameters in a manner more similar to traditional frequentist statistical methods, Bayesian posterior probabilities (POPR) were calculated and used to decide in favour of or against several hypotheses (e.g. H₀: Se_SCT>Se_MC-PCR). The POPR used to test H₀ was calculated as the proportion of Monte Carlo samples for which H₀ was true.

To investigate whether the specified prior knowledge would have affected the posterior estimates of the parameters, we repeated the analysis using uninformative priors for all the parameters, as well as different levels of certainty in the definition of priors for sensitivity and specificity of SCT and specificity of MC-PCR. The different models (i.e. with different priors) were further compared by means of the deviance information criterion (DIC, smaller is better) (14). Reported results refer to the best performing model.

Prior knowledge of test characteristics

Prior for the diagnostic specificity of the SCT

The SCT is currently regarded as the gold standard and the specificity is considered to be close to 1 (15). When priors were used for SCT, the specificity was assumed to be most likely equal to 0.9999, and at least 0.999 with 99% confidence.

Prior for the diagnostic sensitivity of the SCT

Using data on the analytical sensitivity of the SCT at different worm burdens from the experiment by Karamon et al. (10) and combining these with data on the distribution of the worm burden in naturally infected foxes from the study by Hegglin et al. (16), two values for the diagnostic sensitivity of the SCT was obtained, 0.756 and 0.836, respectively (Appendix). In order not to underestimate the sensitivity for SCT, we used 0.836 as the most likely value and concluded that we were 90% sure (expert opinion of the authors) that the sensitivity was not lower than 0.756 (16).

Prior for the diagnostic specificity of the MC-PCR

The MC-PCR used has been shown to have a high specificity (0.999; 95% CI 0.997–1) when evaluated on samples from a low prevalence area (6). However, in high prevalence areas, the specificity may be lower. It has been reported that only around 10–30% of rodents infected with E. multilocularis have lesions containing protoscoleces, that is, they are infectious (12, 17, 18). A fox eating such an infected but not infectious rodent will have E. multilocularis DNA in the intestine and thereby be test positive in the MC-PCR although the fox is not truly infected. This is probably a rare event as when foxes eat more rodents, the probability that at least one rodent will be truly infectious increases. Furthermore, if a fox eats a non-infectious rodent, it will only excrete DNA for a short period, possibly only a day, whereas if it gets truly infected it will excrete DNA for about 2 months. As the effect of infected non-infectious rodents was considered to be small but not negligible (data not shown), a lower most likely value for the prior probability for the specificity of MC-PCR was used, (0.99) with 95% confidence that the true value was at least 0.95.

Results

The estimated test characteristics of the MC-PCR and the SCT are detailed in Table 3. The sensitivity of the MC-PCR was higher compared to the SCT, although this difference was not statistically significant (POPR=0.14). The specificity of both test was very high; however, it was significantly higher for the SCT than for the MC-PCR (POPR<0.001; Table 3).

Table 3.

Posterior median and 95% posterior credible intervals (PCI) of the sensitivity (Se) and specificity (Sp) of the MC-PCR and SCT

Parameter	Estimate	95% PCI	Null hypothesis (H0)	POPR
SeMC-PCR	0.88	[0.80; 0.93]	SeSCT>SeMC-PCR	0.14
SeSCT	0.83	[0.76; 0.88]
SpMC-PCR	0.98	[0.94; 0.99]	SpSCT<SpMC-PCR	<0.001
SpSCT	0.99	[0.99; 1]

POPR, Bayesian posterior probabilities; PCI, posterior credible intervals.

The posterior estimates of prevalence in the two data sets A and B were 0.58 (95% PCI 0.48–0.67) and 0.71 (95% PCI 0.59–0.82], respectively.

Discussion

This study showed that the sensitivity of the MC-PCR is at least as high as the sensitivity of SCT, which is currently considered as the gold standard. Previous studies by Isaksson et al. (6) and Øines et al. (7) have also shown that the analytical sensitivity was very high. Expressed as the limit of detection (LOD), the analytical sensitivity of the MC-PCR was estimated to be 5.3 eggs (95% CI 2.8–9.8 eggs) (6).

The obvious advantage of using PCR is the possibility to use faecal samples instead of whole intestines. The economical savings of not having to shoot, transport, and perform necropsies on whole foxes in order to be able to do SCT is probably quite large, although to the authors’ knowledge, there are no published data on this. In a recent review, it was estimated that the number of samples analysed per person per 5-day period varied between 50 and 150 for SCT, SSCT, and ISCT, whereas when using the MC-PCR 240 samples could be analysed (3). Furthermore, the zoonotic risk for sampling technicians is expected to be much smaller when handling fox scats instead of whole fox carcasses.

Faecal material used in this study was collected from the intestines of shot foxes and not from fox scats because SCT required investigations of the intestinal contents. When collecting fox scats, it can be expected that they may have been defecated several months earlier. A longer exposure in the environment increases the risk that disintegrated worm tissue, possibly present in the fox scat, would decompose. Eggs are more stable in the environment than disintegrated worm parts (19), the latter are, however, probably very rarely present in the fox scats. Despite the use of intestinal contents in the study, the estimates of sensitivity and specificity of MC-PCR are considered to be similar for fox scats.

PCR has previously been reported to be performed directly on faecal samples usually resulting in problems with inhibition, decreasing the sensitivity (20). One approach to overcome the PCR inhibition is to first concentrate taeniid eggs by flotation and sieving. However, using two tests and a serial interpretation (e.g. both tests have to be positive to be considered a positive test result) usually decreases the sensitivity. The MC-PCR overcomes this by using a ‘fishing method’ where DNA is ‘fished’ using hybridization probes attached to magnetic beads, allowing the use of a large amount of sample.

One disadvantage of the MC-PCR, when compared with a test detecting the worm or worm coproantigen, is that, apart from cases where DNA from disintegrated worms is present in faeces, it will have a low sensitivity to detect prepatent infections (21). Al-Sabi et al. (21) report a lower sensitivity (0.16), when using copro-DNA PCR for prepatent infections (2–29 days post-infection). This is in agreement with the test results obtained on a limited number of samples of prepatent infections (6). For the high patent period, 30–70 days after infection, the sensitivity was 1 and in the low patent period, 71–90 days after infection, the sensitivity was 47% (21). This is in accordance with the number of eggs found in the faeces in the different periods of infection in the same study. Furthermore, DNA-based assays cannot be called quantitative in this context, even though there is a strong negative correlation between worm burden and C_q-value when using real-time-PCR assay.

In addition to evaluating the test characteristics of the MC-PCR, this study also provided an estimate of the diagnostic sensitivity and specificity for the SCT. The sensitivity was estimated to be 0.83, slightly but not significantly lower than the sensitivity for the MC-PCR. Karamon et al. (10) have shown that the sensitivity of the SCT decreases when worm burdens are low. Using a total of 40 intestinal samples, the calculated sensitivity of SCT was 30%, 40%, 60%, and 100% in samples enriched with 2, 5, 10, and 30 E. multilocularis worms, respectively. The same authors also concluded that the LOD when testing naturally infected intestines is probably worse (i.e. higher) than in their investigation. The reasons for this is that Echinococcus worms in naturally infected foxes are localised between the villi and are firmly fixed within the mucosal layer; therefore, isolation of these tapeworms is probably less effective than in samples artificially enriched with tapeworms. Moreover, tapeworms in their study were in very good condition, whereas in routine SCT examinations worms are often deformed and fragmented, which makes the detection more difficult. The worms were also stored in 70% ethanol, which may influence their sedimentation properties in the SCT. Despite the shortcomings of the Karamon study, it was considered useful for obtaining prior estimates for the SCT.

The estimated specificity of the MC-PCR was high (0.98, [95% PCI 0.95–0.99]), and almost as high as the one for SCT (0.99, [95% PCI 0.99–1]). Isaksson et al. (6) showed that there is no cross reactivity for other taenia in the MC-PCR. When the test was used in areas with a very low prevalence of E. multilocularis (approximately 0.1%), analysis of 2158 fox scats showed that the specificity was at least 0.999% (95% CI 0.997–1) (6). The slightly lower specificity found in this study could have several explanations. One hypothetical explanation could be that foxes having consumed an infected but not infectious rodent could have E. multilocularis DNA in the intestine and thereby be positive in the MC-PCR although not truly infected. But if the test is used to document freedom from E. multilocularis, it does not matter if the fox is truly infected or not as both cases will reflect the presence of the parasite in the environment.

Latent class analysis allows to estimate test sensitivity and specificity without making reference to a gold standard. However, underlying assumptions must be fulfilled in order to produce reliable estimates. A key assumption is that the results of the diagnostic tests are independent given the disease status. Given that RT-PCR targets the parasite's eggs and SCT the adult worms, it might be reasonable to assume that the two tests are conditionally independent. A second assumption is that the proportion of infected subjects in the two sub-populations differs. According to Toft et al. (22), a small difference in population prevalence (i.e. 10%) results in wider credibility intervals of the posterior estimates of tests accuracies than when larger differences exist between the population prevalences. It can also result in an overestimation of the sensitivity and an underestimation of the specificity. However, the impact of a small difference in disease prevalence is stronger when they are both low, because it means that only few infected subjects are available for the calculations. In our cases, the prevalences in subset A and B were not far apart (0.58 and 0.71, respectively), but they were both high. Furthermore, from posterior inference, the prevalence in A was significantly lower than prevalence in population B in almost all cases (POPR=0.06). The last assumption is that the test properties are constant across the sub-populations. It has been hypothesized that the specificity of MC-PCR might differ between high- and low-prevalence areas. However, in the current study, both sub-populations A and B had high prevalence; therefore, we can assume that the tests performed the same in both groups.

Bayesian analysis allows to incorporate previous knowledge in the form of prior distributions. This not only can help drive the posterior estimates of the parameters, but it can also have a strong influence on the results when the amount of data is scarce or not robust. To investigate whether the available prior knowledge would have affected the posterior estimates of the sensitivity and specificity of MC-PCR and SCT, we repeated the analysis including uninformative priors for all the parameters, as well as using different levels of certainty in our prior distributions. All the tested models produced consistent posterior estimates (data not shown), but when using uninformative priors wider PCIs were obtained. Furthermore, the DIC favoured the selected model; therefore, we can assume that the posterior estimates of sensitivity and specificity of the two diagnostic tests were not misguided by the specified prior information.

Conclusion

In conclusion, this study shows that the sensitivity of the MC-PCR is high [0.88 (95% PCI 0.80–0.93)] and comparable with the sensitivity of the more laborious SCT [0.83 (95% PCI 0.76–0.88)]. This study also shows that the specificity of the MC-PCR is very high [0.98 (95% PCI 0.94–0.99)]. However, when used in low prevalence areas or areas free from the parasite, the specificity has been shown to be even higher [0.999 (95% CI 0.997–1)] (6) and in such areas, a positive result in the MC-PCR should be regarded as a true positive.

Appendix

An estimate of the prior probability of the diagnostic sensitivity of the SCT was obtained using estimates of the analytical sensitivity estimates for different worm burdens (10) and combining these with the proportion of naturally infected foxes that are expected to have these worm burdens using data from Hegglin et al. (16). As the sensitivity estimates obtained from Karamon et al. (10) were only valid for four exact values of worm burden (2, 5, 10 and 30 worms), the data set (16) was divided into two different ways. First, in groups 1, 2–4, 5–9, 10–29 and >29 worms (ds-Low) and then in groups 1–2, 3–5, 6–10, 11–30 and >30 worms (ds-High). The sensitivity of the SCT in each group was assumed to be 0.15, 0.3, 0.4, 0.6 and 1 (ds-Low) and 0.3, 0.4, 0.6, 1 and 1 (ds-High) as described by Karamon et al. (10). However, as Karamon et al. (10) had not estimated the sensitivity for one worm (the first group in ds-Low) we used 0.15, which is half the sensitivity for samples with two worms. The overall sensitivity for SCT, based on ds-Low, was calculated as follows

$$p_1*S{e}_1+p_2*S{e}_2+p_3*S{e}_3+p_4*S{e}_4+p_5*S{e}_5$$ 1

where p ₁ is the proportion of samples in the first group (worm burden=1) and Se ₁ is the sensitivity estimate used for the first group (0.15). The calculation was then repeated in a similar way for ds-High.

The calculation based on ds-Low underestimates the sensitivity as, for example, in a group with 2–4 worms the SCT is assumed to have a sensitivity of 0.3 although this estimate is based on samples with only two worms (10). Conversely, the estimate based on ds-High overestimates the sensitivity.

The prior for the diagnostic sensitivity of SCT was calculated to be 0.756 (ds-Low) and 0.836 (ds-High). However, the estimates of the sensitivity reported by Karamon et al. (10) were overestimated according to the author. Therefore, the estimate 0.836 is overestimated both by Karamon and in the current calculation, whereas the estimate of 0.756 is overestimated by Karamon but underestimated in the current calculation. In order not to underestimate the sensitivity for SCT, we used 0.836 as the most likely value and concluded that we were 90% sure (expert opinion of the authors) that the sensitivity was not below 0.756.

Conflict of interest and funding

The authors have not received any funding or benefits from industry or elsewhere to conduct this study.