Head-to-head evaluation of a commercial real-time PCR assay targeting plasmodial species causing human malaria against a composite reference standard

Abstract

Background

The investigation was conducted to evaluate a robust commercial multiplexed species- or species complex-specific real-time PCR assay for human malaria applicable with mobile cyclers.

Methods

A well-characterized sample set of blood samples from travelers with high pre-test probability of being infected with plasmodial species was assessed with the Plasmodium qPCR panel (Bioeksen, Istanbul, Turkey) against a composite reference standard comprising microscopy following the protocols by World Health Organization in combination with four nucleic acid amplification tests for malaria.

Results

From 1,020 included samples, composite reference standard testing identified 248 (24.3%) as positive for plasmodial DNA, comprising 217x Plasmodium falciparum, 14x Plasmodium vivax, 8x Plasmodium ovale complex, 6x Plasmodium malariae and 3 samples containing DNA of both P. falciparum and P. vivax. Diagnostic accuracy of the evaluated PCR in terms of sensitivity and specificity was 96.8% (240/248) and 99.5% (768/772). The resulting positive and negative predictive values were 98.4% (240/244) and 99.0% (768/776). False-negative results were associated with either submicroscopic parasitemia or parasitemia at the threshold of microscopic detection.

Conclusion

The assessed multiplex real-time PCR assay showed similar performance characteristics like comparable diagnostic assays and thus proved to be suitable for the differentiation of plasmodia in the non-endemic setting.

1. Introduction

Regarding the diagnosis of malaria in both clinical management and routine surveillance, the World Health Organization (WHO) does not recommend the application of molecular assays in its November 2024 guidelines [1]. Reasons for this comprise lacking availability of the technology in resource-poor regions of malaria endemicity and challenging assignment of etiological relevance to positive malaria NAAT (nucleic acid amplification test) results in febrile patients from high-endemicity settings. In such settings, high proportions of the population show low-level parasitemia due to semi-immunity, which is not necessarily of causal relevance to observed clinical symptoms [1].

In resource-rich countries without endemic malaria, however, neither lacking availability of molecular diagnostic platforms nor relevant rates of semi-immunity against malaria are likely to interfere with providing reliable and interpretable molecular diagnostic malaria test results. WHO agrees [1] that molecular malaria testing is associated with high diagnostic sensitivity in case of early infection stages characterized by low parasitemia [2, 3]. Further, NAATs facilitate the diagnosis of mixed plasmodial infections even below the microscopic detection threshold [4, 5]. The latter can be realistically set in the 50 parasites per µL blood range [6], although some authors argue that it may even be slightly lower. Hence, the balance of advantages and disadvantages of molecular malaria testing [1–5] may shift towards the advantages' side in non-endemic settings. This is particularly the case when reliable exclusion of malaria based on the molecular assays' superior sensitivity is desired. It holds especially true for non-endemic settings, where continuous training of laboratory personnel in malaria microscopy is difficult to maintain due to the rare occurrence of plasmodial infections in the diagnostic routine. Nevertheless, not only in travel medicine in non-endemic settings but also in tropical endemic regions, reliability of NAAT-based malaria diagnostics could be confirmed [7–10]. Robustness regarding challenging pre-analytic conditions makes some NAAT applications particularly valuable for the resource-limited tropics [11, 12]. Harsh nucleic acid extraction procedures can further improve diagnostic sensitivity [13].

Historically, molecular malaria testing has been established for more than two decades [14]. Next to traditional real-time polymerase chain reaction (PCR), multiple other molecular platforms for malaria testing, either generic or on species as well as on species complex level have been established [2–5, 16–41]. While malaria severity is currently considered more relevant than plasmodial species for the choice of therapeutic options for acute infection, reliable identification of tertian malaria-associated Plasmodium vivax and Plasmodium ovale complex is necessary in order not to miss the subsequent eradication of hypnocytes in the patient's liver [15]. Those abovementioned NAAT platforms are either laboratory-developed or commercially provided and have shown convincing diagnostic accuracy in previous assessments [2–5, 16–41]. While most assays, which differentiate below the genus level, focus on Plasmodium falciparum, P. vivax, P. ovale complex, Plasmodium malariae and Plasmodium knowlesi, molecular diagnostic discrimination of Plasmodium ovale wallikeri und Plasmodium ovale curtisi below the P. ovale complex level has been introduced as well [32, 39–42]. However, the latter approach is not routinely applied so far due its yet uncertain medical consequences. Other assays not only detect Plasmodium spp. but other diseases with similar symptoms and epidemiology, i.e., fever after travel to or return from the tropics [43, 44]. For the identification of mixed plasmodial infections, both microscopy and immunochromatographic rapid diagnostic test (RDT) assays have been repeatedly described to perform poorer than molecular diagnostic testing [45, 46] in line with the WHO guideline [1]. Further, NAAT approaches have shown to be useful for the reliable discrimination of Plasmodium spp. and Babesia spp. [47], for the diagnosis of emerging zoonotic plasmodial species like Plasmodium cynomolgi [48], for the quantification of plasmodia [48–50], as well as for the molecular detection of determinants of resistance against drugs used for anti-malarial chemotherapy [51]. Notably, however, most available assays provide qualitative or semi-quantitative results without attempts on absolute quantification of plasmodia within the sample materials. Also, potential use of diagnostic specimens, which can be less invasively acquired than human blood, is under ongoing evaluation to further facilitate molecular malaria diagnostics [52, 53].

Contributing to existing evaluation data on molecular diagnostic assays, the here provided study assessed the “Plasmodium qPCR panel” (Bioeksen, Istanbul, Turkey), a commercial real-time PCR assay detecting and differentiating P. falciparum, P. vivax, P. ovale complex, P. malariae and P. knowlesi in human blood. The applicability of the assay with lyophilized reaction chemistry on highly mobile MIC cyclers (Bio Molecular Systems Ltd., London, UK) makes it potentially useful for applications even in remote settings outside high resource areas or standard laboratory conditions. Partly different from other recent approaches on commercial malaria PCR evaluations [54–56], a high sample count was intended to support reliable interpretation of the findings.

2. Methods

2.1. Study design and sample materials

The study was conducted as a comparative assessment with the aim of evaluating a species- or species complex-specific molecular diagnostic assay for human malaria. The evaluation was conducted as a head-to-head comparison against a composite reference standard comprising microscopy as well as three commercial and one laboratory-developed NAAT assays, which had been applied to characterize the used reference materials [57–59]. The blood samples from the assessed sample set had been consecutively taken from travelers and migrants in non-malaria-endemic Germany with the clinical suspicion of malaria for diagnosis at the German National Reference Centre for Tropical Pathogens in Hamburg, ensuring an appropriate pre-test-probability of containing samples with and without plasmodial DNA (deoxyribonucleic acid). The samples had been sent to the German National Reference Centre for Tropical Pathogens within a nine-month period. Residual materials from the diagnostic samples were assessed fully blinded in line with the ethical requirements as detailed below. Consequently, patient specific data cannot be provided, which is an unavoidable deviation from the STARD (Standards for Reporting Diagnostic Accuracy) criteria [60].

2.2. Laboratory diagnostics

Nucleic acid extraction was performed from ethylene diamine tetraacetic acid (EDTA) blood by applying the EZ1 DNA Blood 200 µL Kit (Qiagen, Hilden, Germany) on an automated EZ1 nucleic acid extractor (Qiagen, Hilden, Germany) according to the manufacturer's specifications and as previously evaluated with human samples containing plasmodial DNA [61]. Notably, this extraction scheme differed from the assays licensed by the PCR assay's manufacturer for diagnostic purposes. Following nucleic acid extraction, all samples were tested using the “Plasmodium qPCR panel” (Bioeksen, Istanbul, Turkey). This multiplexed diagnostic assay targets P. falciparum, P. vivax, P. ovale complex, P. malariae and P. knowlesi in two wells. However, it does not detect Plasmodium spp. on generic level. The assay was run on MIC cyclers (Bio Molecular Systems Ltd., London, UK) exactly as described by the manufacturer for in-vitro diagnostic use. Cycle threshold detection of positive samples was conducted with the assay-specific software Sigmoida version 8.6 REV.107 (Bioeksen, Istanbul, Turkey) and confirmed with the cycler-specific software mic pcr version 2.12.6 (Bio Molecular Systems Pty Ltd, Upper Coomera, Queensland, Australia). Test results were analyzed using an assay-specific result managing software as well as the cycler specific software. Cycle threshold-(Ct-)values for statistical evaluation were obtained from the assay-specific program if possible.

All samples had been characterized by and were compared to a composite refence standard [59]. This consisted of applying microscopy of Giemsa-stained thick and thin blood films in line with WHO recommendations [1] as well as of four different NAAT approaches: the genus-specific loop-mediated isothermal amplification-(LAMP-)platform Meridian illumigene/alethia Malaria (Meridian Bioscience Inc., Cincinnati, Ohio, USA), the species-specific “FTD Malaria differentiation” real-time multiplex PCR kit (Fast Track Diagnostics (FTD), Sliema, Malta), the genus-specific RealStar Malaria PCR kit 1.0 (altona Diagnostics, Hamburg, Germany) for screening purposes with subsequent discrimination using the RealStar Malaria S&T PCR kit 1.0 (altona Diagnostics) in case of positive screening results, and a species-specific laboratory-developed real-time PCR assay based on a previously described SybrGreen protocol [30, 31]. All real-time PCR assays of the composite reference standard were run on RotorGene Q cyclers (Qiagen, Hilden, Germany). The commercial assays were conducted as described by the manufacturers, details on the in-house assay are provided in Table 1.

Table 1.

Run conditions of the applied in-house real-time PCR assay according to [37] as part of the composite reference standard on RotorGene Q cyclers (Qiagen, Hilden, Germany)

Oligonucleotides
PCR target	Plasmodium spp. PCR
Target gene	18S rRNA gene
Detection limit	5 × 102 copies/µL
Forward primer PL 1473 F18	5′-TAA-CGA-ACG-AGA-TCT-TAA-3′
Reverse primer PL 1679 R18	5′-GTT CCT CTA AGA AGC TTT-3′
Positive control plasmid insert	5′-TTC-CGA-TAA-CGA-ACG-AGA-TCT-TAA-CCT-GCT-AAT-TAG-CGG-CGA-GTA-CAC-TAT-ATT-CTT-ATT-TGA-AAT-TGA-ACA-TAG-GTA-ACT-ATA-CAT-TTA-TTC-AGT-AAT-CAA-ATT-AGG-ATA-TTT-TTA-TTA-AAA-TAT-CCT-TTT-CCC-TGT-TCT-ACT-AAT-AAT-TTG-TTT-TTT-ACT-CTA-TTT-CTC-TCT-TCT-TTT-AAG-AAT-GTA-CTT-GCT-TGA-TTG-AAA-AGC-TTC-TTA-GAG-GAA-CAT-TGT-G-3′ for P. falciparum,
	5′-TTC-CGA-TAA-CGA-ACG-AGA-TCT-TAA-CCT-GCT-AAT-TAG-CGG-TAA-ATA-CAC-TAT-ATT-CTT-AAG-TGA-AAT-TAG-AAT-ATA-GAT-AAA-TTG-TGC-TAA-TTT-TGA-TTA-AAA-TAT-TAG-AAT-GTT-TTT-TTT-AAT-AAA-AAC-GTT-CTT-TTC-CCT-TTT-TTT-CTT-AAT-TAT-GCA-TAT-TTA-TTT-TTT-TTC-TTC-TTT-TGC-ATA-AGA-ATG-TAT-TTG-CTT-AAT-TGT-AAA-GCT-TCT-TAG-AGG-AAC-GAT-GTG-3′ for P. malariae,
	5′-TTC-CGA-TAA-CGA-ACG-AGA-TCT-TAA-CCT-GCT-AAT-TAG-CGG-CAA-ATA-CGA-TAT-ATT-CTT-ACG-TGG-GAC-TGA-ATT-CGG-TTG-ATT-TGC-TTA-CTT-CGA-AGA-AAA-TAT-TGG-GAT-ACG-TAA-CAG-TTT-CCC-TTT-CCC-TTT-TCT-ACT-TAG-TTC-GCT-TTT-CAT-ACT-GTT-TCT-TTT-TCG-CGT-AAG-AAT-GTA-TTT-GCT-TGA-TTG-TAA-AGC-TTC-TTA-GAG-GAA-CGA-TGT-G-3′ for P. vivax,
	5′-TTC-CGA-TAA-CGA-ACG-AGA-TCT-TAA-CCT-GCT-AAT-TAG-CGG-CGA-ATA-CGT-TAT-ATT-CCT-ACT-TGA-AAT-TGA-ATA-TAG-CTG-AAT-TTG-CTT-ATT-TTG-AAG-AAT-ATA-TTA-GGA-TAC-ATT-ATA-GTG-TCC-TTT-TCC-CTT-TTC-TAC-TTA-ATT-CGC-TAT-TCA-TGC-TGT-TTC-TTT-TTT-GTG-TAG-GAA-TGT-ATT-CGT-TTG-ATT-GTA-AAG-CTT-CTT-AGA-GGA-ACG-ATG-TG-3′ for P. ovale complex,
	5′-TTC-CGA-TAA-CGA-ACG-AGA-TCT-TAA-CCT-GCT-AAT-TAG-CGG-CAA-ATA-CGA-TAT-ATT-CTT-ATG-TAG-AAT-TGA-ATA-TAG-TGG-ATT-TGT-TAG-ATT-TTG-AAG-AAA-ATA-TTG-GAA-TTA-CGT-TAA-ATG-TGA-TTC-CTT-TCC-CTT-TTC-TAC-TTA-ATT-TAC-ATT-TCC-ATC-TAT-TTC-TTT-TTT-GCG-TAT-GAA-TGT-ATT-TGC-TTG-ATT-GTA-AAG-CTT-CTT-AGA-GGA-ACG-ATG-TG-3′ for P. knowlesi
GenBank accession number used for the insert	JQ627151.1 for P. falciparum, EF467831.1 for P. malariae, JQ627158.1 for P. vivax, KF018659.1 for P. ovale complex, KJ917904.1 for P. knowlesi
Reference	[37]
Reaction chemistry
Master Mix	LightCycler FastStart DNA Master Mix SyBR Green I (Roche, Basel, Switzerland)
Reaction volume (µL)	20
Forward primer concentration (pmol µL−1)	500
Reverse primer concentration (pmol µL−1)	500
Final Mg2+ concentration (mM)	6.5
Eluate volume (µL)	2
Run conditions
Initial denaturation	10 min at 95 °C
Cycle numbers	40
Denaturation	10 s at 95 °C
Annealing	5 s at 60 °C
Amplification	20 s at 72 °C
Hold	2 s at 95 °C followed by 30 s at 68 °C
Melting
Ramp from	68–90 °C
Rising	0.6 °C each step
Wait for	90 s of pre-melt conditioning on first step
Wait for	5 s for each step afterwards
Acquire to	Melt A on Green
Final cooling
Cooling	20 s at 40 °C

Expected melting temperatures: 73.5 bis 75.5 °C for P. malariae, 75.5–77.5 °C for P. falciparum, 77.5–79.0 °C for P. ovale complex, 79.0–81.0 °C for P. vivax. CAVE: The expected melting temperature for P. knowlesi is in the 77–78.0 °C range and thus indistinguishable from either P. falciparum or P. ovale complex with this approach, making it unsuitable for settings, in which P. knowlesi is expected. min = minute. s = second.

2.3. Statistics

Descriptive statistics were applied to calculate diagnostic accuracy of the assessed real-time PCR assay in terms of sensitivity and specificity in comparison with the composite reference standard. Additionally, positive and negative predictive values for the assessed population of German travellers and migrants with suspected malaria were calculated. When comparing the assessed NAAT assay with the composite reference standard, a positive NAAT result was considered as most likely true-positive, if it was either confirmed by microscopy, another positive result of one of the NAAT approaches of the composite reference standard or by a matching result in another sample from the same patient during the same disease period in case of repeated sampling. Considering the study's focus on the prove of abundance of plasmodial DNA, positive microscopic results were considered as sufficient but not as a necessarily required for confirming a positive NAAT result as most likely true-positive. A sample was accepted as true-negative, if either fewer than two diagnostic approaches from the composite reference standard confirmed the diagnosis malaria or, alternatively, fewer than one diagnostic approach from the composite reference standard in combination with a positive result in another sample from the same patient during the same disease period in case of repeated sampling. If the latter prerequisite was not fulfilled, single contradicting positive test results from the composition reference standard were considered as most likely non-specific. A summarized visualization of the case definitions used to define positivity and negativity of each sample with the diagnostic composite reference standard is provided in Table 2. Invalid PCR results in malaria-positive samples were counted as false-negative.

Table 2.

Case definitions of true- or false-positive and negative test results within the applied composite reference standard, which was used for the characterization of the sample set for the evaluation

Observation in one of the assays of the composite reference standard	Co-factor	Interpretation	Justification
Positive test result	≥1 additional matching positive test result obtained with another assay in the same sample	True-positive	Minimum concordance between two independent assays within the same sample
Positive test result	No matching positive test result within the same sample with another assay, but matching positive test result in another, temporarily associated sample from the same patient obtained with any assay within the composite reference standard	True positive	Minimum concordance between two temporarily associated samples within the same patient
Positive test result	No matching positive test result within both the same sample and any temporarily associated sample from the same patient	False-positive	Minimum concordance neither achieved between at least two independent assays within the same sample nor between two temporarily associated samples of the same patient
Negative test result	≥2 concordantly positive test results for the parameter obtained with other assays in the same sample	False-negative	Minimum concordance between two positive results of independent assays within the same sample defines positivity of the sample as stated above
Negative test result	One discordantly positive test result for the parameter obtained with another assay in the same sample and matching positive test result in another, temporarily associated sample from the same patient obtained with any assay within the composite reference standard	False-negative	Minimum concordance of positive test results between two temporarily associated samples from the same patient defines positivity as stated above
Negative test result	One discordantly positive test result for the parameter obtained with another assay in the same sample but no matching positive test result in another, temporarily associated sample from the same patient obtained with any assay within the composite reference standard	True-negative	Minimum concordance for the definition of positivity neither achieved between at least two assays within the same sample nor between two temporarily associated samples of the same patient
Negative test result	No discordantly positive result in the same sample	True-negative	Lacking proof of the abundance of plasmodial DNA above the detection threshold of any assay within the composite reference standard in the sample

Differences of Ct values over various categories of microscopically diagnosed parasitemia were calculated applying non-parametric Mann-Whitney testing. The detection threshold of microscopy was defined as 50 parasites per microliter and for the calculations, submicroscopic parasitemia was assigned to this threshold. In addition to the comparison of parasite density categories, non-parametric correlation of parasitemia and Ct-values as indicated by Spearman r was calculated. For the latter two types of calculations, samples from patients with infections due to more than one plasmodial species as indicated by the diagnostic reference standard where excluded. In contrast, samples showing mixed infections with molecular confirmation applying the diagnostic reference standards were categorically assigned to all detected plasmodial species for diagnostic accuracy assessments. The calculations were conducted using the software GraphPad InStat version 3.06 (GraphPad Software, Inc., San Diego, CA, USA).

2.4. Ethics

Ethical clearance for the blinded use of residual diagnostic sample materials for the purpose of diagnostic test comparisons and evaluations without requirement of informed consent was granted by the ethics committee of the Medical Association of Hamburg, Germany (registration number WF-046/16) in accordance to national German laws.

3. Results

3.1. Characterization of the sample population based on the diagnostic composite reference standard

A total of 1020 EDTA blood samples, which had been sent to the German National Reference Center for Tropical Infections within a nine-month period due to the clinical suspicion of malaria, were included in the assessment. The samples had been collected from a total of 815 patients. In addition to the initial samples, one (n = 51), two (n = 29), three (n = 19), four (n = 6) up-to five (n = 3) follow-up samples from the patients were provided, while only one sample was available for n = 707 individuals. In line with the case definitions based on the composite reference diagnostic approach as detailed above, a total of 248/1,020 (24.3%) samples were defined as positive for malaria. Notably, 3 samples were defined as “negative” in spite of individual positive PCR signals in the FTD real-time PCR assay (2x P. falciparum, 1x P. malariae), which contradicted all other test results within the composite reference standard. Therefore, they were considered as most likely false-positive. Notably, the negative samples comprised 14 follow-up samples from patients with malaria (7x P. falciparum, 3x P. vivax, 3x P. ovale complex, and 1x P. malariae), from which neither microscopy nor NAAT succeeded in finding any residual traces of plasmodia. From the 248 defined as “malaria positive”, 213 were found to be positive for P. falciparum, 14 for P. vivax, 8 for P. ovale complex, and 6 for P. malariae applying the composite reference standard. The remaining 7 samples were provided by two patients with proven mixed infection of P. falciparum and P. vivax within at least one of the follow-up samples. Applying the case definitions as detailed above, 3 of these 7 samples were confirmed as positive for both P. falciparum and P. vivax while only P. falciparum could be confirmed within the remaining 4 samples. Submicroscopic samples, in which malaria diagnosis applying the diagnostic composite reference standard was established based on NAAT results only, comprised 73 P. falciparum DNA-containing samples, 3 P. malariae DNA-containing samples, and 1 out of the 7 samples from the two patients with mixed plasmodial infections. In contrast, P. vivax-positive samples and P. ovale complex-positive samples were at least detectable at the microscopic detection threshold defined as 50 parasites/µL. Highest measured parasitemia values were 700,000 parasite/µL for P. falciparum, 50,000 parasites/µL for P. vivax, 20,902 parasites/µL for P. ovale complex and 360 parasites/µL for P. malariae.

3.2. Calculated diagnostic accuracy of the real-time PCR assay, predictive values with the used sample population and description of mismatching samples

Regarding the detection of malaria irrespective of the recorded plasmodial species, the assessed real-time PCR assay showed diagnostic accuracy in terms of sensitivity and specificity of 96.8 and 99.5%, respectively, compared with the diagnostic reference standard if used as a screening tool (Table 3). If samples showing fluorescence curves in the result managing software of the assay but not in the cycler run protocol were excluded, sensitivity and specificity would be 96.4 and 99.7%, respectively (Table 3). Resulting positive and negative predictive values were 98.4 and 99.0%, respectively, or 99.2 and 98.8% in case of exclusion of samples showing fluorescence curves only in the assay result managing software (Table 4).

Table 3.

Diagnostic accuracy estimates of the assessed malaria PCR. P. knowlesi-specific PCR was not assessed due to a lack of samples showing positive results in the assays of the diagnostic reference standards. Mixed infections were assessed for each relevant species-specific PCR

Assay	Sensitivity in n/N; % (95% CI)	Specificity over the whole sample population in n/N; % (95% CI)	Specificity over the subpopulation of malaria-positive samples in line with the case definition in n/N; % (95% CI)
P. falciparum	214/220; 97.3% (94.2–99.0%)	798/800; 99.8% (99.1–100%)	26/28; 92.9% (76.5–99.1%)
P. malariae	6/6; 100% (54.1–100%)	1,014/1,014; 100% (99.6–100%)	242/242; 100% (98.5–100%)
P. ovale complex	7/8; 87.5% (47.3–99.7%)	1,012/1,012; 100% (99.6–100%)	240/240; 100% (98.5–100%)
P. vivax	14/17; 82.4% (56.6–96.2%)	998/1,003; 99.4% (98.7–99.8%)	228/231; 98.7% (96.3–99.7%)
P. vivax (without samples with discrepant results of the assay software and the cycler software)	13/17; 76.5% (50.1–93.2%)	997/1,003; 99.5% (98.8–99.8%)	228/231; 98.7% (96.3–99.7%)
Plasmodium spp.*	240/248; 96.8% (93.8–98.6%)	768/772; 99.5% (98.7–99.9%)	n.a.
Plasmodium spp.* (without samples with discrepant results of the assay software and the cycler software)	239/248; 96.4% (93.2–98.3%)	770/772; 99.7% (99.1–100%)	n.a.

* without requirement of correct species assignment. n.a. = not applicable.

Table 4.

Predictive value estimates for the chosen sample population from a non-endemic setting. P. knowlesi-specific PCR was not assessed due to a lack of samples showing positive results in the assays of the diagnostic reference standards. Mixed infections were assessed for each relevant species-specific PCR

Assay	PPV over the whole sample population in n/N; % (95% CI)	NPV over the whole sample population in n/N; % (95% CI)	PPV over the subpopulation of malaria-positive samples in line with the case definition in n/N; % (95% CI)	NPV over the subpopulation of malaria-positive samples in line with the case definition in n/N; % (95% CI)
P. falciparum	214/216; 99.1% (96.7–99.9%)	798/804; 99.3% (98.4–99.7%)	214/216; 99.1% (96.7–99.9%)	26/32; 81.3% (63.6–92.8%)
P. malariae	6/6; 100% (54.1–100%)	1,014/1,014; 100% (99.6–100%)	6/6; 100% (54.1–100%)	242/242; 100% (98.5–100%
P. ovale complex	7/7; 100% (59.0–100%)	1,012/1,013; 99.9% (99.5–100%)	7/7; 100% (59.0–100%)	240/241; 99.6% (97.7–100%)
P. vivax	14/20; 70% (45.8–88.1%)	997/1,000; 99.7% (99.1–99.9%)	14/17; 82.4% (56.6–96.2%)	228/231; 98.7% (96.3–99.7%)
P. vivax (without samples with discrepant results of the assay software and the cycler software)	13/18; 72.2% (46.6–90.3%)	998/1,002; 99.6% (99.0–99.9%)	13/16; 81.3% (54.3–96.0%)	228/232; 98.3% (95.7–99.5%)
Plasmodium spp.*	240/244; 98.4% (95.9–99.6%)	768/776; 99.0% (98.0–99.6%)	n.a.	n.a.
Plasmodium spp.* (without samples with discrepant results of the assay software and the cycler software)	239/241; 99.2% (97.0–99.9%)	770/779; 98.8% (97.8–99.5%)	n.a.	n.a.

PPV = positive predictive value. NPV = negative predictive value. 95% CI = 95% confidence interval. n.a. = not applicable.

Twenty out of 1,020 (2.0%) samples were misassigned by the evaluated real-time PCR assay in comparison with the diagnostic reference standard (Table 5). A total of 8 samples, which were judged to be malaria-positive according to the composite reference standard, could not be identified as malaria-positive by the assessed real-time PCR assay. The failed detections comprised 6 cases of P. falciparum-positive samples, including five negative and one invalid PCR results. The invalid result was counted like false-negative as described in the methods section. Also, two P. vivax-positive samples were misclassified as negative. As detailed in Table 5, all 8 false-negatives occurred with samples showing either submicroscopic parasitemia (n = 6) or parasitemia at the microscopic detection threshold of ≤50 parasites/µL (1x for the single invalidly tested P. falciparum-positive sample and 1x for a P. vivax-positive sample). In 4 cases, negative samples according to the diagnostic reference standard showed positive PCR-signals (3x P. vivax with Ct-values 26, 28 and 29, 1x P. knowlesi with a Ct-value 29, respectively). Notably, two of these four presumably false-positive results (1x P. vivax, 1x P. knowlesi) did not show visible fluorescence curves in the cycler software. Accordingly, these false-positives were avoidable if both the assay-specific software and the cycler-software would be considered. However, doing so would also remove a Ct-value 30 for P. vivax correctly identified with the assay-specific software but missed with the cycler software for a sample showing a parasitemia of 50/µL in microscopy. In a total of 5 cases, the assessed real-time PCR suggested mixed plasmodial infections, which had not been detected applying the diagnostic reference standard. These cases comprised two additional P. vivax DNA detections (Ct values 24 and 28) in samples positive for P. ovale complex and P. falciparum, respectively, two additional P. falciparum DNA detections (Ct-values 26 and 22) in samples positive for P. malariae and P. ovale complex, respectively, as well as an additional P. knowlesi PCR signal (Ct-value 29 in the assay-specific software and Ct-value 25 in the cycler-specific software) in a sample with a submicroscopic P. falciparum infection. Finally, in case of a P. ovale complex-positive sample showing parasitemia at the microscopic detection threshold of 50/µL, the assay indicated a P. vivax-specific signal with a Ct-value of 26 instead. Notably, this sample was a follow-up sample of another sample identified as positive for P. ovale complex, both applying the composite reference standard and the evaluated real-time PCR assay.

Table 5.

Details on mismatching results

Blinded sample number	Plasmodium spp. according to composite reference standard	Parasitemia (n/µL)	Plasmodium spp. according to the assessed PCR-assay	Ct-value according to assay-specific analysis software	Ct-value according to cycler-specific analysis software
20	P. falciparum	≤50/µL	invalid test result	n.a.	n.a.
195	P. vivax	sub-microscopic	negative	n.a.	n.a.
241	P. falciparum	sub-microscopic	negative	n.a.	n.a.
244	P. falciparum	sub-microscopic	negative	n.a.	n.a.
255	P. falciparum	sub-microscopic	P. knowlesi 1	29	25
272	P. malariae	104/µL	P. falciparum 2	26	26
302	P. ovale complex	50/µL	P. vivax	26	26
417	P. ovale complex	13,000/µL	P. vivax 3	24	24
460	P. falciparum	sub-microscopic	negative	n.a.	n.a.
463	negative	n.a.	P. knowlesi	29	n.a.
503	P. ovale complex	50/µL	P. falciparum 4	22	22
564	P. falciparum	sub-microscopic	negative	n.a.	n.a.
567	P. falciparum + P. vivax	4,600/µL	P. vivax not detected5	n.a.	n.a.
644	negative	n.a.	P. vivax	29	n.a.
721	P. falciparum	sub-microscopic	negative	n.a.	n.a.
749	negative	n.a.	P. vivax	26	26
866	P. vivax	50/µL	P. vivax	30	n.a.6
867	P. vivax	≤50/µL	negative	n.a.	n.a.
889	negative	n.a.	P. vivax	28	28
989	P. falciparum	50/µL	P. vivax 7	28	28

¹ in addition to P. falciparum, which was correctly identified with a Ct-value of 18.

² in addition to P. malariae, which was correctly identified with a Ct-value of 24.

³ in addition to P. ovale complex, which was correctly identified with a Ct-value of 21.

⁴ in addition to P. ovale complex, which was correctly identified with a Ct-value of 17.

⁵ P. falciparum was correctly identified at Ct 21. Ct = cycle threshold.

⁶ correctly identified by assay-specific analysis software only.

⁷ in addition to P. falciparum, which was correctly identified at Ct 22. Ct = cycle threshold. n.a. = not applicable due to negative test results.

Applying the case-definition, Plasmodium species-specific diagnostic sensitivity of the evaluated species-specific assays ranged from 76.5 to 100%. In more details, sensitivities were 76.5% (or 82.4% if the sample evaluated as positive applying the assay-specific software but not the cycler software was included) for P. vivax, 87.5% for P. ovale complex, 97.3% for P. falciparum and 100% for P. malariae, respectively (Table 3). Calculated species-specific specificity values ranged from 99.4 to 100% if the PCR assay was applied with the total population and from 92.9 to 100% if the assay was used for species differentiation in samples already diagnosed as malaria positive (Table 3). Species-specific positive predictive values ranged from 70.0 to 100% over the whole sample population and from 81.3 to 100% for the subpopulation of malaria-positive samples. Species-specific negative predictive values ranged from 99.3 to 100% over the whole sample population and from 81.3 to 100% over the subpopulation of malaria-positive samples (Table 4).

3.3. Associations of measured Ct-values in real-time PCR and microscopically assessed parasitemia

When comparing microscopically measured parasitemia with obtained Ct-values from the assessed PCR assay, significant Ct-value differences over the assessed quantitative categories as shown in Table 6 could be observed for P. falciparum but not for P. malariae, P. ovale complex and P. vivax. In line with this, a significant correlation (Spearman r = −0.9093, P < 0.0001) could be calculated for P. falciparum, while significant correlations were not demonstrated for P. malariae (Spearman r = −0.0308, P > 0.9999), P. ovale complex (Spearman r = 0.3397, P = 0.4444) and P. vivax (Spearman r = −0.4733, P = 0.1201) with low numbers of positive samples. Figure 1 shows trend lines over the measured values for all four plasmodial species. Notably, only samples showing single-species plasmodial infections with the diagnostic reference standard were included in this assessment and only Ct-values classified as true-positive in line with the case definition were considered.

Table 6.

Comparison of measured Ct-values by real-time PCR and microscopically measured parasitemia, based on samples assumed to be truly positive by PCR based on the applied case definitions. Samples from patients with diagnosed plasmodial double infection were excluded from this assessment

Parasites/µL	≤50	>50 – ≤500	>500 – ≤5,000	>5,000 – ≤50,000	>50,000 – ≤500,000	>500,000	P
P. falciparum, median Ct (min., max.), (n)	21.0 (16.0, 24.0), (n = 3)	18.0 (14.0, 22.0), (n = 24)	15.5 (13.0, 22.0), (n = 34)	12.0 (10.0, 16.0), (n = 34)	10.0 (7.0, 18.0), (n = 15)	7.0 (7.0, 7.0), (n = 3)	P < 0.0001
P. malariae, median Ct (min., max.), (n)	18.0 (17.0, 27.0), (n = 3)	22.0 (20.0, 24.0), (n = 2)	n.a.	n.a.	n.a.	n.a.	P = 0.826
P. ovale complex, median Ct (min., max.), (n)	21.0 (16.0, 24.0), (n = 3)	19.0 (18.0, 20.0), (n = 2)	21.0 (21.0, 21.0), (n = 2)	n.a.	n.a.	n.a.	P = 0.4925
P. vivax, median Ct (min., max.), (n)	12.0 (11.0, 30.0), (n = 3)	19.0 (17.0, 21.0), (n = 2)	22.0 (10.0, 23.0), (n = 3)	11.0 (10.0, 12.0), (n = 4)	n.a.	n.a.	P = 0.3818*

min. = minimum. max. = maximum. µL = microliter. Ct = cycle threshold value. n = number. P = significance level according to Kruskal-Wallis testing. n.a. = not applicable. *If the P. vivax sample without detectable fluorescence curves in the cycler software was excluded, the resulting P. vivax Ct-value mean (±SD) would be 11.5 (±0.7) and the median (min., max.) 11.5 (11, 12) for the remaining two true-positive samples within the ≤50 parasites/µL subpopulation, resulting in a P-value of 0.3340 in Kruskal-Wallis testing.

Fig. 1.

Correlation of microscopically assessed parasitemia (x-axis, decadic logarithmic scale) and numeric Ct-values of the assessed PCRs (y-axis) for P. falciparum (A), P. malariae (B), P. ovale complex (C) and P. vivax (D). Samples from patients with diagnosed plasmodial double infection were excluded from this assessment. Blue dots indicate samples, the trend line is shown in black

3.4. Real-time PCR performance with samples from patients with mixed plasmodial infections confirmed by the diagnostic reference standard

For two included patients, the assessment with the diagnostic reference standard had shown mixed plasmodial infections, in both instances comprising P. falciparum and P. vivax DNA. Diagnosis of mixed infections did not succeed in all follow-up samples (Table 7). In one case, the condition was diagnosed not earlier than in the second follow-up sample. Composite reference standard diagnoses of P. falciparum and P. vivax co-infections succeeded in two samples with intermediate parasitemia (5,240 parasites/µL and 4,600 parasites/µL) and one sample with low parasitemia (<50 parasites/µL), but not in samples with parasitemia of more than 10,000 parasites/µL. The here-assessed real-time PCR assay correctly identified two out of three patients with mixed infections in two out of seven samples in line with the composite reference standard results. The two correctly identified samples comprised the ones with parasitemia of 5,240 parasites/µL and <50 parasites/µL, respectively.

Table 7.

Samples from patients with mixed plasmodial infections in line with the case definitions

Sample number	Fully blinded patient identifier	Number of sequential samples	Type of mixed infection	Parasitemia in microscopy in parasites/µL	Molecular confirmation of mixed infection by reference standard	Ct-value P. falciparum	Ct-value P. vivax
556	A	1	P. falciparum & P. vivax	3,424/µL	P. falciparum only detected	13	n.a.
569	A	2	P. falciparum & P. vivax	28,000/µL	P. falciparum only detected	11	n.a.
570	A	3	P. falciparum & P. vivax	5,240/µL	yes	20	25
589	A	4	P. falciparum & P. vivax	sub-microscopic	P. falciparum only detected	27	n.a.
567	B	1	P. falciparum & P. vivax	4,600/µL	yes	21	n.a.
568	B	2	P. falciparum & P. vivax	55,040/µL	P. falciparum only detected	10	n.a.
583	B	3	P. falciparum & P. vivax	<50/µL	yes	21	25

n.a. = not applicable due to negative PCR results.

4. Discussion

The study was conducted to assess the diagnostic accuracy of a multiplex real-time PCR assay for the species-specific detection of human Plasmodium spp. under real life-like conditions using DNA extracted from more than a thousand consecutively collected EDTA blood samples sent to the German National Reference Center for Tropical Pathogens due to the clinical suspicion of malaria. Its diagnostic accuracy in terms of sensitivity and specificity was found to be in a similar range as previously recorded for other species-specific malaria PCR assays with sensitivity >95% and specificity >99% over all assessed samples [56, 57].

4.1. False-negative and false-positive reactions

The few cases of false-negative results comprised submicroscopic samples or samples with low parasitemia at the diagnostic detection threshold. The two missed samples with visible parasitemia at the diagnostic threshold of microscopy comprised a P. vivax-positive one and a P. falciparum-positive one with an invalid result due to a disturbed PCR reaction. In all other instances, samples detectable as malaria-positive applying the diagnostic WHO standard [1] were also found with the real-time PCR assay, in addition to the vast majority of sub-microscopic samples.

Likely false-positive results in line with the study's case definitions were observed for the assays targeting P. falciparum, P. vivax and P. knowlesi but not for P. ovale complex and P. malariae. Ct-values of such additional positive PCR results covered a broad spectrum ranging from 22 to 29 with a right-shifted distribution. Insofar, high Ct-values were not a sure predictor of lacking specificity. True-positive results with high Ct-values were detected as well. A fluorescence signal invisible in the cycler but not in the assay software was not a reliable predictor of false-positive results either. In some, but not all, likely false-positive cases, lacking detectability of fluorescence curves in the cycler-specific software in spite of Ct-values provided by the assay-specific software suggested likely non-specificity of the latter. Cases of such discrepancies between the results of the assay-specific software and the cycler-specific software were restricted to high Ct values ≥29. In such cases, cross-checking of the matching of results provided by the assay-specific software and the cycler-specific software may help avoiding false species-assignments in diagnostic application of the PCR assay.

4.2. Mixed plasmodial infections

The results obtained from samples of the two patients with mixed plasmodial infections using the diagnostic reference standard as well as the here-assessed real-time PCR assay showed that the less frequently abundant species may go undetected by NAAT in spite of overall microscopic parasitemia higher than 10,000 parasites/µL. The few available cases of high parasitemia with likely missed mixed infections make interpretations challenging, although factors like competitive amplification or minor species suppression might have played a role. However, microscopy scored even worse, as none of the samples of the two patients could be microscopically characterized as plasmodial mixed infection, confirming the superior sensitivity of real-time PCR. The latter makes it conceivable that the here-assessed PCR-assay detected low amounts of plasmodial DNA not recorded by the diagnostic reference standard due to sensitivity differences in other samples as well. However, this assumption is speculative and such individual misassignments in the course of the here-described study are unlikely to relevantly affect the observed diagnostic accuracy estimations.

4.3. Low numbers of samples from patients with non-falciparum malaria

Focusing on the diagnostic accuracy estimates on plasmodial species level, low total numbers of samples containing P. ovale complex DNA or P. vivax DNA contributed to low sensitivity values for both and low positive predictive values for P. vivax in spite of only few mismatches compared with the composite reference standard. In contrast, specificity and negative predictive value of the P. falciparum-specific PCR dropped when it was not used for screening purposes over the whole sample population but just for species-level discrimination with samples already identified as malaria-positive. Such differences should be kept in mind when diagnostically applying the assay.

4.4. Lack of samples containing DNA of P. knowlesi and other zoonotic plasmodial species

Diagnostic accuracy estimates were not calculated for P. knowlesi, because the assessed samples did not contain DNA of this plasmodial species. The two positive results in the assessed assay were likely false-positive with high Ct-values for P. knowlesi and could be confirmed as such due to a lacking specific fluorescence curve in the cycler software at least in one instance. We abstained from including sample materials from external laboratory control schemes, because the assessment was not meant to replace the producer's product validation but to assess its performance in a real life-like diagnostic scenario in the low-prevalence setting for human malaria. Although our assessment cannot contribute to information on the sensitivity of the P. knowlesi-specific PCR reaction, an estimation of its specificity when applied under such conditions is yet feasible. Even when assuming the two recorded P. knowlesi signals as likely false-positive, an excellent specificity of 99.8% (99.3%, 100%) is the result. This aligns well with the specificity values observed for the other targeted plasmodial species of the assay. As a consequence for the diagnostic routine situation in the low prevalence setting, it can only be concluded that very late Ct-values for rare pathogens like P. knowlesi need to be interpreted with care. These cases need to be critically discussed with the clinician in charge regarding matching clinical symptoms and epidemiology as well as potential differential diagnoses. This particularly applies if discrepancy between assay-specific software and cycler software in case of Ct values close to the diagnostic detection threshold raises the suspicion of potential non-reliability of recorded signals.

4.5. Parasitemia and Ct-values

Concerning associations between microscopically detected parasitemia and Ct values, the expected negative correlation could only be confirmed for P. falciparum. For P. malariae, P. ovale complex, and P. vivax, the available data points were insufficient to prove a significant correlation in any direction. Due to this lack of information and in contrast to reports on alternative real-time PCR approaches targeting plasmodia [48–50], Ct value-based semi-quantification should be therefore interpreted with care. To a similar extend, the wide distribution of Ct-values at and below the microscopic detection threshold defined as 50 parasites/µL should not be overinterpreted.

4.6. Implications of the limited spectrum of the assay

Although the assessed real-time PCR assay showed high reliability in case of a hypothetical use as a primary diagnostic tool, suitability for this purpose will necessarily depend on the local epidemiology. As mentioned above, not only P. knowlesi was absent in the here-assessed sample population, but also other zoonotic plasmodia less frequently observed in human samples like P. brasilianum, P. cynomolgi, and P. simium [62] were lacking. Since the assessed assay is neither designed to detect these species nor to detect Plasmodium spp. on a generic level, other diagnostic approaches need to be added if plasmodial infections caused by plasmodia other than the five species or species-complexes covered by the assay are suspected. Also, additional molecular diagnostic approaches are required when discrimination of P. ovale curtisi and P. ovale wallikeri below the P. ovale complex level is desired [41–44].

4.7. Study limitations

The study has a number of limitations. First of all, it was based on a limited number of reference samples, including only few specimens with non-falciparum plasmodia and without P. knowlesi. Consequently, the reliability of statistics concerning non-falciparum plasmodia is lower than for the more frequently available P. falciparum DNA-containing samples with broader 95% confidence intervals. On the other hand, this situation represents the epidemiological situation of imported malaria in the non-endemic setting of Germany quite well. Second, although a very broad spectrum of methods was included in the composite reference standard, it cannot be excluded that low amounts of plasmodia DNA, which resulted in seemingly false-positive results of the evaluated assay, were missed by the composite reference approach. Although much effort was taken to characterize the assessed sample set, incorporation bias due to erroneous assignment of positivity and negativity based on the applied diagnostic composite reference standard cannot be completely excluded.

From a technical point of view, it seems worth mentioning that the assessed PCR assay has several limitations. It is run as a two-well approach with an internal amplification control being included in one out of two reaction tubes only. Insofar, the other tube was only indirectly controlled for errors during nucleic acid extraction but not for invalid amplification. It was also run as an open tube approach, which at least theoretically might increase the risk of cross contamination. This was, however, never observed or even suspected during the assessment of more than 1,000 samples.

It is important to disclaim that the applied nucleic acid extraction approach was different from the methods included in the in-vitro diagnostic clearance by the manufacturer. Insofar, it cannot be excluded that mismatches observed in this study might have been avoided by using the recommended nucleic acid extraction schemes, since the mode of nucleic acid extraction has been identified as a relevant factor for the reliability of malaria PCR [13]. On the other hand, the used easy-to-apply and automated nucleic acid extraction proved to be highly reliable as confirmed by the excellent obtained diagnostic accuracy results in comparison with the diagnostic composite reference standard. Insofar, it may be well suited for settings with limited laboratory infrastructure and in areas where regulatory requirements demanding the use of nucleic acid extraction assays explicitly as described by the manufacturer do not apply. Also, even regulatory frameworks like, e.g., the European in-vitro diagnostic regulation, leave room for the extension of scope of diagnostic assays if in-house validation is performed. For this purpose, the here-assessed combination of automated nucleic acid extraction and real-time PCR with adequate diagnostic accuracy may be a promising option. Notably, the few observed cases of false-negative results with the applied nucleic extraction scheme were restricted to either sub-microscopic samples or samples at the microscopic detection threshold, suggesting very limited effects on the diagnostic sensitivity.

5. Conclusions

In spite of the above-mentioned limitations, the assessed real-time PCR assay proved to be suitable for plasmodial discrimination on species level and – depending on the expected local epidemiology concerning non-addressed zoonotic Plasmodium spp. – also as a primary diagnostic tool in case of suspected human malaria. Its diagnostic accuracy was found to be similar to comparable NAAT assays [58, 59]. Its applicability with mobile MIC cyclers and using lyophilized reaction chemistry makes it appear suitable for diagnostic applications outside the standard laboratory setting. The necessary assessments were beyond the scope of the here-presented evaluation though. Thus future studies on the assay's performance in resource-limited field settings should be considered. Also, NAAT should be compared with other technologies, which presently challenge manual microscopy as diagnostic reference standard for the diagnosis of malaria due to increased diagnostic accuracy. In particular, it seems advisable to assess the value of real-time PCR for the primary diagnosis of malaria against emerging and promising diagnostic approaches like artificial intelligence- (AI-) guided microscopy [63] or automated Plasmodium spp. detection in cell sorters [64]. As AI was shown to be superior regarding microscopic pattern recognition compared with human investigators [65], AI-guided micro-morphological detection of malaria might become a challenge for the sensitivity of real-time PCR-guided diagnostic schemes.

Abbreviations

The following abbreviations are used in this manuscript:

95% CI

95% confidence interval

AI

artificial intelligence

Ct

cycle threshold

DNA

deoxyribonucleic acid

EDTA

ethylene diamine tetraacetic acid

µL

microliter

n.a.

not applicable

min

minute

NAAT

nucleic acid amplification test

NPV

negative predictive value

PCR

polymerase chain reaction

PPV

positive predictive value

RDT

rapid diagnostic test

s

second

WHO

World Health Organization

Funding Statement

Funding: The nucleic acid extraction and the molecular assessments for the composite reference standard were funded by the two grants 28K2-S-45 1618 “Evaluation of automated loop-mediated isothermal amplification (LAMP) for easy and rapid proof of malaria for tropical deployments” and 36K2-S-45 1922 “Evaluation and optimization of molecular diagnostic tests for tropical parasitic diseases for surveillance and risk assessment purposes in tropical deployment settings – a German-French cooperation project between the German Armed Forces Hospital Hamburg and the Military Hospital Laveran, Marseille” by the German Ministry of Defence (MoD).