First international external quality assessment of molecular diagnostics for Mers-CoV

Highlights

• •We describe results of the first HCoV-MERS external quality control panel.
• •85% of the 189 returned results had a 100% score.
• •8.1% of laboratories produced false positive results.
• •ORF1b target is not recommended for screenings PCR.

Abstract

Background

Since the discovery of Middle East respiratory syndrome coronavirus (MERS-CoV) in 2012, diagnostic protocols were quickly published and deployed globally.

Objectives

We set out to assess the quality of MERS-CoV molecular diagnostics worldwide.

Study design

Both sensitivity and specificity were assessed using 12 samples containing different viral loads of MERS-CoV or common coronaviruses (OC43, 229E, NL63, HKU1).

Results

The panel was sent to more than 106 participants, of which 99 laboratories from 6 continents returned 189 panel results.Scores ranged from 100% (84 laboratories) to 33% (1 laboratory). 15% of respondents reported quantitative results, 61% semi-quantitative (Ct-values or time to positivity) and 24% reported qualitative results. The major specific technique used was real-time RT-PCR using the WHO recommended targets upE, ORF1a and ORF1b. The evaluation confirmed that RT-PCRs targeting the ORF1b are less sensitive, and therefore not advised for primary diagnostics.

Conclusions

The first external quality assessment MERS-CoV panel gives a good insight in molecular diagnostic techniques and their performances for sensitive and specific detection of MERS-CoV RNA globally. Overall, all laboratories were capable of detecting MERS-CoV with some differences in sensitivity. The observation that 8% of laboratories reported false MERS-CoV positive single assay results shows room for improvement, and the importance of using confirmatory targets.

1. Background

A novel beta-coronavirus, named Middle East respiratory syndrome coronavirus (MERS-CoV), was identified as the cause of severe respiratory disease in humans in the Middle East in 2012 [1], [2]. Within weeks after the initial reports, protocols were published for diagnosing this agent by real-time reverse transcriptase polymerase chain reaction (RT-PCR) [3], [4]. In the first report [3], real-time RT-PCR assays were described for two targets, one located upstream of the envelope gene (UpE) and one within open reading frame 1b (ORF1b). The second report [4] included a diagnostic screening algorithm describing an assay targeting the ORF1a gene, which was more sensitive than the ORF1b assay and therefore recommended as confirmatory assay rather than ORF1b RT-PCR. In the meantime, other laboratories had developed their own assays targeting the nucleocapsid (N) gene, showing similar sensitivity to the upE RT-PCR [5], [6]. The World Health Organization (WHO) recently published updated interim recommendations for laboratory testing of MERS-CoV [7], in which the upE RT-PCR is recommended as first-line screening assay. According to this recommendation, positive MERS-CoV RT-PCR should subsequently be confirmed with either ORF1a, ORF1b, or N gene RT-PCR’s. An alternative for the confirmatory RT-PCR is sequencing of the N gene or RNA dependent RNA polymerase (RdRP) gene (ORF1), as described before [3]. Presently, laboratories around the world have implemented in-house or commercial methods for MERS-CoV molecular diagnostics [8], mostly based on the above described publications [9], but with substantial diversity in nucleic acid extraction methods, RT-PCR (master) mixes and thermocyclers used. External quality assessment (EQA) studies give insight in the possible effects of this variation on assay performance, thereby allowing the participating laboratories to assess the quality of current diagnosis, and identify possible weaknesses and strengths of the range of diagnostic methods that are used.

2. Objectives

With the first MERS-CoV EQA panel we set out to assess the quality of nucleic assay testing (NAT) assays for MERS-CoV worldwide.

3. Study design

3.1. Participants

The information for participating in the MERS-CoV PCR EQA was sent to all members of the European Network for Imported Viral Diseases (ENIVD), external collaborating partners, European national microbiological focal points (NMFP) and diagnostic laboratories in charge of the MERS-CoV diagnostics suggested by the WHO. This EQA was requested by ECDC as part of its joint initiatives with the WHO Office for Europe to support laboratory preparedness building across EU/EEA and European Region Member States. In total, more than 106 participants were invited to take part in the MERS-CoV EQA 2014 study. The participation in the study was free of charge. For benchmarking, all participants received a complete anonymized list of all submitted results with only their own laboratory recognizable. Participants agreed to publication of anonymized results.

3.2. Specimen preparation

The MERS-CoV strain EMC [1] was provided by the Erasmus MC, Rotterdam, The Netherlands. The human coronaviruses (HCoV)-NL63, HCoV-OC43, HCoV-229E were cultivated in the Robert Koch-Institute, Berlin, Germany. All viruses were grown in Vero E6 cells, of which cell culture supernatant was harvested, inactivated by heating (1 h at 60 °C) and gamma irradiated (25 kilo gray). The proficiency test panel included seven samples containing serial 10-fold dilution of MERS-CoV positive cell culture supernatant (1:10–1:10⁷), four samples with common human coronavirus (NL63, OC43, 229E, HKU1) positive culture supernatants, and two negative control sera as specificity controls. The samples were freeze-dried in 2× Lyophilization reagent (Ops diagnostics, NJ, USA) using an EPSILON 2-6D Pilot Freeze Dryer (Martin Christ GmbH, Osterode am Harz, Germany) in 0.5 ml glass vials. The human coronavirus HKU1 was propagated as previously described [10], [11]. No infectious virus was detected when tested for residual infectivity after heat and gamma irradiation by inoculation in cell culture. The participants were assured that they were provided with non-biohazardous material, but recommending to handle all material with care as is recommended for human specimens in general.

3.3. Validation and dispatch of the panel sets

The panel was validated at the Robert Koch-Institute (RKI) and in three reference laboratories (University of Bonn Medical Centre, Germany, Erasmus MC, Rotterdam, The Netherlands and University of Vienna, Austria) using in-house validated methods as well as a commercial assay, for which samples were reconstituted and in 100 μl water and analyzed according to the locally used RT-PCR protocols. The EQA samples were shipped by regular mail or by courier, with instructions for use, and an evaluation form to log results. This included information on the assay protocol, the interpreted result (diagnosis) for each sample, the coronavirus strain identified, the number of genome copies, if possible, as well as any problem encountered.

3.4. Scoring

Based on validation and returned laboratory results, sample 8 was omitted for analysis of the EQA panel, since the concentration of this sample was too low for equal distribution. Returned quantitative results were recalculated to copies/ml for uniformity (copies/μl were multiplied by a factor 1000). To measure the performance of the labs, 1 point per correct result (per sample) was rewarded, and thus 12 points would result in 100% score. Furthermore, to test the differences in performance between PCR targets (with n  > 8 datasets, see Table 1 ), student’s T-test and F-test for variance were calculated by Microsoft Excel 2010.

Table 1.

Number of reported MERS-EQA panel results sorted by result type and target.

Target	Quantitative	Semi quant (Ct value)	Qualitative	Total
upE	17	45	12	74
ORF1a	5	16	5	26
ORF1b	1	13	4	18
N2	0	9	2	11
N3	0	8	2	10
N	0	2	0	2
Nseq	0	0	2	2
ORF16	1	1	0	2
RdRpSeq	0	0	2	2
RNA-pol	1	1	0	2
ORF1ab	0	1	0	1
ORF5/E	0	1	0	1
Unknown	5	17	16	38
Total	30	114	45	189

4. Results

The response rate from 106 participants that had requested the MERS-CoV EQA panel was 93.4% (99 laboratories, see supplementary Table 1). The majority of the laboratories (66.6%) submitted separate results for multiple RT-PCR assays, reflecting their confirmatory diagnostic algorithm, resulting in 189 panel results. A few laboratories did indicate their use of confirmatory PCRs, but only reported one final result. Fig. 1 shows the geographical location of all participants coming from 66 countries. 50% of the participants were located in a European Union (EU) member country, 14% in Asia, 11% in Western Asia (Middle East), 8% in Africa, 7% in a non-EU European country, 4% in Latin America, 3% in North America, and 2% in Oceania according to the United Nations geographic areas.

Fig. 1.

World map of MERS-CoV EQA participants per country.

n = 1; n = 2; n = 3; n = 5; n = 12.

In total, 189 panel results were returned, of which 15% (n = 29) reported quantitative results (copies/μl or copies/ml (qRT-PCR), 61% (n  = 115)) reported semi-quantitative results (Cycle threshold (Ct)-values) from real-time RT-PCRs or time-to-positivity, in case of reverse transcriptase recombinase polymerase amplification (RT-RPA) assay, and 24% (n  = 45) reported qualitative results (positive and negative). In total, 13 genomic targets were used, among which 11 different targets for (real-time) RT-PCR or RT-RPA (n  = 1) assays, and 2 targets for sequencing (N gene and RdRP). This may indicate that a number of unpublished, in-house developed tests were used. Table 1 summarizes the number of panel results returned for each genomic target combined with the analysis type (quantitative, semi-quantitative or qualitative). As recommended by WHO, upE is the most widely used RT-PCR, followed by ORF1a and ORF1b.

4.1. Sensitivity

Overall data showed that the performance of the majority of the laboratories was good, with 85% of the returned results having a score of 100% (12 points) and 9.5% having a score of 92%. Furthermore, 1 lab (0.53%) returned a panel with a score of 86%, 3.7% of the labs scored 83%, 1 lab (0.53%) scored 75% and one lab scored 33%. Table 2 shows the percentage of correct results per MERS-CoV positive sample of the total EQA panel (n  = 7), and the results per assay type, including median (range) per sample for quantitative and semi-quantitative results. Quantitative results showed quite a big variance in the viral load (VL), with the lowest variance for sample #5 (Δ_{maxVL−minVL}  = 3.45 10 log genome equivalents (geq./ml)) and highest for sample #6 (Δ_{maxVL−minVL}  = 4.78 10 log geq./ml), without sample #8 taken into account. These differences were observed despite the fact that most laboratories indicated the use of a quantified in vitro transcribed RNA, provided by the European Virus Archive (EVA) as quantifying standard. To study if this variance is due to the quantification transformation (Ct values into geq./ml) or to the variance of the reported real-time RT-PCR results (Ct values), the viral load variance of the quantitative results described above were recalculated from geq./ml to Ct-values (with arbitrary 100% efficiency, 1 log∼3.3Ct), equalling 11–16 ΔCt. This variation is comparable to the semi-quantitative data of the EQA panel (15.2–21.5 ΔCt for samples #13 and #6 or #9, respectively) and the wide range in reported loads can therefore not be attributed to the quantification transformation.

Table 2.

MERS-CoV specific sensitivity results of the first MERS-CoV EQA panel 2014.

				Quantitative RT-PCR		Semi-quantitative RT-PCR		Qualitative RT-PCR
Sample no.	Origin	Dilution	Correct results total	Correct results	Median viral loadb (range)	Correct results	Median Ct-value (range)	Correct results
#13	MERS-CoV	10−2	99.5%	100.0%	7.59 (5.86–9.70)	99.1%	22.2 (16.0–31.2)	100.0%
#9	MERS-CoV	10−3	99.5%	100.0%	6.79 (3.99–8.70)	100.0%	25.1 (15.0–36.5)	97.8%
#2	MERS-CoV	10−4	98.9%	100.0%	5.88 (3.78–7.26)	99.1%	29.0 (22.0–37.7)	97.8%
#5	MERS-CoV	10−4	98.4%	100.0%	5.78 (3.63–7.08)	99.1%	28.9 (23.0–39.7)	95.6%
#6	MERS-CoV	10−5	96.3%	100.0%	4.84 (2.91–7.70)	96.5%	31.7 (18.0–39.5)	93.3%
#10	MERS-CoV	10−6	84.6%	46.2%	3.83 (1.97–5.46)	87.7%	35.5 (28.6–43.6)	73.3%
#8a	MERS-CoV	10−7	56.5%	43.3%	2.98 (0.56–6.82)	54.4%	37.7 (32.0–41.5)	51.2%

^aSample #8 is omitted for further analysis, since the viral load reached the Poisson distribution-level.

^bViral load in log copies/ml.

Furthermore, we have assessed the difference in yield of specific targets for which more than 8 returned panel results were available (Table 1), to advice on the best targets to use, based on Ct-values. The distributions of Ct values of the semi-quantitative RT-PCR’s targeting UpE [3], ORF1a [4], ORF1b [3], N2 [6], and N3 [6] were compared using Students T-test, after checking that variances were similar (by using F-test)(Table 3 ). The Ct values produced by the ORF1b RT-PCR were significantly higher than those of all other targets in the comparison, whereas no significant differences were observed for the other four targets.

Table 3.

Significance levels of compared semi-quantitative RT-PCR data (Ct-values^a) of most used PCR targets.

Target 1 compared to	Target 2	T-test (pb)	F-test (p-valueb)
upE	ORF1a	0.11	0.59
upE	ORF1b	0.00*	0.75
upE	N2	0.30	0.69
upE	N3	0.08	0.95
ORF1a	ORF1b	0.00*	0.48
ORF1a	N2	0.85	0.69
ORF1a	N3	0.62	0.74
ORF1b	N2	0.00*	0.83
ORF1b	N3	0.00*	0.79
N2	N3	0.56	0.95

^*Significant.

^aMedian Ct values (range) upE 28.7 (16.7–41.7); ORF1a 27.65 (17.3–42.9); ORF1b 32.45 (17.3–43.6); N2 28.35 (15.0–37.6); N3 27.3 (16.0–39.0).

^bSignificance levels were calculated without the data of sample 8, since MERS-CoV was not equally distributed.

4.2. Specificity

To assess the specificity of MERS-CoV molecular diagnostics, two negative controls (sample #1 and #4) and the four known common human coronaviruses, namely HCoV-HKU1 (sample #3), HCoV-OC43 (sample #7), HCoV-229E (sample #11) and HCoV-NL63 (sample #12) were part of the first MERS-CoV EQA panel. HCoV-HKU1 and HCoV-OC43 are beta-coronaviruses similar to MERS-CoV. Two laboratories reported MERS-CoV RNA detected in one of the negative controls (2%). Additionally, 6.1% of the laboratories reported detection of MERS-CoV RNA in the samples spiked with the non-MERS HCoV, of which three in the HCoV-NL63 sample and one laboratory in the HCoV-229E sample alone. However, one laboratory reported MERS-CoV RNA detected in both HCoV-229E and HCoV-HKU1 samples, and one laboratory in the samples containing HCoV-NL63, HCoV-229E and HCoV-HKU1. The false positive results were not associated with a specific assay, as other laboratories using the same protocols did not report this reactivity. This may suggest that these aspecific results originated from contamination rather than non-specific reactions with non-MERS HCoVs or from a different interpretation algorithm of the raw data. Nevertheless, in total 8.1% of the laboratories reported false-positive results.

4.3. Testing for common human coronaviruses

In total, 34 laboratories reported results for the four common hCoV samples. The median (range) Ct-values were calculated from the 11 returned panel results, being 27.7 (20.0–34.0) for HCoV-HKU1, 27.9 (18.0–33.4) for HCoV-OC43, 28.0 (18.0–35.0) for HCoV-229E, and 28.5 (20.0–36.0) for HCoV-NL63. Some of these laboratories missed one or more common HCoVs, being 23.5% HCoV-HKU1, 2.9% HCoV-OC43, 12.1% HCoV-229E, and 12.1% for HCoV-NL63. One laboratory reported having performed the molecular diagnostics for the non-MERS CoVs as well, but was not able to detect any of the common human coronaviruses.

5. Discussion

Since the discovery of MERS-CoV in 2012, diagnostic protocols were quickly deployed globally. In this study, 99 of 106 participants have returned (multiple) results on the first MERS-CoV EQA assessment, with good overall results. A quarter of the laboratories (26.2%) reported quantitative results, though the variance in reported viral concentration was high. This may be due to the fact that there is no international NAT standard (IS) available. Using inactivated complete virus control (provided by the RKI) is recommended, since this takes the variations of the nucleic acid extraction as well as RT-PCR variations into account. However, using a quantified RNA control, provided by European Viral Archive (EVA) merely quantifies the amount of RNA added to the RT-PCR, but does not take variations induced by nucleic acid extractions into account. Though, differences in performance of RT-PCR mixes can be measured using RNA controls. It is known that the variation in viral loads quantified by in-house standards is much higher than if laboratories world-wide use one NAT IS, which can be handled in the exact way as clinical sample (QCMD CMV panels, unpublished results).

In this MERS-CoV EQA panel, almost 90% of the reported results were obtained by in-house real-time RT-PCR, of which the assays published by Corman et al. [3], Corman et al. [4], and Lu X et al. [6] were mostly used. Only one laboratory reported the use of an unpublished assay targeting ORF16 and scored 100%, indicating a well validated in-house assay. A minority of laboratories used other techniques than real-time RT-PCR, like RT-RPA and conventional RT-PCR for MERS-CoV detection. The recommendation of WHO to use confirmatory RT-PCR on a second MERS-CoV genomic target was followed by a majority of the laboratories, though multiple laboratories indicated to have sequencing available for confirmation. Currently there are a number of MERS-CoV specific commercial real-time RT-PCR assays marketed, having the advantage of included (internal and positive) controls in a single kit whose quality is regulated by authorities. Of the reported commercial real-time RT-PCRs (n  = 16), most used were Fast Track Diagnostics (3 labs, 100% score), Genesig Primer Design Path-HCoV-2012 (3 labs, 83–92% score) and Real Star MERSCoV Altona (3 labs, 100% score).

Furthermore, we investigated if there was a difference (in Ct level) between the published targets, taken the reported panel results into account. We found that the Ct values produced by the ORF1b RT-PCR were significantly higher than those of all other targets in the comparison, whereas no significant differences were observed for the other four targets. This combined with the lower number of correct ORF1b results observed in the data (not shown), indicates that ORF1b is the weakest RT-PCR target of the published RT-PCRs, as was suggested by the publication of Corman et al. [4].

The composition of this MERS-CoV EQA panel does not completely reflect the real life performance when testing clinical samples, it merely gives an indication on analytical sensitivity and specificity of the molecular diagnostic test used. Interpreting the results, one needs to realize that the viral concentration in MERS-CoV infected patients in throat swabs, serum and faeces may be in the lower range (>Ct30), depending on timing of sampling. MERS-CoV has not been detected in upper respiratory tract samples (nose swabs) in some infected humans, suggesting the potential for missed cases if only such specimens are collected [12], [13]. In lower respiratory tract samples (trachea aspirates, broncho-alveolar lavages) MERS-CoV concentrations were found to be significantly higher than in upper or middle respiratory tract samples [14], [15], though these samples may not be easy to take from patients with respiratory distress. Taken the fact that most diagnostic samples sent for MERS-CoV molecular diagnostics are throat swabs containing low MERS-CoV RNA concentrations, and that 84.6% of the reported results were correct for sample 10, having the lowest concentration in the panel (median viral load 3.83E3 geq/ml), this may reflect underdiagnosis of MERS-CoV infections due to sensitivity problems in the diagnostic method used by a small portion of the laboratories.

The first external quality assessment MERS-CoV panel gives a good insight in molecular diagnostic techniques and their performances for sensitive and specific detection of MERS-CoV RNA globally. The overall score was good, though for sensitive detection in clinical samples the laboratories are advised to validate another confirmatory target than ORF1b and critically evaluate their results. This survey indicated a robust capability and capacity for detection and confirmation of MERS-CoV in most participating laboratories as of June 2013. The finding that 8.1% of laboratories produced false positive results is disturbing and supports confirmatory testing by either a second RT-PCR target and/or a second laboratory.

Competing interest

None declared.

Funding

This study was funded by the European Center for Disease Prevention and Control (ECDC) by supporting the European Network for Diagnostics of ‘Imported’ Viral Diseases (ENIVD) under the specific contract No.1 ECD.4221-FW/ECDC/2013/012.

Ethical approval

Not required.

Appendix A. Supplementary data

The following are Supplementary data to this article: