Comparison of the analytical sensitivity of seven commonly used commercial SARS-CoV-2 automated molecular assays

Highlights

• •The analytical sensitivities of seven SARS-CoV-2 molecular assays are compared.
• •A single positive clinical specimen quantified by droplet digital PCR was used for the comparison.
• •The Abbott, the Roche, and the Xpert Xpress assays appear more sensitive.

Abstract

The SARS-CoV-2 pandemic has challenged molecular microbiology laboratories to quickly implement and validate diagnostic assays and to expand testing capacity in a short timeframe. Multiple molecular diagnostic methods received FDA emergency use authorization (EUA) and were promptly validated for use nationwide. Several studies reported the analytical and/ or clinical evaluation of these molecular assays, however differences in the viral materials used for these evaluations complicated direct comparison of their analytical performance. In this study, we compared the analytical sensitivity (lower limit of detection, LOD) of seven commonly used qualitative SARS-CoV-2 molecular assays: the Abbott Molecular RealTime SARS-CoV-2 assay, the NeuMoDx™ SARS-CoV-2 assay, the Roche Cobas®SARS-CoV-2 assay, the BD SARS-CoV-2 reagents for BD MAX™ system, the Hologic Aptima® SARS-CoV-2 assay, the Xpert Xpress SARS-CoV-2 test, and the GenMark ePlex SARS-CoV-2 test. The comparison was performed utilizing a single positive clinical specimen that was serially diluted in viral transport media and quantified by the EUA approved SARS-CoV-2 droplet digital PCR (ddPCR) assay. Replicate samples were prepared and evaluated for reproducibility across different molecular assays with multiple replicates per assay. Our data demonstrated that the seven assays could detect 100 % of replicates at a nucleocapsid gene concentration of (N1 = 1,267 and N2 = 1,392) copies/mL. At a one log less concentration, the Abbott, the Roche, and the Xpert Xpress assays detected 100 % of the tested replicates.

1. Introduction

The first case of coronavirus disease-19 (COVID-19) was reported to the World Health Organization (WHO) country office in China on December 31st, 2019 [1]. A highly pathogenic novel coronavirus, SARS-CoV-2 was identified and its genome was characterized in respiratory clinical specimens of the first diagnosed COVID-19 patients in China [2,3]. Since that time, cases of COVID-19 have been reported from every country worldwide with over 11 million confirmed cases. A Public Health Emergency of International Concern was declared by the WHO on January 30th, 2020 with a Public Health Emergency declared in the US on January 31st. On February 4th, the US Food and Drug Administration (FDA) granted an Emergency Use Authorization (EUA) to the Center of Disease and Control (CDC)’s diagnostic molecular assay. On February 29th, the FDA updated their guidance to allow clinical microbiology laboratories and commercial parties to develop in vitro SARS-CoV-2 assays and seek independent EUA [4]. Beginning in March, the number of SARS-CoV-2 molecular diagnostic assays with EUA increased rapidly, many of them being developed for existing, automated and integrated (nucleic acid extraction and RT-PCR) diagnostic systems [4].

Molecular detection of SARS-CoV-2 viral RNA has been used as the gold standard for diagnosis of COVID-19 [5]. Hence, an understanding of the analytical and clinical performance of various SARS-CoV-2 molecular methods is essential. In particular, the analytical sensitivity limitations and the relative sensitivity of different assays is important due to the large number of individuals that may carry transmissible virus asymptomatically in low copy number [6]. In addition, the recognition of initially false negative molecular testing in some patients who ultimately tested positive further highlights the importance of understanding the limit of detection [7]. The initial development of SARS-CoV-2 molecular assays was challenging due to the paucity of clinical specimens, viral controls, and/or material reagents for the validations. Different companies and laboratories used different materials for assessing the analytical and clinical sensitivities of their assays which were quantified differently. For example, the Abbott Molecular RealTime SARS-CoV-2 and Xpert Xpress SARS-CoV-2 assays both utilized recombinant virus for analytical sensitivity determination, while the NeuMoDx™ SARS-CoV-2 Assay utilized genomic RNA, and the GenMark ePlex SARS-CoV-2 test utilized in vitro transcripts for analytical sensitivity determination. These differences, coupled with differences in sample volume, extraction volume and amplification technology has made direct comparison across platforms challenging. Utilizing SARS-CoV-2 droplet digital PCR (ddPCR) [8] quantified single positive clinical specimen, we compared the analytical sensitivity of seven SARS-CoV-2 EUA-granted assays, the Abbott Molecular RealTime, the NeuMoDx™, the Roche Cobas®, the BD SARS-CoV-2 reagents for BD MAX™ system, the Hologic Aptima®, the Xpert Xpress, and the GenMark ePlex SARS-CoV-2 tests. Comparisons were performed using the same quantified specimen and were facilitated by the in-house availability of these assays for SARS-CoV-2 diagnosis at Johns Hopkins Hospital Clinical Laboratory and the Johns Hopkins Center for Innovative Diagnostics for Infectious Diseases.

2. Methods

2.1. Standard and serial dilution preparation

The study was approved by the Johns Hopkins University School of Medicine IRB (IRB00246024). A SARS-CoV-2 positive nasopharyngeal specimen diagnosed at the molecular virology laboratory at John Hopkins Hospital by our original standard-of-care assay, the RealStar® SARS-CoV-2 RT-PCR (Altona Diagnostics, Hamburg, Germany) was used as a standard (Spike target cycle threshold [Ct] = 15) [9]. The specimen was diluted serially in viral transport media (VTM, Copan Diagnostics, Murieta, CA) and dilutions of 1:10⁵ and 1:10⁶ were quantified in duplicates using the EUA approved BioRad ddPCR assay per EUA package insert with the exception of using an automated extraction using the NucliSENS easyMag (bioMérieux, Marcy-l'Étoile, France) as described before [4,10] The average concentration of both the N1 target gene and N2 target gene copies at each dilution was determined to be 10,956 and 10,759 copies/mL (1:10⁵ dilution) and 1,267 and 1,392 copies/mL (1:10⁶ dilution), respectively. An additional dilution was prepared and the concentration was estimated to be 126.7 and 139.2 copies/ mL for both the N1 and N2 gene targets. Replicate samples were prepared for each of these concentrations and ranged from a maximum of ten replicates for the Abbott Molecular SARS-CoV-2 assay (Abbott Molecular, Des Plaines, Il) and NeuMoDx™ SARS-CoV-2 assay (NeuMoDx, Ann Arbor, Michigan) to a low of 3 replicates for the Xpert Xpress SARS-CoV-2 test (Cepheid, Sunnyvale, CA) and the ePlex SARS-CoV-2 test (GenMark, Carlsbad, CA). The number of replicates varied due to the availability of kit reagents and materials necessary to perform each assay.

2.2. Sample evaluation

For each assay, testing was performed per manufacturer instructions: Abbott Molecular RealTime SARS-CoV-2 assay (Abbott Molecular, Des Plaines, Il) [11], NeuMoDx™ SARS-CoV-2 assay (NeuMoDx, Ann Arbor, Michigan) [12], Roche Cobas®SARS-CoV-2 assay [13], Becton Dickinson BD Max reagents (Becton Dickinson, Sparks, MD) [14], Hologic Aptima® SARS-CoV-2 assay (Hologic, Bedford, MA) [15], Cepheid Xpert Xpress SARS-CoV-2 test (Cepheid, Sunnyvale, CA) [16], GenMark ePlex SARS-CoV-2 test (GenMark, Carlsbad, CA) [17]. Although 10 replicates were tested for the majority of platforms, the number of assays performed varied based on availability of reagents and replicates.

3. Results

Results for each assay, the Abbott Molecular RealTime SARS-CoV-2, the NeuMoDx™ SARS-CoV-2, the Roche Cobas®SARS-CoV-2, the BD SARS-CoV-2 reagents for BD MAX™ system, the Hologic Aptima® SARS-CoV-2, the Xpert Xpress SARS-CoV-2, and the GenMark ePlex SARS-CoV-2 tests are summarized in Table 1 . Replicates at the most diluted set (N1 = 126.7 copies/ mL and N2 = 139.2 copies/ mL) were detected 100 % of the time with the Abbott RealTime SARS-CoV-2 assay and the Roche Cobas®SARS-CoV-2 assay. The Xpert Xpress SARS-CoV-2 test did not detect the N2 target in one replicate at this concentration, however because it did detect the E target at that concentration, the result was reported positive. All the tested assays were able to detect all the tested replicates at a one log higher concentration of 1,267 and 1,392 copies/mL for the N1 and N2 gene targets.

Table 1.

Relative Sensitivities of 8 FDA EUA SARS-CoV-2 Molecular Test Platforms.

Platform/Assay	Sample Input* Volume (μl)	Concentrationa	Gene Target	Detected/ Total	%b	Average Ct	Stdevc
Abbott Molecular/ RealTime SARS-CoV-2 assay	600	1,267	RdRp/N	10/10	100	23.71	0.29
		126.7	RdRp/N	10/10	100	27.03	0.65
NeuMoDx™ SARS-CoV-2 Assay	500	1,267	N	5/5	100	33.19	0.23
			Nsp2	5/5	100	33.49	0.36
		126.7	N	8/10	80	33.90	0.33
			Nsp2	3/10	30	33.51	0.73
Roche/ Cobas®SARS-CoV-2 Test	800	1,267	ORF1ab	3/3	100	32.85	0.02
			E	3/3	100	33.61	0.03
		126.7	ORF1ab	9/9	100	34.95	0.81
			E	9/9	100	36.25	1.09
BD SARS-CoV-2 Reagents for BD MAX™ System	750	1,267	N1	3/3	100	32.83	0.15
			N2	3/3	100	32.10	0.26
		126.7	N1	4/8	50	37.33	0.70
			N2	7/8	87.5	36.03	1.91
Hologic/ Aptima® SARS-CoV-2 Assay	500	1,267	Orf1ab	3/3	100	N/A	N/A
		126.7		7/8	87.5	N/A	N/A
Xpert Xpress SARS-CoV-2	300	1,267	E	3/3	100	NC	NC
			N2	3/3	100	NC	NC
		126.7	E	3/3	100	40.20	2.63
			N2	2/3	66.67	40.35	2.33
GenMark/ ePlex SARS-CoV-2 Test	200	1,267	N	3/3	100	N/A	N/A
		126.7		1/3	33.33	N/A	N/A

^aCopies/mL determined by ddPCR for the N1 target gene.

N/A – not applicable, NC – not calculated.

^bPercentage of replicates detected.

^cStandard deviation.

^*The input volume is based on the volume loaded per specimen and not the volume used per reaction.

4. Discussion

In this study, we compared the analytical sensitivities of seven common, commercially available assays for the molecular detection of SARS-CoV-2. Our comparison was performed using the same quantified clinical specimen allowing parallel evaluation. Our results show higher analytical sensitivities of the Roche (Cobas), the Abbott, and the GeneXpert (Xpert Xpress) assays when compared to the NeuMoDx, the BD MAX, the GenMark (ePlex), and the Hologic (Aptima) assays. The analytical sensitivities of all the assays assessed were in a range less than 1,000 copies/ mL indicating excellent analytical performance.

In general, the analytical performance of molecular assays is usually determined by the selection of the assay, the assay design and the selected target. Most of the current molecular diagnostics for SARS-CoV-2 are real-time PCR (RT-PCR) assays, however technologies that include CRISPR (clustered regularly interspaced short palindromic repeats)-based [18] and Loop Mediated Amplification (LAMP) assays have also been developed [19,20]. Most of the assays compared in our study are RT-PCR methods with the exception of the GenMark (ePlex) that involves an initial amplification followed by hybridization and electrochemical detection and the Hologic (Aptima) that uses a transcription-mediated amplification (TMA) technology. Our study identified only slight differences in the analytical sensitivity of the RT-PCR methods and comparable sensitivities using the GenMark (ePlex) and Hologic (Aptima) assays, indicating comparable performance. The assays we evaluated also target different genes in the SARS-CoV-2 genome, mainly the nucleocapsid (N), envelope (E), non-structural protein (Nsp)2, and open reading frame ORF1/2 (Table 1). The N1 and N2 targets within the N gene were recommended by the CDC [21,22] and were selected for their assay, while the WHO recommends an initial screening with the E gene followed by confirmation with the RNA dependent RNA polymerase (RdRp) [23]. Our data, as well as other published data illustrate that the assay performance for SARS-CoV-2 detection is not dictated by the selected gene. However, a recent report concluded higher sensitivity of primers that target the N2 or the E genes [24]. The approach of targeting two genes appears to enhance the assays’ sensitivities (Table 1) and could also reduce the risk of sensitivity reduction associated with genomic polymorphisms [25].

Understanding the analytical performance of SARS-CoV-2 assays is essential for better understanding their clinical performance. Standardized materials for determining the analytical sensitivity or lower limit of detection (LOD) for EUA applications and validations were not available and different companies and laboratories used different approaches that included using in vitro synthesized transcripts (IVT) (e.g. GenMark), cultured virus (e.g. Roche), or SARS-CoV-2 genomic RNA (e.g. NeuMoDx) (Table 2 ). As a step forward for optimizing the validations and collecting comparable assays’ performance data, the FDA currently offers reference panels for EUA applications which should also assist in different assays’ calibrations and on-going monitoring of the performance [26]. Although several papers have been published comparing the characteristics and sensitivities between a few SARS-CoV-2 molecular tests (Table 2) and [10,[27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38]], our study provides a direct comparison between 7 of the most commonly used commercially available assays using the same quantified clinical specimen.

Table 2.

A literature and manufacturer based summary of sensitivity data.

Assay	Assay time	LOD/ materials used/ reference	References
GenMark/ Eplex SARS-Cov-2 Test	1 h 42 min	1 × 105 copies/ mL/ in vitro transcripts / per EUA PI 1,000 copies/mL/ Exact Diagnostics SARS-CoV-2 standard [27,31] 600 copies/mL/ Genomic viral RNA [10]	[10,27,31]
Xpert Xpress SARS-Cov-2	46 minutes	250 copies/ mL/ AccuPlex SARS-CoV-2 reference material/ per EUA PI 100 copies/mL / Exact Diagnostics [31]	[30,31,39]
Roche/ Cobas®SARS-Cov-2 Test	384 specimens/8 h, 3.75 h/ run	0.007 TCID50/mL for SARS-CoV-2 (Orf1ab) and 0.004 TCID50/mL for pan-Sarbecovirus (E) / cultured virus/ per EUA PI	[30,32,37,[39], [40], [41]]
Abbott Molecular/ Realtime SARS-CoV-2 Assay	24−94 samples/run, 6.8 h/run	100 copies/ mL/ Seracare, AccuPlex COVID-19/ per EUA PI	[29,34,36]
Hologic Aptima® SARS-Cov-2 Assay	275 samples/ 8 h, 3.5 h/ run	0.01 TCID50/mL/ inactivated cultured SARS-CoV-2 virus/ per EUA PI	[42]
NeuModx™ SARS-CoV-2 Assay	96 or 288 samples/ 8 h, 70 min to first result	150 copies/mL/ genomic RNA/ per EUA PI
BD SARS-Cov-2 Reagents For BD MAX™ System	24 samples/ 3 h	40 GE/ mL/ genomic RNA/ per EUA PI

NP-nasopharyngeal, PI- package insert, GE- genomic equivalent.

There are limitations to our study, including differences in the sample input volume for each assay, differences in the amount of sample processed and analyzed by the assay and differences in the extraction efficiency between assays, which cannot accurately be accounted for, and thus complicate the comparison. Additionally, we did not evaluate the same number of replicates for each assay due to the availability of supplies and replicates as we used dilutions of one specimen.

We demonstrated similar analytical performance of seven FDA EUA commercial SARS-CoV-2 molecular assays; the most sensitive platforms were the Roche and Abbott. With the critical current and future need for the use of molecular SARS-CoV-2 diagnostics not only for diagnosis, but also for asymptomatic large scale screening, it is essential to identify the most sensitive assays to assure early detection and diagnosis of COVID-19, and inform decisions related to contact isolation, measures that are essential to mitigate the current pandemic.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.