Comparison of eleven in vitro diagnostic assays for the detection of SARS-CoV-2 RNA

Abstract

Transmission mitigation of SARS-CoV-2 requires the availability of accurate and sensitive detection methods. There are several commercial ad hoc molecular diagnostic kits currently on the market, many of which have been evaluated by different groups. However, in low resource settings the availability and cost of these commercial kits can be a limiting factor for many diagnostic laboratories. In such cases alternatives need to be identified. With this in mind, eight commercial reverse transcription quantitative real-time PCR (RT-qPCR) master mixes from Applied Biosystems (Thermo Fisher Scientific), Bio-Rad, Biotech Rabbit, Promega, Qiagen, QuantaBio, Invitrogen (Thermo Fisher Scientific) and Takara using the same commercial primer and probe mix [LightMix® Modular SARS and Wuhan CoV E-gene mix (TIB MolBiol, Germany)] were evaluated. Three ad hoc molecular diagnostic kits [GeneFinder™ COVID-19 Plus RealAmp kit (Osang Healthcare); genesig® Real-Time PCR Coronavirus COVID-19 (Primerdesign); and ViroReal® Kit SARS-CoV-2 & SARS-CoV (Ingenetix)] were also included in the study. The limit of detection was calculated for each assay using serial dilutions of a defined clinical sample. The performances of the assays were compared using a panel of 178 clinical samples and their analytical specificity assessed against a panel of human betacoronaviruses. Inter assay agreement was assessed using statistical tests (Bland-Altman, Fleiss-Kappa and Cohen’s Kappa) and was shown to be excellent to good in all cases. We conclude that all of the assays evaluated in this study can be used for the routine detection of SARS-CoV-2 and that the RT-qPCR master mixes are a valid alternative to ad hoc molecular diagnostic kits.

1. Introduction

SARS-CoV-2 is member of the genus Betacoronavirus within the family Coronaviridae. It is the third member of the family after Severe Acute Respiratory Syndrome Coronavirus (SARS-CoV) and Middle Eastern Respiratory Syndrome Coronavirus (MERS-CoV) to cause severe disease in humans (Coronaviridae Study Group of the International Committee on Taxonomy of Viruses, 2020). The virus has a positive-sense single stranded RNA genome of 29,881 bp in length, consisting of eleven genes. The ORF1ab gene encoding the RNA-dependent RNA polymerase (RdRp), the envelope (E) gene, the nucleocapsid (N) and the spike (S) genes are the most common targets for viral detection by RT-PCR (Corman et al., 2020; Ravi et al., 2020).

As SARS-CoV-2 continues to infect millions of individuals globally, the rapid and accurate confirmation of infections and follow-up of contacts is essential. Testing is a key part of the Strategic Preparedness and Response Plan devised by the World Health Organization at the beginning of the pandemic (WHO, 2020a). RT-qPCR is the most widely used diagnostic test for COVID-19 diagnosis (Corman et al., 2020; Ravi et al., 2020) and, despite the introduction and frequent use of antigen (Ag) lateral flow tests, retains its importance for the confirmation of reactors in Ag-tests and the detection of asymptomatic carriers. As a result, demand for molecular diagnostic kits and reagents has increased enormously and widespread shortages have been reported (ASM, 2020). When the preferred or recommended kits are not available to laboratories, diagnostic alternatives must be identified. This can be difficult particularly in low resource settings where the costs of material and their availability can be a limiting factor.

As of April 2021, the Foundation for Innovative New Diagnostics website lists 141 molecular assays that are commercialized for the diagnosis of COVID-19 (FIND, 2020). Several groups have evaluated this growing list of commercially available kits, many of which have been deemed fit for purpose (Hur et al., 2020; Iglói et al., 2020; van Kasteren et al., 2020; Smith et al., 2020; Zhen et al., 2020a, b; Lu et al., 2020; Poljak et al., 2020; Lieberman et al., 2020; Eberle et al., 2021). However, the evaluation of ready-to-use RT-qPCR master mixes does not appear to have been considered. In this study, eight commercially available RT-qPCR master mixes have been assessed and compared to each other. Three commercially available ad hoc kits [GeneFinder™ (Osang Healthcare); COVID-19 genesig® Real-Time PCR assay (Primerdesign); and Viroreal® (Ingenetix)] were included in this evaluation for additional comparisons.

2. Methods

2.1. Specimen collection and storage

SARS-CoV-2 RNA positive upper respiratory swab samples (collected in universal transport medium or saline solution) with widely varying quantification cycle (Cq)-values were selected from clinical samples received in March 2020 within the frame of COVID-19 diagnostic testing in Austria. Samples were classified as SARS-CoV-2 RNA positive upon initial screening by RT-qPCR, using the LightMix® Modular SARS and Wuhan CoV E-gene mix (TIB MolBiol, Germany) and SuperScript™ III Platinum® One-Step qRT-PCR System with ROX (Invitrogen, Thermo Fisher Scientific, Austria).

Upon completion of diagnostic testing, an aliquot of each positive respiratory swab supernatant (n = 144) was stored at −80 °C. Negative SARS-CoV-2 RNA samples (n = 34) were from upper respiratory swab samples collected during a screening programme targeting Austrian residential/nursing homes, performed in August 2020. The RT-qPCR negative samples were selected only from completely negative cohorts.

2.2. RNA extraction

In August 2020, RNA was prepared from archived SARS-CoV-2 positive and freshly received SARS-CoV-2 negative samples using a commercial magnetic-bead based extraction kit (BioExtract® SuperBall®, BioSellal, France) on the KingFisher™ Flex Purification System (Thermo Fisher Scientific, Austria). Input sample volume was 200 μL; magnetic-bead bound nucleic acids were dissolved in 150 μL of the elution buffer provided with the kit. Eluted RNA from each sample was aliquoted in triplicate in three 96-well PCR plates in order to avoid multiple cycles of freeze-thawing and was shipped on dry ice to the Animal Production and Health Laboratory, Seibersdorf, Austria. Samples were coded and submitted anonymously.

2.3. Primers, probes and real-time instrument

The LightMix® Modular SARS and Wuhan CoV E-gene (TIB MolBiol, Berlin, Germany) containing E gene primers and a FAM-labelled probe was used with the RT-qPCR master mixes (this primer/probe mix, which identifies both SARS-CoV and SARS-CoV-2, has been recently shown to be fit for purpose by Yip et al. (2020). All RT-PCRs were run on a CFX96 Touch Deep Well Real-Time PCR Detection System (Bio-Rad).

2.4. RT-qPCR master mixes

Eight commercial RT-qPCR master mixes were compared (Table 1 and supplementary Table S1).

Table 1.

Details of RT-qPCR Master mixes and ad hoc kits tested.

	Name	Target Gene*	Company
1	TaqMan™ Fast Advanced Master Mix	E	Applied Biosystems/Thermo Fisher Scientific, USA
2	iTaq™ Universal Probes One-Step Kit	E	Bio-Rad, USA
3	Capital™ qPCR Probe Mix,	E	BiotechRabbit, Germany
4	GoTaq® qPCR Master Mix	E	Promega, USA
5	One-Step RT-PCR Kit	E	Qiagen, Germany
6	UltraPlex 1-Step ToughMix	E	QuantaBio, USA
7	One-step Prime ScriptIII RT-PCR Kit	E	Takara, Japan
8	SuperScript™ III One-Step RT-PCR System	E	Invitrogen/Thermo Fisher Scientific, USA
9	ViroReal® Kit SARS-CoV-2 & SARS-CoV	N	Ingenetix, Austria
10	GeneFinderTM COVID-19 Plus RealAmp kit	RdRp, N and E	Osang Healthcare, South Korea
11	genesig® Real-Time PCR Coronavirus COVID-19	Orf1 ab	Primerdesign Ltd, United Kingdom

^*LightMix® Modular SARS and Wuhan CoV E-gene.

2.4.1. TaqMan fast virus 1-Step master mix (Applied Biosystems, cat. 4,444,434)

A 20 μL PCR final reaction volume contained 5 μL of 4X Reaction Mix, 0.5 μL of LightMix® Modular SARS and Wuhan CoV E-gene, 9.5 μL of nuclease-free water and 5 μL of sample RNA. The thermocycling conditions consisted of a reverse transcription at 50 °C for 5 min, followed by denaturation at 95 °C for 20 s and 45 cycles of 94 °C for 15 s and 58 °C for 30 s. Fluorescence was read in the FAM channel.

2.4.2. iTaq™ universal probes one-step kit (Bio-rad, cat. 1,725,141)

A 20 μL PCR final reaction volume contained 10 μL of 2X iTaq Master Mix, 0.5 μL of LightMix® Modular SARS and Wuhan CoV E-gene, 4.25 μL of nuclease-free water, 0.25 μL iScript Reverse Transcriptase and 5 μL of sample RNA. The thermocycling conditions consisted of a reverse transcription at 50 °C for 20 min, followed by denaturation at 95 °C for 5 min and 45 cycles of 94 °C for 10 s and 58 °C for 30 s. Fluorescence was read in the FAM channel.

2.4.3. Capital™ 1-Step qRT-PCR probe master mix (biotech rabbit, cat. BC0503202)

A 20 μL PCR final reaction volume contained 4 μL of 5X Probe Mix, 0.5 μL of LightMix® Modular SARS and Wuhan CoV E-gene, 10.5 μL of nuclease-free water and 5 μL of sample RNA. The thermocycling conditions consisted of a reverse transcription at 50 °C for 10 min, followed by denaturation at 95 °C for 3 min and 45 cycles of 95 °C for 10 s and 58 °C for 30 s. Fluorescence was read in the FAM channel.

2.4.4. SuperScript™ III one-step RT-PCR system with platinum™ taq DNA polymerase (Invitrogen™, cat. # 12,574,026)

A 25 μL PCR final reaction volume contained 12.5 μL of 2X Reaction Mix, 0.625 μL of LightMix® Modular SARS and Wuhan CoV E-gene, 4.475 μL of nuclease-free water, 0.4 μL of MgS0₄ (50 mM), 1 μL of BSA (1 mg/mL), 1 μL of SIII/Enzyme and 5 μL of sample RNA. The thermocycling conditions consisted of a reverse transcription at 55 °C for 10 min, followed by denaturation at 94 °C for 3 min and 45 cycles of 95 °C for 15 s and 58 °C for 30 s. Fluorescence was read in the FAM channel.

Note: One of the first qRT-PCR assays for the detection of SARS-Cov-2 was developed by Corman et al. in January 2020 using the SuperScript™ III One-Step RT-PCR System with Platinum™ Taq DNA Polymerase (Invitrogen) and was deemed a recommended protocol by WHO (Corman et al., 2020). An LOD of 5.2 RNA copies/reaction using E gene specific primers and a high specificity using a panel of human respiratory viruses and bacteria in clinical samples was determined. For this reason, this assay was chosen as the reference comparator for correlation studies and intra-assay agreement calculations and is referred to below as the WHO Recommended Protocol (WRP).

2.4.5. GoTaq ® 1-Step RT-qPCR (Promega, Cat. #A6121)

A 20 μL PCR final reaction volume contained 10 μL of 2X Reaction Mix, 0.5 μL of LightMix® Modular SARS and Wuhan CoV E-gene, 4.1 μL of nuclease-free water, 0.4 μL of RT enzyme and 5 μL of sample RNA. The thermocycling conditions consisted of a reverse transcription at 45 °C for 15 min, followed by denaturation at 95 °C for 5 min and 45 cycles of 95 °C for 15 s and 58 °C for 30 s. Fluorescence was read in the FAM channel.

2.4.6. One-Step RT-PCR (Qiagen, Cat. 210,212)

A 20 μL PCR final reaction volume contained 4 μL of 5X Reaction Mix, 0.5 μL of LightMix® Modular SARS and Wuhan CoV E-gene, 8.9 μL of nuclease-free water, 0.8 μL of dNTPs (10 mM), 1 μL of Enzyme and 5 μL of sample RNA. The thermocycling conditions consisted of a reverse transcription at 50 °C for 30 min, followed by denaturation at 95 °C for 15 min and 45 cycles of 95 °C for 15 s and 58 °C for 30 s. Fluorescence was read in the FAM channel.

2.4.7. UltraPlex 1-Step ToughMix (QuantaBio, Cat. 95166−500)

A 20 μL PCR final reaction volume contained 5 μL of 4X UltraPlex 1-Step ToughMix, 0.5 μL of LightMix® Modular SARS and Wuhan CoV E-gene, 9.5 μL of nuclease-free water and 5 μL of sample RNA. The thermocycling conditions consisted of a reverse transcription at 50 °C for 10 min, followed by denaturation at 95 °C for 3 min and 45 cycles of 94 °C for 10 s and 58 °C for 30 s. Fluorescence was read in the FAM channel.

2.4.8. One-step prime ScriptIII RT-PCR kit (Takara, cat. RR600B)

A 20 μL PCR final reaction volume contained 10 μL of 2X Reaction Mix, 0.5 μL of LightMix® Modular SARS and Wuhan CoV E-gene, 4.5 μL of nuclease-free water and 5 μL of sample RNA. The thermocycling conditions consisted of a reverse transcription at 52 °C for 5 min, followed by denaturation at 95 °C for 10 s and 45 cycles of 95 °C for 5 s and 58 °C for 30 s. Fluorescence was read in the FAM channel.

2.5. Ad hoc molecular diagnostic kits

Three ad hoc molecular diagnostic kits were compared (Table 1 and supplementary Table S1).

2.5.1. ViroReal® kit SARS-CoV-2 & SARS-CoV (Ingenetix, cat DHUV02313)

A 20 μL PCR final reaction volume contained 5 μL of RNA Reaction Mix, 1 μL of SARS-CoV-2 & SARS + RNA IPC-3 Assay Mix, 4 μL of nuclease-free water and 10 μL of sample RNA. The thermocycling conditions consisted of a reverse transcription at 50 °C for 15 min, followed by denaturation at 95 °C for 20 s and 45 cycles of 94 °C for 5 s and 60 °C for 30 s. Fluorescence was read in the FAM and Cy5 channels.

2.5.2. Coronavirus COVID-19 genesig® real-time PCR assay (Primerdesign Ltd. Cat. Z-Path−COVID-19-CE)

A 20 μL PCR final reaction volume contained 10 μL of Oasig™ OneStep 2X RT-qPCR Master Mix, 2 μL of COVID-19 and IEC Mix, 3 μL of nuclease-free water and 5 μL of sample RNA. The thermocycling conditions consisted of a reverse transcription at 55 °C for 10 min, followed by denaturation at 95 °C for 2 min and 45 cycles of 95 °C for 10 s and 60 °C for 1 min. Fluorescence was read in the FAM and VIC channel.

2.5.3. GeneFinder-TM COVID-19 plus RealAmp kit (Osang healthcare; EG-IFMR-45)

A 20 μL PCR final reaction volume contained 10 μL of COVID-19 Plus Reaction Mix, 5 μL COVID-19 Plus Probe Mixture, and 5 μL of sample RNA. The thermocycling conditions consisted of a reverse transcription at 50 °C for 20 min; followed by denaturation at 95 °C for 5 min and 45 cycles of 95 °C for 15 s and 58 °C for 60 s. Fluorescence was read in the FAM, VIC, Texas Red and Cy5 channels:

2.6. Analytical sensitivity

A LightMix® Modular SARS and Wuhan CoV E-gene mix of primers and probes (TIB MolBiol, Germany) was added to all of the RT-qPCR Master mixes thereby targeting the E gene of the SARS-CoV-2. In contrast, the Genesig® targeted the Orf1ab gene, the Viroreal® kit targeted the N gene, and the GeneFinder™ kit targeted the E, N, and RdRP genes simultaneously. It was deemed preferable to use a SARS-CoV-2 positive clinical sample (i.e. C043) in order to determine the analytical sensitivity of the kits instead of a synthetic RNA control. The quantity of total SARS-CoV-2 RNA in the clinical sample was determined by constructing a standard curve using dilutions of a synthetic control of known concentration [the EURM-019 synthetic control (JRC, Directorate for Health, Consumers & Reference Materials, Belgium) consisted of a 880 nt long synthetic RNA containing different viral regions of the E, N, RdRP, and S genes]. The analytical sensitivity of the method was assessed by amplifying six different concentrations (50, 25, 12.5, 6.25, and 3.12 copies/reaction) of quantified SARS-CoV-2 RNA from the clinical sample C043. The limit of detection (LOD) of each kit was evaluated by testing these dilutions in quadruplicate on four separate occasions for a total of 16 replicates. The lower detection limit (LOD) of each assay was determined by probit regression analysis (figures available on request).

The STATGRAPHICS Centurion XV Version 15.2.12 software package (StatPoint Technologies, Warrenton, VA, USA) was used to calculate the predicted detection limits and their confidence limits (CL) using probit analysis and to plot the dose-response curves.

2.7. Comparative analysis and analytical specificity

A panel of 178 clinical samples consisting of positive and negative samples as confirmed by the Department for Molecular Biology at the Austrian Institute for Veterinary Disease Control Mödling (AGES) was tested using each of the RT-qPCR master mixes and commercial kits. AGES has been involved in SARS-CoV-2 diagnosis at a national level since March 2020 and has successfully participated in at least three external SARS-CoV-2 PCR proficiency testing schemes organized by the Medical University of Vienna and by INSTAND (Gesellschaft zur Förderung der Qualitätssicherung in medizinischen Laboratorien e.V, Düsseldorf, Germany).

In addition, RNA from a panel of human coronaviruses consisting of SARS-CoV, human coronavirus 229E, human coronavirus OC43, and MERS-CoV RNA (Vircell, Spain) were tested to determine the cross-reactivity of the assays.

2.8. Correlation and inter-assay agreement

Data distribution was checked using the Shapiro-Wilk normality test implemented in R base and the ggqqplot function of the ggpubr package in R. The cor.test function was used to test the association/correlation between test assays and the WRP (Pearson’s product-moment correlation). For Capital™ 1-Step qRT-PCR Probe Master Mix (Biotech Rabbit) and GeneFinder™(Osang Healthcare), for which the Cq data were not normally distributed, Spearman’s rank correlation was used instead.

To compare the various diagnostic assays two approaches were used; (1) The Fleiss' kappa test was performed on categorical data (positive or negative) to assess the reliability of agreement between all the kits using the Interrater Reliability (irr) package in R. The individual assay agreement with the WRP was determined through the Cohen's kappa coefficient (κ) and confidence bands, using the Visualizing Categorical Data (vcd) package in R; (2) Bland Altman plots (Bland and Altman, 1995)., were produced using R (Hilfiger, 2015), to visually assess the agreement between each assay and the WRP, using the Cq values as numerical data.

3. Results

3.1. Analytical sensitivity

Probit analysis was used to estimate the LOD based on serial 2-fold dilutions (50, 25, 12.5, 6.25, and 3.12 copies/reaction) tested 16 times. When considering the RT-qPCR Master mixes only, there was a 4-fold difference between the lowest LOD values of 5.12 copies/reaction for the One-Step RT-PCR kit (Qiagen) and the highest of 20.89 for the GoTaq® Probe 1-Step RT-qPCR System (Promega) (Table 2 ). The LOD values of the ad hoc kits ranged from 6.37 (GeneFinder™ COVID-19 Plus RealAmp kit) to 25.20 (ViroReal®).

Table 2.

Estimated LOD in copies/reaction for individual assays.

qPCR Ready Mix/Kit	LOD*
TaqmanTM Fast Advanced Master Mix	8.19 (6.57–11.75)
CapitalTM qPCR Probe Mix	5.50 (4.29–8.55)
iTaq™ Universal Probes One-Step Kit	8.51 (6.69–12.67)
GoTaq® qPCR Master Mix	20.89 (15.98–32.59)
One-Step RT-PCR Kit	5.12 (3.85–8.57)
UltraPlex 1-Step ToughMix	6.37 (5.21–9.93)
One-step Prime ScriptIII RT-PCR Kit	5.21 (4.26–7.72)
SuperScript™ III One-Step RT-PCR System	6.38 (5.02−9.69)
ViroReal® Kit SARS-CoV-2 & SARS-CoV#	25.20 (19.90−36.56)
GeneFinderTM COVID-19 Plus RealAmp kit$	6.37 (5.07–9.57)
genesig® Real-Time PCR Coronavirus COVID-19@	15.73 (11.85–25.52)

^*Positivity considered based on amplification of E gene except ^#, N gene; ^@, Orf1ab; ^$, E gene, Orf1ab and N gene.

3.2. Correlation and interassay agreement

Overall, there was a remarkable similarity in the distribution of the Cq values of the 178 tested samples across various kits (Fig. 1 ). The analysis showed that the Cq values were normally distributed for all the assays, except the Capital™ qPCR Probe Mix (Biotech Rabbit) and the GeneFinder™ (Osang Healthcare) kit. The correlation analysis showed that all the tested RT-qPCR master mixes’ and kit's Cq values were significantly correlated to those of the comparator (WRP) with correlation coefficients ranging from 0.925 to 0.973 and a p-value < 2.2e-16 (Table 3 ).

Fig. 1.

Distribution and variability in the Cq values of samples across various test assays. Nucleic acid extracts from 178 clinical samples were tested using eleven assays. The negative amplifications were arbitrarily given the Cq values of 45 for plotting purposes allowing for their visualization above the blue line (set to Cq = 43) (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.).

Table 3.

Interassay agreement calculations using the SuperScript™ III One-Step RT-PCR System (WRP) (Thermo Fisher Scientific) as the comparator.

Kit	Correlation (p-value)	κ-value (95 % CI)	Mean Bias (LLA; ULA)
TaqmanTM Fast Advanced Master Mix	0.962 (< 2.2e-16)	0.916 (0.849−0.982)	0.36 (-2.005; 2.724)
CapitalTM qPCR Probe Mix	0.973(< 2.2e-16) *	0.901 (0.829−0.973)	1.964 (-2.252; 4.18)
iTaq™ Universal Probes One-Step Kit	0.959 (< 2.2e-16)	0.930 (0.870−0.990)	1.263 (-1.188; 3.715)
GoTaq® qPCR Master Mix	0.937 (< 2.2e-16)	0.863 (0.780−0.945)	−0.908 (-3.95; 2.134)
One-Step RT-PCR	0.957 (< 2.2e-16)	0.890 (0.816−0.964)	0.142 (-2.398; 2.638)
UltraPlex 1-Step ToughMix	0.954 (< 2.2e-16)	0.959 (0.912−1)	1.284 (-1.333; 3.901)
One-step Prime ScriptIII RT-PCR Kit	0.925 (< 2.2e-16)	0.866 (0.785−0.946)	−0.751 (-4.054; 2.552)
ViroReal® Kit SARS-CoV-2 & SARS-CoV	0.937 (< 2.2e-16)	0.932 (0.873−0.991)	1.505 (-1.549; 4.558)
GeneFinderTM COVID-19 Plus RealAmp kit	0.942 (< 2.2e-16) *	0.797 (0.704−0.890)	−2.098 (-4.93; 0.733)
genesig® Real-Time PCR Coronavirus COVID-19	0.949 (< 2.2e-16)	0.930 (0.870−0.990)	2.636 (-0.122; 5.394)

LLA = lower limit of agreement; ULA = upper limit of agreement.

^*Spearman's rank correlation rho was used.

There was an excellent agreement (beyond chance) between the eleven assays. This was illustrated by the Fleiss kappa (k) value of 0.871, with a p-value = 0 indicating that the calculated kappa is significantly different from zero. Furthermore, the comparison of each assay with the WRP showed Cohen's kappa coefficient (κ) ranging from 0.797 to 0.959 (Table 3). The Bland-Altman analysis (supplementary files Figure S1) showed a mean bias between each assay ranging from -2.1–2.64. In Fig. 2 , three representative Bland-Altman analysis are shown. The best agreements based on Cq values were between the WRP and the One-Step RT-PCR Kit (Qiagen) followed by GoTaq® qPCR Master Mix (Promega) and the TaqMan™ Fast Advanced Master Mix (Applied Biosystems). The differences between the Cq values of the Takara kit compared the WRP were slightly more evident while the majority of the Cq values from the One-Step RT-PCR Kit (Qiagen), iTaq™ Universal Probes One-Step Kit (Bio-Rad), Capital™ qPCR Probe Mix (Biotech Rabbit), ViroReal® Kit SARS-CoV-2 & SARS-CoV (Ingenetix) and genesig® Real-Time PCR Coronavirus COVID-19 (Primerdesign) assay were lower than the WRP. On the other hand, the Cq values generated by the Genefinder™ COVID-19 Plus RealAmp kit were almost all higher than those of the WRP

Fig. 2.

Bland-Altman plots comparing the WRP to three representative assays. The plots show differences between the Cq values of the WRP and the tested assays against the average of the Cq values: A.WRP vs One-Step RT-PCR (Qiagen), showing an example of almost perfect agreement; B.WRP vs. iTaq™ Universal Probes One-Step Kit (Bio-Rad), showing an example of a tested assay with lower Cq values than the WRP; C.WRP vs GeneFinder™ COVID-19 Plus RealAmp kit (Primerdesign Ltd) showing an example of a tested assay with higher Cq values than the WRP. The red dotted lines represent the lines of identity (i.e. perfect agreement). The grey lines represent the bias between the test assays and the WRP. The grey dotted lines represent the limits (upper and lower) of agreements. Bland-Altman plots for all the tested assays are illustrated in the supplemental material (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.).

3.3. Analytical specificity

None of the assays gave a positive result for human coronavirus 229E, human coronavirus OC43, and MERS-CoV RNA while all, except genesig® Real-Time PCR Coronavirus COVID-19, gave a positive result for SARS-CoV.

4. Discussion

In this study we have compared the performance of both RT-qPCR master mixes and ad hoc kits and have shown that all of the assays in question generated results comparable to those obtained with the WHO recommended protocol (WRP). Therefore, we conclude that the assays are suitable for the detection of SARS-CoV-2 RNA in clinical samples.

To the best of our knowledge, this is the first study to compare the performance of RT-qPCR master mixes in detecting SARS-CoV-2 RNA and so no direct comparisons can be made with previous studies. However, the three commercial ad hoc molecular diagnostic kits used in this study have been evaluated by others.

The Coronavirus COVID-19 genesig® Real-Time PCR assay (Primerdesign Ltd, UK) which has been listed for emergency use by the WHO (WHO, 2020b) has been previously evaluated by two groups (van Kasteren et al., 2020; Eberle et al., 2021) and deemed to be suitable for the routine diagnosis of SARS-CoV-2. The LOD95 in copies/mL of this kit was determined to be 23 (16–123) by van Kasteren and collaborators (van Kasteren et al., 2020) (equivalent to 0.46 copies per reaction). In contrast, Eberle et al. (2021), calculated an LOD of 80 copies per reaction but stated that due to limited reagents only two replicates for the LOD calculation were performed preventing a statistical analysis of these data. Indeed, (van Kasteren et al., 2020) only performed 4 replicates for their LOD95 calculation and therefore, we feel that our LOD95 of 15.73 (11.85–25.52) copies per reaction based on 16 replicates is more statistically robust.

In a recent paper describing a national external quality assessment scheme for the detection of SARS-CoV-2 in Austria, nine of the 52 laboratories participating employed the ViroReal® Kit SARS-CoV-2 & SARS-CoV (Görzer et al., 2020). Of these nine laboratories, only three correctly identified all of the EQA samples provided while the remaining six failed to detect the weakest positive sample (Ct 38.5). This result is reflected in the present study given that the LOD for the ViroReal® Kit SARS-CoV-2 & SARS-CoV was the highest of all the kits and qPCR master mixes tested. However, in our study no significant differences in its capacity to identify positive and negative samples were observed when compared to the other assays.

The GeneFinder™ COVID-19 Plus RealAMP Kit has recently been evaluated by Ong et al. (2020) who concluded that the kit performed well with excellent sensitivity 100 % (95 % confidence interval (CI) 94–100) when used as part of the fully automated molecular diagnostics system ELITe InGenius®.

The LODs of the eight RT-qPCR mixes were comparable to each other (range 5.21–20.89 copies/reaction) and very similar to the LOD value reported by Corman et al. (2020) for the WRP (i.e. 5.2 copies/reaction). In addition, the correlation studies suggested an excellent correlation/association between the paired samples' Cq data (each assay versus the comparator - WRP). Nonetheless, a strong correlation does not necessarily result in a good agreement between tested kits. Hence, we employed two methods to evaluate the agreement between the tested assays and the WRP: the kappa agreement tests using categorical data (positive or negative) and the Bland-Altman analysis using numerical data (Cq values).

The kappa analyses based on categorical data showed an excellent agreement between the eleven assays and also between each assay and the WRP, used as a comparator. When the visualization of the agreement was based on Cq values, the results showed a good agreement only between the SuperScript™ III One-Step RT-PCR System (Invitrogen) and the One-Step RT-PCR Kit (Qiagen) and TaqMan™ Fast Advanced Master Mix (Applied Biosystems). The other assays tended to have either lower Cq values or higher Cq values when compared to the WRP, as seen on the Bland Altman plots (Fig. 2 and supplement Fig. 1). Nevertheless, we must consider the apparent high bias between the different assays with caution. Though the tendency indicates the overall differences between the assays' Cq values, the slight difference between the Cq values observed with the tested assays did not influence their capacity to correctly reveal the presence of SARS-CoV-2 RNA in the sample.

In short, the results indicate that these assays are equally well suited for the detection of SARS-CoV-2. However, it must be remembered that the assays are only as good as the specificity of the primers and probes being used. SARS-CoV-1 variants that can escape detection by RT-PCR or alter analytical sensitivity due to point mutations in viral genes (including the E gene) are already being reported (Artesi et al., 2020; Ziegler et al., 2020; Hasan et al., 2021). Nevertheless, these current data will be of particular importance to those diagnostic laboratories looking for reliable and easily attainable assays for the diagnosis of SARS-CoV-2 when the reagents included in the recommended protocols are not available.

Author contribution

CEL, AS, SRF, FS, IN, and GC conceived the study. WGD, TKBS, KS and AS performed experiments. CEL, WGD, TKBS, KS, AS, IN and GC analysed data. All of the authors contributed to the writing of the manuscript.

Declaration of Competing Interest

The authors report no declarations of interest.

Appendix A. Supplementary data

The following are Supplementary data to this article: