Skip to main content
NCBI home page
Elsevier - PMC COVID-19 Collection logoLink to Elsevier - PMC COVID-19 Collection
. 2021 Sep 11;298:114285. doi: 10.1016/j.jviromet.2021.114285

Validation of the analytical performance of nine commercial RT-qPCR kits for SARS-CoV-2 detection using certified reference material

Di Wang 1, Zhidong Wang 1, Ying Gao 1, Xiao Wu 1, Lianhua Dong 1, Xinhua Dai 1, Yunhua Gao 1,*
PMCID: PMC8434693  PMID: 34520809

Abstract

The ongoing coronavirus disease 2019 (COVID-19) pandemic has become a public health emergency. Although many reverse-transcription PCR (RT-PCR) assays have been developed, their performance, especially sensitivity assessment, has been insufficiently tested. In this study, a preliminary comparison of the analytical sensitivity of nine RT-qPCR kits from different manufacturers was first conducted using a certified reference material derived from the genomic RNA of SARS-CoV-2 as the template. Subsequently, three of the nine kits, comprising two highly sensitive kits (DAAN, Huirui) and one less sensitive kit (Geneodx), were selected for further sensitivity and specificity validation. The results revealed variations in the performance between kits of the two groups. For the two highly sensitive kits, the limits of detection at 95 % probability (LOD95%) were 5.6 copies of the N gene and 3.5 copies of the ORF 1ab per reaction (DAAN), and 6.4 (N) and 4.6 (ORF 1ab) copies per reaction (Huirui). These LOD95% values were approximately 3 to 4-fold better than those of the Geneodx Kit. However, none of these three Kits showed cross-reactivity against 6 other types of human coronaviruses or respiratory viruses. Because most of these commercial kits are approved as in vitro diagnostics (testing specimens without direct human contact), it would be beneficial for their manufacturers to improve the diagnostic capability of these kits and thus reduce the clinical risks associated with false-negative results.

Keywords: SARS-CoV-2, RT-qPCR, Sensitivity, Certified reference material, Limit of detection

1. Introduction

Since its outbreak in December 2019 (Li et al., 2020), the ongoing pandemic of coronavirus disease 2019 (COVID-19) caused by the severe acute respiratory syndrome-related coronavirus 2 (SARS-CoV-2) has placed an enormous burden on society, economy and healthcare systems worldwide (Holshue et al., 2020; Rothe et al., 2020). This enveloped, positive-strand RNA virus is a member of the subgenus Sarbecovirus (Han et al., 2020; Lu et al., 2020) and its genome sequence is closely related to that of the severe acute respiratory syndrome-related coronavirus (SARS-CoV) which caused the 2003 outbreak of SARS disease in humans (Guan et al., 2020). Both SARS-CoV and SARS-CoV-2 are now classified within the virus species Severe acute respiratory syndrome-related coronavirus (Coronaviridae study group of the international committee on taxonomy of viruses, 2020).

Reliable diagnosis is among the foremost priorities in public health interventions in order to confirm suspected cases, contact testing and monitor disease spread. RT-qPCR detection of SARS-CoV-2, regarded as the gold standard in COVID-19 diagnostics, are being applied widely to test for the virus (WHO, 2020a). Among the various PCR targets selected by diagnostic laboratories, the open reading frame 1ab (ORF 1ab), nucleocapsid gene (N) and envelope gene (E) of the SARS-CoV-2 genome are the most frequently chosen targets for RT-qPCR assays (Chu et al., 2020; Corman et al., 2020; Pfefferle et al., 2020).

Managing this once-in-a-century COVID-19 pandemic is an enormous challenge for molecular diagnostics. Many laboratory and commercial assays have been developed to detect SARS-CoV-2, resulting in a flood of new tests into the diagnostic market. The performance characteristics of RT-qPCR assays can vary with the reagents supplied, the thermocycling conditions, instrumentation and sampling protocols, leading to different diagnostic results. For example, a series of false-negatives were diagnosed in cases for which clinical symptoms and computed tomography imaging strongly implicated COVID-19 infection (Winichakoon et al., 2020; Wu et al., 2020; Xie et al., 2020). Wang et al. (2020a, 2020b) also showed that of 68 patients with confirmed COVID-19, 20.6 % had initial and follow-up nasopharyngeal swabs that had tested negative for SARS-CoV-2, but the corresponding sputum specimens all tested positive in RT-qPCR assays. In a systematic review of RT-qPCR assays, false-negative rates ranging from 2% to 33 % were found in repeated tests of samples (Arevalo-Rodriguez et al., 2020). Apart from sub-optimal specimen collection or sub-standard diagnostic operation by personnel, false-negatives could also arise from inadequate sensitivity and reliability of the RT-qPCR assays (Bustin et al., 2009). Supply issues can also be a problem. In February 2020, some laboratories reported delays in regional testing roll-out in various countries caused by reagent contamination (Mögling et al., 2020). Although several vaccines are available and effective in conferring resistance to SARS-CoV-2 (WHO, 2021b, 2021a), the vaccination rates in many countries are despairingly low. Vaccination will also take time to immunize entire populations. Therefore, more and better tests are still required for the large numbers of cases arising daily in subsequent waves of infections as well as for contact tracing.

Molecular diagnostic tests for SARS-CoV-2, such as RT-qPCR, are in vitro diagnostics (IVD) whose used is regulated by local health agencies. In the USA, they are approved and labeled as research use only (RUO) (FDA, 2013) or under emergency use authorisations (EUA) for use in clinical tests (FDA, 2021a, 2021b). EUAs are issued for each individual test with certain conditions of authorization required of the manufacturer and authorized laboratories. Additionally, unlicensed IVDs can be placed under the WHO Emergency Use Listing (EUL) with the ultimate aim of expediting the availability of these products for emergency use (WHO, 2020b). With numerous molecular tests available on the market in China, it is essential that these tests are fully evaluated before being employed by diagnostic clinics. Therefore, the aim of this study was to compare the analytical performance of nine RT-qPCR kits for SARS-CoV-2 diagnosis from different manufacturers in China, including 6 kits approved for in-vitro diagnostics use (IVD EUA), and 3 research use only (RUO) kits which could potentially be approved for IVD use. To standardise the assay, a certified reference material (CRM) genomic RNA of SARS-CoV-2 was used as the standard template. To account for any loss of quality during transportation and storage, the quality and concentration of the CRM was checked using a RT-digital droplet PCR (RT-ddPCR) assay (Dong et al., 2021). The diagnostic kits were first evaluated for analytical sensitivity. Then three kits with different levels of sensitivity were selected for further analysis of their RT-qPCR amplification efficiency, limit of quantification (LOQ) (Pavšic et al., 2016; Kralik and Ricchi, 2017), 95 % limit of detection (LOD95%) (Corman et al., 2020; Pfefferle et al., 2020) and cross-reactivity with other human coronaviruses and respiratory viruses.

2. Materials and methods

2.1. RT-qPCR kits

The nine commercial SARS-CoV-2 RT-qPCR assay kits were purchased from each manufacturer. Information on the target genes and regulatory status of these kits is listed in Table 1 (designated as Kit-1 to Kit-9). Among them, six are approved by the National Medical Products Administration of China (China NMPA) as IVD EUA, as well by the European Union under CE-IVD for use clinical tests (Table 1). Only Kit-3 and Kit-4 target ORF1ab, the N gene and E gene (Table 1). For all these kits, the reagent components were similar, including negative control, positive control, supermixes, primers and probes. Their exact working concentrations, and the sequences of the primers and probes are undisclosed to the public.

Table 1.

Properties and regulatory status of the SARS-CoV-2 RT-qPCR assay kits used in this study.

Kit No. Manufacturer Genes targeted Regulatory status
Kit-1 DAAN ORF 1ab /N China NMPA EUA, CE-IVD, WHO EUL
Kit-2 Huirui ORF 1ab /N RUO
Kit-3 Liferiver ORF 1ab /N/E China NMPA EUA, CE-IVD, WHO EUL
Kit-4 Saint Genomics ORF 1ab /N/E RUO
Kit-5 Sansure ORF 1ab /N China NMPA EUA, CE-IVD, US FDA EUA
Kit-6 Applied Biological \Biological Biological ORF 1ab /N China NMPA EUA, CE-IVD
Kit-7 Geneodx ORF 1ab /N China NMPA EUA, CE-IVD, WHO EUL
Kit-8 GenMag ORF 1ab /N RUO
Kit-9 BioGerm ORF 1ab /N China NMPA EUA, CE-IVD
Open in a new tab

Abbreviations: The targets E = envelope protein gene, N = nucleocapsid protein gene and ORF1ab = open reading frame 1ab, of the SARS-CoV-2 genome.

NMPA EUA = National Medical Products Administration Emergency Use Authorization.

CE-IVD = EU/EC Declaration of Conformity in vitro diagnostics.

RUO, research use only.

WHO EUL = The World Health Organisation Emergency Use Listing.

US FDA EUA = United States Food and Drug Administration Emergency Use Authorization.

2.2. Certified reference material

The SARS-CoV-2 certified reference material (CRM) was received from National Reference Material (CNRM, Product GBW(E)091099) (CNRM, 2021). The product is approved by the China NMPA and contained SARS-CoV-2 genomic RNA isolated from infected patients. Specifications of the CRM provided by the manufacturer are listed in Table 2 . The specified copy number concentrations of the ORF1ab, N and E genes, respectively, were double-checked by RT-ddPCR. After confirming the copy number concentrations, the CRM stock was diluted 10-fold serially four times (S1–S4) with the RNA storage solution provided (4.5 copies to 4.5 × 103 copies per reaction (c/r) for ORF 1ab, and 8.7 copies to 8.7 × 103 c/r for the N gene). Yeast carrier RNA (1 mg/mL) was added to the dilutions to avoid degradation of the CRM. Each dilution was divided into equal aliquots and stored at −80 °C.

Table 2.

Manufacturer’s specifications of the certified reference material (Product GBW(E)091099) for used as the standard in SARS-CoV-2 RT-qPCR assay kits.

Product RT-qPCR target Concentration (copies/μL) Expanded uncertainty (copies/μL) (k = 2)a
2019 Novel Coronavirus (SARS-CoV-2) Genomic RNA E gene 1.06 × 103 1.1 × 102
ORF1ab 8.96 × 102 6.1 × 101
N gene 1.73 × 103 1.3 × 102
Open in a new tab
a

k = coverage factor; k = 2 defines a confidence level of approximately 95 % for the calculation of expanded uncertainty (CNRM;White, 2008).

2.3. RT-Digital droplet PCR assay

The RT-ddPCR assay employed to confirm the copy number concentrations of the ORF1ab, and N and E genes in the CRM were conducted using the One-Step RT-ddPCR Advanced Kit for Probes (Bio-Rad Laboratories, USA) on the QX200 Droplet Digital PCR System (Bio-Rad Laboratories, USA) according to the manufacturer's instructions.

The RT-ddPCR workflow and data analysis were performed as described by Dong et al. (2021). All reagents were supplied in the kit except for the primers and probes. The target sequences and the corresponding RT-ddPCR primers and probes for the ORF1ab, N and E genes were those recommended by China NMPA and the WHO (WHO, 2020a, 2020c). The primers and probes sequences and their concentrations used in the RT-ddPCR assays are listed in Table 3 . They were synthesized by Shanghai Sangon Biotech Co., LTD (Shanghai, China). The RT-ddPCR assay protocol was from Bio-Rad with some optimization. The 20 μL RT-ddPCR reaction contained 5 μL of 4x Supermix, 2 μL of reverse-transcriptase, 1 μL of DTT, 1 μL of primers/probes, 5 μL of the CRM RNA and 6 μL of RNase/DNase-free water. Reaction conditions were 45 °C for 10 min for reverse-transcription, thermal cycling at 95 °C for 5 min, followed by 40 cycles, each of 95 °C for 15 s and 58 °C for 30 s; and enzyme deactivation at 98 °C for 10 min. No reverse-transcriptase (NRT) and no template (NTC) negative controls were included in the experiment. After thermal cycling, the plates were transferred to a droplet reader (Bio-Rad, Laboratories, USA) to read the droplets. The final copy number of the CRM determined by RT-ddPCR was calculated using Eq. (1):

C= ​ −D/Vp​ ​× ln(1−P/N​) (1)

Where C is the copy number per microliter, D is the dilution factor used to dilute the RNA with the PCR master mixture, Vp is the droplet volume, P is the number of positive droplets, N is the total number of accepted droplets. Copy numbers were estimated from four replicates.

Table 3.

The primers and probes, and their concentrations used for the RT-ddPCR quantification of the target genes.

Target Primer Sequence (5′-3′) Reaction amount (nM)
ORF1ab Fw CCCTGTGGGTTTTACACTTAA 600
Rv ACGATTGTGCATCAGCTGA 600
probe FAM-CCGTCTGCGGTATGTGGAAAGGTTATGG-BHQ1 200
N Fw GGGGAACTTCTCCTGCTAGAAT 600
Rv CAGACATTTTGCTCTCAAGCTG 600
Probe FAM-TTGCTGCTGCTTGACAGATT-BHQ1 200
E Fw ACAGGTACGTTAATAGTTAATAGCGT 600
Rv ATATTGCAGCAGTACGCACACA 600
Probe FAM-ACACTAGCCATCCTTACTGCGCTTCG-BBQ 200
Open in a new tab

2.4. Preliminary RT-qPCR evaluation of the commercial kits

The nine RT-qPCR kits for SARS-CoV-2 detection were tested in parallel using freshly thawed samples of each serially diluted CRM S1 to S4 as templates to determine their dose response and sensitivity for detecting ORF1ab and N gene. All the RT-qPCR assays were performed following the manufacturers’ instructions and reaction conditions (Table S1) using the Roche Light Cycler 480 II platform (Roche, Germany) and ABI QuantStudio 12 K Flex (Thermo Fisher Scientific, USA). The reaction volumes totalled 25 μL for all the kits except for the Geneodx kit (Total 50 μL) and the ORF1ab and N gene targets were detected simultaneously in the same reaction. The quantification cycle (Cq) was calculated automatically by the LightCycler 480 Software version 1.5.0 SP3 (Roche). Distilled water was used as the template for negative controls. Each assay was repeated three times. The results of the assay dose response were used to design the concentration of CRM for use in subsequent experiments.

2.5. Detailed RT-qPCR evaluation of three commercial kits with high and low sensitivity

Three of the three kits, comprising two highly sensitive kits (DAAN, Huirui) and one less sensitive kit (Geneodx), were selected for further sensitivity and specificity validation as follows.

2.5.1. Standard curve generation and PCR efficiency (E) calculation

The CRM was diluted 3-fold serially in storage buffer as described above (ranging from 18.5 copies to 4.5 × 103 c/r for ORF 1ab, 11.9 copies to 8.7 × 103 c/r for N gene) and used as templates for RT-qPCR assays with each of the three selected commercial RT-qPCR kits. The Cq values were used to generate a RT-qPCR standard curve for each target gene by plotting the Cq values against the corresponding logarithm (base 10) of the copy number concentrations of the CRM dilutions. The standard curve of each assay is described by the Eq. (2):

y = kx + b (2)

Where x is the logarithm of the copy number concentrations and y is Cq values; k is the slope and b is the intercept of the standard curve. The PCR efficiency (E) was then calculated using equation (3):

E = 10−1/k–1 (3)

2.5.2. Determination of the limit of quantification (LOQ) and limit of detection at 95 % probability of detection (LOD95%)

The limit of quantification (LOQ) is the lowest template concentration that an assay could accurately quantify based on the linear portion of the standard curve. The relative standard deviation (RSD) or coefficient of variance of the LOQ value determined should be ≤ 25 % (Pavšic et al., 2016; Kralik et al., 2017). To determine the LOQ of the RT-qPCR assays of the ORF 1ab and N gene for each of the three kits, a 3-fold serial dilution of the CRM, ranging from 16.7 to 4.5 × 102 c/r, were tested using eight replicates of each assay.

The limit of detection (LOD95%) of an RT-qPCR assay is the lowest concentration at which there is a 95 % probability of detecting the target (Corman et al., 2020; Pfefferle et al., 2020). To determine the LOD95-% of the RT-qPCR assay for each target gene for each kit, a series of two-fold dilution of the S2 CRM sample, ranging from 1.8 to 4.5 × 102 c/r, were used as templates in each assay. The LOD95% was then determined by probit analysis of the assay results from eight replicates. The probit analysis was conducted using the SPSS 17.0 software (SPSS, Chicago, IL, USA).

2.5.3. Analytical specificity analysis of the RT-qPCR kits

The RT-qPCR ORF 1ab and N gene assays of each of the three kits were assessed for cross-reactivity against a panel of viral nucleic acids of human viruses other than SARS-CoV-2. These viral nucleic acids used as templates in the assays, were extracted from confirmed clinical samples, included human coronaviruses (SARS-CoV, Middle East respiratory syndrome coronavirus, HCoV-NL63, HCoV−OC43, HCoV-229E and HCoV-HKU1) and other respiratory viruses (Table S2). CRM template was the positive control and healthy human DNA was used as the template in the negative control. The amount of RNA of each virus was ranged from 1 to 10 pg/μL, 5 μL for each assay.

2.6. Sequence analysis of ORF1ab and N amplicons

The amplification products of the RT-qPCR assays were purified using the QIAquick PCR Purification Kit (Qiagen, Germany) and sent out for sequencing (Sangon Biotech, China). The resulting sequences were aligned using the NCBI Blast tool to confirm the sequence identity of the regions of the ORF1ab and N gene amplified by the assays. The reference full-length sequences of SARS-CoV-2 (GenBank accession number NC_045512.2) and SARS-CoV (AY394997.13) used for sequence alignment were retrieved from the NCBI (http://www.ncbi.nlm.nih.gov/genbank/) database.

3. Results

3.1. Validation of CRM copy number concentration by RT-ddPCR

The dot plots results of RT-ddPCR assays of the CRM to validate the specified concentrations of ORF1ab (Fig. 1 a), N (Fig. 1b) and E (Fig. 1c) showed that all the positive droplets (upper dots) were separated clearly from the negative droplets (lower dots), and only a few scattered dots were observed between them. The copy number concentrations of the three target sequences determined by the RT-ddPCR assays were 921 copies/μL for ORF1ab, 1689 copies/μL for N and 1098 copies/μL for E, while NRT and NTC controls were both negative. These results were statistically the same as the corresponding concentrations specified for the CRM by the manufacturer (Fig. 1d). Thus, the CRM was used as the standard template in subsequent sensitivity analysis of the commercial RT-qPCR kits produced for SARS-CoV-2 detection.

Fig. 1.

Fig. 1

Open in a new tab

Validation of the copy concentrations of ORF1ab, and the N and E genes in the CRM RNA by RT-ddPCR, showing one-dimensional scatter plots of the fluorescent droplet amplitudes for quantifying (a) ORF1ab, (b) N gene and (c) E gene. (d) Comparison of the measured concentrations of the three target genes (ORF1ab, N and E) in the CRM with the corresponding values specified by the manufacturer (expected). Blue dots indicated positive droplets and black dots indicated negative droplets. The bars in (d) indicated one standard deviation of the mean of four replicates.

3.2. Preliminary evaluation of nine RT-qPCR kits

The results of ORF1ab and N gene RT-qPCR assays on the dilution series of CRM S1–S4 using the nine commercial RT-qPCR kits revealed that all these test kits were capable of amplifying the ORF1ab and N gene fragments at dilutions S1 (4.5 × 103 c/r for ORF 1ab and 8.7 × 103 c/r for the N gene) to S3 (4.5 × 101 c/r for ORF 1ab and 8.7 × 101 c/r for the N gene), with Cq values ranging from 25.6–37.7. At the S4 dilution level (4.5 c/r for ORF 1ab and 8.7 c/r for the N gene), the nine kits differed in their ability to detect these genes (Table 4 ). At this level, five kits (Kit-1 to Kit-5) detected both target sequences in all three replicate tests, while two kits (Kit-6 and Kit-8) were unable to detect ORF1ab at all, but detected the N gene in some of the replicates only. In contrast, Kit-7 could not detect either ORF1ab or N in all three replicates (Table 4). Based on the above result, two of the highly sensitive kits (Kit-1 and Kit-2) and the least sensitive Kit-7 were selected as the representatives for further analysis. The RT-qPCR amplification curves generated for ORF1ab and the N gene by these three kits using various template concentrations are shown in Fig. S1. The template dose response results were used to determine the template concentrations used in subsequent experiments.

Table 4.

Cq values of detection of ORF1ab and N gene by the nine RT-qPCR kits at the lowest CRM template concentration (S4). Negative = Cq ≥40.

Kit ORF1ab RT-qPCR (4.5 copies/reaction)
 N RT-qPCR (8.7 copies/reaction)
Average Cq Detected/Tested Average Cq Detected/Tested
Kit-1 35.5 3/3 34.7 3/3
Kit-2 37.0 3/3 36.5 3/3
Kit-3 37.3 3/3 36.8 3/3
Kit-4 37.2 3/3 37.0 3/3
Kit-5 36.7 3/3 36.9 3/3
Kit-6 negative 0/3 37.6 2/3
Kit-7 negative 0/3 negative 0/3
Kit-8 negative 0/3 37.4 2/3
Kit-9 37.2 2/3 36.5 1/3
Open in a new tab

3.3. RT-qPCR standard curves and efficiency, and limit of quantification (LOQ) analysis

From the standard curves, the RT-qPCR efficiencies (E) of the three kits for detecting ORF 1ab and N gene ranged from 91.43–103.10% (R2 ≥ 0.98) (Fig. 2 ). The LOQ of the three kits for detecting ORF 1ab (Fig. 2, left column panels) was 23 (Kit-1), 23 (Kit-2) and 61 (Kit-7) c/r, respectively. The corresponding LOQ for the N gene (Fig. 2, right panels), was 26, 26 and 78 c/r, respectively. Compared with the other two kits, the standard curves of Kit-7 also exhibited a poor linearity range for quantifying the target genes.

Fig. 2.

Fig. 2

Open in a new tab

Standard curves obtained for the RT-qPCR assays ORF1ab and the N gene using the three test kits for detecting SARS-CoV-2. (a) Kit-1; (b) Kit-2; (c) Kit-7. The arrow heads pointed to the LOQ of each assay. Each RT-qPCR assay was the mean of eight replicates; each error bar represented the standard deviation of the mean. For standard curves of targeting ORF 1ab, the template concentrations of Kit-1, Kit-2, Kit-7 were ranged from 23 copies to 4.5 × 103 c/r, 23 copies to 4.5 × 103 c/r and 61 copies to 4.5 × 103 c/r, respectively. For standard curves of targeting N gene, those were ranged from 26 copies to 8.7 × 103 c/r, 26 copies to 8.7 × 103 c/r and 78 copies to 8.7 × 103 c/r.

3.4. Limit of detection (LOD95%)

The results of the probit analysis (Fig. 3 ) of each RT-qPCR assay revealed a LOD95% of 3.5 c/r for ORF1ab and 5.6 copies for the N gene using Kit-1 (Fig. 3a). For Kit-2, the LOD95% was 4.6 copies for ORF1ab and 6.4 copies for N gene (Fig. 3b). In contrast, the resulting LOD95% for Kit-7, the least sensitive kit, was much higher, being 14.3 copies for ORF1ab and 20.4 for the N gene (Fig. 3c).

Fig. 3.

Fig. 3

Open in a new tab

Determination of the LOD95% of the ORF1ab and N gene RT-qPCR assays for the three selected kits by probit analysis based on the Cq values of eight replicates of each CRM dilution tested. (a) Kit-1; (b) Kit-2; (c) Kit-7. Limits of detection (LOD95%) are given in the panel headings as c/r (c/r). The arrow heads pointed to the LOD95% of each assay. The middle solid curve is the probit curve. The outer dotted lines are 95 % confidence intervals (95% CI). In each panel, the y-axis plots the proportion of positive samples among the eight replicate assays performed at each of the template concentrations indicated on the x-axis.

3.5. Specificity of the RT-qPCR kits

Specificity evaluation of the three RT-qPCR kits showed that all of them were specific for SARS-CoV-2 only, with no cross-reactivity with the other human viruses tested (Table S2) or with the healthy human DNA negative control.

Note: The data show the RT-qPCR results for the detection of ORF1ab and the N gene using the CRM RNA as template. Positive = positive detection; NR indicates no reaction (Cq≥40). MERS-CoV = Middle East respiratory syndrome coronavirus. Healthy human DNA used as template negative control was negative.

3.6. Sequence analysis of the target amplicons

Sequence analysis of the amplicons obtained from the RT-qPCR assays of the ORF1ab and N gene using each of the three kits and the CRM RNA as template showed that the sizes of the ORF1ab amplicons obtained by Kit-1, Kit-2 and Kit-7 were 88 bp, 119 bp and 119 bp, respectively, and those of the N gene amplicons were 74 bp, 99 bp and 99 bp, respectively (Fig. 4 ). The nucleotide positions of each of these amplicons, identified by an alignment of their sequences with the SARS-CoV-2 genomic sequence, are shown in Fig. 4a. The results also showed that the amplification targets of Kit-2 and Kit-7 were identical while those of Kit-1 matched to different regions of ORF1ab and N gene, with no overlaps with the amplicons of the other two kits. The results of sequence alignments of the ORF1ab amplicon sequences (Fig. 4b) and the N gene amplicon sequences (Fig. 4c) obtained from the kits showed that their sequences matched completely with the corresponding sequences from the SARS-CoV-2 genome but some mismatches with the SARS-CoV genome were found (Fig. 4b, c).

Fig. 4.

Fig. 4

Open in a new tab

(a) Nucleotide positions of the amplification targets of Kit-1, Kit-2 and Kit-7 on the SARS-CoV-2 genome (accession number NC_045512.2). (b, c) Sequence alignments of the ORF1ab sequences (b) and N sequences (c) of the amplicons generated by Kit 1, and Kits -2 and -7 (Kit-2/7, yielding identical amplicons) with the corresponding sequences of SARS-CoV-2 and SARS-CoV. Mismatches in the sequence alignments are shown as lowercase letters.

4. Discussion

This comparative analysis of the performances of nine RT-qPCR kits indicated that their sensitivities (LOQ and LOD95%) and PCR efficiencies (E) for detecting ORF1ab and N gene differed substantially between them. Comparison of the three selected kits showed that the standard curves of Kit-7 (low sensitivity) exhibited a poor linearity range for quantifying the target genes compared to those of the highly sensitive Kit-1 and Kit-2. However, given that all these kits were designed to detect at least two targets simultaneously, the chances of identifying specimens with low viral loads could be enhanced (LeBlanc et al., 2020).

Comparison of the regulatory status of the three kits showed that both Kit-1 and Kit-7 are EUA approved by China NMPA, IVD approved by EU and approved under WHO EUL for expediting availability, while Kit-2 is approved for RUO. The lower E, LOQ and LOD95% observed for Kit-7 compared to the other two kits suggest that caution is still required in the use of approved IVD products.

To check the accuracy of the LOD95% values obtained for the RT-qPCR assays in this study, we repeated Corman’s PCR assay (Corman et al., 2020) using the reported protocol and reagents, while using the CRM as template. The resulting LOD95% of the RT-qPCR assay for the RdRp gene was 3.2 copies/reaction (95 % confidence intervals: 2.4–8.1) (data not shown) which agreed with the reported value by Corman et al. (2020). Furthermore, the LOD95% of SARS-CoV-2 detection by six RT-qPCR kits, which included DAAN (Kit-1) and Geneodx (Kit-7), was consistent with our finding that the LOD95% differed significantly between Kit-1 and Kit-7 (Wang et al., 2020a, 2020b). The LOD95% difference between them was found to increase 16-fold when they were tested on clinical samples. For Kit-1 and Kit-2, their LOD95% values were roughly in line with the theoretical LOD95% of the RT-qPCR assay of three molecules per reaction according to the Poisson distribution (Burns and Valdivia, 2007; Forootan et al., 2017; Kralik and Ricchi, 2017).

The PCR products of Kit-2 and Kit-7 were identical to those amplified using the primer/probe sets recommended by the Chinese Center for Disease Control and Prevention (CCDC) for SARS-CoV-2 detection. This indicated that the differences in the performance of these kits was influenced by other factors, such as reaction volume (Kit-7 used 50 μL), thermocycling conditions and differences in reagents.

The clinical implication of the performance results observed for these kits may be limited by the lack of clinical sample testing, due to the strict control policy in China. To mimic the features of clinical specimens, the CRM contained total RNA extracted from SARS-CoV-2-infected patients, instead of in vitro-transcribed RNA standards or virion RNA. However, it is still quite possible that these results may be different from those obtained using clinical samples where differences in RNA quality and RT-qPCR inhibitors are likely to exist.

Interestingly, the number of RNA copies present in the CRM differed between the three targets tested according to both the manufacturer’s specification and the RT-ddPCR results of this study. This is consistent with what was found by Chu et al. (2020). Several factors may be responsible, but the presence of subgenomic RNA (sgRNA) in the CRM might be a major contributor (Kim et al., 2020; Alexandersen et al., 2020). Because the CRM was derived from clinical specimens containing SARS-CoV-2 infected cells that expressed sgRNAs, it is not unexpected that more copies of the N and E genes were detected in the CRM (Chu et al., 2020). In addition, differences in the efficiencies of the various steps of the RT-qPCR, such as reverse-transcription, primer annealing, amplicon size, etc., could be other reasons (Schwaber et al., 2019; Niu et al., 2021).

High sensitivity is merely one factor to be considered for SARS-CoV-2 detection. In certain situations, diagnostic laboratories need to meet the unprecedented demand for tests. Accordingly, rapid diagnostic test (RDT) systems may offer the possibility of rapid, simple and portable detection of COVID-19 cases under these situations, despite their relatively lower sensitivity (Patriquin and LeBlanc, 2021). Some studies indicated that a regime of repeated RDTs for target populations (e. g., hospital patients) might increase the probability of identifying SARS-CoV-2 infectious individuals, as well as overcoming the limitation of the poor sensitivity of these tests (Mina et al., 2020; Larremore et al., 2020). However, conditions, such as a minimal acceptable sensitivity and an optimal testing frequency, still need to be well-defined for RDTs to be successful (Patriquin and LeBlanc, 2021).

5. Conclusions

The results of the validation tests on three commercial RT-qPCR kits with differing levels of sensitivities for the ORF1ab and N gene, when tested using CRM as templates, showed that none of them showed cross-reactivity against other tested human respiratory viruses. However, the differences in performance provided by the different kits indicated that the manufacturers need to analyze and further improve their products to increase their diagnostic capability for clinical samples with low viral load. Critically, testing laboratories should conduct appropriate in-house validations before using any new RT-qPCR kits.

Author Contribution

Di Wang: Writing - original draft, Methodology, Writing - review and editing, Visualization. Zhidong Wang: Writing - original draft, Visualization. Ying Gao: Writing - review and editing, Visualization. Xiao Wu: Writing - original draft. Lianhua Dong, Methodology. Xinhua Dai: Supervision, Funding acquisition. Yunhua Gao: Conceptualization, Methodology, Supervision, Funding acquisition.

Funding

This research was funded by Special Fund for National Key R&D Program of China, grant number 2017YFF0204604; AQSIQ-scientific Research in the Public Interest, grant number 201510026; Fundamental Research Funds of National Institute of Metrology, China, grant number 31-ZYZJ2001 and 31-AKY1950.

Declaration of Competing Interest

We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.

Acknowledgments

We thank Liqing Wu and Weifei Zhang for technical assistance.

Footnotes

Appendix A

Supplementary material related to this article can be found, in the online version, at doi:https://doi.org/10.1016/j.jviromet.2021.114285.

Appendix A. Supplementary data

The following is Supplementary data to this article:

mmc1.docx (262KB, docx)

References

  1. Alexandersen S., Chamings A., Bhatta T.R. SARS-CoV-2 genomic and subgenomic RNAs in dignostic samples are not an indicator of active replication. Nat. Commun. 2020;11(1):6059. doi: 10.1038/s41467-020-19883-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Arevalo-Rodriguez I., Buitrago-Garcia D., Simancas-Racines D., et al. False-negative results of initial RT-PCR assays for COVID-19: a systematic review. PLoS One. 2020;15(12):e0242958. doi: 10.1371/journal.pone.0242958. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Burns M., Valdivia H. Modelling the limit of detection in real-time quantitative PCR. Eur. Food Res. Technol. 2007;226(6):1513–1524. doi: 10.1007/s00217-007-0683-z. [DOI] [Google Scholar]
  4. Bustin S.A., Benes V., Garson J.A., et al. The MIQE guidelines: minimum information for publication of quantitative real-time PCR experiments. Clin. Chem. 2009;55(4):611–622. doi: 10.1373/clinchem.2008.112797. [DOI] [PubMed] [Google Scholar]
  5. Chu D.K.W., Pan Y., Cheng S.M.S., et al. Molecular diagnosis of a Novel Coronavirus (2019-nCo-V) causing an outbreak of pneumonia. Clin. Chem. 2020;66(4):549–555. doi: 10.1093/clinchem/hvaa029. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. CNRM, Product GBW(E)091099). Certified Reference Material of 2019 Novel Corona Virus (2019-nCoV) Ribonucleic Acid Genome. https://www.ncrm.org.cn/Repository/05b0d7ab-f48c-4dab-8206-ec554c66314d.pdf.
  7. Corman V.M., Landt O., Kaiser M., et al. Detection of 2019 novel coronavirus (2019-nCoV) by realtime RT-PCR. Euro Surveill. 2020;25(3):23–30. doi: 10.2807/1560-7917.ES.2020.25.3.2000045. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Coronaviridae study group of the international committee on taxonomy of viruses The species severe acute respiratory syndrome-related coronavirus: classifying 2019-nCoV and naming it SARS-CoV-2. Nat. Microbiol. 2020;5:536–544. doi: 10.1038/s41564-020-0695-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Dong L., Zhou J., Niu C., et al. Highly accurate and sensitive diagnostic detection of SARS-CoV-2 by digital PCR. Talanta. 2021;1(224):121726. doi: 10.1016/j.talanta.2020.121726. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. FDA . 2013. Distribution of in Vitro Diagnostic Products Labeled for Research Use Only or Investigational Use Only. Guidance for Industry and FDA Staff. November 2013. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/distribution-vitro-diagnostic-products-labeled-research-use-only-or-investigational-use-only (Accessed 5 September 2021) [Google Scholar]
  11. FDA . 2021. In Vitro Diagnostics EUAs.https://www.fda.gov/medical-devices/coronavirus-disease-2019-covid-19-emergency-use-authorizations-medical-devices/in-vitro-diagnostics-euas [Google Scholar]
  12. FDA . 2021. In Vitro Diagnostics EUAs - Molecular Diagnostic Tests for SARS-CoV-2.https://www.fda.gov/medical-devices/coronavirus-disease-2019-covid-19-emergency-use-authorizations-medical-devices/in-vitro-diagnostics-euas-molecular-diagnostic-tests-sars-cov-2 [Google Scholar]
  13. Forootan A., Sjoback R., Bjorkman J., et al. Methods to determine limit of detection and limit of quantification in quantitative real-time PCR (qPCR) Biomol. Detect. Quantif. 2017;12:1–6. doi: 10.1016/j.bdq.2017.04.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Guan Y., Zheng B.J., He Y.Q., et al. Isolation and characterization of viruses related to the SARScoron-avirus from animals in southern China. Science. 2020;302(5643):276–278. doi: 10.1126/science.1087139. [DOI] [PubMed] [Google Scholar]
  15. Han Q., Lin Q., Jin S., et al. Coronavirus 2019-nCoV: a brief perspective from the front line. J. Infect. 2020;80(4):373–377. doi: 10.1016/j.jinf.2020.02.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Holshue M.L., DeBolt C., Lindquist S., et al. First case of 2019 novel coronavirus in the United States. N. Engl. J. Med. 2020;382(10):929–936. doi: 10.1056/NEJMoa2001191. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Kim D., Lee J.Y., Yang J.S., et al. The architecture of SARS-CoV-2 transcriptome. Cell. 2020;181(4):914–921. doi: 10.1016/j.cell.2020.04.011. e10. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Kralik P., Ricchi M. A basic guide to real time PCR in microbial diagnostics: definitions, parameters, and everything. Front. Microbiol. 2017;8:108. doi: 10.3389/fmicb.2017.00108. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Larremore D.B., Wilder B., Lester E., et al. Test sensitivity is secondary to frequency and turnaround time for COVID-19 surveillance. medRxiv. 2020 doi: 10.1101/2020.06.22.20136309. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. LeBlanc J.J., Gubbay J.B., Li Y., et al. Real-time PCR-based SARS-CoV-2 detection in canadian laboratories. J. Clin. Virol. 2020;128 doi: 10.1016/j.jcv.2020.104433. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Li Q., Guan X., Wu P., et al. Early transmission dynamics in Wuhan, China, of novel coronavirus infected pneumonia. N. Engl. J. Med. 2020;382(13):1199–1207. doi: 10.1056/NEJMoa2001316. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Lu R., Zhao X., Li J., et al. Genomic characterisation and epidemiology of 2019 novel coronavirus: implications for virus origins and receptor binding. Lancet. 2020;395(10224):565–574. doi: 10.1016/S0140-6736(20)30251-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Mina M.J., Parker R., Larremore D.B. Rethinking Covid-19 test sensitivity-a strategy for containment. N. Engl. J. Med. 2020;383(22):e120. doi: 10.1056/NEJMp2025631. [DOI] [PubMed] [Google Scholar]
  24. Mögling R., Meijer A., Berginc N., et al. Delayed laboratory response to COVID-19 caused by molecular diagnostic contamination. Emerg Infect Dis. 2020;26(8):1944–1946. doi: 10.3201/eid2608.201843. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Niu C.Y., Dong L.H., Gao Y.H., et al. Quantitative analysis of RNA by HPLC and evaluation of RT-dPCR for coronavirus RNA quantification. Talanta. 2021;228 doi: 10.1016/j.talanta.2021.122227. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Patriquin G., LeBlanc J.J. SARS-CoV-2 sensitivity limbo - How low can we go? Int. J. Infect. Dis. 2021;103:23–24. doi: 10.1016/j.ijid.2020.11.138. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Pavšic J., Žel J., Milavec M. Assessment of the real-time PCR and diff ;erent digital PCR platforms for DNA quantification. Anal. Bioanal. Chem. 2016;408:107–121. doi: 10.1007/s00216-015-9107-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Pfefferle S., Reucher S., Norz D., et al. Evaluation of a quantitative RT-PCR assay for the detection of the emerging coronavirus SARS-CoV-2 using a high throughput system. Euro Surveill. 2020;25(9):1–5. doi: 10.2807/1560-7917.es.2020.25.9.2000152. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Rothe C., Schunk M., Sothmann P., et al. Transmission of 2019-nCoV infection from an asymptomatic contact in Germany. N. Engl. J. Med. 2020;382(10):970–971. doi: 10.1056/NEJMc2001468. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Schwaber J., Andersen S., Nielsen L. Shedding light: the importance of reverse transcription efficienc-y standards in data interpretation. Biomol. Detect. Quantif. 2019;17 doi: 10.1016/j.bdq.2018.12.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. ang K., Zhang X., Sun J., et al. Differences of SARS-CoV-2 shedding duration in sputum and naso-pharyngeal swab specimens among adult inpatients with COVID-19. Chest. 2020;158(5):1876–1884. doi: 10.1016/j.chest.2020.06.015. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Wang X., Yao H., Xu X., et al. Limits of detection of 6 approved RT–PCR kits for the Novel SAR-S-Coronavirus-2 (SARS-CoV-2) Clin. Chem. 2020;66(7):977–979. doi: 10.1093/clinchem/hvaa099. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. White G.H. Basics of estimating measurement uncertainty. Clin. Biochem. Rev. 2008;29(Suppl 1(Suppl 1)):S53–S60. [PMC free article] [PubMed] [Google Scholar]
  34. WHO . 2020. Coronavirus Disease 2019 (COVID-19) Situation Report – 39.https://www.who.int/docs/default-source/coronaviruse/situation-reports/20200228-sitrep-39-covid-19.pdf [Google Scholar]
  35. WHO . 2020. Emergency Use Listing Procedure for in Vitro Diagnostics.https://www.who.int/teams/regulation-prequalification/eul/in-vitro-emergency-use-listing-procedure [Google Scholar]
  36. WHO . 2020. Laboratory Testing for 2019 Novel Coronavirus (2019-nCoV) in Suspected Human Cases.https://www.who.int/publications/i/item/10665-331501 [Google Scholar]
  37. WHO . 2021. COVID-19 Clinical Management: Living Guidance.https://www.who.int/publications/i/item/clinical-management-of-covid-19 [Google Scholar]
  38. WHO . 2021. Status of COVID VAX. 19 August 2021. https://extranet.who.int/pqweb/sites/default/files/documents/Status_COVID_VAX_19August2021.pdf (Accessed 5 September 2021) [Google Scholar]
  39. Winichakoon P., Chaiwarith R., Liwsrisakun C., et al. Negative Nasopharyngeal and oropharyngeal swab does not rule out COVID-19. J. Clin. Microbiol. 2020;58(5) doi: 10.1128/JCM.00297-20. e00297-20. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Wu J., Liu J., Zhao X., et al. Clinical characteristics of imported cases of COVID-19 in Jiangsu province: a multicenter descriptive study. Clin. Infect. Dis. 2020;71(15):706–712. doi: 10.1093/cid/ciaa199. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Xie X., Zhong Z., Zhao W., et al. Chest CT for typical 2019-nCoV pneumonia: relationship to negative RT-PCR testing. Radiology. 2020;296(2):41–45. doi: 10.1148/radiol.2020200343. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

mmc1.docx (262KB, docx)

Articles from Journal of Virological Methods are provided here courtesy of Elsevier

RESOURCES