Analytical performance of rapid nucleic acid detection assays and routine RT-qPCR assays for detection of SARS-CoV-2 in Shanghai, China in 2022

Abstract

Diagnostic accuracy of COVID-19 varies among different assays. In this study, the analytical performance of 1 rapid nucleic acid detection assay (Coyote assay) and 2 routine RT-qPCR assays (BioGerm assay and DaAn assay) was evaluated, using 1196 clinical samples. Disagreement in the results of 2 paired targets occurred in all 3 assays. The Coyote assay failed to detect 15 samples, and the DaAn assay failed to detect 5 samples. The Cohen's kappa coefficient was 0.970 between the BioGerm and DaAn assays, 0.907 between the Coyote and BioGerm assays, and 0.936 between the Coyote and DaAn assays. The positive percent agreement, and negative percent agreement of the Coyote assay were 84.04%, and 100%, respectively. Our study revealed that the results of the Coyote, BioGerm, and DaAn assays were highly consistent, which provided reference for the application of these assays for diagnosis of COVID-19.

1. Introduction

The COVID-19 pandemic caused by SARS-CoV-2 has created enormous burdens to national healthcare systems worldwide and resulted in unrecoverable damage in terms of massive loss of life. The majority of patients infected with SARS-CoV-2 develop mild respiratory symptoms, including fever and cough [1]. SARS-CoV-2 is more likely to affect older individuals and those with cardiovascular disease, chronic lung disease, diabetes, and obesity, resulting in more severe symptoms, including acute respiratory distress syndrome and multi-organ failure [2]. As of May 24, 2022, there have been more than 523 million confirmed cases of COVID-19 globally, resulting in 6.2 million deaths [3].

Accurate diagnosis of infected individuals followed by appropriate quarantine is critical to interrupt the spread of SARS-CoV-2. Comprehensive diagnosis of COVID-19 requires the combination of clinical manifestations and laboratory examination targeting the pathogen SARS-CoV-2. As the proportion of mild and asymptomatic infection keeps increasing, it is challenging to distinguish infected from healthy individuals. Various immunoassays have been developed for detection of specific serum antibodies against SARS-CoV-2. However, the window period of antibody generation impairs diagnostic efficiency in the early stage of infection [4,5]. Therefore, nucleic acid detection (NAD) based on RT-qPCR is recommended as the gold standard for COVID-19 diagnosis, given the high sensitivity and specificity [6].

However, detection of SARS-CoV-2 RNA by RT-qPCR is limited due to the requirement of certified clinical laboratories and qualified technicians for conventional NAD, the time from sample collection to obtaining results, especially for emergency cases, and the continuous evolution of SARS-CoV-2. Genetic mutations affect not only the pathogenic characteristics of viruses, but also the performance of commercial NAD assays. The currently circulating variant Omicron is the most mutated form of SARS-CoV-2, and the BA.2 lineage is the main pathogen of the COVID-19 pandemic in the Shanghai region in the spring of 2022 [7,8]. The Shanghai Municipal Health Commission confirmed 57,819 infections and 586 deaths associated with the Omicron variant from Feb 26 to May 24, 2022, in addition to more than 590,000 asymptomatic infections [9]. Hence, extensive application of assays for rapid detection of SARS-CoV-2 with equivalent diagnostic efficiency of RT-qPCR analysis is warranted.

Although currently available commercial rapid NAD assays and RT-qPCR mainly target the relatively conserved open reading frame 1ab gene (ORF1ab), the nucleocapsid (N) protein gene, or the envelop (E) protein gene, rather than the spike (S) protein, a mutation hotspot of the Omicron variant, their diagnostic efficiency should be adequately assessed for detection of Omicron considering its current dominance [7,10]. Therefore, the aim of the present study was to compare the limit of detection (LOD), reproducibility, and agreement of the rapid NAD assay (Coyote assay) to those of the routine NAD assays (BioGerm and DaAn assays) for testing clinical samples collected from patients with suspected infection with the Omicron variant.

2. Materials and methods

2.1. Preparation of samples and reference standard material

A total of 1196 nasopharyngeal samples collected from individuals with suspected or diagnosed SARS-CoV-2 infections from March 26 to May 12, 2022 were selected for this study. Reference standard material containing heat-inactivated SARS-CoV-2 RNA was purchased from Shanghai BioGerm Medical Technology Co., Ltd. (Shanghai, China). All reference standard material were diluted to appropriate concentrations prior to testing in parallel with the Coyote assay (Coyote Bioscience Co. Ltd., Beijing, China), BioGerm assay (Shanghai BioGerm Medical Technology Co., Ltd.), and DaAn assay (DaAn Gene Co., Ltd., Guangzhou, China). The assays results were interpreted in accordance with the instructions provided by the respective manufacturers.

2.2. Coyote assay

Nasopharyngeal swabs were fully eluted in preservation solution and a 15 μL aliquot was mixed with 15 μL of respiratory tract specimen treatment reagent in a centrifuge tube. SARS-CoV-2 RNA was amplified using a DirectDetect™ SARS-CoV-2 Detection Kit (PCR-Fluorescence Probe) (Coyote Bioscience Co., Ltd.) with the FlashDetect™ Flash 20 Nucleic Acid Fast Detection System (Coyote Bioscience Co., Ltd.). Each 52 μL PCR reaction volume contained 15 μL of the above-mentioned mixture, 35 μL of PCR reagent, and 2 μL of Coyote flash enhance buffer. The PCR amplification protocol included 1 cycle at 42 °C for 3 minutes followed by 15 preamplification cycles at 96 °C for 3 seconds and 55 °C for 5 seconds, and 30 amplification cycles at 96 °C for 3 seconds and 55 °C for 10 seconds. A period of approximately 30 minutes was required to obtain results from the Coyote assay targeting ORF1ab and the N. The detailed result interpretation criteria are shown in Supplementary Table 1.

2.3. BioGerm assay

The nucleic acids of SARS-COV-2 were extracted utilizing the BioGerm Nucleic Acid Extraction and Purification kit (magnetic beads method) in accordance with the manufacturer's instructions. Briefly, 200 μL of each sample was added to a 1.5 mL reaction tube containing 500 μL of extraction buffer, 15 μL of protease K, and 4 μL of magnetic beads. After binding, the magnetic beads were removed, and the purified RNA was eluted and amplified using a Novel Coronavirus (2019-nCoV) NAD Kit (PCR-Fluorescence Probing) (Shanghai BioGerm Medical Biotechnology Co., Ltd.). Each 25 μL PCR reaction volume contained 5 μL of purified RNA. The PCR amplification protocol included 1 cycle at 50 °C for 10 minutes, followed by 1 cycle at 95 °C for 5 minutes and 45 cycles at 95 °C for 10 seconds and 60 °C for 40 seconds. A period of approximately 95 minutes was required to obtain results from the BioGerm assay targeting ORF1ab and the N. The detailed result interpretation criteria are shown in Supplementary Table 1.

2.4. DaAn assay

RNA extraction was conducted using the DaAn Gene Nucleic Acid Extraction Kit (DaAn Gene Co., Ltd.). Briefly, 200 μL of each sample were added to 250 μL of nucleic acid extraction and purification reagent. The nucleic acid was eluted with 50 μL of elution buffer. The RNA was amplified with the Novel Coronavirus (2019-nCoV) Real Time Multiplex RT-PCR kit (DaAn Gene Co., Ltd.). Each 25 μL PCR reaction volume contained 5 μL of purified RNA. The PCR amplification protocol included 1 cycle at 50°C for 15 min, followed by 1 cycle at 95 °C for 15 minutes and 45 cycles at 94 °C for 15 seconds and 55 °C for 45 seconds. A period of approximately 60 minutes was required to obtain results from the DaAn assay targeting ORF1ab and the N. The detailed result interpretation criteria are shown in Supplementary Table 1.

2.5. Verification of the LOD and reproducibility of the 3 assays

To verify the LOD of 3 selected assays, the reference material was diluted to concentrations equal to or below the LODs declared by the respective manufacturers. Each dilution was tested 5 times per day in a single run for 3 consecutive days. Positive detection rates at each concentration were calculated to estimate the LOD. Verification of intra- and inter-assay reproducibility was conducted at concentrations higher than the calculated LOD. Of 5 replicates applied in the intra-assay stage, 4 was included in the inter-assay stage. Negative controls were tested by each assay at the same time. Positive and negative detection rates and coefficients of variation were calculated to determine the reproducibility of the 3 assays.

2.6. Agreement among the 3 assays

In total, 1196 clinical samples were tested. Based on the agreement of results of a single NAD assay or across all 3 assays, the samples were classified into different groups. The detailed detection results and quantification cycle (Cq) values were rigorously recorded for deeper analysis. The Cohen's kappa coefficient was calculated to further estimate the consistency among the BioGerm, DaAn, and Coyote assays. The positive percent agreement, negative percent agreement, positive predictive value, and negative predictive value of the Coyote assay were calculated when 2 routine NAD assays were taken as reference methods.

2.7. Statistical analysis

The final results of 3 assays were collected and analyzed. The Cohen's kappa coefficient, 95% confidence interval (CI), and quartile of Cq values were calculated using IBM SPSS Statistics for Windows, version 22.0. (IBM Corporation, Armonk, NY). The distribution of Cq values was analyzed using GraphPad Prism 8.0 software (GraphPad Software, Inc., San Diego, CA).

3. Results

3.1. Comparison of the LOD of 3 assays

The claimed LOD was 400 copies/mL for the Coyote assay and 500 copies/mL for the BioGerm and DaAn assays. Thus, the reference standard material was diluted from 2000 to 400 copies/mL for the Coyote assay, from 10⁶ to 500 copies/mL for the BioGerm assay, and from 5 × 10⁵ to 500 copies/mL for the DaAn assay. At the designated concentrations, all 3 NAD assays accurately detected ORF1ab and the N. The assays showed substandard positive detection rates at lower concentrations, so the LODs verified in our study were consistent with the claimed LODs.

3.2. Comparison of the reproducibility of 3 assays

Reference standard material was diluted to 2000 copies/mL to verify the reproducibility. The positive detection rates of all 3 assays were 100% (Supplementary Table 2). To be detailed, the intra-assay CV of the Coyote assay for detecting ORF1ab and N was 0.835% and 1.301%, and the inter-assay CV was 1.114% and 2.134%. The intra-assay CV of the BioGerm assay for detecting ORF1ab and N was 0.958% and 0.663%, and the inter-assay CV was 1.035% and 1.095%. The intra-assay CV of the DaAn assay for detecting ORF1ab and N was 0.921% and 0.585%, and the inter-assay CV was 2.791% and 2.750%. The CV values were all ≤5%, and the coincidence rate of negative controls was 100%, which met the requirements declared by the manufacturer.

3.3. Agreement within each assay

Clinical testing was conducted using 1196 samples collected from March 26 to May 12, 2022. For each NAD assay, the results were classified as positive (2-targets positive), partially positive (1-target positive), or negative (Table 1 ). The DaAn assay yielded the highest percentage of positive results (82/1196; 6.86%), followed by the BioGerm assay (79/1196; 6.60%) and Coyote assay (77/1196; 6.52%). The BioGerm assay (15/1196; 1.34%) was more likely to yield partially positive results as compared to the DaAn assay (7/1196; 0.59%) and Coyote assay (2/1196; 0.17%). In most cases, partially positive results were caused by failure of ORF1ab.

Table 1.

Agreement within each NAD assay.

	Coyote	Coyote ORF1ab	Coyote N	BioGerm	BioGerm ORF1ab	BioGerm N	DaAn	DaAn ORF1ab	DaAn N
Positive	77 (6.52%)	77	77	79 (6.60%)	79	79	82 (6.86%)	82	82
Partially positive	2 (0.17%)	0	2	15 (1.34%)	0	15	7 (0.59%)	1	6
Negative	1117 (93.32%)	1117	1117	1102 (92.06%)	1102	1102	1107 (92.56%)	1107	1107
Total	1196 (100%)			1196 (100%)			1196 (100%)

NAD = nucleic acid detection.

The results of analysis of the Cq values are presented in Fig. 1 A. For positive detection of ORF1ab and the N, the Cq values were 6.42 to 28.09 and 4.85 to 25.10 for the Coyote assay, 16.74 to 40.00 and 15.26 to 38.44 for the BioGerm assay, and 9.30 to 29.61 and 8.07 to 29.42 for the DaAn assay, respectively. For partially positive samples, the positive marker tended to be close to the assays' cut-off value.

Fig. 1.

Comparison of Cq ranges of each target among different groups. (A) Comparison of Cq ranges among samples with positive and partially positive results for paired targets. (B) Comparison of Cq ranges among samples with similar and nonsimilar results of the BioGerm and DaAN assays. Data are presented with quartiles. Cq = quantification cycle.

Next, the degree of disagreement in the results within the assay in selected Cq value ranges was investigated (Table 2 ). Those Cq value ranges were chosen because the low value was the lowest Cq value where a disagreement was seen and the upper value was the cut-off Cq value. Specifically, 25% of samples with Cq values of 20 to 27 for the Coyote N, 83% of samples with Cq values of 35 to 40 for the BioGerm N, and 40% samples with Cq values of 25 to 30 for the DaAn N yielded negative results for ORF1ab. Only 1 (9%) of 11 samples with Cq values of 25 to 30 for DaAn ORF1ab yielded negative results for the N (data not shown).

Table 2.

Association between partially positive results and Cq values around the cut-off values.

Assay	Cq of N	Total	Positive for ORF1ab	Negative for ORF1ab
Coyote	20−27	8	6 (75%)	2 (25%)
BioGerm	35−40	18	3 (17%)	15 (83%)
DaAn	25−30	15	9 (60%)	6 (40%)

Cq = quantification cycle.

3.4. Agreement among the 3 assays

Based on the agreement of the results among the 3 assays for the same sample, the 1196 samples were divided into 4 groups (Table 3 ). The 3 assays yielded the same results for 1181 (98.75%) samples (group A, 79 samples with positive results; group B, 1102 samples with negative results). The results of the remaining 15 (1.25%) samples differed among the assays (i.e., positive with the routine NAD assays but negative with the rapid NAD assay). Among these 15 samples, 10 (66.7%) yielded positive results with the BioGerm and DaAn assays (group C), while 5 (33.3%) yielded positive results with only the BioGerm assay (group D). All three assays were more likely to yield positive results for both ORF1ab and the N in group A. For the samples with differing results in groups C and D, the BioGerm and DaAn assays tended to yield partially positive results.

Table 3.

Agreement among the Coyote, BioGerm, and DaAn assays.

	Coyote ORF1ab	Coyote N	BioGerm ORF1ab	BioGerm N	DaAn ORF1ab	DaAn N	n
Agreement: n = 1181 (98.75%)
Group A: Positive for all assays	+	+	+	+	+	+	76
	+	+	-	+	+	+	1
	-	+	-	+	-	+	1
	-	+	-	+	+	-	1
Group B: Negative for all assays	-	-	-	-	-	-	1102
Disagreement: n = 15(1.25%)
Group C: Positive for 2 routine NAD assays	-	-	+	+	+	+	2
	-	-	+	+	-	+	1
	-	-	-	+	+	+	3
	-	-	-	+	-	+	4
Group D: Positive for 1 routine NAD assay	-	-	-	+	-	-	5

NAD = nucleic acid detection.

The range of Cq values for both ORF1ab and N differed among groups A, C, and D (Fig. 1B). The results for ORF1ab and the N were either negative or positive with Cq values concentrated around the cut-off values in groups C and D. The Cq values were characteristically lower in group A. Hence, the Cq values of each target in the different groups were further stratified (Supplementary Table 3), which revealed distinct distributions across the targets or groups. For group A, the Cq values of ORF1ab and the N were concentrated at 5 to 25 and 5 to 25 with the Coyote assay, 15 to 35 and 15 to 35 with the BioGerm assay, and 15 to 35 and 10 to 30, respectively, with the DaAn assay. For group C, the Cq values of the N were concentrated at 35 to 40 with the BioGerm assay and 25 to 30 with the DaAn assay. For group D, the Cq values of the N were concentrated at 35 to 40 with the BioGerm assay.

Cohen's kappa coefficient was calculated to further evaluate the agreement among the three assays (Supplementary Table 4). Agreement was highest between the BioGerm and DaAn assays (κ = 0.970, 95% CI [0.945, 0.995]). The results of the Coyote assay were more in agreement with those of the DaAn assay (κ = 0.936, 95% CI [0.897, 0.975]) than the BioGerm assay (κ = 0.907, 95% CI [0.860, 0.954]).

3.5. Clinical performance of the coyote assay

Next, the clinical performance of the Coyote assay was compared to that of the DaAn and BioGerm assays as reference methods. The 94 samples that yielded positive results with either the BioGerm or DaAn assay were defined as “true positives,” while the 1102 samples that yielded negative results with both assays were defined as “true negatives.” As shown in Supplementary Table 5, among the true positive samples, the Coyote assay yielded positive results for 79 and negative results for 15, thus the results of 1117 samples were classified as negative (positive percent agreement = 84.04%; negative percent agreement = 100%; positive predictive value = 100%; negative predictive value = 98.66%).

4. Discussion

More than 60 mutations have been identified in the Omicron variant, significantly changing the pathogenic characteristics of SARS-CoV-2 and thus bringing new challenges to countries [11]. A meta-analysis revealed that the pooled proportion of nonsevere disease (97.9%) and asymptomatic infection (25.5%) among Omicron-positive individuals were significantly higher than those of Delta-positive individuals [12]. Moreover, compared to the Delta variant, the transmissibility of Omicron is 3.31-fold higher [13]. Consequently, the difficulty of diagnosing Omicron infection according to clinical manifestations was increased and the requirement of the efficiency of diagnostic methods was enhanced.

The emergent outbreak of COVID-19 drove the development of NAD assays over a relatively short period, and the evolution of the virus may have led to poorer performance [14]. Variations in the primer- and/or probe-binding regions may result in failure to detect SARS-CoV-2 RNA. The C26340U mutation in the SARS-CoV-2 genome was associated with the failure of E gene target of the cobas® SARS-CoV-2 test (Roche Diagnostics GmbH, Mannheim, Germany) [15]. Several single-point mutations, including C29200T, C29200A, and C29197T, have been associated with the failure of N gene target of the Xpert® Xpress assay (Cepheid, Sunnyvale, CA, USA) [16], [17], [18], [19]. Partial ORF1ab gene failure was once described in the BA.2.12.1 lineage of the Omicron, which contained ORF1ab synonymous mutations C11674T and T15009C [20]. Holland et al. reported that C636T, T651C, 641∆6, and A638G mutations in the Delta lineage could result in N gene target failure of the TaqPath COVID-19 Combo Kit [21]. As well, the 21765 to 21770 genomic deletion (spike ∆69−70) of the Alpha variant led to S gene target failure of this kit [22]. S gene target failure was a hallmark of the Alpha variant, and was observed in the BA.1 lineage of Omicron, but not the BA.2 lineage [20]. It can be seen that SARS-Cov-2 variants differently affected the efficiency of diagnostic assays. Given these factors, the performance and application of NAD assays in clinical use must be further evaluated.

Since the discovery of SARS-CoV-2, multiple in vitro diagnostic assays have been approved for clinical use by the National Medical Products Administration of China. In the present study, the performance of three NAD assays targeting ORF1ab and the N of SARS-CoV-2 was compared. Overall, ORF1ab was more likely to go undetected, leading to discrepancies in the results of 2 targets of the same assay (Table 1). Similarly, Gdoura et al. reported that 94.9% of the discrepancy in the results of the DaAn assay were caused by a single positive amplification of the N when detecting the Alpha variant [23]. On the contrary, Wang et al. identified more mutations to the targets of probes and/or primers based on the N versus the E and RdRp sequences of the ORF1ab fragment [24]. Thus, target failure varied according to different mutations and SARS-CoV-2 variants. Interestingly, almost all samples partially positive with the DaAn assay were partially positive with BioGerm assay, while samples that tested partially positive with the BioGerm assay could be positive for both ORF1ab and the N with the DaAn assay. Nevertheless, it was uncertain whether DaAn ORF1ab possessed better performance or it was variability in detecting samples with low viral load, reflected by Cq values around the cut-off value.

There were notable inconsistencies in the results across the assays (Table 3). Most samples in group A yielded positive results for both targets. Of note, samples that yielded negative results with the BioGerm assay overlapped those in group B, indicating an NPV of 100% with the BioGerm assay. Group D included 5 samples that yielded positive results with the BioGerm assay only. From this perspective, the BioGerm assay was more sensitive for testing of the N gene of the Omicron variant as compared to the DaAn assay. Despite several inconsistencies, the results of the Coyote, BioGerm, and DaAn assays showed considerable agreement, as confirmed by Cohen's kappa coefficient, which ranged from 0.907 to 0.970. Besides, the Coyote assay showed considerable sensitivity and specificity. Taken together, the three assays demonstrated equivalent analytical performance for diagnosis of COVID-19.

For detection of diluted reference standard material, the Coyote assay was more sensitive, as the verified LOD was 400 copies/mL, lower than that of the BioGerm and DaAn assays. For detection of clinical samples containing Omicron, the Coyote assay yielded several false negative results. The decreased sensitivity may be associated with variables of detection assays, including master mix components, the polymerase enzyme used, primer design, and cycling conditions. It is also worth noting that the first samples allocated to group C and D were isolated on April 21 and 25, 2022, respectively, as daily new asymptomatic infections in Shanghai began to decline. However, the sample size in this study was relatively small and the sequencing of isolated RNA was infeasible owing to the quarantine policy, which limited the explanation of above-mentioned phenomenons. Future large-scale studies with standard methods, such as digital PCR and genome sequencing, are warranted to further investigate the causes of contradictory results.

In conclusion, there was considerable agreement between the results of the selected rapid NAD assay and routine RT-qPCR assays. The time required for the rapid NAD assays is significantly shorter, suggesting advantages for screening of large populations. Considering the possibility of false negative results, rapid NAD assays would better serve as a complement to routine RT-qPCR and the results of rapid NAD assays should be interpreted with caution.

Declaration of competing interest

The authors declare that they have no known competing interests.