Performance verification of five commercial RT-qPCR diagnostic kits for SARS-CoV-2

Abstract

Background

Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), which has caused a global pandemic beginning in 2020, can be detected by reverse-transcription polymerase chain reaction (RT-PCR). However, owing to the urgent need for a large number of detection kits, the time spent researching and developing these kits has been shortened during the pandemic, and the kits that are being used commercially have not undergone full and independent evaluation. To ensure the accuracy of SARS-CoV-2 test results, performance verification of commercial Real-Time quantitative PCR (RT-qPCR) kits is required.

Methods

The performance of five commercial RT-qPCR diagnostic kits for SARS-CoV-2 used in China was evaluated using a coronavirus disease 2019 (COVID-19) RNA liquid performance verification reference product-manufactured by Guangzhou Bondson (BDS) Biotechnology Co., Ltd.,Guangzhou, China-that uses droplet digital RT-PCR technology combined with fluorescence quantitative PCR. The five kits of Novel Coronavirus 2019-nCoV nucleic acid detection kit (RT-qPCR method) evaluated were Da An (Da An Gene Co., Ltd. of Sun Yat-sen University), Liferiver (Shanghai ZJ Bio-Tech Co., Ltd.), Kinghawk (Beijing Kinghawk Pharmaceutical Co., Ltd.), eDiagnosis (Wuhan Easy Diagnosis Biomedicine Co., Ltd.), and Maccura (Maccura Biotechnology Co., Ltd.). Performance verification criteria included the coincidence rate, limit of detection (LoD), cross-reactivity, precision, and anti-interference. Finally, through the BDS performance verification reference product kit, clinical samples are used to verify its clinical diagnostic efficacy.

Results

The coincidence rate was 100% for all kits except for Kinghawk, which was 95%. The LoD for Da An, eDiagnosis and Maccura was 250copies/mL, and it was 1000 copies/ml for Liferiver. Kinghawk was not able to detect its advertised LoD of 500 copies/ml. The cross-reactivity test results were all negative. Moreover, all kits had a coefficient of variation less than 5%; however, Liferiver showed the best precision. Da An, Liferiver, and eDiagnosis showed higher sensitivity to the nucleocapsid (N) gene than they did to the open reading frame (ORF) 1ab genes. Anti-interference results for all five kits were positive. The results of clinical diagnostic efficacy were that the specificity of the four kits was 1.000 (0.877–1.000), the sensitivity of Da An was 1.000 (0.850–1.000), Liferiver was 0.964 (0.798–0.998), Maccura was 0.893 (0.706–0.972), and eDiagnosis was 0.857 (0.664–0.953).

Conclusions

All commercial RT-qPCR diagnostic kits for SARS-CoV-2 passed the BDS performance verification, except for Kinghawk (batch No:20200608113) which failed to detect the LoD of 500 copies/mL. Da An and Liferiver have excellent clinical diagnostic specificity and sensitivity. This study can provide guidance for the selection or optimization of RT-qPCR diagnostic test kits for SARS-CoV-2.

1. Introduction:

An outbreak of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2; a novel coronavirus) in December 2019 is responsible for the coronavirus disease 2019 (COVID-19) pandemic [1]. SARS-CoV-2 is an enveloped RNA virus that belongs to the genus β-coronavirus. The coronavirus open reading frame (ORF), nucleocapsid (N) protein, and envelope (E) protein genes are the most commonly used targets for the detection of SARS-CoV-2 [2]. SARS-CoV-2 infection is highly contagious and can spread through respiratory droplets, aerosols, contact, and feces [3]. As of December 2, 2021, COVID-19 has spread to more than 192 countries and regions, with more than 264.13 million confirmed cases and 5.23 million deaths worldwide(https://gisanddata.maps.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467b48e9ecf6). COVID-19 has also caused unpredictable economic losses. Hence, there is an urgent need for a fast, sensitive, and inexpensive detection method to identify SARS-CoV-2 infection and ultimately curb the development and spread of COVID-19 [4].

RT-qPCR can measure the amount of viral RNA in real-time and is considered the gold standard for SARS-CoV-2 detection [5]. In the Seventh Edition of China’s New Coronary Diagnosis and Treatment Guidelines, one of the diagnostic criteria for COVID-19 is a positive RT-qPCR result, and one of the discharge criteria is that respiratory tract specimens should yield two consecutive negative SARS-CoV-2 test results [6]. Therefore, nucleic acid detection of SARS-CoV-2 is important for proper diagnosis and for appropriate handling of patients. Unfortunately, this technology also has a false negative rate that cannot be ignored [7], [8]. Sensitivity is affected by multiple factors, including the inherent characteristics of the selected patient, such as disease stage and viral load, method of collection, sample management, and performance of the test kits used [9], [10]. Among these, evaluating the performance of different test kits could greatly help identify the cause of false negatives [11].

Currently, there are several different SARS-CoV-2 RT-qPCR kits that fully serve the needs for prevention and control of the COVID-19 pandemic and provide an important guarantee for the diagnosis of COVID-19. However, owing to the urgent need for diagnostic kits, research and development related to nucleic acid detection kits for SARS-CoV-2 have been hastened, resulting in issues such as the conducting of clinical trials that are too small-scaled, insufficient scientific data support, and the use of commercial nucleic acid detection kits that have not been fully and independently evaluated. In this study, using the new coronavirus ribonucleic acid (COVID-19 RNA) liquid performance verification RT-PCR reference product (Guangzhou Bondson Biotechnology Co., Ltd., Guangzhou, China) as a reference, we evaluated the performance of five commercial RT-PCR diagnostic kits for SARS-CoV-2. This study provides a reference for clinical laboratories for selecting the SARS-CoV-2 nucleic acid detection kits to be used.

2. Materials and methods

2.1. Materials

2.1.1. Performance verification reference product

We used the COVID-19 RNA liquid performance verification reference product BDS (Guangzhou Bondson Biotechnology Co. Ltd.; batch number 2020001) that is based on the SARS-CoV-2 pseudovirus culture medium containing the important characteristic SARS-CoV-2 genes, namely, the full-length N, E, and ORF1ab genes. The reference product is uniform and stable and has good interoperability with clinical samples since it uses droplet digital PCR (ddPCR) combined with fluorescence quantitative PCR. It can be used by laboratories to evaluate the performance of commercial RT-qPCR diagnostic test kits for SARS-CoV-2 (COVID-19 RNA liquid performance verification reference instructions, Guangzhou Bondson).

2.1.2. SARS-CoV-2 diagnostic kits

Five commercial RT-qPCR diagnostic test kits of Novel Coronavirus 2019-nCoV nucleic acid detection kit (RT-qPCR method) for SARS-CoV-2, i.e., Da An (Da An Gene Co., Ltd. of Sun Yat-sen University, Guangzhou, China; batch number 2020030), Liferiver (Shanghai ZJ Bio-Tech Co., Ltd., Shanghai, China; batch number P20200512), Kinghawk (Beijing Kinghawk Pharmaceutical Co., Ltd., Beijing, China; batch number 20200608113), eDiagnosis (Wuhan Easy Diagnosis Biomedicine Co., Ltd., Wuhan, China; batch number 200606), and Maccura (Maccura Biotechnology Co., Ltd., Chengdu, China; batch number 0520251), were used in this study. Basic information on these RT-qPCR diagnostic kits for SARS-CoV-2 is shown in Table 1 .

Table 1.

Basic information on five commercial RT-qPCR diagnostic test kits for SARS-CoV-2 evaluated in this study.

Kit	Gene	Template quantity/μl	Reaction system/μl	Number of cycles/Pieces	PCR reaction time (min)	Internal standard type	Declaration of detection limit (copies/ml)	Results interpretation criteria
								Positive	Suspicious	Negative
Da An	ORF1ab, N	5	25	45	118	Endogenous	500	Ct ≤ 40		Ct > 40
Liferiver	ORF1ab, N, E	5	25	45	90	Exogenous	1,000	Ct ≤ 43		Ct > 43
Kinghawk	ORF1ab, N	5	25	40	98	Endogenous	500	Ct < 38	38 < Ct < 40	No Ct
eDiagnosis	ORF1ab, N	5	25	40	82	Endogenous	500	Ct < 38	38 ≤ Ct < 40	Ct ≥ 40
Maccura	ORF1ab, N, E	20	40	40	88	Exogenous	1,000	Ct ≤ 38		Ct > 38 or No Ct

2.1.3. Nucleic acid instrumentation

The Stream SP96 automatic nucleic acid extraction instrument (Da An Gene Co., Ltd. of Sun Yat-sen University) was used for nucleic acid extraction. RT-qPCR was conducted using the SLAN-96P real-time PCR system (Shanghai Hongshi Biotechnology Co., Ltd., Shanghai, China).

2.2. Methods

2.2.1. RT-qPCR

The RT-qPCR cycle parameters were set according to the kit instructions. The baseline was usually automatically set by the instrument. The baseline adjustment principle included selecting the area with a stable fluorescence signal before exponential amplification, avoiding signal fluctuation at the beginning of fluorescence acquisition, and reducing the threshold cycle (Ct)/quantification cycle (Cq) value of the sample with the earliest exponential amplification by 1–2 cycles at the end. The principle of setting threshold line is to make the threshold line in the exponential phase of amplification curve. RT-qPCR data are traditionally analyzed by estimating the Ct in which the fluorescent signal generated by the probe emission crosses the threshold line. Refer to the instructions of the corresponding kits for the baseline and threshold settings. Here, the negative and positive test results were judged according to the specific kit instructions.

2.2.2. Coincidence rate verification

There were 10 positive reference samples in BDS, numbered P1–P10, and 10 negative patient samples from previous tests, numbered P11–P20, totaling 20 samples. Nucleic acid extraction was carried out using a Da An nucleic acid extraction kit (Da An Gene Co., Ltd. of Sun Yat-sen University). Amplification was performed using the five commercial RT-qPCR diagnostic kits for SARS-CoV-2 as mentioned above, using the same Hongshi SLAN-96P real-time PCR system (Shanghai Hongshi Biotechnology Co., Ltd., Shanghai, China). A kit was considered to have passed verification if the coincidence rate of negative and positive results was ≥ 95%.

2.2.3. Limit of detection

The concentrations of the limit of detection (LoD) reference products L1–L5 provided in BDS were 200,000, 20,000, 2,000, 500, and 250 copies/ml, respectively. Detection was performed 3 times for each sample, and the lowest concentration at which all three results were positive was considered as the initial screening LoD. Then, detection at the LoD was performed for each sample 20 consecutive times in different batches (five batches were tested, and the test was repeated four times for each batch) for statistical analysis. The lowest concentration level, with a detection rate of 100% positive results, was taken as the LoD for each kit.

2.2.4. Cross-reactivity

Twenty analysis-specific reference samples, N1–N20, were included in BDS (Table 2 ). Nucleic acid extraction was performed using the Da An kit and then amplification was carried out using the five commercial RT-qPCR diagnostic kits for SARS-CoV-2 with the same amplification instrument. Each sample was tested three times. To pass in terms of cross-reactivity, all results should be negative for cross-reactivity.

Table 2.

Pathogenic microorganisms used to evaluate the cross-reaction of five commercial RT-qPCR diagnostic test kits for SARS-CoV-2 evaluated in this study.

ID	Pathogen	ID	Pathogen
N1	Pseudovirus positive samples containing human coronavirus HCoV-OC43 RNA	N11	Positive samples containing adenovirus (inactivated virus)
N2	Pseudovirus positive samples containing human coronavirus HCoV-HKU1 RNA	N12	Positive samples containing enterovirus (inactivated virus)
N3	Pseudovirus positive samples containing human coronavirus HCoV-229E RNA	N13	Positive samples containing Mycoplasma pneumoniae (inactivated Mycoplasma)
N4	Pseudovirus positive samples containing human coronavirus HCoV-NL63 RNA	N14	Positive samples containing EB virus (inactivated virus)
N5	Pseudovirus positive samples containing novel coronavirus SARS RNA	N15	Positive samples containing human cytomegalovirus (inactivated virus)
N6	Pseudovirus positive samples containing MERS RNA of Middle East respiratory syndrome virus	N16	Positive samples containing Mycobacterium tuberculosis (inactivated bacterial solution)
N7	INFA RNA positive samples containing influenza A virus (inactivated virus)	N17	Human genome DNA sample
N8	INFB RNA positive samples containing influenza B virus (inactivated virus)	N18	Human genome DNA sample
N9	Positive samples containing respiratory syncytial virus type A + B (inactivated virus)	N19	Negative samples not containing COVID-19 RNA
N10	Positive samples containing human parainfluenza virus (inactivated virus)	N20	Negative samples not containing COVID-19 RNA

2.2.5. Precision

The precision reference products in BDS include low concentration (R1, 2,000 copies/mL) and medium concentration (R2, 20,000 copies/mL) products. The five commercial RT-qPCR diagnostic kits for SARS-CoV-2 were used for amplification using the same amplification instrument. One batch was measured per day for 5 d, and the measurement of each batch was repeated four times. Thus, 20 measurements were taken at each concentration. Precision verification was considered passed if the precision coefficient of variation (CV) value was < 5%.

2.2.6. Anti-interference ability

The ability of the five commercial kits to detect SARS-CoV-2 in interference reference samples I1–I3 (that contained 30 g/dL hemoglobin, 6 g/dL albumin, and 100 µg/mL ribavirin and azithromycin for 2,000 copies/mL positive samples) was assessed. The concentration of the positive samples in the interfering substance was considered to be a weak positive concentration, which was set based on 2–5x the LoD of the kits on the market. The concentration of interference was set according to the anti-interference ability of the kit. The detection of each sample was performed in three replicates for each commercial kit. If the three replicates were positive, the results were positive, then that particular kit was considered to pass the anti-interference verification.

2.2.7. Diagnosis efficacy

We recruited 28 COVID-19 patients and 35 none-COVID-19 controls in Dalian, and collected pharyngeal swabs. We verified the clinical diagnostic efficacy of clinical samples that for the kits that have passed the BDS performance verification reference product. The sensitivity, specificity, PPV (Positive Predictive Value) and NPV (Negative Predictive Value) were calculated.

2.2.8. Statistical analysis

SPSS 19.0 (IBM, Armonk, NY, USA) and GraphPad Prism 5.0 (GraphPad, San Diego, CA, USA) were used for statistical analyses. The Mann–Whitney U test was employed for comparisons between two groups. The Kruskal–Wallis test was used to compare three independent groups. A P-value < 0.05 was considered significant. In order to evaluate the detection efficiency and diagnostic value of the RT-qPCR kits, we calculated the sensitivity, specificity, PPV, NPV and its 95% confidence interval (CI).

3. Results

3.1. Coincidence rate

The coincidence rate for all five diagnostic kits was 100%, except for Kinghawk. One of the 10 positive reference products was not detected by Kinghawk, while all 10 negative reference products yielded negative results. Hence, the Kinghawk coincidence rate was 95% (Table 3 ).

Table 3.

Coincidence rate of the five commercial RT-qPCR diagnostic test kits for SARS-CoV-2 evaluated in this study.

ID	Da An		Liferiver			Kinghawk		eDiagnosis		Maccura
	ORF1ab	N	ORF1ab	N	E	ORF1ab	N	ORF1ab	N	ORF1ab	N	E
P1	35.46	34.41	36.17	34.22	32.45	36.28	37.28	36.13	33.71	33.5	33.49	31.8
P2	36.49	35.91	37.37	36.26	37.08	37.57	37.07	38.16	35.9	35.61	36.79	33.21
P3	29.44	27.87	30.72	28.5	26.8	31.63	30.6	30.69	28.43	27.7	25.76	28.05
P4	33.2	31.56	34.73	32.3	30.82	34.91	34.36	34.14	32	31.75	29.28	31.8
P5	31.37	29.89	32.92	30.87	29.2	33.88	32.76	32.88	30.71	30.22	30.36	27.99
P6	36.81	34.34	36.42	35.14	32.95	37.19	No ct	37.58	34.72	34.03	34.19	31.78
P7	31.71	30.03	32.54	30.7	28.79	33.81	32.34	32.06	30.09	29.04	30.63	27.34
P8	26.52	24.93	27.71	25.87	23.73	29.67	27.3	27.21	25.22	23.90	22.45	25.51
P9	28.95	26.84	29.99	27.99	25.62	30.97	29.58	29.61	27.33	26.61	26.85	24.36
P10	34.26	32.05	35.47	33.44	30.98	35.45	34.72	34.41	32.30	31.04	32.16	29.26
P11	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct
P12	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct
P13	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct
P14	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct
P15	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct
P16	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct
P17	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct
P18	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct
P19	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct
P20	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct	No ct
Positive mean	32.42	30.78	33.40	31.53	29.84	34.14	32.89	33.29	31.04	30.34	30.20	29.11
Positive SD	3.44	3.56	3.18	3.36	3.90	2.68	3.35	3.51	3.37	3.62	4.30	2.97
Coincidence rate (%)	100	100	100	100	100	100	95	100	100	100	100	100

3.2. LoD

Kinghawk failed to detect the initial screening concentration of 1,000 copies/mL, even after repeating the test. The preliminary screening LoD of Liferiver was 500 copies/mL, while that of Da An, eDiagnosis, and Maccura was 250 copies/mL in each case (Fig. 1 ). Moreover, it was observed that as the concentration of the reference product decreased, the Ct value gradually increased (Fig. 1). The confirmed LoD of Liferiver was 1,000 copies/mL and that of Da An, eDiagnosis, and Maccura was 250 copies/mL (Fig. S1). Since ≥ 100% of the positive results met expectations, the kits passed verification. The LoD of Da An, eDiagnosis, and Maccura was 250 copies/mL and for Liferiver it was 1,000 copies/mL, indicating that these four kits met their respective declared LoD. Meanwhile, Kinghawk did not meet the LoD of 500 copies/mL declared by its manufacturer. It is also worth noting that the values of the verification results of the LoD of Da An, eDiagnosis, and Maccura (250 copies/mL) were lower than those declared by their respective manufacturer (500copies/mL, 500copies/mL and 1,000 copies/mL).

Fig. 1.

Preliminary screening experiments of the five commercial RT-qPCR diagnostic test kits for SARS-CoV-2 evaluated in this study. Data represent the mean cycle threshold (Ct) values of each nucleic acid reference product concentration measurement (n = 3). A: Open reading frame (ORF) 1ab gene; B: nucleocapsid (N) gene.

Using the LoD values, we calculated the 95% confidence interval (CI) Ct values of the genes detected by the different kits. The 95% CI of Da An ORF1ab was 38.28–39.08 and N was 35.93–36.26; Liferiver ORF1ab was 37.99–38.93, N was 36.49–37.44, and E was 34.15–34.80; eDiagnosis ORF1ab was 37.89–38.75 and N was 35.35–35.98; and Maccura ORF1ab was 35.65–36.27, N was 34.41–35.40, and E was 35.32–35.83.

3.3. Cross-reactivity

The list of the specific pathogenic microorganisms used to assess the cross-reactivity of each diagnostic test kit is shown in Table 2. All test results for cross-reactivity were negative, indicating that the five commercial RT-qPCR diagnostic test kits for SARS-CoV-2 had strong specificity and that the presence of the pathogenic microorganisms listed in Table 2 would not affect the ability of the test kits to detect SARS-CoV-2.

3.4. Precision evaluation

The precision of the five commercial RT-qPCR diagnostic kits for SARS-CoV-2 was in line with that declared by their respective manufacturers (<5% CV, Table 4 ). The distribution of Ct values of 20 test results of each commercial RT-qPCR diagnostic test kit for SARS-CoV-2 are shown in Fig. 2 . As illustrated, Liferiver had the smallest CV for ORF1ab gene detection in reference product R1 (1.05%; Table 4), while Maccura had the largest (2.47%; Table 4). Similarly, Liferiver had the smallest CV for N gene detection in R1 (1.21%; Table 4), while eDiagnosis had the largest (2.74%; Table 4). For R2, Liferiver had the smallest CV for ORF1ab gene detection (0.65%; Table 4), while Da An had the largest (2.00%; Table 4). Again, Liferiver had the smallest CV for N gene detection (0.65%; Table 4), while Maccura had the largest (1.97%; Table 4). Taken together, our results indicate that Liferiver exhibited the best precision for the detection of the ORF1ab and N genes at low and median concentrations.

Table 4.

R1 and R2 precision results of the five commercial RT-qPCR diagnostic test kits for SARS-CoV-2 evaluated in this study.

	Da An		Liferiver			Kinghawk		eDiagnosis		Maccura
	ORF1ab	N	ORF1ab	N	E	ORF1ab	N	ORF1ab	N	ORF1ab	E	N
Mean (R1)	34.94	32.78	35.65	33.45	31.43	34.90	35.51	34.74	32.61	32.41	30.39	32.71
SD (R1)	0.75	0.74	0.37	0.40	0.47	0.43	0.73	0.76	0.89	0.80	0.93	0.87
CV% (R1)	2.14	2.26	1.05	1.21	1.50	1.23	2.05	2.19	2.74	2.47	3.07	2.65
Mean (R2)	30.59	28.75	31.20	29.01	27.09	31.86	30.58	30.92	29.11	27.91	26.14	29.18
SD (R2)	0.61	0.27	0.20	0.20	0.21	0.58	0.40	0.29	0.55	0.24	0.27	0.57
CV% (R2)	2.00	0.92	0.65	0.68	0.79	1.83	1.31	0.94	1.90	0.86	1.05	1.97

Fig. 2.

Precision results of the five commercial RT-qPCR diagnostic test kits for SARS-CoV-2 evaluated in this study. (A) Cycle threshold (Ct) for detection of the ORF1ab gene in R1. (B) Ct for the detection of the N gene in R1. (C) Ct values for the detection of ORF1ab in R2. (D) Ct values for the detection of the N gene in R2. (E) Comparison of mean Ct values for detection of ORF1ab and N genes in R1. (F) Comparison of mean Ct values for detection of ORF1ab and N genes in R2.

In the precision, the ORF1ab and N genes of the five commercial RT-qPCR diagnostic test kits for SARS-CoV-2 were compared. At the low concentration level of R1, the Ct values of Kinghawk and Maccura were lower for the ORF1ab gene than the N gene, whereas the Ct values of Da An, Liferiver, and eDiagnosis were all significantly lower for the detection of the N gene than the ORF1ab gene (P < 0.05; Fig. 2E). At the medium concentration level of R2, the Ct value of Maccura was higher for the N gene than the ORF1ab gene, whereas Da An, Liferiver, Kinghawk, and eDiagnosis had significantly lower Ct values for the N gene than the ORF1ab gene (P < 0.05; Fig. 2F). Therefore, the sensitivity of Da An, Liferiver, and eDiagnosis for the N gene was higher than that for the ORF1ab gene at both low and median concentrations of the reference product.

3.5. Anti-interference ability

The anti-interference test results showed that all five commercial RT-qPCR diagnostic kits for SARS-CoV-2 were able to detect ORF1ab and N genes in the I1–I3 samples, indicating that 30 g/dL hemoglobin, 6 g/dL albumin, and 100 µg/mL ribavirin and azithromycin, respectively, did not interfere with SARS-CoV-2 nucleic acid detection. Moreover, the five test kits showed no statistical differences in the detection of the ORF1ab and N genes in the I1–I3 samples (Fig. 3 A, B).

Fig. 3.

Anti-interference test results of the five commercial RT-qPCR diagnostic kits for SARS-CoV-2 evaluated in this study. Detection of the (A) ORF1ab gene and (B) N gene in the I1–I3 samples containing 30 g/dL hemoglobin, 6 g/dL albumin, and 100 µg/mL ribavirin and azithromycin, respectively.

3.6. Diagnosis efficacy

We evaluated the clinical diagnostic efficacy of the four RT-qPCR kits that have passed the BDS performance verification reference product: Da An, Liferiver, eDiagnosis, and Maccura, and the results are shown in Table 5 . The specificity of all the four kits was 1.000 (0.877–1.000). Da An has the highest sensitivity of 1.000 (0.850–1.000), PPV of 1.000 (0.850–1.000), NPV of 1.000 (0.877–1.000), followed by Liferiver, with sensitivity of 0.964 (0.798–0.998) and PPV of 1.000 (0.845–1.000) , NPV is 0.972 (0.838–0.999), and eDiagnosis has the lowest sensitivity of 0.857 (0.664–0.953), PPV is 1.000 (0.828–1.000), NPV is 0.897 (0.748–0.967).

Table 5.

Diagnosis efficacy of RT-qPCR kits for SARS-CoV-2 nucleic acid detection.

	COVID-19 (n = 28)		None-COVID-19 (n = 35)
Test kits	Positive	Negative	Positive	Negative	Sensitivity(95 %CI)	Specificity(95 %CI)	PPV(95 %CI)	NPV(95 %CI)
Da An	28	0	0	35	1.000(0.850–1.000)	1.000(0.877–1.000)	1.000(0.850–1.000)	1.000(0.877–1.000)
Liferiver	27	1	0	35	0.964(0.798–0.998)	1.000(0.877–1.000)	1.000(0.845–1.000)	0.972(0.838–0.999)
eDiagnosis	24	4	0	35	0.857(0.664–0.953)	1.000(0.877–1.000)	1.000(0.828–1.000)	0.897(0.748–0.967)
Maccura	25	3	0	35	0.893(0.706–0.972)	1.000(0.877–1.000)	1.000(0.834–1.000)	0.921(0.775–0.979)

4. Discussion

The rapid and accurate detection of SARS-CoV-2 plays a key role in determining the treatment plan for COVID-19 patients and for controlling the spread of infection [12]. RT-qPCR has a high sensitivity and specificity; therefore, it is currently used as the main method for COVID-19 diagnosis [13]. Thus, when SARS-CoV-2 nucleic acid detection is the gold standard for COVID-19 diagnosis, the sensitivity and accuracy of the diagnostic test kits are particularly important. Under normal circumstances, it takes 3–5 years to obtain a registration certificate for nucleic acid kits, during which the kit needs to be continuously optimized and verified with a large number of clinical samples. Inevitably, there are some flaws in the quality and performance parameter settings of many commercial kits, and manufacturers are constantly updating and optimizing their kits. One of these flaws is that the LoD advertised by manufacturers is based on the best extraction method and best experimental conditions before the test kit leaves the factory; there is often a lack of clinical verification data in such scenarios. Therefore, when performing SARS-CoV-2 nucleic acid testing, the laboratory, clinician, or end-user should first evaluate the performance of the test kit [11].

In Ecuador, Byron Freire-Paspuel et al. validated the performance of several SARS-CoV-2 nucleic acid detection kits. The results showed that nCoV QS (MiCo BioMed, South Corea) and Genome (ABM, Richmond, Canada) had low clinical sensitivity of 66.7% and 75.0%, respectively. The former lacked RNA quality control probe, and the latter had a limit of detection estimated to be over 8.000 copies/mL, both of which were not authorized by the FDA (Food & Drug Administration) or EUA (emergency use authorization) [14], [15]. Isobollo (M monitor, South Korea) and Accupower (Bioner, South Korea) had sensitivity of 61.9% and 78.9% respectively, and do not have EUA approval either from the FDA or from the Korean disease control and prevention agency, but it is currently available in Ecuador for SARS CoV-2 clinical diagnosis [16], [17]. Accupower (bioner, South Korea) had a sensitivity of 75.5%, and its limit of detection estimated to be bigger than 20,000 copies/mL and was not authorized by the FDA or EUA [18]. Allplex (Seegene, South Korea) and Viasure (CerTest Biotec, Spain) had high sensitivities of 96.5% and 91.9%, respectively, and limit of detection of 4000 copies/mL and 2000 copies/mL [18], [19]. Through the performance verification of the above kits, we can see that some kits fail to pass the performance verification. They are still used in developing countries such as Ecuador although they are not authorized, which can easily cause false negatives and thus is detrimental to the control of the epidemic. In addition, it should be noted that the presence of high CT values indicates a decrease in sensitivity, especially for specimens with low viral load. Therefore, before clinical use of authorized SARS-CoV-2 diagnostic kits, appropriate clinical performance evaluation studies must be conducted by regulatory agencies in developing countries such as Ecuador, especially if these kits lack FDA or country-of-origin clinical use authorization. Hur et al. [20] assessed 4 commercial kits for SARS-CoV-2 RT-qPCR approved for emergency use in Korea, providing a reference for Korea’s SARS-CoV-2 nucleic acid detection testing. Altamimi et al. [21]evaluated 12 RT-qPCR commercial kits from different countries for SARS-CoV-2 detection; however, their study evaluated only sensitivity and specificity. Wang et al. [22]assessed the limits of detection of 6 approved SARS-CoV-2 RT–qPCR kits in China using ddPCR references without more parameters. In the current study, we evaluated 5 RT-qPCR commercial kits in China as to coincidence rate, limit of detection, specificity, precision, anti-interference performance and diagnosis efficacy parameters.

In our evaluation, we used a COVID-19 RNA liquid performance verification reference product combined with clinical diagnosis efficacy, which allowed for an independent and comprehensive performance verification of five commercial SARS-CoV-2 RT-qPCR diagnostic kits currently used in China. Among the five tested kits, Da An, Liferiver, eDiagnosis, and Maccura met the coincidence rate, detection limit, specificity, precision, and anti-interference performance parameters declared by their respective manufacturers. The coincidence rate verification results showed that the coincidence rate of Kinghawk was 95%, while the coincidence rate for all other kits was 100%. The LoD for Da An, eDiagnosis, and Maccura was 250copies/mL, the LoD for Liferiver was 1,000 copies/mL. The values for the LoD of Da An, eDiagnosis, and Maccura were lower than those declared by their manufacturers. Kinghawk did not meet the declared LoD of 500 copies/mL. Precision verification results showed that the CV of the five test kits was < 5% and that Liferiver had the best precision. Moreover, we found that different kits had different levels of sensitivity to ORF1ab and N genes, such that Da An, Liferiver, and eDiagnosis were more sensitive to N gene detection than they were to ORF1ab. The specificity of all the four kits was 1.000 (0.877–1.000). Da An has the highest sensitivity of 1.000 (0.850–1.000), followed by Liferiver, with sensitivity of 0.964 (0.798–0.998) and eDiagnosis has the lowest sensitivity of 0.857 (0.664–0.953). In Ecuador, Byron Freire-Paspuel et al. performed clinical performance validation of SARS-CoV-2 diagnostic kits such as Da An, eDiagnosis and Sansure Biotech, manufactured in China and authorized by the CDC on an emergency basis. Their results showed that Da An has a limit of detection of 2,000 copies/mL and 100% of sensitivity [15]. They found an excellent clinical performance and analytical sensitivity for both kits with sensitivity values of 100% and 95.3% and estimated limits of detection of 500 copies/mL and 1,000 copies/mL, for eDiagnosis and Sansure Biotech kits, respectively [23]. Prior to the clinical use of the authorized SARS-CoV-2 diagnostic kit, it is best for local regulators to conduct appropriate clinical performance evaluation studies.

Here, our one limitation is that we only show a brief comparative analysis of the performance of the kits analyzed, and this may not be fully representative of the detection performance of other batches of kits. Our another limitation is that Kinghawk kit only carried out the performance verification of BDS products, and lacked the diagnostic efficacy verification of clinical samples. The reason is that Kinghawk was short of supply of the same batch of kits when carrying out the clinical diagnostic efficacy test, so the corresponding clinical diagnostic efficacy test has not been carried out. If the condition, should carry on Kinghawk’s clinical diagnosis efficacy verification. In conclusion, we evaluated the performance of five commercial RT-qPCR diagnostic test kits used to detect SARS-CoV-2 and also provided references for the selection of kits for clinical laboratories, facilitating the optimization of commercial RT-qPCR diagnostic test kits. Such measures should reduce the clinical risks associated with false-negative results and more effectively detect and control the spread of COVID-19 throughout the world. The variation and changes in SARS-CoV-2 in multiple geographic locations indicates that regular genetic screening is needed. In our future work, we plan to assess genetic changes in the virus and their impacts on detection of SARS-CoV-2 using existing commercial kits.

5. Conclusion

Owing to the urgent need for a large number of SARS-CoV-2 detection kits, the time spent researching and developing these kits has been shortened during the pandemic, and the kits that are being used commercially have not undergone full and independent evaluation. To ensure the accuracy of SARS-CoV-2 test results, performance verification of commercial RT-qPCR kits is required. The performance of five commercial RT-qPCR diagnostic kits for SARS-CoV-2 used in China was evaluated using a coronavirus disease 2019 (COVID-19) RNA liquid performance verification reference product—manufactured by Guangzhou Bondson (BDS) Biotechnology Co., Ltd.,Guangzhou, China—that uses droplet digital PCR technology combined with fluorescence quantitative PCR. The result was that all commercial RT-qPCR diagnostic kits for SARS-CoV-2 passed the BDS performance verification, except for Kinghawk (batch No: 20200608113) which failed to detect the LoD of 500 copies/mL. Then, we evaluated the clinical diagnostic efficacy of the four RT-qPCR kits that have passed the BDS performance verification reference product: Da An, Liferiver, eDiagnosis, and Maccura. The specificity of all the four kits was 1.000 (0.877–1.000). Da An has the highest sensitivity of 1.000 (0.850–1.000), followed by Liferiver, and eDiagnosis has the lowest sensitivity of 0.857 (0.664–0.953). This study provides a reference for clinical laboratories for selecting the SARS-CoV-2 nucleic acid detection kits to be used, helps manufacturers to optimize kits and also provides laboratory methods for clinical laboratories to verify the performance of SARS-CoV-2 kits.

Ethics statement

The study was conducted in accordance with the International Coordinating Council for Clinical Trials and the Helsinki Declaration and was approved by the Hospital Ethics Review Committee (Ethics No 2020PS564K(X2)). Informed consents were obtained as required.

Author Contributions

All authors have accepted responsibility for the entire content of this manuscript and approved its submission. Mei Yang and Shuang cao were responsible for BDS performance verification experimental operations; Yong Liu and Zhijie Zhang were responsible for clinical diagnostic efficacy experimental design and data collection; Rui Zheng was responsible for the diagnosis of clinical patients and experimental design; Yuzhong Li, Jie Zhou, Chengguo Zong, and Desheng Cao were responsible for the operation of the clinical diagnostic efficacy experiment; Xiaosong Qin and Mei Yang were responsible for experimental design and article writing.

Fundings

This work was supported by Scientific Research Projects Related to Prevention and Control of Coronavirus Disease 2019 (COVID-19) of China Medical University, the 345-Talent Project of Shengjing Hospital of China Medical University, the Guangdong Province Major Key Projects of Industrial Technology [201902010003], the Emergency Scientific Research Project for Prevention and Control of COVID-19 of Liaoning Province (the fourth batch, No. 2020JH2/10300175).

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary material

The following are the Supplementary data to this article: