Laboratory verification of an RT‐PCR assay for SARS‐CoV‐2

Abstract

Background

Reverse transcription‐polymerase chain reaction (RT‐PCR) is an extremely common clinical method for detecting pathogens, particularly for emerging infectious diseases such as the new coronavirus disease (COVID‐19). Currently, detection of the RNA from the novel coronavirus SARS‐CoV‐2 is the gold standard for establishing a COVID‐19 diagnosis. This study evaluates the characteristic performance of the analytical system in a clinical laboratory.

Methods

A commercial SARS‐CoV‐2 RNA RT‐PCR Kit used in a clinical laboratory is assessed based on ISO 15189 verification requirements. A multiple real‐time RT‐PCR assay for the RdRP, N, and E genes in SARS‐CoV‐2 is verified.

Results

The analytical system exhibits good analytical sensitivity (1000 copies/mL) and specificity (100%); however, the values of 86.7% and 100% for analytical accuracy deserved attention, compared with two other types of methods. Overall, the kit is potentially useful for SARS‐CoV‐2 diagnostic testing and meets the verification requirements.

Conclusion

Compliance with international standards, such as ISO 15189, is valuable for clinical laboratories and for improving laboratory medicine quality and safety. Normalization is essential for obtaining reliable results from the SARS‐CoV‐2 RNA RT‐PCR assay. This study aims to develop an improved SARS‐CoV‐2 verification framework compared with traditional molecular diagnostic methods, given the urgency of implementing new assays in clinical laboratories.

Verification of interference and cross‐reactivity for analytical specificity. The cross‐reactivity test was conducted with influenza A and B viruses, Bordetella pertussis, and the other coronaviruses (SARS, MERS, NL63, HKU1, 229E, and OC43); the interference test was performed by adding mucin. No significant difference was observed.

1. INTRODUCTION

Human coronaviruses (hCoVs, including hCoV‐229, hCoV‐OC43, hCoV‐NL63, and hCoV‐HKU1) have historically been considered unimportant pathogens that cause the common cold in humans. However, severe acute respiratory syndrome coronavirus (SARS‐CoV) in 2002, Middle East respiratory syndrome coronavirus (MERS‐CoV) in 2012, and novel coronavirus (SARS‐CoV‐2) in 2019 have caused fatal respiratory diseases and outbreaks by crossing the species barrier for human‐to‐human transmission and have produced the most widespread global health events. ¹ , ² To date, the ultimate scope and effects of the SARS‐CoV‐2 outbreak area are currently unclear as the situation is rapidly evolving in other countries worldwide, ³ , ⁴ , ⁵ increasing the importance of a laboratory diagnosis; thus, testing experience in mature laboratories appears meaningful.

Historically, the goal of an infectious disease diagnosis has been to detect the source of the infection and enable an appropriate response. ⁶ The technological advantages of molecular diagnostic methods, such as next‐generation sequencing (NGS) and real‐time fluorescent PCR technology, have become particularly valuable for the detection of infectious agents that cannot be cultured or are difficult to culture. ⁷

RT‐PCR has been applied to detect pneumonia‐associated viral nucleic acids in SARS‐CoV‐2 infections during this outbreak and has been used to confirm COVID‐19. The uncharacteristic public debate regarding the high incidence of false‐negative results may have arisen, because many kits do not perform well in terms of sensitivity, which depends not only on the quality of the sample but also on the quality of the reagents, such as primers/probes.

In this study, RT‐PCR assay for SARS‐CoV‐2 testing is verified. Overall, we believe that an international standard, such as the ISO 15189 certification in our laboratory, is of considerable importance. With quality requirements as the guideline, we focus on maximally improving verification of molecular diagnoses in clinical laboratories.

2. MATERIALS AND METHODS

2.1. Study design

A kit specific for SARS‐CoV‐2 RNA is evaluated in our laboratory. Analytical accuracy, sensitivity, specificity, and stability are the main verification parameters. A high‐level sample of 2.0 × 10⁴ copies per milliliter (copies/mL) of SARS‐CoV‐2 pseudovirus (BDS Biotech) calculated by droplet digital PCR (ddPCR) and fluorescence quantitative PCR (qPCR) served as a reference material. A low‐level sample was diluted from the reference material of 2.0 × 10³ copies/mL to 1.0 × 10³ copies/mL, and the concentration was calculated by the standard curve constructed based on the standard product. The Department of Clinical Laboratory is a China National Accreditation Service for Conformity Assessment (CNAS)‐certified laboratory, and the ISO 15189 system has been in stable operation in this laboratory for 3 years. All medical staff involved in PCR testing were authorized and qualified professionals who obtained official permission to conduct these tests in China. Written informed consent was obtained from all patients, and the study protocol was approved by the Ethics Committee of Zhejiang University. The experimental methods were conducted in accordance with approved guidelines.

2.2. Viral nucleic acid extraction and RT‐PCR

RNA was extracted from samples arranged in pairs using a Viral RNA Extraction Kit and the Ex2400 extraction system (Liferiver). A sample volume of 300 µL was used for RNA extraction, and the elution volume was 50 µL. RT‐PCR was performed using a Novel Coronavirus Real‐Time RT‐PCR Kit (Liferiver) according to the manufacturer's protocol. Primers and probes were designed to target SARS‐CoV‐2 (GenBank accession number: MN908947). The following one‐step PCR protocol was used: one cycle at 45°C for 10 minutes and 95°C for 3 minutes, followed by 45 cycles at 95°C for 15 seconds and 58°C for 30 seconds, with single‐point fluorescence detection at 58°C. Finally, 25 µL of the total PCR volume was used according to the manual protocol. The detection limit of the RdRP qRT‐PCR assays was approximately 1.0 × 10³ copies/mL. Crossing point (Cp) values were used to determine SARS‐CoV‐2. Following color difference compensation, amplified viral fragments were detected in the FAM, HEX/VIC/JOE, and Cal Red 610/ROX/TEXAS RED fluorescence channels with a LightCycler 480II (Roche). Samples containing the RdRP gene with Cp values ≤ 43.0 were considered as positive for SARS‐CoV‐2 RNA. Samples containing the RdRP gene with Cp values > 43.0 or the N or E gene alone were considered as negative.

2.3. Analytical accuracy

The positive predictive value (PPV) and negative predictive value (NPV) indicate the analytical accuracy. At least 5 negative and positive samples (which should include weak positive/low‐amplification samples), and generally no fewer than 10 samples, should be selected for the final calculation. In accordance with the patient sample detection procedure, two reference methods and a candidate method are used for parallel detection. The assessment criterion is the performance declared by the manufacturer of the kit. Both reference kits were officially recommended by registration with the National Medical Products Administration.

2.4. Analytical specificity

Analytical specificity in terms of cross‐reactivity is determined by testing nine pathogens, influenza A/B viruses, Bordetella Pertussis, and other six coronaviruses, including SARS, MERS, NL63, HKU1, 229E, and OC43, which were prepared by the National Center for Clinical Laboratories in China. In this study, influenza A/B viruses and B pertussis were commonly detected in patients with infectious pneumonia. Influenza A and B RNAs were identified using kits (Liferiver), while B pertussis DNA was detected by another kit (Yilifang) according to the manufacturer's protocol. Analytical specificity in the interference test is performed by adding mucin (Worthington) to high‐level and low‐level samples, separately. A nonsignificant difference in viral load indicates no interference from mucin in any respiratory specimen.

2.5. Limit of detection

With reference to the CNAS‐GL039:2019 guidelines, certified reference materials declared that the limit of detection (LOD) concentration should be diluted. In this study, a low‐level sample with a viral load of 1.0 × 10³ copies/mL was used. The assessment criteria of the LOD test required detection in more than 90% of the samples (18 of the 20 positive samples). The mean value and standard deviation (SD) were calculated with reference to the Cp values of the RdRP/N/E genes, and viral loads were calculated by the standard curve of the RdRP gene.

2.6. Statistical analysis

Viral load was calculated by plotting Cp values onto the standard curve constructed based on the standard product (BDS Biotech), and the Cp values of the amplification curve were also recorded. Statistical analysis was performed using SPSS Statistics version 23 (IBM). A P value of ≤.05 was considered significant.

3. RESULTS

3.1. Performance characteristics of the SARS‐CoV‐2 RT‐PCR assay

Validated examination procedures were subject to independent verification in our laboratory. Analytical accuracy, sensitivity, and specificity are the essential verification parameters in qualitative testing; repeatability and stability are optional parameters. However, we evaluated all performance parameters of the SARS‐CoV‐2 assay in this study (Figure 1).

FIGURE 1.

Analytical accuracy with reference methods (A) and (B). With reference method (A), 100% PPV, NPV, and accuracy were observed, while 83.3% (62.6%~95.3%, 95% CI) PPV, 100% (54.1%~100%, 95% CI) NPV, and 86.7% (69.3%~96.2%, 95% CI) accuracy were observed with reference method (B)

3.1.1. Accuracy

To evaluate accuracy, 30 positive samples from patients with a respiratory disease (20 positive and 10 negative samples were tested with other RT‐PCR reagents) were tested in our analytical system. The results were consistent with those derived using reference method (a), with a positive predictive value (PPV) of 100% and a negative predictive value (NPV) of 100%. However, 83.3% (62.6%~95.3%, 95% confidence interval (CI)) of PPV and 100% of NPV (54.1%~100%, 95% CI) were calculated for reference method (b), which showed 86.7% (69.3%~96.2%, 95% CI) accuracy. The two reference methods have different analytical sensitivities of 1000 copies/mL and 500 copies/mL, and the kit that we used is more similar to reference method (a) (Figure 1).

3.1.2. Sensitivity

We evaluated the low‐amplification control for sensitivity testing to calculate the LOD of the analytical system, which is defined as the lowest concentration (1 × 10³ copies/mL for viral load) at which 100% of positive samples are detected. Viral RNA in 20 samples (the logarithmic value of viral load for the RdRP gene was 3.27 ± 0.30 (mean ± SD)) was detected with a positive rate of 100% in this study (Table 1). Differences in sensitivity exist in different target regions of SARS‐CoV‐2, and the Cp values (mean ± SD) for the RdRP, N, and E genes were 38.02 ± 2.50, 35.30 ± 4.32, and 33.73 ± 3.16, respectively (Table 1).

TABLE 1.

Analytical LOD and repeatability

Sample No.	The limit of detection concentration (1000 copies/mL)
Target gene(s)	Viral load (log10 (copies/mL))	RdRP (Cp)	N (Cp)	E (Cp)
1	2.87	42.79	22.69	30.91
2	3.54	34.96	32.73	31.71
3	3.58	34.44	32.56	31.09
4	3.40	36.59	32.97	32.50
5	3.63	33.91	33.94	30.33
6	3.43	36.21	32.71	30.98
7	3.24	38.40	36.91	37.17
8	3.26	38.27	36.48	36.14
9	2.92	40.52	35.36	35.52
10	3.25	38.44	35.51	34.79
11	3.79	34.92	33.24	18.18
12	3.52	36.73	36.8	31.9
13	3.14	39.14	39.98	33.56
14	3.15	39.10	38.59	37.82
15	3.54	36.56	31.44	29.73
16	3.27	38.29	36.85	NA
17	3.04	39.78	37.83	35.92
18	3.27	38.30	34.75	34.33
19	2.87	40.83	41.78	41.52
20	2.64	42.30	42.94	NA
Mean ± SD	3.27 ± 0.30	38.02 ± 2.50	35.30 ± 4.32	33.73 ± 3.16

Repeatability precision was determined by testing the limit of detection in a sample with a viral load of 1000 copies/mL, and results showed 100% detection at the LOD.

Abbreviation: NA, undetectable result.

3.1.3. Specificity

The analytical specificity of the SARS‐CoV‐2 RT‐PCR assay was evaluated by adding influenza A and B RNAs, B pertussis, and the other coronavirus (SARS, MERS, NL63, HKU1, 229E, and OC43) DNA into two levels of positive samples. The RT‐PCR results did not show cross‐reactivity with the influenza A/B viruses, Bordetella pertussis, and other coronaviruses (Figure 2).

FIGURE 2.

Verification of interference and cross‐reactivity for analytical specificity. The cross‐reactivity test was conducted with influenza A and B viruses, Bordetella pertussis, and the other coronaviruses (SARS, MERS, NL63, HKU1, 229E, and OC43); the interference test was performed by adding mucin. No significant difference was observed

3.1.4. Interference

Mucin was added to two levels of positive samples until a final concentration of 1 mg/mL was achieved. Based on the interference results, 1 mg/mL mucin did not exert a significant effect on the RT‐PCR assay (Figure 2).

3.2. Challenge study

Most kits for SARS‐CoV‐2 RNA testing have limitations, such as a brief development time, insufficient testing in preclinical trials, ongoing evolution of gene mutations, ¹ , ⁸ and sample collection limitations, among others. A verification assay was performed in a clinical laboratory with ISO 15189 certification. Due to the outbreak of COVID‐19, the amount of commercial reagent available for clinical trials is less sufficient than previously available amounts; thus, its clinical application is truly challenging in most clinical laboratories.

4. DISCUSSION

SARS‐CoV‐2 was identified as a new species of coronavirus similar to SARS‐CoV. Fortunately, the government of China made a massive effort to control the spread of the virus by implementing robust measures to detect cases early, isolate and care for patients, and trace contacts. Currently, molecular diagnostic reagent kits are being rapidly pushed toward application for SARS‐CoV‐2 RNA testing, allowing early recognition and isolation of individuals with COVID‐19. Until June 20, 2020, 22 reagent kits for SARS‐CoV‐2 RNA testing have been approved by the National Medical Products Administration in China. Since then, SARS‐CoV‐2 has been detected in several other countries; however, the number of tests and the sensitivity of reagents have caused a bottleneck effect. The analytical sensitivity is often referred to as the LOD, and the lowest actual concentration in a specimen should be detected. ⁷

Relevant validation and verification reports evaluating the performance of the SARS‐CoV‐2 analytical system in public use are not available. We consider this paper to be the first one to focus on improving internal quality control. This study was performed in a SARS‐CoV‐2 RNA testing laboratory that acquired ISO 15189 accreditation from the CNAS three years ago. From 29 January 2020 to 29 February 2020, 1759 tests for the SARS‐CoV‐2 RNA were completed in our laboratory.

Currently, SARS‐CoV‐2 is spreading relatively rapidly throughout China and worldwide. Clinical laboratories must improve the quality management of SARS‐CoV‐2 RNA tests, particularly in some laboratories with poor operational conditions in developing countries. Before a new test is used in a clinical laboratory, the performance characteristics of the procedure must be confirmed. ⁹ ISO 15189 defines the term “verification” as “confirmation through the provision of objective evidence, that specified requirements have been fulfilled.” Therefore, analytical verification of qualitative tests is absolutely evidenced by the accuracy, sensitivity, specificity, and interference data. ¹⁰ , ¹¹ , ¹²

The Real‐Time Multiplex RT‐PCR Kit (detection of 3 genes) can be performed basically consistently with the manufacturer's statement. The verification performance of the SARS‐CoV‐2 RNA test satisfied the assessment criteria, with a good result for the LOD, PPV, NPV, cross‐reactivity, and interference. However, we must consider single fluorescence channel interference in the LightCycler 480II, and detection of the N and E genes by the kit showed instability. The reagents for SARS‐CoV‐2 RNA assay are currently available in single‐, double‐, and triple‐target genes, which have different judgment criteria for the results. Taking the triple‐target detection reagent as an example, some reagents require two target genes to be positive at the same time for a positive determination. Some reagents require two in three target genes to be positive (where RdRp must be positive), but if RdRp alone is positive in the retest, the result can also be considered positive; however, if RdRp is negative, even if both N and E are positive, the result cannot be positive for SARS‐CoV‐2. The number of target genes and determination rules is related to the selection of the overall methodology design (including primers, probes, and detection areas) when the reagent is established. In our study, differences were found in the sensitivity of the three target genes, and the E gene has a 20% (2/20) missed detection rate (Table 1). Overall, reagent manufacturers should establish reagent methods based on certain scientific and experimental data. Both the PPV and NPV were 100% against the reference method (a), which has the same LOD concentration of 1000 copies/mL. To our surprise, a PPV of 83.3% and an NPV of 100% were obtained for reference reagent (b), which has higher sensitivity with an LOD concentration of 500 copies/mL. In our study, this kit for SARS‐CoV‐2 RNA detection may not produce results consistent with the manufacturer's statement of 100% accuracy. Therefore, verification of the accuracy with other reference methods is needed in future studies.

The LOD is another important performance characteristic of both quantitative and qualitative tests. Analytical performance at the low concentration limit is often defined as the ability of the test to diagnose a disease. The manual provided with the kit indicated that a concentration of 1000 copies/mL with a Cp value of 43 was considered the minimal concentration. At present, no international or national standard substance of SARS‐CoV‐2 RNA is available; however, reference materials of SARS‐CoV‐2 RNA can be acquired, and the quantitative value can be determined by ddPCR. The reference material containing 2.0 × 10³ copies/mL was diluted to 1.0 × 10³ copies/mL as low‐level positive samples for testing (Table 1). At the same time, the viral load in the tests was determined by fitting a standard curve with different concentrations of the reference material to the measured values (Figure 3). In addition, a Cp value of 38.02 ± 2.50 (RdRP gene) represents the minimal concentration, but a Cp value of 43 is considered the minimal concentration according to the manual of the SARS‐CoV‐2 RNA Kit.

FIGURE 3.

Standard curve of SARS‐CoV‐2 RNA. The concentrations of the reference materials are 2.0 × 10⁴/1.0 × 10⁴/5.0 × 10³/1.0 × 10³ copies/mL. The x‐axis shows the logarithmic values of viral load, and the y‐axis represents the Cp values of RNA amplification

Cross‐reactivity and interference are two components of analytical specificity. False‐positive results may be caused by contamination with other organisms and nucleic acids. Various organisms may exist in specimens and cause similar clinical symptoms; thus, we selected the respiratory organisms influenza A and B viruses, B pertussis, and other coronaviruses (SARS, MERS, NL63, HKU1, 229E, and OC43) to test cross‐reactivity, as simulated samples of bacteriophage virus‐like particles, which were prepared by genetic engineering methods with no risk of biological transmission. The viral loads of the other coronaviruses determined by ddPCR are as follows: 3.0 × 10⁶ copies/mL of SARS, 1.6 × 10⁶ copies/mL of NL63, 9.4 × 10⁵ copies/mL of HKU1, 2.7 × 10⁶ copies/mL of 229E, 1.0 × 10⁴ copies/mL of MERS, and 1.0 × 10⁶ copies/mL of OC43. On the other hand, interfering substances present in specimens might affect polymerase activity or inhibit DNA/RNA amplification, and these effects must be assessed. Actually, skill in nucleic acid extraction procedures, including delaying the nucleic acid lysis time and performing immediate centrifugation in the last step for bead precipitation, is often helpful for inactivating or removing interfering substances.

In addition, our study had some limitations. First, we need to compare more RNA extraction reagents to identify a suitable reagent, because the quality of the nucleic acid extraction reagent is a key element for the success of testing, which depends on the adsorption efficiency of the magnetic bead, the sample size for nucleic acid extraction, and the elution volume. Nucleic acid detection is a process of extraction followed by amplification. Second, the results of this study do not reflect the advantages of the three target genes used for detection, such as high sensitivity and specificity, and the interpretations provided by the kits are not very clear. The interpretation rules for single‐ or double‐positive target genes will lead to differences in the interpretation of positive results (negative/positive). We are not sure that the proportion of false‐positive samples among all single‐target gene (N or E gene) positive samples exists. A sample positive for the RdRP gene alone requires consistent retesting to determine positivity; thus, what are the false‐positive rates without retesting? Accordingly, we suggest that reagent manufacturers need more data to optimize reagent performance.

In summary, verification of the SARS‐CoV‐2 RNA test was consistent with the product requirements, and the detection system basically meets the detection performance stated in the kit. Finally, we postulate that this study will be useful for other clinical laboratories, before using a new analytical system for molecular diagnosis.

AUTHORS' CONTRIBUTIONS

All authors confirm that they contributed to the content of this paper. W Xu and ZQ Zhu assisted with sample testing. YP Wu and XP Xia were responsible for the study design and data interpretation and were major contributors to manuscript editing and critical revision of the article. All authors read and approved the final manuscript. We are grateful to all colleagues in our laboratory for their support.

Wu Y, Xu W, Zhu Z, Xia X. Laboratory verification of an RT‐PCR assay for SARS‐CoV‐2. J Clin Lab Anal. 2020;34:e23507 10.1002/jcla.23507