Evaluation of Intra- and Interlaboratory Variations in SARS-CoV-2 Real-Time RT-PCR Through Nationwide Proficiency Testing

Abstract

Objective

This study aimed to examine the intra- and interlaboratory variations of cycle threshold (Ct) values using the nationwide proficiency testing for SARS-CoV-2.

Methods

Triplicated strong-positive contrived samples duplicated weak-positive contrived samples, and 2 negative samples were transported to participating laboratories in October 2021.

Results

A total of 232 laboratories responded. All except 4 laboratories correctly answered. Six false-negative results, including 2 false-negatives with Ct values beyond the threshold and 1 clerical error, were noted from weak-positive samples. Intralaboratory variations of Ct values of weak-positive and strong-positive samples were not acceptable (Ct > 1.66) in 17 and 7 laboratories, respectively. High interlaboratory variations of Ct values (up to 7 cycles) for the 2 commonly used polymerase chain reaction (PCR) reagents were observed.

Conclusion

The overall qualitative performance was acceptable; intralaboratory variation was acceptable. However, interlaboratory variations of Ct values were remarkable even when the same PCR reagents were used.

During the coronavirus disease 2019 (COVID-19) pandemic, diagnostics is an important armamentarium in dealing with COVID-19. Several reported proficiency testing (PT) programs, with up to 930 participating laboratories for SARS-CoV-2 in Austria, China, Europe, South Korea, and the US, showed excellent performance and preparedness.^1–7 Cycle threshold (Ct) values are generally accepted as semiquantitative estimates of SARS-CoV-2 in samples and clinically regarded as indicators of infectivity in the real world.⁸ However, professional bodies discourage the clinical application of Ct values in qualitative testing.^9,10 Interlaboratory variation of Ct values was observed in reports from Austria and the US; therefore, caution needs to be exercised when interpreting Ct values of SARS-CoV-2 testing.^1,7,11 However, intralaboratory precision of Ct value has not yet been analyzed. Therefore, this study aimed to evaluate the status of SARS-CoV-2 reporting of laboratories in South Korea, a country with a low prevalence of SARS-CoV-2, and assess intralaboratory and interlaboratory variations of SARS-CoV-2 testing.

Materials and Methods

PT Scheme

Participation in the nationwide PT conducted by the Korean Association of External Quality Assessment Service (KEQAS) was mandatory for 232 laboratories performing SARS-CoV-2 real-time reverse transcription polymerase chain reaction (rRT-PCR) tests in Korea. Specimen preparation, validation and transport, and data reporting were performed as described previously.⁴ The Korea Research Institute of Standards and Science (KRISS) SARS-CoV-2 Proficiency Panel was adopted. This PT panel is composed of triplicated strong-positive samples, duplicated weak-positive samples, and 2 negative samples. Two-level concentrations of the positive reference material were prepared using the entire SARS-CoV-2 genome inserted into a lentivirus vector, and the negative reference materials included the human RNase P gene. Strong-positive samples and weak-positive samples showed E gene Ct values of 26.74 ± 0.16 and 32.79 ± 0.39 when tested in triplicate with eMAG (bioMérieux) and Allplex SARS-CoV-2 Assay (Seegene), respectively. These samples were transported to participating laboratories in October 2021, and responses were returned within 4 days. Due to the lack of personal identifiers and patient data in this study, the institutional review board of Asan Medical Center waived the ethics review (#2021-1772).

Data Analysis

Only samples showing ≥80% agreed response with the expected results were submitted for qualitative evaluation as recommended by the Clinical and Laboratory Standards Institute (CLSI). Intralaboratory variations were calculated using the maximum difference in the Ct values of RNA-dependent RNA polymerase (RdRp) genes for strong-positive samples and weak-positive samples, tested using PCR reagents in the same laboratory. The difference from false-negative responses was discarded from this analysis. The maximum difference of >1.66, reflecting a difference of 0.5 log concentration, was considered unacceptable for strong-positive and weak-positive samples. Meanwhile, interlaboratory variations were analyzed by comparing the Ct values of RdRp with extraction kits and PCR reagents using the box-and-whisker plot. MedCalc 20.015 (MedCalc Software) and Excel 2016 (Microsoft) were used for the descriptive statistical analyses.

Results

A total of 232 laboratories, including 35 public laboratories (26 laboratories operated by public health bodies, 5 army laboratories, and 4 national quarantine stations), participated. RNA extraction kits, extraction devices, PCR platforms, and PCR reagents were varied along with the protocols used by participating institutions (Supplemental Table 1).

All participating laboratories, except 4 laboratories, answered correctly. False-negative results from weak-positive samples were reported from 4 laboratories, as described in TABLE 1. Two laboratories using the Biosewoom Real-Q Direct SARS-CoV-2 Detection Kit incorrectly responded for 1 of the weak-positive samples; one completely missed 1 weak-positive sample for any of the target genes. Another laboratory incorrectly responded due to clerical error, and the fourth laboratory detected E gene with a Ct value >38 for both samples. However, the positive threshold of the Seasun Biomaterials U-TOP COVID-19 Detection Kit Plus was at Ct value 38.

TABLE 1.

Six False-Negative Cases From Four Participating Laboratories

Laboratory	Sample	Target Gene	Reported Ct Value	Other Target Gene Ct Value	PCR Reagent Used	Extraction Kit Used
1	WPS #2	E	ND	RdRp 35.93	Biosewoom Real-Q Direct SARS-CoV-2 Detection Kit	Alphagene Nucleic Acid Extraction Kit
2	WPS #1	RdRp	ND	E ND	Biosewoom Real-Q Direct SARS-CoV-2 Detection Kit	Real-Prep Viral DNA/RNA Kit
2	WPS #1	E	ND	RdRp ND	Biosewoom Real-Q Direct SARS-CoV-2 Detection Kit	Real-Prep Viral DNA/RNA Kit
3	WPS #1	RdRp	30.78	E 32.46	SDbiosensors STANDARD M nCoV Real-Time Detection Kit	Libex Viral DNA and RNA Extraction Kit
4	WPS #1	E	38.30	S 36.30, N 35.20, RdRp 33.90	Seasun Biomaterials U-TOP COVID-19 Detection Kit Plus	Others
4	WPS #2	E	39.90	S 37.00, N 36.50, RdRp 35.70	Seasun Biomaterials U-TOP COVID-19 Detection Kit Plus	Others

Ct, cycle threshold; E, envelope; N, nucleocapsid; ND, not detected; PCR, polymerase chain reaction; RdRp, RNA-dependent RNA polymerase; S, spike; WPS, weak-positive sample.

Intralaboratory variations of Ct of weak-positive and strong-positive samples were not acceptable (>1.66) in 17 (7.3%) and 7 (3.0%) laboratories, respectively (FIGURE 1). A majority (62.9%) of the participating laboratories used PCR reagents produced by SDbiosensors STANDARD M nCoV Real-Time Detection Kit (93; 40.1%) and Seegene Allplex SARS-CoV-2 Assay (53; 22.8%), as shown in Supplemental Table 1. Interlaboratory variation for these 2 PCR reagents is depicted in FIGURE 2. The ranges of the Ct values of RdRp were 21–28 (SDBiosensors) and 25–30 (Seegene) for strong-positive samples and 28–35 (SDBiosensors) and 32–37 (Seegene) for weak-positive samples.

FIGURE 1.

Intralaboratory variation of cycle threshold (Ct) values of RdRp for strong-positive (A) and weak-positive (B) samples. Intralaboratory variation was defined as the maximum difference between Ct values of RdRp from the same sample in the same laboratory. Negative results for strong-positive and weak-positive samples were excluded.

FIGURE 2.

Interlaboratory variation of cycle threshold (Ct) values of RdRp gene along with extraction kits for laboratories using SDbiosensors reagents (n = 93) and Seegene reagents (n = 53). Strong-positive samples using SDbiosensors reagents (A) and Seegene reagents (B). Weak-positive samples using SDbiosensors reagents (C) and Seegene reagents (D). Extraction kits: 1, AdvanSure R (LG Chem); 2, Alphagene Nucleic Acid Extraction Kit (Alphagene); 3, EZ1 Advanced XL RNA Card (Qiagen); 4, Genolution Viral NA Kit (Genolution); 5, Libex Viral DNA and RNA Extraction Kit (Tianlong); 6, NucleiSens easyMAG (bioMérieux); 7, QIAamp Viral RNA Mini Kit (Qiagen); 8, QIAcube Kit (Qiagen); 9, QIAsymphony DSP Virus/Pathogen Kit (Qiagen); 10, Real-Prep Viral DNA/RNA kit (Bioneer); 11, Seegene ProPrep (Seegene); 12, Seegene STARMag (Seegene); 13, Smart LabAssist Extraction Kit (TANBead); 14, TANBead Optipure Prep (TANBead); 15, Viral Nucleic Acid (small or large) Volume Kit (Roche); 16, other kits.

Discussion

This PT study showed the performance of commonly used molecular assays for SARS-CoV-2 detection in laboratories in South Korea. The overall qualitative performance of the participating laboratories was acceptable, and intralaboratory variation was acceptable in the vast majority (89.7%). Considering the well-known variability of weak-positive samples, 97.0% of participating laboratories reported Ct values with acceptable intralaboratory variability.

Six false-negative results were reported in this PT. However, only 1 laboratory missed 1 weak-positive sample without positivity of any target genes; the agent used in this laboratory targeted only 2 genes, RdRp and E genes. Other cases with false-negative results of a single target could be reexamined due to positivity in another gene. In light of the continuous emergence of novel SARS-CoV-2 variants,¹² PCR reagents targeting multiple genes have advantages over those with only 2 targets considering the dropout phenomenon.^13,14 However, not only PCR reagents but also extraction and sample preparation should be rigorously reviewed for satisfactory reporting of SARS-CoV-2.

Intralaboratory variation was acceptable for most laboratories. With this finding, follow-up of the Ct value in the same institution seems reasonable. However, variability can occur even in the sample acquisition process.^10,15 Furthermore, sampling using a nasopharyngeal swab, as recommended by the Centers for Disease Control and Prevention, is very painful and difficult in some instances¹⁶; thus, Ct values obtained on the same day from the same patient can be varied even if the same protocol is performed in the same institution. Therefore, caution is required in monitoring patients with COVID-19 in the same institution.

Conversely, interlaboratory variation of the Ct values for the 2 commonly used PCR reagent companies were observed up to 7 cycles, as shown in FIGURE 2. Previous reports from the US and Austria^1,7,11 pointed to remarkable interlaboratory variation. One report from Austria demonstrated a Ct value range for RdRp of 25.1 to 37.7 from a single sample.¹ The study noted interassay variation, in addition to sample volume used during extractions, as a major cause for this variation.^1,11 In the present study, high interlaboratory variations between laboratories using the same reagent and extraction protocol were observed. This finding is in line with the US study,⁷ which reported an interlaboratory variation of up to 14 cycles between laboratories using the same testing systems. Therefore, we discourage clinicians from interpreting the Ct values reported from other institutions, even if the same PCR reagent was used.

This study has some limitations. First, positive samples showing high viral loads were not included in this study; Ct values of all positive samples were higher than 20. Therefore, false-positives resulting from cross-contamination were not evaluated. Second, this PT was conducted in 1 country, so most laboratories used the PCR reagents from only 4 companies. International PT is required to investigate intra- and interlaboratory variations of various PCR reagents.

Conclusion

In conclusion, this study showed that the overall performance of the participating laboratories was satisfactory. However, a few laboratories with unsatisfactory results were also noted. In addition to evaluating the performance of the participating laboratories, PT can also examine the SARS-CoV-2 rRT-PCR protocols, including reagents and extraction methods. Interlaboratory variations in SARS-CoV-2 testing were remarkable even if the same extraction method and PCR reagent were applied. Therefore, attention is needed when using the Ct value to estimate the clinical status of patients with COVID-19, including their infectivity.

Supplementary Data

Supplemental figures and tables can be found in the online version of this article at  www.labmedicine.com

Glossary

Abbreviations

Ct

cycle threshold

PCR

polymerase chain reaction

COVID-19

coronavirus disease 2019

PT

proficiency testing

KEQAS

Korean Association of External Quality Assessment Service

rRT-PCR

real-time reverse transcription polymerase chain reaction

CLSI

Clinical and Laboratory Standards Institute

RdRp

RNA-dependent RNA polymerase

Funding

This work was supported by Korea Centers for Disease Control and Prevention (grant No. 4837-301) and the Korean Health Technology R&D Project through Korea Health Industry Development Institute (KHIDI), funded by Ministry of Health and Welfare, Republic of Korea (grant No. HI18C2383).