Performance of RT-qPCR detection of SARS-CoV-2 in unextracted nasopharyngeal samples using the Seegene Allplex^TM 2019-nCoV protocol

Abstract

The rapid spread of the Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) has led the world to a pandemic. Therefore, rapid, sensitive, and reproducible diagnostic tests are essential to indicate which measures should be taken during pandemics. We retrospectively tested unextracted nasopharyngeal samples from consecutive patients with suspected SARS-CoV-2 infection (n = 334), and compared two different Ct cut-off values for interpretation of results using a modified Allplex protocol. Its performance was evaluated using the USA Centers for Disease Control and Prevention (CDC) as reference. The reduction on Ct cut-off to 35 increased the test NPA from 79.65 to 88.00 %, reducing the number of false positives, from 10.48 to 6.29 %, resulting in an almost perfect agreement between the Allplex and the CDC protocol (Cohen’s Kappa coefficient = 0.830 ± 0.032). This study demonstrates that the Seegene Allplex™ 2019-nCoV protocol skipping the viral RNA extraction step using the Ct cut-off of 35 is a rapid and efficient method to detect SARS-CoV-2 in nasopharyngeal samples.

1. Introduction

The global pandemic declared by the World Health Organization (WHO) in 2020 is the result of respiratory disease COVID-19 in the occurrence of acute severe respiratory syndrome coronavirus 2 (SARS-CoV-2). By October 2021, over 238 million cases had been diagnosed (World Health Organization, 2021). Therefore, the validation of diagnostic methods is indispensable, requiring procedures with high reliability as well as accurate repeatability, showing agreement between replicates and decreasing the chance of false positives or false negatives. Valid methods that allow a precise estimation of the infected people are essential to know the levels of etiological agent dissipation and to be able to take timely containment measures (Bezier et al., 2020; World Health Organization, 2021).

Due to its great reliability, reverse transcription quantitative real-time polymerase chain reaction (RT-qPCR) is the gold standard method for the molecular detection of SARS-CoV-2. It is based on the extraction of viral RNA sampled in the nasopharyngeal swab from the patient, followed by reverse transcription for the quantitative amplification of viral cDNA (El Jaddaoui et al., 2021). Until now, to detect SARS-CoV-2 using RT-qPCR, different diagnostic kits have been developed (Park et al., 2020). The type of assay as well as the specific genetic regions used as a target are extremely important for the reliability of the viral diagnosis of COVID-19 (Habibzadeh et al., 2021).

Most COVID-19 molecular diagnostic kits target viral genes encoding structural proteins: spike (S), envelope (E), membrane (M) and nucleocapsid (N). Nevertheless, non-structural proteins are also part of protocols such as the RNA-dependent RNA polymerase (RdRp) (Melo-Vallès et al., 2021). The Allplex™ 2019-nCoV (Seegene) was designed to amplify the E, N and RdRP genes, in a multiplex assay (Farfour et al., 2020). This protocol could also be adapted to reduce the processing time of tests, suppressing the RNA extraction step by a quick thermal and enzymatic lysis, followed by E, N and RdRP genes amplification using RT-qPCR (Freppel et al., 2020). As stated in the protocol proposed by Freppel et al. (2020), samples were diluted at 1:5 in proteinase K solution. After incubation at 50 °C for 15 min, samples underwent thermal lysis at 90 °C per 3 min in a thermocycler and then cooled to 4 °C before the RT-qPCR test. According to the same authors, proteinase K reduces the number of invalid results and increases sensitivity to samples with low viral load.

In this study, we investigate the analytical performance of RT-qPCR detection of SARS-CoV-2 in unextracted nasopharyngeal samples using the Seegene Allplex™ 2019-nCoV protocol adapted by Freppel and co-workers (Freppel et al., 2020) in comparison to the CDC standard method (USA CDC, 2020) and propose a different Ct cut-off for interpretation of results using the Allplex protocol.

2. Material and methods

2.1. Sample collection and experimental design

Nasopharyngeal swabs (flexible swabs with 3 mL of saline; n = 334) from consecutive patients seeking specialized care due to suspicion of SARS-CoV-2 infection were collected by the Municipal Health Department of Dois Vizinhos-Paraná-Brazil, from February to April 2021, following the American Centers for Disease Control and Prevention guidelines for collection, transportation and storage of samples (USA CDC, 2020). The Molecular Biology Laboratory at the Universidade Tecnológica Federal do Paraná in Dois Vizinhos handled all tests and data to preserve patient anonymity. All samples were stored at −80 °C until analysis.

SARS-CoV-2 diagnostic was firstly performed according to the USA CDC 2019-Novel Coronavirus (2019-nCoV) RT-qPCR Diagnostic Panel, chosen as reference test (USA CDC, 2020). The CDC protocol was the first approved for emergency use by the USA Food and Drug Administration (FDA) and it has been adopted by many countries as the main protocol (Giri et al., 2021). Next, the same samples were retrospectively tested using the modified protocol for unextracted nasopharyngeal swab samples using the Allplex™ 2019-nCoV Assay kit, according to Freppel et al. (2020).

2.2. The Allplex protocol

The modified Allplex protocol (Freppel et al., 2020) was evaluated in an attempt to accommodate the use of Allplex™ 2019-nCoV (Seegene, Seoul, Korea) kits by the Multiuser Core Laboratory of Biological Analysis and the Molecular Biology (BioMol) laboratory at the Universidade Tecnológica Federal do Paraná. Preliminary tests using the manufacture’s protocol provided a significant number of invalid results with a high number of false positive results. A similarly high number of false positive results were obtained when a different RT-qPCR detection system was used, such as the QIAquant 96 5plex (Qiagen, MD, USA) (see web-only Supplementary Table S1). Briefly, thawed samples were diluted 1:5 in RNase-free water with 125 μg/mL Proteinase K (Bioclin, Minas Gerais, Brazil). 50 μL of diluted samples were incubated at 50 °C for 15 min followed by thermal lysis at 90 °C for 3 min in the CFX96 Touch Real-Time PCR Detection System (Bio-Rad, CA, USA). The multiplex RT-qPCR (Allplex™ 2019-nCoV assay, Seegene, Seoul, Korea) was carried out in duplicate at 50 °C for 20 min (step 1), 95 °C for 15 min (step 2) and 45 cycles at 95 °C for 15 s (step 3) and 58 °C for 30 s (step 4) using the CFX96 Touch Real-Time PCR Detection System. The fluorescence signal was detected on step 4 using the channels FAM (E gene), HEX (Internal Control), Cal Red 610 (RdRP gene), and Quasar 670 (N gene), and evaluated using the CFX Maestro Software (BioRad) according to the manufacturer's instructions. Positive (2019-nCoV PC, Allplex) and non-template control reactions were performed for every RT-qPCR run. Positive, negative and invalid results were interpreted according to Allplex™ 2019-nCoV assay manual using the Ct ≤ 40 cut-off (Seegene, 2021) compared to an alternative cut-off at Ct ≤ 35. The lower Ct cut-off was chosen due to the high number of false positive results obtained in preliminary data (see web-only Supplementary Table S1). To check the reproducibility of results obtained with the modified Allplex protocol, a subset of samples was analyzed using the QIAquant 96 5plex (Qiagen, MD, USA) and the Chromo4™ System (Bio-Rad, CA, USA) equipment.

2.3. The CDC protocol

For virus RNA extraction, 120 μL of thawed samples were incubated at 56 °C for 10 min with 330 μL of lysis solution and transferred to the Maxwell® RSC Instrument cartridges for RNA extraction according to manual instructions (Maxwell® 16 Viral Total Nucleic Acid Purification Kit, Promega, Wisconsin, USA). Undiluted RNA samples were subjected to a simplex RT-qPCR using the QuantiTect Probe RT-qPCR Kit (QIAGEN, Hilden, Germany) at 50 °C for 30 min (step 1), 95 °C for 15 min (step 2), and 45 cycles at 94 °C for 15 s (step 3) and 55 °C for 1 min (step 4) using the CFX96 Touch Real-Time PCR Detection System. Data collection occurred on step 4 using the FAM channel for all targets (N1, N2, and RP) and it was evaluated using the CFX Maestro Software. Human specimen control (HSC), Positive (2019-nCoV_N_Positive Control, Integrated DNA Technologies Inc, Iowa, USA), and Non-template control reactions were performed in every RT-qPCR run. The interpretation of the results (positive, negative, invalid or inconclusive) was performed according to the CDC 2019-Novel Coronavirus (2019-nCoV) Real-Time RT-qPCR Diagnostic Panel (USA CDC, 2020).

2.4. Statistical analysis

To characterize the cycle threshold (Ct) intervals for each protocol, unmatched nonparametric Ct values (dependent variable) from samples considered positive and negative (independent variables) were subjected to the Mann-Whitney distribution test with 5% statistical significance. Correlation among Ct values for viral gene targets was calculated using the nonparametric Spearman correlation test. A unique ID was attributed to each nasopharyngeal sample for protocol performance comparison using the Cohen's Kappa coefficient and Yate's continuity corrected chi-square test with a 95 % confidence interval. Statistical analyses were performed using GraphPad Prism (GraphPad Software, California, USA). Positive per cent agreement (PPA) and negative per cent agreement (NPA) were calculated as indicated in the formulas bellow. True positive and true negative samples were detected with the CDC protocol. Samples with invalid and inconclusive results were excluded from analysis.

$$PPA=\frac{Positivesamplesinbothprotocols}{Truepositivesamples+Falsenegativesamples}\times 100$$

$$NPA=\frac{Negativesamplesinbothprotocols}{Truenegativesamples+Falsepositivesamples}\times 100$$

3. Results

Distinct Ct intervals were observed with the Allplex detection kit when positive and negative samples were compared. A difference was detected for the RdRP (p < 0.0001, Fig. 1 c; p = 0.0001, Fig. 1f) and the N (p < 0.0001; Fig. 1d, g) genes. For the E gene, no signal was detected in any of the samples considered negative, according to the Allplex kit manufacturer’s manual (Fig. 1e, h) (Seegene, 2021). These results demonstrate that all target genes clearly displayed different Ct intervals between positive and negative samples regardless of the Ct cut-off applied. Similar results were observed for viral gene targets (N1 and N2) of the CDC protocol (p < 0.0001; Fig. 1a, b).

Fig. 1.

Box plot of Ct intervals for viral targets N1 and N2 using the CDC Protocol (a, b) and RdRP, N, and E using the Allplex detection kit (c-h) between samples with Positive and Negative outcomes. For the Allplex kit an additional comparison was performed between Positive samples according to two different Ct cut-offs: 35 (i-k) (** for p ≤ 0.01; *** for p ≤ 0.001 and **** for p ≤ 0.0001). Box-and-whiskers plot: central line in box is median, bottom line of box is first quartile (25 %), top line of box is third quartile (75 %), bottom of whiskers is the smallest value, and top of whiskers is largest value.

No differences were observed between Ct intervals of positive samples according to different Ct cut-offs (35 and 40) except for the N gene (p = 0.0019; Fig. 1j). When a later cycle cut-off was chosen to identify positive samples, the upper limit of the Ct interval for the N gene was significantly higher, from 36.29 (Ct 35 cut-off) to 39.89 (Ct 40 cut-off), resulting in more samples being considered as positive, being further away from the results obtained with the CDC protocol (see web-only Supplementary Table S2).

Next, we verified how closely the Allplex target genes correlate with each other using the Pearson correlation test (r) of Ct values of E and N genes (Fig. 2 b, c), E and RdRP genes (Fig. 2d, e), and N and RdRP genes (Fig. 2f, g) from positive samples according to each Ct cut-off (35 and 40). A positive correlation was observed among Ct values for each pair analysed (Fig. 2; p < 0.0001). Higher correlation coefficients (>0.95) were observed between E and RdRP followed by E and N genes while the correlation between N and RdRP genes showed the lowest coefficient (<0.88). The highest correlation coefficient was observed between N1 and N2 gene targets of the CDC protocol (p < 0.0001; r = 0.9899; Fig. 2a).

Fig. 2.

Scatter plots of Ct values for viral targets. CDC Protocol: N1 x N2 (a); Allplex detection kit: E x N (b, c), E x RdRP (d, e), and N x RdRP (f, g). For the Allplex protocol, two Ct cut-off values were considered to define "positive" outcomes: Ct ≤ 40 (black dots) and Ct ≤ 35 (brown dots).

Although a substantial agreement between the CDC protocol and the Allplex detection kit with a Ct cut-off of 40 was observed, indicated by a Cohen's Kappa coefficient of 0.757 (Table 1a), the frequency of positive and negative outcomes obtained with these protocols was different (p = 0.0091; Table 2a). From the 139 negative samples using the Allplex protocol, 137 were indeed negative, whilst only 127 were correctly detected as positive out of 162. The positive percent agreement (PPA) between these groups was 98.4496 % while the negative percent agreement (NPA) was only 79.6512 % (see web-only Supplementary Table S3).

Table 1.

Agreement between CDC and Allplex (Ct ≤ 40 and Ct ≤ 35) results [Cohen's Kappa coefficient, standard error of the mean (SEM), and confidence interval (95 % CI)].

a
		CDC		Total	Kappa	SEM	95 % CI
		Negative	Positive
Allplex Ct ≤ 40	Negative	137	2	139	0.757	0.037	0.685 - 0.828
	Positive	35	127	162
	Total	172	129	301

b
		CDC		Total	Kappa	SEM	95 % CI
		Negative	Positive
Allplex Ct ≤ 35	Negative	154	4	158	0.830	0.032	0.767 - 0.894
	Positive	21	119	140
	Total	175	123	298

Kappa scale: Kappa < 0: No agreement; 0.00−0.20: Slight agreement; 0.21−0.40: Fair agreement; 0.41−0.60: Moderate agreement; 0.61−0.80: Substantial agreement; 0.81–1.00: Almost perfect agreement (Landis and Koch, 1977).

Table 2.

Frequency comparison of test results between kits considering two different Ct cut-offs for the Allplex protocol [Chi-Square, p value, and degrees of freedom (df)].

a
	Allplex Ct ≤ 40	CDC	Chi-Square; p value; df
Negative	139 (46.18 %)	172 (57.14 %)	6.812; 0.0091; 1
Positive	162 (53.82 %)	129 (42.86 %)
	301 (100 %)	301 (100 %)

b
	Allplex Ct ≤ 35	CDC	Chi-Square; p value; df
Negative	158 (53.02 %)	175 (58.72 %)	1.742; 0.1869; 1
Positive	140 (46.98 %)	123 (41.28 %)
Total (%)	298 (100 %)	298 (100 %)

A higher NPA (88 %) was observed when a lower Ct cut-off was used to detect positive samples with the Allplex kit compared to the CDC protocol. A PPA of 96.7479 % was obtained between these groups. From the 158 negative samples using the Allplex Ct ≤ 35 protocol, 154 were negative using the CDC test, whilst 119 were correctly detected as positive out of 140 (see web-only Supplementary Table S4). Moreover, an almost perfect agreement between these protocols was detected as indicated by a Cohen's Kappa coefficient of 0.83 (Table 1b), and no differences were found when outcome frequencies were compared by Chi-Square analysis (p = 0.1869; Table 2b).

The total number of samples with valid results varied according to the Ct cut-off applied since it affected the number of invalid and/or inconclusive results (see web-only Supplementary Tables S3 and S4).

When samples were divided as positive or negative according to the CDC protocol, heatmaps of Ct values for each viral target (Fig. 3 for negative samples and web-only Supplementary Fig. S1) show a higher discrepancy between protocols when a later cycle cut-off was applied for the Allplex group (Fig. 3a). Moreover, all three targets of the Allplex detection kit were amplified in some of the true negative samples, including the E gene, which showed no amplification in negative samples according to the Allplex kit’s manual (Seegene, 2021). Among those three, the N gene showed the highest degree of disagreement.

Fig. 3.

Heatmap of Ct values for paired negative samples (Allplex-CDC) according to CDC protocol. (a) Allplex protocol results using the Ct 40 cut-off and (b) Ct 35 cut-off. ND = not detected.

Result reproducibility is crucial for a diagnostic test, especially in a pandemic context. When the modified Allplex protocol was applied to a subset of 28 samples (14 positive and 14 negative samples according to the CDC protocol), the Ct cut-off of 40 only showed fair agreement, at most, between different equipment (Cohen's Kappa coefficient of 0.075 ± 0.207 for CFX96 vs QIAquant and 0.241 ± 0.161 for CFX96 vs Chromo4™) while the lower Ct cut-off provided better moderate to substantial agreement coefficients (Cohen's Kappa coefficient of 0.562 ± 0.159 for CFX96 vs QIAquant and 0.781 ± 0.116 for CFX96 vs Chromo4™)(see web-only Supplementary Table S5).

4. Discussion

The present study shows an almost perfect agreement between the use of a modified protocol to detect SARS-CoV-2 in unextracted nasopharyngeal samples with the Allplex kit (Freppel et al., 2020) and the reference test based on the USA CDC protocol (USA CDC, 2020), as long as a lower Ct (Ct = 35) is considered as a cut-off.

The use of the standard Ct cut-off of 40 significantly affected the Ct interval for the N gene, resulting in more samples being considered positive, consequently increasing the number of false-positive tests in this group. In previous studies using the Allplex kit with extracted samples, higher Ct values when amplifying the N gene were also observed, in comparison to the other two viral targets (E and RdRP) (Farfour et al., 2020; Freppel et al., 2020; Lee et al., 2021). This may be due to genomic variability (Ceraolo and Giorgi, 2020; Freppel et al., 2020; Wang et al., 2020). The N gene is responsible for the production of a nucleocapsid structural protein and it is a target in both protocols. While for the CDC protocol two different regions of the N gene are amplified (N1 and N2), the Allplex kit has only one target sequence for the N gene. Recent data suggest that the N gene may be undergoing indel mutations, making it more difficult to detect the virus in the Allplex assay (Lee et al., 2021). Such mutations could affect the Allplex protocol’s reliability for this particular target.

The Allplex is a multiplex kit and the other two gene targets, E and RdRP, showed similar performances when different Ct cut-offs were compared. According to Allplex™ 2019-nCoV assay manual guidelines (Seegene, 2021), the detection of the E gene should always accompany at least one of the other two viral targets, N or RdRP, for a sample to be considered positive. Moreover, the European Center for Disease Prevention and Control also does not recommend the use of gene E alone for the SARS-CoV-2 diagnostics due to its low specificity (Colton et al., 2021). In this study, when samples were tested as positive or negative using the Allplex protocol, we observed the amplification of the E gene only in positive samples regardless of the Ct cut-off applied. Another study about the original Allplex protocol performance has reported a PCR efficiency of 105 % for the E gene (van Kasteren et al., 2020) and a similar high efficiency was also reported by Hur et al. (2020), but with reproducible results obtained only when a lower Ct cut-off was applied for the E gene. Nevertheless, when the CDC protocol was used as the reference to determine which samples were indeed positive or negative, the E gene did not show the same specificity as some true negative samples also displayed its amplification (Fig. 3). This may be related to specific mutations in this gene that have already been observed in SARS-CoV-2 genomes on different continents, signaling the widespread of this mutation (Tahan et al., 2021).

The third target of the Allplex protocol is the RdRP gene, which encodes an RNA-dependent RNA polymerase, part of the SARS-CoV-2 replication–transcription complex (Slanina et al., 2021). Like the N gene, its amplification alone is enough for a sample to be considered positive (Seegene, 2021). Unlike the N gene, the Ct interval for the RdRP gene was not affected by the Ct cut-off applied using the Allplex protocol. Moreover, of all correlation plots among viral gene targets, the one between the RdRP and the N gene presented the lowest correlation coefficient.

Other authors compared several commercial RT-qPCR Kits approved for COVID-19 testing, with the Allplex kit presenting excellent performance with approximately 100 % specificity and precision (Garg et al., 2021; Hur et al., 2020). However, when we used the Allplex kit to detect SARS-CoV-2 from unextracted nasopharyngeal samples, considering Ct cut-off of 40 (recommended by the manufacturer) we noticed an increase in the number of false positive results. Additionally, we observed that the modified Allplex protocol was not significantly different from the reference diagnostic test only when the Ct cut-off of 35 was applied.

5. Conclusions

This study demonstrates that the use of the Seegene Allplex™ 2019-nCoV on unextracted nasopharyngeal samples step is an efficient method to detect SARS-CoV-2, as long as a Ct cut-off of 35 is applied. Considering the Ct cut-off reduction only affected the number of positive samples according to the amplification of the N gene, we propose the use of a lower Ct cut-off to increase the test NPA reducing the chance of false positives mainly due to a lower specificity of the N gene when only one region is amplified. Further studies applying genotyping techniques would help shedding light on the mechanisms behind the diagnostic performance difference obtained with the Allplex detection kit.

CRediT authorship contribution statement

Flavia R.O. Barros: Conceptualization, Methodology, Formal analysis, Visualization, Writing - original draft, Writing - review & editing. Deborah C.A. Leite: Conceptualization, Methodology, Validation, Writing - original draft, Writing - review & editing. Larissa J. Guimarães: Investigation, Writing - original draft. Juliana M.H. Lopes: Investigation. Marina W. Vasconcelos: Investigation, Writing - original draft. Luciane X. Ferreira: Investigation, Writing - original draft. Sandrieli Gonçalves: Investigation, Writing - original draft. Victor G.F. Pastre: Investigation, Writing - original draft. Gabriela Pereira: Investigation, Writing - original draft. Alex B. Trentin: Investigation, Writing - original draft. Naiana C. Gabiatti: Investigation, Writing - original draft. Betty C. Kuhn: Investigation, Writing - original draft. Juliana M.K.C. Perseguini: Investigation, Writing - original draft. Simone N. Wendt: Funding acquisition, Writing - original draft. Nédia C. Ghisi: Supervision, Funding acquisition, Project administration, Writing - original draft.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following is Supplementary data to this article:

Data availability

Data will be made available on request.