Abstract
We evaluated the diagnostic performance of 4 commercially NAAT for detecting SARS-CoV-2 RNA, Influenza type A/B virus and RSV. Included tests were the Allplex™ SARS-CoV-2 fast PCR Assay (RNA extraction-free), Allplex™ RV Master Assay, Allplex™ SARS-CoV-2 fast MDx Assay (LAMP) and Aptima™ SARS-CoV-2/Flu Assay (RT-TMA). The assays’ performance characteristics were determined using nasopharyngeal swabs from 270 patients with suspected SARS-CoV-2 infection. A total of 215 SARS-CoV-2 positive, 55 negative nasopharyngeal swabs and 19 bacteria strains were included. The sensitivities and specificities for detecting SARS-CoV-2, Influenza type A virus and RSV ranged between 81.8% and 100% with extremely good agreements (κ ≥ 86.8 %). The Aptima™ SARS-CoV-2/Flu Assay introduced a new result parameter, that is, TTime. Here, we showed that TTime may be used as a surrogate for Ct-value. We concluded that all assays assessed in this study can be used for routine detection of SARS-CoV-2, Influenza type A virus and RSV.
Keywords: SARS-CoV-2, COVID-19, RT-PCR, RT-TMA, Comparison
1. Introduction
Accurate and rapid diagnostic tools are pivotal in combating infectious diseases by identifying cases for tract and tracing and to mitigate the burden of disease [1]. Nucleic acid amplification tests (NAATs) have been rapidly developed to identify patients with respiratory virus infection, to aid in both treatment and disease control decisions. Respiratory viruses with highest impact on global disease burden are currently SARS-CoV-2, Influenza type A/B virus and RSV [2]. To date, numerous commercially available assays have been developed for these viral respiratory tract infections [3]. Although information on the analytical performances is available for most diagnostic performance characteristics requires independent evaluations before implementation.
The aim of this study was to evaluate the diagnostic performance of 4 new commercially available nucleic acid amplification tests (NAAT) for detecting SARS-CoV-2 RNA, Influenza type A/B virus and RSV at primary health care level, designed to improve turn-around time (TAT) and test capacity within routine laboratory facilities.
2. Materials and methods
2.1. Study population and sample collection
Patients having a fever (>38°C) and coughing or shortness of breath were suspected for a SARS-CoV-2 infection. These patients were sampled from the oral and nasal cavity using a single nasopharyngeal swab. We included nasopharyngeal swabs in Aptima® Multitest Swab (Hologic Inc., Marlborough, MA), eSwab® (COPAN Diagnostics Inc., Brescia, Italy) and Transwab® (MWE Medical Wire, Corsham, United Kingdom) from patients presented at primary health care facilities or nursing homes based on results from either the (1) Allplex™ SARS-CoV-2 Assay, or (2) Allplex™ SARS-CoV-2/FluA/B/RSV Assay (Seegene Inc., Seoul, Republic of Korea) or (3) Aptima™ SARS-CoV-2 Assay (Hologic Inc., Marlborough, MA) between November 2021 and April 2022.
2.2. Reference results
Nasopharyngeal swabs were tested in both the Allplex™ SARS-CoV-2/FluA/B/RSV Assay and Aptima™ SARS-CoV-2 Assay. Swabs were defined as positive reference samples when tested positive in both these assays. Likewise swabs were defined as negative reference samples when tested negative in both assays. This study used only anonimized data extracted from routine diagnostics, therefore no ethical committee approval was obtained.
2.3. Seegene Allplex assays and Hologic Aptima assays
RNA was extracted using the Seegene STARMag 96 × 4 Universal Cartridge Kit on the Seegene STARlet system. The Allplex™ SARS-CoV-2 fast PCR Assay (RNA extraction-free RT-PCR), Allplex™ RV Master Assay (conventional RT-PCR) and Allplex™ SARS-CoV-2 fast MDx Assay (RNA extraction-free loop mediated isothermal amplification (LAMP)) were all performed according to the manufacturer's instructions using the CFX96 Real-Time Detection System (Bio-Rad, Hercules, CA). Results were analyzed using the Seegene software and displayed with the Seegene viewer on the monitor.
The Aptima™ SARS-CoV-2/Flu Assay was performed on the Panther system (Hologic Inc.; Marlborough, MA) that utilizes the combined technologies of target capture, real-time Transcription Mediated Amplification (RT-TMA), and detection of amplicons using fluorescently labeled torches. Qualitative results were accompanied with TTime. Supplementary Table 1 describes the specifications of these assays. Raw data in this study is available in supplementary Table 3.
2.4. PCR efficiency and statistical analyses
To calculate the PCR efficiency, a 10-fold dilution of the first WHO International Standard for SARS-CoV-2 RNA (NIBSC code 20/146) [4] was tested for each of the RT-PCR based Seegene assay. PCR efficiency (E) was calculated using the formula (-1 + 10−1/slope)*100. The inter- and intra-assay variations were calculated by assessing the same positive sample in 10 different runs and in the same run, respectively. Sensitivity, specificity, accuracy and Cohen's kappa with its 95% confidance intervals (95% CIs) were estimated. For the specificity analysis, 19 bacteria strains were added, that is, H. influenzae (n = 5), S. pneumonia (n = 5), S. aureus (n = 5), P. aeruginosa (n = 4). Delta Ct-value (ΔCt) was defined as the absolute difference in Ct between obtained Ct from the Allplex™ SARS-CoV-2/FluA/B/RSV Assay and the Allplex™ SARS-CoV-2 fast PCR Assay, Allplex™ RV Master Assay or Allplex™ SARS-CoV-2 fast MDX Assay. All statistical analyses were performed in RStudio (version 2022.07.1), using R version 4.2.1 [5].
3. Results
3.1. Study population and samples
Nasopharyngeal swabs in Aptima® Multitest Swab (n = 112), eSwab® (n = 112) and Transwab® (n = 46) were collected from 270 patients presented at primary health care facilities (n = 125) or nursing homes (n = 145) with a median age of 81.0 (IQR: 66.6–88.4) and 78.9 (IQR: 58.1–87.0) years, respectively. For the comparison, 215 positive and 74 negative SARS-CoV-2 nasopharyngeal swabs were included. Among the 74 SARS-CoV-2 negative samples, 29 nasopharyngeal swabs were positive for other respiratory viruses (17 Influenza type A virus, 1 Influenza type B virus and 11 RSV).
3.2. PCR efficiency, inter- and intra-assay variation
The Allplex™ SARS-CoV-2 fast PCR Assay showed efficiencies of 76% (95% CI: 59–104), 79% (95% CI: 68–94) and 75% (95% CI: 68–83) for the E-, RdRP- and N-gene, respectively. The efficiencies were 71% (95% CI: 67–76) and 66% (95% CI: 63–70) for the S/N- and RdRp-gene in the Allplex™ RV Master Assay, respectively. The R2 for each target gene in these 2 Allplex™ assays approached 1. The inter- and intra-assay variations in each of the target genes in all Allplex™ assays were very low (Supplementary Table 2).
3.3. Diagnostic performances
The sensitivities and specificities for detecting SARS-CoV-2 of all assays ranged between 93.1% and 100% with extremely good agreements (κ ≥ 95.3%) (Table 1 ). For the detection of Influenza type A virus, the sensitivity was slightly lower (82.4%, 95% CI: 56.6–96.2) in the Allplex™ RV Master Assay compared to the Aptima™ SARS-CoV-2/Flu Assay, while the sensitivity for detecting RSV was 81.8% (95% CI; 48.2–97.7) in the Allplex™ RV Master Assay.
Table 1.
Analytical performance for the detection of SARS-CoV-2, Influenza type A/B virus or RSV.
| Total (n = 289) | Sensitivity (%) | Specificity (%) | Accuracy | Kappa | |
|---|---|---|---|---|---|
| SARS-CoV-2 (n = 215) | Allplex™ SARS-CoV-2 fast PCR assay | 100 (98.3–100) | 98.6 (92.5–100) | 99.7 (98.1–100) | 99.1 (97.2–100) |
| Allplex™ RV master assay | 100 (98.3–100) | 98.6 (92.5–100) | 99.7 (98.1–100) | 99.1 (97.2–100) | |
| Allplex™ SARS-CoV-2 fast MDX assay | 99.1 (96.7–99.9) | 100 (95.0–100) | 99.3 (97.5–99.9) | 98.2 (95.6–100) | |
| Aptima™ SARS-CoV-2/flu assay | 100 (98.3–100) | 93.1 (84.5–97.7) | 98.3 (96.0–99.4) | 95.3 (91.1–99.4) | |
| Influenza type A virus (n = 17) | Allplex™ RV master assay | 82.4 (56.6–96.2) | 99.6 (98.0–100) | 98.6 (96.5–99.6) | 86.8 (73.9–99.6) |
| Aptima™ SARS-CoV-2/flu assay | 100 (80.5–100) | 98.9 (96.8–99.8) | 99.0 (97.0–99.8) | 91.3 (81.6–100) | |
| Influenza type B virus (n = 1) | Allplex™ RV master assay | - | 100 (98.7–100) | - | - |
| Aptima™ SARS-CoV-2/flu assay | - | 100 (98.7–100) | - | - | |
| RSV (n = 11) | Allplex™ RV master assay | 81.8 (48.2–97.7) | 100 (98.7–100) | 99.3 (97.5–99.9) | 89.6 (75.3–100) |
For determining sensitivity and specificity, positive and negative reference samples were used as defined by being positive and negative in both the Allplex™ SARS-CoV-2/FluA/B/RSV Assay and Aptima™ SARS-CoV-2 Assay, respectively.
3.4. Comparison of Ct-values across the Seegene assays
Due to differences in detecting target genes across Allplex™ assays, mean Ct-value across target genes in each assay were calculated and Ct-value of the RdRp-gene was used for comparison. The average ΔCt in both, mean Ct and RdRp-gene Ct, were positive for the Allplex™ SARS-CoV-2 fast PCR Assay and the Allplex™ SARS-CoV-2 fast MDX Assay, while the average ΔCts were negative for the Allplex™ RV Master Assay (Table 2 and Supplementary Fig. 1).
Table 2.
Comparison of Ct-value from Allplex™ SARS-CoV-2/FluA/B/RSV Assay with other Allplex™ assays and the relationship between Ct-value from Allplex™ assay and TTime.
| Assays | Allplex™ SARS-CoV-2/FluA/B/RSV Assay |
Aptima™ SARS-CoV-2/Flu Assay |
||||||
|---|---|---|---|---|---|---|---|---|
| Ct (mean) |
Ct (RdRp-gene) |
TTime |
||||||
| ΔCt (95%CI) | function | R2 | ΔCt (95%CI) | function | R2 | function | R2 | |
| Allplex™ SARS-CoV-2 fast PCR assay | 4.04 (3.8-4.2) | y = 0.927x – 2.29 | 0.93 | 4.16 (3.9-4.4) | y = 0.881x – 1.25 | 0.92 | y = 1.64x + 1.75 | 0.91 |
| Allplex™ RV master assay | -2.56 (-2.7–2.4) | y = 0.972x + 3.23 | 0.95 | -2.22 (-2.4–2.0) | y = 0.906x + 4.51 | 0.94 | y = 1.71x + 7.52 | 0.94 |
| Allplex™ SARS-CoV-2 fast MDX assay | 6.28 (6.0-6.6) | y = 0.699x + 0.81 | 0.85 | 8.31 (7.9-8.7) | y = 0.579x + 1.95 | 0.79 | y = 1.24x + 3.84 | 0.85 |
| Allplex™ SARS-CoV-2/FluA/B/RSV assay | - | - | - | - | - | - | y = 1.76x + 4.49 | 0.97 |
Ct = cycle threshold; ΔCt = delta Ct; 95% CI = 95% confidence interval;R2 = R-squared; x = Ct value obtained from Allplex™ SARS-CoV-2/FluA/B/RSV or TTime obtained from Aptima™ SARS-CoV-2/Flu Assay.
The comparison of mean Ct-value and RdRp-gene Ct-value between the Allplex™ SARS-CoV-2/FluA/B/RSV assay with other the Allplex™ assays is visualized in supplementary figure 1. Supplementary Fig. 2 depicted the relationship between mean Ct-value of all Allplex™ assays and TTime.
3.5. Relationship between Ct-value and TTime
Linear relationship between mean Ct-values of the Allplex™ Assay and obtained TTime from the Aptima™ SARS-CoV-2/Flu Assay with high R2 were observed for the Allplex™ SARS-CoV-2 fast PCR Assay, Allplex™ RV Master Assay and Allplex™ SARS-CoV-2 fast MDX Assay (Table 2 and Supplementary Fig. 2).
4. Discussion
In this study, we describe the performance characteristics of 4 commercially available NAATs for the detection of SARS-CoV-2, Influenza type A/B virus and RSV. All assays showed exellent sensitivities, specificities, accuracy, agreement, intra- and inter-assay precision and linearity of results. In addition, a positive average ΔCt in the Allplex™ SARS-CoV-2 fast PCR Assay and Allplex™ SARS-CoV-2 fast MDX Assay was observed, suggesting an overall higher analytical sensitivity. In contrast, a negative average ΔCt (-2.2) was observed in the Allplex™ RV Master Assay.
Shortening the conventional NAAT workflow by omiting the nucleic acid extraction step or using LAMP are appealing alternatives for reducing TAT. This allows for increased productivity and testing capacity within current laboratory facilities. We showed that both, a nucleid acid extraction free protocol (Allplex™ SARS-CoV-2 fast PCR Assay) and LAMP (Allplex™ SARS-CoV-2 fast MDX Assay) can successfully replace conventional NAAT while maintaining equivalent or in case of analytical sensitivity, even better performance characteristics.
The main differences between the Allplex™ and Aptima™ assays are detection methods (i.e., RT-PCR with or without RNA extraction, LAMP, RT-TMA) and loading methods (i.e., batch-wise, random-access). On one hand the Allplex™ assays have lower PCR running time and thereby improve throughput and testing capacity. On the other hand however, random-access and a higher loading capacity for the Panther system required less handling time. Lasty, requirements for improvement of total TAT may vary, depending on for example, geographical location, healthcare settings or extra-laboratory conditions.
Every positive result from the Aptima™ SARS-CoV-2/Flu Assay was accompanied with a TTime. Although experience with TTime is currently limited, it is proposed as the time when a certain cut-off based on the total relative light unit (RLU) was exceeded [6]. In contrast to RLU, Ct-values (RT-PCR based assays) have been related to viral load and are broadly used as indicators for disease stage or severity [7], [8], [9], [10]. To our best knowledge, this is the first study providing a linear relationship between TTime and Ct-values, and thus TTime may be used as a surrogate for Ct-value. However, the regression equations were different for each of the Allplex™ assays. Differences are most likely due to primer- and probe design, inhibitors or RNA detection methods [11].
Our study has several limitations that should be taken into account. First, clinical data (e.g., signs and symptoms) were not available and we could not distinguish between SARS-CoV-2 infected patients and asymptomatic carriers. Consequently, we solely provide performance characteristics by comparing with our defined reference result. Second, due to transmission prevention measures during the COVID-19 pandemic, the occurances of other respiratory pathogens were low [12]. Therefore we were not able to adequately assess the performance characteristics for Influenza type B virus detection and not at all for Adenovirus, human rhinovirus, parainfluenzavirus and metapneumovirus. Third, we did not attempt to further investigate discordant results, as the Ct-values were high in the majority of these samples. Additonal investigation such as sequencing analysis or identification of SARS-CoV-2 variants may also explain these discordant results.
In conclusion, when taken all findings into consideration, all commercially available NAAT included in this study can be used for reliable identification of SARS-CoV-2, Influenza type A virus and RSV in a routine laboratory setting.
Declaration of Competing Interest
The authors declare no competing interests.
Acknowledgments
Acknowledgments
The authors would like to thank staffmembers of the department of medical microbiology for selecting, storage and preparation of samples.
Funding
The authors received no specific funding for this work.
Authors contributions
Conceptualization and study design: CH, NvL, KM, KT; Data curation: CH, ZP, MvG, AvH, NP, JB, MvE; Data analysis: CH, KT; Interpretation of data: all authors; Drafting the original manuscript: KT; Review and editing manuscript: all authors. All authors approved this manuscript version to be submitted.
Footnotes
Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.diagmicrobio.2023.115970.
Appendix. Supplementary materials
References
- 1.Walker PGT, Whittaker C, Watson OJ, Baguelin M, Winskill P, Hamlet A, et al. The impact of COVID-19 and strategies for mitigation and suppression in low- and middle-income countries. Science. 2020;369:413–422. doi: 10.1126/science.abc0035. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.WHO Coronavirus (COVID-19) dashboard. Available from:https://covid19.who.int/. (4th December 2022).
- 3.FindDx test directory. Available from:https://www.finddx.org/covid-19/test-directory/. (4th December 2022).
- 4.Bentley E, Mee ET, Routley S, Mate R, Fritzsche M, Hurley M, et al. WHO Expert Committee on Biological Stadardization. 2020. Collaborative study for the establishment of a WHO international standard for SARS-CoV-2 RNA. 4th December 2022. [Google Scholar]
- 5.R Core Team (2022). R: A language and enviroment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Available from: http://www.R-project.org/. ( 5th June 2023).
- 6.Aptima® SARS-CoV-2/Flu Assay (Panther® System). https://www.fda.gov/media/144590/download. (4th December 2022).
- 7.Wang Y, Zhang L, Sang L, Ye F, Ruan S, Zhong B, et al. Kinetics of viral load and antibody response in relation to COVID-19 severity. J Clin Invest. 2020;130:5235–5244. doi: 10.1172/JCI138759. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Wolfel R, Corman VM, Guggemos W, Seilmaier M, Zange S, Muller MA, et al. Virological assessment of hospitalized patients with COVID-2019. Nature. 2020;581:465–469. doi: 10.1038/s41586-020-2196-x. [DOI] [PubMed] [Google Scholar]
- 9.Zhao J, Yuan Q, Wang H, Liu W, Liao X, Su Y, et al. Antibody responses to SARS-CoV-2 in patients with novel Coronavirus disease 2019. Clin Infect Dis. 2020;71:2027–2034. doi: 10.1093/cid/ciaa344. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Zou L, Ruan F, Huang M, Liang L, Huang H, Hong Z, et al. SARS-CoV-2 viral load in upper respiratory specimens of infected patients. N Engl J Med. 2020;382:1177–1179. doi: 10.1056/NEJMc2001737. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Chen X, Xia S. Sensitive methods for detection of SARS-CoV-2 RNA. Methods Microbiol. 2022;50:1–26. doi: 10.1016/bs.mim.2021.06.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Reukers DFM, van Asten L, Brandsema PS, Dijkstra F, Hendriksen JMT, Hooiveld M, et al. Annual report Surveillance of COVID-19, influenza and other respiratory infections in the Netherlands: winter 2021/2022. Available from: https://www.rivm.nl/publicaties/annual-report-surveillance-of-covid-19-influenza-and-other-respiratory-infections-in-0. (4th December 2022).
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.