Abstract
Aims
The aim of this study was to evaluate the performance of the automated Elecsys® SARS-CoV-2 antigen assay compared to RT-PCR taken as the gold standard for SARS-CoV-2 detection.
Methods
225 nasopharyngeal swabs were randomly collected among which 123 were tested positive and 102 negatives for SARS-CoV-2 based on RT-PCR. Antigen dosing were performed on a Cobas 8000 e801 analyzer.
Results
The antigen test diagnosed SARS-CoV-2 infection status with an overall sensitivity of 65,85% (95% CI 56,76–74,16%), a specificity of 100% (95% CI 96,49–100%) with a Cut-off value ≥ 1. When the cut-off value for the antigen assay was set to > 0,673 COI, the accuracy reached its highest level with a sensitivity of 74,8% (95% CI 66,2 – 82,2%) and a specificity of 97,1% (95% CI 91,6 – 99,4%). Imprecision was estimated in accordance with manufacturer's claims.
Conclusions
We obtained an overall sensitivity of 65,85% (95% CI 56,76–74,16%) and a specificity of 100% (95% CI 96,49–100%), slightly higher than the results reported by the manufacturer. Yet, it remains relatively low comparatively to what is generally acceptable for these antigenic assays (a relative sensitivity of 80%). We also noticed that the accuracy could reach its highest level if the cut-off is set above 0,673 which is lower than established by the manufacturer. Thus, our results suggest that the Elecsys® SARS-CoV-2 Antigen assays, should be improved prior to be used in a SARS-Cov-2 screening strategy. However, if one antigenic assay could demonstrate acceptable performance, it might be centralized in clinical laboratories, keeping the RT-PCR in a second phase for confirmation.
Keywords: Automated antigen immunoassay, Sensitivity, SARS-CoV-2 testing strategy, COVID-19
1. Introduction
A novel coronavirus, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), emerged in December 2019 in Wuhan, China [1]. Since the beginning of the Coronavirus Infectious Disease-19 (COVID-19) pandemic, laboratories have been using nucleic acid amplification tests (NAATs), such as real-time reverse transcription polymerase chain reaction (RT-PCR), to detect SARS-CoV-2 [2]. Molecular techniques are considered as the "gold standard" for the diagnosis of SARS-CoV-2 infection due to their great sensitivity/specificity. However, the use of these techniques faced certain constraints such as the scarcity of resources (consumables, reagents) and relative extended turnaround time [3].
In addition to molecular tests, other strategies have been developed especially serological (detection of specific SARS-CoV-2 antibodies) and antigenic assays, mostly rapid test.
Some countries have included in their screening and testing strategy for SARS-CoV-2 algorithms including the use of antigenic rapid detection tests (Ag-RDTs) for symptomatic and asymptomatic patients [4,5].
Rapid antigenic tests for SARS-CoV-2 are inexpensive and can return results within 18 min [5,6] and most of them have received either Food and Drug Administration (FDA) Emergency Use Authorization (USA) or CE-IVD mark for use in asymptomatic and symptomatic persons within the first 5–12 days after symptom onset [7,8].
Though Ag-RDTs are substantially less sensitive than NAAT, they offer the possibility of rapid, inexpensive, and early detection of the most infectious COVID-19 cases in appropriate settings. The World Health Organization (WHO) recommends a minimum of 80% sensitivity and 97% specificity for Ag-RDTs that can be used for patients with symptoms consistent with COVID-19 [9].
Ag-RDTs have been highly evaluated [10 –12] and widely used for population screening and even as POCT. New automated versions of antigen testing are now available on the market, but data on their performance is still scarce.
Currently, there is a real need for alternative assays such as automated antigen detection tests, which in contrast to automated antibody tests, can detect the presence of the virus itself in respiratory samples and could be an alternative to the RT-PCR in the management of next phases of the Covid-19 pandemic. Very few data are available supporting the use of automated antigenic tests in routine laboratory as an alternative for first-stage screening of positive cases, then using the RT-PCR in a second stage as a confirmatory test, probably due to the relative lower sensibility of antigenic tests [3,10,11].
The aim of this study was to evaluate the performance of the automated Elecsys® SARS-CoV-2 antigen assay compared to RT-PCR, taken as the gold standard, for SARS-CoV-2 detection.
2. Material and methods
2.1. Clinical samples
A convenience sample of two hundred and twenty-five (225) nasopharyngeal specimens from outpatients (known or suspected exposure) with a request for SARS-CoV-2 RT-PCR analysis were randomly collected using flocked or polyester-tipped swabs, placed in 3 mL of Viral Transport Media (VTM) (Biotech Corporation Co., Ltd.) with a composition of NaCl at 0.8%. This container tube is FDA approved for use as a viral transport medium and has been validated in our laboratory for daily use for Sars-CoV-2 RT-PCR. The guidelines for the management of potentially contagious samples in Laboratory have been strictly respected, following the indications of WHO and CDC [2, 9]. Our study respected and followed the principles of the Declaration of Helsinki. The VTMs were stored at 4 °C until nucleic acid extraction. Total nucleic acids were extracted within 5 h after swab collecting. All samples were collected during April 2021. Data related to clinical symptoms were unavailable.
2.2. Analytical performance
Precision was assessed by considering repeatability and between-day variability using quality control (QC) materials provided by the manufacturer by measurement of quadruplicates during 5 successive days.
2.3. RT-PCR test
RT-PCR testing was performed using the Allplex ™ 2019-nCoV assay (Seegene Technologies Inc, South Korea) on a CFX96 thermocycler (BioRad, Hercules, CA, US), following an automated RNA extraction performed with the Genomic STARlet liquid handler (Hamilton, Nv, US).
The RT-PCR targeted 4 specific genes: E, RdRP, S, N and expressed results in cycle threshold (Ct) values. All samples with a valid result for Target-N and/or Target-E were included in our study.
The remaining VTM from each nasopharyngeal swab was immediately frozen at −20 °C after the RT-PCR analyses, in accordance with the manufacturer's recommendations for the antigenic testing (less than 14 days and only once frozen) [6].
2.4. SARS-CoV-2 automated antigen test
Automated qualitative detection of the SARS-CoV-2 nucleocapsid antigen was performed using the Elecsys® SARS-CoV-2 Antigen assay (Roche Diagnostics, Mannheim, Germany) on Cobas® 8000 e801 analyzer (Roche Diagnostics, Mannheim, Germany)
The principle of the method is an ElectroChemiLuminescence ImmunoAssay (ECLIA), a double-antibody sandwich assay. The analytical testing time is 18 min [6]. Before analyzing samples, a previous viral inactivation step was mandatory using the SARS-CoV-2 Extraction solution C Cobas®. This solution is intended for reduction for viral load in swabs transported in a virus transport solution, including sterile saline. In our study, we used 1 mL per sample of thawed viral transport media from each nasopharyngeal swab and 100 µL of the Solution then we vortexed each sample for 5 s. All samples were analyzed within 15 min after the inactivation step.
The results of SARS-CoV-2 Antigen are expressed in cut-off index (COI) with a qualitative interpretation based on the following: COI < 1.0 Non-reactive for SARS‑CoV‑2 antigen and COI ≥ 1.0 Reactive for SARS‑CoV‑2 antigen.
2.5. Data management and statistical analyzes
Prior to proceeding analysis with collected samples, we first evaluated the precision by an analysis of variance (ANOVA) to estimate repeatability and intermediate precision.
Based on the Ct values obtained from the RT-PCR assay, we assigned samples into different subcategories. We have included 51 patients with undetectable (Negative) RT-PCR results with a Ct ≥ 40, to assess possible interferences in the immunoassay method. This cohort is described in Fig. 1 .
Fig. 1.
Samples assignation in subcategories based on RT-PCR Ct values.
As the RT-PCR was considered as the gold standard, sensitivity was defined as the proportion of SARS-CoV-2 positive samples by the antigen test initially categorized as positive by RT-PCR (Ct ≤ 35). Specificity was defined as the proportion of samples identified as negative by the antigen test initially categorized as negative by RT-PCR. Positive and negative predictive values (PPV, NPV) were also calculated for the overall sample and relative sensitivity were evaluated for sample subcategories.
To establish the optimal cut-off value, a receiver-operating-characteristic (ROC) curve was plotted. The non-parametric empirical method was used to estimate the area under the curve (AUC). The Elecsys® SARS-CoV-2 antigen cut-off value was determined by identifying an inflection point on the validation ROC curve, corresponding to a combination of high specificity and sensitivity. Categorical variables were expressed as frequencies with 95% confidence intervals (95% CI).
Statistical analyses were performed using MedCalc Statistical Software version 19.3 (MedCalc Software Ltd, Ostend, Belgium) and Excel 2016 (Microsoft, Redmont; USA).
3. Results
Intermediate precision for the Elecsys SARS-CoV-2 antigen assay across 5 days revealed coefficient of variations of 2.2%, remaining between the manufacturers claimed values of 1.7 – 5.7% for Cobas e 801 modules. For repeatability, we obtained the same coefficient of variation (1.1%) than one reported by the manufacturer [6].
3.1. Comparison between RT-PCR and the antigen automated test
We performed the automated antigen test on 225 nasopharyngeal swabs, including 123 positives and 102 negatives samples, as determined by RT-PCR.
The median antigen level of the PCR-positive samples was 4.56 COI (range 0.40–22,762 COI) and that of the PCR-negative samples was 0.45 COI (range 0.33–0.80). The mean antigen level of the PCR-positive samples was significantly higher than of the PCR-negative samples (p < 0.01) (Fig. 2 ).
Fig. 2.
Distribution of the values obtained in Ag SARS-COV-2 (COI) in relation to the result of the RT-PCR.
The concordance between RT-PCR and Ag-RDT results is shown in Table 1 .
Table 1.
Results obtained from the RT-PCR and the Ag tests.
| RT-PCR | ||
|---|---|---|
| Ag interpretation | Negative | Positive |
| Negative | 102 | 42 |
| Positive | 0 | 81 |
| N = 102 | N = 123 | |
When the RT-PCR results were used as a reference, the antigen test diagnosed SARS-CoV-2 infection status with a sensitivity of 65.85% (95% CI 56.76–74.16%) and a specificity of 100% (95% CI 96.49–100%), PPV = 100%, NPV = 70.83 (95% CI 75.61–86.20%) and an accuracy = 81.33% (95% CI 75.62–86.20%).
We also evaluated the relative sensitivity obtained for positive cases. Subcategorized according to Ct values (Table 2 ). With a Ct ≤ 25, we obtained a relative sensitivity of 100%.
Table 2.
Relative sensitivity for subcategories.
| Positive RT-PCR | Relative sensitivity (%) |
|---|---|
| Ct ≤ 20; N = 46 | 100 (95% CI 92.12–100) |
| 20 < Ct < 25; N = 18 | 100 (95%CI 92.13–100) |
| 25 < Ct>< 30; N = 30 | 46.67 (95% CI 28.34–65.67) |
| 30 < Ct < 35; N = 29 | 3.44 (95% CI 0.08–17.76) |
A ROC curve was constructed to determine the best cut-off index according to the World Health Organization (WHO) recommendation for Ag-RDTs performance (minimum of 80% sensitivity and 97% specificity). When the cut-off for the antigen level was set above 0.673 COI (95% CI 91.6 – 99.4), the accuracy reached its highest level. ROC analyses yielded an AUC value of 0.91 ± 0.0194 (95% CI 0.8644–0.9438) suggesting that the antigen test accurately detected SARS-CoV-2 (Fig. 3 ) with a sensitivity of 74.8% (95% CI 66.2 – 82.2%) and a specificity of 97.1% (95% CI 91.6 – 99.4%).
Fig. 3.
Receiver operating characteristic curve for the Ag SARS-Cov-2 test.
4. Discussion
Most testing strategies implemented for Covid-19 have included the use of rapid antigenic tests, based on recommendations from international organizations, such as the WHO [9]. However, there are very few discussions about automated tests, which if they demonstrate good performance, could be a viable alternative to rapid tests. In the present study, we evaluated the performance of an automated antigenic test (Elecsys® SARS-CoV-2 Antigen) based on ECLIA and compared results with those of the RT-PCR.
In this investigation we obtained an overall sensitivity of 65.85% (95% CI 56.76–74.16%) and a specificity of 100% (95% CI 96.49–100%), slightly higher than those reported by the manufacturer, a sensitivity of 60.5 (95% CI 55.5–65.4%) and a specificity of 99.9% (95% CI 99.6–100%) (6). Otherwise, Pray et al. obtained a poor sensitivity of 41.2% and a specificity of 98.4% [13] in their evaluation of a SARS antigen fluorescent Immunoassay (Quidel corp., CA, US) and Hirotsu et al., who evaluated another ECLIA based test (Lumipulse, Fujirebio, Japan) obtained a sensitivity of 55.2% and a specificity of 99.6% for this automated antigenic test, lower than our results [14]. Yin et al. evaluated the same automated test than Hirotsu et al. (in comparison with four other Ag-RDTs) and obtained an overall sensitivity of 86.7% for the automated assay but specifically for asymptomatic inpatients [5]. Lefever et al. evaluated a quantitative Ag test (DiaSorin, Saluggia, Italy) and obtained an overall sensitivity and specificity of, respectively 67.7% and 100%. But when considering samples with high viral load, this sensitivity rises to 87.4%−100% [15]. These findings are comparable with ours (sensitivity = 100%) for all Ct values below 25, reflecting a high viral load.
Performance discrepancies between these studies could nevertheless be explained by the difference of methodologies applied, the variability of cut-off values set for the different antigen tests but also by the reference RT-PCR used [16].
Considering results obtained in the studies listed above, we observe that the main disadvantages of antigenic automated tests could be their relative lower sensitivity as also reported for Ag-RDTs.
We also have evaluated the cut-off index fixed by manufacturer for the qualitative interpretation of the presence of disease (COI ≥ 1). According to our results, this cut-off value could be reevaluated, to increase the test performance. In fact, if lowered to a value of 0.673, the accuracy reached its highest level, increasing sensitivity to 74.8% (95% CI 66.2 – 82.2%), specificity to 97.1% (95% CI 91.6 –99.4%), the AUC to 0.91(95% CI 0.86–0.94%), PPV to 100%, NPV to 70.83 (95% CI 75.61–86.20%) and accuracy to 81.33% (95% CI 75.62–86.20%).
Taking all this into account, we can argue that the automated antigenic test we evaluated have an excellent specificity but the main challenge steads in the sensitivity, which is directly correlated to the Ct value thus the viral load.
As rapid antigenic tests are already in use in many parts of the world for SARS-Cov-2 screening, despite their relatively low sensitivity, we suggest that if one automated antigenic test reports high levels of sensitivity and specificity, it could be centralized in clinical laboratories (under the supervision of laboratory experts) and could be used in diagnostic algorithms for population screening and even diagnosis. But these algorithms should take in account some criteria as symptoms, risk contact, intensity of virus circulation, etc.
Though NAAT offers higher sensitivity than Ag testing, Ag delivers results more rapidly and identifies the most highly infectious individuals, equivalent to NAAT in averting transmission, with a shorter turnaround time less than 30 min after arrival at the laboratory [17].
Furthermore, it would be interesting to define an exact cut-off point with further studies and define an interval of COI between which the RT-PCR could be used as confirmation diagnostic test as is already in place for other pathologies [18] but also contemplated by some national organizations [19].
5. Conclusion
The Elecsys® SARS-CoV-2 Antigen have an overall similar performance than other Ag assays (sensitivity less than 70% and specificity higher than 95%). In our study, we obtained a low sensitivity of 65.85% (95% CI 56.76–74.16%) and a specificity of 100% (95% CI 96.49–100%), slightly higher than results reported by the manufacturer. The accuracy reached its highest level when the cut-off was set above 0,673, which is lower than established by the manufacturer.
Our results suggest that the Elecsys® SARS-CoV-2 Antigen assay, as other current available automated antigenic assays, cannot be efficiently used in a SARS-Cov-2 screening strategy without considering other factors such as related to the current epidemiology of the pathology (transmission dynamics) and the clinical status of patients. However, if one antigenic assay could demonstrate acceptable sensitivity (>80%) and specificity (>97%), it might be centralized in clinical laboratories, keeping the RT-PCR in a second phase for confirmation.
Declaration of Competing Interest
All the authors declare no conflicting interests.
Acknowledgments
We wish to thank the staff of Cerba HealthCare Belgium for its daily technical assistance. We also thank Roche Diagnostics Belgium for providing the reagents free of charge.
References
- 1.Zhu N., Zhang D., Wang W., Li X., Yang B., Song J., Zhao X., Huang B., Shi W., Lu R., Niu P., Zhan F., Ma X., Wang D., Xu W., Wu G., Gao G.F., Tan W., China Novel Coronavirus Investigating and Research Team A novel coronavirus from patients with pneumonia in China, 2019. N. Engl. J. Med. 2020;382(8):727–733. doi: 10.1056/NEJMoa2001017. Feb 20Epub 2020 Jan 24. PMID: 31978945; PMCID: PMC7092803. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Center for Diseases Control. Coronavirus disease . US Department of Health and Human Services, CDC; 2020; Atlanta, GA: 2019. (COVID-19): Interim Guidance for Antigen Testing for SARS-CoV-2.https://www.cdc.gov/coronavirus/2019-ncov/lab/resources/antigen-tests-guidelines.html Accessed on 23 July,2021. [Google Scholar]
- 3.Peeling R.W., Olliaro P.L., Boeras D.I., Fongwen N. Scaling up COVID-19 rapid antigen tests promises and challenges. Lancet Infect. Dis. 2021 doi: 10.1016/S1473-3099(21)00048-7. Feb 23S1473-3099(21)00048-7.Epub ahead of print. PMID: 33636148; PMCID: PMC7906660. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Boum Y., Fai K.N., Nicolay B., Mboringong A.B., Bebell L.M., Ndifon M., Abbah A., Essaka R., Eteki L., Luquero F., Langendorf C., Mbarga N.F., Essomba R.G., Buri B.D., Corine T.M., Kameni B.T., Mandeng N., Fanne M., Bisseck A.Z., Ndongmo C.B., Eyangoh S., Hamadou A., Ouamba J.P., Koku M.T., Njouom R., Claire O.M., Esso L., Epée E., Mballa G. Performance and operational feasibility of antigen and antibody rapid diagnostic tests for COVID-19 in symptomatic and asymptomatic patients in Cameroon: a clinical, prospective, diagnostic accuracy study. Lancet Infect. Dis. 2021 doi: 10.1016/S1473-3099(21)00132-8. Mar 25S1473-3099(21)00132-8Epub ahead of print. Erratum in: Lancet Infect Dis. 2021 Apr 16; PMID: 33773618; PMCID: PMC7993929. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Yin N., Debuysschere C., Decroly M., et al. SARS-CoV-2 Diagnostic Tests: Algorithm and Field Evaluation From the Near Patient Testing to the Automated Diagnostic Platform. Frontiers in Medicine. 2021;8:650581. doi: 10.3389/fmed.2021.650581. PMID: 33889587; PMCID: PMC8055843. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Elecsys® SARS-CoV-2 Antigen. Method sheet v1 (Dec 2020). Available from: https://diagnostics.roche.com/global/en/products/params/elecsys-sars-cov-2-antigen-test.html. Accessed on July 19, 2021.
- 7.Food and Drug Administration In vitro diagnostics EUAs. Silver spring, MD: US department of health and human services. Food Drug Adm. 2020 https://www.fda.gov/medical-devices/coronavirus-disease-2019-covid-19-emergency-use-authorizations-medical-devices/in-vitro-diagnostics-euas. Accessed 23 July 2021. [Google Scholar]
- 8.FIND, SARS-CoV-2 diagnostic pipeline. https://www.finddx.org/covid-19/pipeline/?avance=all&type=Rapid±diagnostic±tests&test_target=Antigen&status=CE-IVD§ion=show-all&action=default. Accessed July 23, 2021.
- 9.World Health Organization (WHO). Antigen-detection in the diagnosis of SARS-CoV-2 infection using rapid immunoassays: interim guidance. WHO/2019-nCoV/Antigen_Detection/ 2020.13.
- 10.Scohy A., Anantharajah A., Bodéus M., Kabamba-Mukadi B., Verroken A., Rodriguez-Villalobos H. Low performance of rapid antigen detection test as frontline testing for COVID-19 diagnosis. J. Clin. Virol. 2020;129 doi: 10.1016/j.jcv.2020.104455. Aug;Epub 2020 May 21. PMID: 32485618; PMCID: PMC7240272. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Lanser L., Bellmann-Weiler R., Öttl K.W., Huber L., Griesmacher A., Theurl I., Weiss G. Evaluating the clinical utility and sensitivity of SARS-CoV-2 antigen testing in relation to RT-PCR Ct values. Infection. 2020 Nov 13:1–3. doi: 10.1007/s15010-020-01542-0. Epub ahead of print. PMID: 33185807; PMCID: PMC7662025. [DOI] [PMC free article] [PubMed]
- 12.Porte L., Legarraga P., Vollrath V., Aguilera X., Munita J.M., Araos R., et al. Evaluation of a novel antigen-based rapid detection test for the diagnosis of SARS-CoV-2 in respiratory samples. Int. J. Infect. Dis. 2020:328–333. doi: 10.1016/j.ijid.2020.05.098. Oct; 99Epub 2020 Jun 1. PMID: 32497809; PMCID: PMC7263236. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Pray I.W., Ford L., Cole D., Lee C., Bigouette J.P., et al. CDC COVID-19 surge laboratory group. Performance of an antigen-based test for asymptomatic and symptomatic SARS-CoV-2 testing at two university campuses - Wisconsin, September-October 2020. MMWR Morb. Mortal. Wkly. Rep. 2021;69(5152):1642–1647. doi: 10.15585/mmwr.mm695152a3. Jan 1PMID: 33382679. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Hirotsu Y., Maejima M., Shibusawa M., Nagakubo Y., Hosaka K., Amemiya K., et al. Comparison of automated SARS-CoV-2 antigen test for COVID-19 infection with quantitative RT-PCR using 313 nasopharyngeal swabs, including from seven serially followed patients. Int J Infect Dis. 2020;99:397–402. doi: 10.1016/j.ijid.2020.08.029. Oct: PMID: 32800855; PMCID: PMC7422837. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Lefever S., Indevuyst C., Cuypers L., Dewaele K., Yin N., Cotton F., et al. Comparison of the quantitative diasorin liaison antigen test to RT-PCR for the diagnosis of COVID-19 in symptomatic and asymptomatic outpatients. J. Clin. Microbiol. 2021 doi: 10.1128/JCM.00374-21. Apr 13: JCM.00374-21Epub ahead of print. PMID: 33849953. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Lippi G., Simundic A.M., Plebani M. Potential preanalytical and analytical vulnerabilities in the laboratory diagnosis of coronavirus disease 2019 (COVID-19) Clin. Chem. Lab. Med. 2020;58(7):1070–1076. doi: 10.1515/cclm-2020-0285. Jun 25PMID: 32172228. [DOI] [PubMed] [Google Scholar]
- 17.Kendall E.A., Arinaminpathy N., Sacks J.A., Manabe Y.C., Dittrich S., Schumacher S.G. et al., medRxiv 2020.12.16.20248357; 10.1101/2020.12.16.20248357.
- 18.WHO Information for the molecular detection of influenza viruses. http://www.who.int/influenza/gisrs_laboratory/collaborating_centres/list/en/index.html.January 2020. Accessed on July 8, 2021.
- 19.https://www.cdc.gov/flu/professionals/diagnosis/overview-testing-methods.htm Accessed on July 8, 2021.