Abstract
Given the ongoing pandemic, there is a need to identify SARS-CoV-2 and differentiate it from other respiratory viral infections in various critical settings. Since its introduction, rapid antigen testing is spreading worldwide, but diagnostic accuracy is extremely variable and often in disagreement with the manufacturer’s specifications. Our study compared the clinical performances of two microfluidic rapid antigen tests towards a molecular assay, starting from positive samples. A total of 151 swabs collected at the Microbiology and Virology Laboratory of A.O. “SS Antonio e Biagio e C. Arrigo” (Alessandria, Italy) for the diagnosis of SARS-CoV-2 were simultaneously tested to evaluate accuracy, specificity, and agreement with the RT-qPCR results. Both assays showed an overall agreement of 100% for negative specimens, while positive accuracy comprised between 45.10% and 54.90%. According to the manufacturer’s instructions, the greatest correlation between the antigenic and molecular assays was observed for the subset with high viral load (18/19, 94.74%), while it dramatically decreased for other subsets. Moreover, the ability to differentiate between SARS-CoV-2 and Flu provides an added value and could be addressed in an epidemic context. However, an in-house validation should be performed due to differences observed in performance declared by manufacturers and those actually obtained.
Keywords: SARS-CoV-2, COVID-19, third-generation assay, microfluidic
1. Introduction
Coronavirus disease (COVID-19) is an infectious disease caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), a single-stranded positive-sense RNA virus with four main structural proteins: spike, envelope, membrane, and nucleocapsid [1]. It belongs to the human coronavirus family (HCoV), of which six were already known to cause disease. Among these, four are known as human endemic coronaviruses (HCoV-229E, HCoV-NL63, HCoV-OC43, HCoV-HKU1) responsible for acute self-limiting common-cold symptoms, while the other two (SARS-CoV and MERS-CoV) cause outbreaks of severe lower respiratory tract infection [2]. A quick and accurate detection of SARS-CoV-2 infected individuals is important to minimize the spread of COVID-19. However, it immediately represented a substantial challenge for healthcare systems [3]. Qualitative reverse transcription qPCR (RT-qPCR) testing of specimens collected at a level of the upper respiratory tract (nasopharyngeal and oropharyngeal secretion) was the first diagnostic approach developed for reliable detection of viral RNA, and it is still considered the gold standard for COVID-19 diagnosis [4]. Nevertheless, molecular testing is expensive, time consuming and requires adequately skilled staff; moreover, it can take several hours (average 3–4 h) to obtain a result. On the other hand, rapid antigen testing for the detection of SARS-CoV-2 is spreading worldwide, allowing a quick diagnosis in many different community settings; however, diagnostic accuracy is extremely variable, with sensitivity ranging from 0% to 94% [5]. Rapid tests are based on lateral flow immunochromatography using antibodies against SARS-CoV-2 proteins (antigens), present in respiratory tract specimens, and detected mainly the viral nucleoprotein, less frequently the spike proteins [6]. Nonetheless, several immunoassays revealed an overall diagnostic sensitivity that settled at around 70%, lower than the minimum diagnostic sensitivity (≥80%) required by the Task Force on COVID-19 (International Federation of Clinical Chemistry and Laboratory Medicine, IFCC) and World Health Organization (WHO) [7,8]. In addition to manual lateral flow immunoassays (first and second generation) and laboratory-based chemiluminescent tests (fourth generation), the microfluidic assays (third generation), of which the LumiraDx SARS-CoV-2 Ag test represents the prototype for rapidity, handiness, and versatility as a decentralized testing device, could emerge as a reliable diagnostic alternative [9]. Since not only SARS-CoV-2 but also influenza viruses A and B are highly transmissible and share some overlapping signs and symptoms (cough, sore throat, fever, headache, respiratory distress), a quick and accurate differential diagnosis represents a pivotal step in order to prevent further spread of the viral disease and providing a suitable treatment [10]. For these reasons, combo antigen tests have been developed for accurate diagnosis of SARS-CoV-2 and differentiation from other respiratory infections. Although the analytical performance (i.e., accuracy, sensitivity, and specificity) of these newly introduced tests have been determined in manufacturers’ studies, they may diverge in routine clinical practice and in their use among the general population [11]. Hence, considering the biological and technical features, an in situ evaluation should be performed at different hospital settings before introducing a new antigenic assay for SARS-CoV-2 detection.
Thus, due to the abovementioned aspects, the aim of our study was a retrospective analysis focused on clinical performances of the antigenic tests LumiraDx SARS-CoV-2 & Flu A/B (LumiraDx Limited, UK) and LumiraDx SARS-CoV-2 Ag Ultra (LumiraDx Limited, UK) for detection of SARS-CoV-2, starting from clinical samples collected in viral transport medium (VTM) with different viral loads.
2. Materials and Methods
A total of 151 nasopharyngeal swabs were collected and stored at Microbiology and Virology Laboratory of A.O. “SS Antonio e Biagio e C. Arrigo” (Alessandria, Italy) for diagnosis of SARS-CoV-2 infection. All samples were tested according to WHO guidelines [12] and collected in 3 mL of liquid universal transport medium (UTM) (Copan, Italy). Each specimen was assessed for detection of SARS-CoV-2 via RT-qPCR (Xpert® Xpress SARS-CoV-2, Cepheid, Sunnyvale, CA, USA) used as standard of care (SoC).
Furthermore, clinical specimens were assigned to four groups: negative SARS-CoV-2 (100 samples), very high viral load (Ct value 17–25; 19 samples), high viral load (Ct value 25–30; 23 samples), and moderate viral load (Ct value 30–36; 9 samples).
All samples were simultaneously tested with antigenic tests LumiraDx SARS-CoV-2 & Flu A/B and LumiraDx SARS-CoV-2 Ag Ultra according to manufacturer instructions.
The LumiraDx immunoassays herein reported were based on rapid microfluidic immunofluorescence for qualitative detection of SARS-CoV-2 Nucleocapsid proteins (LumiraDx SARS-CoV-2 Ag Ultra) and Influenza Type A/B Nucleocapsid proteins (tests LumiraDx SARS-CoV-2 & Flu A/B) in order to differentiate the etiological agents of viral disease. The assay principle relied on fluorescent latex nanoparticles coated with monoclonal antibodies and magnetic beads targeted to the abovementioned viral antigenic structures.
Briefly, 100µL of UTM was transferred in a vial and collected with a swab until complete retention. Subsequently, the soaked swab was transferred into extraction buffer, loaded onto test strip for both tests, and analysed with LumiraDx instrument according to the manufacturer’s instructions. Within 12 min, the test result (positive or negative) was reported.
For each specimen group, both antigenic methods were compared in terms of accuracy, specificity, and agreement towards the molecular SoC and between them.
3. Results
According to RT-qPCR, on a total of 151 samples included in our analysis, 51 samples (33.8%) tested positive, while 100 (66.2%) were negative. On positive samples, the overall median Ct was 27.5 ± 5.99 (IQR: 19.8–30.0). According to the abovementioned criteria, specimens with detected viral load were divided into three groups as follows: 19 samples at very high viral load (Median Ct: 18.5 ± 2.69, IQR: 17.0–20.5), 23 at high viral load (Median Ct: 28.5 ± 1.69, IQR: 27.3–30.0), and 9 with moderate viral load (Median Ct: 34.5 ± 0.89, IQR: 34.0–34.5) (Table 1).
Table 1.
Descriptive analysis of tested samples.
| Positive Samples | Negative Samples | Total | ||||
|---|---|---|---|---|---|---|
| N. (%) | 51 (33.8%) | 100 (66.2%) | 151 (100%) | |||
| Ct Values of positive samples | ||||||
| Mean | Median | Standard Deviation (SD) |
Interquartile Range (IQR) | Min | Max | |
| Overall | 26.0 | 27.0 | 5.9 | 10.3 | 17.0 | 35.0 |
| Very high | 19.2 | 18.5 | 2.7 | 3.5 | 17.0 | 25.0 |
| High | 28.4 | 28.5 | 1.7 | 2.7 | 25.5 | 30.0 |
| Moderate | 34.1 | 34.5 | 0.9 | 0.5 | 32.0 | 35.0 |
Looking at the diagnostic accuracy of LumiraDx SARS-CoV-2 Ag Ultra towards the SoC, the specificity for negative samples was 100%, while the accuracy was 54.90 (CI: 40.34% to 68.87%). The overall agreement was 0.62 (CI: 0.48 to 0.75). Focusing on samples with very high viral load accuracy and inter-rater agreement were, respectively, 94.74% (CI: 73.97% to 99.87%) and 0.97 (CI: 0.90 to 1), while those with high viral load decreased (accuracy: 43.48, CI: 23.19% to 65.50%; k-value: 0.56, CI: 0.35 to 0.76). No samples with moderate viral load yielded a positive result for the antigenic test (Table 2 and Figure 1).
Table 2.
Diagnostic performances of the two microfluidic antigenic tests.
| Accuracy | Specificity | Cohen’s Kappa Coefficient | |
|---|---|---|---|
| LumiraDx SARS-CoV-2 Ag Ultra vs. RT-qPCR | |||
| Overall | 54.90% | 100.00% | 0.62 |
| Very high | 94.74% | 0.97 | |
| High | 43.48% | 0.56 | |
| Moderate | 0.00% | 0.00 | |
| LumiraDx SARS-CoV-2 & Flu A/B vs. RT-qPCR | |||
| Overall | 45.10% | 100.00% | 0.52 |
| Very high | 94.74% | 0.97 | |
| High | 21.74% | 0.31 | |
| Moderate | 0.00% | 0.00 | |
Figure 1.
Specificity and accuracy of the two microfluidic antigenic tests according to subsets samples.
Instead, focusing on the diagnostic accuracy of LumiraDx SARS-CoV-2 & Flu A/B towards the SoC, the specificity for negative samples was 100%, while the accuracy was 45.10 (CI: 31.13% to 59.66%). The overall inter-rater agreement was 0.52 (CI: 0.38 to 0.66). For samples with very high viral load accuracy and agreement were, respectively, 94.74% (CI: 73.97% to 99.87%) and 0.97 (CI: 0.90 to 1) and more reduced for those with high viral load (accuracy: 21.74, CI: 7.46% to 43.70%; k-value: 0.31, CI: 0.09 to 0.52). Again, no samples with moderate viral load yielded a positive result for the antigenic test compared with RT-qPCR (Table 2 and Figure 1).
The inter-rater agreement of LumiraDx SARS-CoV-2 Ag Ultra and SARS-CoV-2 & Flu A/B resulted in 0.88 (CI: 0.77 to 0.98). A complete agreement was obtained for samples with high viral load (k-value: 1), while for those with high viral load, the inter-rater agreement was 0.64 (CI: 0.36 to 0.93) (Table 3).
Table 3.
Comparison of diagnostic performance between the two antigenic tests.
| SARS-CoV-2 Ultra (N. Pos) | SARS-CoV-2 & Flu A/B (N. Pos) | Cohen’s Kappa Coefficient |
|
|---|---|---|---|
| Overall | 28 | 23 | 0.88 |
| Very high | 18 | 18 | 1.00 |
| High | 10 | 5 | 0.64 |
| Moderate | 0 | 0 | 0.00 |
4. Discussion
In this study, we determined the performance characteristics of the two assays, the LumiraDx SARS-CoV-2 Ag Ultra and LumiraDx SARS-CoV-2 & Flu A/B, for the detection of SARS-CoV-2 virus in respiratory samples and compared the results with RT-qPCR, used as reference.
Since their introduction in autumn 2020, antigenic tests have become a key part of testing strategies, thus suggesting their deployment in order to detect potential infectivity and help control the spread of infection rather than for the purpose of clinical diagnosis [6]. Despite rapid antigen tests reported to have good performances (agreement >90% for positive RT-qPCR results), a wide range of clinical accuracy, often lower than declared by the manufacturers, was observed in clinical samples [13]. A comparison of several commercial lateral flow/chemiluminescent assays showed values ranging from 20.0% to 90.0% [14,15,16,17], nearly always lower than the value reported in the kits IFU. Herein, we observed overall accuracies of 54.90% and 45,10% for LumiraDx SARS-CoV-2 Ag Ultra and LumiraDx SARS-CoV-2 & Flu A/B, respectively. However, focusing on a subset of samples with higher viral load (cycle threshold range: 17–25), this value amounted to 94.74% for both methods, more than what was declared by the manufacturer (LumiraDx SARS-CoV-2 Ag Ultra: 92.7%; LumiraDx SARS-CoV-2 & Flu A/B: 95.5%). In the range of 25–30 cycle threshold, the diagnostic accuracy dramatically decreased, especially for the SARS-CoV-2/Flu A/Flu B assay (21.74%). Both kits appeared not able to detect positive samples with lower viral loads (Cycle threshold > 30). Considering that individuals with fever and respiratory symptoms showed higher nasopharyngeal SARS-CoV-2 viral loads in terms both of digital PCR (Viral load quantification) and qPCR (Ct values) than those without those symptoms [18,19], the antigenic tests abovementioned might provide a useful assay to screen symptomatic patients in critical settings. Moreover, the possibility to use the same instrument to read results obtained by both kits may provide the ability to rapidly switch between them according to the pandemic context (i.e., presence or absence of contemporary Flu peak and increase in SARS-CoV-2 infection) [20].
Interestingly, we observed a higher accuracy for LumiraDx SARS-CoV-2 Ag Ultra towards LumiraDx SARS-CoV-2 & Flu A/B, despite the declared limit of detection of this latter was lower (80 TCID50/mL vs. 800 TCID50/mL for LumiraDx SARS-CoV-2 Ag Ultra). This could probably be due to the different SARS-CoV-2 strains used for the evaluation of the test in terms of isolate and supplier. In our opinion, according to the real performance observed in clinical settings of those declared [6], this stressed the need for in-house validations of antigenic assays in order to provide the most suitable testing strategy. Thus, we should consider the added value of these assays in terms of cost, rapidity, and impact on the clinical management of symptomatic patients.
In this study, we focused on the antigenic detection of the SARS-CoV-2 virus, not evaluating the interaction with Flu A or B or other respiratory viruses. Moreover, due to the small sample size, no impact of variants on clinical performances was assessed. Finally, it is noteworthy that our results were obtained via UTM samples tested according to the manufacturer’s instructions, not directly from collected swabs. Thus, there is a dilution factor to be considered. Future analysis will be addressed to evaluate the relationship between the two assays in clinical settings, patients’ symptomatology, and epidemic context.
In summary, rapid antigenic tests could provide useful support to molecular testing, especially in critical settings, but the advantages and limitations should be evaluated via an in-house analysis.
Acknowledgments
We thank LumiraDx Italy for their support in publishing this work.
Author Contributions
Conceptualization, A.R. (Andrea Rocchetti), A.M., and A.R. (Annalisa Roveta); methodology, S.S. and S.P.; validation, E.S., C.B. and M.C.; investigation, V.P., S.C. and M.G.; data curation, S.S.; writing—original draft preparation, P.B., V.P. and S.S.; writing—review and editing, P.B. and V.P; supervision, A.M. and A.R. (Andrea Rocchetti). All authors have read and agreed to the published version of the manuscript.
Institutional Review Board Statement
This study was performed according to the declaration of Helsinki principles and our institution’s ethical standards. Ethical approval was not required because this was a retrospective analysis of data from samples collected as part of standard care, and those included in the database were deidentified before access. No personal information was stored in the study database. No patient intervention occurred in relation to the obtained results.
Informed Consent Statement
Not applicable.
Data Availability Statement
The data contained in this manuscript are available upon request.
Conflicts of Interest
The authors declare no conflict of interest.
Funding Statement
This research received no external funding.
Footnotes
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
References
- 1.Naqvi A.A.T., Fatima K., Mohammad T., Fatima U., Singh I.K., Singh A., Atif S.M., Hariprasad G., Hasan G.M., Hassan I. Insights into SARS-CoV-2 genome, structure, evolution, pathogenesis and therapies: Structural genomics approach. Biochim. Biophys. Acta-Mol. Basis Dis. 2020;1866:165878. doi: 10.1016/j.bbadis.2020.165878. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Yin Y., Wunderink R.G. MERS, SARS and other coronaviruses as causes of pneumonia. Respirology. 2018;23:130–137. doi: 10.1111/resp.13196. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Bottino P., Zanotto E., Sidoti F., Pastrone L., Piva R., Mereu E., Costa C., Cavallo R. Extraction Bottleneck in the Diagnosis of SARS-CoV-2: Evaluation of an Alternative Protocol Derived from Veterinary Use. Microorganisms. 2023;11:535. doi: 10.3390/microorganisms11020535. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Corman V.M., Landt O., Kaiser M., Molenkamp R., Meijer A., Chu D.K., Bleicker T., Brünink S., Schneider J., Schmidt M.L., et al. Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR. Euro. Surveill. Bull. Eur. Sur. Les Mal. Transm. Eur. Commun. Dis. Bull. 2020;25:2000045. doi: 10.2807/1560-7917.ES.2020.25.3.2000045. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Leli C., Di Matteo L., Gotta F., Cornaglia E., Vay D., Megna I., Pensato R.E., Boverio R., Rocchetti A. Performance of a SARS-CoV-2 antigen rapid immunoassay in patients admitted to the emergency department. Int. J. Infect. Dis. 2021;110:135. doi: 10.1016/j.ijid.2021.07.043. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Scheiblauer H., Filomena A., Nitsche A., Puyskens A., Corman V.M., Drosten C., Zwirglmaier K., Lange C., Emmerich P., Müller M., et al. Comparative sensitivity evaluation for 122 CE-marked rapid diagnostic tests for SARS-CoV-2 antigen Germany, September 2020 to April 2021. Eurosurveillance. 2021;26:2100441. doi: 10.2807/1560-7917.ES.2021.26.44.2100441. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Antigen-Detection in the Diagnosis of SARS-CoV-2 Infection: Interim Guidance, 6 October 2021 n.d. [(accessed on 5 September 2023)]. Available online: https://apps.who.int/iris/handle/10665/345948.
- 8.Bohn M.K., Lippi G., Horvath A.R., Erasmus R., Grimmler M., Gramegna M., Mancini N., Mueller R., Rawlinson W.D., Menezes M.E., et al. IFCC interim guidelines on rapid point-of-care antigen testing for SARS-CoV-2 detection in asymptomatic and symptomatic individuals. Clin. Chem. Lab. Med. 2021;59:1507–1515. doi: 10.1515/cclm-2021-0455. [DOI] [PubMed] [Google Scholar]
- 9.Lippi G., Henry B.M., Plebani M. LumiraDX SARS-CoV-2 Antigen Test for Diagnosing Acute SARS-CoV-2 Infection: Critical Literature Review and Meta-Analysis. Diagnostics. 2022;12:947. doi: 10.3390/diagnostics12040947. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Widyasari K., Kim S., Kim S., Lim C.S. Performance Evaluation of STANDARD Q COVID/FLU Ag Combo for Detection of SARS-CoV-2 and Influenza A/B. Diagnostics. 2023;13:32. doi: 10.3390/diagnostics13010032. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Dinç H.Ö., Karabulut N., Alaçam S., Uysal H.K., Daşdemir F.O., Önel M., Tok Y.T., Sirekbasan S., Agacfidan A., Gareayaghi N., et al. Evaluation of the Diagnostic Performance of a SARS-CoV-2 and Influenza A/B Combo Rapid Antigen Test in Respiratory Samples. Diagnostics. 2023;13:972. doi: 10.3390/diagnostics13050972. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Technical Guidance Publications, n.d. [(accessed on 2 May 2021)]. Available online: https://www.who.int/emergencies/diseases/novel-coronavirus-2019/technical-guidance-publications.
- 13.Schneider U.V., Forsberg M.W., Leineweber T.D., Jensen C.B., Ghathian K., Agergaard C.N., Mortensen K.K., Cohen A., Jørgensen C.S., Larsen H., et al. A nationwide analytical and clinical evaluation of 44 rapid antigen tests for SARS-CoV-2 compared to RT-qPCR. J. Clin. Virol. 2022;153:105214. doi: 10.1016/j.jcv.2022.105214. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Krüger L.J., Tanuri A., Lindner A.K., Gaeddert M., Köppel L., Tobian F., Brümmer L.E., Klein J.A., Lainati F., Schnitzler P., et al. Accuracy and ease-of-use of seven point-of-care SARS-CoV-2 antigen-detecting tests: A multi-centre clinical evaluation. EBioMedicine. 2022;75:103774. doi: 10.1016/j.ebiom.2021.103774. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Mak G.C., Lau S.S., Wong K.K., Chow N.L., Lau C.S., Lam E.T., Chan R.C., Tsang D.N. Analytical sensitivity and clinical sensitivity of the three rapid antigen detection kits for detection of SARS-CoV-2 virus. J. Clin. Virol. Off. Publ. Pan Am. Soc. Clin. Virol. 2020;133:104684. doi: 10.1016/j.jcv.2020.104684. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Denzler A., Jacobs M.L., Witte V., Schnitzler P., Denkinger C.M., Knop M. Rapid comparative evaluation of SARS-CoV-2 rapid point-of-care antigen tests. Infection. 2022;50:1281–1293. doi: 10.1007/s15010-022-01810-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Thommes L., Burkert F.R., Öttl K.-W., Goldin D., Loacker L., Lanser L., Griesmacher A., Theurl I., Weiss G., Bellmann-Weiler R. Comparative evaluation of four SARS-CoV-2 antigen tests in hospitalized patients. Int. J. Infect. Dis. IJID Off. Publ. Int. Soc. Infect. Dis. 2021;105:144–146. doi: 10.1016/j.ijid.2021.02.052. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Hastie E., Amogan H., Looney D., Mehta S.R. Association between SARS-CoV-2 Viral Load and Patient Symptoms and Clinical Outcomes Using Droplet Digital PCR. Viruses. 2023;15:446. doi: 10.3390/v15020446. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Brizuela M.E., Goñi S.E., Cardama G.A., Zinni M.A., Castello A.A., Sommese L.M., Farina H.G. Correlation of SARS-CoV-2 Viral Load and Clinical Evolution of Pediatric Patients in a General Hospital From Buenos Aires, Argentina. Front. Pediatr. 2022;10:883395. doi: 10.3389/fped.2022.883395. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Frediani J.K., Parsons R., McLendon K.B., Westbrook A.L., Lam W., Martin G., Pollock N.R. The New Normal: Delayed Peak SARS-CoV-2 Viral Loads Relative to Symptom Onset and Implications for COVID-19 Testing Programs. MedRxiv Prepr. Serv. Health Sci. 2023 doi: 10.1101/2023.05.09.23289735. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
The data contained in this manuscript are available upon request.