Abstract
At the time of writing, FIND has listed four CE-marked SARSCoV-2 antigen tests. We evaluated the recently CE-approved rapid POCT SD-Biosensor for SARS-CoV-2 nucleoprotein detection in nasopharyngeal secretions from 330 patients admitted to the Emergency Room for a suspect of COVID-19 and travelers returning home from high risk countries. Sensitivity, specificity, accuracy, negative and predictive values were consistent with the use of the test to mass-screening for SARS-CoV-2 surveillance.
Keywords: Sars CoV-2, Antigen test, Mass-screening, Point-of-care-testing, Cell culture
1. Background
In the context of COVID-19 pandemic, the development of rapid and easy-to-perform diagnostic methods is of high priority, to shorten the time of result-reporting, but this is a condition that demands rapid and cost-efficient approaches. The currently gold standard for the detection of SARS-CoV-2 relies on viral RNA amplification by real-time RT-PCR (RT-PCR) and requires few hours before results release [1]. The pandemic highlighted the limits of production and trade for molecular tests as we are facing a worldwide shortage of reagents. Point-of-care diagnostic tests (POCTs) for detecting viral antigens in clinical samples would be very helpful for the diagnosis of COVID-19 [2] either as mass-screening or first aid tests at the emergency room. At the time of writing, the Foundation for Innovative New Diagnostics (https://www.finddx.org/) has listed four CE-marked rapid SARS-CoV-2 antigen tests, which are primarily lateral flow immunochromatographic assays from nasopharyngeal swab (NP) with result reporting in less than 30 min. The evaluation of these tests by the scientific community, even if limited, outlines the major concern of false-negative results due to low viral loads. In fact, recently published works from different countries evaluating commercial and in-house POCTs for SARS-COV-2 showed a concordant high level of specificity (from 99.5 to 100 %), but a wide range of sensitivity (30 %–93.9 % [3,4]. Therefore, great efforts for the implementation of antigen test performance are currently on-going.
2. Objective
To evaluate a recently CE-approved POCT, the STANDARD Q COVID-19 Ag (SD-Biosensor, RELAB, I), for the detection of SARS CoV-2 nucleoprotein in NP swabs in comparison with the gold standard RT-PCR. POCT performances were studied in terms of sensitivity, specificity, and negative and predictive values to assess the contribution of this test to mass-screening for COVID-19.
3. Study design
The STANDARD Q COVID-19 Ag (R-Ag) was applied to 330 patients of two different populations from the phase-1 and -3 of the pandemic according to the Italian Government measures (http://www.governo.it/it/coronavirus-misure-del-governo): 185 randomly selected patients (mean age 44.6, 95 %CI: 40.7–48.6) referring at the Emergency Rooms of two Infectious Disease reference centers in North-Italy (ASL Città di Torino, Turin and San Martino University Hospital, Genoa) with symptoms and sign consistent with of COVID-19 from March 3rd to May 1st, 2020 and 145 travelers (mean age 35.9, 95 % CI: 32.7–39.1) returning home from European high risk countries (Croatia, Spain and Malta) in August, 2020.
Samples were collected in COPAN UTM medium (COPAN, I) and processed for SARS CoV-2 by RT-PCR using different methods: Seegene Allplex® 2019 n-CoV Assay (N = 159), DiaSorin Simplexa® (n = 28), and Cobas 6800 Roche® (N = 118). Cycle threshold (Ct) values were recorded and the mean Ct through different reactive genes was used as a proxy for the viral load. In a minority of samples, COVID-19 Ag antigen test (R-Ag) was run on left-over of diagnostic samples from March and April stored at −20 °C (13/185, 7%), then in parallel with RT-PCR in all the others. Three hundred ul of UTM were mixed to the R-Ag extraction buffer provided by the kit, three drops were applied to the solid device and covered with the proper film. Results were manually read after 15−30 min.
Statistical analysis was performed with R statistical framework: Shapiro-Wilk normality test on Ct values, Student’s T test for comparison between the mean Ct values from R-Ag-positive and R-Ag-negative, ROC curve (pROC package), and Cohen's kappa coefficient (psych package).
4. Results
Detection rates of SARS CoV-2 by R-Ag and RT-PCR were 23.3 % (77/330) and 33 % (109/330), respectively; no false positive with R-Ag were observed (Table 1 ). R-Ag sensitivity, specificity, negative and positive predictive values were 70.6 %, 100 %, 87.4 % and 100 %, respectively, compared with RT-PCR. Concordance between the two techniques was 90.3 % (Cohen’s k = 0.76, 95 % CI: 0.69−0.84).
Table 1.
Results of the SD-Biosensor antigen (R-Ag) test compared to RT-PCR in the samples tested. R-Ag sensitivity, specificity, negative and positive predictive values were 70.6 %, 100 %, 87.4 % and 100 %, respectively, compared with RT-PCR. Concordance between the two techniques was.90.3 %.
| R-Ag + N. |
R-Ag – N. |
Total N. |
Positive % |
Negative % |
|
|---|---|---|---|---|---|
| RT-PCR+ | 77 | 32 | 109 | 70.6 | |
| RT-PCR- | 0 | 221 | 221 | 100 | |
| Total | 77 | 253 | 330 | 23.3 |
In diagnostic samples (N = 185) R-Ag was positive in 75/104 (72.1 %) RT-PCR positive swabs and negative in 81/81 (100 %) with a sensitivity, specificity, negative and positive predictive values of 72.1 %, 100 %, 73.6 % and 100 %, respectively. Concordance between the two techniques was 84.3 % (Cohen’s k = 0.7, 95 % CI: 0.61−0.8). In screening samples, (N = 145) R-Ag was positive in 2/5 (40 %) RT-PCR positive swabs (asymptomatic patients) and negative in 140/140 (100 %) RT-PCR negative samples (overall sensitivity, specificity, negative and positive predictive values: 40 %, 100 %, 100 % and 97.9 %, respectively, compared to RT-PCR). Concordance between the two techniques was 97.9 % (Cohen’s k = 0.56, 95 % CI: 0.12–1).
The mean Ct values across viral genes by RT-PCR was used as a proxy for viral load; we observed a significantly lower value for concordant RT-PCR-positive/R-Ag-positive samples (22.3, IC95 % 22.26–22.34, range 12.3–36), than that in discordant RT-PCR-positive/R-Ag-negative samples (32.1, IC95 % 32.05–32.15; range 23.7–38.1) (p-value <<0.0001) (Fig. 1 ). Ct values for the two R-Ag groups showed a normal distribution (p-value = 0.39, R-Ag-positive; p = 0.36, R-Ag-negative). According to different mean Ct classes (≤25, 25–28, 28–30, 30–35, >35) the detection rate of R-Ag was 100 % for samples with a Ct <28 and decreased to 38.5 %, 26.7 %, and 9.1 % in the other ranks. Fig. 2 shows the ROC curve for R-Ag to estimate the Ct threshold value for R-Ag to detect a positive swab (equal to 27.7). In Table 2 a simulation of negative and positive predictive (NPV, PPV) values according to disease prevalence is shown.
Fig. 1.
Mean Ct across viral genes values for SD-Biosensor antigen (R-Ag) positive and negative tests in studied samples. A subset of R-Ag negative/RT-PCR positive samples were subjected to viral cell culture with negative results after 21-days of incubation.
Fig. 2.
ROC curve of sensitivity and specificity of the SD-Biosensor antigen (R-Ag) test in studied samples.
Table 2.
Simulation of Negative and Positive Predictive (NPV, PPV) values according to disease prevalence.
| PPV | NPV | |
|---|---|---|
| Prevalence of COVID-19 | Diagnostic samples (N = 185) | |
| 0.5 % | 100 % | 99.86 % (95 % CI 99.81−99.90) |
| 1 % | 100 % | 99.72 % (95 %CI 99.62−99.79) |
| 2 % | 100 % | 99.43 % (95 % CI 99.23−99.58) |
| 5 % | 100 % | 98.55 % (95 % CI 98.04−98.93) |
| 10 % | 100 % | 96.99 % (95 % CI 95.95−97.78) |
| Prevalence of COVId- 19 | Screening samples (N = 145) | |
| 0.5 % | 100 % | 99.70 % (95 % CI 99.39−99.85) |
| 1 % | 100 % | 99.4 % (95% CI 98.78−99.70) |
| 2 % | 100 % | 98.79 % (95 % CI 97.56−99.4) |
| 5 % | 100 % | 96.94 % (95% CI 93.93−98.48) |
| 10 % | 100 % | 93.75 % (95 % CI 88.0−96.84) |
In a small subset of R-Ag negative/RT-PCR positive samples (15/32) with Ct > 30 we inoculated NP swab in Vero cells. After subculturing with sequential blind passages, in the absence of any cytopathic effect, all supernatants tested negative for SARS CoV-2 by RT-PCR [1] after 21-day incubation.
5. Discussion
To control COVID-19 pandemic, improvement of SARS CoV-2 diagnosis with easy, rapid and cost-efficient approaches is urgently required. POCTs for the detection of SARS-CoV-2 antigens are quite promising; however, the principal concerns are the false-negative rate due to low viral loads [[3], [4], [5], [6], [7], [8]].
The STANDARD Q COVID-19 Ag identified 70.6 % of RT-PCR positive sample, respectively, with no false positive results (100 % of specificity). Using the Ct value by RT-PCR as a proxy for viral load, R-Ag-positive samples had a significantly lower Ct than that of R-Ag-negative samples; the majority of discordant RT-PCR-positive/R-Ag-negative samples reported negative results when cell-cultured. Therefore, R-Ag false negative results were found in samples with a low viral load, consistent with low viable virus and low infectiousness [9,10]. A major limit of our study was that the test was assessed in suboptimal conditions using UTM samples instead of on-site NP swabs.
The clinical performance of POCTs largely depends on the circumstances in which they are used, and the appropriate setting should be identified. In agreement with recently published works, our data confirm that this POCT is effective during the acute/recent phase of the disease within a few days after symptoms onset when the viral load in the upper respiratory tract is at its peak [[3], [4], [5],11]. The adoption of POCT for SARS CoV-2 testing is certainly more suitable in point of care centers for mass screening where the prevalence of COVID-19 is much lower and the pre-test probability of not having the disease is higher than that in the patients admitted to the emergency room were the pre-test probability of having COVID-19 is significantly higher and false negative results are relevant for the correct management of patients.
The main advantages of POCTs for antigen testing are rapidity, easy of interpretation, limited technical skill and infrastructure required, and this continues to make them worth pursuing. Lastly, the adoption of POCTs in mass screening testing could decrease the burden on virology laboratories that have been overwhelmed during the last COVID-19 pandemics, and the shortage of reagent they are facing.
Funding
We thank RELAb for the donation of the STANDARD Q COVID-19 SD-Biosensor kits to pursue the study. No other specific grant from public funding agencies was received.
CRediT authorship contribution statement
Francesco Cerutti: Writing - original draft, Formal analysis, Conceptualization. Elisa Burdino: Investigation, Validation. Maria Grazia Milia: Investigation. Tiziano Allice: Investigation, Validation. Gabriella Gregori: Investigation, Validation. Bianca Bruzzone: Data curation. Valeria Ghisetti: Conceptualization, Writing - review & editing, Supervision.
Declaration of Competing Interest
The authors report no declarations of interest.
References
- 1.Corman V.M., Landt O., Kaiser M., Molenkamp R., Meijer A., Chu D.K., et al. Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR. Eurosurveillance [Internet] 2020;25(3) doi: 10.2807/1560-7917.ES.2020.25.3.2000045. https://www.eurosurveillance.org/content/10.2807/1560-7917.ES.2020.25.3.2000045 Jan 23 [cited 2020 Feb 11]. Available from: [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Mak G.C., Cheng P.K., Lau S.S., Wong K.K., Lau C., Lam E.T., et al. Evaluation of rapid antigen test for detection of SARS-CoV-2 virus. J. Clin. Virol. 2020;129(August) doi: 10.1016/j.jcv.2020.104500. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Porte L., Legarraga P., Vollrath V., Aguilera X., Munita J.M., Araos R., et al. Evaluation of novel antigen-based rapid detection test for the diagnosis of SARS-CoV-2 in respiratory samples. Int. J. Infect. Dis. 2020;(June) doi: 10.1016/j.ijid.2020.05.098. S1201971220304057. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Scohy A., Anantharajah A., Bodéus M., Kabamba-Mukadi B., Verroken A., Rodriguez-Villalobos H. Low performance of rapid antigen detection test as frontline testing for COVID-19 diagnosis. J. Clin. Virol. 2020;129(August) doi: 10.1016/j.jcv.2020.104455. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Lambert-Niclot S., Cuffel A., Le Pape S., Vauloup-Fellous C., Morand-Joubert L., Roque-Afonso A.-M., et al. Evaluation of a rapid diagnostic assay for detection of SARS CoV-2 antigen in nasopharyngeal swab. J. Clin. Microbiol. 2020 doi: 10.1128/JCM.00977-20. May 13; JCM.00977-20, jcm;JCM.00977-20v1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Harcourt J., Tamin A., Lu X., Kamili S., Sakthivel S.K., Murray J., et al. Severe acute respiratory syndrome coronavirus 2 from patient with 2019 novel coronavirus disease, United States. Emerg. Infect. Dis. [Internet] 2020;26(6) doi: 10.3201/eid2606.200516. http://wwwnc.cdc.gov/eid/article/26/6/20-0516_article.htm Jun [cited 2020 May 10]. Available from: [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Mavrikou S., Moschopoulou G., Tsekouras V., Kintzios S. Development of a portable, ultra-rapid and ultra-sensitive cell-based biosensor for the direct detection of the SARS-CoV-2 S1 spike protein antigen. Sensors. 2020;20(May (11)):3121. doi: 10.3390/s20113121. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Mertens P., De Vos N., Martiny D., Jassoy C., Mirazimi A., Cuypers L., et al. Development and potential usefulness of the COVID-19 Ag respi-strip diagnostic assay in a pandemic context. Front. Med. 2020;7(May (225)) doi: 10.3389/fmed.2020.00225. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Scola B.L. Viral RNA load as determined by cell culture as a management tool for discharge of SARS-CoV-2 patients from infectious disease wards. Eur. J. Clin. Microbiol. Infect. Dis. 2020 doi: 10.1007/s10096-020-03913-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Bullard J., Dust K., Funk D., Strong J.E., Alexander D., Garnett L., et al. Predicting infectious SARS-CoV-2 from diagnostic samples. Clin. Infect. Dis. 2020;(May) doi: 10.1093/cid/ciaa63è118. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Omi K., Takeda Y., Mori M. SARS-CoV-2 qRT-PCR Ct value distribution in Japan and possible utility of rapid antigen testing kit [Internet] Infect. Diseases (except HIV/AIDS) 2020 doi: 10.1101/2020.06.16.20131243. Jun [cited 2020 Aug 12]. Available from: [DOI] [Google Scholar]