Performance of the Sofia SARS-CoV-2 rapid antigen test as frontline test in a university hospital, Germany

Abstract

The rapid and reliable detection of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is of high importance for individual patient care and hospital infection prevention. We aimed to evaluate the performance of the Sofia SARS-CoV-2 antigen rapid diagnostic test (Ag-RDT) in comparison to real-time reverse-transcription polymerase chain reaction (RT-PCR). We conducted a prospective, monocentric cross-sectional study in an emergency department of a German university hospital from November 2020 to March 2021. We tested all samples using both Sofia SARS-CoV-2 Ag-RDT and real-time RT-PCR. A total of 7877 patients were included. Overall sensitivity of the Ag-RDT was 62.9% and specificity was 99.4%. Sensitivity varied across study months, whereas specificity remained high. Sensitivity increased to 94.2% in samples with a cycle threshold (Ct)-value ≤25. The Sofia Ag-RDT proved to be a rapid tool to detect samples with high viral loads (Ct-value ≤25) and might thus help to identify infectious patients.

1. Background

Rapid and reliable diagnostics are instrumental in containing the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic. Hospitals are among the most vulnerable institutions and admission testing has become current standard of practice at many hospitals worldwide.

RT-PCR is the current gold standard to diagnose SARS-CoV-2 infection but is hampered by rather long turnaround time and the dependency on reagents and instrumentation. In addition, trained laboratory personnel are needed, which limits its application in remote and/or resource-limited settings. Antigen rapid diagnostic tests (Ag-RDTs) can be done directly at the point of care within very short time, are less costly compared to RT-PCR, and easy to perform. To enhance read-out, some Ag-RDT are equipped with a device to measure fluorescence. These characteristics allow for frequent diagnostic testing which is critical to contain community spread or for admission testing in the health care setting. A number of studies have been conducted to determine the clinical performance of Ag-RDT assays in different cohorts and study settings e.g., in outpatient SARS-CoV-2 testing facilities or to a lesser extent in hospitals [1,2]. The World Health Organization (WHO) recommends a sensitivity of ≥80% of Ag-RDT in comparison to RT-PCR [1], [2], [3], [4]. In previous studies, the Sofia Ag-RDT demonstrated a sensitivity among asymptomatic patients between 35.7% and 78.6% and among symptomatic patients between 72.1% and 93.8%, while specificity of at least 96.9% have been reported [2,[5], [6], [7]]. However, only a small number of studies using this assay were conducted in a hospital setting. We were able to show a sensitivity of 65% in a pilot study [8]. Of note, varying prevalences, and the surge of novel variants of concern (VOC) may influence the performance of Ag-RDT [9].

2. Objectives

To address this, we aimed to evaluate the Sofia Ag-RDT in a German university hospital in a real-world setting. The study was conducted over a period of 5 months beginning November 1, 2020 to March 31, 2021. This period was characterized by a substantial increase of COVID-19 cases and the introduction of the VOC B.1.1.7 (Alpha) in Germany in early 2021.

3. Materials and methods

3.1. Patients, study setting, and sample collection

We prospectively enrolled all hospitalized patients irrespective of respiratory symptoms admitted to the emergency department (ED) at Bielefeld Hospital, a university hospital with 1300 beds in northern Germany from  November 1 to March 31, 2021. The study was conducted over a time period of 5 months with 7-day incidence ranging between 73 and 974 per 100,000 inhabitants in Bielefeld, which peaked at the start of the study period, and subsequently declined [10]. The end of the study period was characterized by the rise of the SARS-CoV-2 B.1.1.7 (Alpha) variant of concern (VOC), which was detected in 78.5% of cases by the end of March [11]. Trained medical personnel took 2 nasopharyngeal swab samples per patient using both nostrils. One nasopharyngeal swab was collected in universal transport medium for molecular testing within the next 24 hours, followed by another nasopharyngeal swab for immediate on-site antigen testing. We retrospectively extracted data on COVID-19 associated symptoms (i.e., cough, dyspnea, loss of smell/taste, and fever) from the medical charts of all RT-PCR positive patients.

3.2. Rapid antigen detection assay

We used the Sofia antigen fluorescent immunoassay (FIA) (Quidel, Kornwestheim, Germany) according to the manufacturer's instructions in Europe [12]. This version also accepts nasopharyngeal swabs as sample type (Package Insert – DE, IT, FR, ES). Of note, the FDA EUA-authorized version recommends the use of anterior nasal swabs only. The Sofia FIA is a sandwich-based lateral flow assay and provides automated and user-independent read out using the Sofia 2 FIA analyzer.

3.3. RT-PCR

As reference test, we used nucleic acid testing (NAT) in 2 different laboratories (certified according to DIN EN ISO 9001 and accredited according to DIN EN ISO/IEC 17025). NAT assays included the VIASURE SARS-CoV-2 RT-PCR (CerTest Biotec S.L.), the RIDA®GENE SARS-CoV-2 (r-biopharm, Darmstadt, Germany) and RIDA®GENE Flu & SARS-CoV-2 (r-biopharm, Darmstadt, Germany), the Xpert®Xpress SARS-CoV-2 (Cepheid, Frankfurt, Germany), the LightMix® Modular Sarbecovirus E-gene (TIB Molbiol, Berlin, Germany), and the Allplex™ SARS-CoV-2/FluA/FluB/RSV (Seegene, Seoul, Korea) assay. The VIASURE detects the ORF1ab and N gene, the RIDA®GENE either the E gene or the E gene and RdRp gene, the Xpert the E and the N gene, the LightMix® the E gene, and the Allplex™ the S, N, and RdRP gene. Cycle threshold (Ct)-values were recorded for each positive NAT sample (depending on assay; lowest Ct-value if multiple gene targets are included and distinguished in the assay). All RT-PCR assays yielded comparable performance in manufacturer independent studies [13], [14], [15]. We used the INSTAND e.V. Bezugsprobe 1 (10.000.000 SARS-CoV-2 RNA copies/mL) and 2 (1.000.000 SARS-CoV-2 RNA copies/mL) to calibrate our assays. Assays were able to detect 1.000.000 RNA copies/mL at a Ct-value of approximately 25 [16]. The Ct-value is a surrogate measure for SARS-CoV-2 virus concentration and a Ct-value of ≤25 has been shown to correlate with infectivity [17], [18], [19].

3.4. Statistical analyses

Data were collected using SPSS software 24. Statistical analyses were performed using R (Version 4.1.0). We determined sensitivity, specificity, positive and negative predictive value (PPV, NPV) and 95% CI for Ag-RDT using real-time RT-PCR results as reference standard. Mann-Whitney U test and Tukey´s multiple comparison test was used for Ct value comparisons; P-values <0.05 were considered statistically significant. The performance measures of accuracy were compared using χ² test.

3.5. Ethics

Ethical approval was obtained (Az 2020-870-f-S; AeKWL/WWU Muenster).

4. Results

A total of 7877 patients were included and tested by both Ag-RDT and RT-PCR. Median age was 67 years (range 1−107 years), and 3640 (46%) were males. Of note, only 127/7877 (1.6%) patients were ≤18 years of age. Fourteen samples (0.2%) yielded invalid Ag-RDT results and were excluded from further analysis. We retested invalid samples with a new sample and all yielded a valid negative result.

Overall, 284/7863 (3.6%) tested positive using Ag-RDT and 375/7863 (4.8%) were positive using RT-PCR. Concordant results were observed in 7676/7863 (97.6%) samples and discordant results were recorded in 187 samples. Overall sensitivity of Ag-RDT in comparison to RT-PCR was 62.9% (95% confidence interval [CI], 57.8%–67.8%) and specificity was 99.4% (95% CI, 99.2%–99.5%) (Table 1 ). Of note, sensitivity increased to 94.2% (194/205 patients) when only samples with Ct-values ≤25 were included. We compared Ct-values of Ag-RDT positive/RT-PCR positive (n =  236) samples and Ag-RDT negative/RT-PCR positive (n = 139) samples. Mann-Whitney U test showed a significant lower median Ct-value (median 21, range 11−36) for Ag-RDT positive/RT-PCR positive compared to Ag-RDT negative/RT-PCR positive samples (median 32.2, range 18−41) (Mann-Whitney U test, P < 0.001, Fig. 1 ). In detail, only 11 Ag-RDT negative/RT-PCR positive samples had Ct-values ≤25.

Table 1.

Test performance of Ag-RDT in comparison to RT-PCR among patients presenting to the emergency department, n = 7863.

		Ag-RDT
		Negative, n = 7579	Positive, n = 284	Sensitivity, %	Specificity, %	PPV, %	NPV, %
RT-PCR	Negative, n = 7488	7440	48	62.9 (95% CI 58−68)	99.4 (95% CI 99−100)	83.1 (95% CI, 78−87)	98.2 (95%CI 98−98)
	Positive, n = 375	139	236

PPV = positive predictive value; NPV = negative predictive value

Fig. 1.

Box plot diagrams comparing the median, interquartile range (IQR) and range of Ct-values between true positive (TP) and false negative (FN) results for the AG-RDT samples (n = 375) (Mann-Whitney, P < 0.001).

We retrieved clinical data for all 375 RT-PCR-positive cases from the hospital based information system. Of these, 62 (16.5%) showed none of 4 COVID-19 associated symptoms, i.e., cough, dyspnea, loss of smell/taste, and fever. Of note, 33 (53.2%) of these patients without COVID-19 associated symptoms tested negative using Ag-RDT and the median Ct value in these Ag-RDT negative samples was 33.5 (range 20.5 – 41). We did not see RT-PCR/Ag-RDT positive patients without COVID-19 associated symptoms in our cohort. Using RT-PCR positive samples we calculated a sensitivity for the Ag-RDT of 72.1% (95% CI 64.9−79.3) in symptomatic patients and of 46.8% (95% CI, 34.4%−59.2%) in asymptomatic patients (χ² test, P < 0.001). In addition, median Ct values differed (Mann-Whitney U test, P < 0.001).

To appreciate the dynamics of the pandemic, we calculated sensitivity and specificity, positive predictive value (PPV) and negative predicitve value (NPV) per study month (Table 2 ). Lowest sensitivity was determined in January 2021 with 53.9%. Next, we plotted Ct-values separately for each month (Fig. 2 ). We were able to show that median Ct-values were lower in December 2020 compared to January 2021. Median Ct-value in December was 23.0 (interquartile range [IQR] 19.0−29.7) and in January the median Ct-value was 28.0 (IQR 22.5−33.0) (p=0,008; Tukey´s multiple comparison test). All other median Ct-values were not significantly different. This data is in line with data from Table 2, where highest sensitivity of 74.6% was seen in December and the lowest in January with 53.9%.

Table 2.

Sensitivity, specificity, positive, and negative predictive value of Ag-RDT across study months.

	November 2020, n = 1391	December 2020, n = 1605	January 2021, n = 1572	February 2021, n = 1572	March 2021, n = 1723
Sensitivity, %	57.1 (95% CI, 46−67)	74.6 (95% CI, 66−82)	53.9 (95% CI, 42−65)	60.7 (95% CI, 41−78)	58.0 (95% CI, 43−72)
Specificity, %	99.3 (95% CI, 99−100)	99.3  (95% CI, 99−100)	98.9 (95% CI, 98−99)	99. 8 (95% CI, 99−100)	99.5 (95% CI, 99−100)
Test prevalence, %	6.5	8.1	4.8	1.8	2.9
Positive predictive value (PPV), %	85.3 (95% CI, 74−93)	89.8  (95% CI, 83−95)	70.7 (95% CI, 57−82)	85.0 (95% CI, 62−97)	78.4 (95% CI, 62−90)
Negative predictive value (NPV), %	97.1 (95% CI, 96−98)	97.8 (95% CI, 97−98)	97.4 (95% CI, 97−98)	99.3 (95% CI, 99−100)	98.8 (95% CI, 98−99)

Fig. 2.

Box plot diagrams comparing the median, IQR and of Ct values across study months of all RT-PCR positive samples. Statistical differences only observed between December and January (Tukey, P = 0.008).

The sensitivity varied depending on the Ct-values (Fig. 3 ): sensitivity was 100% (95% CI, 93%−100%) for Ct-values of <18, decreased to 98% (95% CI, 91%−100%) at Ct-values of 18 to 20.9 and further declined to 95% (95% CI, 87%−99%) at Ct-values of 21 to 23.9. Between Ct-values of 24 to 26.9 the sensitivity was 73% (95% CI, 57%−85%). At Ct-values of 27 to 29.9 the sensitivity was 44% (95% CI, 29%−60%) and found to be lowest at Ct-values between 30 and 41 with 15% (95% CI, 9%−23%). Of note, a Ct-value of 25 approximately corresponds to 1.000.000 SARS-CoV-2 RNA copies per mL indicating infectivity [20].

Fig. 3.

Sensitivity of Ag-RDT depending on Ct-value of RT-PCR. Error bars indicate 95% confidence interval.

5. Discussion

Here we report a clinical sensitivity of 62.9% (95% CI 57.8%–67.8%) and specificity of 99.4% (95% CI, 99.2%−99.5%) for the Sofia Ag-RDT compared to RT-PCR among patients presenting to the ED of a large university hospital. The measures of accuracy varied by Ct-values, symptom status, and test prevalence.

The WHO defines a sensitivity of ≥80% and specificity of ≥97% as the acceptability criteria for SARS-CoV-2 Ag-RDT. While the overall sensitivity is lower than the accetable threshold, the sensitivity inversely increased with Ct-values and sensitivity at Ct-values ≤25 was 94% and 100% at Ct-values of ≤18. This is in concordance to other reports [2,21,22]. Several studies aimed to assess the sensitivities of Ag-RDTs compared to RT-PCR and cell culture [21,[23], [24], [25]]. All studies observed an increase in sensitivities when cell culture was the reference standard, suggesting the use of Ag-RDT as a tool for detecting infectious patient samples. Pray et al. performed cell culture on 18 samples which were assessed by Sofia Ag-RDT as false negative, of which 2 viral isolations were obtained [25]. Taking this into consideration the here reported sensitivity of 94% at Ct-values ≤25 suggests that Sofia Ag-RDT might be suitable for identifying infectious patient samples in the ED setting.

The dependence of sensitivity on viral loads is in line with previous studies [2,23]. Stratified by months we found the largest difference in sensitivity in the months where Ct-values varied with statistical significance. By contrast, clinical specificity of the Sofia Ag-RDT was excellent throughout, which is a reassuring finding.

Intriguingly, a number of studies evaluated the Sofia Ag-RDT and reported sensitivities ranging from 72% to 80% in symptomatic persons [5,25]. To the best of our knowledge only 2 studies tested the Sofia Ag-RDT in a clincal setting using nasopharnygeal swabs and obtained comparable results with sensitivities of 66% and 76%, respectively [2,5]. In disconcordance to those studies we found differences in accuracy and Ct-values between symptomatic and asymptomatic patients. In part, this might be explained by differences in sampling types (nasopharyngeal versus anterior nasal swabs).

False-positive results were observed but to a lesser extent than false-negative ones. A higher rate of false-positive results might be associated with a low prevalvence setting, which was not the case in our study. A total of 139 false negative results was seen, which is a concerning result for the hospital setting but needs to be interpreted in light of a high median Ct-value.

Depending on test prevalence varying NPV and PPV were determined. As expected, test performance was better at higher test prevalence of SARS-CoV-2. Here, test prevalence never exceeded 10% despite covering a time frame of 5 months and the beginning of the third wave in Germany. WHO recommended acceptable PPVs of 62.8 and 78% for prevalences between 5% and 10%. Based on our study data the 95% confidence interval of the PPVs of November, January, February, and March fall below these thresholds. This finding challenges the use of Ag-RDT irrespective of the course of the pandemic.

Technically, the Sofia is equipped with a device for automated and improved read-out. The device can be connected to the hospital-based information system which facilitates documentation and reporting of test results. In our study, we did not encounter any technical problems with the system rendering it useful for routine testing. Reassuring, only a small fraction of Ag-RDT tests results returned invalid. This rate is lower than previously reported [25]. Importantly, most SARS-CoV-2 patients were identified using the Sofia Ag-RDT and the short time to result allows for optimized patient management and likely saves resources. For clinical practice e.g., in a hospital ED, it remains important to combine clinical data, and results of the Ag-RDT. Current guidelines recommend the use of Ag-RDTs in patients with symptoms compatible with COVID-19 and re-testing of asymptomatic but Ag-RDT negative patients after 1 to 2 days [26]. This is in line with the procedures of the Infection Control Unit at Klinikum Bielefeld and our findings, where sensitivity of AG-RDT was higher in symptomatic compared to asymptomatic RT-PCR positive patients. As a caveat, false negative Ag-RDT results in a hospital setting are a concern as these patients might give rise to nosocomial transmissions. In these cases, the use of RT-PCR, and adherence to infection prevention measures is recommended until final confirmation [27].

A strength of our study is the large number of unselected patients, which were all uniformly tested using both methods and its prospective nature. In addition, our study period covered times of varying SARS-CoV-2 test prevalence, and the introduction of the SARS-CoV-2 Alpha VOC. Unlike previous studies we included patients of all age groups. However, now that the Delta VOC emerged worlwide follow-up studies are warranted. Of note, Bourassa et al. recently demonstrated drop outs of the Sofia 2 Ag-RDT due to a D399N nucleocapsid gene mutation of SARS-CoV-2. Reassuringly, preliminary data showed that the Delta VOC had no impact on the perfromance of Ag-RDT [28].

This study has several limitations. First, data of symptom status have been obtained retrospectively from the medical charts, thus rely on proper documentation, and is prone for underestimation. Limited resources prevented us from reviewing the medical charts of every included patient. In addition, no systematic information on symptom onset in relation to date of testing was available to us and no information about symptom status of the test negative patients. Third, we did not perform cell culture due to a lack of resources to detemine infectivity but extrapolated this using Ct-values. Furthermore Ct-values have been obtained from 6 different assays and differences in the performance of the RT-PCR assays might have influenced the results of our study into an unknown direction. This study evaluated patients from the emergency department so test performance might differ in other test settings.

In conclusion, the sensitivity of the Sofia Ag-RDT in comparison to RT-PCR in a real-world hospital setting was below the threshold recommended by WHO. However, the Sofia Ag-RDT proved to be a reliable tool to rapidly identify high viral loads (Ct-value ≤25), which might be promising in detecting infectious patients. Only a small fraction of Ag-RDT negative/RT-PCR positive samples yielded SARS-CoV-2 concentrations, which were likely to be infectious.

Funding

This work was in part funded by the Bundesministerium für Bildung und Forschung (COVIRES; Fkz. 13GW0540C).

Declaration of competing interests

The authors have no relevant competing interest to disclose in relation to this work.

Ethical approval

Ethical approval was obtained (Az 2020-870-f-S; AeKWL/WWU Muenster).

Authors’ contribution

Study concept and design: LB, BR, MW, MP; data acquisition and laboratory analysis: LB, OK, JK, RK-H, BR, MW; Manuscript – first draft: LB, TD, AF; Supervision: MP; critical revision of manuscript: All. All authors read and approved the final manuscript.

Acknowledgments

We kindly acknowledge expert technical assistance of all laboratory technicians.