Comparative evaluation of four SARS-CoV-2 antigen tests in hospitalized patients

Abstract

Objectives

Rapid identification of infected subjects is a cornerstone for controlling a pandemic like the current one with the SARS-CoV-2. Easy to handle antigen tests can provide timely results, which is of particular importance in a primary care setting. However, concerns exist regarding their sensitivity, which led us to evaluate four commercially available tests in patients hospitalized for COVID-19.

Methods

We analyzed in parallel nasopharyngeal/oropharyngeal swabs from 154 consecutive patients admitted to our department with moderate to severe COVID-19, using quantitative RT-PCR (Cobas, Roche) and up to four antigen tests from different distributors. Antigen test results were linked to Ct (cycle threshold) values as markers for patients’ infectivity.

Results

We found that two out of four antigen tests correctly identified subjects with high viral loads (Ct ≤ 25), and three out of four tests detected more than 80% of subjects with a Ct ≤ 30, which is considered the threshold for infectivity. However, one test investigated had a poor clinical performance. When investigating subjects with Ct values >30, we found that the antigen test was still positive in up to 45% of those cases.

Conclusion

Most antigen tests had a sufficient sensitivity to identify symptomatic subjects infected with SARS-CoV-2 and with transmissible infection. On the other hand, antigen testing may not be suitable to identify loss of infectivity in COVID-19 subjects during follow-up. Newly introduced antigen tests need to be validated in a clinical or primary care setting to define their clinical usefulness.

Introduction

A central component for controlling a pandemic, like the current one caused by the SARS-CoV-2 with no specific available therapy, is the rapid identification and containment of infected individuals (Weissleder et al., 2020). Easily accessible and relatively inexpensive antigen tests offer an easy-to-use diagnostic tool to quickly identify such patients. Antigen tests are useful in primary and emergency care settings because they do not require laboratory equipment. They have been shown to aid in the specific diagnosis of respiratory infections such as influenza, specifically during the cold season when many respiratory viruses with similar clinical symptoms are circulating (Moriyama et al., 2020). However, a major drawback for the use of antigen tests is their lack of sensitivity which is sometimes much lower within a clinical setting than described in the user's manual (Lanser et al., 2020), as also shown for influenza bedside tests during the 2009 pandemic (Bellmann-Weiler et al., 2011). Several antigen tests for rapid detection of SARS-CoV-2 have been introduced into clinical practice to diagnose symptomatic patients. However, information on their sensitivity in symptomatic patients in a real-life clinical setting is scarce (Diao et al., 2020). Therefore, we performed a comparative analysis of four commercially available SARS-CoV-2 antigen tests in symptomatic COVID-19 patients who needed hospitalization and compared their analytical outcomes with the results, and cycle threshold (Ct) values obtained with a commercially available RT-PCR test.

Methods

We investigated consecutive COVID-19 patients admitted to our inpatient ward at the Department of Internal Medicine II, the Medical University of Innsbruck, from August to the end of October 2020.

Oropharyngeal swabs were collected from a total of 154 patients (median age of 69 years (18–92), 35.7% women and 64.3% men) and analyzed for the presence of SARS-CoV-2 viral RNA (target ORF1a/b and B-CoV target E-Gene) by RT-PCR employing the Cobas® apparatus (Roche Diagnostics GmbH, Mannheim, Germany). In parallel, we took nasopharyngeal swabs from the same patients for SARS-CoV-2 antigen detection using one to four antigen tests from different manufacturers, namely (in alphabetical order of distributor) Panbio™ COVID-19 Ag Rapid test (Abbott, Chicago, Illinois), Novel Coronavirus (2019-nCov) Antigen Detection Kit (CLMSRDL, Sichuan Mass Spectrometry Biotechnology Co., Ltd, Chengdu, Sichuan), DIAQUICK COVID-19 Ag Cassette (DIALAB, Wiener Neudorf, Austria) and SARS-CoV-2 Rapid Antigen Test (Roche Diagnostics Deutschland GmbH, Mannheim, Germany). Tests were performed by expert staff at the bedside using swabs provided in the antigen test kits as instructed in the user manual at the time. The tests were immediately analyzed so that the investigators were blinded to Ct values.

We retrospectively calculated the sensitivity and positive predictive value for the tests, using the RT-PCR results and Ct values as reference. All statistics were calculated with IBM SPSS® Statistics 27.0. We defined Ct values under or equal to 25 in the RT-PCR as highly contagious, values between 25 and up to 30 as contagious, and values above 30 as no longer contagious, according to the recommendations of the German Robert Koch Institute (RKI) (Laferl et al., 2020).

The study was approved by the ethics committee of the Innsbruck Medical University (ID of ethical vote: 1167/2020) and conformed to the Declaration of Helsinki's principles; informed consent of the patients was obtained.

Results

When studying patients with a Ct value ≤25, which reflects the population with the highest viral loads and thus the highest infectivity, we found that two tests were able to detect all affected patients, whereas the remaining tests had sensitivities of 83.3% and 60%, respectively (Table 1 ). We then examined the four antigen tests’ clinical performance, including all patients with a Ct value ≤30 (Diao et al., 2020). We observed that the sensitivity of the tests was between 45.2% and 88.9%, indicating large variances in the clinical usefulness of those tests to identify infected individuals with a high probability. Thus, the positive predictive values (PPV) of the antigen tests to detect patients with transmissible infections (Ct values ≤30) were between 52.5 (CI 45.2–59.6%) and 73.8 (CI 62.3–82.8%) (Table 1). Three out of four tests had a sensitivity above 79%. However, one out of the four antigen tests showed correct positive results in only 60% of patients with Ct values ≤30.

Table 1.

Comparison of rapid antigen tests of four different distributors

Test distributora	PCR Ct ≤ 25	PCR Ct ≤ 30	PCR Ct > 30	PPV (CI) PCR Ct cutoff 30	Total number of samples
	N (Ag+/Ag−) % positive (95% CI)	N (Ag+/Ag−) % positive (95% CI)	N (Ag+/Ag−) % positive	PPV (CI 95%)
Abbotta	18 (15/3) 83.3%(58.6–96.4%)	39 (31/8) 79.5% (63.5–90.7%)	43 (11/32) 25.6%	73.8 (62.3–82.8%)	82
CLMSRDLa	10 (6/4) 60%(26.2–87.8%)	31 (14/17) 45.2%(27.3–64.0%)	54 (5/49) 9.3%	73.7 (52.7–87.6%)	85
DIALABa	12 (12/0) 100%(73.5–100%)	36 (32/4) 88.9%(73.9–96.9%)	63 (29/34) 46.0%	52.5 (45.2–59.6%)	99
Rochea	9 (9/0) 100%(66.4–100%)	32 (27/5) 84.4%(67.2–94.7%)	39 (16/23) 41.0%	62.8 (53.0–71.7%)	71

Confidence intervals (CI 95%) for antigen tests sensitivities in relation to ct values ≤25 and ≤30 are given in parentheses. Sensitivity and positive predictive value infections (PPV) using PCR Ct values as reference (>30 defined as non-infectious and ≤30 and ≤25 as infectious); N = number.

^aFor details of tests, see Methods section.

To determine whether the antigen tests would remain positive despite non-infectivity, we swabbed patients after prolonged hospitalization during follow-up with Ct values above 30, as proposed by the RKI (Laferl et al., 2020). We found that in subjects with Ct values >30, 9.3% to 46.0% of samples showed positive antigen test results.

Discussion

This single-center study presents a clinical evaluation and comparison of four commercially available COVID-19 antigen tests, using RT-PCR as a reference.

Our study's strength is that the analyses were performed at a single center with experienced personnel and standardized procedures for sampling and processing of nasopharyngeal/oropharyngeal swabs, which were immediately analyzed in parallel by RT-PCR and antigen tests. This provides information on the actual value of antigen detection kits as a point of care method for identifying infected COVID-19 patients and estimates the clinical utility of such tests to diagnose symptomatic patients in a primary care setting. A limitation of our study is that we cannot provide information on the tests’ specificity as we only investigated hospitalized patients with already confirmed COVID-19.

A further limitation of our study is the usage of Ct-values to define levels of infectivity. Ct-values can vary considerably, either because of inconsistent sampling methods or based on differences between RT-PCR methods, which currently lack standardized reference materials (Buchta et al., 2020). To reliably determine infectivity based on Ct-values, further validation and international standardization of the available quantitative RT-PCR assays is required.

Our study indicates that some antigen tests have an excellent sensitivity to identify infected patients with COVID-19 like symptoms needing hospitalization, specifically those with higher viral loads and thus higher infectivity, which has already been shown for antigen tests other than those used by us (Lambert-Niclot et al., 2020). In our opinion, this makes them a suitable tool in primary care situations to rapidly detect SARS-CoV-2 infected individuals among symptomatic subjects. Nevertheless, one out of four CE-certified antigen tests showed inferior clinical performance.

However, we cannot provide information on whether those tests perform similarly well in subjects who are less severely affected by the infection. Moreover, our results do not allow speculation on the usage of antigen tests in strategies aimed at screening asymptomatic patients in the incubation phase of the infection, although some governments have implemented such strategies. Currently, the WHO does not recommend rapid immunoassays for unselected screening for COVID-19 but instead recommends it for testing individuals with a high pre-test probability of being infected (e.g., local outbreak situation in semi-closed communities or groups for early detection and isolation, health workers during outbreaks, etc.), especially when an analysis by PCR is not immediately available (World Health, 2020).

To have a definitive answer to these critical public health issues, further evaluations of these tests with parallel PCR analyses have to be carried out during screening events, which will also provide more information regarding the respective tests’ specificity.

Numerous studies evaluating POCT antigen tests have recently emerged in the literature. Strömer et al. compared the SARS-CoV-2 NADAL COVID-19® Ag POCT with RT-PCR and found that the POCT was more likely to be positive with lower Ct values (Strömer et al., 2020). Our results regarding the Panbio™ COVID-19 Ag Rapid test by Abbott are concordant with the findings of Albert et al. (Albert et al., 2020); due to the absence of reported Ct values, no direct comparison is possible between our study and that by Olearo et al., nonetheless in accordance with our finding they also detected an increase in sensitivity with higher viral loads (Olearo et al., 2020).

Our comparative evaluation of the antigen tests was extended to patients being considered non-infectious according to the recommendations of the RKI (Laferl et al., 2020). These investigations showed persistence of positivity in many subjects even with Ct values above 30 and lack of COVID-19 specific symptoms. This suggests that antigen tests may not serve as a suitable tool to determine either persisting infectivity or end of infectivity in subjects suffering from COVID-19.

Funding

There was no funding source for this study.

Ethical approval

The study was approved by the ethics committee of the Innsbruck Medical University (ID of ethical vote: 1167/2020) and conformed to the principles of the Declaration of Helsinki.

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have influenced the work reported in this paper.