Determining the reliability of rapid SARS-CoV-2 antigen detection in fully vaccinated individuals

Abstract

Background

Rapid antigen detection tests (RADT) are commonly used as SARS-CoV-2 diagnostic tests both by medical professionals and laypeople. However, the performance of RADT in vaccinated individuals has not been fully investigated.

Objectives

RT-qPCR and rapid antigen detection testing were performed to evaluate the performance of the Standard Q COVID-19 Ag Test in detecting SARS-CoV-2 breakthrough infections in vaccinated individuals.

Study design

Two swab specimens, one for RT-qPCR and one for RADT, were collected from vaccinated individuals in an outpatient clinic. For comparison of RADT performance in vaccinated and unvaccinated individuals, a dataset already published by this group was used as reference.

Results

During the delta wave, a total of 696 samples were tested with both RT-qPCR and RADT that included 692 (99.4%) samples from vaccinated individuals. Of these, 76 (11.0%) samples were detected SARS-CoV-2 positive by RT-qPCR and 45 (6.5%) samples by the Standard Q COVID-19 Ag test. Stratified by Ct values, sensitivity of the RADT was 100.0%, 94.4% and 81.1% for Ct ≤ 20 (n=18), Ct ≤ 25 (n=36) and Ct ≤ 30 (n=53), respectively. Samples with Ct values ≥ 30 (n=23) were not detected. Overall RADT specificity was 99.7% and symptom status did not affect RADT performance. Notably, RADT detected 4 out of 4 samples of probable Omicron variant infection based on single nucleotide polymorphism analysis.

Conclusion

Our results show that RADT testing remains a valuable tool in detecting breakthrough infections with high viral RNA loads.

1. Introduction

In combating the SARS-CoV-2 pandemic, a variety of measures has been taken to break transmission chains [1], [2], [3], [4]. For example, rapid antigen detection tests (RADT) have been implemented as a public health strategy in numerous countries to screen for SARS-CoV-2 infected individuals [5], [6], [7], [8]. Since the introduction of RADT, multiple publications have shown that antigen tests can reliably detect individuals with high SARS-CoV-2-RNA loads, who pose an increased risk of SARS-CoV-2 transmission (> 6 log₁₀ copies/ml) [9], [10], [11], [12].

Several COVID-19 vaccines with high efficacy against severe illness and death have been approved since the end of 2020 [13], [14], [15], [16], [17], [18]. Nevertheless, as vaccine-induced serum antibody levels decline over time [19], [20], [21], an increasing incidence of breakthrough infections is reported amongst vaccinated individuals [22], [23], [24]. Therefore, regular screening measures to identify potentially infectious vaccinated individuals are of critical importance. Given the positive impact of immunization on viral clearance [25, 26], one might expect overall sensitivity of RADT to decline. However, the question whether RADT performance is indeed affected in vaccinated individuals remains unanswered. Here, we set out to address this question testing 696 individuals in a side-by-side comparison using RADT and RT-qPCR at our infectious diseases outpatient clinic.

2. Materials and methods

2.1. Study design

Routine SARS-CoV-2 testing by RT-qPCR was performed between the 06^th of December 2021 and 13^th of December 2021 in the outpatient clinic of the University Hospital Cologne, Department I of Internal Medicine, Cologne, Germany. Individuals from the general population with symptoms suggestive for COVID-19, or exposure to SARS-CoV-2, as well as hospital staff for routine screening were eligible for testing. For quality control of RADT performance in vaccinated breakthrough infections, two swab specimens, one from the oro-/nasopharynx for RT-qPCR and one from the nasopharynx for RAD testing, were collected by trained personnel after oral consent. Swabs for RT-qPCR were transferred into virus transport and preservation medium (biocomma®, Shenzhen, China). The on-site RADT was performed using the second swab. Data on vaccination and symptom status were retrospectively retrieved from patient charts and analyzed upon consent of the Institutional Review Board of the University of Cologne. Participants were considered fully vaccinated 14 days after one dose of Ad26.COV2.S or two doses of BNT162b2, mRNA-1273, and/or ChAdOx1-S, and 14 days after at least one dose of an approved vaccine (timepoint: 06^th December 2021) in case of a prior SARS-CoV-2 infection more than 6 months ago. For direct comparison of vaccinated and unvaccinated individuals a previous dataset from Korenkov and Poopalasingam et al. (sample collection: October 2020-January 2021) was used [10].

2.2. Rapid antigen detection test

The Standard Q COVID-19 Ag Test (SD Biosensor Inc., Suwon-si, Republic of Korea/Hoffmann La Roche AG, Basel, Switzerland) was performed according to the manufacturer's instructions. RADT with any visible test lines were considered positive, if the control line was also present.

2.3. Real time reverse transcription PCR

Different SARS-CoV-2 RT-qPCR protocols were used for detection of SARS-CoV-2. In brief, the following methods were used: (i) cobas® SARS-CoV-2 test kit running on the cobas 6800® (Roche Diagnostics, Mannheim, Germany) was used for 385 (55.32%) samples according to the manufacturer's instructions. (ii) SARS-CoV-2 AMP kit on the Alinity m (Abbott, Illinois, USA) was used for 62 (8.91%) samples. (iii) Multiplex RT-qPCR with LightMix® SarbecoV E-gene (TIB Molbiol, Berlin, Germany) running on the Panther Fusion® (Hologic, Wiesbaden, Germany) was used for 2 (0.29%) samples. (iv) For 244 (35.06%) specimens, samples from up to ten asymptomatic employees were pooled and tested for SARS-CoV-2 infection using the methods (i) and (ii). All samples within a negative pool were considered negative. For positive pools all samples were retested individually using the methods (i) and (ii). (v) Xpert® Xpress SARS-CoV-2 (Cepheid, Sunnyvale, USA) test kit was used for 3 (0.43%) samples according to the manufacturer's instructions. To enable comparison of Cycle threshold (Ct) values between different RT-qPCR protocols, Ct values were translated into copies/ml and converted to a cobas 6800®-adjusted Ct value. Detailed descriptions of the RT-qPCR protocols and Ct value conversion can be found in previous publications [10, 27].

2.4. SARS-CoV-2 Mutation Analysis

The presence or absence of spike gene mutations suggestive of the Delta or Omicron variants of SARS-CoV-2 was determined using the VirSNiP L452R and S371L/S373P Mutation Assays (TIB Molbiol). In this probe-based assay, single nucleotide mutations resulting in melting temperature differences are determined by melting curve analysis following PCR amplification. RNA for mutational analysis was extracted from Virus Transport Medium using the MagNA Pure 96 DNA and Viral NA Large Volume Kit (Hoffmann La Roche AG, Basel, Switzerland), and SARS-CoV-2 RNA PCR and subsequent melting curve analyses were performed on a LightCycler 480 (Hoffmann La Roche AG, Basel, Switzerland).

2.5. Statistical analysis

Sensitivity, specificity, as well as positive and negative predictive values were calculated using RT-qPCR results as a reference. Contingency tables were used to evaluate RADT result and vaccination status. The Wilson/Brown method was used to calculate confidence intervals (CI) and the Mann-Whitney U-test (MWU) was performed to compare differences between medians. Comparison of sensitivity curves was done based on the mean profiles of each curve. P-values <0.05 were considered significant. Data analysis was performed using Microsoft Excel 16.44 (Microsoft) and GraphPad Prism 9 (GraphPad Software, Inc.).

2.6. Ethics

The performed analyses in the manuscript were exempt from applying for ethical approval by the Institutional Review Board of the University of Cologne. Under german law no separate ethics application to and statement of ethical approval by the local ethics committee is required for performing purely retrospective clinical studies.

3. Results

3.1. RT-qPCR and RADT testing in a vaccinated cohort under real-life conditions

For the validation of RADT performance in vaccinated individuals, two consecutive swabs, one from the oro- and nasopharynx for RT-qPCR and one from the nasopharynx for the RADT, were collected at our outpatient clinic. A total of 696 samples were tested, of which 692 (99.4%) were collected from vaccinated individuals. At the time of sampling, 274 (39.4%) individuals reported symptoms (Fig. 1 A). 420 (60.3%) samples were obtained from female and 276 (39.7%) from male participants with an overall median age of 31 years (IQR: 26 - 41; Fig. 1 B). For SARS-CoV-2 RT-qPCR positive individuals, the median adjusted Ct value was 26.0 (IQR: 20.8 - 31.0; Fig. 1 C).

Fig. 1.

Cohort description. (A) Distribution of vaccination and symptom status among participants. (B) Age and gender distribution of the cohort. (C) Distribution of cycle threshold (Ct) values (adjusted to cobas 6800®).

3.2. Reliable detection of breakthrough infections with high viral RNA load by RADT

A SARS-CoV-2 breakthrough infection in vaccinated individuals (n=692) was detected in 76 (11.0%) cases by RT-qPCR. The L452R-mutation typical for the delta variant was detected in 53 (69.7%) and the S371L/S373P mutations typical for the omicron variant in 4 (5.3%) cases. In the remaining 19 (25.0%) samples, mutation analysis was not performed or not successful because of low viral RNA loads. The Standard Q COVID-19 Ag Test yielded a positive result in 45 (6.5%) vaccinated participants. 5 assays (0.7%) were classified invalid as the sample did not flow through sufficiently (Fig. 2 A). Paired samples of positive RADT had a median Ct of 21.0 (IQR: 18.5 - 25.0) by RT-qPCR while RT-qPCR-positive samples that were negative by RADT had a median Ct value of 32.0 (IQR: 28.0 - 34.0). For symptomatic individuals, the median Ct was 21.8 (IQR: 20.0 - 28.0) compared to 30.5 (IQR: 24.8 - 33.0) for asymptomatic individuals (Fig. 2 B). Using the RT-qPCR results as reference, 43 RADT results (6.3%) were true positive, 610 (88.8%) true negative, 32 (4.7%) false negative and 2 (0.3%) false positive. Overall RADT sensitivity and specificity were determined to be 57.3% (95%CI: 46.1 - 67.9) and 99.7% (95%CI: 98.8 - 99.9) for vaccinated compared to 42.9% (95%CI: 36.4 - 49.6) and 99.9% (95%CI: 99.6 – 100.0) in our previous unvaccinated cohort, respectively. Stratified by adjusted Ct values, sensitivities of the RADT were 100.0% (95%CI: 82.4-100.0) for Ct ≤ 20, 94.4% (95%CI: 81.9 - 99.0) for Ct ≤ 25, 81.1% (95%CI: 68.6-89.4) for Ct ≤ 30 and 0% (95%CI: 0.0 - 14.3) for Ct > 30 in vaccinated individuals. Statistical analysis of the curve progressions of RADT sensitivities plotted by Ct values and SARS-CoV-2-RNA loads based on the mean profiles of each curve showed a decrease in RADT performance in vaccinated individuals (p=0.0251 and p=0.4507, respectively) (Table 1 ; Figure 2 C). For symptomatic vaccinated individuals, RADT sensitivity and specificity were 71.1% (95%CI: 56.6 - 82.3) and 100.0% (95%CI: 98.3 – 100.0), respectively (Table 1), compared to 36.7% (95%CI: 21.9 - 54.5) and 99.5% (95%CI: 98.1% - 99.9%) in asymptomatic vaccinated individuals. Of note, all 4 probable Omicron variant cases with adjusted Ct values of 16, 18, 20 and 22 were detected by the Standard Q COVID-19 Ag Test. We conclude that the Standard Q COVID-19 Ag Test reliably detects SARS-CoV-2 breakthrough infections with high viral RNA loads. For vaccinated individuals with Ct values between 20 and 30, a decrease in RADT sensitivity was observed when compared to rapid antigen detection in unvaccinated individuals from a previous study (51.2% vs. 86.5%) [10].

Fig. 2.

SARS-CoV-2 detection by RT-qPCR and RADT in vaccinated individuals. (A) Proportion of positive and negative RT-qPCR and RADT results. (B) Distribution of adjusted Ct values and viral RNA loads in copies/ml by vaccination and symptom status (p<0.0001 and p=0.0005, respectively; MWU). RADT results are displayed in corresponding colors. (C) The sensitivity of the RADT is stratified by adjusted Ct values (p=0.0251) and viral RNA loads (p=0.4507) for vaccinated and unvaccinated participants. If the single sample with a Ct value of 24 (n=1; sensitivity 0%) is excluded, statistical analysis shows no significant difference in curve progression for adjusted Ct values (p=0.0912).

Tables 1.

Performance data of the Standard Q COVID-19 Ag test in vaccinated individuals.

Subgroups	Total no.	Sensitivity [% (95% CI)]
Overall	687	57.3 (46.1-67.9)a
Ct ≤ 20	18	100.0 (82.4-100.0)
Ct ≤ 25	36	94.4 (81.9-99.0)
Ct ≤ 30	53	81.1 (68.6-89.4)
Ct > 30	22	0.0 (0.0-14.9)
Symptomatic	271	71.1 (56.6-82.3)b
Asymptomatic	416	36.7 (21.9-54.5)c

^aPPV, 95.6% (95% CI = 85.2 to 99.2); NPV, 95.0% (95% CI = 93.1 to 96.5); Specificity, 99.7% (95% CI = 98.8 to 99.9).

^bPPV, 100.0% (95% CI = 89.3 to 100.0); NPV, 94.6% (95% CI = 90.9 to 96.8); Specificity, 100.0% (95% CI = 98.3 to 100.0).

^cPPV, 84.6% (95% CI = 57.8 to 97.3); NPV, 95.3% (95% CI = 92.8 to 97.0); Specificity, 99.5% (95% CI = 98.1 to 99.9).

4. Discussion

Since the approval of several COVID-19 vaccines by regulatory authorities [13], [14], [15], more than 10 billion doses have been administered to date [28, 29]. Due to the efficacious immune response in vaccinated individuals [16], [17], [18], discussions have emerged whether previously described performance data of RADT are still valid [9], [10], [11], [12]. Although vaccine-induced immunity has been associated with expedited viral clearance, analyses of viral kinetics have described peak viral RNA loads to be similar in vaccinated and unvaccinated individuals [25, [30], [31], [32]]. This suggests the performance of the lateral flow immunochromatographic assay to be unaffected by vaccination status in high viral RNA load samples.

To investigate the reliability of rapid SARS-CoV-2 antigen detection in vaccinated individuals, we obtained paired samples for RT-qPCR and on-site RADT, of which 687 were eligible for analysis. As identified in prior evaluations, the performance of tested RADT seems to mainly be dependent on the viral RNA load of the investigated samples [9], [10], [11], [12]. The observed difference in RADT performance between symptomatic and asymptomatic cases can, therefore, be explained by higher viral RNA loads in symptomatic individuals, which is consistent with RADT performance in unvaccinated subjects. Comparing collected data from vaccinated participants with our previous findings in an unvaccinated cohort [10], the Standard Q COVID-19 RADT shows a similar performance for samples with high viral RNA loads.

For samples with Ct values between 20 and 30 (i.e., medium viral RNA loads) the RADT showed a minor loss of power in vaccinated subjects. Besides the difference of vaccinated versus unvaccinated individuals, we cannot exclude other potential reasons for the detected difference including changed swab locations (nasopharynx and oro- and nasopharynx) in the different study settings, different predominantly circulating virus variants (D614G vs Delta), and the limited sample size of this sub-cohort with medium viral loads. However, despite slightly reduced sensitivity amongst vaccinated individuals, the negative predictive value (NPV) of the test was high with 95.0%.

The study is subject to some limitations including a limited sample size as stated above. For the evaluation of test performance in Omicron variant cases, larger investigations are needed. Additionally, for the comparison between vaccinated and unvaccinated subjects, data from unvaccinated individuals was used from a previous publication for which swabs were obtained from the oro- and nasopharynx. Retrieved data on symptoms are limited to symptom status and were not specified in terms of quality or duration. Also, this study was restricted to the Standard Q COVID-19 Ag test and may not be representative for other RADT kits.

In conclusion, rapid antigen testing reliably detected samples with high viral RNA loads including 4 samples of the newly emerged Omicron variant among vaccinated individuals. Although potentially less sensitive for samples with medium (20 - 30) Ct values in vaccinated than unvaccinated cases, one might still suspect that subjects with low viral RNA loads who had a negative RADT result present a reduced risk of SARS-CoV-2 spreading at the time of testing, as demonstrated for individuals post vaccination [35, 36]. Therefore, the RADT remains a valuable complementary tool in combating the pandemic by reliably detecting Delta breakthrough infections in vaccinated individuals that are of higher likelihood to transmit SARS-CoV-2.

Funding

This work was supported by the Bundesministerium für Bildung und Forschung (Federal Ministry of Education and Research): grant 01KX2021 („B-FAST“, „NaFoUniMedCovid19“).

Author contributions

Conceptualization: NP, MK, FK

Methodology: NP, MK, FK

Investigation: NP, MK, AA, JS, IF

Visualization: NP, MK

Funding acquisition: FK

Project administration: NP, MK

Supervision: MH, HG, CL, EH, FK

Writing – original draft: NP, MK

Writing – review & editing: HG, CL, FK

Data and materials availability

All raw data will be made available upon request.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.