SARS‐CoV‐2 rapid antigen test in comparison to RT‐PCR targeting different genes: A real‐life evaluation among unselected patients in a regional hospital of Italy

Abstract

We assessed the performance of the Panbio rapid antigen detection (RAD) test for the detection of severe acute respiratory syndrome coronavirus 2 (SARS‐CoV‐2) infection and we compared it with the routine reverse transcriptase‐polymerase chain reaction (RT‐PCR)‐based molecular test in a population of 4167 unselected patients admitted to IRCCS Sacro Cuore Don Calabria Hospital. Analysis stratified by cycling threshold (C _t) value of SARS‐CoV‐2 gene targets indicated that antigen (Ag)‐positive C _t values were significantly lower compared to Ag‐negative values (p < 0.0001). Overall, we found discordance in 140, tested negative by RAD and positive by RT‐PCR, and in 4 resulted positive by RAD and negative by RT‐PCR. RAD test achieved a sensitivity and specificity of 66.82% and 99.89%, respectively. The positive predictive value was shown to be 97.87% while the negative predictive value was shown to be 97.62%. In our context, the RAD test showed a reliable diagnostic response in subjects that displayed high C _t values, corresponding to high viral load, while low ability was displayed to identify positive cases with medium‐low C _t values, thus presenting low viral load and where confirmatory RT‐PCR was needed. Our finding supports the use of the RAD test in real‐life settings where a high volume of swabs is being processed but with caution when interpreting a positive test result in a low prevalence setting.

1. INTRODUCTION

Identification of people infected with severe acute respiratory syndrome coronavirus 2 (SARS‐CoV‐2) is an essential prerequisite for controlling the pandemic spreading. Reverse transcriptase‐polymerase chain reaction (RT‐PCR) molecular represents the gold standard for the diagnosis of viral infection, it is very sensitive and accurate and remains the reference method for diagnosing coronavirus disease 2019. However, nucleotide‐based testing of viral RNA is expensive time‐consuming, and required specialized laboratory settings, in terms of personnel and instrumentation.

Coronavirus rapid antigen detection (RAD) tests, with the appropriate application in the context of the pandemic, may contribute to the overall diagnostic capacity, offering benefits in terms of response times, and costs for the healthcare system, especially in situations in which the possibility of performing a molecular test on nasopharyngeal swab could be limited. ¹ The use of RAD can be recommended to test people, regardless of symptoms, when a high positive percentage is expected, for example, that approximates or exceed 10%. ² , ³ Despite the lower sensitivity (Sn) when compared with the molecular assays, the RAD is a potentially highly valuable test in terms of surveillance, to track and prevent the spread of infection. ² The antigen (Ag) tests, based on the immunochromatographic principle, essentially detect SARS CoV‐2 nucleocapsid protein (N), they are performed at or near the place where a specimen is collected, and they provide results onsite within few minutes. However, due to the different methods applied, RAD tests tend to be less sensitive than RT‐PCR tests, being more prone to false‐negative results, therefore every suspected case must be confirmed by a molecular test.

Information on the performance of RAD tests is limited and the Sn of first‐generation Ag is overall low. ³ European Center for Disease Prevention and Control (ECDC) agrees with the minimum requirements of accuracy established by the World Health Organization (WHO) for a RAD diagnostic test: ≥80% Sn, ≥97% specificity (Sp). ⁴ , ⁵ Therefore, different companies developed the RAD tests for second, third, and fourth generations to meet appropriate criteria established by WHO.

To examine the impact of a coronavirus disease 2019 (COVID‐19) RAD on a real‐world setting, we have evaluated the assay performance of a second‐generation Panbio™ COVID‐19 Ag Rapid Test and compared it with different RT‐PCR tests in a cohort of patients attending our hospital. To note that in the course of the study, a further third‐generation Ag assay was implemented (FREND COVID‐19 NanoEntek Inc.); however, the sample size was not considered large enough for comparative analysis.

2. MATERIALS AND METHODS

2.1. Study population

This retrospective study was performed on 4167 nasopharyngeal swabs of unselected patients, who were referred to IRCCS Sacro Cuore Don Calabria Hospital. The study period was November 28, 2020 to May 27, 2021. Inclusion criteria were the presence of a SARS‐CoV‐2 RAD result combined with an RT‐PCR test result. For each subject, two concomitantly nasopharyngeal swabs were collected; one swab was tested with the RAD test at the point of care, according to the manufacturer's instructions, whereas the other was processed for the routine SARS‐CoV‐2 RT‐PCR. Data were retrieved from our internal Laboratory Information Management System database and anonymized. Collected data includes the date of collection, age, sex, RAD, and RT‐PCR result.

2.2. SARS‐CoV‐2 testing

Panbio™ COVID‐19 RAD (Abbot Diagnostics Jena GmbH) was performed immediately after sampling, following the manufacturer's instructions. Nasopharyngeal swab specimens for SARS‐CoV‐2 RT‐PCR were analyzed within 24 h from the collection at the Department of Infectious, Tropical Diseases and Microbiology. Briefly, RNA was extracted from 200 µl of swabs transport medium using the automated Microlab Nimbus workstation (Hamilton) coupled to a Kingfisher Presto system (Thermo Fisher Scientific), according to the manufacturer's instructions. RT‐PCR was performed using three alternative methods: (i) the Bosphore SARS‐CoV‐2/Flu/RSV IVD panel (Anatolia Geneworks), targeting the Orf1ab, N, and the E gene; (ii) Real‐Time PCR SARS‐CoV‐2/Flu/RSV Panel Kit on a NeuMoDx™ molecular system (Qiagen) targeting N and Nsp2 gene; (iii) an in‐house direct quantitative RT‐PCR developed with the CDC 2019‐nCoV rRT‐PCR Diagnostic Panel assay (Integrated DNA Technologies, Inc.) using the PrimeDirect Probe RT‐PCR Mix (TaKaRaBio) on a CFX96 Touch Real‐Time PCR Detection System (Bio‐Rad). The latter assay targets the N1 and N2 gene regions.

2.3. Ethics statement

The study was conducted in accordance with the European Union general data protection regulation 2016/679 and good clinical practices. Ethical clearance was obtained from the competent ethics committees (Prot No. 44284/2021, Comitato Etico per la Sperimentazione clinica delle Provincie di Verona e Rovigo).

2.4. Statistical analysis

Descriptive statistics were generated to define the study population. Kruskal–Wallis one‐way analysis of variance (ANOVA) was applied for comparisons between three or more groups of gene targets. A p < 0.05 was defined as the level of significance. Agreement between RAD assay and RT‐PCR tests was assessed using Cohen's κ statistics. Graphpad (GraphPad Software) and MedCalc (MedCalc Software) were used to perform statistical analyses.

3. RESULTS

3.1. Demographic characteristics of the study population

For this study, we analyzed the data collected from 4167 nasopharyngeal swabs. The demographic characteristics for all patients studied are summarized in Table 1. There was no difference in the proportion of females compared to males (50.2% and 49.8%, respectively). The majority of the samples were from the Emergency Room, accounting for 65.19%.

TABLE 1.

Descriptive statistic of population study

Demographics	n	%
Population	4167
Gender
Female	2093	50.22
Male	2074	49.77
Age (years)	Female	Male
Minimum	0.20	0.3
25% Percentile	33.0	40.40
Mean	53.63	57.19
Standard deviation	24.42	22.86
Median	52.90	60.20
75% Percentile	76.05	76.10
Maximum	100.6	99.90
Department	n	%
Emergency Room	2679	64.29
Obstetrics Gynecology	527	12.64
Occupational Medicine Surveillance	509	12.21
Outpatients	149	3.57
Surgery	108	2.59
Oftalmology	56	1.34
Orthopedic	48	1.15
Urology	29	0.69
General medicine	13	0.31
Others	49	1.17

3.2. Diagnostic performance of RAD

To evaluate the “on‐field” diagnostic performance of the RAD test in the discriminating presence or absence of SARS‐CoV‐2, we calculated Sn, Sp, positive predictive value (PPV), and negative predictive value (NPV) in our cohort, using the RT‐PCR test as the gold standard. Out of 4167 total swabs, 422 (10.12%) were tested positive and 3745 (89.87%) were tested negative by RT‐PCR. Focusing on the 422 RT‐PCR positive samples, the RAD test was able to detect 282 (66.82%) as positives, missing 140 (33.18%) samples. The two tests resulted to be also discordant on negative cases, where four (0.10%) samples, resulted positive by RAD and negative by RT‐PCR (Table 2). RAD test achieved a Sn and Sp of 66.82% (95% confidence interval [CI], 62.11–71.30) and 99.89% (95% CI, 99.73–99.97), respectively (Table 2). Considering an estimated disease prevalence ranging from 2.88% to 8.84% (GIMBE Foundation; https://www.gimbe.org/), an average prevalence of 6.79% has been considered for PPV and NPV calculation. PPV was shown to be 97.87% (95% CI, 94.51–99.19) while the NPV was shown to be 97.62% (95% CI, 97.28–97.91). When the diagnostic performance was evaluated according to age group (<30, 30–60, and >60 years) RAD Sn and Sp seemed to be higher in the lowest age group 0–30 years (age group: Sn, Sp; <30: 74.36, 99.67; 30–60: 59.09, 99.93; >60: 72.12, 99.94). Finally, the comparison between our estimations, showed a substantial agreement of RAD (Cohen's κ, 0.77) when compared to RT‐PCR.

TABLE 2.

RAD test performance compared with reference standard RT‐PCR

	RT‐PCR		Total	Sensitivity% (95% CI)	Specificity% (95% CI)	PPV% (95% CI)a	NPV% (95% CI)a
Ag‐RDT	Positive	Negative
Positive	282 (6.09%)	4	286 (6.94%)	66.82 (62.11– 71.30)	99.89 (99.73– 99.97)	97.87 (94.51– 99.19)	97.62 (97.28– 97.91)
Negative	140	3741	3881 (93.13%)
Total	422 (10.12%)	3745 (89.87%)	4167
Cohen's κ statistics		0.77

Abbreviations: Ag‐RDT, antigen rapid diagnostic test; CI, confidence interval; NPV, negative predictive value; PPV, positive predictive value; RAD, rapid antigen detection; RT‐PCR, reverse transcriptase‐polymerase chain reaction

^a Estimated prevalence of 6.79% (GIMBE Foundation).

3.3. Distribution of cycling threshold values in RAD positive and negative swabs

We focused on the overall population with an RT‐PCR positive result, evaluating cycling threshold (C _t) values between RAD positive (+) and negative (−) specimens. Analysis on samples stratified by C _t value groups (<20, 20–30, 30–35, and >35) indicated that C _t values of RAD + samples were significantly lower compared to values of RAD‐ samples (mean diff ± SE, 9.30 ± 0.60, p < 0.0001) even though for the medium C _t an overlapping between RAD+ and RAD− samples was present (Figure 1A). Then, we evaluated the distribution of C _t values across the gene targets used in the SARS‐CoV‐2 diagnostic routine. Five different gene targets were used: N1, N2, Orf1ab/N, N, Nsp2. We explored these C _t distributions statistically by ANOVA analysis that reported significantly lower C _t values for N and Nsp2 (p < 0.0001) in RAD+ and RT‐PCR+, compared to the N1 and N2 targets (Figure 1B). In discordant results (RAD− and RT‐PCR+) solely Nsp2 showed the significative difference when compared with the N1 and N2 targets, despite relatively lower C _t values were mostly observed also with N target (Figure 1C). Subsequently, we compared the C _t values of SARS‐CoV‐2 targets in relation to the Ag test result. Our data demonstrated a mean C _t value for all targets, significantly lower in Ag‐positive compared to the Ag‐negative specimens (ANOVA, p < 0.0001) (Figure 2).

FIGURE 1.

(A) Distribution of C _t values by RAD tested positive and negative. Statistical analysis was performed by Student's t‐test (***p < 0.0001). (B, C) C _t values in RAD+/RT‐PCR+ and in RAD−/RT‐PCR+ specimens, respectively. Data are presented as box and whisker plots, showing median (horizontal line), boxes representing the 25th to 75th percentiles, whiskers representing minimum and maximum values. ***p < 0.001, **p < 0.001 was determined by one‐way ANOVA. ANOVA, analysis of variance; RAD, rapid antigen detection; RT‐PCR, reverse transcriptase‐polymerase chain reaction

FIGURE 2.

Comparison of C _t values of SARS‐CoV‐2 RT‐PCR gene targets according to RAD result. Each dot plot represents an individual C _t value. One‐way ANOVA was performed (***p < 0.0001). ANOVA, analysis of variance; RAD, rapid antigen detection; RT‐PCR, reverse transcriptase‐polymerase chain reaction; SARS‐CoV‐2, severe acute respiratory syndrome coronavirus 2

4. DISCUSSION

Since the start of the SARS‐CoV‐2 pandemic, the diagnostic ability to detect infected people in time manner has been crucial for the management of viral infection. RAD tests have significantly reduced delays in the test results, allowing a more rapid decision for clinical intervention and preventive measures, but they are not without potential risks regarding diagnostic test accuracy. In the present study, we have evaluated the diagnostic performance of Panbio™ COVID‐19 Ag Rapid Test with routine RT‐PCRs analyzing different SARS‐CoV‐2 genes, in a cohort of 4167 suspected subjects. The evaluation of the data showed some discrepancies between the molecular and the RAD test: 3.39% of the false‐negative results were observed for high C _t values, whereas we found concordance between RAD and RT‐PCR test at medium‐lower C _t, reflecting the ability of the RAD test to detect better high viral load in presumably symptomatic subjects. This was successfully established as described by Platten et al., ⁶ demonstrating that RAD is frequently negative in PCR positive samples with C _t values above 24–28. A further discrepancy was observed in 0.09% of swabs that resulted positive to the Ag test but negative by RT‐PCR. A partial reason for this discordance might be due to errors that might have affected the preanalytical phase (e.g., sample collection) or the data collection (e.g., subjective RAD reading).

Overall, the RAD test revealed a moderate Sn (66.82%) and good Sp (99.89%) in our study. The manufacturer reported a Sn of 98.1% (95% CI, 93.2%–99.8%) and a Sp of 99.8% (95% CI, 98.6%–100.0%) when performed in symptomatic subjects (n = 104), while when tested in asymptomatic subjects (n = 483), the Sn was lowered to 66.0% (95% CI, 51.2%–78.8%). Literature data are wide‐ranging in terms of overall accuracy for this test. Studies conducted on pure symptomatic patients reported an Sn varying from 87% to 71%, ⁷ , ⁸ , ⁹ whereas a study conducted on asymptomatic patients reported a very low Sn for this test (55.3%). ¹⁰ Our data, collected from a mixed population of symptomatic and asymptomatic subjects, is in line with the other two studies reporting an overall Sn of about 60%–70% in heterogeneous patients. ¹¹ , ¹² Anyhow, the results showed a lower Sn as compared to ECDC and WHO‐recommended Sn of an effective RAD. This observed moderate Sn might be due to a possible high proportion of asymptomatic subjects in our population since we did not collect data on symptoms. Another possible explanation could be linked to different testing times, performed in the early or late phase of infection. Because infection of SARS‐CoV‐2 occurs in a large proportion of the population with the asymptomatic presentation, precautions have to be posed when interpreting results of RAD test in these subjects.

Conversely, we found a high Sp near to 100%, demonstrating a low occurrence of false‐positive outcomes of Ag test. Similar Sp was also observed by other authors. ³ , ¹¹ , ¹² Diagnostic accuracy measures of a test are strictly dependent on the prevalence of the disease. For our study, the mean estimated prevalence of SARS‐CoV‐2 infection was 6.79% (GIMBE Foundation), thus the calculated NPV was 97.87% indicating that potentially 21.3/1000 persons could be falsely positive, whereas PPV was 97.62% revealing that potentially 23.8 infected on 1000 person could be missed. This leads to use the RAD test with more prudence when interpreting test results, especially in low prevalence settings, in which the increasing amount of false‐positive may cause detrimental social and economic consequences. A seminal contribution has been made by Kretschmer et al., ¹³ showing as false‐positive results may have a high economic burden in terms of direct and indirect costs arising from performed tests and subsequent quarantine. Finally, no substantial differences were observed between the different amplified genes and the results of the Ag test, confirming that the current genes used in molecular diagnostics are all efficient.

The main limitation of the present study is the lack of clinical data since our finding came from a large cohort of subjects tested irrespective of clinical presentation; therefore, we cannot correlate the Sn of the Ag test with the onset of symptoms. Secondarily, despite RAD tests were meticulously performed at the point of care following the manufacturer's instruction, many variables (e.g., subjectivity in reading) may have influenced the result of the test. Moreover, since the data reported here come from real daily activity, it was not possible to repeat the Ag test or the molecular tests for discordant cases and the RT‐PCR result was considered for the diagnosis. Collectively, notwithstanding these limitations, our finding supports the use of RAD test in high prevalence settings, where a high volume of swabs is being processed, and, hence, a first prompt result of infectivity is essential, improving clinical management and rapid effectiveness of isolation of positive cases, but with limited performance in low prevalence situations where less accuracy might have implications on epidemic management with the associated high economic burden. Recently, third‐generation Ag tests demonstrated excellent correlation with the RT‐PCR. ¹⁴ Recently our hospital adopted the FREND COVID‐19 third‐generation test. Although the number of executed tests is limited, our preliminary results, provided evidence of similar performances of second‐generation Ag tests (data not shown). However, this issue may constitute the object of future studies. Lately, genetic variants of the SARS‐CoV‐2 virus holding mutations in the N protein must be carefully monitored to evaluate the possible influence on RAD tests that use it as a target. ¹⁵

CONFLICT OF INTERESTS

The authors declare that there are no conflict of interests.

AUTHOR CONTRIBUTIONS

Davide Treggiari: writing – original draft and formal analysis. Chiara Piubelli and Francesca Perandin: writing – review and editing, and supervision. Sara Caldrer, Manuela Mistretta, Andrea Ragusa, Pierantonio Orza, Barbara Pajola, Donatella Piccoli, Antonio Conti, Carlo Lorenzi, Valentina Serafini, and Marco Boni: data curation and investigation. Francesca Perandin: conceptualization

Treggiari D, Piubelli C, Caldrer S, et al. SARS‐CoV‐2 rapid antigen test in comparison to RT‐PCR targeting different genes: a real‐life evaluation among unselected patients in a regional hospital of Italy. J Med Virol. 2022;94:1190‐1195. 10.1002/jmv.27378

DATA AVAILABILITY STATEMENT

The data that support the findings of this study are openly available in Zenodo at https://doi.org/10.5281/zenodo.5521462