A Real-World Comparison of SARS-CoV-2 Rapid Antigen Testing versus PCR Testing in Florida

ABSTRACT

The reported sensitivity of rapid, antigen-based diagnostics for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection varies. Few studies have evaluated rapid antigen tests in real-world settings or among large populations. Beginning October 2020, Florida offered individuals presenting for SARS-CoV-2 testing PCR testing if they tested positive by the Abbott BinaxNOW COVID-19 antigen (Ag) card, were symptomatic, or required or requested PCR testing. We compared results among individuals who received both types of tests at four publicly accessible testing sites across Florida. We calculated the positive percent agreement (PPA) between the two test types by symptom status. Subsequently, we evaluated the PPA among individuals regardless of symptoms with lower cycle threshold values (<30). Overall, 18,457 individuals were tested via both methods, of which 3,153 (17.1%) were positive by PCR. The PPA for the Abbott BinaxNOW COVID-19 Ag card using the PCR comparator was 49.2% (95% confidence interval [CI], 47.4% to 50.9%). Among symptomatic individuals the PPA was 51.9% (95% CI, 49.7% to 54.0%). When restricted to positive PCR tests with a cycle threshold value of <30, regardless of symptom status, the PPA was 75.3% (95% CI, 72.8% to 77.6%). The PPA of the Abbott BinaxNOW COVID-19 Ag card compared with PCR was lower than that previously reported. Our findings may reflect the performance of the BinaxNOW antigen test in real-world settings.

INTRODUCTION

The ability of infected individuals to transmit severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) in the absence of symptoms (1), means that infection surveillance efforts are critical for controlling the pandemic. Surveillance efforts currently predominantly use real-time PCR assays, which can be expensive, require significant laboratory infrastructure, and take 24 to 48 h to return results. Rapid antigen-based detection methods have been proposed as an alternative strategy, facilitating more-rapid test turnaround times and reduced costs; however, results of studies evaluating the sensitivities of such assays vary considerably (2–5).

In August 2020, the Food and Drug Administration issued an emergency use authorization for the Abbott BinaxNOW COVID-19 antigen (Ag) card, a rapid antigen assay with a reported sensitivity of 97.1% among symptomatic individuals within 7 days of exposure (2). In a follow-up community-based study among symptomatic and asymptomatic individuals, the sensitivity of that same assay was 93.3% when limited to individuals with a PCR cycle threshold value of less than 30 (indicating a high viral load), inclusive of asymptomatic individuals (n = 95) (3), though both studies were limited by sample size. Subsequent work has identified similarly high sensitivity of the BinaxNOW COVID-19 Ag card, particularly when limited to individuals with lower cycle threshold values (6). However, evaluation of the performance of the Abbott BinaxNOW COVID-19 Ag card among larger sample sizes in real-world settings is warranted in order to further understand the test performance.

Some experts argue that adequate infection surveillance depends more on test turnaround time and test frequency than on test sensitivity (7, 8). However, without large-scale validation of an assay’s sensitivity, the true utility of such rapid diagnostics remains uncertain. Thus, we aimed to compare the Abbott BinaxNOW COVID-19 Ag card with PCR among a population who received both tests in Florida.

MATERIALS AND METHODS

Between October and November 2020, the State of Florida offered individuals presenting for testing the option of the Abbott BinaxNOW COVID-19 Ag card or both the antigen card and standard Food and Drug Administration (FDA) emergency use-authorized PCR (9) on the same day for a subset of individuals (symptomatic individuals, individuals who tested positive on the BinaxNOW COVID-19 Ag card, individuals who required PCR testing for work or travel, or individuals who specifically requested PCR testing). Results of both tests were available to patients. We analyzed results from all individuals who were tested with both types of tests as well as the cycle threshold values for each positive PCR test from four counties: Leon, Miami-Dade, Lee, and Broward.

We compared test performances by calculating the positive and negative percent agreement (PPA and NPA, respectively) among individuals asymptomatic and symptomatic at the time of testing. Subsequently, we evaluated the PPA among individuals regardless of symptoms but with lower cycle threshold values (<30, as was previously suggested) (3, 10).

Specimens that were collected for PCR testing included health care worker-observed self-collected oral fluid swabs, self-collected anterior nares swabs, and clinician-collected nasopharyngeal swabs. Specimens tested using PCR were transported in RNA preservative medium (DNA/RNA Shield solution; Zymo Research Corp., Irvine, CA) to a large, commercial high-complexity Clinical Laboratory Improvement Amendments (CLIA)-compliant laboratory for testing in Washington, DC. Testing was conducted with a modified FDA-authorized Centers for Disease Control and Prevention protocol, as previously reported (11). A cycle threshold value of 30 in the specific PCR assay that was used corresponded to approximately 3,000 viral copies per ml (range 1,500 to 6,000 copies per ml) of Zymo solution. The laboratory PCR protocol included a cycle threshold value cutoff of ≥40.0 for negative results; cycle threshold values of ≥36.0 and less than 40.0 triggered automatic repeat testing of the specimen with repeated extraction from the original sample. Anterior nares specimens were collected via flocked swabs (Copan Diagnostics, Murrieta, CA), while posterior nasopharyngeal specimens were collected with a synthetic swab (Becton Dickinson and Company, Franklin Lakes, NJ).

Specimen collection for the Abbott BinaxNOW COVID-19 Ag card was performed across all four field sites, in tents, pods, and personal vehicles, and was overseen by the Florida Department of Emergency Management. Swabs were collected either by trained certified nursing assistants or by health care worker-observed self-collection. All sites were instructed to store BinaxNOW COVID-19 Ag cards in accordance with the package insert instructions (2), with quality control and validation performed on each new kit (one kit contained 40 tests) prior to use. Per the instructions from the Florida Department of Emergency Management, all BinaxNOW swabs were collected prior to the collection of the PCR swab. Instructions for specimen processing were standardized across all sites, with swabs immediately placed onto the BinaxNOW cards, which were then timestamped and tracked for quality control.

The Mass General Brigham institutional review board deemed that the analysis of deidentified data did not constitute human subjects research (2020P003530).

RESULTS

During the study period, 18,457 individuals were tested via Ag and PCR. Of those, 3,153 (17.1%) were positive by PCR, 958 (30.4%) of whom reported no symptoms at the time of testing. Symptom status was unknown among 88 (2.8%) of those that were PCR positive. PCR testing was performed on 16,709 oral fluid specimens, on 975 anterior nares specimens, and on 773 nasopharyngeal specimens. The median cycle threshold value among oral fluid specimens that tested positive by PCR was 31.9 (interquartile range, 27.1 to 35.1). The median cycle threshold value was higher among asymptomatic individuals, at 32.6 (interquartile range, 28.1 to 35.3), while among symptomatic individuals, the median cycle threshold value was 31.5 (interquartile range, 26.5 to 35.0).

Overall, the PPA for the Abbott BinaxNOW COVID-19 Ag card with PCR was 49.2% (95% confidence interval [CI], 47.4% to 50.9%). The PPA was 51.9% (95% CI, 49.7% to 54.0%) among symptomatic individuals. Using oral fluid specimens as the comparator for PCR testing regardless of symptom status, the PPA for the Abbott BinaxNOW COVID-19 Ag card was 49.4% (95% CI, 47.5% to 51.2%), while among anterior nares specimens, the PPA was 47.5% (95% CI, 39.1% to 56.1%); among nasopharyngeal swab specimens, the PPA was 46.1% (95% CI, 37.3% to 55.1%). When comparing among tests with cycle threshold values of <30, regardless of symptom status or specimen type, the PPA was 75.3% (95% CI, 72.8% to 77.6%). Table 1 shows the PPA among each specimen type by symptom status and overall, while the Fig. 1 shows the cycle threshold values and BinaxNOW COVID-19 Ag card results for the tests that were positive by PCR.

TABLE 1.

Positive and negative percent agreement between the Abbott BinaxNOW COVID-19 Ag card and PCR by symptom status from a community-based sample in Florida, 2020

FIG 1.

Abbott BinaxNOW COVID-19 Ag card results by SARS-CoV-2 viral cycle threshold value from a community-based sample in Florida, 2020.

DISCUSSION

Our study among a large community-based sample in Florida compared the Abbott BinaxNOW COVID-19 Ag card with PCR to evaluate the positive percent agreement. We found a lower frequency of SARS-CoV-2 antigen detection by the Abbott BinaxNOW COVID-19 Ag card than what was reported previously (2, 3). Though the agreement between the tests was improved when restricting the analysis to positive PCR tests with lower cycle threshold values, the agreement was still lower than in previous studies (2, 3). Part of the explanation for such a difference may be that our results constitute a real-world comparison in a large sample versus performance estimates from carefully selected often enriched clinical study populations or smaller real-world studies.

Another explanation for the difference in our findings may be how the testing was performed. While all sites were given similar instructions, a post hoc analysis revealed that the PPA from one site was somewhat higher (61% from Lee County) than the other sites (49% from Miami-Dade County, 48% from Broward County, and 49% from Leon County, P = 0.06). That finding may also be explained by differences in viral load distribution among individuals at the various testing sites, with individuals presenting to Lee County, for example, presenting more recently after infection. Further, there may have been some variation in adherence to recommended specimen collection and testing methods. It is possible, for example, that swab collection varied by testing facility (either by a trained clinician or health care worker-observed self-collection or by the duration or vigor of nasal swabbing, etc.) in a manner that impacted test positivity. A recent study showed that the sensitivity of the Abbott BinaxNOW COVID-19 Ag card may be reduced when swabs are self-collected compared to when swabs are collected by trained clinic staff (12). However, we did not collect data on how compliant sites were with collection and testing instructions or know which specific specimens were self-collected or clincian-collected.

Furthermore, there were a high proportion of results with cycle threshold values greater than 30 among those who tested positive. The distribution of cycle threshold values in the testing population may impact test performance (13). Additionally, a small proportion of those tests that were positive by PCR with high cycle threshold values may have been false-positive results. Finally, worth considering are variants of concern (14). An in vitro study concluded that four different rapid SARS-CoV-2 antigen tests performed reliably with two genotypic variants (15). A real-world evaluation of the performance of rapid antigen tests against SARS-CoV-2 variants is warranted; however, we do not suspect the presence of variants in our population explains the low PPAs.

Thus, questions remain about the optimal uses of such rapid tests. While increased frequency of testing certainly appears to be an important strategy for infection surveillance (7), an assay which fails to detect up to one-quarter of the cases with high viral loads may not be sufficient. Single use of such assays, such as at the entrance to indoor venues or events, will likely fail to detect a sufficient number of infectious cases to be a useful prevention strategy. However, repeated testing over time in the same population using rapid antigen testing may prove to be a cost-effective method (8, 16); further research is needed.

An additionally important consideration is the specificity of the test. Our results revealed a high negative percent agreement (98.8%), similar to that previously reported (2, 3). A recent study, however, noted that even small imperfections in testing specificity may result in a high number of false-positive tests when implemented on a national scale in populations with a low frequency of true infection (17), overwhelming testing infrastructure, and contact notification systems. Thus, false-positive results create major challenges and consume already limited resources in public health systems. Confirmation of rapid posttest results with PCR should be considered.

One strategy to mitigate such a negative impact would be to target specific hot spots of localized spread, as SARS-CoV-2 appears to spread heterogeneously throughout the population (18).

In all, our results provide important contextualization of the performance of a single rapid antigen-based SARS-CoV-2 test. The promise of infection surveillance using less-sensitive but significantly more-rapid assays is great; however, the sensitivity of the current iteration of such assays may be too low to use in a single-test strategy. Further work is needed to investigate the utility of rapid antigen tests in other settings and at different testing frequencies as well as the development of assays with improved sensitivity.

Our study had several limitations. All sample collection was done within the same state; thus, the generalizability to populations in other states may be limited. Additionally, the notable proportion of individuals whose symptom status at the time of testing was unknown makes conclusions about the overall performance less certain. To address that uncertainty, the results are presented by symptom status (symptomatic, asymptomatic, and unknown). The presence or absence of symptoms likely reflects differences in viral load at the time of testing and thus might impact the performance of the antigen test. Finally, we had little clinical and exposure data and thus could not analyze the impact of factors such as recency of exposures, severity of symptoms, and others that may impact testing results. Overall, however, we feel those limitations do not negate the importance of our findings.

Conclusions.

We demonstrated that the positive percent agreement of the Abbott BinaxNOW COVID-19 Ag card compared with PCR was lower than among previous reports. Furthermore, the impact of cycle threshold values, and thus viral load, on the positive percent agreement was less than previously reported. Further research is needed into optimal uses of rapid SARS-CoV-2 Ag testing in public health prevention efforts.