Performance of High Throughput SARS-CoV-2 Antigen Testing Compared to Nucleic Acid Testing

Abstract

Objective

Independent assessment of SARS-CoV-2 antigen (COV2Ag) tests remains important as varying performance between assays is common. We assessed the performance of a new high-throughput COV2Ag test compared to SARS-CoV-2 nucleic acid amplification tests (NAAT).

Methods

A total of 347 nasopharyngeal samples collected from January to October 2021 were assessed by NAAT as part of standard-of-care testing (CDC LDT or GeneXpert System, Cepheid) and COV2Ag using the ADVIA Centaur CoV2Ag assay (Siemens Healthineers).

Results

Among NAAT positive specimens we found 82.4% agreement and in NAAT negative specimens we found 97.3% agreement (overall agreement 85.6%). In symptomatic persons, COV2Ag agreed with NAAT 90.0% (n = 291), and in asymptomatic persons, 62.5% (n = 56). Agreement between positive NAAT and COV2Ag increased at lower cycle threshold (Ct) values.

Conclusion

The COV2Ag assay exceeded the World Health Organization minimum performance requirements of ≥ 80% sensitivity and ≥ 97% specificity. The COV2Ag assay is helpful for large scale screening efforts due to high-throughput and reduced wait times.

Consistent emergence of new variants of concern in the COVID-19 pandemic requires clinical laboratories to continue SARS-CoV-2 clinical testing even in the face of widespread vaccine availability in many countries. Nucleic acid amplification tests (NAAT) remain the most sensitive testing clinically available for the detection of SARS-CoV-2 infection. The SARS-CoV-2 antigen (COV2Ag) tests are critical in supporting efforts to identify infections and control the transmission of the virus. They are generally more portable, faster, easier to perform, and tend to detect infection in people most likely to have transmittible infection.^1,2 These COV2Ag tests are an important component of a comprehensive mitigation strategy for SARS-CoV-2 spread as they offer the ability to quickly screen people participating in activities with high transmissibility risk, protect immune compromised populations, and mitigate risk in health care settings.^1–4 High-throughput COV2Ag tests allow maximization of resources for many of these risk mitigation strategies.^1,5

Independent assessment of SARS-CoV-2 antigen tests remains important as varying sensitivity and specificity between assays is common and standard regulatory assessments have not been possible in the face of the COVID-19 pandemic.^1,2,5 Additionally, clinical laboratory-based COV2Ag tests allow for better analytic control and improved sensitivity and specificity.⁵ We leveraged banked samples to assess the performance and understand the limitations of a new high-throughput COV2Ag test compared to SARS-CoV-2 nucleic acid tests in a large urban hospital system.

Materials and Methods

Cohort

Remnant nasal and nasopharyngeal swabs in viral transport media following NAAT for SARS-CoV-2 were stored at −80°C until use (the longest storage time was 1 year). The NAAT-positive samples were collected from 3 time periods representing distinct variants of SARS-CoV-2 as determined in our patient population: January 1 to 20, 2021 (pre-alpha variant predominance; n = 136; pre-alpha), April 30 to June 7, 2021 (alpha variant B.1.1.7 predominance; n = 35; alpha), and September 14 to October 24, 2021 (delta variant predominance; n = 101; delta). Negative cases were collected from December 5 to 9, 2021 (n = 75) as negative cases were not banked as part of our institutional protocol. The SARS-CoV-2 NAAT was performed as part of standard-of-care testing on the CDC LDT as previously described⁶ or the GeneXpert System using either the Xpert Xpress SARS-CoV-2 or SARS-CoV-2/Flu A/Flu B/RSV (Cepheid, Sunnyvale, CA, USA). In cases where more than 1 cycle threshold (Ct) value was produced by the NAAT, the lowest value was considered for comparison purposes.

Minimum necessary clinical variables were obtained via chart review. The electronic medical record at our institution includes a required field for SARS-CoV-2 testing purpose. This field was used to distinguish symptomatic from asymptomatic as many patients did not have sufficient documentation to further stratify by disease severity, outcomes, or specific symptomatology. Available choices for this SARS-CoV-2 testing purpose field include preprocedural, test required for facility transfer, symptomatic patient, test required prior to inpatient hospice, testing required prior to returning to work, close contact with COVID-19 patient, and unknown. Demographic and applicable disease information are summarized in TABLE 1. This work was performed under the auspices of the University of Pittsburgh Institutional Review Board Study #20040220.

TABLE 1.

Demographics and clinical characteristics

	Total Cohort	NAAT Positive	NAAT Negative
Total number (%)	347 (100%)	272 (78%)	75 (22%)
Female, n (%)	207 (60%)	172 (63%)	35 (47%) a
Male, n (%)	139 (40%)	100 (37%)	39 (52%) a
Age median (IQR)	51 (32–66)	47 (31–62)	64 (49–73)
Symptomatic, n (%)	291 (84%)	232 (85%)	59 (79%)
Asymptomatic, n (%)	56 (16%)	40 (15%)	16 (21%)
Transfers, n (%)	42 (12%)	31 (11%)	11 (15%)
pre-alpha, n (%)	136 (39%)	136 (50%)	NA
alpha, n (%)	35 (10%)	35 (13%)	NA
delta, n (%)	101 (29%)	101 (37%)	NA

NA, not applicable.

^a One NAAT Negative subject was of unknown gender.

Testing

All testing was performed in the College of American Pathologists-accredited University of Pittsburgh Medical Center Clinical Laboratories in compliance with local regulations for patient testing. Specimens were thawed, aliquoted, and neutralized per manufacturer instructions. Briefly, 1 mL of sample was incubated with 2 drops of lysis reagent for 15 minutes before storage at −20°C for 3 weeks prior to testing.

The ADVIA Centaur CoV2Ag assay is an automated sandwich immunoassay that uses mouse monoclonal antibodies to detect SARS-CoV-2 nucleocapsid antigen and provides an index value with a threshold of > 1.0 being reactive. All samples were thawed and tested on the same day. Calibration and quality control materials were within manufacturer’s specifications. Precision was verified using 10 replicates of standard material at 2 levels with a resulting coefficient of variation of 30.1% (mean 0.150) for negative and 0.6% (mean 204.46) for high materials.

Analysis

Data was collected and collated using Excel (Microsoft Corporation, Redmond, WA, USA), and R Studio 2021.09.0 Build 351 with R version 4.1.1. Analyses were performed in Excel, R Studio, and GraphPad Prism 7 (GraphPad Software, San Diego, CA, USA). Subgroup comparisons were analyzed using the Mann-Whitney nonparametric rank sum test. Box-and-whisker plots are plotted using the Tukey method.

Results

We found that overall COV2Ag had 85.6% agreement with NAAT (n = 347; 95% confidence interval [CI] 81.5–89.1). Agreement between COV2Ag and NAAT among time periods sampled had overlapping CIs. Specifically, pre-alpha 82.4%(95%CI 74.9–88.3), alpha 65.7% (95%CI 47.8–80.9), and delta 88.1% (95%CI 80.2–93.7). Among NAAT positive patients we found 82.4% agreement (n = 272; 95%CI 77.3–86.7) and in NAAT negative patients, we found 97.3% agreement (n = 75; 90.7–99.7). In the 2 COV2Ag-positive and NAAT-negative specimens, the COV2Ag index values were < 1.4 (positive threshold > 0.99).

Among symptomatic subjects, COV2Ag agreed with NAAT 90.0% (n = 291; 95%CI 86.0–93.2), and among asymptomatic subjects, there was 62.5% (n = 56; 95%CI 48.5–75.1) agreement. The Ct values for NAAT positive samples were significantly different between symptomatic and asymptomatic subjects, with mean values of 22.6 (n = 232) and 31.3 (n = 40), respectively (P = 9.03E-08, FIGURE 1A). Agreement between positive NAAT and COV2Ag increased at lower Ct values (FIGURE 1B). There was 90.8% agreement for Ct values < 35 (n = 240; 95%CI 86.5–94.2), 98.1% agreement for < 30 cycles (n = 213; 95%CI 95.3–99.5), and 100% agreement for < 25 cycles (n = 177; 95%CI 97.9–100). Concordance between positive NAAT and COV2Ag was low at higher Ct values, with 25.4% agreement for Ct values > 30 cycles (n = 59, 95% CI 15.0–38.4) and 19.4% agreement for Ct values > 35 cycles (n = 31, 95% CI 7.5–37.5).

FIGURE 1.

Comparison of SARS-CoV-2 nucleic acid amplification test Ct values between cohort subcategories. A, NAAT positive symptomatic cases (n = 232) compared to asymptomatic cases (n = 40) Ct values. Tukey box and whisker plot. Mann-Whitney P < .0001. B, NAAT-positive Ct values for discordant (n = 48) and concordant (n = 224) samples between NAAT and COV2Ag. Tukey box and whisker plot. Mann-Whitney P < .001. C, Cohort parsed into days from symptom onset and percent agreement and NAAT median Ct values compared. Bars indicate percent agreement between NAAT and COV2Ag, left y-axis; line indicates NAAT-positive median Ct value, right y-axis. n(95%CI) for days −1 to 1 through 8 to 10, respectively: 103(84.1–95.9), 89(79.0–93.7), 61(81.9–97.3), 24(78.9–99.9), 10(34.8–93.3).

Concordance between COV2Ag and NAAT, when assessed for days from symptom onset, demonstrated insignificant variation between days with overlapping confidence intervals (FIGURE 1C). Median Ct values among NAAT positive symptomatic patients had little variation between days −1 (1 patient tested positive before a procedure with symptom onset the following day) and day 7. The increase after day 7 may be due to the small sample number collected after day 7, as reflected in the wide confidence interval (FIGURE 1C).

Discussion

We found that the COV2Ag assay exceeded the WHO minimum performance requirements of ≥ 80% sensitivity and ≥ 97% specificity for SARS-CoV-2 antigen assays.⁷ The overlapping confidence intervals between variant collection times indicated reasonably comparable detection between variants, likely due to the multiple monoclonal antibody nature of the assay. The data also reassured us that at −80°C, the banked samples were stable for over 1 year. The alpha portion of our cohort was underrepresented with approximately one-third the sample size of the pre-alpha and delta collection time periods. This is reflected in the large confidence interval, and there may be differences in detection between these variants that our study was not sufficiently powered to observe.

As with other groups, we found that antigen assay agreement with NAAT was higher at lower Ct values.^2,4,5,8 There is ongoing debate around the use of high Ct values in SARS-COV-2 NAAT to indicate current SARS-COV-2 positivity because high Ct values can persist for months following SARS-COV-2 infection, potentially reflecting detection of viral RNA but not infectious viral particles.^9–12 We found no significant differences in percent agreement between COV2Ag and NAAT in symptomatic patients when assessing by days from symptom onset. This is likely a reflection of the similar median Ct values for each of these groups.

The COV2Ag assay has been considered helpful for large scale screening efforts due to high-throughput and reduced wait times as well as for procedural screening prior to patient surgery or care facility transfers, which benefit from faster turnaround times to free hospital space. For screening efforts in asymptomatic persons, it is notable that agreement with NAAT is reduced in our and other’s studies,^5,13 which may reflect the higher Ct values noted in these cases (FIGURE 1A). Ultimate implementation with appropriate utility should be determined by local laboratory and clinical medical directors.

Glossary

Abbreviations

COV2Ag

SARS-CoV-2 antigen

NAAT

nucleic acid amplification tests

Ct

cycle threshold

CI

confidence interval

Funding

Siemens Healthineers provided some funding for this work.

Conflict of Interest

SW and OPP have received speaking honoraria from Siemens Healthineers.