Performance characteristics of five antigen-detecting rapid diagnostic test (Ag-RDT) for SARS-CoV-2 asymptomatic infection: a head-to-head benchmark comparison

Abstract

Background

Mass testing for early identification and isolation of infectious COVID-19 individuals is efficacious for reducing disease spread. Antigen-detecting rapid diagnostic tests (Ag-RDT) may be suitable for testing strategies; however, benchmark comparisons are scarce.

Methods

We used 286 nasopharyngeal specimens from unexposed asymptomatic individuals collected between December 2020 and January 2021 to assess five Ag-RDTs marketed by Abbott, Siemens, Roche Diagnostics, Lepu Medical, and Surescreen.

Results

For the overall sample, the performance parameters of Ag-RDTs were as follows: Abbott assay, sensitivity 38.6% (95%CI 29.1–48.8) and specificity 99.5% (97–100%); Siemens, sensitivity 51.5% (41.3–61.6) and specificity 98.4% (95.3–99.6); Roche, sensitivity 43.6% (33.7–53.8) and specificity 96.2% (92.4–98.5); Lepu, sensitivity 45.5% (35.6–55.8) and specificity 89.2% (83.8–93.3%); Surescreen, sensitivity 28.8% (20.2–38.6) and specificity 97.8% (94.5–99.4%). For specimens with cycle threshold (Ct) <30 in RT-qPCR, all Ag-RDT achieved a sensitivity ≥70%. The modelled negative- and positive-predictive value for 1% prevalence were >99% and <50%, respectively.

Conclusions

When screening unexposed asymptomatic individuals, two Ag-RDTs achieved sensitivity ≥80% for specimens with Ct<30 and specificity ≥96%. The estimated negative predictive value suggests the suitability of Ag-RDTs for mass screenings of SARS-CoV-2 infection in the general population.

Introduction

Mass testing for early identification and isolation of individuals infected with the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), irrespective of symptoms, is potentially an efficacious strategy to reduce disease transmission.¹ Recent advances on the validation of Antigen-detecting Rapid Diagnostic Tests (Ag-RDTs) show promise to replace central laboratory techniques for epidemiological control of the SARS-CoV-2 through mass testing.

Reverse transcription-polymerase chain reaction (RT-qPCR) is the current gold standard for identifying the presence of the SARS-CoV-2 in respiratory specimens.² More recently, transcription-mediated amplification (TMA) of the SARS-CoV-2 genome has been added to the repertoire of nucleic acid amplification tests (NAAT) for SARS-CoV-2 detection.³ Despite their high sensitivity, NAATs are associated with drawbacks that limit their use for community-based testing strategies, including the need for laboratory-processing, high cost, and long turnaround from sampling to results release. Furthermore, there is cumulative evidence indicating that the period of NAAT positivity in infected individuals largely exceeds the time window in which infectious viral particles can be isolated from the respiratory tract, raising doubts about the epidemiological meaning of a NAAT positive result.⁴

Ag-RDTs, commonly used in diagnosing other infectious diseases, have emerged as an alternative tool that meets the requirements for frequent testing at the point-of-care: rapid turnaround time, low cost, and ease-of-use.⁵ Overall, Ag-RDTs have lower sensitivity than NAATs; however, clinical validation studies have consistently reported increasing sensitivities in specimens with higher viral loads. These findings, along with the growing body of evidence on the lack of infectivity of cases with low viral load6, 7, 8, 9 and the potential long-tail of positivity when using highly sensitive methods such as PCR, suggest that frequent testing with Ag-RDTs―even those with low sensitivity―may be more effective than less frequent testing with RT-qPCR or TMA for mass screening campaigns to improve SARS-CoV-2 control.^{9 , 10}

The performance parameters of Ag-RDTs are mostly based on testing respiratory specimens from clinically suspected cases11, 12, 13, 14 and contacts after exposure to a positive case.15, 16, 17, 18 However, the sensitivity bias associated with the viral load leads to high heterogeneity in the reported performance parameters, which strongly depend on the disease status and potential exposure (e.g., symptomatic vs. asymptomatic, contact vs. unexposed) of tested individuals. This heterogeneity precludes comparative analyses between tests assessed in different studies and challenges benchmarking of Ag-RDTs. Furthermore, head-to-head comparisons are scarce, particularly in samples from asymptomatic individuals, the target population of community-based screening strategies.^{19 , 20} In this study, we used fresh nasopharyngeal samples collected in routine mass screening campaigns of unexposed asymptomatic individuals to perform a head-to-head comparison of five Ag-RDTs.

Methods

Study design

As part of the surveillance program for pandemic control in Catalonia (North-East Spain), the local government launched NAAT-based systematic screenings in areas at high risk of an outbreak. The University Hospital Germans Trias i Pujol processed nasopharyngeal specimens collected in a healthcare area in North-East Spain (i.e., Metropolità Nord) with a catchment population of ∼1,400,000 people. These samples enabled us to assess the Ag-RDTs in line with The Foundation for Innovative New Diagnostics (FIND) target product profile for lateral flow assays that directly detect antigens of SARS-CoV-2 antigen assays,²¹ which recommends at least 100 known negative samples and 100 known positive samples with a documented RT-PCR result. In this study, we used samples collected between December 2020 and January 2021 (i.e., during the third wave of the epidemic in Spain) with RT-qPCR results available (i.e., data on cycle threshold [Ct]) to perform a head-to-head assessment of five Ag-RDTs. Samples with invalid results in any of the assessed Ag-RDTs were excluded from the analysis.

All samples used in this analysis had been collected in the setting of a public health surveillance program, and data were handled according to the General Data Protection Regulation 2016/679 on data protection and privacy for all individuals within the European Union and the local regulatory framework regarding data protection. The study protocol was approved by the ethics committee of Hospital Germans Trias i Pujol (Badalona, Spain).

Procedures

Samples consisted of nasopharyngeal swabs collected by health care workers during mass testing of unexposed asymptomatic individuals living in areas at high risk of an outbreak. Swab specimens were placed into sterile tubes containing viral transport media (DeltaSwab Virus, Deltalab; or UTM Universal Transport Medium, Copan). The reference test (i.e., RT-qPCR) was performed on fresh samples stored at 2–8 °C for up to 24 h; samples were then stored up to 12 h at 2–8 °C until their use for the five Ag-RDTs.

RNA for RT-qPCR tests were extracted from fresh samples using the viral RNA/Pathogen Nucleic Acid Isolation kit for the Microlab Starlet or Nimbus platforms (Hamilton, USA), according to the manufacturer's instructions. PCR amplification was conducted according to the recommendations of the 2019-nCoV RT-qPCR Diagnostic Panel of the Centers for Disease Control and Prevention (CDC) (REF), using the Allplex™ 2019-nCoV assay (Seegene, South Korea) on the CFX96 (Bio-Rad, USA) in line with manufacturer's instruction. Briefly, a 25 μL PCR reaction mix was prepared that contained 8 μL of each sample's nucleic acids, 2019-nCoV positive and negative controls, 5 μL of 2019-nCoV MOM (primer and probe mix) and 2 μL of real-time one-step Enzyme. Thermal cycling was performed at the following conditions: 20 min at 50 °C for reverse transcription, followed by 15 min at 95 °C, and then 45 cycles of 15 s at 94 °C and 30 s at 58 °C. An RT-qPCR was considered positive according to the manufacturer's instructions.²²

Index tests included the following Ag-RDTs: PanBio^TM COVID-19 Ag Rapid test (Abbott), CLINITEST® Rapid COVID-19 Antigen Test (Siemens), SARS-CoV-2 Rapid Antigen Test (Roche Diagnostics), SARS-CoV-2 Antigen Rapid Test Kit (Lepu Medical), and COVID-19 Coronavirus Rapid Antigen Test Cassette (Surescreen). Supplementary Table 1 provides further details regarding the specifications of each test. All Ag-RDT determinations were performed in parallel by two blinded technicians, who used approximately 100 μL of 1:2 mix of each kit buffer and the sample previously homogenised. Samples were applied directly to the test cassette and incubated for 15 min at room temperature before reading results at the naked eye, according to the manufacturer instructions (i.e., the presence of any test line (T), no matter how faint, indicates a positive result).

Outcomes and statistical analysis

We calculated that a sample size of at least 73 positive specimens and 165 negative specimens would give 80% power to estimate overall sensitivity and specificity of Ag-RDT assays in our study. We based our calculation on the expected sensitivity and specificity in asymptomatic population of 65% and 96%,^{17 , 23} respectively, fixed precision of the point estimate of 2.5%, and confidence level of 95%. The calculation was in line with FIND recommendations for assessing Ag-RDTs that retrospective assessments should include a minimum of 100 samples per RT-PCR result.²¹

The primary analysis of the head-to-head comparison was the sensitivity and specificity of each Ag-RDT. Sensitivity and specificity were calculated as defined by Altman and Bland,²⁴ and reported as a percentage and the exact binomial 95% confidence interval (CI). Sensitivity was also analysed in a subset of samples with Ct<30, considered at high risk of transmission.

Secondary analyses were done assessing discordance between results obtained in each Ag-RDTs. Positive and negative-predictive values for each Ag-RDT at population prevalence between 1% and 15% for SARS-CoV-2 infection were modelled²⁵ and plotted with the exact binomial 95% CI.²⁶ All analyses and plots were performed using R version 3.6.²⁷

Role of the funding source

The funders of the study had no role in the study conception, design, conduct, data analysis, or writing of the report. All authors had full access to all the data in the study and had final responsibility for the decision to submit for publication.

Results

Our sample collection included 316 fresh nasopharyngeal swabs from unexposed asymptomatic individuals who had a RT-qPCR result available. Of these, 30 were excluded because of lack of documented Ct value (n=25), incomplete results due to limited sample volume (n=1), or invalid results in any of the Ag-RDTs (n=4, all of them in the Lepu assay), resulting in a study set of 286 samples: 101 (35.3%) with positive RT-qPCR result and 185 (64.7%) with negative RT-qPCR result (Fig. 1 ).

Fig. 1.

Flow-chart of sample inclusion.

All samples were nasopharyngeal swabs collected from unexposed asymptomatic individuals during screening campaigns.

The Ct value of samples with positive RT-qPCR result was <30 in 30 (29.7%) samples, 30-to-35 in 46 (45.5%), and >35 in 25 (24.8%). The overall sensitivity and specificity of the analysed Ag-RDTs ranged from 28.7% to 51.5% and 89.2% to 99.5%, respectively (Table 1 ). When considering only RT-qPCR positive samples with Ct <30 (i.e., indicates a high concentration of viral genetic material which is typically associated with a higher risk of infectivity),²⁸ the sensitivity of Ag-RDTs increased to 76.7% (95% CI 57.7–90.7) for the Abbott assay; 86.7% (69.3–96.3) for the Siemens Assay; 83.3% (65.3–94.4) for the Roche assay; 83.3% (65.3–94.4) for the Lepu assay; and to 70% (50.6–85.3%) for the Surescreen assay (Fig. 2 ).

Table 1.

Sensitivity and specificity of the antigen-detecting rapid diagnostic tests for SARS-CoV-2.

	Abbott	Siemens	Roche	Lepu	Surescreen
Overall sensitivity	38•61% (29•09–48•82)	51•49% (41•33–61•55)	43•56% (33•72–53•8)	45•54% (35•6–55•76)	28•71% (20•15–38•57)
Detected	39	52	44	46	29
Not detected	62	49	57	55	72
Total PCR+	101	101	101	101	101
Sensitivity in specimens with Ct<30	76•67% (57•72–90•07)	86•67% (69•28–96•24)	83•33% (65•28–94•36)	83•33% (65•28–94•36)	70% (50•6–85•27)
Detected	23	26	25	25	21
Not detected	7	4	5	5	9
Total PCR+	30	30	30	30	30
Specificity	99•46% (97•03–99•99)	98•38% (95•33–99•66)	96•22% (92•36–98•47)	89•19% (83•8–93•27)	97•84% (94•56–99•41)
Detected	1	3	7	20	4
Not detected	184	182	178	165	181
Total PCR-	185	185	185	185	185

All samples were nasopharyngeal swabs collected from unexposed asymptomatic individuals during mass screening campaigns. Sensitivity and specificity results are presented with the 95% confidence interval.

Fig. 2.

Sensitivity of the antigen-detecting rapid diagnostic tests according to the cycle threshold value of the RT-qPCR analysis.

Bars show the 95% confidence interval of the estimated sensitivity.

Of the 286 samples analysed by Ag-RDTs, 222 (77.6%) had concordant results across all Ag-RDT assessed. The 29 samples with concordant positive results across Ag-RDTs were all PCR-positive. Conversely, 37 (19.2%) of 193 specimens with negative results in all Ag-RDTs were PCR positive. Fig. 3 shows the distribution of Ag-RDT results in samples with discordant results. The Ag-RDT that most often yielded a positive result in samples with negative results in all other Ag-RDTs was the Lepu assay (n=23; 35.9%), followed by the Siemens assay (n=10, 15.6%). Table S2 summarises the cycle threshold distribution across discordances.

Fig. 3.

Discordance analysis between Ag-RDTs.

Bars show the number of samples for each discordance pattern. Black dots and grey dots indicate the assays showing positive and negative results in each discordance pattern. Table S2 summarizes the cycle threshold distribution across discordances.

To provide an estimate of misidentified cases―either false-positive or false-negative cases―that can be used for making decisions in the public health setting, we modelled the positive and negative predictive value for a prevalence range consistent with a mass screening of unexposed asymptomatic individuals (Fig. 4 A). For the overall study sample, the estimated positive predictive value (PPV) at a 1% prevalence ranged from 4.1% to 41.9%, with the Lepu assay and the Abbott assay, respectively (Table S3). The estimated PPVs notably increased for the <30 Ct subgroup of samples (Fig. 4B), and when prevalence in the population was higher. The estimated negative predictive value (NPV) at 1% prevalence ranged from 99.3% to 99.5%, with the Surescreen assay and the Siemens assay, respectively.

Fig. 4.

Positive predictive value and negative predictive value according to pre-test probabilities.

A: overall sample (n= 286). B: samples with cycle threshold <30 in the RT-qPCR assay. Table S3 provides detailed values and confidence intervals for predicted false negative and false positives in the investigated prevalence.

Discussion

In this study, we compared head-to-head the sensitivity and specificity of five Ag-RDTs to screen SARS-CoV-2 infected individuals with unknown exposure and no clinical suspicion of COVID-19. Four of the tested Ag-RDTs (i.e., Abbott, Siemens, Roche, and Surescreen assays) showed a specificity higher than 96%. Regarding sensitivity, despite it was low for the overall sample (range 29–51%), the corresponding values for the subset of samples with a RT-qPCR value Ct <30 were higher than 80% for the Siemens, Roche, and Lepu assays. This finding is of particular interest for the proposed use of Ag-RDT as a reliable alternative to RT-qPCR for the rapid detection of individuals with higher risk of infectivity in mass screening of asymptomatic individuals. Pre-clinical studies have persistently reported a very low infectious capacity of respiratory specimens with viral loads below 10⁶ genome copies/mL, which usually correspond to a Ct of approximately 29–31.^{5 , 8 , 29} These findings align with the significant increase of the secondary attack rate for values of Ct <30,³⁰ indicating higher infectiousness among individuals with viral loads below this Ct threshold.

Although sensitivity and specificity are important intrinsic characteristics of a test, the number of expected errors when using the test for screening purposes strongly depends on the prevalence of the infection in the screened sample. Hence, positive and negative predictive values are a mainstay for making public health decisions regarding the use of a test. The reported prevalence of SARS-CoV-2 infection in PCR-based untargeted screenings of the general population typically ranges between 1% and 3%, depending on the virus transmission context.^{23 , 31} In low prevalence settings, Ag-RDTs will have a high NPV but a low PPV. According to our estimate, the NPV for SARS-CoV-2 infections at 1% prevalence was higher than 99% for all test, suggesting that a negative test may not require confirmation. In contrast, the PPV at 1% prevalence was lower than 50% in all tests, suggesting that a positive result will need immediate confirmation by RT-qPCR, even for highly specific assays.

Our study has several strengths and limitations. We used the same fresh set of samples for assessing five different Ag-RDTs and the sample size met the FIND recommendation for retrospective assessments of the clinical performance of these tests. Furthermore, to our knowledge, this is the first head-to-head comparison of Ag-RDT in asymptomatic screenings, an intended use proposed by various authors.^{5 , 10 , 17 , 23} On the other hand, our study was limited by the small number of specimens with Ct <30, a threshold deemed of interest for the use of Ag-RDT in screenings of the general population. In our sample, specimens below this threshold accounted for 30%; however, other authors have reported proportions of nearly 60% in random screenings of the general population.²³ Of note, we used specimens in transport medium. This approach is convenient for mass screening strategies in which individuals with positive Ag-RDT results may need further diagnostic confirmation by PCR. However, only one manufacturer (i.e., the Roche assay) provided instructions on how to process samples collected in virus transport medium. The consistency of our results across assays, particularly regarding negative results, suggests that the use of this media had a little or negligible impact on test performance. Finally, it is worth mentioning that all nasopharyngeal swabs in our analysis were collected by trained healthcare professionals. According to a recent report of lateral flow viral antigen detection devices, the positivity rate might be lower in screenings performed by non-trained people.⁹

Our results provide policymakers with evidence on the use of Ag-RDT for mass screening of unexposed, asymptomatic individuals. Two commercial, widely available assays can be used for SARS-CoV-2 antigen testing to achieve sensitivity in specimens with a Ct<30 and specificity of at least 80% and 96%, respectively. While these tests may overlook SARS-CoV-2 infection with low viral loads, they accurately detect individuals with high viral loads and, therefore, at higher risk of transmission. Our findings also support the idea that Ag-RDTs can be used for mass screening in low prevalence settings and accurately rule out a highly infectious case in such setting. In models according to population prevalence, all Ag-RDTs will have a NPV >99% and a PPV<50% at 1% prevalence. Our results, together with the cumulative evidence on the limited overlapping between PCR positivity and the presence of infectious viral particles in the respiratory tract, encourage the design of public health interventions for containing viral COVID-19 spread that shift from positivity testing to infectivity testing. The low cost and short turnaround time of Ag-RDTs, which ease frequent testing, are additional advantages over assays better suited for diagnostic use like NAATs. In low-income countries with limited laboratory resources, the trade-off between targeted PCR analyses and massive screenings with Ag-RDTs should be carefully considered.

CRediT authorship contribution statement

Bàrbara Baro: Conceptualization, Methodology, Validation, Data curation, Writing – original draft. Pau Rodo: Methodology, Validation, Data curation. Dan Ouchi: Validation, Formal analysis. Antoni E. Bordoy: Data curation. Emilio N. Saya Amaro: Methodology, Data curation. Sergi V. Salsench: Methodology, Data curation. Sònia Molinos: . Andrea Alemany: Data curation. Maria Ubals: Data curation. Marc Corbacho-Monné: Data curation. Pere Millat-Martinez: Writing – review & editing, Data curation. Michael Marks: Writing – review & editing. Bonaventura Clotet: Funding acquisition, Writing – review & editing. Nuria Prat: Writing – review & editing. Oriol Estrada: Funding acquisition, Resources. Marc Vilar: Funding acquisition, Resources. Jordi Ara: Funding acquisition, Writing – review & editing. Martí Vall-Mayans: Writing – review & editing. Camila G-Beiras: Writing – review & editing. Quique Bassat: Funding acquisition, Writing – review & editing. Ignacio Blanco: Conceptualization, Funding acquisition, Writing – review & editing. Oriol Mitjà: Conceptualization, Writing – original draft, Funding acquisition.

Declaration of Competing Interest

We declare no conflicts of interest.