Analytical performances of the point-of-care SIENNA™ COVID-19 Antigen Rapid Test for the detection of SARS-CoV-2 nucleocapsid protein in nasopharyngeal swabs: A prospective evaluation during the COVID-19 second wave in France

Abstract

Objectives

We herein assessed the analytical performances of the antigen-rapid diagnostic test (Ag-RDT) SIENNA™ COVID-19 Antigen Rapid Test Cassette (Nasopharyngeal Swab) (Salofa Oy, Salo, Finland), targeting the SARS-CoV-2 N nucleocapsid protein, for the diagnosis of COVID-19 in hospitalized patients with suspected SARS-CoV-2 infection, by reference to real-time RT-PCR (rRT-PCR).

Methods

Nasopharyngeal swabs were collected from patients with COVID-19-like illness during the second epidemic wave in Paris, France, among which 100 and 50 were positive and negative for SARS-CoV-2 RNA, respectively.

Results

Overall, the Ag-RDT showed high sensitivity, specificity, positive and negative predictive values of 90.0%, 100.0%, 100.0% and 98.1%, respectively, as well as high or almost perfect agreement (93.3%), reliability assessed by Cohen’s κ coefficient (0.86), and accuracy assessed by Youden’s J index (90%) to detect SARS-CoV-2. The analytical performances of the Ag-RDT remained high in the event of significant viral excretion (i.e., N gene C_t values ≤33 by reference rtRT-PCR), while the sensitivity of the Ag-RDT dropped to 69.6% with low or very low viral shedding (C_t > 33).

Conclusions

The SIENNA™ Ag-RDT presents excellent analytical performances for viral loads ≤33 C_t, classically corresponding to situations of symptomatic COVID-19 and/or proven contagiousness.

Introduction

The coronavirus disease 2019 (COVID-19) pandemic continues to spread across the world. Hence, there is an urgent need for rapid, simple, and accurate tests to diagnose severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection. While currently recommended nucleic acid amplification tests (NAAT), such as real-time reverse transcription polymerase chain reaction (rtRT-PCR) assays, remain the gold standard cornerstone for the diagnosis of SARS-CoV-2 infection (Smithgall et al., 2020, Rai et al., 2021), immunological methods can also be used to detect viral antigens (Dinnes et al., 2020, Li and Li, 2020, Rai et al., 2021). Indeed, performing rtRT-PCR is expensive, time consuming, and requires special equipment and qualified operators. Faster, cheaper, and easier to use alternative tools could be represented by novel point-of-care antigen-detecting rapid diagnostic tests (Ag-RDT) (Dinnes et al., 2020). Ag-RDT relies on direct detection of SARS-CoV-2 viral proteins produced by replicating virus in nasal swabs and other respiratory secretions, often the virus N nucleocapsid protein, preferred because of its relative abundance and conserved structure, or other viral proteins such as the spike protein (Li and Li, 2020). Most Ag-RDTs use sandwich catching by anti-SARS-CoV-2 monoclonal antibodies to detect viral antigens in the simple-to-use lateral flow immunoassay format allowing results in <30 min. Around 150 Ag-RDTs for SARS-CoV-2 infection are now commercially available or in development (FindDx, 2021). However, there is significant variability reported with respect to their diagnostic performances and a lack of external validation for many of the available tests, which still require clinical validation (Dinnes et al., 2020, Mattiuzzi et al., 2020, Favresse et al., 2021, Fitzpatrick et al., 2021, Schildgen et al., 2021).

The aim of our study was to evaluate the Ag-RDT SIENNA™ COVID-19 Antigen Rapid Test Cassette (Nasopharyngeal Swab) (Salofa Oy, Salo, Finland; manufactured under license of T&D Diagnostics Canada Pvt. Ltd., Halifax, Canada), for the diagnosis of COVID-19 in hospitalized patients with suspect SARS-CoV-2 infection during the second wave of the COVID-19 epidemic in Paris, France. The results of this test were compared with qualitative and quantitative results obtained in parallel using rtRT-PCR as reference test.

Material and methods

Antigen test

The Ag-RDT SIENNA™ COVID-19 Antigen Rapid Test Cassette (Nasopharyngeal Swab) is designed for the qualitative detection of SARS-CoV-2 N nucleocapside protein in nasopharyngeal secretions through monoclonal antibodies. The test was performed according to the manufacturer's instructions by mixing nasopharyngeal secretions with 300 μL of the extraction buffer in a tube. Then, three drops (# 80 mL) were added to the appropriate well. When nasopharyngeal secretions cross the strip, passive diffusion allows the solubilized conjugate to migrate with the sample and react with the anti-SARS-CoV-2 antibodies immobilized on the membrane. A control line allows the correct migration of the sample and the reliability of the test to be assessed. According to the manufacturer’s “Instruction for use” (IFU) (reference 102,241), visual interpretation of the results starts 10 min up to 20 min after deposition. Samples identified as positive at both the control line and test line are regarded as SARS-CoV-2 antigen-positive, and samples having only the control line are regarded as SARS-CoV-2 antigen-negative. All necessary reagents to perform the assay are provided by the manufacturer and no assay-specific, specialized equipment is needed. According to the IFU, the assay kits are stable when stored at 2–30 °C.

Clinical specimens and procedures

The analytical performances of the Ag-RDT SIENNA™ COVID-19 Antigen Rapid Test Cassette (Nasopharyngeal Swab) were retrospectively evaluated on a biological collection of archived nasopharyngeal swabs frozen at −80 °C, including 100 positive and 50 negative RNA swabs from SARS-CoV-2 by reference multiplex rtRT-PCR (Allplex™ 2019-nCoV Assay, Seegene, Seoul, Korea), a reliable molecular assay for the diagnosis of SARS-CoV-2 infection (Farfour et al., 2020). All assays were used according to manufacturers’ recommendations. Visual interpretation of the study Ag-RDT result was performed after 15 min.

Statistical analyses

Data were entered into an Excel database and analyzed using IBM® SPSS® Statistics 20 software (IBM, SPSS Inc., Armonk, New York, USA). Medians were calculated for quantitative variables. The results were presented along with their 95% confidence interval (CI) using the Wilson score bounds for categorical variables (Newcombe, 1998). The results of SARS-CoV-2 RNA detection by the multiplex rRT-PCR were used as the reference standard to estimate the sensitivity and specificity of the study Ag-RDT, with corresponding 95% CI. The concordance between study Ag-RDT and multiplex molecular detection of SARS-CoV-2 RNA was assessed by percent agreement corresponding to the observed proportion of identical results between Ag-RDT compared to rRT-PCR detection. The reliability between the study Ag-RDT and the multiplex molecular detection of SARS-CoV-2 RNA was estimated by Cohen’s κ coefficient (Cohen, 1960), and the degree of agreement was determined as ranked by Landlis and Koch (1977). The accuracy of the study Ag-RDT to correctly diagnose SARS-CoV-2 infection was estimated by Youden’s J index (J = sensitivity + specificity − 1) (Youden, 1950). Positive predictive values (PPV) and negative predictive values (NPV) were calculated according to the Bayes formulae, by considering the official reported prevalence of SARS-CoV-2-RNA positivity in symptomatic patients in the Paris area, France, on 17th November 2020, e.g. near the peak of the second wave epidemic in France (Santé publique France, 2020; https://www.santepubliquefrance.fr/).

Ethics statement

The study was used as a clinical evaluation of the continuous quality improvement program and COVID-19 management measures performance evaluation, according to the national law on the accreditation of medical biology laboratories (Journal Officiel de la République Française, 2010), providing an exemption from informed consent application, according to the French public health code (Code de la Santé Publique, article L 1121-1.1; https://www.legifrance.gouv.fr/). The dataset was completely anonymous and did not contain any identifiable personal health information.

Results

The vast majority [(84/90 (93.3%)] of positive results appeared within the first five minutes, and frequently [(29/90 (32.3%)] within one minute. The analytical results are shown in Table 1 . Overall, the Ag-RDT SIENNA™ COVID-19 Antigen Rapid Test Cassette (Nasopharyngeal Swab) showed high sensitivity, specificity, PPV and NPV of 90.0%, 100.0%, 100.0% and 98.1%, respectively (Table 1), as well as high or almost perfect agreement (93.3%), reliability assessed by Cohen’s κ coefficient (0.86), and accuracy assessed by Youden’s J index (90%) to detect SARS-CoV-2.

Table 1.

Analytical performances of the SIENNA™ COVID-19 SARS-CoV-2 Antigen Rapid Test Cassette (Nasopharyngeal Swab) rapid diagnostic test for the qualitative detection of the N protein of SARS-CoV-2 using 100 positive and 50 negative nasopharyngeal swab samples by reference rRT-PCRⁱ, according to their N gene C_t values.

				SIENNA™ COVID-19 Antigen Rapid Test Cassettea
		N gene Ct by rRT-PCR (median; range)	N	TP (n)	FN (n)	Sensitivityb(% [95% CI])g	Specificityb(% [95% CI])	Agreementc	Concordanced	Youden’s J indexe	PPVf(% [95% CI])	NPVf(% [95% CI])
N gene Cth of positive samples by reference rRT-PCRi	≤20	19 (9-20)	25	25	0	100.0 [99.9–100.0]	100.0 [99.9–100.0]	100.0	1.0	1.0	100.0 [99.9–100.0]	100.0 [99.9–100.0]
	21–33	25 (21–33)	52	49	3	94.2 [87.9–100.0]	100.0 [99.9–100.0]	97.0	0.94	0.94	100.0 [99.9–100.0]	98.9 [96.1–100.0]
	>33	35 (34–37)	23	16	7	69.6 [50.7–88.0]	100.0 [99.9–100.0]	90.4	0.76	0.70	92.1 [86.0 –95.7]	94.4 [88.5–100.0]
	All Ct values	24 (9–37)	100	90	10	90.0 [84.1–95.9]	100.0 [99.9–100.0]	93.3	0.86	0.90	100.0 [99.9–100.0]	98.1 [94.6–100.0]

C_t: cycle threshold; FN: false negative; FP: false positive; NPV: negative predictive value; PPV: positive predictive value; rRT-PCR: real-time reverse transcription-polymerase chain reaction; TP: true positive; TN: true negative.

^aNasopharyngeal samples were collected with a flocked swab, then discarded in 1 ml of physiological saline, and further stored frozen at −80 °C before reuse;

^bAll 50 swab samples negative by reference rRT-PCR were found negative by the test SIENNA™ COVID-19 Antigen Rapid Test Cassette (Nasopharyngeal Swab); the values of TN and FN were 50 and 0, respectively;

^cAgreement = TP + TN/TP + FP + TN + FN, expressed in percentage;

^dThe Cohen’s k coefficient calculation was used to estimate the concordance (Cohen, 1960) and interpreted according the Landis and Koch scale (Landlis and Koch, 1977), as follows: <0 as indicating no agreement, 0–0.20 as slight, 0.21–0.40 as fair, 0.41–0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1 as almost perfect concordance;

^eThe accuracy of the test SIENNA™ COVID-19 Antigen Rapid Test Cassette (Nasopharyngeal Swab) to correctly diagnose SARS-CoV-2 infection was estimated by Youden’s J index (J = sensitivity + specificity − 1) (Youden, 1950);

^fPPV and NPV were calculated according to the Bayes’s formulae, by taking into account the official reported prevalence of SARS-CoV-2-RNA positivity in COVID-19-suspected patients in Paris’s area, France, of 16.2% on 17th November 2020 [Santé publique France, 2020; https://www.santepubliquefrance.fr/];

^g95% confidence intervals in brackets were calculated by using the Wilson score bounds;

^hThe Ct values of N gene detection by the reference Seegene rRT-PCR were used to classify nasopharyngeal samples according to their level of SARS-CoV-2 RNA excretion; C_t of 20 and 33 were taken as thresholds of very high and high SARS-CoV-2 RNA excretion, respectively, as previously stated (Centers for Disease Control and Prevention, 2020b; Jefferson et al., 2020; Société Française de Microbiologie, 2020; Yu et al., 2020);

ⁱThe CE IVD-marked Allplex™ 2019-nCoV Assay (Seegene, Séoul, Korée) constituted the reference multiplex rRT-PCR for SARS-CoV-2 RNA detection. This assay detects three target genes of SARS-CoV-2 (E, RdRP and N genes) (Farfour et al., 2020).

These analytical performances were further stratified according to the cycle threshold (C_t) values of the N gene detected by reference rRT-PCR considering C_t-related criteria of very high (C_t ≤ 20) and high (C_t ≤ 33) SARS-CoV-2 RNA excretion. Indeed, viral loads with C_t > 33 are considered to be low and correspond to moderate or very low viral excretion (Centers for Disease Control and Prevention, 2020b; Jefferson et al., 2020; Société Française de Microbiologie, 2020; Yu et al., 2020). Conversely, samples with C_t ≤ 33 have a significant SARS-CoV-2 viral load, as it is the case in individuals symptomatic for COVID-19 or contagious. C_t values ≤20 indicate very high viral shedding (Jefferson et al., 2020, Yu et al., 2020).

There were two distinct situations. In the event of significant viral loads (high or very high) in real-time PCR (C_t ≤ 33), the Ag-RDT SIENNA™ COVID-19 Antigen Rapid Test Cassette (Nasopharyngeal Swab) showed excellent analytical performances, with sensitivities between 97.9% and 100.0%, specificities between 99.9% and 100.0% (no false positive results observed), PPV ​​between 99.9% and 100.0% and NPV ​​between 96.1% and 100.0%. In the event of low or very low viral loads (C_t > 33), the sensitivity of the RDT SIENNA™ COVID-19 Antigen Rapid Test Cassette (Nasopharyngeal Swab) showed reduced analytical performances with 69.6% sensitivity, while its specificity remained high (>99.0%).

Discussion

We herein evaluated the analytical performances of the novel point-of-care Ag-RDT SIENNA™ COVID-19 Antigen Rapid Test Cassette (Nasopharyngeal Swab) by reference to multiplex rRT-PCR for SARS-CoV-2 RNA detection as gold standard in a real-life community setting. In the event of significant viral excretion (i.e., N gene C_t values below 33 by reference rRT-PCR), the study Ag-RDT showed high sensitivity (≥94.0%) and specificity (≥99.0%) for SARS-CoV-2 RNA detection, with excellent concordance, reliability and accuracy with the reference multiplex rRT-PCR, and PPVs and NPVs above 98.0%. The sensitivity of the study Ag-RDT dropped to 69.6% with low or very low viral shedding (C_t > 33). Taken together, these observations demonstrate that the study Ag-RDT harbored excellent analytical performances, which makes it suitable to be used as point-of-care Ag-RDT in various hospital and non-hospital settings where rapid diagnosis of SARS-CoV-2 is necessary.

The study Ag-RDT fulfilled the current World Health Organization (WHO)’s recommendations for a screening Ag-RTD stating that, at minimum, Ag-RDTs would need to correctly identify significantly more cases than they would miss (sensitivity ≥80%) and would have very high specificity (≥97–100%) (World Health Organization, 2020a). Furthermore, analytical performances of comparable order as those of our study Ag-RDT were previously reported for some Ag-RDTs in lateral flow immunoassay format (Cerutti et al., 2020, Chaimayo et al., 2020, Diao et al., 2021, Favresse et al., 2021, Linares et al., 2020, Schildgen et al., 2021, Toptan et al., 2020, Weitzel et al., 2020), while several studies have reported much lower sensitivity levels contrasting with always high specificity (Albert et al., 2020, Dinnes et al., 2020, Scohy et al., 2020, Yamayoshi et al., 2020, Osterman et al., 2021, Torres et al., 2021).

These large variations in sensitivity according to the studies led us to analyze our results according to the estimated viral load in SARS-CoV-2 in the samples. In our series, we have stratified the nasopharyngeal samples according to the level of viral excretion, indirectly evaluated by the value of the C_t of the N gene according to the reference rRT-PCR, in order to calculate the performance of the study Ag-RDT at different proposed cut-offs for contagiousness. Our results clearly show that the analytical performances of the study Ag-RDT were much better in the event of a high viral load, i.e., in the case of significant viral excretion. These observations demonstrate the interest of the study Ag-RDT as a rapid rule-in test for COVID-19 with samples at high viral load, in symptomatic patients for example, and point to caution with its use as a singular rule-out test especially in the setting of samples with lower viral loads. The SARS-CoV-2 RNA positive subpopulation of our clinical samples collection was characterized by a wide range of C_t-values with medium and low C_t-values dominating. This allowed the calculation of sensitivity and specificity values with higher relevance for clinical practice. The C_t-dependent evaluation showed a very good sensitivity for highly and moderately SARS-CoV-2 positive samples (C_t ≤ 33). In contrast, the sensitivity of the assay with specimens containing only a limited viral load was low. Thus, the study Ag-RDT for SARS-CoV-2 antigen detection may have a limited suitability for the determination of the SARS-CoV-2 infection status of patients. COVID-19 infection would not be detected in patients in the early or late phase of the infection typically associated with a low viral load. However, differentiation between contagious and non-contagious individuals may be possible with this assay. Samples with C_t-values >33 usually do not allow culturing of the virus indicating low infectivity (Jefferson et al., 2020, La Scola et al., 2020). Such individuals may be regarded as non-contagious despite carrying low virus loads (Zou et al., 2020). This differentiation of individuals may be of particular importance for the decision on access to susceptible individuals, for example in nursing homes or in many other medical circumstances. Similar observations of dramatic decrease of sensitivity of Ag-RDT for SARS-CoV-2 antigen detection at C_t thresholds around 25–33 were previously reported (Yamayoshi et al., 2020, Favresse et al., 2021, Krüttgen et al., 2021), confirming that Ag-RDTs were most effective to identify RT-PCR positive symptomatic patients or asymptomatic subjects with high viral loads in their respiratory secretions (i.e., C_t values ≤33).

Our results on the evaluation of the analytical performances of the SIENNA™ Ag-RDT confirm the differential interest of Ag-RDT for the detection of SARS-CoV-2 antigens depending on the level of viral load of the sample analyzed, underlined by several authors (Guglielmi, 2020, Mattiuzzi et al., 2020, Mina et al., 2020), and international (World Health Organization, 2020a) and national (Food and Drug Administration, 2020, Journal Officiel de la République Française, 2020) recommendations.

Firstly, Ag-RDT may be used to detect SARS-CoV-2 infected symptomatic individuals suffering from COVID-19-like symptoms with high viral loads and has potential in determining highly contagious individuals (Food and Drug Administration, 2020, Journal Officiel de la République Française, 2020, World Health Organization, 2020a). According to the last WHO ad interim guidance for SARS-CoV-2 antigen testing (World Health Organization, 2020a), these Ag-RDTs could be used preferentially where reference molecular assays are unavailable or laboratory services are overloaded, and shall be specifically used in settings where molecular testing is not immediately available (World Health Organization, 2020a). For example, in France, nurses, pharmacists, and general practitioners have been authorized in October 2020 to perform Ag-RDT for SARS-CoV-2 antigen detection in patients suspected of COVID-19 infection in medical settings (Journal Officiel de la République Française, 2020), especially when RT-PCR will be difficult to carry out or delayed. In emergency settings, Ag-RDTs could be useful to quickly recognize patients with COVID-19-like illness for first care and isolation from negative patients, in order to avoid SARS-CoV-2 nosocomial infection (Möckel et al., 2021). When the pretest probability for receiving positive test results for SARS-CoV-2 is elevated (e.g., in symptomatic individuals or in persons with a known COVID-19 exposure during contact tracing), a negative antigen test result should be confirmed by NAAT (Centers for Disease Control and Prevention, 2020a, Centers for Disease Control and Prevention, 2020b).

Secondly, data are lacking on Ag-RDT performance in asymptomatic persons to inform expanded screening testing to rapidly identify and isolate infected persons (Centers for Disease Control and Prevention, 2020b). The lower diagnostic performances of Ag-RDTs (especially the higher false negative rate) may leave many asymptomatic or mildly symptomatic COVID-19 patients undiagnosed and not isolated from the community (Mattiuzzi et al., 2020). Nonetheless, it is important to consider that the real infectivity and efficiency of viral transmission in asymptomatic patients with low SARS-CoV-2 viral load, as detected by NAATs but not with Ag-RDT (e.g., cycle thresholds >30–33), is still a matter of open debate (Guglielmi, 2020, Mattiuzzi et al., 2020). Briefly, asymptomatic or pre-symptomatic SARS-CoV-2 infected individuals are considered to be significantly less contagious than symptomatic patients (Buitrago-Garcia et al., 2020), and not to be very infective, especially during the later stage of their infection (Cheng et al., 2020, Walsh et al., 2020), with lower viral inoculation associated with a much higher likelihood of developing only mildly symptomatic or even totally asymptomatic illness (Zhang et al., 2020). Then, adult patients with a false negative Ag-RDT for SARS-CoV-2 antigen detection and symptom onset at least one week earlier typically have a low SARS-CoV-2 RNA concentration and likely are past the infectious period (Mattiuzzi et al., 2020). Finally, despite their relatively low analytic sensitivity, Ag-RDTs for SARS-CoV-2 antigen detection can be beneficial even for detecting infected individuals who are asymptomatic, pre-symptomatic and without known or suspected exposure to SARS-CoV-2 (Mina et al., 2020), in various congregate settings like a long-term care facility or correctional facility, workplace, school, or airport (Shaimoldina and Xie, 2020).

Thirdly, population screening testing to identify infectious individuals presents one possible means to break enough transmission chains to suppress the ongoing pandemic and reopen societies, with or without a vaccine. The faster turnaround time of the Ag-RDT can help limit transmission by more rapidly identifying infectious persons for isolation, particularly when used as a component of serial testing strategies (Larremore et al., 2020).

Otherwise, Ag-RDTs for SARS-CoV-2 antigen detection likely would also be useful in low- and middle-income countries and are recommended by the WHO (Jacobs et al., 2020, Ndwandwe et al., 2020, World Health Organization, 2020b).

This study has several limitations. The date of symptoms onset was unknown. Furthermore, the study included only symptomatic patients with suspected SARS-CoV-2 infection, and therefore asymptomatic patients with a priori lower viral load were not evaluated. In addition, the interpretation of analytical performances for high C_t values ​​must take into account thawing of samples and additional dilution prior to sample deposition, which makes interpretation of results much less reliable at these low viral loads. Finally, the possibility of false-negative results exists, especially if the infectious viral load is low. Thus, our study was retrospective and conducted on frozen samples, which can lead to selection and sample quality bias, such as inadequate sample integrity of virus targets because of multiple freeze–thaw steps of sample processing (Yu et al., 2017, Brukner et al., 2020).

In conclusion, the study Ag-RDT presents excellent analytical performances for viral loads ≤33 C_t, classically corresponding to situations of symptomatic COVID-19 and/or proven contagiousness, which allows in practice the recommendation of the SIENNA™ Ag-RDT for routine clinical use in most diagnostic indications of SARS-CoV-2 infection. The test can also be offered to test asymptomatic individuals in many situations of potential exposure to SARS-CoV-2 and in mass screening at the population level.

Contributions

RSMB, HP, DV and LB have conceived and designed the research; RSMB performed the experiments and statistical analyses; HP, RSMB, HP, DV and LB analyzed the results and drafted the manuscript.

Declaration of interest

The authors report no conflicts of interest. The authors alone are responsible for the content and the writing of the paper.

Funding

No grant was received from the test manufacturers.