Diagnostic performance of CerTest and Panbio antigen rapid diagnostic tests to diagnose SARS-CoV-2 infection

Abstract

Objectives

Antigen rapid diagnostic tests (Ag-RDT) have been developed as reliable tools to control the SARS-CoV-2 pandemic. The objective of our study was to evaluate the diagnostic performance of two Ag-RDTs.

Methods

We evaluated CerTest SARS-CoV-2 Ag One Step Card Test and Panbio COVID-19 Ag Rapid Test Device Ag-RDTs. We included 320 nasopharyngeal samples: 150 PCR negative samples to assess the specificity and 170 PCR positive samples to evaluate the sensitivity. We also evaluated their sensitivity according to cycle threshold (Ct) values and the time from the onset of symptoms. Tests were compared using the McNemar’s test and agreement was evaluated using the kappa score (k).

Results

Both Ag-RDTs showed a specificity of 100 %. Overall sensitivity was 53.5 % for CerTest and 60.0 % for Panbio. For samples with $\text{C}\text{t}\le$ 25, sensitivity was 94.0 % for CerTest and 96.4 % for Panbio (p = 0.500). Regarding samples with Ct>25, sensitivity was 14.0 % for CerTest and 24.4 % for Panbio (p = 0.004). Sensitivity for samples within the first 5 days after the onset of symptoms were 84.8 % for CerTest and 91.3 % for Panbio (p = 0.250) and notably decreased for samples taken after the fifth day. Both Ag-RDTs showed an excellent agreement between them (agreement = 96.7 %, k = 0.920). Agreement with PCR was also excellent for high viral load samples (Ct<25) for CerTest (98.0 %, k = 0.954) and Panbio (98.8 %, k = 0.973).

Conclusions

CerTest SARS-CoV-2 and Panbio COVID-19 Ag showed excellent performance and agreement results for samples with high viral loads (Ct $\le$ 25) or samples taken within the first 5 days after the onset of symptoms.

1. Introduction

The pandemic due to SARS-CoV-2 that started in Wuhan in December 2019 has caused as of November 10, 2020, more than 49.7 million cases and over 1.2 million deaths worldwide [1]. A key point to control this pandemic is an early and accurate diagnosis of SARS-CoV-2 infection. This allows the establishment of infection control measures and systematic tracing of close contacts of COVID-19 cases. Polymerase chain reaction (PCR) is the reference method for COVID-19 diagnosis. However, these assays require trained personnel, specialized equipment and take several hours to perform. Antigen rapid diagnostic tests (Ag-RDTs) have been developed to overcome these limitations. During the first wave of the pandemic, these assays were not recommended due to poor sensitivity results [2,3]. However, the interest in these Ag-RDTs remained as they could be employed as point of care tests (POC), presented a lower cost than PCR assays and could improve the turnaround time for results [4,5]. Because of that, it has been proposed that these advantages could overcome the sensitivity limitation, especially where PCR is unavailable or when excessive turnaround times preclude clinical utility [[6], [7], [8]]. Because of that, there is an increasing number of commercialized Ag-RDTs that meet the diagnostic performance requirements established by the World Health Organization [[8], [9], [10], [11], [12], [13], [14]]. Our objective was to evaluate the diagnostic performance of two of these Ag-RTDs.

2. Methods

2.1. Population and study period

The study was performed between 8th and 20th October 2020. We included 320 consecutive nasopharyngeal samples from patients with suspicion of COVID-19 that were attended in our hospital and associated primary healthcare centers. Each sample corresponded to one single patient. The study was performed on two groups of patients:

PCR negative patients: we included 150 PCR negative samples. They were employed to assess the specificity of Ag-RDTs.

PCR positive patients: we included 170 PCR positive samples to evaluate the sensitivity.

For both groups, we employed nasopharyngeal swabs submitted in 3 mL of universal transport medium (UTM). Nasopharyngeal swabs and UTM tubes were provided by Vircell (Vircell, S.L., Granada, Spain) and Deltalab (Deltalab, Barcelona, Spain) laboratories. Nasopharyngeal samples were processed for PCR upon arrival at the laboratory and were later cryopreserved at -20 °C until their analysis by Ag-RDTs.

2.2. Diagnostic methods

2.2.1. Molecular techniques

A summary of molecular techniques is shown in Supplementary Table 1. Briefly, RNA amplification was performed using three Real-Time PCR platforms: Viasure SARS-CoV-2 Real Time PCR Detection Kit (Certest Biotech S.L., Zaragoza, Spain; which detected SARS-CoV-2 ORF1ab and N genes), Allplex SARS-CoV-2 assay (Seegene, Seoul, South Korea; detected genes: E, RdRP, S and N) and GeneFinder COVID-19 Plus RealAmp Kit (Osang Healthcare Co., Gyeonggi, South Korea; detected genes: E, RdRP and N). All equipments were employed according to the manufacturer's instructions for both the handling and the interpretation of the results. We considered as positive PCR samples those in which all genes included in each RT-PCR assay were positive (see Supplementary Table 1). Nasopharyngeal samples were tested using one or other PCR platform indistinctly, according to the usual laboratory workflow.

2.2.2. Antigen rapid detection tests

The characteristics of the evaluated Ag-RDTs (including sample processing and interpretation) are summarized in Supplementary Table 2. These Ag-RDTs were CerTest SARS-CoV-2 Ag One Step Card Test (Certest Biotec S.L., Zaragoza, Spain) and Panbio COVID-19 Ag Rapid Test Device (Abbot Rapid Diagnostics GmbH, Jena, Germany). Both Ag-RDTs detected SARS-CoV-2 nucleoprotein antigens. Both tests were carried out under biosafety conditions. In order to avoid interferences, the reading and interpretation of the results was blind between Ag-RDTs (CerTest results reading was made without knowing the result of Panbio for each sample and vice versa).

2.3. Clinical data

Demographic (age, sex) and clinical variables of the study population (symptoms, time from the onset of symptoms) were obtained from the medical records in PCR positive patients. Additionally, we recovered the Cycle threshold (Ct) value corresponding to N gene for all PCR positive samples in order to assess SARS-CoV-2 viral load. This value in RT-PCR refers to the moment in which amplification occurs and it is inversely correlated with the viral load in these samples (lower Ct values indicate higher viral loads) [15].

2.4. Statistical analysis

Continuous variables were expressed as median and interquartile range (IQR) and categorical variables as proportions. Sensitivity and specificity with 95 % confidence intervals (95 %CI) were calculated using RT-PCR as gold standard. We evaluated the overall sensitivity of these tests and also we analyzed the sensitivity according to the Ct value for N gene using a cutoff of Ct = 25 [11] (group 1: high viral load, Ct $\le$ 25; group 2: low viral load, Ct>25) and the time from the onset of symptoms, with a cutoff of 5 days ($\le$ 5 days, 6–10 days, > 10 days). Agreement between different techniques was evaluated using the Cohen’s kappa score [16] and the McNemar’s test. For these comparisons, a p value less than or equal to 0.05 was considered significant. Statistical analysis was performed using Stata/IC 13.1 (StataCorp, Texas, USA).

3. Results

3.1. Summary of the study population

Regarding those 170 PCR positive patients, median age was 51 years (IQR: 38–68) and 89 (52.4 %) were male. 134 (78.8 %) patients presented COVID-19 symptoms and 26 (15.3 %) were asymptomatic patients with a prior contact with COVID-19 case. Most frequent symptoms were cough (54.5 %), followed by fever (41.0 %), dyspnea (25.4 %), anosmia (21.6 %) and myalgia (18.7 %). Information regarding symptoms was unavailable for ten patients and for another six of symptomatic patients, days from the onset of symptoms to sample obtention was unavailable. Regarding the origin of samples, 85 (50.0 %) were delivered from our primary healthcare centers, 34 samples (20.0 %) were from hospitalized patients, 36 (21.2 %) came from the emergency department and 15 samples (8.8 %) came from the occupational health department of our hospital.

3.2. Diagnostic performance according Ct values

Diagnostic performance results according Ct values are summarized in Table 1 and Fig. 1 . Both Ag-RDTs showed a specificity of 100 % (95 %CI: 97.6–100.0) and an overall sensitivity of 53.5 % (45.7–61.2) for CerTest and 60.0 % (52.2–67.4) for Panbio and this difference was statistically significant (p = 0.001). Sensitivity increased for samples with high viral loads (Ct $\le$ 25). For these samples, sensitivity was 94.0 % (86.7–98.0) for CerTest and 96.4 % (89.9–99.3) for Panbio, but this difference was not statistically significant (p = 0.500). On the other hand, for samples with low viral load (Ct>25), both tests showed poor sensitivity results, as sensitivity was 14.0 % (7.4–23.1) for CerTest and 24.4 % (15.8–34.9) for Panbio (p = 0.004).

Table 1.

Diagnostic performance of the evaluated Ag-RDTs according to viral load.

	PCR positive			PCR negative			High viral load (Ct$\le$25)			Low viral load (Ct>25)
No. samples	170			150			84			86
Ct values	25.2 (20.2–29.7)			–			20.2 (17.9–22.2)			29.7 (27.0–31.3)
	Positive samples	Sensitivity	p value	Positive samples	Specificity	p value	Positive samples	Sensitivity	p value	Positive samples	Sensitivity	p value
CerTest	91/170	53.5 (45.7–61.2)	0.001	0/150	100.0 (97.6–100.0)	1.000	79/84	94.0 (86.7–98.0)	0.500	12/86	14.0 (7.4–23.1)	0.004
Panbio	102/170	60.0 (52.2–67.4)		0/150	100.0 (97.6–100.0)		81/84	96.4 (89.9–99.3)		21/86	24.4 (15.8–34.9)

Statistics: values are expressed as absolute count (percentage) and median (interquartile range). Sensitivity and specificity results are expressed as percentage with 95 %CI. P-values were calculated by the McNemar’s test. Significant differences are shown in bold. Abbreviations: Ag-RDT: antigen rapid diagnostic test; Ct: cycle threshold; 95 %CI: 95 % confidence interval; p-value: level of significance.

Fig. 1.

Positivity rates for Ag-RDTs according to Ct values.

Abbreviations: Ag-RDT: antigen rapid diagnostic test; Ct: cycle threshold.

3.3. Diagnostic performance according time from the onset of symptoms

Table 2 and Fig. 2 summarize the diagnostic performance results according to the time from the onset of symptoms. Both Ag-RDTs showed high sensitivity in samples taken within the first 5 days from the onset of symptoms: 84.8 % (71.1–93.7) for CerTest and 91.3 % (79.2–97.6) for Panbio, p = 0.250). Sensitivity decreased significantly from the sixth day from the onset of symptoms, reaching a sensitivity of 25.9 % (11.1–46.3) for Panbio and 18.5 % (6.3–38.1) for CerTest from 10 days (p = 0.500).

Table 2.

Diagnostic performance of the evaluated Ag-RDTs according to time from the onset of symptoms.

Time from the onset of symptoms	$\le$ 5 days (n = 46)			6 – 10 days (n = 55)			> 10 days (n = 27)
	Positive samples	Sensitivity	p value	Positive samples	Sensitivity	p value	Positive samples	Sensitivity	p value
CerTest	39/46	84.8 (71.1–93.7)	0.250	27/55	49.1 (35.4–62.9)	0.063	5/32	18.5 (6.3–38.1)	0.500
Panbio	42/46	91.3 (79.2–97.6)		32/55	58.2 (44.1–71.3)		7/27	25.9 (11.1–46.3)

Statistics: sensitivity results are expressed as percentage with 95 %CI. P-values were calculated by the McNemar’s test. Significant differences are shown in bold. Abbreviations: Ag-RDT: antigen rapid diagnostic test; 95 %CI: 95 % confidence interval; p-value: level of significance.

Fig. 2.

Positivity rates for Ag-RDTs according to the days from the onset of symptoms.

Abbreviations: Ag-RDT: antigen rapid diagnostic test.

3.4. Agreement results between diagnostic techniques

Agreement results are summarized in Table 3 . Agreement between Ag-RDTs was excellent for overall samples (agreement = 96.7 %, k = 0.920) and also for high viral load samples (Ct $\le$ 25, agreement = 99.2 %, k = 0.982). Agreement of Ag-RDTs with PCR was moderate in overall samples (Panbio: 79.6 %, k = 0.596; CerTest: 76.4 %, k = 0.531) but regarding high viral load samples, they showed almost perfect agreement (Panbio: 98.8 %, k = 0.973; CerTest: 98.0 %, k = 0.954)

Table 3.

Agreement between Ag-RDTs.

Agreement with RT-PCR	Agreement (%)	Kappa
Overall samples
CerTest	76.4	0.531
Panbio	79.6	0.596
High viral load samples (Ct$\le$25)
CerTest	98.0	0.954
Panbio	98.8	0.973

Agreement between Ag-RDTs	Agreement (%)	Kappa
Overall samples
CerTest vs Panbio	96.7	0.920
High viral load samples (Ct$\le$25)
CerTest vs Panbio	99.2	0.982

Statistics: values are expressed as percentage. Kappa scores over 0.81 (almost perfect agreement) are shown in bold. Abbreviations: Ag-RDT: antigen rapid diagnostic test; Ct: cycle threshold.

4. Discussion

Our study shows that CerTest and Panbio Ag-RDTs are reliable to diagnose SARS-CoV-2 infection. They showed excellent performance characteristics when they were performed in samples with high viral load or samples taken within the first five days after the onset of symptoms. Moreover, both techniques showed a specificity of 100 % and excellent levels of agreement between them and for high viral load samples, sensitivity results over 90 % and excellent levels of agreement with PCR.

Ag-RDTs have demonstrated their usefulness to control the SARS-CoV-2 pandemic. Panbio Ag-RDT has been the most frequently evaluated test [11,12,[17], [18], [19]]. However, in the context of the escalating new waves of this pandemic, the number of alternative assays provided by other manufacturers is growing exponentially [9,10,12,14,19,20]. In this context, proper validations of these new tests are critical before their use in clinical practice [21]. Regarding Panbio Ag-RDT, our results are in line with those reported by other authors, as we showed a specificity of 100 %, a sensitivity over 90 % for high viral load samples and over 85 % for patients within five days post symptoms onset [11,12,[17], [18], [19]]. Furthermore, we also showed that these tests could be performed on the same UTM sample that can later be used for PCR. In this way, it could be not necessary to obtain a sample for the antigen test and another for PCR. However, diluting the sample in UTM and later in the test buffer reduces antigen concentrations in the final sample when comparing with direct inoculation in the buffer, thus potentially reducing sensitivity.

Some authors have pointed out that, besides the lower sensitivity of Ag-RDTs compared to PCR, they improve the turnaround time for results, which is key to interrupt transmission chains in order to control the spread of this pandemic [6,7,21,22]. In line with this, some diagnostic algorithms recommend the use of these tests as the first step for symptomatic patients within the first five days after the onset of symptoms [[21], [22], [23]]. Our results support the use of CerTest and Panbio tests for that purpose. However, it should be noted that the evidence regarding the use of these tests is mainly focused on symptomatic patients with less than 5–7 days since the onset of symptoms and, from that period of time, the sensitivity of these tests is insufficient. Because of that, Ag-RDTs cannot be considered as a replacement to RT-PCR, but a complement for the diagnosis of SARS-CoV-2 infection. On the other hand, some authors have pointed out that these tests can be useful also for asymptomatic individuals with high viral loads, being particularly useful in situations of high prevalence of the disease [24]. These results would mean that Ag-RDTs could be used as a screening tool in asymptomatic populations to detect individuals with high infectious capacity. However, there is needed more evidence to confirm the usefulness of Ag-RDTs for such kind of interventions.

Our study presents some limitations: first, it is a retrospective study that has been performed on nasopharyngeal samples delivered to our laboratory. There would be needed additional clinical validation studies to reinforce our conclusions. Second, it has been conducted in a single institution and we have analyzed the results of two among all commercialized Ag-RDTs. Consequently, our conclusions should not be extrapolated to other available Ag-RDTs and more prospective multicenter studies and meta-analysis are needed to establish the usefulness of other Ag-RDTs. However, to the best of our knowledge, this study constitutes the first comparative evaluation of CerTest and Panbio Ag-RDTs and our findings indicate that these tests could be reliable tools for the diagnosis of COVID-19 and the control of this pandemic.

Funding

This research was funded by Certest Biotec S.L. (Zaragoza, Spain). Each manufacturer provided Panbio and CerTest devices.

Informed consent

Since the present study is retrospective, informed consent was not required.

Ethical approval

The study was conducted according to the ethical requirements established by the Declaration of Helsinki. The Ethics Committee of Hospital Universitario Príncipe de Asturias (Madrid) approved the study (protocol code: Antígeno-COVID).

Author contributions

Study concept and design: FPG and JCG.

Patients’ selection and clinical data acquisition: FPG, JR, PGH, TA, RPT and ML.

Sample processing: IPR, ALB and HMG.

Statistical analysis and interpretation of data: FPG.

Writing of the manuscript: FPG and JCG.

Critical revision of the manuscript for relevant intellectual content: JCG.

Supervision and visualization: JCG.

All authors read and approved the final manuscript.

Declaration of Competing Interest

The authors report no declarations of interest.

Appendix A. Supplementary data

The following are Supplementary data to this article: