Evaluation of three rapid lateral flow antigen detection tests for the diagnosis of SARS-CoV-2 infection

Abstract

Introduction

The COVID-19 pandemic has led to high demand of diagnostic tools. Rapid antigen detection tests have been developed and many have received regulatory acceptance such as CE IVD or FDA markings. Their performance needs to be carefully assessed.

Materials and Methods

158 positive and 40 negative retrospective samples collected in saline and analyzed by a laboratory-developed RT-PCR test were used to evaluate Sofia (Quidel), Standard Q (SD Biosensor), and Panbio™ (Abbott) rapid antigen detection tests (RADTs). A subset of the specimens was subjected to virus culture.

Results

The specificity of all RADTs was 100 % and the sensitivity and percent agreement was 80 % and 85 % for Sofia, 81 % and 85 % for Standard Q, and 83 % and 86 % for Panbio™, respectively. All three RADTs evaluated in this study reached a more than 90 % sensitivity for samples with a high viral load as estimated from the low Ct (Cycle threshold) values in the reference RT-PCR. Virus culture was successful in 80 % of specimens with a Ct value <25.

Conclusions

As expected, the RADTs were less sensitive than RT-PCR. However, they benefit from the speed and ease of testing, and lower price as compared to RT-PCR. Repeated testing in appropriate settings may improve the overall performance.

1. Introduction

The superior analytical sensitivity and specificity makes RT-PCR the primary diagnostic tool for the detection of SARS-CoV-2 in respiratory samples [1,2]. Rapid antigen detection tests (RADTs) can be cheap and easy to use compared to RT-PCR. Contrary to PCR, antigen tests do not amplify the detected target, rendering them generally less sensitive [3]. The SARS-CoV-2 RADTs are relatively new in the market, without extensive evaluation. They appear to show variable performance: according to a systematic review published in August 2020, RADT sensitivity varied from 0% to 94 % [4]. We evaluated three CE IVD marked SARS-CoV-2 RADTs: Sofia (Quidel), Standard Q COVID-19 Ag (SD Biosensor), and Panbio™ (Abbott). In each of these tests, the sample is suspended or diluted in a sample inactivation medium and transferred to a test device. If SARS-CoV-2 antigen is present above a threshold concentration, a line will appear in the test device within 15 min. For Panbio™ and Standard Q, the line is visible for a naked eye, and for Sofia, detection of the fluorescent signal is automated.

2. Materials and methods

The evaluation was performed at the diagnostic laboratory of Helsinki University Hospital and Helsinki and Uusimaa Hospital District, Finland. Research permit HUS/157/2020 (Helsinki University Hospital, Finland) was obtained from the local review board.

2.1. Evaluated tests

We evaluated the performance of Quidel Sofia SARS FIA (Quidel, San Diego, CA), Standard Q COVID-19 Ag test (SD Biosensor, Republic of Korea), and Panbio™ (Abbott Diagnostic GmbH, Jena, Germany) according to each manufacturer’s guidelines for samples in virus transport medium. All evaluated tests are intended for fresh swab samples, so this is off-label use and leads to dilution of samples, see Table 1 .

Table 1.

Main properties of the three RADTs evaluated in this study.

	Quidel Sofia SARS Antigen FIA	SD Biosensor SARS-CoV-2 Rapid Antigen test	Abbott Panbio™ COVID-19 Ag Rapid TEST Device NP
Intended use	Nasopharyngeal and nasal swabs	Nasopharyngeal swabs	Nasopharyngeal swabs
Target	Nucleocapsid protein	Nucleocapsid protein	Nucleocapsid protein
Interpretation of results	Sofia or Sofia2 instrument	Visual	Visual
Procedural control/visible control line	Yes/No	Yes/Yes	Yes/Yes
External controls	Yes	Available, but not included in the kit	Yes
LoD, as reported by the manufacturer	1.13 × 102 TCID50/mL	3.12 × 102.2 TCID50/mL	2.5 × 101.8 TCID50/mL
Sensitivity, as reported by the manufacturer	96.67%	96.52 %	93.33 %
Specificity, as reported by the manufacturer	100 %	99.68 %	99.45 %
Incubation time	15 min	15−30 minutes	15−20 minutes
Sample volume	120 μL	3 drops	5 drops
Sample volume when using a frozen VTM sample μL	250	350	150
Sample dilution when using a frozen saline sample in this study, approximately	1:6	1:4	1:10
Regulatory certification	CE-IVD, FDA EUA	CE-IVD	CE-IVD
Lots used in this evaluation	143,489	QCO3020105	41ADF024A

2.2. Reference test

The samples were originally analyzed with a laboratory-developed RT-PCR test (LDT) based on the method by Corman and others [5] and modified by us [6] to detect the N gene target of SARS-CoV-2.

2.3. Patient samples

A total of 198 nasopharyngeal swabs in 0.9 % saline taken between April and November 2020 and stored in −20 °C were available for the study.

The analytical performance evaluation was conducted with 102 samples: 40 LDT negative and 62 positive samples, whose Ct (Cycle threshold) values varied between 14.73–35.71 (the positive control Ct was 26.35–32.66).

To investigate the test performance in an outpatient setting, another set of LDT positive samples from adults from outpatient clinics and drive-through testing sites was included by selecting samples systematically backwards from 18 November until 96 specimens were reached, i.e. November 1–18 2020. Samples from hospitals and occupational health services were excluded, as well as samples that had less than the required 750 μL for three RADTs left. Due to variable amount of test devices available, eighty-seven samples were analyzed by Sofia, 90 by Panbio™ and 96 by Standard Q.

2.4. Virus culture

59 specimens of the PCR positive subset used for analytical performance evaluation was subjected to virus isolation experiments in Vero E6 TMRPSS2 cells as described in [7].

2.5. Statistics

Concordance of the results was examined in McNemars’s test. Level of statistical significance was set at P < 0.05. Cohen’s kappa coefficient was computed to assess the agreement between the methods by chance and Mann-Whitney U test was used to compare the Ct value medians. Statistical analysis was performed using SPSS/PASW statistical program package, version 25 (IBM SPSS Inc., Chicago, IL, USA).

3. Results

The performance of the Sofia, Standard Q and Panbio™ RADTs in comparison with the LDT RT-PCR was assessed by analysing 188, 198 and 190 specimens, respectively. The three RADTs provided valid results for all specimens. The specificity for all three tests was 100 %: all RT-PCR negative samples tested negative. In all, the results of Sofia were 84.57 % consistent with the reference RT-PCR with a kappa value of 0.636 (P < 0.001). The Standard Q and Panbio™ tests achieved overall agreements of 84.85 % and 86.32 % as compared to the RT-PCR with kappa values of 0.633 (P < 0.001) and 0.660 (P < 0.001), respectively. For all of the RADTs, the observed difference to the reference PCR was statistically significant (P < 0.001). The median Ct values of the specimens positive with RADTs were significantly lower than those of the specimens with false negative results (P < 0.001 for all RADTs) (Table 2 ).

Table 2.

The performance of RADTs, all samples, all RT-PCR positive samples, and samples grouped according to Ct values of the N gene target in the LDT RT-PCR. Tested, n, number of tested samples. Same as ref: number of samples tested the same as reference LDT RT-PCR test, positive or negative. Tested pos, n, number of samples that were positive in RADT. Ag⁻/Ag⁺: Ct median of samples that were negative in RADT (Ag⁻) and Ct median of samples that were positive in RADT (Ag⁺). ¹For virus culture experiments, at this rowthere are Ct median of samples that were negative in virus culture and Ct median of samples that were positive in virus culture.

		SOFIA QUIDEL	STANDARD Q SD BIOSENSOR	PANBIO ABBOTT	VIRUS CULTURE
All	Tested, n	188	198	190	59
samples	Same as ref	159	168	164	18
	Agreement	84.57 %	84.85 %	86.32 %	30.51 %
	95 % CI	78.72−89.04	79.19−89.18	80.70−90.49	20.25−43.15
	Ct median (Ag−/Ag+)1	31.75/20.97	31.17/20.95	31.08/20.98	29.33/19.261
All RT-PCR positive	Tested, n	148	158	152	59
	Tested pos, n	119	128	126	18
samples	Sensitivity	80 %	81 %	83 %	31 %
	Agreement	80.41 %	81.01 %	82.89 %
	95 % CI	73.28−86.00	74.19−86.36	76.12−88.05
RT-PCR	Tested, n	89	97	92	20
positives,	Tested pos, n	88	96	90	16
Ct<25	Sensitivity	99 %	99 %	98 %	80 %
RT-PCR	Tested, n	34	35	34	19
positives,	Tested pos, n	28	24	26	2
Ct 25−29.99	Sensitivity	82 %	69 %	76 %	11 %
RT-PCR	Tested, n	123	132	126	39
positives,	Tested pos, n	116	120	116	18
Ct<30	Sensitivity	94 %	91 %	92 %	46 %
RT-PCR	Tested, n	25	26	26	20
positives,	Tested pos, n	3	8	10	0
Ct≥30	Sensitivity	12 %	31 %	38 %	0%
RT-PCR	Tested, n	40	40	38	ND
negatives	Tested pos, n	0	0	0

At the analytical evaluation of the tests we used 40 negative and 62 positive samples in LDT, that were chosen to include a wide range of Ct values. When Ct in RT-PCR was <25, sensitivity of all three RADTs was 100 % and virus culture 80 %. When Ct in RT-PCR was ≥30, the sensitivity was much lower (Table 2).

The Ct values in the 96 positive samples of the adult outpatients were 10.74−32.49. (The positive control Ct was 25.56−27.96.) In all, 75 samples (78 %) had a Ct value <25, sixteen (17 %) had Ct values of 25−29.99 and five (5%) ≥30. The sensitivity values of RADTs were 87 % (Panbio™), 89 % (Standard Q) and 92 % (Sofia) but reached 92–96 % for samples with Ct <30 (Fig. 1 ).

Fig. 1.

Sensitivity of the RADTs in comparison to Ct values in RT-PCR. Samples include all the LDT RT-PCR positive nasopharyngeal samples from adult outpatients at HUSLAB in 1.-18. November 2020 that had enough sample volume left for three antigen tests. N = 96, however, in some cases, not all three RADTs were performed to all samples. Each dot indicates one samples. Ct <25: 75 samples; Ct 25-29.99: 16 samples; Ct ≥30: 5 samples.

4. Discussion

We evaluated three lateral-flow RADTs for the detection of SARS-CoV-2 infection in nasopharyngeal swab samples collected in 0.9 % saline. Our results agree with previous observations that RADTs perform well in detecting culture positive samples and samples with high viral loads [[8], [9], [10], [11]]. Compared to RT-PCR, the sensitivity of RADTs is low: in the present study the sensitivity was only 12–38 % in samples with Ct≥30 (Table 2).

However, the distribution of Ct values of RT-PCR positive samples in settings where antigen detection tests could primarily be used is unknown. We wanted to approach the real-life setting by testing all positive outpatient adult samples tested by LDT in our laboratory during 2.5 weeks in November 2020. The testing strategy in Finland in November 2020 assumed patients to have at least mild symptoms of SARS-CoV-2 infection. Thus, this panel is unlikely to include asymptomatic or presymptomatic patients. In November 2020 the sampling was available in average within 12 h of seeking for testing. Although medical records of outpatients were not available, we assume the majority of this panel to consist of samples taken within a couple of days from symptom onset. 78 % of samples had a Ct value <25, and 95 % <30, in our LDT. This may justify using antigen tests as a diagnostic tool even when their sensitivity compared to nucleic acid testing is poor for samples with high Ct values. Careful consideration between the advantages (ease, speed, costs) and disadvantages (lower sensitivity) needs to be conducted in the selection of appropriate testing strategies [11,12]. If RADTs are considered, laboratories need to perform independent clinical validations, as manufacturers’ declaration of performance may not include satisfactory selection of clinical samples. WHO has stated SARS-CoV-2 a biohazard risk for laboratory personnel [13], which needs to be accounted for in RADT testing. In addition, RATDs are not easily scalable for mass testing.

The present study is limited by its retrospective nature. Ideally, samples for RADT evaluation should be collected according to manufacturer’s instructions. Instead, we used frozen samples in saline. This may lead to underestimation of the sensitivity of the antigen detection tests.

All three RADTs, Sofia, Standard Q and Panbio™, were specific but less sensitive than RT-PCR. However, RADTs do have other advantages over PCR tests, such as shorter turnaround time and lower price. Together with the availability of testing outside central laboratories, these features enable repeated testing of individuals, which may increase the sensitivity of a testing regimen as a whole [14]. Indeed, simple and cheap RADTs with a satisfactory performance, such as the three tests evaluated in this study, make repeated testing a realistic diagnostic approach in settings in which RT-PCR is not readily or quickly available.

Declaration of Competing Interest

The authors report no declarations of interest.