Field evaluation of COVID‐19 antigen tests versus RNA based detection: Potential lower sensitivity compensated by immediate results, technical simplicity, and low cost

Abstract

One year into the coronavirus disease 2019 (COVID‐19) pandemic, diagnostic strategies, although central for contact tracing and other preventive measures, are still limited. To meet the global demand, lower cost and faster antigen tests for severe acute respiratory syndrome coronavirus 2 (SARS‐CoV‐2) detection are a convenient alternative to the gold standard reverse transcription‐polymerase chain reaction (RT‐PCR) assay. We tested laboratory‐based RT‐PCR RNA detection and two rapid antigen detection (RAD) tests, based on the immunochromatography test for nucleocapsid protein of SARS‐CoV‐2 (COVID‐19 Ag ECO Test, ECO Diagnóstica, and Panbio COVID‐19 Ag Rapid Test Abbott). Paired collection and testing were done in a small prospective open study in three clinical services in São Paulo, constituted of mostly symptomatic volunteers at collection (97%, 109/112) for a median of 4 days (interquartile range: 3–6), ranging from 1 to 30. Among the 108 paired RT‐PCR/RAD tests, results were concordant in 96.4% (101/108). The test's performance was comparable, with an overall sensitivity of 87% and a specificity of 96%. These observations add to other data that suggest that antigen tests may provide reasonable sensitivity and specificity and deserve a role to improve testing strategies, especially in resource‐limited settings.

Research Highlights

• Testing and isolating individuals transmitting SARS‐CoV‐2, generally with high viral load, is crucial to pandemic control.

• We documented comparable performance (96%) of two rapid test to standard RT‐qPCR.

• Lower cost and immediate results of rapid antigen tests may allow innovative containment measures or, at least, complement current testing limitations, specially in resource limited setting.

1. INTRODUCTION

After 1 year of the coronavirus disease 2019 (COVID‐19) pandemic, the world has over 90 million confirmed cases and 2 million deaths. Brazil is one of the hardest‐hit countries, with one in every thousand Brazilians dead due to COVID‐19 during this first year (https://covid19.who.int/, accessed January 18, 2021). The State of São Paulo recorded 1,702,294 of these confirmed cases of the new coronavirus and 51,566 deaths (http://www.saude.sp.gov.br, accessed January 31, 2021), and is the State with more cases and deaths from COVID‐19. The scale and impact of the epidemic are multifactorial, but limitations in diagnosis can be listed as one of the main bottlenecks. The use of molecular detection to guide the correct tracking of contacts is limited or non‐existent in Brazil as in some other places, and restrictions are even more evident in identifying asymptomatic cases, as a contact of an infected individual, and most of the limited capacity is mainly used to confirm hospitalized cases.

To control the COVID‐19 pandemic, improvement of detection with easy, rapid, and cost‐efficient approaches is urgently required. ¹ There are several obstacles to proper molecular detection by real‐time reverse transcription‐polymerase chain reaction (RT‐PCR) of the severe acute respiratory syndrome coronavirus 2 (SARS‐CoV‐2), the gold standard for laboratory confirmation of infection. Serological tests are erroneously adopted as a diagnostic tool by some, giving a false impression that they have been properly evaluated for infection. ² In contrast, rapid antigen detection (RAD) tests may identify acute COVID‐19 by recognition of the virus antigen itself. They are less laborious, require short training and are relatively inexpensive when compared to RT‐PCR. ³

Doubts about the quality of the rapid test results are one of the obstacles to its acceptance. We conducted a comparative and paired study of RT‐PCR and antigen tests in three services with different profiles. The objective was to evaluate the feasibility of the use of SARS‐CoV‐2 RAD tests and compare their performance to laboratory‐based viral RNA detection test (LBT).

2. MATERIALS AND METHODS

The study prospectively enrolled consecutive health service users, suspected of being infected with SARS‐CoV‐2 with up to 10 days of symptoms, recruited from 3 services in the metropolitan region of São Paulo, Brazil. The volunteers were included if agreed and signed an informed consent allowing the paired test and anonymous data registry. The RT‐PCR was offered by the reference laboratory from the State network for the diagnosis of COVID‐19, with paired material collected for one RAD test technology, either COVID‐19 Ag ECO Test, ECO Diagnóstica (Eco), or Panbio COVID‐19 Ag Rapid Test Abbott (Panbio). In some cases, LBT was repeated. For the RT‐PCR test, the volunteer was asked to gargle with 10 ml of saline solution, 3 ml of which were collected at a conic tube and combined with two swabs, one for each nostril (nasopharynx). This material was refrigerated and sent to the reference laboratory. Briefly, extracted RNA was submitted to RT‐PCR using different commercial kits based on protocols from two reference laboratories. The test used aimed at the detection of viral genes (e.g., N, E, and RdRp), including GeneFinder COVID‐19 Plus RealAmp kit (OSANG Healthcare) and Allplex (Seegene), which are based on the Charité protocol. ⁴ For some samples, it was applied IDT primers and GoTaq Probe 1‐Step RT‐qPCR System (Promega), based on the CDC protocol ⁵ to detect two genes (N1 and N2). The samples were considered positive according to the acceptance criteria of each kit. As recommended for the Influenza assay, Human RNAse P allowed the assessment of the quality of the sample and the presence of inhibitors, and reactions with a cycle threshold (C _t) up to 35 were considered valid.

Both RAD was carried out as specified by the manufacturers, with a single swab for collecting nasopharynx secretion in one nostril, subsequently immersed in the buffer solution, and then dripped onto the test plate. RAD is a qualitative membrane‐based immunoassay (immunochromatography) for the detection of Nucleocapsid protein of SARS‐CoV‐2 in nasopharyngeal samples. After 15 min the membrane was observed for control and test bands, which were differentiate by its intensity as none (not reactive, no band was observed); low (less visible than the control); and high (more visible than the control). The test was considered positive if the control was reactive and any intensity was observed at the test band.

2.1. Statistical analysis

We used the Shapiro–Wilk normality test to evaluate if data were normally distributed. Continuous variables were presented as the median and interquartile range (IQR) (25–75), Mann–Whitney U test to compare medians. Categorical variables, expressed as numbers (percentages), were compared by the χ ² test or Fisher's exact test as appropriate. Interrater agreement was calculated using kappa statistics. A two‐sided p‐value < .05 were considered significant. Statistical analyses were performed with SPSS v22 or STATA 10. Specificity and sensitivity with 95% confidence intervals (95% CI) and calculated the positive predictive value (PPV), considering the RT‐PCR result as the gold standard, using MedCalc (https://www.medcalc.org/calc/diagnostic_test.php).

3. RESULTS

From August 06 to December 14, 2020 the study enrolled 112 patients with clinical suspicion of COVID‐19 (median age: 40 years; range: 1–94 years), 70% from public services (36 in a referral outpatient clinic in Santo André and 43 inpatients at the Mauá Municipal Hospital), and 33 (30%) from a private hospital in Mauá. Nasopharyngeal secretion RT‐PCR was performed concomitantly with RAD in 108/112 patients (Panbio 39.3%, N = 44 and ECO 60.6%, N = 68). Only one RAD (Panbio) showed an invalid result and was repeated, giving a positive result.

The detection rate of SARS‐CoV‐2 was 27.7% (31/108) by RT‐PCR and 28% (31/112) by RAD. Among the 108 paired RT‐PCR/RAD tests, results were concordant in 96.4% (101/108). The overall sensitivity was 87% (95% CI: 70–96) and the specificity was 96% (95% CI: 89–99), with a Kappa of 0.84, p < .0001. This analysis includes two cases with positive antigen test (both Panbio) initially undetected by RT‐PCR but subsequently detected in samples repeated 1 and 3 days after the initial RT‐PCR.

3.1. Demographic, clinical, and laboratory data are shown in Table 1

Table 1.

Demographic, clinical, and laboratory characteristics study patients with paired RT‐PCR and RAD

	Any RAD	Panbio	Eco	p
Gender (male), N %	52, 46%	25, 57%	27, 40%	.09
Age (years)	40 (IQR: 31–56)	47.9 (IQR: 30–62)	39.4 (IQR: 31–54)	.36
Symptoms (days)	4 (IQR: 3–6)	4 (IQR: 3–5)	4 (IQR: 2–6)	.90
RNA detected (RT‐qPCR)	31/108, 28%	9/42, 21%	22/66, 32%	.30
Antigen detected (RAD)	31/112, 28%	11/44, 25%	20/68, 29%	.67
Concordance	101/108, 96%	40/42, 96%	61/66, 97%	.70
Sensitivity (%)	87 (95% CI: 70–96)	100 (95% CI: 66–100)	82 (95% CI: 60–95)
Specificity (%)	96 (95% CI: 89–99)	94 (95% CI: 80–99)	98 (95% CI: 88–100)

Note: Continuous variables presented as median and interquartile range (IQR) and categorical variables as proportions. p Value calculated using the Mann–Whitney U test for continuous variables and the χ ² test, or Fisher's exact test for categorical variables.

Abbreviations: CI, confidence interval; COVID‐19, coronavirus disease 2019; IQR, interquartile range; RAD, rapid antigen detection (COVID‐19 Ag ECO Test [Eco] and PanbioTM COVID‐19 Ag Rapid Test Abbott [Panbio]); RT‐PCR, reverse transcription polymerase chain reaction; RT‐qPCR, quantitative RT‐PCR.

This article is being made freely available through PubMed Central as part of the COVID-19 public health emergency response. It can be used for unrestricted research re-use and analysis in any form or by any means with acknowledgement of the original source, for the duration of the public health emergency.

Upon clinical revision, some cases did not meet enrollment criteria and had no COVID‐19 related symptoms or had longer than 10 days of symptoms. However, almost all cases were symptomatic at collection (109/112 97%), with a median of 4 (IQR: 3–6) days of symptom, ranging from 1 to 30. Only 3 hospitalized cases had symptoms over 10 days: 11 days, with an Eco reactive test and RT‐PCR detected, and two cases with both nonreactive Eco and an RT‐PCR not detected from 20 to 30 days of symptoms.

In seven cases the results of the RAD and RT‐PCR were divergent (Table 2). RAD was reactive in three cases (ECO, N = 2 and Panbio, n = 1) with RT‐PCR not detected. In two of these three cases, the intensity of the Ag test band was tenuous (Panbio and ECO). The third case with a band intensity greater than that of the control (Panbio) had a condition compatible with carcinomatous lymphangitis resulting from esophageal neoplasia with a nonspecific lung tomography finding.

Table 2.

Cases with discordant RT‐qPCR and RAD results

Case	Age (years)	Symptoms (days)	Trademark	RAD	Band intensity	RT‐qPCR	Ct
1	52	10	Eco	Nonreactive	None	Detectable	22/37
2	31	7	Eco	Nonreactive	None	Detectable	40/30/31
3	54	10	Eco	Nonreactive	None	Detectable	26
4	47	3	Eco	Nonreactive	None	Detectable	25/23/23
5	63	5	Eco	Reactive	Lowa	Not detectable	0/0/0
6	67	4	Panbio	Reactive	Highb	Not detectable	0/0/0
7	29	4	Panbio	Reactive	Lowa	Not detectable	0/0/0

Abbreviations: RAD, rapid antigen detection (COVID‐19 Ag ECO Test [Eco] and Panbio COVID‐19 Ag Rapid Test Abbott [Panbio]); RT‐qPCR, quantitative reverse transcription polymerase chain reaction.

^aLess visible than the control.

^bMore visible than the control; cycle threshold (C _t): three results referred to genes RdRp, N and E respectively; two results referred to genes N1 and N2; one result referred to gene E.

This article is being made freely available through PubMed Central as part of the COVID-19 public health emergency response. It can be used for unrestricted research re-use and analysis in any form or by any means with acknowledgement of the original source, for the duration of the public health emergency.

We compared the lower C _t obtained from the three RT‐PCR targets to SARS‐CoV‐2 genes. The C _t from the discordant results (negative antigen/positive RT‐PCR) was similar to that of concordant cases (25 vs. 27; p = .6).

We evaluated the positive and negative predictive value of RAD use at five different hypothetical prevalence rates (Table 3). For lower prevalence, like 10%, the tests have a PPV) below 80%, but at higher prevalence scenarios, as above 50%, the PPV values are above 94%.

Table 3.

Positive and negative predictive value of RAD at different COVID‐19 prevalence values (%)

	DP (%)	Positive predictive value	Negative predictive value	Accuracy
	10	71.3% (95% CI: 45–88)	98.5% (95% CI: 96–99)	95.2% (95% CI: 89–98)
Any RAD	20	84.8% (95% CI: 65–94)	96.8% (95% CI: 92–99)	94.3% (95% CI: 88–98)
	30	90.6% (95% CI: 76–97)	94.6% (95% CI: 87–98)	93.4% (95% CI: 87–97)
	50	95.7% (95% CI: 88–99)	88.1% (95% CI: 75–95)	91.6% (95% CI: 85–96)
	70	98.1% (95% CI: 95–99)	76.2% (95% CI: 56–89)	89.8% (95% CI: 83–95)
	10	64.7% (95% CI: 32%–88%)	100%	94.6% (95% CI: 83–99)
Panbio	20	80.5% (95% CI: 52–94)	100%	95.2% (95% CI: 84–99)
	30	87.6% (95% CI: 65–96)	100%	95.8% (95% CI: 85–100)
	50	94.3% (95% CI: 81–98)	100%	97% (95% CI: 86–100)
	70	97.5% (95% CI: 91–99)	100%	98.2% (95% CI: 88–100)
	10	80% (95% CI: 36–97)	98% (95% CI: 95–99)	96.1% (95% CI: 88–99)
Eco	20	90% (95% CI: 56–98)	95.6% (95% CI: 90–98)	94.6% (95% CI: 86–99)
	30	93.9% (95% CI: 69–99)	92.6% (95% CI: 84–97)	94% (95% CI: 84–98)
	50	97.3% (95% CI: 84‐100)	84.3% (95% CI: 69–93)	89.8% (95% CI: 80–96)
	70	98.8% (95% CI: 82–100)	69.7% (95% CI: 49–85)	86.6% (95% CI: 76–94)

Abbreviations: CI, confidence interval; COVID‐19, coronavirus disease 2019; DP, disease prevalence; RAD, rapid antigen detection test (COVID‐19 Ag ECO Test [Eco] and Panbio COVID‐19 Ag Rapid Test Abbott [Panbio]).

This article is being made freely available through PubMed Central as part of the COVID-19 public health emergency response. It can be used for unrestricted research re-use and analysis in any form or by any means with acknowledgement of the original source, for the duration of the public health emergency.

4. DISCUSSION

In this study, we compared face‐to‐face two test modalities to detect SARS‐CoV‐2 infection, a laboratory‐based RT‐PCR test, and two commercially available antigen tests. Although the test results showed some discrepancies, with positive antigens tests not detected in the RNA test, as well as negative antigens with positive RNA detection, the overall agreement of the procedures was high (96%), suggesting reasonable comparability. With many products in the market, few studies had evaluated these two tests. Although the number of tests is small and the study may not provide by itself a conclusive assessment, it adds to other data that suggest that antigen tests may provide a reasonable testing alternative.

The median days of symptoms in our study were 4 days (IQR: 3–6), with an overall sensitivity (87%, 95% CI: 70–96). The results with Panbio were better, but with a large CI due to the small sample size. It is however comparable to studies that evaluated RAD Panbio in patients with up to 7 days of symptoms, as Albert study, with 96% (95% CI: 67.0–88.8), ⁶ and Linares study, with 86.5% (95% CI: 75.5–97.5) ⁷ and higher than that observed at the Fenollar study, ³ with the sensitivity of 79% 144 out of 182 among symptomatic patients. In a study at the University of Genève for the Swiss Federal Office of Public Health with an open enrollment that included asymptomatic contacts, ⁹ they observed sensitivity of 86% and 100% specificity among 535 paired analysis. The discrepancies were associated with cases with higher Ct, that is, lower viral load. We found only one study evaluating the RAD Eco, with concordant results in 121 out of 139 (87.1%), among symptomatic patients, the majority (81%) with symptoms up to 7 days. ⁸ It is lower than we observed (97%) but within the CI.

In some of these studies, as well as evaluations that assessed other rapid antigen tests with lower sensitivity, the discordance of results to viral or RNA presence ¹⁰ were mostly related to cases with lower viral load, reinforcing the notion that these tests would perform better in higher viral load, more infectious cases. ¹¹

Of the cases with discordant results, three were considered false‐positive RAD based on the gold standard methodology. Although antigen assays are more susceptible to potential false‐positive results, one of these cases had anosmia and had a subsequent seroconversion (confirmed by Elecsys Anti‐SARS‐CoV‐2 immunoassay test; Roche) but it was not re‐tested for antigen or RNA. Moreover, it was observed two cases of positive RAD, which were non‐detected RT‐PCR, but after few days a new sample for RT‐PCR was collected and then the presence of SARS‐CoV‐2 was detected. For this reason, it was considered that these two cases presented results in agreement for both methodologies. The findings in this study suggest that RAD sometimes may indicate the viral presence before RT‐PCR.

We tested the predictive value at different prevalent scenarios. As can be seen in Table 3, only at a prevalence of 10%, that Brazil had at the early months of the pandemic, have these tests a low PPV. As the pandemic expands and prevalence increases, so does the power of a positive test result to represent a true positive. They may help to guide public policies according to the prevalence observed in a given area.

Our study has many limitations, including the small sample size, the inability, due to limitation of tests available, to compare head‐to‐head the two RADs evaluated, as well as a limited follow‐up of cases to confirm testing results or to evaluate clinical progression. However, the study had some distinctions that are worth mentioning. First, we did the test during routine services after brief, in‐service training of nurses and other health care workers, suggesting the simplicity and adaptability of this testing modality to real‐world conditions. We opted for replacing the nasopharynx swab with saline gargle in our routine and at this study, due to recurrent scarcity of swabs, but also to minimize the exposure of health workers to the riskier oropharyngeal swab collection. Another point is the reports of a higher viral load in the samples obtained by gargle compared to those by swab pharyngeal. ¹² , ¹³ Therefore, by combining the two potential sources of RNA we may maximize sensitivity and decrease health worker risk. Our study involved three health units with a low‐tech laboratory, with no capacity to carry out RNA‐based tests. These clinical sites had reference laboratories to conduct RNA tests, but that is not a situation in many parts of the country as well as in many areas of the world. Although conducted in a specific area of Brazil, it may be generalized to other regions of the country and maybe other places of the world struggling for a more efficient COVID‐19 testing scenario. It would be of special relevance to places without easy access to proper cold chain transport to reference laboratories. Even when transport is no a limitation, RNA testing, as other laboratory procedures that need to be performed in other facilities, involves several steps from the collection to the result, which can impact the quality in resource‐limited settings. Even after reaching a reference laboratory, equipment, supplies, and specialized technicians are at strained limits in many places. The return of results to the health unit and/or the patient are steps needed before an actual infected patient finally gets into isolation.

If the LBT can bring quality and sensitivity, the time from collection to actual access to test results represents a period of potential transmissibility. Even with recommendations to assume a potential positivity while waiting for the results, this period may represent less protection of third parties, and the immediate access (e.g., few minutes) to the results of the RAD can bring a significant advantage in this aspect. In some settings, the need for the patient to return to get test results is not only an additional difficulty that RAD eliminates, and brings less opportunity for transmission, as many patients have to use public transportation to go to the health center to obtain those results. That means transmission opportunities not only on the way to the unit but as well as during circulation within the unit. These potential advantages should be considered in a comprehensive strategy that uses tests not only to inform the individual but as part of a public health policy to minimize the spread of the virus.

Therefore, the point of care test, RAD, provides an almost immediate result, while RT‐PCR requires at the best scenario a few hours and most cases days for the result. Cost and, especially, accuracy become important issues in the selection of the test. Although the results of our study show a relatively high agreement in both test modalities, some studies in the literature have shown less concordance. As much as a negative result in RT‐PCR does not rule out infection, the same is valid for RAD.

Unlike RT‐PCR tests and RAD, with better positivity in the first week, antibody detection tests may provide some information for population‐level surveys but may be less suitable for individual assessment. ² Since the appearance of antibodies depends on time to elicit an immune response, the diagnosis of COVID‐19 by serological methods is generally more efficient after eight days of illness when the sensitivity of serological assays exceeds that of nucleic acid tests. ⁸ Considering that, for mild cases, the first week represents the period of greatest transmissibility, ¹⁴ these tests are not applicable to indicate isolation. Several rapid tests based on the detection of antibodies, lateral flow immunoassay, from different companies are available, but many lack adequate validation of its performance regarding both sensitivity and specificity. ¹⁵ In places where rapid antibody tests are being used to minimize diagnosis bottlenecks, the option for an antigen test may be more adequate.

Brazil has never implemented a comprehensive strategy to tackle the COVID‐19 pandemic, with a lack of national coordination, the result of a government that denies the scale of the epidemic and the science. The Brazilian Ministry of Health attributes the number of deaths to noncompliance with its guidelines for early treatment with hydroxychloroquine. ¹⁶ There are few preventive initiatives at the municipal and state level, and some with different restrictive strategies. For example, there are municipalities, separated by a street, in which a shopping center is open on one side of the street, and the other side of the street, with restrictions on non‐essential activities. The increasing number of cases, deaths, and the stretch of the health system's capacities to its limit constitutes a serious threat to the sustainability of the health system and the ability to respond appropriately to the situation, express the result of these incoherencies. Proper testing and contact tracing may lessen this burden.

5. CONCLUSION

In a scenario of case screening in the first week of symptoms related to COVID‐19, the use of rapid antigen testing shows good comparability with laboratory‐based RNA detection. Both tests evaluated, the COVID‐19 Ag ECO Test (ECO Diagnóstica) and the Panbio COVID‐19 Ag Rapid Test (Abbott), approaches the criteria defined by World Health Organization for these tests of 80% sensitivity and 97% specificity. ¹⁷ The logistical advantages of point‐of‐care testing can supersede its limitations and provide a valuable tool to improve the diagnosis of COVID‐19, contributing to the control of transmission in the community.

STUDY GROUP

Melissa Cristina de Araújo Dias, Jaqueline Lima Gerbase, Ana Carolina Gaidis, Elis Muriel Marques Monti, Renira Gonçalves Santos Lima, Debora de Menezes Tanabe, Helio Yoshimoto, Deise Lopes Arantes, Humberto Barjud Onias, Isadora Id Limongelli, Letícia Fiorio Baptista, Silvia Costa dos Santos Marques, Pedro Mendes Lage, and Fernanda Matsuda Manzoni.

CONFLICT OF INTERESTS

The authors declare that there are no conflict of interests.

ETHICS STATEMENT

The study was approved by the institutional ethical committee (CAAE: 21164819.7.0000.0082) and all participants provided informed consent for dual testing and use of anonymized data for research purposes.

Matsuda EM, Campos IB, Oliveira IP, Colpas DR, Carmo AMS, Brígido LFM. Field evaluation of COVID‐19 antigen tests versus RNA based detection: Potential lower sensitivity compensated by immediate results, technical simplicity, and low cost. J Med Virol. 2021;93:4405–4410. 10.1002/jmv.26985