Abstract
Objectives
We aimed to evaluate the accuracy of the Panbio™ Ag-RDT at primary health care (PHC) centers and test sites in symptomatic patients and close contacts, using the Reverse-Transcription Polymerase Chain Reaction (RT-PCR) test as the gold standard.
Methods
The study was conducted in four PHC centers and two test sites in Mallorca, Spain. Consecutive patients older than 18 years, attending the sites for RT-PCR testing were included. Two nasopharyngeal samples were collected, one for RT-PCR and the other was processed on-site using the Panbio™ rapid antigen test kit for SARS-CoV-2. The sensitivity and specificity were calculated using RT-PCR as the reference, and the predictive values using the pretest probability results for each analyzed group.
Findings
A total of 1369 participants were included; mean age 42.5 ± 14.9 years and 54.3% women. The overall prevalence was 10.2%. Most participants (70.6%) presented within 5 days of the onset of symptoms. The overall sensitivity was of 71.4% (95% CI: 63.1%, 78.7%), the specificity of 99.8% (95% CI: 99.4%, 99.9%), the positive predictive value of 98.0% (95% CI: 93.0%, 99.7%) and a negative predictive value of 96.8% (95% CI: 95.7%, 97.7%). The sensitivity was higher in symptomatic patients, in those arriving within 5 days since symptom onset and in those with high viral load.
Interpretation
Ag-RDT had relatively good performance characteristics in suspected symptomatic patients within five days since the onset of symptoms. However, our data do not support the sole use of Panbio™ Ag-RDT in asymptomatic individuals.
Funding
None
Keywords: COVID-19, Rapid antigen test, Sar-CoV-2, Primary care
Background
The COVID-19 pandemic is a significant challenge to the populations and health-care systems of countries throughout the world. The early detection of infected persons using massive and accurate testing, contact tracing, and rapid isolation are effective in slowing virus transmission. A Reverse-Transcription Polymerase Chain Reaction (RT-PCR) is currently the gold standard for diagnosis, but it has certain inconveniences, such as limited access to disposables and reagents in some regions, high cost, long processing time, and a need for specialized laboratories and trained personnel.
Rapid antigen detection tests (Ag-RDTs) were available soon after the COVID-19 pandemic began. These simple and inexpensive tests use lateral flow assays to detect proteins from the SARS-CoV-2 active infection. Nevertheless, research conducted early during the COVID-19 pandemic reported they had unsatisfactory diagnostic performance1, especially a low sensitivity2. A Cochrane systematic review of 8 studies of 5 different antigen tests reported an average sensitivity of 56.2% (95% confidence interval [CI]: 29.5%, 79.8%) and a mean specificity of 99.5% (95% CI: 98.1%, 99.9%)3.
More recent studies have reported improved accuracy of Ag-RDTs, especially for patients with high viral loads, i.e. patients who present within 5 days since symptom onset2. On 11 September, the World Health Organization (WHO) published an interim guidance that recommended use of an Ag-RDT for diagnosis of COVID-19 when the RT-PCR test is not available. The WHO recommended that an Ag-RDT test must have a sensitivity of at least 80% and a sensitivity of at least 97%, based on the gold-standard RT-PCR test4. A patient with a positive Ag-RDT result within 5 days of symptom onset can be considered to have a SARS-CoV-2 infection because these individuals are more likely to have high viral loads. However, a negative result should be interpreted with caution, especially in patients with high pretest probability and confirmatory RT-PCR test following a negative Ag-RDT test is recommended5.
Ag-RDTs are promising point-of-care alternatives because they can be produced at low cost in high volume, are easy to use, and provide results within 15 min, thus allowing quicker clinical decisions. However, the RT-PCR test remains the gold standard for diagnosis because prospective and real-world data on the performance of Ag-RDTs are currently limited4. Nevertheless, for low- and middle-income countries that lack the resources to implement national RT-PCR testing strategies, the WHO has announced the availability of 120 million Ag-RDT kits6. Moreover, the Bill & Melinda Gates Foundation executed separate volume guarantee agreements with Abbott and SD Biosensor for production of an Ag-RDT.
Europe is currently experiencing a second wave of the COVID-19 pandemic and there is a greater demand for diagnostic testing. Fast diagnostic testing using an inexpensive, point-of-care, easy-to-use, and rapid technique might help alleviate the burdens experienced by testing laboratories and caregivers in primary healthcare (PHC) centers and COVID-19 test sites7. The increased demand for RT-PCR tests can cause delays in reporting of positive results and lead to delays in contact tracing, and thus have negative consequences for control of the COVID-19 pandemic.
Most previous studies that evaluated Ag-RDTs for COVID-19 examined symptomatic patients, were conducted in the setting of hospital emergency services, and examined patients who presented with moderate or severe symptoms of COVID-19. Thus, rigorous studies are needed before Ag-RDTs can be used for the diagnosis of SARS-CoV-2 in the setting of PHC centers or in the community.
The Panbio™ rapid antigen test kit for SARS-CoV-2 (Abbott Diagnostic GmbH, Jena, Germany) is a qualitative test using specimens from nasopharyngeal swabs. The manufacturer reported that the sensitivity for symptomatic patients was 93.3% overall and 98.2% in those RT-PCR cycle threshold (Ct) ≤33, and that the specificity was 99.4%8.
The main aim of this study was to evaluate the performance of the Panbio™ Ag-RDT at PHC centers and test sites in symptomatic patients and close contacts, using the RT-PCR test as the gold standard.
Methods
Design and setting
This prospective diagnostic study was conducted in Mallorca (Balearic Islands, Spain) from October 2–25, 2020. Two testing locations (COVID-EXPRESS) that cover the city of Palma and 4 PHC centers (Santa Ponça, Alcudia, Inca, and Coll d'en Rebassa) were included.
Study population
Individuals were invited to participate if they were older than 18 years, were not previously diagnosed with COVID-19, attended one of the above-named settings for RT-PCR testing, had symptoms suggestive of infection with referral by a general practitioner (GP), or had a close contact with another patient with an RT-PCR-confirmed infection.
All potentially eligible participants were asked to sign an informed consent document and to answer a short questionnaire that asked about the reasons for RT-PCR testing (referral by a GP due to symptoms, close contact, others); socio-demographic information (sex and age); and the presence of symptoms, type of symptoms, and number of days since symptom onset or close contact to a positive SAR-CoV-2 patient.
SARS-CoV-2 testing
Trained nurses collected two consecutive nasopharyngeal sample swabs for the RT-PCR test and the Ag-RDT and interpreted the results of the AG-RDT on-site.
RT-PCR
Within 24 h of collection, one nasopharyngeal swab was sent for processing to Son Espases University Hospital, Microbiology Service without any additional information of the participants. RNA extraction was performed using the MagMAX™ Viral/Pathogen Nucleic Acid Isolation Kit (ThermoFisher) and amplification was performed using the TaqPath™ COVID-19 CE-IVD RT-PCR Kit and QuantStudio™ (ThermoFisher). The viral load was expressed as Ct for three genes (ORF, N, and S).
Ag-RDT
The other nasopharyngeal swab was processed on-site using the Panbio™ Ag-RDT (Abbott Diagnostic GmbH, Jena, Germany) and the results were interpreted within 15 min following the manufacturer's instructions. This kit detects the presence of the nucleocapsid (N) protein on a membrane-based immunochromatography assay. For a positive result with the Abbot Panbio™ test, a test line must form in the result window. A visible control line is required to indicate a test result is valid. The sample from the swab is mixed with approximately 300 μl of buffer, and then 5 drops are dispensed into the device. Neither the test line nor the control line are visible in the result window prior to the specimen dispensation on the device
Statistical analysis
To evaluate the accuracy of the Panbio™ Ag-RDT, the initial prevalence of COVID-19 was estimated as 15%, the marginal error as 5%, and the sensitivity as 90%. Thus, it was necessary to test 927 participants.
The sensitivity, specificity, and their 95% CIs were calculated using RT-PCR as the reference. Sensitivity analysis was stratified by the declared reason for performing the RT-PCR test, symptoms, days since symptom onset or exposure, and Ct-value. Predictive values and 95% CIs were estimated using the pretest probability results for each analyzed group. Means and standard deviations were used to describe population characteristics and for descriptive analysis. All statistical calculations were performed using Stata 13 (StataCorp, College Station, TX, USA) with the Stata DIAGT module.
Ethics and funding
This study was conducted following the Declaration of Helsinki and was approved by the Balearic Research Ethics Committee (IB 4350/20 PI on 30/09/2020) and by the Mallorca Primary Care Research Commission. Each participant was asked to sign an informed consent agreement before inclusion. This study was promoted by the Balearic Public Health Service and no external funding was received.
Patient and public involvement
No patients were involved in setting the research question and we did not seek public engagement in the design of the laboratory aspects of the study, nor were they involved in developing plans for the design of the study. No patients were asked to advise on interpretation or writing up of results.
Results
Enrollment and characteristics of patients
We initially identified 1412 potentially eligible subjects visited consecutively in multiple PHC centers in Mallorca (Fig. 1 ). Twenty-seven individuals (1.9%) declined participation, mostly because of a lack of time or anticipation of discomfort from sample collection, and we were unable to retrieve Ag-RDT results for another 16 participants. The final sample consisted of 1369 participants, their mean age was 42.5 ± 14.9 years, and 54.3% were women. The overall prevalence of COVID-19 was 10.2%, and there were 140 positive RT-PCR tests and 102 positive Ag-RDTs. We excluded 7 RT-PCR results from Panbio performance characteristics analysis (3 because of incorrect labeling that could not be recovered and 4 because of inconclusive results).
Fig. 1.
Enrollment and clinical characteristics of patients who received the RT-PCR test and the Panbio™ test.
We analyzed the characteristics of all participants (Table 1 ). Most appointments for RT-PCR testing were because of close contact with a confirmed positive COVID-19 individual (54.8%) or because of symptoms suggestive of COVID-19 and referral by a PHC professional (36.7%); the other 116 individuals (8.5%) were considered as unknown, because there were referred by PHC professionals without declaring the reason.
Table 1.
Demographic and clinical characteristics of enrolled patients.
| N (%) | RT-PCR+ (N,%) | Ag-RDT+ (N,%) | |
|---|---|---|---|
| Entire sample* | 1369 (100%) | 140 (10.2%) | 102 (7.5%) |
| Age (mean, SD) | 42.5 ± 14.9 | 41.5 ± 14.8 | 42.8 ± 14.0 |
| ≤20 years | 71 (5.2%) | 8 (0.6%) | 4 (0.3%) |
| 21–30 years | 272 (19.9%) | 32 (2.3%) | 19 (1.4%) |
| 31–40 years | 313 (22.9%) | 31 (2.3%) | 23 (1.7%) |
| 41–50 years | 302 (22.1%) | 31 (2.3%) | 28 (2.0%) |
| 51–60 years | 227 (16.6%) | 21 (1.5%) | 18 (1.3%) |
| 61–70 years | 132 (9.6%) | 12 (0.9%) | 6 (0.4%) |
| >70 years | 52 (3.8%) | 5 (0.4%) | 4 (0.3%) |
| Sex | |||
| Women | 744 (54.3%) | 71 (5.2%) | 53 (3.9%) |
| Man | 625 (45.7%) | 69 (5.0%) | 49 (3.6%) |
| Reason for testing | |||
| Symptoms | 503 (36.7%) | 55 (4.0%) | 44 (3.2%) |
| Close contact | 750 (54.8%) | 76 (5.6%) | 55 (4.0%) |
| Unknown | 116 (8.5%) | 9 (0.7%) | 3 (0.2%) |
| Declaring symptoms | |||
| Yes | 680 (49.7%) | 87 (6.4%) | 72 (5.3%) |
| No | 689 (50.3%) | 53 (3.9%) | 30 (2.2%) |
| Fever | 252 (18.4%) | 49 (3.6%) | 42 (3.1%) |
| Cough | 301 (22.0%) | 41 (3.0%) | 37 (2.7%) |
| Sore throat | 310 (22.6%) | 33 (2.4%) | 28 (2.0%) |
| Chest pain | 61 (4.5%) | 8 (0.6%) | 6 (0.4%) |
| Shortness of breath | 92 (6.7%) | 12 (0.9%) | 10 (0.7%) |
| Tiredness | 251 (18.3%) | 45 (3.3%) | 37 (36.3%) |
| Muscle/joint pain | 223 (16.3%) | 44 (3.2%) | 39 (2.8%) |
| Headache | 341 (24.9%) | 53 (3.9%) | 48 (3.5%) |
| Diarrhea | 135 (9.9%) | 9 (0.7%) | 9 (0.7%) |
| Vomiting | 50 (3.7%) | 4 (0.3%) | 4 (0.3%) |
| Loss of smell | 54 (3.9%) | 17 (1.2%) | 13 (0.9%) |
| Loss of taste | 64 (4.7%) | 19 (1.4%) | 14 (1.0%) |
| Skin involvement | 10 (0.7%) | 2 (0.1%) | 2 (0.1%) |
| Unable to move/speak | 2 (0.1%) | 1 (0.1%) | 1 (0.1%) |
| Other | 125 (9.1%) | 14 (1.0%) | 13 (0.9%) |
| Not known | 1 (0.1%) | 1 (0.1%) | 1 (0.1%) |
| Days since SO/CC | |||
| ≤5 | 967 (70.6%) | 101 (7.4%) | 80 (5.9%) |
| >5 | 215 (15.7%) | 19 (1.4%) | 14 (1.0%) |
| Unknown | 187 (13.6%) | 20 (1.5%) | 8 (0.5%) |
| Viral load | |||
| N gene (mean, SD) | 20.3 ± 6.5 | N/A | N/A |
| S gene (mean, SD) | 21.9 ± 6.5 | N/A | N/A |
| ORF gene (mean, SD) | 21.0 ± 6.7 | N/A | N/A |
| N gene Ct<25 | 98 (73.1%) | 98 (73.1%) | 86 (87.8%) |
| N gene Ct=25.0–29.9 | 26 (19.4%) | 26 (19.4%) | 10 (38.5%) |
| N gene C ≥ 30.0 | 10 (7.5%) | 10 (7.5%) | 2 (0.1%) |
RT-PCR: reverse transcription polymerase chain reaction, Ag-RDT: rapid antigen diagnostic test, SO: symptom onset, CC: close contact, Ct: RT-PCR cycle threshold.
*4 RT-PCR results were inconclusive and 3 were unknown.
Almost half of the subjects (49.7%) reported symptoms within 7 days prior to testing, and the most frequently reported symptoms were headache (24.9%), sore throat (22.6%), cough (18.4%), and tiredness (18.3%). Most participants (70.6%) presented within 5 days of the onset of symptoms or close contact, although these data were unavailable for 13.6% of the participants. The SARS-CoV-2 N gene viral load was not obtained for 6, 7 for S gene and 8 for ORF gene, out of 140 patients with positive RT-PCR results due to sample mislabeling. The mean viral load values of the other 134 positive RT-PCR results Ct values below 25 (indicating high viral load) for all three genes (N: 20.3 ± 6.5, S: 21.9 ± 6.5, ORF: 21.0 ± 6.7). We considered a Ct of 25 to 29.9 as moderate viral load and a Ct above 30 as low viral load. More than 70% of the analyzed participants (n = 98) with available Ct values had high viral loads for the N gene.
Overall test accuracy
Our analysis of the overall performance of the Panbio™ test (Table 2 ) indicated a pretest prevalence of 10.2%, a sensitivity of 71.4% (95% CI: 63.1%, 78.7%), and a specificity of 99.8% (95% CI: 99.4%, 99.9%). This corresponded to a positive predictive value (PPV) of 98.0% (95% CI: 93.0%, 99.7%) and a negative predictive value (NPV) of 96.8% (95% CI: 95.7%, 97.7%).
Table 2.
Overall sensitivity, specificity, and predictive values of the Panbio™ test.
| Prevalence (%) | Sensitivity% (95% CI) | Specificity% (95% CI) | PPV% (95% CI) | NPV% (95% CI) | |
|---|---|---|---|---|---|
| Overall (N = 1362) | 10.2% | 71.4% | 99.8% | 98.0% | 96.8% |
| (63.1%, 78.7%) | (99.4%, 99.9%) | (93.0%, 99.7%) | (95.7%, 97.7%) | ||
| Overall: reason for testing | |||||
| GP referral for symptoms (N = 502) | 10.9% | 80.0% | 100.0% | 100.0% | 97.6% |
| (67.0%, 89.5%) | (99.1%, 100%) | (91.9%, 100%) | (95.7%, 98.7%) | ||
| Close contact (N = 745) | 10.2% | 69.7% | 99.7% | 96.3% | 96.6% |
| (58.1%, 79.7%) | (98.9%, 99.9%) | (87.4%, 99.9%) | (95.0%, 97.8%) | ||
| Unknown (N = 115) | 7.8% | 33.3% | 100.0% | 100.0% | 94.6% |
| (7.4%, 70.0%) | (96.5%, 100%) | (29.2%, 100%) | (88.7%, 98.0%) | ||
| Overall: symptoms | |||||
| Yes (N = 677) | 12.8% | 80.4% | 99.6% | 97.2% | 97.1% |
| (70.5%, 88.1%) | (98.7%, 99.9%) | (90.3%, 99.6%) | (95.5%, 98.3%) | ||
| No (N = 685) | 7.7% | 56.6% | 100.0% | 100.0% | 96.4% |
| (42.3%, 70.1%) | (99.4%, 100%) | (88.4%, 100%) | (94.7%, 97.7%) | ||
GP: general practitioner, 95%CI: 95% confidence interval, RT-PCR: Reverse-Transcription Polymerase Chain Reaction, Ag-RDT: rapid antigen diagnostic test.
Comparison of individuals referred by their GPs due to symptoms with individuals who had close contact with another patient indicated better test sensitivity (80.0%, 95% CI: 67.0%, 89.5% vs. 69.7%, 95% CI: 58.1%, 79.7%) and specificity (100%, 95% CI: 99.1%, 100% vs. 99.7%, 95% CI: 98.9%, 99.9%) in the former group. The test sensitivity was particularly poor (33.3%, 95% CI: 7.4%, 70.0%) in individuals with unknown reasons for testing. Considering all the participants reporting symptoms, the test sensitivity was similar to those with symptoms and referral by a GP (80.4%, 95% CI: 70.5%, 88.1%), but was considerably lower in asymptomatic subjects (56.6%, 95% CI: 42.3%, 70.1%). Notably, the test specificity was above 99.6% in all analyzed groups.
Test accuracy based on days since symptom onset or close contact
As noted above, most patients (n = 963) received tests within 5 days since symptom onset or since the close contact (Table 3 ). Nevertheless, the overall test sensitivity for these patients was only 77.2% (95% CI: 67.6%, 84.7%), below the minimal sensitivity recommended by WHO for Ag-RDTs (80%). However, the test sensitivity was acceptable for patients who reported symptoms (n = 556; 83.1%, 95% CI: 71.9%, 90.5%) and for patients referred by their GPs for symptoms (n = 418; 86.0%, 95% CI: 71.3%, 94.2%).
Table 3.
Accuracy of the Panbio™ test in patients tested 5 or fewer days (top) or more than 5 days (bottom) since symptom onset or close contact.
| Prevalence (%) | Sensitivity (%, 95% CI) | Specificity (%, 95% CI) | PPV (%, 95% CI) | NPV (%, 95% CI) | |
|---|---|---|---|---|---|
| ≤5 days | |||||
| Overall (N = 963) | 10.4% | 77.2% | 99.7% | 97.5% | 97.4% |
| 67.6%, 84.7% | 99.0%, 99.9% | 90.4%, 99.5% | 96.0%, 98.3% | ||
| Overall symptomatic (N = 556) | 12.7% | 83.1% | 99.5% | 96.7% | 97.5% |
| 71.9%, 90.5% | 98.3%, 99.9% | 87.6%, 99.4% | 95.6%, 98.6% | ||
| Overall asymptomatic (N = 407) | 7.3% | 63.3% | 100.0% | 100.0% | 97.1% |
| 43.9%, 79.4% | 99.0%, 100% | 82.3%, 100% | 94.8%, 98, 5% | ||
| GP referral for symptoms (N = 418) | 10.2% | 86.0% | 100.0% | 100.0% | 98.4% |
| 71.3%, 94.2% | 99.0%, 100% | 90.5%, 100% | 96.4%, 99.3% | ||
| Overall close contacts (N = 507) | 11.0% | 71.4% | 99.5% | 95.2% | 96.5% |
| 57.5%, 82.3% | 98.2%, 99.9% | 82.5%, 99.1% | 94.3%, 97.9% | ||
| Symptomatic close contacts (N = 117) | 23.0% | 77.7% | 97.7% | 91.3% | 93.6% |
| 57.2%, 90.6% | 91.4%, 99.6% | 70.4%, 98.4% | 86.0%, 97.3% | ||
| Asymptomatic close contacts (N = 390) | 7.4% | 65.5% | 100% | 100% | 97.3% |
| 45.6%, 82.0% | 98.9%, 100% | 82.3%, 100% | 95.0%, 98.7% | ||
| >5 days | |||||
| Overall (N = 213) | 8.9% | 73.6% | 100.0% | 100.0% | 97.4% |
| 48.5%, 89.8% | 98.1%, 100% | 76.8%, 100% | 93.9%, 99.0% | ||
| Overall symptomatic (N = 108) | 12.0% | 61.6% | 100.0% | 100% | 95% |
| 32.2%, 84.8% | 96.1%, 100% | 63.0%, 100% | 88.1%, 98.1% | ||
| Overall asymptomatic (N = 105) | 5.7% | 100.0% | 100.0% | 100.0% | 100.0% |
| 54.0%, 100% | 96.3%, 100% | 54.0%, 100% | 96.3%, 100% | ||
| GP referral for symptoms (N = 75) | 13.3% | 50.0% | 100.0% | 100.0% | 92.8% |
| 20.1%, 79.8% | 94.4%, 100% | 47.8%, 100% | 83.4%, 97, 3% | ||
| Overall close contacts (N = 135) | 6.7% | 100.0% | 100.0% | 100.0% | 100.0% |
| 66.3%, 100% | 97.1%, 100% | 66.3%, 100% | 97.1%, 100% | ||
| Symptomatic close contacts (N = 30) | 10.0% | 100.0% | 100.0% | 100.0% | 100.0% |
| 29.2%, 100% | 87.2%, 100% | 29.2%, 100% | 87.2%, 100% | ||
| Asymptomatic close contacts (N = 105) | 5.7% | 100.0% | 100.0% | 100.0% | 100.0% |
| 54.0%, 100% | 96.3%, 100% | 54.0%, 100% | 96.3%, 100% | ||
GP: general practitioner, 95%CI: 95% confidence interval, RT-PCR: Reverse-Transcription Polymerase Chain Reaction, Ag-RDT: rapid antigen diagnostic test.
The test sensitivity was also unacceptable for patients tested within 5 days since the close contact, reporting symptoms (n = 117; 77.7%, 95% CI: 57.2%, 90.6%) or not at the moment of testing (n = 390; 65.5%, 95% CI: 45.6%, 82.0%). These general trends in test sensitivity and specificity were similar when the time period was prolonged from 5 to 7 days (Supplementary Table S1). Analysis of test sensitivity according to patients’ symptoms are provided in Supplementary Table S2.
Effect of viral load on test sensitivity
We assessed the sensitivity according to the viral gene load (Fig. 2 ). The overall test sensitivity for patients with high viral loads of the N gene (n = 134; Ct < 25) was 87.7% (95% CI: 79.5%, 93.5%). Analysis of separate categories of patients with high viral loads indicated the test sensitivity was above 80% even in asymptomatic patients (86.2%, 95% CI: 68.3%, 96.1%). Notably, test sensitivity decreased considerably for patients with higher Ct values (lower viral loads). Analysis of S and ORF gene viral loads also indicated acceptable sensitivity of the test for patients with high viral loads, but not for patients with low viral loads (Supplementary Table S2).
Fig. 2.
Sensitivity of the Panbio™ test in patients with different clinical status and N gene viral load.
False-negative and false-positive results
There were false-negative Ag-RDT results in 40 of the 140 patients (28.6%) who had positive RT-PCR results. These individuals were mostly between 21 and 40 years-old (52.5%) and half of them were women. Among these 40 patients, the main reason for testing was for being in close contact with a SARS-CoV-2 confirmed patient within 5 days; 23 patients (57.3%) reported no symptoms when arriving at the testing site. Among those for whom data on N gene viral load were available (n = 36), 12 patients had high viral loads, 16 had moderate viral loads, and 8 had low viral loads.
There were false-positive Ag-RDT results in 2 of the 1222 patients (0.1%) who had negative RT-PCR results. Both of these patients were women, between 41 and 50 years-old, were tested because of close contact with a known patient, and had symptoms during the previous 5 days (headache, tiredness, and/or cough).
Inter-observer agreement
Two independent and blinded observers, one of whom was an expert evaluator, performed visual interpretations of 68 Ag-RDT results. The interobserver agreement was 100% (Supplementary Table 4).
No side effects were reported when collecting the nasopharyngeal swabs.
Discussion
This study was an evaluation of the Panbio Ag-RDT for SARS-CoV-2 in real-life PHC settings and test sites. Most patients visit PHC centers when they have mild or moderate symptoms of COVID-19 or after contact tracing. We found that the overall test sensitivity was 71.4% (95% CI: 63.1%, 78.7%), the sensitivity for symptomatic patients was 80.4% (95% CI: 70.5%, 88.1%), and the sensitivity for patients who reported symptoms within the previous 5 days was 83.1% (95% CI: 71.9%, 90.5%). The overall specificity was 99.8% (95% CI: 99.4%, 99.9%). Based on a prevalence of 10.2%, the PPV was 98.0% (95% CI: 93.0%, 99.7%) and the NPV was 96.8% (95% CI: 95.7%, 97.7%).
We found that test sensitivity was higher in samples that had high viral loads (Ct < 25), even in patients who were asymptomatic at the moment of testing (86.2%, 95% CI: 68.3%, 96.1%). This is an important finding because some asymptomatic individuals with SARS-CoV-2 infections (i.e., “super spreaders”) might have a substantial impact on the spreading of this virus 199. The Ag-RDT provides fast results, thus facilitating early identification, rapid isolation of the patient, and early contact tracing of highly contagious cases10 , 11.
Strengths and limitations
The present study is the largest evaluation of the Panbio Ag-RDT in samples from symptomatic and asymptomatic patients in real-world context. This is a strength of our study because this is the setting where the majority of the patients with mild symptoms (up to 75%)12 and close contacts are visited and followed-up. We incorporated COVID-19 testing into the usual care and management of patients with suspected SARS-CoV-2 infection and less than 2% of participants denied taking part in the study.
Our study has several limitations. We only assessed one type of Ag-RDT that targeted the SARS-CoV-2 N (nucleocapsid) protein, although other tests are available. Also, we only examined two types of testing scenarios —referral by a GP based on symptoms and close contact with a confirmed case — and therefore cannot make inferences about the applicability of this test in other scenarios (screening of nursing-homes, workplaces, etc.). Additional rigorous studies are needed to establish the optimal performance characteristics for Ag-RDTs that have different protein targets, are employed in different specific settings, and that have different pretest probabilities.
There is no standardized method to establishing the infectiousness of a patient with confirmed SARS-CoV-2 based on viral load. Ct values can vary among studies according to the type of test and target gene. We considered low viral load as a Ct value above 30 in patients with positive RT-PCR results13, and assumed that such individuals can be considered non-infectious. However, our results reinforce the presence of positive relationships between a positive Ag-RDT result and high viral load in all three analyzed genes (N, S, and ORF).
Comparation with other studies
The test sensitivity in our patients was lower than that provided by the manufacturer (93.3% overall and 98.2% in those with Ct≤33), but in line with previous studies. Two studies conducted in Spain that included symptomatic patients who attended a PHC center and hospital emergency wards reported the overall test sensitivity was 79.6% and the sensitivity was 86.5% in those with high viral loads14 , 15. Other studies of symptomatic patients in the Netherlands and Switzerland found similar test sensitivity (85.4%, 81.0%, and 72.6%) and a positive correlation between Ag-RDT positivity and SARS-CoV-2 viral load16 , 17. A study in France that examined positive and negative frozen RT-PCR samples reported a much lower overall sensitivity of only 35.3%18; however, test sensitivity was greater in samples collected within 3 days since the onset of the symptoms and in samples from patients with high viral loads. Two preprints (non peer-reviewed studies) reported test sensitivities of 91.7%19 and 97.1%10. An evaluation report in Spain performed in symptomatic patients who visited hospital emergency rooms reported a sensitivity of 98.2%20. All these previous studies reported excellent test specificity.
Policy implications
Our results support the use of the Ag-RDT for symptomatic patients within 5 days since the onset of the symptoms in PHC setting. A positive Ag-RDT result may be considered as an active COVID-19 infection based on high specificity and PPV data. Even though the interim WHO guideline recommends use of the Ag-RDT for testing in low-income countries or when RT-PCR testing is not available, several European countries have included this new diagnostic tool as part of their national testing strategy (Belgium, France, Germany, Greece, Italy, and others)21. Validation reports and pilot studies are being conducted to establish the diagnostic performance of these tests, and this may lead to future changes in the indications for the Ag-RDT16 , 18.
Our data do not support the sole use of the Ag-RDT in asymptomatic individuals, as a negative result does not exclude the disease. We obtained an overall NPV of 96.8% in a population that had a prevalence of 10.2%, so at least 3 of 100 persons were potentially missed. Moreover, the false-negative Ag-RDT results obtained in our sample were mostly individuals who had close contact with a COVID-19 confirmed patient, were asymptomatic and most important, 12/36 participants with negative Ag-RDT result but positive RT-PCR, had high viral loads. Several other studies found that the Ag-RDT missed diagnoses in some patients with high viral loads14 , 16 , 22. This might have a substantial impact from a public health point of view, because a negative Ag-RDT result especially in individuals with high pretest probabilities must be interpreted cautiously, and a confirmatory test should be considered5.
There are several new approaches used to overcome the low sensitivity of the Ag-RDT, in an effort to incorporate this test as a reliable diagnostic tool for massive testing to be used for monitoring and controlling outbreaks. One approach is the complementary use of the Ag-RDT with clinical diagnostic evaluations and another approach is the use of repeated testing23. A recent study using the enhanced epidemiological SIDHRE-Q model concluded that frequent Ag-RDT testing overcame the limitation of low test sensitivity, suggesting this might be an effective method to control SARS-CoV-2 transmission24. However, future research must confirm these findings in real-world settings, as not all diagnostic tests are useful for screening25.
Our results show that a point-of-care Ag-RDT has good performance characteristics in suspected symptomatic patients within five days since the onset of symptoms in PHC. However, a negative Ag-RDT result must be considered presumptive when the pretest probability is high, and a confirmatory test might be required. Further studies are needed to examine the accuracy of the Ag-RDT in different settings with lower pretest probabilities.
Author's contributions
OB, PL, AL, EC and JL designed and implemented the study, validated and analyzed the data, wrote the draft and revised the paper.
AR and AO undertook PCR, coordinated training for antigen test performance and participate in the protocol design, monitored the whole data collection, revised the draft paper.
PP designed the collection tools design, validated and analyzed the data, revised the drafted paper.
The members of the COVID-19 Primary Care Research Group design the protocol, implemented the study, monitored the data collection and revised the draft paper.
All authors approved the final version of the paper.
Funding
This research did not receive any specific grant from funding agencies in the public, commercial or not-for-profit sectors. The Balearic Public Health Service (Servei de Salut de les Illes Balears) had endorsed the study, bought Panbio™ COVID-19 Antigen Rapid Test Device and provided the human and logistic resources for its accomplishment. No external funding was received. Abbott had no role in the design, study management, data analysis, result interpretation or in the writing of the paper.
Ethics
This study was conducted following the Declaration of Helsinki and was approved by the Balearic Research Ethics Committee (IB 4350/20 PI on 30/09/2020) and by the Mallorca Primary Care Research Commission. All participants had signed the informed consent before inclusion.
Data sharing
The anonymized database set is available at http://doi.org/10.5281/zenodo.4264502.
Declaration of Competing Interest
The authors declared no conflict of interest.
Acknowledgments
We thank the nurses from primary health care centers and test sites that collected samples and performance the antigen test on- site without whom this work would not be possible, and all participants of the study that voluntarily participate in the study.
Footnotes
COVID-19 Primary Care Research Group members (alphabetical order)
Layla Aoukhiyad – pharmacist, Servei de Salut de les Illes Balears; email: layla.aoukhiyad@ibsalut.es
Ricardo Arcay – medical microbiologist resident, Son Espases University Hospital Microbiology Unit; email: ricardo.arcay@ssib.es
Javier Arranz – general practitioner, Escola Graduada Primary Care Health Center; email: jarranz@ibsalut.caib.es
Irene Barrionuevo – nurse, Son Espases University Hospital Emergency Unit; email: irenem.barrionuevo@ssib.es
Bernardino Comas – general practitioner, Son Espases University Hospital Emergency Unit; email: bernardino.comas@ssib.es
Sara Contreras – biologist, Primary Health Care Research Unit; email: scontreras3@ibsalut.caib.es
María Fernandez-Billon – medical microbiologist resident, Son Espases University Hospital Microbiology Unit; email: maria.fernandez-billon@ssib.es
J. Luis Ferrer – nurse, Santa Ponça Primary Health Care Center; email: jlferrer@ibsalut.caib.es
Pablo Fraile – medical microbiologist resident, Son Espases University Hospital Microbiology Unit; email: pablo.fraile@ssib.es
Marina García – general practitioner, Inca Primary Care Health Center; email: marinagarcia@ibsalut.caib.es
Carla Iglesias – medical microbiologist resident, Son Espases University Hospital Microbiology Unit; email: carla.iglesias@ssib.es
Miquel Llobera – nurse, Santa Ponça Primary Health Care Center; email: mllobera@ibsalut.caib.es
M. Consolación Méndez – director of nursing, Mallorca Primary Health Care Directorate; email: mmendez@ibsalut.caib.es
Susana Munuera – general practitioner, Servei de Salut de les Isles Balears; email: susana.munuera@ibsalut.es
Victoria Pascual – nurse, COVID Express Testing Center; email: avpascual@ibsalut.caib.es
Carlos Raduan – general practitioner medical director, Mallorca Primary Health Care Directorate; email: craduan@ibsalut.caib.es
Ana Requena – nurse, Coll d'en Rabassa Primary Care Health Center; email: arequena2@ibsalut.caib.es
Antonia Roca – general practitioner medical director, Mallorca Primary Health Care Directorate; email: aroca@ibsalut.caib.es
Pedro Rull – general practitioner, Son Espases University Hospital Emergency Unit; email: pere.rull@ssib.es
Silvia Vallcaneras – nurse, Alcudia Primary Care Health Center; email: svallcaneras@ibsalut.caib.es
Verónica Vega – nurse, COVID Express Testing Center. email: vvega@ibsalut.caib.es
Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.jinf.2021.02.014.
Appendix. Supplementary materials
References
- 1.Vandenberg O., Martiny D., Rochas O. Considerations for diagnostic COVID-19 tests. Nat Rev Microbiol [Internet] 2020 doi: 10.1038/s41579-020-00461-z. Oct 14; Available from. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.European Centre for Disease Prevention and Control. Rapid Risk Assessment: increased transmission of COVID-19 in the EU/EEA and the UK – thirteenth update [Internet]. 2020 [cited 2020 Nov 6]. Available from: https://www.ecdc.europa.eu/en/publications-data/covid-19-risk-assessment-increased-transmission-thirteenth-update
- 3.Dinnes J., Deeks J.J., Adriano A., Berhane S., Davenport C., Dittrich S. Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection. Cochrane Database Syst Rev. 2020;2020(August 26(8)) doi: 10.1002/14651858.CD013705. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.World Health Organization. Antigen-detection in the diagnosis of SARS-CoV-2 infection using rapid immunoassays. 2020 [cited 2020 Sep 18];369 (6508):1167–8. Available from: https://www.who.int/publications/i/item/antigen-detection-in-the-diagnosis-of-sars-cov-2infection-using-rapid-immunoassays
- 5.Center for Disease Control and Prevention. Interim Guidance for Rapid Antigen Testing for SARS-CoV-2 CDC. 2020; Available from: https://www.cdc.gov/coronavirus/2019-ncov/lab/resources/antigen-tests-guidelines.html
- 6.World Health Organization. Global partnership to make available 120 million affordable, quality COVID-19 rapid tests for low- and middle-income countries [Internet]. 2020. Available from: https://www.who.int/news-room/detail/28-09-2020-global-partnership-to-make-available-120-million-affordable-quality-covid-19-rapid-tests-for-low–and-middle-income-countries
- 7.FINDDX. Rapid Diagnostic Tests for Covid-19 [Internet]. FIND because diagnosis matters. 2020. Available from: https://www.finddx.org/covid-19/sarscov2-eval-antigen/
- 8.Panbio COVID-19 Ag Rapid Test Device | Abbott Point of Care Testing [Internet]. Available from: https://www.globalpointofcare.abbott/es/product-details/panbio-covid-19-ag-antigen-test.html
- 9.Krüger L., Gaeddert M., Köppel L., Denkinger CM for the study team Evaluation of the accuracy, ease of use and limit of detection of novel, rapid, antigen-detecting point-of-care diagnostics for SARS-CoV-2. medRxiv. 2020 doi: 10.1101/2020.10.01.20203836. [Internet]2020.10.01.20203836. Available from. [DOI] [Google Scholar]
- 10.van Beek J., Igloi Z., Boelsums T., Fanoy E., Gotz H., Molenkamp R. From more testing to smart testing: data-guided SARS-CoV-2 testing choices. medRxiv. 2020 doi: 10.1101/2020.10.13.20211524. [Internet]2020.10.13.20211524. Available from. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Canadian Agency for Drugs and Technologies in Health. Rapid Point-of-Care Antigen Testing for SARS-CoV-2 Infection [Internet]. 2020. Available from: https://www.canada.ca/en/public-health/services/diseases/2019-novel-coronavirus-infection/guidance-documents/use-rapid-antigen-detection-tests.html
- 12.Candel F.J., Barreiro P., San Román J., Abanades J.C., Barba R., Barberán J. Recommendations for use of antigenic tests in the diagnosis of acute SARS-CoV-2 infection in the second pandemic wave: attitude in different clinical settings. Rev Española Quimioter [Internet] 2020 doi: 10.37201/req/120.2020. Available from. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.García F., Melón S., Navarro D., Paño J.R., Galán J.C. Organización del diagnóstico de SARS-CoV-2 y estrategias de optimización [Internet]. 2020. Available from: https://www.seimc.org/documentos-cientificos/recomendaciones-institucionales
- 14.Albert E., Torres I., Bueno F., Huntley D., Moya E., Fernandez-Funtes M.A., et al. Field evaluation of a rapid antigen test (Panbio COVID-19 Ag Rapid Test Device) for the diagnosis of COVID-19 in primary healthcare centers. medRxiv [Internet]. 2020;pre-print (96):2020.10.16.20213850. Available from: https://doi.org/10.1101/2020.10.16.20213850
- 15.Linares M., Pérez-Tanoira R., Carrero A., Romanyk J., Pérez-García F., Gómez-Herruz P. Panbio antigen rapid test is reliable to diagnose SARS-CoV-2 infection in the first 7 days after the onset of symptoms. J Clin Virol [Internet] 2020;133 doi: 10.1016/j.jcv.2020.104659. Available from. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Centre for Emerging Viral Diseases Université de Geneve. Validation report: SARS-CoV-2 Antigen Rapid Diagnostic Test [Internet]. 2020. Available from: https://www.hug.ch/sites/interhug/files/structures/laboratoire_de_virologie/documents/Centre_maladies_virales_infectieuses/ofsp_rdt_report_gcevd_27.10.2020.pdf
- 17.van der Moeren N., Zwart V., Lodder E., den Bijllaardt W Van, Van Esch H, Stohr J. Performance evaluation of a SARS-Cov-2 rapid antigentest test performance in the community in the Netherlands. Medrxiv [Internet] 2020:1–24. doi: 10.1101/2020.10.13.20211524. Available from. [DOI] [Google Scholar]
- 18.Fourati S., Audureau É., Chevaliez S., Pawlotsky J.-.M. Évaluation de la performance diagnostique des tests rapides d ’ orientation diagnostique antigéniques [Internet]. 2020. Available from: https://www.aphp.fr/contenu/evaluation-de-la-performance-diagnostique-des-tests-rapides-dorientation-diagnostique
- 19.Alemany A., Baro B., Ouchi D., Ubals M., Corbacho-monné M. Analytical and Clinical Performance of the Panbio COVID-19 Antigen-Detecting Rapid Diagnostic. 2020; https://doi.org/10.1101/2020.10.30.20223198 [DOI] [PMC free article] [PubMed]
- 20.Comisión de Salud Pública del Consejo Interterritorial del Sistema Nacional de Salud. Estrategia de detección precoz, vigilancia y control de COVID-19. Minist Sanid Gob España Inst Salud Carlos III [Internet]. 2020;1–16. Available from: https://www.mscbs.gob.es/profesionales/saludPublica/ccayes/alertasActual/nCov/documentos.htm
- 21.European Commission. Commission Recommendation of 28.10.2020 on COVID-19 testing strategies, including the use of rapid antigen tests. 2020; Available from: https://ec.europa.eu/info/live-work-travel-eu/health/coronavirus-response/public-health_en
- 22.Gremmels H., Winkel B.M.F., Schuurman R., Rosingh A., Rigter N.A.M., Rodriguez O. Real-life validation of the Panbio COVID-19 antigen rapid test (Abbott) in community-dwelling subjects with symptoms of potential SARS-CoV-2 infection. medRxiv. 2020 doi: 10.1101/2020.10.16.20214189. [Internet]2020.10.16.20214189. Available from. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Mina M.J., Parker R., Larremore D.B. Rethinking COVID-19 test sensitivity — a strategy for containment. N Engl J Med. 2020:1–2. doi: 10.1056/NEJMp2025631. [DOI] [PubMed] [Google Scholar]
- 24.Nash B., Badea A., Reddy A., Bosch M., Salcedo N., Gomez A.R. The impact of high frequency rapid viral antigen screening on COVID-19 spread and outcomes: a validation and modeling study 2 3. medRxiv. 2020 2020.09.01.20184713. [Google Scholar]
- 25.Raffle A.E., Pollock A.M., Harding-Edgar L. Covid-19 mass testing programmes. BMJ. 2020;370:m3262. doi: 10.1136/bmj.m3262. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.