Key Points
Question
Can untrained persons correctly perform and interpret the results of SARS-CoV-2 rapid antigen detection tests (RADT), and can performance be optimized?
Findings
In this cross-sectional study of 278 participants self-performing SARS-CoV-2 RADT in an intended-use setting, the accuracy of RADT interpretation was poor when the manufacturer’s instructions were used. A modified quick reference guide was associated with significantly better user performance.
Meaning
These findings suggest that longitudinal mass RADT testing for SARS-CoV-2 could be accurately self-performed in an intended-use setting but there are potential interventions to optimize performance.
This cohort study assesses the execution and interpretation of SARS-CoV-2 rapid antigen detection tests when performed by untrained persons an intended-use setting.
Abstract
Importance
Longitudinal mass testing using rapid antigen detection tests (RADT) for serial screening of asymptomatic persons has been proposed for preventing SARS-CoV-2 community transmission. The feasibility of this strategy relies on accurate self-testing.
Objective
To quantify the adequacy of serial self-performed SARS-CoV-2 RADT testing in the workplace, in terms of the frequency of correct execution of procedural steps and accurate interpretation of the range of possible RADT results.
Design, Setting, and Participants
This prospective repeated cross-sectional study was performed from July to October 2021 at businesses with at least 2 active cases of SARS-CoV-2 infection in Montreal, Canada. Participants included untrained persons in their workplace, not meeting Public Health quarantine criteria (ie, required quarantine for 10 days after a moderate-risk contact with someone infected with SARS-CoV-2). Interpretation and performance were compared between participants who received instructions provided by the manufacturer vs those who received modified instructions that were informed by the most frequent or most critical errors we observed. Data were analyzed from October to November 2021.
Exposures
RADT testing using a modified quick reference guide compared with the original manufacturer’s instructions.
Main Outcomes and Measures
The main outcome was the difference in correctly interpreted RADT results. Secondary outcomes included difference in proportions of correctly performed procedural steps. Additional analyses, assessed among participants with 2 self-testing visits, compared the second self-test visit with the first self-test visit using the same measures.
Results
Overall, 1892 tests were performed among 647 participants, of whom 278 participants (median [IQR] age, 43 [31-55] years; 156 [56.1%] men) had at least 1 self-testing visit. For self-test visit 1, significantly better accuracy in test interpretation was observed among participants using the modified quick reference guide than those using the manufacturer’s instructions for reading results that were weak positive (64 of 115 participants [55.6%] vs 20 of 163 participants [12.3%]; difference, 43.3 [95% CI, 33.0-53.8] percentage points), positive (103 of 115 participants [89.6%] vs 84 of 163 participants [51.5%]; difference, 38.1 [95% CI, 28.5-47.5] percentage points), strong positive (219 of 229 participants [95.6%] vs 274 of 326 participants [84.0%]; difference, 11.6 [95% CI, 6.8-16.3] percentage points), and invalid (200 of 229 participants [87.3%] vs 252 of 326 participants [77.3%]; difference, 10.0 [95% CI, 3.8-16.3] percentage points). Use of the modified guide was associated with improvements on self-test visit 2 for results that were weak positive (difference, 15.4 [95% CI, 0.7-30.1] percentage points), positive (difference, 19.0 [95% CI, 7.2-30.9] percentage points), and invalid (difference, 8.0 [95% CI, 0.8-15.4] percentage points). For procedural steps identified as critical for test validity, adherence to procedural testing steps did not differ meaningfully according to instructions provided or reader experience.
Conclusions and Relevance
In this cross-sectional study of self-performed SARS-CoV-2 RADT in an intended-use setting, a modified quick reference guide was associated with significantly improved accuracy in RADT interpretations.
Introduction
Ongoing transmission of SARS-CoV-2 is frequently driven by asymptomatic or presymptomatic individuals.1,2 Mass deployment of lateral-flow rapid antigen detection tests (RADT) for serial screening of asymptomatic persons has been proposed for preventing community transmission, with the aim of promptly detecting individuals most likely to be infectious.3,4,5,6 This strategy may be even more relevant with the emergence of the Omicron (B.1.1.529) variant7 that may evade immunity afforded by vaccination.
The feasibility of longitudinal mass testing relies on implementation of self-performed RADT where people live, work, or attend school. However, several factors impede this deployment. These include regulatory requirements that RADT be performed by trained health care workers in some countries8 and lack of data on the accuracy of self-administration in their intended-use settings.9 Similarly, to our knowledge, no studies have assessed the impact of repeated lateral-flow testing on user performance.
In this study, we report the results of a prospective field evaluation of serial self-performed RADT for SARS-CoV-2 among untrained, asymptomatic persons in their workplaces. The primary objective was to quantify the adequacy of self-testing, in terms of the frequency of correct execution of procedural steps and the accuracy of interpretation of the range of possible RADT results. We also aimed to compare results using the instructions provided by the manufacturer with results using modified instructions that were informed by the most frequent or most critical errors we observed.
Methods
This cross-sectional study was approved by the Research Ethics Board of the Research Institute of the McGill University Health Centre. All participants provided written informed consent. The study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline for cross-sectional studies.
Study Design and Setting
In this prospective, repeated, cross-sectional study, businesses in Montreal, Canada, with at least 2 active cases of SARS-CoV-2 infection within a 14-day period were identified by the Montreal Department of Public Health. Businesses with newly identified outbreaks were contacted by the study team by telephone and offered participation in the study via the deployment of 1 of 4 mobile teams on-site. Among outbreak businesses, priority was given to contacting those with greater than 50 employees to increase the probability of identifying participants with unsuspected SARS-CoV-2 infection.10 For businesses that agreed to participate, study visits were scheduled twice weekly for 2 weeks (4 visits). Study data were collected and managed using REDCap.11
Any worker in outbreak businesses who could provide written informed consent was considered eligible to participate. Participants were not compelled by the study team or by their employer to attend all 4 study visits and could withdraw from further testing if they no longer wished to participate.
Data Collection
Data on self-reported participant age, gender, race and ethnicity, language preference (English or French), COVID-19 symptoms, exposures, and vaccination were collected. Race and ethnicity were self reported and categorized as Asian, Black, Hispanic, Indigenous, mixed, White, and other (including Algerian, Arab, Bulgarian, Canadian, Chilean, Guatemalan, Haitian, Italian, Mediterranean, Moroccan, Peruvian, and Portuguese). Race and ethnicity and other sociodemographic measures were recorded to account for the heterogeneity of SARS-CoV-2 community transmission. Additionally, we assessed testing results, adequacy of performance of test procedures, and result interpretation.
Diagnostic Testing Procedures
The RADT used was the Panbio COVID-19 Ag Rapid Test Device (Abbott Laboratories). At each visit, trained study personnel instructed participants on procedures to perform self-collected nasal midturbinate swab specimens. On visits 1 and 3, all postspecimen collection steps and result interpretations were performed by study personnel. On visits 2 (first self-testing visit) and 4 (second self-testing visit), these steps were performed by the participant. During self-testing, study personnel assessed 13 RADT procedural steps performed by the participant as correct or incorrect. If any of 6 steps deemed to be critical to testing integrity were assessed as incorrectly performed, the participant was informed at the end of the procedure and testing was redone by study personnel. Participants with a positive RADT result were offered confirmatory laboratory-based polymerase chain reaction (PCR) testing at a local testing center.
Result Interpretation Using a Proficiency Panel
A proficiency panel of 7 RADT test results was created using serial dilutions of the positive control reagent and aimed to evaluate participants’ ability to interpret the range of possible signal intensities of the test line, including results classified as weak positive, positive, strong positive, invalid, and negative (eFigure 1 in the Supplement). The panel was printed in color at actual size and high resolution on cardboard. At visits 2 and 4, participants were asked to interpret the 7 RADT results, in addition to their own RADT result.
Intervention
Participants were initially given access to the complete manufacturer’s instructions, the visual quick reference guide (eFigure 2 in the Supplement), and no time limit during self-testing visits. Study personnel did not give any additional guidance. Written authorization from the manufacturer was obtained for reproduction of their quick reference guide in this study. Following a planned interim analysis, we developed a modified quick reference guide (eFigure 3 in the Supplement) that addressed the most frequently observed user errors in RADT procedural steps and interpretation, and we deployed it subsequently.
The salient elements of the modified quick reference guide included the addition of clear and simple text to the pictograms in the manufacturer’s instructions. Critical procedural steps were emphasized with bold red text. These stressed using the correct quantity of buffer to specimen tubes, adding the recommended volume of specimen to the rapid test device, and reading the result at the prescribed time interval. Test interpretation was guided by a series of questions and tailored images: “1. Is the test INVALID?”; “2. Is the test POSTIVE?”; and “3. Is the test NEGATIVE?” Prior to implementation in the study, linguistic elements in both English- and French-language versions of the modified quick reference guide were refined through consultation with colleagues who work mainly with migrants and refugees not fluent in either language.
Outcomes
Among all participants with at least 1 self-testing visit, we compared the group using our modified quick reference guide with the group using the original manufacturer’s instructions. Among participants with 2 self-testing visits, we compared the second self-test visit with the first self-test visit. For each comparison, our primary outcome was the difference in the proportion of correctly interpreted RADT proficiency panel results. Our secondary outcome was the difference in proportion of correctly performed procedural steps.
Sample Size Estimate
Recruitment of participants using the manufacturer’s instructions (study phase 1) proceeded until the overall weighted mean of the accurate readings could be estimated with 6.0% precision at the 95% confidence level. A modified quick reference guide informed by the analysis of phase 1 data was expected to yield at least a 12.0–percentage point improvement in RADT reading accuracy vs the manufacturer's instructions. Based on an expected overall weighted mean of the accurate readings of 60.0% (95% CI, 54.0%-66.0%) in phase 1, the number of readings required to achieve an accuracy of 72.0% with 6.0% precision was 216 readings per result category (invalid, negative, weak positive, positive, and strong positive). At the rate of 2 readings per person per visit, this corresponds to a recruitment of 108 people using the modified instructions.
Statistical Analysis
Baseline characteristics were summarized according to self-testing visit and reference guide type. Estimates of 95% CIs around a proportion or around the difference of 2 proportions were calculated according to a binomial distribution using the Wilson Score method. We used the term sensitivity to refer to the proportion of accurately identified RADT result interpretations of positive and of invalid results; specificity referred to accurate RADT result interpretation of negative results.
Generalized linear mixed-effects models were fit to evaluate the association of the use of the modified quick reference guide with accurate RADT proficiency panel result interpretation. Outbreak business and participant were modeled as a random intercepts; participant age (in years), gender, RADT result type (positive, strong positive, negative, or invalid) and an interaction term between RADT result type and instruction type were included as covariates (fixed effects).
Data were analyzed using R statistical software version 3.5.2 (R Project for Statistical Computing). Because of the small number of variables to be analyzed and because missingness was no greater than 5%, a complete case analysis was performed for the modeling. Statistical significance was assessed by using 2-tailed tests, with α = .05. Data were analyzed from October to November 2021.
Results
From July 7, to October 8, 2021, 236 outbreak businesses were identified (Table 1; Figure 1); 168 businesses (71.2%) were contacted and 13 businesses (5.5%) participated. The most frequently cited reason for declining participation was a disruption to work shifts. Overall, 1892 tests were performed among 647 participants, with a mean (SD) of 2.9 (1.26) visits per participant, of which 278 participants (median [IQR] age, 43 [31-55] years; 156 [56.1%] men) had at least 1 self-testing visit, including 115 participants with 198 visits with the modified quick reference guide) (Table 2; eTable in the Supplement). RADT result was positive in 3 of 1892 tests (0.2%), of which 1 result was confirmed positive and 2 were found to be negative by PCR testing.
Table 1. Business and Participant Recruitment.
Business or participant | No. |
---|---|
Businesses with outbreaks identified by Public Health | 236 |
Businesses with outbreaks contacted by study team | 168 |
Businesses with outbreaks visited by study team | 13 |
Participants tested | 647 |
Participants with ≥1 dose of any COVID-19 vaccine at time of testing, No. (%) | 565 (89.1)a |
Participants who underwent ≥1 self-test | 278 |
Participant self-testing visits | 451 |
Participant testing visits | |
Total | 1892 |
With positive rapid test results, No (%) | 3/1892 (0.16) |
Positive rapid test results confirmed by PCR, No. (%) | 1/1892 (0.05)b |
Abbreviation: PCR, polymerase chain reaction.
Among 634 of 647 participants who answered the question about vaccination status.
All positive test results underwent confirmatory PCR testing on the same day according to the study protocol.
Table 2. Self-reported Demographic Characteristics Among Participants With at Least 1 Self-testing Visit.
Characteristic | Participants, No. (%) (N = 278) |
---|---|
Age | |
Median (IQR), y | 43 (31-55) |
Not reported | 14 (5.0) |
Gender | |
Female | 109 (39.2) |
Male | 156 (56.1) |
Do not wish to disclose | 1 (0.3) |
Not reported | 12 (4.3) |
Language preference | |
English | 38 (13.7) |
French | 227 (81.6) |
Not reported | 13 (4.7) |
Business type | |
Manufacturing | 43 (15.5) |
Office | 196 (70.5) |
Retail | 27 (9.7) |
Not reported | 12 (4.3) |
Race and ethnicity | |
Asian | 10 (3.6) |
Black | 12 (4.3) |
Hispanic | 14 (5.0) |
Indigenous | 0 |
Mixed | 5 (1.8) |
White | 202 (72.7) |
Othera | 22 (7.9) |
Not reported | 13 (4.7) |
Health conditions and habitsb | |
Chronic respiratory condition | 17 (6.1) |
Current cigarette smoking | 32 (11.5) |
Past cigarette smoking | 41 (14.7) |
Diabetes | 12 (4.3) |
Heart disease | 6 (2.1) |
Hypertension | 17 (6.1) |
None of the above | 145 (52.1) |
Other | 16 (5.7) |
Symptoms reported on the day of recruitmentc | |
Aches and pains | 7 (2.5) |
Chest pain or pressure | 2 (0.7) |
Diarrhea | 2 (0.7) |
Difficulty breathing or shortness of breath | 4 (1.4) |
Dry cough | 3 (1.1) |
Fatigue | 20 (7.2) |
Fever | 1 (0.3) |
Headache | 12 (4.3) |
Loss of smell or taste | 1 (0.3) |
Rash on skin or discoloration of fingers or toes | 2 (0.7) |
Sore throat | 12 (4.3) |
None | 199 (71.6) |
Other | 2 (0.7) |
Vaccination against COVID-19 | 237 (85.2) |
1 Dose received | 27 (9.7) |
2 Doses received | 210 (75.5) |
Other self-reported race and ethnicity included Algerian, Arab, Bulgarian, Canadian, Chilean, Guatemalan, Haitian, Italian, Mediterranean, Moroccan, Peruvian, and Portuguese.
Participants could self-report 1 or more health condition. Other conditions included cystic fibrosis, Crohn disease, hyperthyroidism, irritable bowel syndrome, and allergies.
Participants could self-report 1 or more symptoms. Other symptoms included runny nose and wet cough.
At the first self-test visit, significantly better accuracy in interpreting proficiency panel RADT results was observed among participants using the modified quick reference guide compared with those using the manufacturer’s instructions for results that were weak positive (64 of 115 participants [55.6%] vs 20 of 163 participants [12.3%]; difference, 43.3 [95% CI, 33.0-53.8] percentage points), positive (103 of 115 participants [89.6%] vs 84 of 163 participants [51.5%]; difference, 38.1 [95% CI, 28.5-47.5] percentage points), strong positive (219 of 229 participants [95.6%] vs 274 of 326 participants [84.0%]; difference, 11.6 [95% CI, 6.8-16.3] percentage points), and invalid (200 of 229 participants [87.3%] vs 252 of 326 participants [77.3%]; difference, 10.0 [95% CI, 3.8-16.3] percentage points). Use of the modified guide, compared with using the manufacturer’s instructions, was associated with statistically significant improvements on self-test visit 2 for results that were weak positive (46 of 83 participants [55.4%] vs 36 of 90 participants [40.0%]; difference, 15.4 [95% CI, 0.7-30.1] percentage points), positive (73 of 83 participants [87.9%] vs 62 of 90 participants [68.9%]; difference, 19.0 [95% CI, 7.2-30.9] percentage points), and invalid (149 of 166 participants [89.7%] vs 147 of 180 participants [81.7%]; difference, 8.0 [95% CI, 0.8-15.4] percentage points) (Figure 2).
In multivariable models accounting for outbreak business and participant age and gender, the modified quick reference guide was associated with higher accuracy in result interpretation for self-test visit 1 (adjusted odds ratio, 13.7 [95% CI, 4.9-37.8]) but not for self-test visit 2 (adjusted odds ratio, 0.9 [95% CI, 0.1-5.5]). For procedural steps identified as critical for the validity of test results, adherence did not differ meaningfully according to instructions provided or reader experience (Figure 3).
Discussion
In this prospective cross-sectional study of self-performed SARS-CoV-2 RADT in an intended-use setting, the accuracy of signal interpretation was proportional to test-band intensity and was extremely low for tests with weak bands. An evidence-based modified quick reference guide improved signal interpretation by untrained workers to a level of accuracy similar to that of professional technologists for other RADTs12,13 but was not associated with improving the proper execution of procedural steps. Repeated self-testing was associated with improved signal interpretation, although the difference was less apparent with modified instructions, as initial accuracy was already high. These results complement recent data on how consumers act on the results of at-home COVID-19 self-test kits.14
Self-testing using RADT has been studied for other infections. Among users attending a pretravel clinic, only 70.6% of all malaria RADT strips were correctly interpreted by untrained users.15 In several studies of malaria RADT, weak positive test bands consistently have correlated with lower sensitivity.12,13,15 Our study agrees with these findings for SARS-CoV-2 RADT, and provides evidence for a simple and effective potential corrective intervention.
It is notable that during the study period, Health Canada had not yet authorized self-performance of RADT by nonprofessionals. Because of this, the instructions for use of the RADT used in this test and all other RADT sold in Canada during this period stated for professional use only. These instructions may differ from those for the same products sold in other countries. Also, while the RADT used in this study is not available in the United States, several comparable products are sold on the US market.
Study strengths include the recruitment of participants from diverse racial and ethnic, socioeconomic, and educational backgrounds. The modified quick reference guide was informed by the most frequent or most critical errors observed and validated with different participants. Further, we assessed the independent outcomes associated with modified instructions and repeated self-testing.
Limitations
This study has some limitations. One limitation was the small proportion of eligible businesses that entered the study. This proportion depended on several factors other than the agreement of the businesses to participate, such as our team’s capacity (ie, our four 2-person mobile teams could not always visit all the businesses that agreed to participate) and the decision not to attend businesses that agreed to participate but where the last recorded COVID-19 case was more than 14 days earlier than the day our teams could visit. Additionally, a single RADT was assessed in this study, with uncertain generalizability to the full diversity of available RADTs.
Conclusions
This cross-sectional study found that longitudinal mass SARS-CoV-2 RADT testing could be accurately self-performed and provides evidence for a potential intervention to optimize performance. Self-administered RADTs are inexpensive and can be decentralized and implemented at scale for serial mass testing of asymptomatic persons to break chains of transmission and reduce SARS-CoV-2 incidence.16,17 This use case may become more pertinent with the emergence of new SARS-CoV-2 variants of concern.
References
- 1.World Health Organization . WHO Coronavirus (COVID-19) Dashboard. Accessed November 23, 2021. https://covid19.who.int
- 2.Li R, Pei S, Chen B, et al. Substantial undocumented infection facilitates the rapid dissemination of novel coronavirus (SARS-CoV-2). Science. 2020;368(6490):489-493. doi: 10.1126/science.abb3221 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Mina MJ, Parker R, Larremore DB. Rethinking COVID-19 test sensitivity—a strategy for containment. N Engl J Med. 2020;383(22):e120. doi: 10.1056/NEJMp2025631 [DOI] [PubMed] [Google Scholar]
- 4.Mina MJ, Andersen KG. COVID-19 testing: one size does not fit all. Science. 2021;371(6525):126-127. doi: 10.1126/science.abe9187 [DOI] [PubMed] [Google Scholar]
- 5.Mina MJ, Peto TE, García-Fiñana M, Semple MG, Buchan IE. Clarifying the evidence on SARS-CoV-2 antigen rapid tests in public health responses to COVID-19. Lancet. 2021;397(10283):1425-1427. doi: 10.1016/S0140-6736(21)00425-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Petersen I, Crozier A, Buchan I, Mina MJ, Bartlett JW. Recalibrating SARS-CoV-2 antigen rapid lateral flow test relative sensitivity from validation studies to absolute sensitivity for indicating individuals shedding transmissible virus. Clin Epidemiol. 2021;13:935-940. doi: 10.2147/CLEP.S311977 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.World Health Organization . Classification of Omicron (B.1.1.529): SARS-CoV-2 variant of concern. Accessed November 27, 2021. https://www.who.int/news/item/26-11-2021-classification-of-omicron-(b.1.1.529)-sars-cov-2-variant-of-concern
- 8.Health Canada . Authorized medical devices for uses related to COVID-19: list of authorized testing devices. Accessed November 24, 2021. https://www.canada.ca/en/health-canada/services/drugs-health-products/covid19-industry/medical-devices/authorized/list.html
- 9.Dinnes J, Deeks JJ, Adriano A, et al. ; Cochrane COVID-19 Diagnostic Test Accuracy Group . Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection. Cochrane Database Syst Rev. 2020;8(8):CD013705. doi: 10.1002/14651858.CD013705 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Campbell JR, Dion C, Uppal A, Yansouni CP, Menzies D. Systematic testing for SARS-CoV-2 infection among essential workers in Montréal, Canada: a prospective observational and cost assessment study. medRxiv. Preprint posted online May 14, 2021. doi: 10.1101/2021.05.12.21256956 [DOI] [PMC free article] [PubMed]
- 11.Harris PA, Taylor R, Minor BL, et al. ; REDCap Consortium . The REDCap consortium: building an international community of software platform partners. J Biomed Inform. 2019;95:103208. doi: 10.1016/j.jbi.2019.103208 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Mukadi P, Gillet P, Lukuka A, et al. External quality assessment of reading and interpretation of malaria rapid diagnostic tests among 1849 end-users in the Democratic Republic of the Congo through Short Message Service (SMS). PLoS One. 2013;8(8):e71442. doi: 10.1371/journal.pone.0071442 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Vernelen K, Barbé B, Gillet P, Van Esbroeck M, China B, Jacobs J. Photo-based external quality assessment of malaria rapid diagnostic tests in a non-endemic setting. PLoS One. 2018;13(8):e0201622. doi: 10.1371/journal.pone.0201622 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Woloshin S, Dewitt B, Krishnamurti T, Fischhoff B. Assessing how consumers interpret and act on results from at-home COVID-19 self-test kits: a randomized clinical trial. JAMA Intern Med. 2022;182(3):332-341. doi: 10.1001/jamainternmed.2021.8075 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Trachsler M, Schlagenhauf P, Steffen R. Feasibility of a rapid dipstick antigen-capture assay for self-testing of travellers’ malaria. Trop Med Int Health. 1999;4(6):442-447. doi: 10.1046/j.1365-3156.1999.00419.x [DOI] [PubMed] [Google Scholar]
- 16.Pavelka M, Van-Zandvoort K, Abbott S, et al. ; CMMID COVID-19 working group; Inštitút Zdravotných Analýz . The impact of population-wide rapid antigen testing on SARS-CoV-2 prevalence in Slovakia. Science. 2021;372(6542):635-641. doi: 10.1126/science.abf9648 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Larremore DB, Wilder B, Lester E, et al. Test sensitivity is secondary to frequency and turnaround time for COVID-19 screening. Sci Adv. 2021;7(1):eabd5393. doi: 10.1126/sciadv.abd5393 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.