Head-to-head comparison of the accuracy of saliva and nasal rapid antigen SARS-CoV-2 self-testing: cross-sectional study

Abstract

Background

The diagnostic accuracy of unsupervised self-testing with rapid antigen diagnostic tests (Ag-RDTs) is mostly unknown. We studied the diagnostic accuracy of a self-performed SARS-CoV-2 saliva and nasal Ag-RDT in the general population.

Methods

This large cross-sectional study consecutively included unselected individuals aged $\ge$ 16 years presenting for SARS-CoV-2 testing at three public health service test sites. Participants underwent molecular test sampling and received two self-tests (the Hangzhou AllTest Biotech saliva self-test and the SD Biosensor nasal self-test by Roche Diagnostics) to perform themselves at home. Diagnostic accuracy of both self-tests was assessed with molecular testing as reference.

Results

Out of 2819 participants, 6.5% had a positive molecular test. Overall sensitivities were 46.7% (39.3–54.2%) for the saliva Ag-RDT and 68.9% (61.6–75.6%) for the nasal Ag-RDT. With a viral load cut-off (≥ 5.2 log10 SARS-CoV-2 E-gene copies/mL) as a proxy of infectiousness, these sensitivities increased to 54.9% (46.4–63.3%) and 83.9% (76.9–89.5%), respectively. For the nasal Ag-RDT, sensitivities were 78.5% (71.1–84.8%) and 22.6% (9.6–41.1%) in those symptomatic and asymptomatic at the time of sampling, which increased to 90.4% (83.8–94.9%) and 38.9% (17.3–64.3%) after applying the viral load cut-off. In those with and without prior SARS-CoV-2 infection, sensitivities were 36.8% (16.3–61.6%) and 72.7% (65.1–79.4%). Specificities were > 99% and > 99%, positive predictive values > 70% and > 90%, and negative predictive values > 95% and > 95%, for the saliva and nasal Ag-RDT, respectively, in most analyses. Most participants considered the self-performing and result interpretation (very) easy for both self-tests.

Conclusions

The Hangzhou AllTest Biotech saliva self Ag-RDT is not reliable for SARS-CoV-2 detection, overall, and in all studied subgroups. The SD Biosensor nasal self Ag-RDT had high sensitivity in individuals with symptoms and in those without prior SARS-CoV-2 infection but low sensitivity in asymptomatic individuals and those with a prior SARS-CoV-2 infection which warrants further investigation.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12916-022-02603-x.

Background

A molecular test, mainly real-time reverse-transcriptase polymerase chain reaction (RT-PCR), is considered the reference test for SARS-CoV-2 infection detection [1]. However, molecular tests may take up to 24 h to deliver a result. Tested individuals are asked to quarantine until they receive a result, which has personal and societal consequences. SARS-CoV-2 rapid antigen diagnostic tests (Ag-RDTs) have shown promising diagnostic accuracies [2–6]. These Ag-RDTs require no or minimal equipment, provide a result within 15–30 min and can be performed in a range of settings. While Ag-RDTs were initially introduced for use by trained staff at test sites, they can now also be bought in various outlets for self-testing. Such self-testing, without supervision of a trained professional, may potentially lower the threshold to testing and would allow individuals to obtain a test result quickly and at their own convenience. This in turn could support the early detection of infectious cases and reduce community transmission [7].

Previous studies of self-performed nasal Ag-RDTs showed sensitivities of close to 80% for the Becton Dickinson nasal Ag-RDT in a mixed population of asymptomatic and symptomatic individuals and 82.5% for the SD Biosensor nasal Ag-RDT in symptomatic individuals [8, 9]. Studies employing the SD Biosensor and Abbott Ag-RDTs have shown that supervised nasal self-sampling might be a reliable alternative to nasopharyngeal sampling by a trained professional [10, 11]. However, the sample sizes of these studies were modest, and the nasal self-sampling was supervised instead of self-performed in the home setting. The diagnostic accuracy evidence of self-performed saliva Ag-RDTs is scarce. A recent study found poor performance of four different (unspecified) self-collected saliva Ag-RDTs in a mixed population of symptomatic and asymptomatic individuals, with sensitivities varying from 3.6 to 32.8%, increasing to 5.3% and 41.0% when a “cell culture viability” cut-off was used [12]. A recent saliva Ag-RDT study employing self-sampling supervised by a trained professional and testing by that trained professional in a mixed population of symptomatic and asymptomatic individuals found a sensitivity of 66%, which increased to 89% when a cycle threshold (Ct) < 30 cut-off was used [13].

We conducted a large-scale prospective cross-sectional diagnostic accuracy study in the Netherlands of a self-performed saliva and a self-performed nasal Ag-RDT, using a molecular test as the reference standard for each, by head-to-head comparison. We included individuals presenting for routine SARS-CoV-2 testing at Dutch public health service test sites regardless of their reason for testing, vaccination status, and symptomatology at the time of sampling. A secondary aim was to evaluate user experiences and preferences for both self-performed Ag-RDTs.

Methods

The study is reported according to the STARD 2015 guidelines: an updated list of essential items for reporting diagnostic accuracy studies [14].

Study design and population

This large prospective cross-sectional diagnostic test accuracy study was embedded within the Dutch public testing infrastructure. Public testing in the Netherlands, by default molecular testing, is free-of-charge but only available for government-approved test indications. At the time of the study (9 to 26 September 2021), testing indications included having symptoms of a potential SARS-CoV-2 infection; having been identified as a close contact of a SARS-CoV-2 index case via traditional contact-tracing or the contact-tracing app regardless of symptomatology at the time of notification; having tested positive on a nasal self-test performed outside of this study; or having returned from a country on the government’s list of high risk countries [15]. Participants were recruited consecutively at three Dutch public health service COVID-19 test sites, located in West-Brabant (Roosendaal), Central- and Northeast Brabant (Tilburg), and Rotterdam-Rijnmond (Zuidland). Individuals were considered eligible if they were aged 16 years or older and willing and able to sign a digital informed consent in Dutch.

The study was conducted when the SARS-CoV-2 prevalence was 8.2% with the Delta variant as the dominant variant in the Netherlands (99.9%) during the entire study period [16–18].

Inclusion procedure

Individuals who attended one of the participating test sites for a routine molecular SARS-CoV-2 test were asked by the test site staff whether they were willing to participate. If interested, they received a participant information letter, the saliva and the nasal Ag-RDT together with an instruction manual, and an email with a study participation link to access study documentation. Next, trained test site staff took a swab for routine molecular testing (see below). Participants were asked to provide informed consent electronically via the participation link after arriving home, to subsequently perform both self-tests as soon as possible but within 3 h (the saliva test first, followed by the nasal test), and to complete a short online baseline questionnaire. This included questions on demographics; presence, type, and onset of COVID-19-related symptoms; indication for testing; vaccination status including type of vaccine and vaccination date(s); the results of the two Ag-RDTs that they had just performed; and their user experiences with and opinions about both Ag-RDTs (Additional file: material 1). Participants whose online questionnaire was not completed within 3 h of their test site visit were contacted by a call center with the request to perform both self-tests and complete the questionnaire as soon as possible.

Ten days after their test site visit, participants received an email asking them to complete an online follow-up questionnaire. The follow-up questionnaire included questions on COVID-19-related symptoms and SARS-CoV-2 testing during follow-up (Additional file: material 2) to capture any infections that may have been missed by the baseline molecular test.

Specimen collection and testing

Molecular reference test sampling was performed by trained test site staff. While molecular testing was always used as the reference standard, the three test sites used slightly different sampling methods and the three affiliated centralized laboratories used slightly different molecular testing methods (Additional file: material 3). Briefly, the Roosendaal site combined oropharyngeal-nasal sampling with RT-PCR testing on a Roche cobas 8800 platform. The Tilburg site combined oropharyngeal and nasopharyngeal sampling with the Abbott Alinity M SARS-CoV-2 assay or in-house RT-PCR [19]; samples that tested positive by RT-PCR in Tilburg were subsequently tested on the Roche cobas 8800 platform in Roosendaal to obtain Ct values for viral load calculation. The Zuidland site combined oropharyngeal and nasopharyngeal sampling with RT-PCR on a Roche cobas 6800 platform.

Participants performed the saliva and nasal Ag-RDTs themselves in their own homes (i.e., unsupervised) according to the instructions in the manual that they received at the test site; those instructions were identical to those provided by the manufacturers but translated into Dutch. The saliva Ag-RDT that we used was the COVID-19 Antigen Rapid Test (Oral Fluid) for Self-testing by Hangzhou AllTest Biotech Co. Ltd. and is a so-called spitting test: participants had to spit in a funnel connected to a tube. The nasal Ag-RDT that we used was the SD Biosensor SARS-CoV-2 Rapid Antigen Test Nasal for self-testing, distributed by Roche Diagnostics (Additional file: material 3). Both tests are CE-marked, and the Hangzhou AllTest Biotech was the only saliva self-test with CE-marking at the time of study conception. Participants interpreted their Ag-RDT test results visually in accordance with the instructions, and this interpretation was always done before they had received their molecular test result. Vice versa, the Ag-RDTs results were not available to those in the study team assessing the molecular test results. Participants received their molecular test results from the public health services test site that they attended conform routine practice to direct any further management (such as quarantine advice, if applicable).

Outcomes and statistical analyses

The primary outcomes were the diagnostic accuracies (sensitivity, specificity, positive and negative predictive values with corresponding exact 95% confidence intervals [CI] [20]) of each self-test, with molecular testing as the reference standard. We performed a complete cases analysis because the number of individuals without molecular test or Ag-RDT results was very low (n = 147 (5.0%) for the saliva Ag-RDT, and n = 131 (4.4%) for the nasal Ag-RDT); Fig. 1).

Fig. 1.

Flow of study participants. Saliva = COVID-19 Antigen Rapid Test (Oral Fluid) for Self-testing by Hangzhou AllTest Biotech Co., Ltd., Nasal = SD Biosensor SARS-CoV-2 Rapid Antigen Test Nasal for self-testing by Roche Diagnostics

Secondary outcomes were diagnostic accuracies stratified by presence of symptoms at the time of sampling (yes or no), COVID-19 vaccination status (vaccinated with at least one dose yes or no), having had a prior SARS-CoV-2 infection (yes or no), gender (female or male), and age (≥ 16 to ≤ 40 or > 40 to ≤ 65 or > 65 years). Additional secondary outcomes included all of the above but after using a viral load cut-off as a proxy of infectiousness (≥ 5.2 log10 SARS-CoV-2 E-gene copies/mL), which was the viral load cut-off above which 95% of people with a positive molecular test had a positive virus culture in a recent study by our group [3]. In addition, we assessed the user experiences and preferences for both self-performed Ag-RDTs and tested for overall differences across Ag-RDTs using a McNemar-Bowker test and for differences across categories using a McNemar test.

Finally, using the follow-up questionnaire, we determined whether participants who received a negative molecular test result at baseline had tested positive in the subsequent 10 days by either molecular test or Ag-RDT.

Sample size calculation

Previous nasal Ag-RDTs performance studies in symptomatic individuals found sensitivities around 85% when performed by trained staff [2] and around 80% when used as a self-test [8]. A recent accuracy study in The Netherlands that quantified the accuracy of saliva Ag-RDT performed by trained professionals found an overall sensitivity of 66% and around 89% when using a Ct < 30 cut-off [13]. We therefore based our sample size calculation on an expected sensitivity of 80% for each self-performed Ag-RDT, with a margin of error of 7%, type I error of 5%, and power of 80%. Hence, we aimed for approximately 140 positive molecular reference tests per Ag-RDT evaluation. When planning the study (June 2021), we anticipated a SARS-CoV-2 prevalence (based on molecular test positivity rates) in our target population of around 5%, and closely monitored molecular test positivity rates over time to prolong recruitment as needed.

Results

Between 9 and 26 September 2021, 2950 individuals participated in the study (Fig. 1). An Ag-RDT result with matching molecular reference test result were available for 2803 saliva Ag-RDT users (95.0%) and 2819 nasal Ag-RDT users (95.5%).

The demographic characteristics of study participants are presented in Table 1 and Additional file: Table S1. The mean age was 41 years (standard deviation 15.5) and 61% was female. The majority was vaccinated (85% once; 74% twice), had not had a previous SARS-CoV-2 infection (87%), and was symptomatic at the time of sampling (83%). These characteristics were comparable across test sites, although participants presenting at the Zuidland location were less often vaccinated than those presenting in Roosendaal and Tilburg (76.8% vs. 85.2% vs. 87.5%).

Table 1.

Baseline characteristics of the study population, stratified by type of Ag-RDT

Test result available from the reference test and…	Saliva Ag-RDT	Nasal Ag-RDT
Sample size	N = 2803	N = 2819
Age [years], mean (SD)a	40.9 (15.5)	40.8 (15.5)
Sex, female, n (%)b	1711 (61.3)	1717 (61.1)
Highest level of education, n (%)c
Primary	49 (1.7)	48 (1.7)
Secondary	1329 (47.4)	1332 (47.3)
College/University, Bachelor’s degree	974 (34.7)	988 (35.0)
College/University, Master’s degree	418 (14.9)	416 (14.8)
Vaccination statusd
Not vaccinated	404 (14.4)	421 (14.9)
No information on vaccination status	9 (0.3)	9 (0.3)
Vaccinated with at least one dose, n (%)	2390 (85.3)	2389 (84.7)
Type of vaccine, n (%)e
Astra Zeneca	292 (10.4)	291 (10.3)
Janssen	151 (5.4)	152 (5.4)
Moderna	270 (9.6)	270 (9.6)
Pfizer	1663 (59.3)	1663 (59.0)
Unknown	14 (0.5)	13 (0.5)
Number of vaccinations received, n (%)e
1	308 (11.0)	305 (10.8)
2	2081 (74.2)	2083 (73.9)
Unknown	1 (0.0)	1 (0.0)
At least one prior SARS-CoV-2 infection, n (%)f	366 (13.1)	372 (13.2)
Less than 2 months ago	24 (6.6)	24 (6.5)
2 to 6 months ago	99 (27.0)	100 (26.9)
6 to 12 months ago	209 (57.1)	214 (57.5)
More than 12 months ago	34 (9.3)	34 (9.1)
Testing information, n (%)!
Asymptomatic—close contact of confirmed SARS-CoV-2 infected household member	113 (4.1)	115 (4.1)
Asymptomatic—close contact of other confirmed SARS-CoV-2 infected individual	138 (4.9)	136 (4.8)
Asymptomatic—other	233 (8.4)	236 (8.4)
Symptoms at time of sampling, n (%)g	2306 (82.7)	2319 (82.6)
Symptom onset, n (%)h
On day of sampling	185 (8.0)	187 (8.0)
A day before sampling	931 (40.1)	932 (40.0)
Two days before sampling	672 (29.0)	684 (29.3)
Three or more days before sampling	518 (22.3)	516 (22.1)
Unknown	13 (0.6)	13 (0.6)
Type of symptoms (self-reported), n (%)h,i
Common cold	2111 (91.0)	2118 (90.8)
Shortness of breath	388 (16.7)	391 (16.8)
Fever	417 (18.0)	417 (17.9)
Coughing	1254 (54.1)	1253 (53.7)
Loss of taste or smell	113 (4.9)	109 (4.7)
Muscle ache	245 (10.6)	247 (10.6)
Other symptoms	196 (8.5)	202 (8.7)

In the Netherlands, individuals are notified of a close contact by the Dutch public health service test-and-trace program and/or the Dutch contact tracing mobile phone application (the CoronaMelder app) and/or an individual with a confirmed SARS-CoV-2 infection (index case)

Out of the total group of participants, 2746 had a test result available for both tests

SD standard deviation

^aAge was not available from 10 and 11 participants that had a saliva test result and that had a nasal test result, respectively

^bSex not available from 11 and 12 participants that had a saliva test result and that had a nasal test result, respectively

^cLevel of education was not available from 33 and 35 participants that had a saliva test result and that had a nasal test result, respectively

^dCOVID-19 vaccination status not available from 9, and 9 participants, including 0, and 0 with a positive molecular test result in those with a saliva test result and nasal test result, respectively

^ePercentage calculated as proportion of those vaccinated

^fPrevious SARS-CoV-2 infection information not available from 10 and 10 participants, including 0 and 0 with a positive molecular test result, in those that had a saliva test result and those that had a nasal test result, respectively

^gSymptoms not available for 13, and 13 participants, including 0, and 0 with a positive molecular test result, in those that had a saliva test result, and those that had a nasal test result, respectively

^hPercentage calculated as proportion of those with symptoms at time of sampling

ⁱTotals add up to a number higher than the number of individuals with symptoms at the time of sampling because individuals could report more than one symptom

Table 2 and Fig. 2 show the results of the primary analysis and the secondary stratified analyses. Additional file: Fig. S1 presents them after the application of a viral load cut-off as a proxy for infectiousness. Additional file: Tables S1 and S2 show 2 × 2 tables for both Ag-RDTs. The main findings are presented in the text below.

Table 2.

Diagnostic accuracy parameters for the saliva and nasal Ag-RDTs. Values are percentages (95% confidence interval) unless stated otherwise

Analysis	No	Prevalencea [%]	Sensitivity [%] (95%CI)	Specificity [%] (95%CI)	PPV [%] (95%CI)	NPV [%] (95%CI)
COVID-19 antigen rapid test (oral fluid) for self-testing (Hangzhou AllTest Biotech Co., Ltd.)
Primary analysis	2803	6.5	46.7 (39.3 to 54.2)	99.0 (98.5 to 99.4)	76.6 (67.6 to 84.1)	96.4 (95.6 to 97.1)
Secondary (stratified) analysis
Infectiousness viral load cut-offb,$	2788	5.1	54.9 (46.4 to 63.3)	98.8 (98.3 to 99.2)	70.9 (61.5 to 79.2)	97.6 (97.0 to 98.2)
Symptoms present at samplingc
Yes	2306	6.5	51.0 (42.7 to 59.2)	98.8 (98.3 to 99.2)	75.5 (66.0 to 83.5)	96.6 (95.8 to 97.4)
No	484	6.4	25.8 (11.9 to 44.6)	99.8 (98.8 to 100)	88.9 (51.8 to 99.7)	95.2 (92.8 to 96.9)
Vaccinated (at least one)d
Yes	2390	4.8	47.0 (37.6 to 56.5)	99.0 (98.5 to 99.4)	71.1 (59.5 to 80.9)	97.4 (96.6 to 98.0)
No	404	16.6	46.3 (34.0 to 58.9)	98.8 (97.0 to 99.7)	88.6 (73.3 to 96.8)	90.2 (86.7 to 93.1)
Previous SARS-CoV-2 infectione
Yes	366	5.5	20.0 (5.7 to 43.7)	99.1 (97.5 to 99.8)	57.1 (18.4 to 90.1)	95.5 (92.9 to 97.4)
No	2427	6.7	50.0 (42.1 to 57.9)	99.0 (98.5 to 99.4)	77.9 (68.7 to 85.4)	96.5 (95.7 to 97.2)
Sexf
Female	1711	6.3	38.0 (28.8 to 47.8)	99.2 (98.6 to 99.6)	75.9 (62.4 to 86.5)	96.0 (94.9 to 96.9)
Male	1081	6.8	60.3 (48.1 to 71.5)	98.7 (97.8 to 99.3)	77.2 (64.2 to 87.3)	97.2 (96.0 to 98.1)
Age [years]g
≥ 16 to ≤ 40	1450	6.3	41.3 (31.1 to 52.1)	98.9 (98.2 to 99.4)	71.7 (57.7 to 83.2)	96.1 (95.0 to 97.1)
> 40 to ≤ 65	1135	6.3	50.0 (38.0 to 62.0)	99.2 (98.4 to 99.6)	80.0 (65.4 to 90.4)	96.7 (95.5 to 97.7)
> 65	208	8.2	64.7 (38.3 to 85.8)	99.0 (96.3 to 99.9)	84.6 (54.6 to 98.1)	96.9 (93.4 to 98.9)
SD Biosensor SARS-CoV-2 Rapid Antigen Test Nasal by Roche Diagnostics (‘Nasal’),
Primary analysis	2819	6.4	68.9 (61.6 to 75.6)	99.5 (99.2 to 99.8)	91.2 (85.1 to 95.4)	97.9 (97.3 to 98.4)
Secondary (stratified) analysis
Infectiousness viral load cut-offb,$	2804	5.1	83.9 (76.9 to 89.5)	99.5 (99.2 to 99.7)	90.2 (83.9 to 94.7)	99.1 (98.7 to 99.5)
Symptoms present at samplingc
Yes	2319	6.4	78.5 (71.1 to 84.8)	99.5 (99.2 to 99.8)	92.1 (86.0 to 96.2)	98.5 (97.9 to 99.0)
No	487	6.4	22.6 (9.6 to 41.1)	99.6 (98.4 to 99.9)	77.8 (40.0 to 97.2)	95.0 (92.6 to 96.8)
Vaccinated (at least one)d
Yes	2389	4.5	70.1 (60.5 to 78.6)	99.6 (99.2 to 99.8)	88.2 (79.4 to 94.2)	98.6 (98.0 to 99.0)
No	421	17.3	67.1 (55.1 to 77.7)	99.4 (97.9 to 99.9)	96.1 (86.5 to 99.5)	93.5 (90.5 to 95.8)
Previous SARS-CoV-2 infectione
Yes	372	5.1	36.8 (16.3 to 61.6)	99.2 (97.5 to 99.8)	70.0 (34.8 to 93.3)	96.7 (94.3 to 98.3)
No	2437	6.6	72.7 (65.1 to 79.4)	99.6 (99.3 to 99.8)	92.9 (86.9 to 96.7)	98.1 (97.5 to 98.6)
Sexf
Female	1717	6.2	68.9 (59.1 to 77.5)	99.6 (99.1 to 99.8)	91.2 (82.8 to 96.4)	98.0 (97.2 to 98.6)
Male	1090	6.7	69.9 (58.0 to 80.1)	99.5 (98.9 to 99.8)	91.1 (80.4 to 97.0)	97.9 (96.8 to 98.7)
Age [years]g
16 to ≤ 40	1463	6.4	66.7 (56.1 to 76.1)	99.5 (99.0 to 99.8)	89.9 (80.2 to 95.8)	97.8 (96.9 to 98.5)
> 40 to ≤ 65	1139	6.1	72.9 (60.9 to 82.8)	99.6 (99.0 to 99.9)	92.7 (82.4 to 98.0)	98.2 (97.3 to 98.9)
> 65	206	7.8	68.8 (41.3 to 89.0)	99.5 (97.1 to 100)	91.7 (61.5 to 99.8)	97.4 (94.1 to 99.2)

PPV positive predictive value, NPV negative predictive value

^aSARS-CoV-2 infection based on molecular test result

^bViral load cut-off for infectiousness, defined as viral load above which 95% of people with a positive RT-PCR test result had a positive viral culture⁶, was 5.2 log10 SARS-CoV-2 E-gene copies/mL

^cSymptoms not available for 13, and 13 participants, including 0, and 0 with a positive molecular test result, in those that had a saliva test result, and those that had a nasal test result, respectively

^dCOVID-19 vaccination status not available from 9, and 9 participants, including 0, and 0 with a positive molecular test result in those with a saliva test result and nasal test result, respectively

^ePrevious SARS-CoV-2 infection information not available from 10 and 10 participants, including 0 and 0 with a positive molecular test result, in those that had a saliva test result and those that had a nasal test result, respectively

^fSex not available from 11 and 12 participants that had a saliva test result and that had a nasal test result, respectively

^gAge was not available from 10 and 11 participants that had a saliva test result and that had a nasal test result, respectively

^$Viral load unavailable for 15 individuals that had a saliva test result and 15 individuals that had a nasal test result

Fig. 2.

Sensitivities with 95% confidence intervals of the Ag-RDT-molecular reference standard test comparisons stratified according to symptomatology, COVID-19 vaccination status, previous infection status, sex, and age. The vertical line indicates the sensitivity of the Ag-RDT in the overall study population, and the number of positive molecular tests out of the total or subgroup between parentheses

Self-performed saliva Ag-RDT

Overall test accuracy

SARS-CoV-2 molecular test positivity was 6.5% (182/2803) and overall sensitivity was 46.7% (85/182; 95% CI 39.3–54.2%; Table 2, Fig. 2). Among those with a positive molecular test result, the percentage of participants with a viral load above the cut-off as a proxy for infectiousness was 78.0% (142/182). Using this viral load cut-off, the overall sensitivity was 54.9% (78/142; 46.4–63.3%; Table 2, Fig. 2). Specificities were around 99% in both analyses; positive and negative predictive values are presented in Table 2.

Stratified analyses

The sensitivity was 51.0% (77/151; 42.7–59.2%) in participants who were symptomatic at the time of testing and 25.8% (8/31; 11.9–44.6%) in participants who were asymptomatic (Table 2, Fig. 2). After application of a viral load cut-off as a proxy for infectiousness, these sensitivities were 57.3% (71/124; 48.1–66.1%) and 38.9% (7/18; 17.3–64.3%), respectively (Additional file: Fig. S1). The sensitivity was 20.0% (4/20; (5.7–43.7%) in participants who had had a previous SARS-CoV-2 infection and 50.0% (81/162; 42.1–57.9%) in participants who had never had a prior infection (Table 2, Fig. 2). After applying the viral load cut-off, these sensitivities were 50.0% (4/8; 15.7–84.3%) and 55.2% (74/134; 46.4–63.8%), respectively (Additional file: Fig. S1). In males the sensitivity was higher than in females, 60.3% (44/73; 48.1–71.5%) versus 38.0% (41/108; 28.8–47.8%), and increased slightly by increasing age (Table 2, Fig. 2). We found no evidence of a differential impact on diagnostic accuracy by COVID-19 vaccination status. The sensitivities after application of a viral load cut-off as a proxy for infectiousness are presented in Additional file: Fig. S1. Specificities were > 99% and positive predictive values > 70% and negative predictive values > 9 5% in most analyses (Table 2).

Self-performed nasal Ag-RDT

Overall test accuracy

SARS-CoV-2 molecular test positivity was 6.4% (180/2819) and overall sensitivity was 68.9% (124/180; 61.6–75.6%; Table 2, Fig. 2). Among those with a positive molecular test result, the percentage of participants with a viral load above the cut-off as a proxy for infectiousness was 79.4% (143/180). Using this viral load cut-off, the overall sensitivity was 83.9% (120/143; 76.9–89.5%; Table 2, Fig. 2). Specificities were 99.5% in both analyses; positive and negative predictive values are presented in Table 2.

Stratified analyses

The sensitivity was 78.5% (117/149; 71.1–84.8%) in participants who were symptomatic at the time of testing and 22.6% (7/31; 9.6–41.1%) in participants who were asymptomatic (Table 2, Fig. 2). After application of a viral load cut-off as a proxy for infectiousness, these sensitivities were 90.4% (113/125; 83.8–94.9%) and 38.9% (7/18; 17.3–64.3%), respectively (Additional file: Fig. S1). On average, the viral load was lower in asymptomatic than in symptomatic individuals with a positive molecular reference test (Additional file: Fig. S2). The sensitivity was 36.8% (7/19; 16.3–61.6%) in participants who had had a previous SARS-CoV-2 infection and 72.7% (117/161; 65.1–79.4%) in participants who had never had a prior infection. After applying the viral load cut-off, these sensitivities were 100.0% (7/7; 59.0–100.0%) and 83.1% (113/126; 75.7–89.0%), respectively (Additional file: Fig. S1). We found no evidence of a differential impact on diagnostic accuracy by COVID-19 vaccination status, sex, and age (Fig. 2, Additional file: Table S3). Specificities were > 99% and positive predictive values > 90% and negative predictive values > 95% in most analyses (Table 2).

Diagnostic test accuracy results for both Ag-RDTs were the same when the analysis populations were limited to the 2746 participants for whom all three test results were available (data not shown).

User experiences and preferences

Participants indicated that taking a sample was easier for the self-performed saliva Ag-RDT than for the self-performed nasal Ag-RDT (p < 0.0001 by McNemar-Bowker test), with a larger proportion of participants indicating it was easy or very easy for the saliva Ag-RDT than for the nasal Ag-RDT (80.4% vs. 66.8% (p < 0.001 by McNemar test). Most participants reported that reading the test result was easy or very easy for both tests (92.2% vs. 92.4%; Table 3; p = 0.82 by McNemar test). Distributions of change scores for both usability components are presented in Additional file: Fig. S3. After performing both Ag-RDTs, but before receiving the molecular test result, 55.6% of the participants reported to prefer the saliva over the nasal Ag-RDT, 9.3% reported to prefer the nasal Ag-RDT, and 32.8% indicated to have no preference.

Table 3.

Usability of the saliva and nasal Ag-RDTs

N = 2950	Saliva Ag-RDT	Nasal Ag-RDT
Difficulty taking a samplea
Very easy	1286 (43.6)	726 (24.6)
Easy	1087 (36.8)	1245 (42.2)
Medium	380 (12.9)	613 (20.8)
Hard	117 (4.0)	265 (9.0)
Very hard	16 (0.5)	35 (1.2)
Unknown	64 (2.2)	66 (2.2)
Difficulty reading the test resulta
Very easy	1997 (67.7)	1940 (65.8)
Easy	724 (24.5)	785 (26.6)
Medium	110 (3.7)	128 (4.3)
Hard	39 (1.3)	23 (0.8)
Very hard	15 (0.5)	9 (0.3)
Unknown	65 (2.2)	65 (2.2)
The user manual was clear
Agree	2533 (85.9)	n/a
In doubt	266 (9.0)	n/a
Disagree	86 (2.9)	n/a
Unknown	65 (2.2)	n/a
The test went well
Agree	2514 (85.2)	n/a
In doubt	322 (10.9)	n/a
Disagree	50 (1.7)	n/a
Unknown	65 (2.2)	n/a
If I could, I would use the test at home
Agree	2554 (86.6)	n/a
In doubt	89 (3.0)	n/a
Disagree	239 (8.1)	n/a
Unknown	68 (2.3)	n/a
Do you prefer one test over the other
No	969 (32.8)
Yes, saliva Ag-RDT preferred	1639 (55.6)
Yes, nasal Ag-RDT preferred	275 (9.3)
Unknown	67 (2.3)

n/a not available

^ap-value < 0.0001 for an omnibus symmetry test result for a paired contingency table (McNemar-Bowker test). The unknown category was not included in this test

Follow-up

Follow-up information was available for 72% of participants (Table S3), of whom 1994 participants had a negative molecular test result at baseline. Of the latter group, 249/1994 (12.5%) were re-tested within 10 days, and 7/249 (2.8%) tested positive, wherein we did not know whether it was a new infection, or the initial molecular test was false negative.

Discussion

This largest diagnostic accuracy evaluation of two unsupervised self-performed Ag-RDTs to date showed a low overall sensitivity (46.7%) of the Hangzhou AllTest Biotech saliva self-test. Applying a viral load cut-off as a proxy for infectiousness did not improve the overall sensitivity meaningfully (54.9%) nor in any of the studied subgroups, with all sensitivities remaining far below the WHO standard of 80% [21].

The study showed better performance of the SD Biosensor nasal self-test with an overall sensitivity of 68.9%, increasing to 83.9% when the viral load cut-off was applied. The sensitivities were much higher in the 2319 symptomatic participants than in the 487 asymptomatic participants (78.5% and 22.6%, respectively) and in the 2417 individuals who never had COVID-19 in the past compared to the 372 individuals who did (72.7% and 36.8%, respectively), reaching sensitivities (with sufficient precision) above the WHO-recommended 80% in symptomatic individuals and in individuals without a previous infection after applying the viral load cut-off. The sensitivities in asymptomatic individuals and in individuals who had had COVID-19 in the past have wider 95% confidence intervals and should therefore be interpreted with caution. We recommend additional research in those groups. The diagnostic accuracy of this nasal self-test did not differ by COVID-19 vaccination status, sex, and age.

Discussion of the saliva self-test results

We identified two previous studies in the scientific literature; the sensitivities observed in our study were in between those found in the two studies. A Czech study evaluated four saliva Ag-RDTs and found sensitivities of 15% for a saliva test that required spitting in a cup, and 3.6%, 25.5%, and 32.8% for saliva tests requiring sucking on a sponge, in comparison with RT-PCR [12]. These sensitivities improved slightly after using a “cell culture viability” cut-off but remained well below 50%. The sampling was done by the participants themselves but supervised by trained personnel. Samples sizes were modest, ranging from 98 to 407 participants per evaluated test. A recent Dutch study of the SD Biosensor saliva test with 789 participants found a sensitivity of 66.1%, increasing to 88.6% when Ct < 30 and to 96.7% when viral culturability was used as cut-offs [13]. In the Dutch study, saliva was collected by letting nasal and cough discharge drool into a collection device and was supervised by trained test site staff. Sensitivity was lower (60%) in asymptomatic participants but only 10 asymptomatic participants tested RT-PCR positive. We tested the analytical performance of the Hangzhou and SD Biosensor lateral flow test devices on calibrated samples and found that both test devices performed (equally) well (Supplement 3). We therefore hypothesize that the widely ranging sensitivity results for saliva Ag-RDTs may be due to high variability in saliva sampling methods (spitting vs. sucking vs. drooling) and/or high variability in the quantity and quality of sample self-obtained by different individuals. Furthermore, saliva specimens may on average contain lower SARS-CoV-2 viral loads than upper respiratory tract samples. Studies have shown that saliva viral loads are usually sufficiently high for detection by molecular methods [22–24], but they may not be sufficiently high for detection by self-performed Ag-RDTs.

We saw trends of reduced diagnostic accuracy in persons without symptoms or with previous SARS-CoV-2 infection. These trends were like the trends that we observed for the nasal self-test and are discussed below. We also saw trends by gender and age, with sensitivities for men and for persons aged over 65 reaching around 60%, which is still well below the WHO-recommended 80% [21]. The saliva Ag-RDT evaluation studies to date did not stratify by gender and age [12, 13], and the nasal Ag-RDT studies, including the nasal self-test that we evaluated in this study, did not show these trends [3, 6]. We recommend that future saliva Ag-RDT evaluations stratify by gender and age to investigate this further.

Discussion of the nasal self-test results

The diagnostic performance of Ag-RDTs combined with nasopharyngeal sampling done by trained personnel or by individuals themselves have been evaluated extensively by us and others [2–6, 8, 9, 12]. The above-mentioned Czech study also evaluated an Ag-RDT in combination with anterior nasal sampling done by trained personnel [12]. These studies found good performance (70–80%) with nasopharyngeal sampling but lower performance with anterior nasal sampling (45–55%) and also lower performance in asymptomatic individuals (50–60%), regardless of self or professional sampling. Our study showed that nasal self-sampling with the SD Biosensor Ag-RDT provided good sensitivity, which equaled the sensitivity of Ag-RDTs found in other studies in which the nasal sampling was done by a trained professional [10, 11], but only for individuals who have symptoms at the time of testing. We found a very low sensitivity of this self-performed nasal Ag-RDT of only 23% in asymptomatic individuals, which is much lower than the sensitivities found in our previous studies using nasopharyngeal or oropharyngeal combined with nasal sampling done by trained personnel [3, 6]. This difference in performance persisted after applying a viral load cut-off. It is currently unclear why the sensitivities of the nasal Ag-RDT self-test differed depending on the presence of symptoms, even after applying the viral load cut-off. We tested the analytical performance of the SD Biosensor lateral flow test device on calibrated samples and found that the test device itself performed well (Supplement 3). We hypothesize that the difference in sensitivity may be explained by the difference in viral load distributions in asymptomatic and symptomatic individuals. In addition, it may be more difficult for asymptomatic individuals (i.e., with a dry nose) to retrieve sufficient nasal fluid by self-swabbing. The former hypothesis is supported by the fact that the sensitivity of the saliva self-test was also lower in asymptomatic than symptomatic individuals, but both hypotheses might play a role.

We also found a low sensitivity (36.8%) of the nasal self-test in individuals who had had COVID-19 in the past. These results should be interpreted with caution due to the small group sizes: only 19 participants with a positive molecular test reported having had COVID-19 (16 of whom were symptomatic at the time of testing), and only seven of them had a viral load above the viral load cut-off (six of whom were symptomatic). However, similar trends were observed for the saliva self-test in this study and for the SD Biosensor Ag-RDT conducted by trained staff in a previous study [6]. In that study, sensitivities were 54.5% for oropharyngeal-nasal sampling and 68.4% for nasopharyngeal sampling in individuals with prior infection, and 75.8% and 75.0%, respectively, in individuals without prior infection. The low sensitivity in individuals with a prior infection may be explained by lower viral loads in this group (in the current study, 12/19 participants with a prior infection were below the viral load cut-off compared to 113/161 in those without a prior infection), with some of them potentially carrying viral RNA in the absence of a productive infection (i.e., no viral antigen production). Another explanation might be that individuals who have had COVID-19 have circulating anti-nuclear capsid (N) protein antibodies [25]. These anti-N antibodies might bind to the N protein that is produced during the new infection, hampering the binding of monoclonal antibodies against the N-protein in the test device. It should be noted that we found a smaller reduced sensitivity of the BD Veritor Ag-RDT conducted by trained staff (oropharyngeal-nasal sampling) in individuals with and without a prior infection (64.6% versus 70.1%), so this effect may be test device-specific [6].

Strengths and limitations of this study

Strengths of this study include the large overall sample size covering multiple test sites nationwide, the collection of samples for the reference test and two Ag-RDTs in the same individuals within a few hours allowing for a head-to-head comparison of the two self-tests, the fact that sampling was done by the participants themselves without any supervision conform the real-world context of self-testing, that the index test was blinded for the reference test result and vice versa, and the use of a proxy for infectiousness. Furthermore, the follow-up information showed that very few infections were missed by the molecular reference tests.

Our study also has some limitations. First, the reference standards that we used were molecular tests, but platforms and test kits used differed among the centralized laboratories. However, the diagnostic accuracies of all molecular tests used are similarly high [26, 27], and we therefore believe that this has not influenced our findings significantly. In addition, Ct values used to calculate viral loads were determined by different yet comparable platforms (Additional file: material 3). Second, we used the viral load cut-off above which 95% of people with a positive RT-PCR test result had a positive virus culture as a proxy of infectiousness. Although this cut-off is not fully evidence based [3], it is a best estimate based on current knowledge and less arbitrary than using Ct cut-offs of 25 or 30 as is often done [28, 29]. In the current study, we relied on infectiousness viral load cut-offs that were determined in our previous study in a mainly unvaccinated population (the proportion of vaccinated individuals in the present study reached 85% at the end of the study) and when a different SARS-CoV-2 variant was dominant [3]. Whether this would have impacted the applied viral load cut-offs is unknown, but vaccination itself did not influence any of the test sensitivities. Third, our sample size calculation was based on the primary analysis and the diagnostic accuracy parameters are less precise for the secondary stratified analyses. Fourth, participants were not blinded to the results of the saliva Ag-RDT when interpreting the result of the nasal Ag-RDT, which could have potentially biased the test outcome assessment of the nasal Ag-RDT. We do, however, believe that the impact of this limitation is small considering that the interpretation of the test results was considered (very) easy by > 95% of participants, and the performance of each Ag-RDT was substantially different and higher for the last performed nasal Ag-RDT. If outcome assessment was biased, the diagnostic performance of the self-tests would likely have been more similar. Fifth, we had some, though very limited, missing index test data (5%). We did not perform multiple imputation techniques because the group with missing data was very similar to the group with complete data, suggesting that data was missing completely at random.

Policy implications

Ag-RDTs for self-use are widely available in the Netherlands. Until recently, the recommendation was to use them when asymptomatic prior to having contacts (such as going to school, events, or work) and visit a public health test site for molecular testing when symptomatic. Individuals whose self-test was positive are (still) asked to visit a public health test site for confirmatory testing. The SD Biosensor nasal self-test that we evaluated in this study is one of the self-tests that is commercially available in the Netherlands, although we do not know its market share. Our results indicate that the SD Biosensor nasal self-test sensitivity among individuals with mild symptoms is similar to Ag-RDT sensitivities found in studies where it was applied to professionally obtain upper respiratory tract samples. Based on those results, the Dutch Outbreak Management Team (OMT) that advises the Ministry of Health, Welfare, and Sports regarding COVID-19 policy recommended expanding nasal self-testing to individuals with mild symptoms. The OMT stressed that self-tests are not advised in vulnerable persons, in individuals meeting vulnerable persons, and in case of more severe symptoms, and that a negative self-test result is not sufficient for ending quarantine for contacts of a confirmed case. However, individuals testing negative by nasal self-testing would be allowed to go to work (if not working with vulnerable persons) or school despite their mild symptoms, preferably using mouth-nose masks and testing again a day later in case the initial test result was negative. Individuals testing positive by nasal self-testing would have to self-isolate and visit a public health test site for confirmatory molecular testing, to keep track of virus spreading and to allow for contact-tracing. All these nuances require careful communication, including on the implications of false-negative test results.

The very low sensitivity of the SD Biosensor nasal self-test in asymptomatic individuals is worrisome, even though the a-priori probability of being infected is lower in asymptomatic than symptomatic individuals. In addition, the potentially reduced sensitivity of the SD Biosensor nasal self-test in individuals who have had COVID-19 in the past is also worrisome. This is especially important because an increasing proportion of the population will have had COVID-19. We call for additional research in these two specific subgroups. We also recommend that persons who tested negative by a self-test continue to adhere to the general preventive measures such as physical distancing, wearing mouth-nose masks, and washing hands.

The SD Biosensor nasal self-test is only one of the commercially available self-tests. We recommend that all available self-tests are evaluated urgently by independent researchers, also addressing the relevant subgroups. Finally, in high-risk situations, such as testing of vulnerable people in care facilities, severely ill patients, or healthcare workers, we recommend molecular testing at all times, which is already in line with current policy.

Conclusions

The Hangzhou AllTest Biotech saliva self Ag-RDT is not reliable for SARS-CoV-2 detection, overall, and in all studied subgroups. The SD Biosensor nasal self Ag-RDT had high sensitivity in individuals with COVID-19 like symptoms and in those without a prior SARS-CoV-2 infection but a very low sensitivity in asymptomatic individuals and in those with a prior SARS-CoV-2 infection warrants further investigation.

Abbreviations

Ag-RDTs

Rapid lateral flow antigen diagnostic tests

CI

Confidence interval

COVID-19

Coronavirus disease 2019

IQR

Interquartile range

Nasal Ag-RDT

SD Biosensor SARS-CoV-2 Rapid Antigen Test Nasal for self-testing, distributed by Roche Diagnostics

RT-PCR

Real-time reverse-transcriptase polymerase chain reaction

SARS-CoV-2

Severe acute respiratory syndrome coronavirus-2

Saliva Ag-RDT

COVID-19 Antigen Rapid Test (Oral Fluid) for Self-testing by Hangzhou AllTest Biotech Co. Ltd.

STARD 2015

Standards for Reporting of Diagnostic Accuracy Studies 2015

Authors’ contributions

KGMM initiated the study. ES, RPV, IKV, WvdB, SDP, EL, MH, RM, ZI, CW, IV, CRSN-I, WGHH, JAJWK, SvdH, JHHMvdW, and KGMM designed the study. ES, RPV, IKV, CRSN-I, and KGMM coordinated the study. WvdB, SDP, JS, RM, and ZI were responsible for laboratory analyses and data processing. ES, RPV, and IKV verified the underlying data. ES performed the statistical analysis and verified the underlying data in close collaboration with RPV and KGMM. ES, RPV, JHHMvdW, and KGMM drafted the first version of the manuscript. All authors critically read the manuscript and provided feedback. All authors read and approved the final manuscript, and approved its submission. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Funding

This study was funded by the Dutch Ministry of Health, Welfare, and Sport. The funder had no role in the study design; collection, analysis, and interpretation of data; writing of the report and decision to submit the paper for publication.

Availability of data and materials

Individual participant data collected during the study will be available, after deidentification of all participants. Data will be available to researchers who provide a methodologically sound proposal to achieve the aims in the approved proposal. Proposals should be directed to the corresponding author to gain access to the data. Data requestors will need to sign a data sharing agreement. The study protocol is available upon request by contacting Karel Moons at k.g.m.moons@umcutrecht.nl.

Declarations

Ethics approval and consent to participate

Not required because the study was judged by the METC Utrecht to be outside the scope of the Dutch Medical Research Involving Human Subjects Act (protocol No 21–470 /C). All participants signed an informed consent form before any study procedure.

Consent for publication

Not applicable.

Competing interests

All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: support from the Dutch Ministry of Health, Welfare, and Sport for the submitted work; no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years; no other relationships or activities that could appear to have influenced the submitted work.