Accuracy of rapid antigen detection test for nasopharyngeal swab specimens and saliva samples in comparison with RT-PCR and viral culture for SARS-CoV-2 detection

Abstract

Introduction

Rapid antigen detection (RAD) tests are convenient tools for detecting the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) in clinics, and testing using saliva samples could decrease the risk of infection during sample collection. This study aimed to assess the accuracy of the SARS-CoV-2 RAD for testing of nasopharyngeal swab specimens and saliva samples in comparison with the RT-PCR tests and viral culture for detecting viable virus.

Methods

One hundred seventeen nasopharyngeal swab specimens and 73 saliva samples with positive results on RT-PCR were used. Residual samples were assayed using a commercially available RAD test immediately, and its positivity was determined at various time points during the clinical course. The concordance between 54 nasopharyngeal swab samples and saliva samples that were collected simultaneously was determined. Viral culture was performed on 117 samples and compared with the results of the RAD test.

Results

The positive rate of RAD test using saliva samples was low throughout the clinical course. Poor concordance was observed between nasopharyngeal swab specimens and saliva samples (75.9%, kappa coefficient 0.310). However, a substantially high concordance between the RAD test and viral culture was observed in both nasopharyngeal swab specimens (86.8%, kappa coefficient 0.680) and saliva samples (95.1%, kappa coefficient 0.643).

Conclusions

The sensitivity of the SARS-CoV-2 RAD test was insufficient, particularly for saliva samples. However, a substantially high concordance with viral culture suggests its potential utility as an auxiliary test for estimating SARS-CoV-2 viability.

1. Introduction

The coronavirus disease (COVID-19) pandemic caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has emerged as a serious concern worldwide. To control the spread of the SARS-CoV-2 infection, it is essential to isolate infected patients based on a laboratory-confirmed diagnosis. Reverse transcription PCR (RT-PCR) is a gold standard investigation in the clinical diagnosis of COVID-19; however, RT-PCR requires special equipment, skilled medical technologists, and considerable time (several hours) to generate test results. Rapid antigen detection (RAD) tests are a type of point-of-care testing to detect SARS-CoV-2-specific antigen proteins based on a lateral flow assay, which can be performed within 30 min without necessitating any special equipment or training. However, the samples for RAD tests generally comprise nasal or nasopharyngeal swab (NPS) samples and, therefore, the risk of infection among medical staff during sample collection is inevitable. Saliva, which can be used as an alternative sample in RT-PCR for SARS-CoV-2, can be self-collected by patients and thus reduce the exposure risk of medical staff [1,2]. However, there is a lack of studies that demonstrate the accuracy of the RAD test in saliva samples [3]. Furthermore, although the viability of the virus is important for predicting its communicability, clinical studies on RAD test results and virus viability are scarce [4]. Therefore, we conducted an observational study to assess the accuracy of the SARS-CoV-2 RAD test of NPS and saliva samples in comparison with the RT-PCR tests and viral culture for detecting viable virus.

2. Methods

In this single-centre study, we used NPS specimens and saliva samples collected for SARS-CoV-2 RT-PCR testing from July 1 to October 8, 2020 at the Keio University Hospital (Tokyo, Japan). The NPS specimens and saliva samples were collected simultaneously in some patients to assess the accuracy of RT-PCR testing of saliva samples [5]; otherwise, either an NPS specimen or a saliva sample alone were collected. NPS specimens were collected by trained medical staff using a FLOQ SWAB and a Copan UTM container (Copan, Brescia, Italy), and saliva samples were self-collected by patients themselves by spitting into sterile containers (ASIAKIZAI, Tokyo, Japan) after 1 min of salivation. SARS-CoV-2 real-time RT-PCR was performed using the LightCycler96 (Roche, Basel, Switzerland) and the 2019 Novel Coronavirus Detection Kit (Shimadzu, Kyoto, Japan) by using N1 and N2 primers and probes in accordance with the manufacturer’s instructions [6]. The threshold cycle (Ct) values <40 for either primer were considered to be indicative of a positive result. Among these specimens, RT-PCR-positive samples with enough residual volume were subjected to RAD tests on the day of sample collection. This study was approved by the Ethics Committee of Keio University School of Medicine (20200063 and 20190337), and by the Research Ethics Review Committee of the Institute of Medical Science, the University of Tokyo (approval number 2019-71-0201). Individual informed consents were waived according to the retrospective nature of this study. RAD tests were performed using the Espline SARS-CoV-2 RAD kit (FUJIREBIO, Tokyo, Japan), which was purchased from Alfresa Co. (Tokyo, Japan) and used in accordance with the manufacturer’s instructions. The tip of the swab in the kit was soaked in the residual Copan UTM medium containing the NPS specimens or saliva samples, then immersed in extraction reagents, and rolled in the reagents 10 times. The samples processed in extraction reagents were left for 5 min, and two drops were added to the test strip. After 30 min, the test results were judged by at least two medical technologists.

Furthermore, to assess the viability of the virus in the sample when the patients’ consent was obtained for sample use, the refrigerated residual samples were sent to biosafety level 3 facilities, and viral culture was performed to detect infective virus using VeroE6/TMPRSS2 cells in Dulbecco’s Modified Eagle Medium containing 10% fetal calf serum (FCS), 1 mg/mL G418, 100 units/mL penicillin, 100 μg/mL streptomycin, and 5 μg/mL Plasmocin Prophylactic (InvivoGen, San Diego, CA, USA). A 24-well plate containing VeroE6/TMPRSS2 cell culture monolayer at a low crowding density (70–95% confluent) was prepared. After the medium was discarded, 100 μL of sample was added to the cells and incubated for 1 h at 37 °C under 5% CO₂. Then, 0.5 mL of growth medium was added and incubated at 37 °C under 5% CO₂ for 1 week until a cytopathogenic effect was observed (supplemental file. Fig. S1). [7].

Information about the date of sample collection, patient age, sex, and date of symptom onset were obtained from the electronic hospital medical records. The date of symptom onset was defined as the date when fever, respiratory symptoms, olfactory disorder, or taste disorder were first observed in symptomatic patients, and as the date that the first RT-PCR-positive sample was collected in presymptomatic or asymptomatic patients. The duration after symptom onset was calculated as the interval (days) between sample collection and symptom onset. The relationship between the RAD test results and the days after symptom onset and the Ct values of RT-PCR were assessed. The Ct values of the N1 prime probe sets were used for the assessment because the elevation of the amplification curve for the N2 primer probe sets was not observed in some samples according to the difference in the amplification efficacy among the two primer probe sets. For the NPS specimens and saliva samples that were collected simultaneously, the concordance between the RAD tests between the two samples was calculated. Additionally, among the viral culture samples, the concordance between the RAD test results and viral culture results was assessed.

Fisher’s exact test for categorical outcomes and Mann–Whitney U test for continuous variables were used for statistical analysis. A p-value of less than 0.05 was considered statistically significant. All statistical analyses were performed using SPSS version 25 (IBM, Chicago, IL, USA) and Prism version 8 (GraphPad, San Diego, CA, USA).

3. Results

A total of 190 RT-PCR positive samples obtained from 72 patients were used. One hundred and seventeen samples were NPS specimens and 73 were saliva samples. Patient age, sex ratio, and number of days from onset were equivalent for the two sample types. The Ct values of the saliva samples were slightly lower than those of the NPS specimens although the difference was not statistically significant (Table 1 ).

Table 1.

Baseline patient characteristics.

Characteristics		NPS	Saliva
		(N = 117)	(N = 73)
Age (median, IQR)		47 (32–61)	48 (34–59)
Sex
	Male	77	46	p = 0.756
	Female	44	27
Days from onset, (median, IQR)		8 (6–11)	9 (6–11)	p = 0.817
Ct values of RT-PCR (N1 set)		32.43 (27.01–36.59)	30.45 (26.75–32.91)	p = 0.072
Collection
	Simultaneous (NPS and saliva)	54	54
	Either NPS or saliva alone	63	19
Viral Culture Performed		76	41
	Positive results	23	2	p = 0.001
	Negative results	53	39

Seventy-two percent of NPS specimens that were collected within 4 days of symptom onset were positive for the RAD test. However, the RAD test positivity rate of NPS specimens collected 5 days after symptom onset was less than 30%, and those of saliva samples were lower than 30% in each time period (Fig. 1 ).

Fig. 1.

Time course from symptom onset and RAD test positivity. The correlation between days from COVID-19 symptom onset and RAD test results of nasopharyngeal swab (NPS) specimens and saliva samples was examined. Black bar shows the number of RAD test-positive samples, and the grey bar shows the negative samples. Except for the NPS specimens that were collected within 4 days from symptom onset that yielded high positivity, the RAD test positivity was generally low.

The median Ct values of RAD test-positive samples were lower than that of negative NPS specimens (23.39 vs. 34.31, p < 0.001) and saliva samples (22.63 vs. 30.96, p < 0.001) (Fig. 2 ).

Fig. 2.

Real-time RT-PCR Ct values of the samples and RAD test results. Median Ct values (N1 sets) were lower in RAD test-positive samples than in negative samples among nasopharyngeal swab (NPS) specimens and saliva samples.

Fifty-four NPS specimens and saliva samples were collected simultaneously. The RAD test of saliva samples was positive in seven samples, whereas that of NPS specimens was positive in 16 samples. The concordance rate was only 75.9% (kappa coefficient, 0.310; Table 2 ).

Table 2.

Concordance of RAD test results between NPS specimens and saliva samples collected simultaneously.

Sample		NPS
		Positive	Negative
Saliva	Positive	5	2	7	Concordance	75.9%
	Negative	11	36	47	kappa	0.310
		16	38	54

Viral cultures were performed for 117 samples, and 25 samples (21.4%) tested positive on viral culture. The concordance rate of RAD test results and viral culture results were substantially high (86.8%) in NPS specimens (kappa coefficient 0.680) and 95.1% in saliva samples (kappa coefficient 0.643) (Table 3 ).

Table 3.

Concordance of RAD test results and viral culture test results.

Samples	RAD	Viral Culture
		Positive	Negative
All samples, (NPS + Saliva)	(n)	25	92	Concordance	89.7%	Kappa	0.695
	Positive	19	6	Sensitivity	76.0%	PPV	76.0%
	Negative	6	86	Specificity	93.5%	NPV	93.5%
NPS	(n)	23	53	Concordance	86.8%	Kappa	0.680
	Positive	17	4	Sensitivity	73.9%	PPV	81.0%
	Negative	6	49	Specificity	92.5%	NPV	89.1%
Saliva	(n)	2	39	Concordance	95.1%	Kappa	0.643
	Positive	2	2	Sensitivity	100%	PPV	50.0%
	Negative	0	37	Specificity	94.9%	NPV	100%

PPV, positive predictive value; NPV, negative predictive value.

4. Discussion

SARS-CoV-2 RAD tests are convenient and can be performed easily even at the patient’s bedside, although their low sensitivity is of considerable concern for making differential diagnosis of suspected acute respiratory infection cases in COVID-19 pandemic. Scohy et al. reported that the RAD test detected only 42.1% of RT-PCR-positive samples [8]; similarly, this study revealed the poor sensitivity of RAD tests. Therefore, it might be difficult to substitute the RAD test for RT-PCR testing in COVID-19 diagnosis.

In particular, a poor concordance of the RAD test results between NPS specimens and saliva samples was demonstrated in this study, and this is compatible with previous reports of the poor sensitivity of the RAD test using saliva samples [3]. Additionally, Diao et al. mentioned the high accuracy of RAD testing of NPS specimens collected immediately after the onset of symptoms [9], and this study identified the high sensitivity of RAD testing in NPS specimens collected within 4 days from symptom onset. However, even in the early phase, the sensitivity of the RAD test of saliva samples was low, which demonstrates that the clinical utility of RAD testing of saliva samples is limited in our study setting.

It is interesting that a substantially high concordance was observed between RAD test results and viral culture in both NPS specimens and saliva samples. In most cases, the samples with positive viral culture showed Ct values of less than 25 [10]. The Ct value threshold of RAD test positivity was approximately 25 in saliva samples and 30 in NPS samples. Therefore, it was simple for viral culture-positive samples to also become RAD positive. Despite this confounding relationship, this fact implies RAD test can be a potential auxiliary diagnostic tool for estimating the infectivity of the COVID-19 patients. Patients with the viable virus are thought to be highly contagious, and strict isolation is required. RT-PCR, which can detect not only viable virus but also fragmented RNA, is known to continue to test positive for as long as 1 month from symptom onset [11]. Therefore, negative conversion of RT-PCR is an unsatisfactory criterion for ending isolation, as it can result in unnecessarily prolonged isolation. Although the sensitivity of viral culture for detecting viable virus is uncertain, only viral culture can detect viable virus. Therefore, negative conversion of viral culture is an ideal criterion for ending the isolation of COVID-19 patients; however, viral culture requires special equipment and experienced staff trained in biosafety protocols. Although the sample size is insufficient to draw a firm conclusion, the negative predictive value of the RAD test for predicting the viral culture result suggests that one or two negative RAD test results is potentially useful as a criterion for assessing the viability of SARS-CoV-2.

This study has several limitations. First, RAD test samples of NPS specimens were not collected directly from the patient but were obtained from residual UTM medium. Therefore, the viral load of the samples might be lower than that of the directly collected samples. Second, the sensitivity of viral culture is difficult to determine. Therefore, a negative viral culture does not always mean the absence of communicability, although viral culture is the only available means for assessing the infectivity of the sample at this time. Third, although the false positivity of RAD test is problematic [12], this study could not evaluate the specificity of RAD tests compared to RT-PCR because RT-PCR-negative samples were not included in this study. Finally, the variance of the accuracy among commercial kits was not studied. Therefore, to determine the external validity of the study results, further studies including a variety of kits and large sample sizes are essential to optimise the usage of RAD kits in the clinic management of COVID-19.

In conclusion, although the sensitivity of the SARS-CoV-2 RAD test was insufficient for ruling out COVID-19 diagnosis, particularly when using saliva samples, the substantially high concordance of the RAD test and viral culture suggests its potential utility in estimating SARS-CoV-2 viability.

Funding

This study was funded by Research Funds of Keio University School of Medicine and partly supported by a Research Program on Emerging and Re-emerging Infectious Diseases (grant no. JP19fk0108113), the Japan Initiative for Global Research Network on Infectious Diseases (J-GRID) (JP19fm0108006), and Japan Program for Infectious diseases Research and Infrastructure (JP20wm0125002) of the Japan Agency for Medical Research and Development (AMED) and by the National Institutes of Allergy and Infectious Diseases-funded Center for Research on Influenza Pathogenesis (grant no. HHSN272201400008C).

Contributors

YU conceived, designed the study, analysed and interpreted the data, and wrote the manuscript. MN performed the assays and analysed the data. WA, TN, YF, YS, and KI performed the assays. MN, WA, TN, RI, YF, YS, KI, TS, YK, NH, and MM discussed the data, and critically reviewed and revised the manuscript. All authors have approved the final version of the manuscript for publication.

Declaration of competing interest

None.

Appendix A. Supplementary data

The following is the Supplementary data to this article: