Abstract
Background
Direct detection tests for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) that bypass complicated nucleic acid/antigen purification steps are promising tools for the rapid diagnosis of coronavirus disease 2019 (COVID-19).
Methods
To determine the analytical and clinical diagnostic performances of the direct detection assays, we compared 6 direct molecular detection assays, including two loop-mediated isothermal amplification (LAMP) assays and one lateral flow antigen assay, against the reference extraction-based RT-PCR assay using 183 respiratory samples (87 nasopharyngeal swabs, 51 saliva samples, and 45 sputum samples).
Results
Analytical sensitivity analysis showed that the direct RT-PCR assay of Toyobo exhibited the lowest LOD of 1,000 copies/mL. Compared with the 80 positive and 103 negative samples based on the reference assay, the Toyobo assay had the highest positive percent agreement (PPA) of 96.3%, followed by the two direct RT-PCR assays of Takara and Shimadzu and one LAMP assay of Eiken (86.3–87.5%). The Fujirebio antigen assay had the lowest PPA of 44.7% among the assays tested. The negative percent agreement of these direct detection assays was 100%, except for the Eiken assay (96.3%).
Conclusions
Large differences in PPA existed among the direct detection tests. Laboratories need to take these characteristics into consideration before implementing these assays.
Keywords: SARS-CoV-2, RT-PCR, Lateral flow antigen assay, Direct detection
1. Introduction
The use of reliable detection tests for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is important for the diagnosis of coronavirus disease 2019 (COVID-19) [1]. Molecular tests using viral nucleic acid extraction/purification and reverse transcription-PCR (RT-PCR) can provide highly sensitive and specific results and remain the gold standard [2]. However, we have experienced challenges in response to the massive demand for RT-PCR testing under global supply shortages and the need for skilled laboratory professionals [3].
Molecular detection assays based on RT-PCR or reverse transcription-loop-mediated isothermal amplification (RT-LAMP), which directly utilize patient samples in their reactions and bypass viral nucleic acid extraction/purification steps (hereinafter referred to as direct assays), use less plastics and reagents and can decrease the time, labor, and cost [4]. Lateral flow immunoassays also detect SARS-CoV-2 antigen directly from specimens and can be easily performed and employed as point-of-care testing without the need for trained technicians [5]. However, concerns exist for their diagnostic performances; a false-positive result can significantly limit social and personal activities, and a false-negative result allows for the potential spread of COVID-19. It is also important to understand the differences in the diagnostic performances of the available assays to utilize them in appropriate settings or to switch them when a shortage occurs. Current data for comparison of analytical sensitivities and/or diagnostic performances among available direct detection assays are lacking or insufficient due to different methods used to evaluate each assay (e.g., type and usage of positive control material [e.g., inactivated virus in negative sample matrix that simulates clinical specimen or synthetic RNA template in an RT-PCR], type of negative sample matrix, and number of replicates for analytical sensitivity analysis and clinical samples for clinical studies). In this study, we aimed to determine the analytical sensitivities and clinical diagnostic performances of six direct molecular detection assays and one lateral flow antigen assay, which will enable direct comparison of the performances among assays.
2. Material and methods
2.1. Clinical specimens
We included 183 respiratory samples (nasopharyngeal swab, n = 87; saliva, n = 51; sputum, n = 45) collected from patients who were suspected to have COVID-19 between March and July 2020 at Kyoto University Hospital and Kyoto City Hospital. We performed studies in parallel with different methods using the same samples. Further details are given in Supplementary material.
2.2. Direct detection assays
Table 1 shows a summary of the direct detection assays used in this study. Only acceptable specimen types for each assay were tested. All commercial kits were used according to the manufacturers’ instructions. We included four commercial direct RT-PCR kits: 2019 Novel Coronavirus Detection kit (Shimadzu, Kyoto, Japan; hereinafter referred to as Shimadzu), SARS-CoV-2 gene detection kit (Kyokuto Pharmaceutical Industrial; Kyokuto), SARS-CoV-2 Direct Detection RT-qPCR Kit (Takara Bio, Otsu, Japan; Takara), and SARS-CoV-2 Detection Kit -Multi- (Toyobo, Osaka, Japan; Toyobo). These kits use a similar workflow of mixing raw respiratory samples with sample treatment reagents, followed by heating, addition of RT-PCR reagents, and detection of fluorescent signals on real-time PCR machines. The QuantStudio® 5 Real-Time PCR System (Thermo Fisher Scientific, Waltham, MA, USA) was used for all of the assays except for the Takara kit, in which the LightCycler® 480 System II was used (Roche, Basel, Switzerland). The Toyobo kit was also tested using 5 µL of extracted RNA in a 25 µL reaction volume because it officially supported a workflow using extracted RNA. RNA was extracted from 140 µL of the sample using the QIAamp® Viral RNA Mini Kit (Qiagen, Hilden, Germany) and was eluted in a final volume of 60 µL, following the manufacturer's protocol of “Purification of Viral RNA (Spin Protocol)”.
Table 1.
Summary of the detection assays used in this study.
| Assay principle, name, manufacturer, regulatory status | Detection target | Applicable respiratory specimens | Internal control included? | Volume of template/reaction (µL) | Positive interpretive criteria | Reagent costa | Reaction time (min.) |
|---|---|---|---|---|---|---|---|
| Direct RT-PCR | |||||||
| 2019 Novel Coronavirus Detection kit (241–09560–91), Shimadzu (Kyoto, Japan), RUO-C | Two regions in N gene (CDC N1/N2 primer/probe sets) | Allb | Yes | 5/20 | Ct < 40 for either N1 or N2 | $2250/100 tests | 5c+88 |
| SARS-CoV-2 gene detection kit (551–69460–4), Kyokuto (Tokyo, Japan), RUO | Two regions in N gene (NIID N/N2 primer/probe sets) | Nasopharyngeal swabs | Yes | 2/17 | Ct < 40 for either N or N2 | $900/50 tests | 5c+80 |
| SARS-CoV-2 Direct Detection RT-qPCR Kit (RC300a), Takara, RUO-C | Two regions in N gene (NIID N/N2 primer/probe sets) in one channel | Allb | No | 8/50 | Ct ≤ 40 for N/N2 | $1200/100 tests | 5c+66 |
| SARS-CoV-2 Detection Kit -Multi- (NCV-403), Toyobo, RUO-C | Two regions in N gene (modified CDC N1/N2 primer/probe sets) | Allb | Yes | 8/51 | Ct < 40 for either N1 or N2 | $980/100 tests | 5c+66 |
| Direct RT-LAMP | |||||||
| Loopamp® SARS-CoV-2 detection kit (LMP403), Eiken, IVD | RdRP and N genes | Saliva | No | 0.24d/25 | Increase in the turbidity within 35 min (interpreted by a turbidimeter) | $968/48 tests | 35 |
| Yamazaki, in-house, RUO | ORF1ab, S, and ORF7 genes | Allb | No | 10e/50 | Increase in the fluorescence within 25 min | $3/test | 5c+25 |
| Lateral flow immunoassay | |||||||
| ESPLINE® SARS-CoV-2 (260319), Fujirebio, IVD | SARS-CoV-2 antigen | Nasopharyngeal swabs | Yes | 13f/20 | Visual detection of test lines | $72/10 tests | 5f+30 |
| RT-PCR | |||||||
| SARS-CoV-2 Detection Kit -Multi- (NCV-403), Toyobo, RUO-C | Two regions in N gene (modified CDC N1/N2 primer/probe sets) | Allb | Yes | 5g/25 | Ct < 40 for either N1 or N2 | $980/200 tests | 66 |
| NIID N2, in-house, RUO-C | N gene (NIID N2 primer/probe set) | Allb | No | 5g/20 | Ct < 40 | $2/test | 68 |
| CDC N1/N2, in-house, RUO | Two regions in N gene (CDC N1/N2 primer/probe sets) | Allb | Yes (separate reaction) | 5g/20 | Ct < 40 for both N1 and N2h | $6/test | 88 |
RUO-C, research use only but approved for clinical diagnostic use in Japan; RUO, research use only; RT-LAMP, reverse transcription loop-mediated isothermal amplification; IVD, in-vitro diagnostics in Japan; NIID, National Institute of Infectious Disease in Japan; CDC, Centers for Disease Control and Prevention.
Calculated at an exchange rate of 100 yen = $1. Only the cost for each reaction reagent was included.
Includes nasopharyngeal swabs in viral transport media, as well as saliva and sputum specimens.
Time of the heat treatment before the reactions.
100 µL of saliva was mixed with 4 mL of proprietary extraction reagent (Loopamp® viral RNA extraction reagent for influenza; LMP801), 10 µL of which was used for the reaction (10 µl of 41-fold diluted sample was used.).
Ten microliters of saliva or sputum or 5 µL of nasopharyngeal swabs diluted with 5 µL of TE buffer (pH 8.0) were heated at 95 °C for 5 min and mixed with the in-house LAMP reagent.
A 20 µL mixture was obtained from 20 µL of nasopharyngeal samples and 10 µL of proprietary concentrated sample treatment solution (20 µl of 1.5-fold diluted sample was used), which was incubated at room temperature for 5 min before application onto the cassette.
Five microliters of extracted RNA template, which corresponded to approximately 11.7 µl of raw sample, was used for the RT-PCR. The raw sample volume was calculated by the fact that 140 µl of raw sample underwent extraction and was eluted in 60 µl.
Samples were defined as positive when both the N1 and N2 assays were positive with Ct values <40. Samples were defined as negative when both the N1 and N2 assays were negative and the RNaseP assay (control reaction) was positive. Samples were defined as inconclusive when either N1 or N2 was positive and the other was negative.
Two LAMP assays were performed: Loopamp® SARS-CoV-2 detection kit (Eiken Chemical, Tokyo, Japan; hereinafter referred to as Eiken) and an in-house colorimetric assay developed by Yamazaki et al. [6] (see Supplementary material for details). For the Eiken kit, the reaction was monitored using a LoopampEXIA® real-time turbidimeter (Eiken Chemical). For Yamazaki's in-house assay, the reaction was monitored using QuantStudio5.
A commercial lateral flow immunoassay kit for antigen testing, ESPLINE® SARS-CoV-2 (Fujirebio, Tokyo, Japan; hereinafter referred to as Fujirebio), was also included. The test line of the strip was observed by the naked eye.
2.3. Reference assays
The National Institute of Infectious Disease in Japan (NIID) N2 and the US Centers for Disease Control and Prevention (CDC) N1/N2 RT-PCR assays were regarded as the reference standards (Table 1). The reference assays were performed as described previously [7]. Results of the reference assays were blinded to the performers of the direct detection assays. Further details are given in Supplementary material.
2.4. Analytical sensitivity
We determined the limit of detection (LOD) using a minimum of six replicates of twofold or tenfold serial dilutions of phosphate-buffered saline with the heat-inactivated SARS-CoV-2 strain (ATCC® VR-1986HK™) starting from 1000 genome copies/mL sample. The LOD was defined as the lowest concentration at which 19 of 20 (95%) replicates were positive. Phosphate-buffered saline was used since a negative matrix that mimics clinical samples could not be employed due to the absence of specimen types that were common to all assays (Table 1).
2.5. Statistical analysis
The agreement of the assays was assessed by Cohen's kappa concordance coefficient. The positive percent agreement (PPA) and negative percent agreement (NPA) were compared using the McNemar test. The PPA of different specimen types was compared using Fisher's exact test. The Ct values were compared using the Kruskal–Wallis test or a Mann–Whitney U test. A p-value <0.05 was considered statistically significant. All statistical analyses were performed using SAS® Studio 3.8 (SAS Institute Inc., Cary, NC). Visualization of the Ct values was conducted using R (https://cran.r-project.org) and ggplot2 (https://ggplot2.tidyverse.org).
3. Results
3.1. Analytical sensitivity
The analytical sensitivity analysis showed that among the seven direct detection assays, Toyobo exhibited the lowest LOD of 1000 copies/mL (Table 2). The other three RT-PCR assays showed LODs between 2500–50,000 copies/mL and the two RT-LAMP assays showed higher LODs between 50,000–100,000 copies/mL. The larteral flow immunoassay Fujirebio had a much higher LOD of 2500,000 copies/mL. When purified RNA was used instead of raw sample, the Toyobo assay showed a twofold lower LOD.
Table 2.
Analytical sensitivity of the direct detection assays.
| Assay principle, manufacturer | Limit of detectiona, genome copies/mL sample (95% CI) | Viral genome copies/mL sample, positive rate (no. of replicates, positive/tested) |
|||
|---|---|---|---|---|---|
| Dilution 1 | Dilution 2 | Dilution 3 | Dilution 4 | ||
| Direct RT-PCR | |||||
| Shimadzu | 2500 | 5000, 100% (10/10) | 2500, 100% (20/20) | 1000, 90% (18/20) | 500, 80% (8/10) |
| Kyokuto | 50,000 | 100,000, 100% (10/10) | 50,000, 95% (19/20) | 25,000, 73% (8/11) | 10,000, 36% (4/11) |
| Takara | 10,000 | 25,000, 100% (10/10) | 10,000, 87% (19/20) | 5000, 80% (16/20) | 2500, 40% (4/10) |
| Toyobo | 1000 | 2500, 100% (10/10) | 1000, 100% (20/20) | 500, 63% (12/19) | 250, 33% (3/9) |
| Direct RT-LAMP | |||||
| Eiken | 100,000 | 250,000, 100% (10/10) | 100,000, 100% (20/20) | 50,000, 79% (11/14) | 25,000, 71% (10/14) |
| Yamazakib | 50,000 | 100,000, 100% (10/10) | 50,000, 100% (20/20) | 25,000, 60% (6/10) | 10,000, 50% (3/6) |
| Lateral flow immunoassay | |||||
| Fujirebio | 2500,000 | 5000,000, 100% (10/10) | 2500,000, 100% (20/20) | 1000,000, 0% (0/6) | |
| RT-PCR | |||||
| Toyobo | 500 | 1000, 100% (10/10) | 500, 100% (20/20) | 250, 90% (18/20) | 100, 50% (3/6) |
RT-LAMP, reverse transcription loop-mediated isothermal amplification; CI, confidence interval.
The copy numbers per reaction for the direct assays can be calculated by multiplying the values of template volume (e.g., 0.005 for the SHIMADZU assays). For the Toyobo assay using purified RNA (RT-PCR), the copy numbers per reaction for the direct assays can be calculated by multiplying the values by 0.0117 (140 µL sample, 60 µL elution in RNA purification step, and 5 µL of purified RNA for reaction).
Five microliters of template (test condition for the nasopharyngeal swabs) was used for the analysis.
3.2. Reference assays
The results of the reference NIID N2 and CDC N1/N2 assays were concordant, except for one nasopharyngeal sample. The discordant sample was inconclusive on the CDC assay (N1 negative and N2 positive) and was positive on the NIID N2 assay. The chart review revealed that the patient had a history of close contact with a COVID-19 patient, presented with pneumonia, and was treated for COVID-19. Therefore, this sample was considered a true positive. The Ct values of the NIID N2 assay indicated that the sputum samples had lower viral loads than the saliva samples (Fig. 1 ).
Fig. 1.
Ct values of the reference NIID N2 assay for the 80 true positive samples.
Box-and-whisker plots are shown with individual values, according to the specimen types. Statistical comparison of the Ct values among the three specimen types found significant differences (p = 0.02). The Ct values of the sputum samples were significantly higher than those of saliva.
3.3. Diagnostic performance
The diagnostic performances of the direct detection assays used to evaluate the clinical samples are summarized in Table 3. The Toyobo assay had the highest PPA of 96.3%, followed by the Takara (87.5%), Eiken (87.5%), and Shimadzu (86.3%) assays. The Kyokuto assay had a PPA of 71.9%; however, as many as 28% of the samples were judged invalid due to the absence of amplification of the internal control. Invalid results were not observed in any of the other assays. The Yamazaki and Fujirebio assays had low PPAs of 58.8% and 44.7%, respectively. The PPAs varied according to the specimen types (Table 4). For testing the nasopharyngeal swabs, the Toyobo and Shimadzu assays showed excellent PPAs of ≥97.4%, followed by the Takara assay (89.5%). For testing the saliva samples, the Toyobo, Shimadzu and Takara assays showed ≥95.8% PPAs and that of Eiken (87.5%) was lower. The Yamazaki assay had a similar PPA (83.3%) to the Eiken assay. For testing the sputum samples, all of the assays (Shimadzu, Takara, and Yamazaki) except the Toyobo assay showed significantly lower PPAs (33.3%–66.7%) than the reference standard. All of the assays exhibited an NPA of 100%, except the Eiken assay, which had one false-positive result (96.3% NPA). The Toyobo assay that utilized extracted RNA showed 100% concordant results with the reference.
Table 3.
Overall diagnostic performance of the direct detection assays.
| Assay principle, manufacturer | Number of positive/negative samples tested | PPA (95% CI) | NPA (95% CI) | Kappa (95% CI) |
|---|---|---|---|---|
| Direct RT-PCR | ||||
| Shimadzu | 80/103 | 86.3%a (76.7–92.9%) | 100% (96.4–100%) | 0.88 (0.80–0.95) |
| Kyokuto | 38/49b (nasopharyngeal swabs only) | 71.9%a (53.2–86.3%) | 100% (88.7–100%) | 0.72 (0.55–0.89) |
| Takara | 80/103 | 87.5%a (78.2–93.9%) | 100% (96.4–100%) | 0.89 (0.81–0.96) |
| Toyobo | 80/103 | 96.3% (89.4–99.3%) | 100% (96.4–100%) | 0.97 (0.92–1) |
| Direct RT-LAMP | ||||
| Eiken | 24/27 (saliva only) | 87.5% (67.6–97.4%) | 96.3% (81.0–100%) | 0.84 (0.69–1) |
| Yamazaki | 80/103 | 58.8%a (47.1–69.7%) | 100% (96.4–100%) | 0.62 (0.50–0.73) |
| Lateral flow immunoassay | ||||
| Fujirebio | 38/49 (nasopharyngeal swabs only) | 44.7%a (28.6–61.7%) | 100% (92.7–100%) | 0.48 (0.31–0.65) |
| RT-PCR | ||||
| Toyobo | 80/103 | 100% (95.4–100%) | 100% (96.4–100%) | 1c |
RT-LAMP, reverse transcription loop-mediated isothermal amplification; PPA, positive percent agreement; NPA, negative percent agreement; CI, confidence interval.
P < 0.05 in comparison with the defined reference standard.
Tests for 24 samples (27.6%; 6 true positive and 18 true negative samples) were invalid due to the absence of positive signals for the internal control target. Repeat testing yielded the same results. If these samples were included in the calculation, the PPA was 60.5% (43.3–76.0%), the NPA was 63.3% (48.2–76.6%), and the Kappa coefficient was 0.24 (0.03–0.45).
95% CI could not be calculated.
Table 4.
Diagnostic performance of the direct detection assays according to the specimen types.
| Assay, manufacturer | Nasopharyngeal swab, 38/49a |
Saliva, 24/27a |
Sputum, 18/27a |
||||||
|---|---|---|---|---|---|---|---|---|---|
| PPA (95% CI) | NPA (95% CI) | Kappa (95% CI) | PPA (95% CI) | NPA (95% CI) | Kappa (95% CI) | PPA (95% CI) | NPA (95% CI) | Kappa (95% CI) | |
| Direct RT-PCR | |||||||||
| Shimadzu | 97.4% (86.1–100%) | 100% (92.7–100) | 0.98 (0.93–1) | 95.8% (78.9–99.9%) | 100% (87.2–100) | 0.96 (0.88–1) | 50.0%b (26.0–74.0%) | 100% (87.2–100) | 0.55 (0.30–0.79) |
| Kyokuto | 71.9%b,c (53.2–86.3%) | 100%c (88.7–100%) | 0.72c (0.55–0.89) | NA | NA | NA | NA | NA | NA |
| Takara | 89.5% (75.2–97.1%) | 100% (92.7–100) | 0.91 (0.81–1.00) | 100% (86.2–100) | 100% (87.2–100) | 1d | 66.7%b (40.9–86.7%) | 100% (87.2–100) | 0.71 (0.49–0.92) |
| Toyobo | 100% (90.7–100) | 100% (92.7–100) | 1d | 100% (86.2–100) | 100% (87.2–100) | 1d | 83.3% (58.5–96.5%) | 100% (87.2–100) | 0.86 (0.70–1) |
| Direct RT-LAMP | |||||||||
| Eiken | NA | NA | NA | 87.5% (67.6–97.4%) | 96.3% (81.0–100%) | 0.84 (0.69–1.00) | NA | NA | NA |
| Yamazaki | 55.3%b (38.3–71.4%) | 100% (92.7–100) | 0.58 (0.42–0.75) | 83.3% (62.6–95.3%) | 100% (87.2–100) | 0.84 (0.69–0.99) | 33.3%b (13.3–59.1%) | 100% (87.2–100) | 0.38 (0.13–0.62) |
| Lateral flow immunoassay | |||||||||
| Fujirebio | 44.7%b (28.6–61.7%) | 100% (92.7–100) | 0.48 (0.31–0.65) | NA | NA | NA | NA | NA | NA |
| RT-PCR | |||||||||
| Toyobo | 100% (90.7–100) | 100% (88.7–100) | 1d | 100% (86.2–100) | 100% (87.2–100) | 1d | 100% (81.4–100) | 100% (87.2–100) | 1d |
PPA, positive percent agreement; NPA, negative percent agreement; CI, confidence interval; NA, not applicable.
Number of true positive/total samples.
P < 0.05 in comparison with the defined reference standard.
Tests for 24 samples (27.6%; 6 true positive and 18 true negative samples) were invalid due to the absence of positive signals for the internal control target. Repeat testing yielded the same results. If these samples were included in the calculation, the PPA was 60.5% (43.3–76.0%), the NPA was 63.3% (48.2–76.6%), and the Kappa coefficient was 0.24 (0.03–0.45).
95% CI could not be calculated.
4. Discussion
We performed a manufacturer-independent evaluation of direct molecular and antigen detection assays. The analytical sensitivities varied according to the assays. The LODs of all of the direct detection assays (Table 2 ) were higher than those of the reference assays (NIID N2, 391 copies/mL and CDC N1/N2, 256 copies/mL) that were quantified using the same SARS-CoV-2 strain by our group [7]. The LODs of the Toyobo and Shimadzu assays were 1000–2500 copies/mL and were superior to those of the other direct RT-PCR assays and RT-LAMP assays (one to two orders of magnitude higher copies/mL). At least three possible factors may be associated with the higher LODs compared with the reference assays and the variations in the LODs of these direct assays. First, a lower sample volume was used for the direct molecular assays. Approximately 11.7 µl of sample was used for the reference RT-PCR when 140 µl of sample underwent extraction and was eluted in 60 µl, and 5 µl of elute was used for the reaction. In contrast, 2–10 µl was used for direct RT-PCR, and 0.24 µl or 10 µl was used for RT-LAMP (Table 1). Second, the specimen lysis methods (heat, enzymatic, chemical lysis, or combination of these) were different among the assays. The heating conditions, the addition of proteinase K treatment or polyvinylsulfonic acid can influence the detection sensitivity [4, 8, 9]. Third, the mastermixes, primers, probes, and thermal cycling conditions of the RT-PCR may be associated [10, 11]. Of these, the primers and probes may not explain the difference in the 4 direct RT-PCR assays evaluated in this study. These assays utilized the CDC or NIID primer/probe sets that were used in the reference assays, and genetic variations that may compromise detection sensitivities have rarely been observed [10]. The antigen assay Fujirebio showed an LOD that was four orders of magnitude higher for copies/mL (2.5 million copies/mL) than those of the reference assays. This value is concordant with a previous study that found that the LOD of Fujirebio (1.2 million copies/mL) was one of the best among the 19 lateral flow antigen assays studied [12].
The differences in the LODs correlated with the PPAs for clinical specimens (Table 3 ), but they differed according to the specimen type (Table 4 ). Some direct detection assays only can evaluate limited specimen types, and we examined only valid specimen types (Table 1) in this study. The PPAs for testing the sputum with these assays were generally low, which made the overall PPAs of the applicable assays for sputum samples (Toyobo, Shimadzu, Takara, and Yamazaki) lower. The low PPAs of the assays are considered to be associated with the lower viral loads in sputum samples (Fig. 1), and it is difficult to speculate the suitability of sputum for direct detection. For nasopharyngeal swabs and saliva, the direct RT-PCR assays of Toyobo and Shimadzu had a good performance of >95% PPAs, and the performance of Takara followed. The Kyokuto assay yielded many invalid results for nasopharyngeal samples and was considered unsuitable for clinical tests. The direct RT-LAMP assays of Eiken and Yamazaki had PPAs of approximately 85% for saliva, which were lower than the direct RT-PCR assays. Eiken RT-LAMP is considered less sensitive even if it is used with extracted RNA [7, 13], which may explain the lower PPAs of RT-LAMP than the RT-PCR assays. Only one Japanese study evaluated multiple direct detection assays to evaluate the saliva of laboratory confirmed positive patients, and the samples had been collected a median of seven days after hospital admission [14]. Compared with the 80.6% sensitivity of the NIID N2 reference assay, the Takara, Shimadzu, Eiken and Fujirebio assays showed sensitivities of 76.7%, 78.6%, 70.9%, and 11.7%, respectively. These values of the direct RT-PCR assays correspond to an approximately 90% PPA against the reference assay, which is comparable to our results. The Fujirebio assay exhibited a lower PPA than our results for nasopharyngeal swabs, but it was not approved for saliva. Two studies that examined the clinical performance of the Fujirebio assay for nasopharyngeal swabs reported PPAs of 39–48% against reference RT-PCR assays [15, 16]. These values are in agreement with the PPA in this study (44%). Overall, the RT-LAMP and lateral flow antigen assays showed lower PPAs than the direct RT-PCR assays, but they can be used for patients who are expected to have high viral loads, such as symptomatic patients with early disease.
The RT-LAMP Eiken assay produced a false-positive result. The RT-LAMP method is challenged by low specificity because of the use of multiple primers that are potentially associated with nonspecific amplification [1]. The reported pooled specificity of the direct RT-LAMP assays was as low as 96% [17]. Concerns for false positives have also been reported for lateral flow antigen assays. In a screening study with 71,847 tests, the Abbott and BD direct antigen assays showed 30–77% false positive rates in a low prevalence setting (0.05%) [18]. Following these observations, it may be prudent to perform confirmatory RT-PCR tests in a low prevalence setting.
For the quality assurance of molecular detection tests, it is important to include internal control reactions to avoid false negatives resulting from insufficient extraction and/or PCR inhibition [19]. However, RT-LAMP usually cannot discriminate between the signals of different targets and thus cannot include internal controls. The Eiken and Yamazaki RT-LAMP assays also lacked internal controls (Table 1). This limits the clinical utility of RT-LAMP assays, especially when they are combined with direct detection methods, which have a higher risk of reaction inhibition.
The study limitations included a relatively small sample size of each specimen type, potential sampling bias, and the lack of paired samples for all specimen types. We could not evaluate differences in diagnostic performance according to the sample type. Additionally, there is a lack of analytical specificity analysis, clinical information, repeated measurements by multiple investigators, detailed examinations for the samples that yielded discrepant results among different assays, and a genomic variation analysis in this study. The strengths included the evaluation of multiple direct detection assays using the same clinical samples in the absence of manufacturer-independent assessments for several of the assays.
5. Conclusions
We found that large differences in analytical sensitivities and PPAs for the clinical specimens existed among the seven direct detection assays. Based on our results, the direct RT-PCR assays of Toyobo, Shimadzu and Takara were useful for evaluating nasopharyngeal swabs and saliva. The direct RT-LAMP assays of Eiken and Yamazaki for evaluating saliva are faster than direct RT-PCR assays and can be used in a resource-poor setting. However, it is noted that their PPAs were inferior to direct RT-PCR assays, and they lacked internal controls. The use of the lateral flow antigen Fujirebio assay may be limited to only high-risk populations. These direct detection assays help increase the number of tests and decrease the turnaround time even under supply shortages, but clinical laboratories need to take these characteristics into consideration before implementing these assays.
Ethics approval
This study was performed in line with the principles of the Declaration of Helsinki. The Ethics Committee of Kyoto University Graduate School and the Faculty of Medicine approved this study (R2379) and waived the need to obtain informed consent from each patient.
CRediT authorship contribution statement
Yasufumi Matsumura: Conceptualization, Methodology, Formal analysis, Investigation, Data curation, Writing – original draft, Writing – review & editing. Wataru Yamazaki: Resources, Investigation, Writing – review & editing. Taro Noguchi: Resources, Investigation, Writing – review & editing. Masaki Yamamoto: Resources, Investigation, Writing – review & editing. Miki Nagao: Resources, Investigation, Writing – review & editing, Funding acquisition.
Declaration of Competing Interest
The authors declare the following financial interests/personal relationships which may be considered as potential competing interests:
Y.M. received research funds from Toyobo, Beckman Coulter, and Precision System Science. M.N. received research funds from Beckman Coulter and Precision System Science. The funding organizations had no role in the study design, data collection and interpretation, or the decision to submit the work for publication.
Acknowledgments
We thank Tsunehiro Shimizu and Akihiko Hayashi (Kyoto City Hospital, Kyoto, Japan) for their technical assistance. Funding: This research was supported by the COVID-19 Private Fund (to the Shinya Yamanaka laboratory, CiRA, Kyoto University). The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.
Footnotes
Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.jcvp.2023.100138.
Appendix. Supplementary materials
References
- 1.Mardian Y., Kosasih H., Karyana M., Neal A., Lau C.Y. Review of current COVID-19 diagnostics and opportunities for further development. Front. Med. (Lausanne) 2021;8 doi: 10.3389/fmed.2021.615099. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Sethuraman N., Jeremiah S.S., Ryo A. Interpreting diagnostic tests for SARS-CoV-2. JAMA. 2020;323:2249–2251. doi: 10.1001/jama.2020.8259. [DOI] [PubMed] [Google Scholar]
- 3.Nolte F.S., Babady N.E., Buchan B.W., Capraro G.A., Graf E.H., Leber A.L., et al. Responding to the challenges of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2): perspectives from the association for molecular pathology infectious disease subdivision leadership. J. Molec. Diagn. 2020;22:968–974. doi: 10.1016/j.jmoldx.2020.06.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Smyrlaki I., Ekman M., Lentini A., Rufino de Sousa N., Papanicolaou N., Vondracek M., et al. Massive and rapid COVID-19 testing is feasible by extraction-free SARS-CoV-2 RT-PCR. Nat. Commun. 2020;11:4812. doi: 10.1038/s41467-020-18611-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Tali S.H.S., LeBlanc J.J., Sadiq Z., Oyewunmi O.D., Camargo C., Nikpour B., et al. Tools and techniques for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2)/COVID-19 detection. Clin. Microbiol. Rev. 2021;34:e00228. doi: 10.1128/CMR.00228-20. -20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Yamazaki W., Matsumura Y., Thongchankaew-Seo U., Yamazaki Y., Nagao M. Development of a point-of-care test to detect SARS-CoV-2 from saliva which combines a simple RNA extraction method with colorimetric reverse transcription loop-mediated isothermal amplification detection. J. Clin. Virol. 2021;136 doi: 10.1016/j.jcv.2021.104760. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Matsumura Y., Shimizu T., Noguchi T., Nakano S., Yamamoto M., Nagao M. Comparison of 12 molecular detection assays for severe acute respiratory syndrome Coronavirus 2 (SARS-CoV-2) J. Mol. Diagn. 2021;23:164–170. doi: 10.1016/j.jmoldx.2020.11.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Lübke N., Senff T., Scherger S., Hauka S., Andrée M., Adams O., et al. Extraction-free SARS-CoV-2 detection by rapid RT-qPCR universal for all primary respiratory materials. J. Clin. Virol. 2020;130 doi: 10.1016/j.jcv.2020.104579. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Chu A.W., Chan W.M., Ip J.D., Yip C.C., Chan J.F., Yuen K.Y., et al. Evaluation of simple nucleic acid extraction methods for the detection of SARS-CoV-2 in nasopharyngeal and saliva specimens during global shortage of extraction kits. J. Clin. Virol. 2020;129 doi: 10.1016/j.jcv.2020.104519. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Penarrubia L., Ruiz M., Porco R., Rao S.N., Juanola-Falgarona M., Manissero D., et al. Multiple assays in a real-time RT-PCR SARS-CoV-2 panel can mitigate the risk of loss of sensitivity by new genomic variants during the COVID-19 outbreak. Int. J. Infect. Dis. 2020;97:225–229. doi: 10.1016/j.ijid.2020.06.027. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Visseaux B., Collin G., Houhou-Fidouh N., Le Hingrat Q., Ferré V.M., Damond F., et al. Evaluation of three extraction-free SARS-CoV-2 RT-PCR assays: a feasible alternative approach with low technical requirements. J. Virol. Methods. 2021;291 doi: 10.1016/j.jviromet.2021.114086. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Cubas-Atienzar A.I., Kontogianni K., Edwards T., Wooding D., Buist K., Thompson C.R., et al. Limit of detection in different matrices of 19 commercially available rapid antigen tests for the detection of SARS-CoV-2. Sci. Rep. 2021;11:18313. doi: 10.1038/s41598-021-97489-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Karino M., Harada M., Yamada C., Fukuoka K., Sugo M., Hanada H., et al. Evaluation of the efficacy of LAMP-based SARS-CoV-2 detection with simple RNA extraction from nasopharyngeal swabs: a prospective observational study. PLoS ONE. 2021;16 doi: 10.1371/journal.pone.0260732. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Nagura-Ikeda M., Imai K., Tabata S., Miyoshi K., Murahara N., Mizuno T., et al. Clinical evaluation of self-collected saliva by quantitative reverse transcription-PCR (RT-qPCR), Direct RT-qPCR, reverse transcription-loop-mediated isothermal amplification, and a rapid antigen test to diagnose COVID-19. J. Clin. Microbiol. 2020;58:e01438. doi: 10.1128/JCM.01438-20. -20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Basso D., Aita A., Padoan A., Cosma C., Navaglia F., Moz S., et al. Salivary SARS-CoV-2 antigen rapid detection: a prospective cohort study. Clin. Chim Acta. 2021;517:54–59. doi: 10.1016/j.cca.2021.02.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Aoki K., Nagasawa T., Ishii Y., Yagi S., Kashiwagi K., Miyazaki T., et al. Evaluation of clinical utility of novel coronavirus antigen detection reagent, Espline® SARS-CoV-2. J. Infect. Chemother. 2021;27:319–322. doi: 10.1016/j.jiac.2020.11.015. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Subsoontorn P., Lohitnavy M., Kongkaew C. The diagnostic accuracy of isothermal nucleic acid point-of-care tests for human coronaviruses: a systematic review and meta-analysis. Sci. Rep. 2020;10:22349. doi: 10.1038/s41598-020-79237-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Sexton M.E., Kraft C.S., Tang Y.-.W. Routine antigen testing is not a substitute for health care worker vaccination against SARS-CoV-2. J. Clin. Microbiol. 2021;59:e01564. doi: 10.1128/JCM.01564-21. -21. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Burkardt H.J. Standardization and quality control of PCR analyses. Clin. Chem. Lab. Med. 2000;38:87–91. doi: 10.1515/CCLM.2000.014. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.