Abstract
Molecular testing for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is the mainstay for accurate diagnosis of the infection, but the diagnostic performances of available assays have not been defined. We compared 12 molecular diagnostic assays, including 8 commercial kits using 155 respiratory samples (65 nasopharyngeal swabs, 45 oropharyngeal swabs, and 45 sputum) collected at two Japanese hospitals. Sixty-eight samples were positive for more than one assay and one genetic locus, and were defined as true-positive samples. All the assays showed a specificity of 100% (95% CI, 95.8%–100%). The N2 assay kit of the US Centers for Disease Control and Prevention and the N2 assay of the Japanese National Institute of Infectious Disease (NIID) were the most sensitive assays with 100% sensitivity (95% CI, 94.7–100), followed by the Centers for Disease Control and Prevention N1 kit, E assay by Corman, and Japanese National Institute of Infectious Disease N2 assay multiplex with internal control reactions. These assays are reliable as first-line molecular assays in laboratories when combined with appropriate internal control reactions.
Accurate detection tests for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) are important to combat the coronavirus disease 2019 (COVID-19) pandemic.1 Various molecular diagnostic assays have been developed and used worldwide,1, 2, 3, 4 but the differences in their diagnostic performances remain poorly understood. In this study, the authors aimed to compare the performance of 12 molecular assays.
Materials and Methods
Clinical Specimens
A total of 923 upper or lower respiratory tract samples (nasopharyngeal swabs and oropharyngeal swabs in viral transport media or sputum) were collected from 446 patients who were suspected to have COVID-19 between January and May 2020 at Kyoto University Hospital and Kyoto City Hospital. In this study, All SARS-CoV-2–positive samples (n = 68) from 39 patients with COVID-19 and 87 negative samples that were randomly selected from the remaining 855 samples were included. These negative samples were obtained from 69 patients, including 2 patients with COVID-19, of which samples were collected for follow-up testing. The N2 assay developed by the National Institute of Infectious Disease (NIID) in Japan4 was employed as the routine assay and was used for the above sample selection process.
RNA Extraction
Sputum samples were liquefied using semialkaline protease (Sputazyme; Kyokuto Pharmaceutical Industrial, Tokyo, Japan) before further processing. The respiratory samples were prospectively stored at −80°C after stabilization by mixing an equal volume of DNA/RNA Shield (2X concentrate; Zymo Research, Irvine, CA). The thawed samples were centrifuged at 20,000 × g for 2 minutes. RNA was extracted from 140 μL of the supernatant using the QIAamp Viral RNA Mini Kit (Qiagen, Hilden, Germany) with RNA extraction controls—5 μL of LightMix Modular EAV RNA Extraction Control (EAV; Roche, Basel, Switzerland) or 10 μL of MS2 phage (Thermo Fisher Scientific, Waltham, MA)—and eluted in a final volume of 60 μL.
Molecular Assays
Table 1 shows the molecular assays evaluated in this study. Real-time RT-PCR was performed using N1, N2, and RNaseP (RP) internal control assays developed by the Centers for Disease Control and Prevention (2019-nCoV CDC EUA kit, obtained from Integrated DNA Technologies, Coralville, IA), N2 assay developed and distributed by the NIID in Japan4 (with/without EAV), and N and E assays developed by Charité in Germany1 (Corman) with TaqPath 1-Step RT-qPCR Master Mix, CG (Thermo Fisher Scientific). The LightMix Modular assays (Roche) for envelope protein (E), RNA-dependent RNA polymerase (RdRP), and nucleocapsid protein (N) genes multiplexed with EAV, the Real-Time Fluorescent RT-PCR kit for detecting 2019-nCoV (BGI Biotechnology, Wuhan, China), and the TaqPath COVID-19 Combo Kit (Thermo Fisher Scientific) were also tested according to the manufacturers' instructions. The above reactions were performed using a LightCycler 480 System II (Roche), and threshold cycle (CT) values were determined by the second derivative maximum method, except for the CDC N1/N2 and TaqPath COVID-19 Combo Kit assays, which were performed using Applied Biosystems 7500 Fast or QuantStudio5 Real-Time PCR Systems (Thermo Fisher Scientific) using a fixed threshold of 0.1. A loop-mediated isothermal amplification (LAMP) assay was performed using a Loopamp SARS-CoV-2 detection kit LMP403 and LoopampEXIA real-time turbidimeter (Eiken Chemical, Tokyo, Japan).
Table 1.
Summary of the Molecular Assays Used in This Study
| Assay | Target gene (position in SARS-CoV-2 genome∗) | Internal control | Volume of template RNA/reaction, μL | Thermal cycling condition | PCR reagent | Reaction time, minutes | Regulatory status |
|---|---|---|---|---|---|---|---|
| CDC N1 kit | N (28,286 to 28,357) | RNaseP in separate reaction | 5/20 | 10 minutes at 50°C, 2 minutes at 95°C, 45 cycles of 3 seconds at 95°Cand 30 seconds at 55°C | TaqPath | 88 | EUA |
| CDC N2 kit | N (29,163 to 29,229) | RNaseP in separate reaction | 5/20 | 10 minutes at 50°C, 2 minutes at 95°C, 45 cycles of 3 seconds at 95°C and 30 seconds at 55°C | TaqPath | 88 | EUA |
| CDC RP kit | Human RNaseP | - | 5/20 | 10 minutes at 50°C, 2 minutes at 95°C, 45 cycles of 3 seconds at 95°C and 30 seconds at 55°C | TaqPath | 88 | EUA |
| NIID N2 | N† (29,142 to 29,280) | None | 5/20 | 15 minutes at 50°C, 2 minutes at 95°C, 45 cycles of 3 seconds at 95°C and 30 seconds at 60°C | TaqPath | 68 | RUO‡ |
| NIID N2 with EAV | N† (29,142 to 29,280) | EAV Extraction Control kit | 5/20 | 15 minutes at 50°C, 2 minutes at 95°C, 45 cycles of 3 seconds at 95°C and 30 seconds at 60°C | TaqPath | 68 | RUO |
| Corman E | E§ (26,268 to 26,380) | None | 5/20 | 15 minutes at 50°C, 2 minutes at 95°C, 45 cycles of 3 seconds at 95°C and 30 seconds at 60°C | TaqPath | 68 | RUO |
| Corman N | N† (28,555 to 28,682) | None | 5/20 | 15 minutes at 50°C, 2 minutes at 95°C, 45 cycles of 3 seconds at 95°C and 30 seconds at 60°C¶ | TaqPath | 68 | RUO‡ |
| Roche E kit | E | EAV Extraction Control kit | 5/20 | 5 minutes at 55°C, 5 minutes at 95°C, 45 cycles of 5 seconds at 95°C, 15 seconds at 60°C, and 15 seconds at 72°C | LightCycler | 65 | RUO‡ |
| Roche RdRP kit | RdRP | EAV Extraction Control kit | 5/20 | 5 minutes at 55°C, 5 minutes at 95°C, 45 cycles of 5 seconds at 95°C, 15 seconds at 60°C, and 15 seconds at 72°C | LightCycler | 65 | RUO |
| Roche N kit | N | EAV Extraction Control kit | 5/20 | 5 minutes at 55°C, 5 minutes at 95°C, 45 cycles of 5 seconds at 95°C, 15 seconds at 60°C, and 15 seconds at 72°C | LightCycler | 65 | RUO‡ |
| Thermo Combo kit | ORF1ab, S, N | MS2 phage extraction control | 5/20 | 10 minutes at 53°C, 2 minutes at 95°C, 40 cycles of 3 seconds at 95°C and 30 seconds at 60°C | Included | 67 | CE-IVD, EUA, JP-IVD |
| BGI kit | ORF1ab (3180 to 3280) | Human beta-actin gene | 10/30 | 20 minutes at 50°C, 10 minutes at 95°C, 40 cycles of 15 seconds at 95°C and 30 seconds at 60°C | Included | 90 | CE-IVD, EUA, JP-IVD |
| LAMP kit | RdRP, N | Not included | 10/25 | 35 minutes at 62.5°C | Included | 35 | JP-IVD |
CDC, Centers for Disease Control and Prevention; CE-IVD, European conformity-in vitro diagnostics; EUA, the US Food and Drug Administration Emergency Use Authorization; JP-IVD, in vitro diagnostics in Japan; LAMP, loop-mediated isothermal amplification; LightCycler, LightCycler Multiplex RNA Virus Master; NIID, National Institute of Infectious Disease in Japan; RUO, research use only; TaqPath, TaqPath 1-Step RT-qPCR Master Mix, CG.
GenBank accession number MN908947 (https://www.ncbi.nlm.nih.gov/nuccore).
FAM-TAMRA probe was used.
RUO, but approved for clinical diagnostic use in Japan. The Corman N assay is combined with the NIID N2 assay, and the Roche N assay is combined with the Roche E assay.
FAM-BHQ-1 probe was used.
Modified from the original condition (58°C). For the Corman N assay, the NIID recommended reaction was at 60°C.4
Analytical Sensitivity
The limit of detection (LOD) of each assay was determined using a minimum of six replicates of 2-fold or 10-fold serial dilutions of the heat-inactivated SARS-CoV-2 strain (ATCC VR-1986HK) starting from 1000 genome copies/mL. If an assay could not detect 1000 genome copies/mL, higher concentrations were assayed. The 95% LOD was calculated using probit analysis.
Statistical Analysis
At the time of manuscript preparation, no gold standard existed. In this study, to ensure the presence of SARS-CoV-2 RNA and to avoid false positives, a sample was defined as positive when positive test results were obtained for more than one genetic locus and assay, and the others were defined as negative. The agreement of the assays was assessed by the Cohen's kappa concordance coefficient. The sensitivity and specificity were compared using the McNemar’s test. The sensitivity of different specimen types was compared using the Fisher's exact test. The CT values were compared using the Kruskal-Wallis test or a U-test. P <0.05 was considered statistically significant. All statistical analyses were performed using SAS Studio software version 3.8 (SAS Institute Inc., Cary, NC).
Ethics Approval
This study was performed in line with the principles of the Declaration of Helsinki. The Ethics Committee of Kyoto University Graduate School and the Faculty of Medicine approved this study (R2379) and waived the need for obtaining informed consent from each patient.
Results
Analytical sensitivity analysis showed that the LODs of the CDC N1, CDC N2, NIID N2 (with/without EAV), Corman E, and BGI assays were as low as <500 genome copies/mL sample (range, 240 to 391 copies, corresponding to 2.8 to 4.6 copies/reaction) (Table 2 ). The Thermo Combo and Roche E assays followed with LODs of <1000 genome copies/mL sample. The LODs of the Roche RdRP and N assays were higher than those of the abovementioned assays, ranging from 2441 to 31,151 genome copies/mL sample.
Table 2.
Analytical Sensitivity of 12 Molecular Assays
| Assay | Limit of detection,∗ genome copies/mL sample (95% CI) | Viral genome copies/mL sample, positive rate (positive replicates, n/tested replicates, n) |
||||
|---|---|---|---|---|---|---|
| Dilution 1 | Dilution 2 | Dilution 3 | Dilution 4 | Dilution 5 | ||
| CDC N1 kit | 256 (144–1334) | 1000, 100% (6/6) | 500, 100% (10/10) | 250, 90% (9/10) | 100, 80% (8/10) | 50, 50% (3/6) |
| CDC N2 kit | 240 (148–851) | 1000, 100% (6/6) | 500, 100% (10/10) | 250, 90% (9/10) | 100, 78% (7/9) | 50, 33% (2/6) |
| NIID N2 | 391 (227–1455) | 1000, 100% (6/6) | 500, 100% (10/10) | 250, 80% (8/10) | 100, 50% (3/6) | 50, 33% (2/6) |
| NIID N2 with EAV | 391 (227–1455) | 1000, 100% (6/6) | 500, 100% (10/10) | 250, 80% (8/10) | 100, 50% (3/6) | 50, 33% (2/6) |
| Corman E | 327 (196–1123) | 1000, 100% (6/6) | 500, 100% (10/10) | 250, 80% (8/10) | 100, 83% (5/6) | 50, 17% (1/6) |
| Corman N | 4923 (2507–20,893) | 10,000, 100% (6/6) | 5000, 90% (9/10) | 2500, 83% (5/6) | 1000, 83% (5/6) | 500, 17% (1/6) |
| Roche E kit | 785 (504–3566) | 2500, 100% (6/6) | 1000, 100% (10/10) | 500, 67% (4/6) | 250, 50% (3/6) | 100, 0% (0/6) |
| Roche RdRP kit | 11,610† | 100,000, 100% (6/6) | 10,000, 100% (10/10) | 5000, 0% (0/6) | 2500, 17% (1/6) | 1000, 0% (0/6) |
| Roche N kit | 31,151 (17,828–236,207) | 100,000, 100% (6/10) | 50,000, 100% (10/10) | 25,000, 83% (5/6) | 10,000, 67% (4/6) | 5000, 17% (1/6) |
| Thermo Combo kit | 767 (480–2521) | 2500, 100% (6/6) | 1000, 100% (14/14) | 500, 80% (8/10) | 250, 50% (3/6) | 100, 33% (2/6) |
| BGI kit | 257 (139–1452) | 1000, 100% (6/6) | 500, 100% (10/10) | 250, 90% (9/10) | 100, 83% (5/6) | 50, 50% (3/6) |
| LAMP kit | 2441 (1337–19,566) | 5000, 100% (6/6) | 2500, 100% (10/10) | 1000, 67% (8/12) | 500, 50% (3/6) | 250, 33% (2/6) |
Calculated using probit analysis. The copy numbers per reaction can be calculated by multiplying the values by 0.0117 (0.0233 for the BGI and LAMP assays, which utilize 10 μL of PCR template).
95% CI could not be calculated.
Among a total of 155 clinical samples (65 nasopharyngeal swabs, 45 oropharyngeal swabs, and 45 sputum), 68 samples (35 nasopharyngeal swabs, 15 oropharyngeal swabs, and 18 sputum) were positive for more than one assay and one genetic locus, and were defined as true-positive samples; the other samples were considered true negative. A full list of the results with CT values is available as Supplemental Table S1.
All the assays exhibited a specificity of 100%, whereas the sensitivity varied (Table 3 ). The CDC N1, CDC N2, NIID N2 (with/without EAV), and Corman E assays were the most sensitive assays, with ≥95.6% sensitivity. These five assays displayed high overall agreement compared with the reference standard (kappa values of ≥0.96) and between any two of them (kappa values of ≥0.95). The CDC N2 and NIID N2 assays exhibited 100% sensitivity; thus, their results were equal to the defined reference standard. The sensitivities of the remaining seven assays (Corman N, Roche E, Roche RdRP, Roche N, Thermo Combo, BGI, and LAMP assays; ≤88.2%) were significantly lower than those of the most-sensitive assays.
Table 3.
Overall Diagnostic Performance of 12 Molecular Assays
| Assay | Sensitivity (95% CI) | Specificity (95% CI) | Kappa (95% CI) |
|---|---|---|---|
| CDC N1 kit∗ | 98.5% (92.1%–100%) | 100% (95.8%–100%) | 0.99 (0.96–1) |
| CDC N2 kit∗ | 100% (94.7%–100%) | 100% (95.8%–100%) | 1† |
| NIID N2 | 100% (94.7%–100%) | 100% (95.8%–100%) | 1† |
| NIID N2 with EAV‡ | 95.6% (87.6%–99.1%) | 100% (95.8%–100%) | 0.96 (0.91–1) |
| Corman E | 98.5% (92.1%–100%) | 100% (95.8%–100%) | 0.99 (0.96–1) |
| Corman N | 69.1%§ (56.7%–79.8%) | 100% (95.8%–100%) | 0.72 (0.60–0.83) |
| Roche E kit‡¶ | 86.8%§ (76.3%–93.8%) | 100% (95.8%–100%) | 0.88 (0.80–0.96) |
| Roche RdRP kit‡¶ | 42.6%§ (30.7%–55.3%) | 100% (95.8%–100%) | 0.46 (0.33–0.58) |
| Roche N kit‡¶ | 67.6%§ (55.2%–78.5%) | 100% (95.8%–100%) | 0.70 (0.59–0.82) |
| Thermo Combo kit‡ǁ | 85.3%§ (74.6%–92.8%) | 100% (95.8%–100%) | 0.87 (0.78–0.95) |
| BGI kit‡∗∗ | 88.2%§ (78.1%–94.8%) | 100% (95.8%–100%) | 0.89 (0.82–0.97) |
| LAMP kit | 80.9%§ (69.5%–89.5%) | 100% (95.8%–100%) | 0.83 (0.73–0.92) |
All samples yielded positive signals in separate CDC RNaseP reactions. The CDC N1 assay was negative, but the CDC N2 assay was positive for two true-positive samples. Repeat testing showed that one sputum sample was positive for both assays, whereas results of the other nasopharyngeal sample were unchanged. Thus, the former was considered positive, and the latter was considered inconclusive as the results of the CDC assay.
95% CI could not be calculated.
All reactions yielded positive signals for control targets.
P < 0.05 in comparison with the defined reference standard.
Cutoff was defined by two cycles higher than the observed CT value for 10 copies according to the manufacturer's instructions (E, 36.7; RdRP, 40; N, 39.3). When the fixed cutoff shown in the instructions was used (E, 36; RdRP, 39; N, 37), the sensitivity was changed as follows: E, 83.8%; RdRP, 36.8%; N, 50.0%, and the specificity was unchanged.
Seven samples were positive for only the N gene that warranted repeat testing. Repeat testing showed that four samples (two true-positive sputum samples, one true-positive pharyngeal sample, and one true-negative sputum sample) were negative for all genes, and these were considered negative. The other three true-positive sputum samples were positive again for only the N gene and were considered positive.
Four samples were positive, but the CT values were >38, which warranted repeat testing. Repeat testing showed that two true-negative pharyngeal samples and one true-negative nasopharyngeal sample were negative, and they were considered negative. The other true-positive sputum sample was positive again with a CT value of 39.12 and was considered positive.
The CDC protocol requires both N1 and N2 assays, and a sample will be considered positive if both produce positive results. In this study, one true-positive nasopharyngeal sample was positive only for the N2 assay even after retesting. The sample was considered inconclusive, and the performance of the CDC protocol was considered the same as the CDC N1 assay. The NIID protocol includes both NIID N2 and Corman N assays, and a sample will be considered positive if either assay produces a positive result. In this study, 69.1% of samples were positive for both assays, and 30.9% were positive for only the N2 assay. The protocol by Corman recommended an E assay that detects SARS-related viruses (Sarbecovirus) as a first-line screening assay and then SARS-CoV-2–specific RdRP assay for confirmatory testing.1 This approach defined only 49.2% of the Roche E assay–positive samples as SARS-CoV-2, although a single positive result of the Corman E or Roche E assay can be interpreted as SARS-CoV-2 positive in the absence of other Sarbecovirus. Assays with multiplexed internal control reactions and the CDC RNaseP assay yielded positive signals for all samples.
Table 4 shows the diagnostic performances for each specimen type. Nasopharyngeal swabs had a higher sensitivity than those of the other samples in all assays. These differences in the sensitivities were statistically significant in the Corman N assay for sputum samples and Roche N assay for oropharyngeal swabs and sputum samples. Viral loads in nasopharyngeal swabs, which were presumed from the CT values of CDC N2 and NIID N2 assays, were higher in nasopharyngeal swabs (median [interquartile range], 27.1 [23.6 to 31.1] and 29.7 [26.3 to 33.3], respectively) than those in oropharyngeal swabs (31.5 [29.9 to 35.0] and 33.0 [32.0 to 34.6]) or sputum (30.0 [25.6 to 33.5] and 30.9 [28.3 to 34.0]), but the differences did not reach statistical significance (P = 0.11 and 0.16 by comparison among 3 specimen types, respectively). The abundance of human cells, which were presumed from the CT values of the CDC RNaseP assay, was higher in sputum samples (25.6 [23.6 to 27.8]) than in nasopharyngeal or oropharyngeal swabs (28.2 [26.9 to 29.6]; P < 0.001 and 28.8 [26.9 to 31.0]; P < 0.001, respectively).
Table 4.
Diagnostic Performance of 12 Molecular Assays According to Specimen Types
| Assay | Nasopharyngeal swab, 35/65∗ |
Oropharyngeal swab, 15/45∗ |
Sputum, 18/45∗ |
|||
|---|---|---|---|---|---|---|
| Sensitivity (95% CI) | Kappa (95% CI) | Sensitivity (95% CI) | Kappa (95% CI) | Sensitivity (95% CI) | Kappa (95% CI) | |
| CDC N1 kit | 100% (90.0%–100%) | 1† | 93.3% (68.0%–99.9%) | 0.95 (0.85–1) | 100% (81.4%–100%) | 1† |
| CDC N2 kit | 100% (90.0%–100%) | 1† | 100% (78.2%–100%) | 1† | 100% (81.4%–100%) | 1† |
| NIID N2 | 100% (90.0%–100%) | 1† | 100% (78.2%–100%) | 1† | 100% (81.4%–100%) | 1† |
| NIID N2 with EAV | 100% (90.0–100) | 1† | 93.3% (68.0%–99.9%) | 0.95 (0.85–1) | 88.9% (65.2%–98.7%) | 0.91 (0.77–1) |
| Corman E | 100% (90.0%–100%) | 1† | 93.3% (68.0%–99.9%) | 0.95 (0.85–1) | 100% (81.4%–100%) | 1† |
| Corman N | 82.9%‡ (66.3%–93.5%) | 0.82 (0.67–0.96) | 60.0%‡ (32.2%–83.7%) | 0.67 (0.43–0.91) | 50.0%‡§ (26.0%–74.0%) | 0.55 (0.30–0.79) |
| Roche E kit | 94.3% (80.8%–99.3%) | 0.94 (0.85–1) | 73.3%‡ (44.9%–92.3%) | 0.79 (0.58–0.99) | 83.3% (58.5%–96.5%) | 0.86 (0.70–1) |
| Roche RdRP kite | 48.6%‡ (33.9%–68.7%) | 0.49 (0.31–0.68) | 33.3%‡ (38.3%–88.2%) | 0.73 (0.51–0.95) | 38.9%‡ (17.3%–64.3%) | 0.43 (0.19–0.68) |
| Roche N kit | 82.9%‡ (66.3%–93.5%) | 0.82 (0.67–0.96) | 53.3%‡§ (26.5%–78.8%) | 0.60 (0.35–0.86) | 50.0%‡§ (26.0%–74.0%) | 0.55 (0.30–0.79) |
| Thermo Combo kit | 91.4% (76.9%–98.2%) | 0.91 (0.80–1) | 73.3%‡ (44.9%–92.3%) | 0.79 (0.58–0.99) | 83.3% (58.5%–96.5%) | 0.86 (0.70–1) |
| BGI kit | 94.3% (80.8%–99.3%) | 0.94 (0.85–1) | 80.0% (51.9%–95.7%) | 0.84 (0.67–1) | 83.3% (58.5%–96.5%) | 0.86 (0.70–1) |
| LAMP kit | 91.4% (76.9%–98.2%) | 0.91 (0.80–1) | 66.7%‡ (38.3%–88.2%) | 0.73 (0.51–0.95) | 72.2%‡ (58.5%–96.5%) | 0.86 (0.70–1) |
The specificities (95% CI) of all the assays were 100% (88.4%–100%) for nasopharyngeal swabs, 100% (88.4%–100%) for oropharyngeal swabs, and 100% (87.2%–100%) for sputum samples.
Number of true positive/total samples.
95% CI could not be calculated.
P < 0.05 in comparison with the defined reference standard.
P < 0.05 in comparison with the sensitivity for nasopharyngeal swabs.
Among 39 patients with true-positive results, 15 patients had two samples that were simultaneously collected from different sites. Of these, 14 patients had true-positive results for both samples (nasopharyngeal swabs and sputum samples, 8 patients; nasopharyngeal and oropharyngeal swabs, 4 patients; oropharyngeal swabs and sputum samples, 2 patients). The remaining 1 patient had 1 true-positive oropharyngeal swab and 1 true-negative sputum. Among 69 patients with true-negative results, 12 patients had multiple samples that were simultaneously collected from different sites (nasopharyngeal and oropharyngeal swabs, 11 patients; nasopharyngeal, oropharyngeal swabs, and sputum samples, 1 patient).
Discussion
The current diagnosis of COVID-19 mainly relies on RT-PCR tests.5 Manufacturer-independent evaluation of the molecular assays, including commercial kits that utilize otherwise-extracted RNA templates, was performed. It was found that the specificity was perfect for all the assays and that the CDC N1, CDC N2, NIID N2, and Corman E assays were the most sensitive and highly concordant.6 Genetic variations that may compromise sensitivity of the CDC N1, N2, and Corman E assays have been rarely observed as of week 21 of 2020.7 False negatives by the other assays occurred among low-copy number samples (presenting high CT values by the CDC N2 or NIID N2 assay) (Supplemental Table S1), suggesting a lack of sensitivity of these assays.
The Roche assays were based on Corman's assays1 but had lower sensitivity for their E and N assays. This is likely due to lower CT cutoffs for the Roche assays, rather than differences in reagents and reaction conditions (Table 1 and Supplemental Table S1). Previous studies reported that the N assay was less sensitive than the E assay,8 and the RdRP assay was less sensitive than the Roche E assay.3 The low sensitivity of the Roche RdRP and N assays was concordant with their high LODs (Table 2). The BGI assay exhibited a lower sensitivity than those of the other assays, which displayed similarly low LODs. This may be due to the inclusion of human gene internal controls in the same reaction, which could prevent amplification of viral genes, especially in human genome-enriched samples. The LAMP assay can be used in a resource-poor setting and has the fastest assay time due to its isothermal reaction. However, the Eiken LAMP kit used in this study had a low sensitivity and no control reaction.
Viral loads vary depending on sampling methods and specimen types. One study reported higher viral loads in nasopharyngeal swabs than in oropharyngeal samples,9 and another study employing digital PCR reported viral loads were the highest in sputum followed by oropharyngeal swabs and then nasopharyngeal swabs.10 The lower sensitivities observed for oropharyngeal and sputum samples in this study may not represent different viral loads in specimen types because most of our samples were not paired. It might be related to the facts of higher viral loads (lower CT values) in nasopharyngeal swabs and/or higher loads of human genes (lower CT values of the CDC RNaseP Assay) in sputum samples.
To avoid false negatives due to technical errors such as extraction problems or PCR inhibition, it is recommended to include internal control reactions. The CDC assays were designed to be combined with a separate internal control reaction (Table 1). Differing from multiplex assays that incorporate internal controls such as Roche, Thermo Combo, or BGI kits, this approach needs extra reagents, time, and space in a reaction plate but can be combined with other in-house assays (NIID N2 or Corman E) without any modification. For the multiplex approach, the NIID N2 assay was selected to be multiplexed with the Roche EAV kit, resulting in the similar performance as the original NIID N2 assay.
To date, two published reports have compared performances of multiple RT-PCR assays using clinical samples. Nalla et al11 compared CDC N1/N2/N3 and Corman E/RdRP assays among 10 SARS-CoV-2–positive samples. They reported that the CDC N2 and Corman E assays were the most sensitive. van Kasteren et al12 compared seven commercial kits, including 13 positive and 6 negative samples. When compared with the Corman E assay, the R-Biopharm AG (Darmstadt, Germany) performed the best, followed by BGI, KH Medical (Hanam-si, South Korea), and Seegene (Seoul, South Korea). These reports are in agreement with this study’s findings.
The study limitations included a relatively small sample size of each specimen type and lack of paired samples for all specimen types, clinical information, measurements by multiple investigators, and genomic variation analysis. Strengths included evaluation of multiple molecular assays using the same clinical samples. In addition, the NIID N2 assay was validated with EAV control reaction, and the performance of the Thermo Combo kit and Eiken LAMP kit was determined for the first time.
In conclusion, the CDC EUA kit (N1/N2/RNaseP), NIID N2 with/without EAV, and Corman E assays were found to be the most-sensitive assays. They are feasible as references and clinical diagnostic tests until commercial kits with internal control reactions or fully automated systems that have high diagnostic performances are available without supply shortages in clinical laboratories. Continuous efforts to improve COVID-19 diagnostics are important to control this pandemic.
Acknowledgments
We thank Akihiko Matsuo and Akihiko Hayashi (Kyoto University Hospital, Kyoto, Japan) for their technical assistance.
Footnotes
Supported by the COVID-19 Private Fund (Shinya Yamanaka Laboratory, CiRA, Kyoto University, Japan).
Disclosures: None declared.
Supplemental material for this article can be found at http://doi.org/10.1016/j.jmoldx.2020.11.007.
Author Contributions
Y.M. is the guarantor of this work and, as such, had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Y.M. conceived and designed the study. Y.M., T.S., T.N., S.N., M.Y., and M.N. contributed materials and data collection. Y.M. performed the experiments and analyzed the data. Y.M. wrote the manuscript; and all authors reviewed and approved the final version of the manuscript.
Supplemental Data
Assay results (CT values) for each sample. Positive samples are in bold.
References
- 1.Corman V.M., Landt O., Kaiser M., Molenkamp R., Meijer A., Chu D.K.W., Bleicker T., Brünink S., Schneider J., Schmidt M.L., Mulders D.G.J.C., Haagmans B.L., van der Veer B., van den Brink S., Wijsman L., Goderski G., Romette J.-L., Ellis J., Zambon M., Peiris M., Goossens H., Reusken C., Koopmans M.P.G., Drosten C. Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR. Euro Surveill. 2020;25:2000045. doi: 10.2807/1560-7917.ES.2020.25.3.2000045. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Waggoner J.J., Stittleburg V., Pond R., Saklawi Y., Sahoo M.K., Babiker A., Hussaini L., Kraft C.S., Pinsky B.A., Anderson E.J., Rouphael N. Triplex real-time RT-PCR for severe acute respiratory syndrome coronavirus 2. Emerg Infect Dis. 2020;26:1633–1635. doi: 10.3201/eid2607.201285. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Chan J.F.-W., Yip C.C.-Y., To K.K.-W., Tang T.H.-C., Wong S.C.-Y., Leung K.-H., Fung A.Y.-F., Ng A.C.-K., Zou Z., Tsoi H.-W., Choi G.K.-Y., Tam A.R., Cheng V.C.-C., Chan K.-H., Tsang O.T.-Y., Yuen K.-Y. Improved molecular diagnosis of COVID-19 by the novel, highly sensitive and specific COVID-19-RdRp/Hel real-time reverse transcription-PCR assay validated in vitro and with clinical specimens. J Clin Microbiol. 2020;58:e00310–e00320. doi: 10.1128/JCM.00310-20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Shirato K., Nao N., Katano H., Takayama I., Saito S., Kato F., Katoh H., Sakata M., Nakatsu Y., Mori Y., Kageyama T., Matsuyama S., Takeda M. Development of Genetic diagnostic methods for Novel Coronavirus 2019(nCoV-2019) in Japan. Jpn J Infect Dis. 2020;73:304–307. doi: 10.7883/yoken.JJID.2020.061. [DOI] [PubMed] [Google Scholar]
- 5.Sethuraman N., Jeremiah S.S., Ryo A. Interpreting diagnostic tests for SARS-CoV-2. JAMA. 2020;323:2249–2251. doi: 10.1001/jama.2020.8259. [DOI] [PubMed] [Google Scholar]
- 6.Landis J.R., Koch G.G. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174. [PubMed] [Google Scholar]
- 7.Peñarrubia L., Ruiz M., Porco R., Rao S.N., Juanola-Falgarona M., Manissero D., López-Fontanals M., Pareja J. Multiple assays in a real-time RT-PCR SARS-CoV-2 panel can mitigate the risk of loss of sensitivity by new genomic variants during the COVID-19 outbreak. Int J Infect Dis. 2020;97:225–229. doi: 10.1016/j.ijid.2020.06.027. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Okamaoto K., Shirato K., Nao N., Saito S., Kageyama T., Hasegawa H., Suzuki T., Matsuyama S., Takeda M. An assessment of real-time RT-PCR kits for SARS-CoV-2 detection. Jpn J Infect Dis. 2020;73:366–368. doi: 10.7883/yoken.JJID.2020.108. [DOI] [PubMed] [Google Scholar]
- 9.Zou L., Ruan F., Huang M., Liang L., Huang H., Hong Z., Yu J., Kang M., Song Y., Xia J., Guo Q., Song T., He J., Yen H.-L., Peiris M., Wu J. SARS-CoV-2 viral load in upper respiratory specimens of infected patients. N Engl J Med. 2020;382:1177–1179. doi: 10.1056/NEJMc2001737. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Yu F., Yan L., Wang N., Yang S., Wang L., Tang Y., Gao G., Wang S., Ma C., Xie R., Wang F., Tan C., Zhu L., Guo Y., Zhang F. Quantitative detection and viral load analysis of SARS-CoV-2 in infected patients. Clin Infect Dis. 2020;71:793–798. doi: 10.1093/cid/ciaa345. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Nalla A.K., Casto A.M., Huang M.-L.W., Perchetti G.A., Sampoleo R., Shrestha L., Wei Y., Zhu H., Jerome K.R., Greninger A.L. Comparative performance of SARS-CoV-2 detection assays using seven different primer/probe sets and one assay kit. J Clin Microbiol. 2020;58 doi: 10.1128/JCM.00557-20. e00557-00520. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.van Kasteren P.B., van der Veer B., van den Brink S., Wijsman L., de Jonge J., van den Brandt A., Molenkamp R., Reusken C.B.E.M., Meijer A. Comparison of seven commercial RT-PCR diagnostic kits for COVID-19. J Clin Virol. 2020;128:104412. doi: 10.1016/j.jcv.2020.104412. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Assay results (CT values) for each sample. Positive samples are in bold.