Comparison of a laboratory-developed test targeting the envelope gene with three nucleic acid amplification tests for detection of SARS-CoV-2

Highlights

• •Four nucleic acid amplification tests for SARS-CoV-2 RNA demonstrated comparable performance using clinical specimens.
• •A limited number of discrepancies were observed in specimens with low viral loads.
• •The isothermal amplification assay was slightly less sensitive but was one hour faster than the other methods.
• •Assay selection requires consideration of test performance, instrument/reagent availability, turnaround and throughput.

Abstract

Background

Numerous nucleic acid amplification tests, including real-time, reverse transcription PCR (rRT-PCR) and isothermal amplification methods, have been developed to detect SARS-CoV-2 RNA, including many that have received emergency use authorization (EUA). There is a need to assess their test performance relative to one another.

Objectives

The aim of this study was to compare the test performance of a high complexity laboratory-developed rRT-PCR EUA from Stanford Health Care (SHC) targeting the SARS-CoV-2 envelope (E) gene with other tests: the Atila isothermal amplification assay targeting the nucleocapsid (N) gene and open reading frame 1ab (ORF1ab), the Altona E and spike (S) multiplex, real-time RT-PCR, and the US Centers for Disease Control and Prevention (CDC) N1 and N2 rRT-PCRs.

Study Design

A diagnostic comparison study was performed by testing nasopharyngeal samples from persons under investigation for coronavirus disease 2019 (COVID-19). Assay performance was assessed by percent agreement and Cohen’s kappa coefficient.

Results

Positive percent agreement with the SHC EUA reference assay was 82.8 % (95 % confidence interval (CI) 65.0 to 92.9) for Atila, 86.7 % (95 % CI 69.7 to 95.3) for the Altona E and S targets, and 86.7 % (95 % CI 69.7 to 95.3) and 90.0 % (95 % CI 73.6 to 97.3), for the CDC N1 and N2 targets, respectively. All assays demonstrated 100 % negative percent agreement. Kappa coefficients ranged from 0.86 to 0.92, indicating excellent agreement.

Conclusions

Performance was comparable among the SARS-CoV-2 nucleic acid amplification methods tested, with a limited number of discrepancies observed in specimens with low viral loads.

1. Background

Accurate diagnostics for the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) are critical to identifying and managing individuals with Coronavirus Disease-2019 (COVID-19). The current standard of care for diagnosis of active infection is the detection of viral RNA from respiratory specimens by real-time, reverse transcription polymerase chain reaction (rRT-PCR). [[1], [2], [3], [4]] There are, however, a limited number of isothermal amplification methods that have been developed for the detection of SARS-CoV-2 RNA. One such isothermal assay is the Atila iAMP COVID-19 detection kit, which targets the open reading frame 1ab (ORF1ab) region and nucleocapsid (N) gene in a single channel and utilizes the detection RNase P in a separate channel as the internal control. To evaluate this isothermal method, as well as two additional rRT-PCR assays, the N1 and N2 components of the US Centers for Disease Control and Prevention (CDC) assay and a commercial multiplex assay targeting the envelope (E) and spike (S) genes (Altona Diagnostics), we tested nasopharyngeal samples from individuals under investigation for COVID-19 using the Stanford Health Care (SHC) Clinical Virology Laboratory EUA assay as reference. Given the need to rapidly deploy SARS-CoV-2 diagnostic tests in response to the COVID-19 pandemic, these data are important to inform laboratory decision-making and to guide clinical management.

2. Objectives

The aim of this study was to assess the test performance of the Atila, Altona, and CDC assays compared to the SHC EUA for the qualitative detection of SARS-CoV-2.

3. Study Design

3.1. Ethics statement

This study was approved by the Stanford University Institutional Review Board (protocol #48973).

3.2. Clinical specimens and reference testing

Eighty nasopharyngeal swab samples in viral transport medium (VTM) collected between March 18 and March 28, 2020, from patients under investigation for SARS-CoV-2 submitted to the Stanford Health Care Clinical Virology Laboratory for diagnostic testing, were included in this study. The SHC test was performed as described in the EUA documentation. [4,5] Briefly, Total nucleic acids were extracted from 500 μL VTM on the QIAsymphony SP using the QIAsymphony DSP Virus/Pathogen Midi Kit (both from Qiagen, Germantown, MD) according to the manufacturer’s recommendations, and eluted in 60 μL buffer AVE. SARS-CoV-2 RNA was detected using previously described primer and probe sequences targeting the E gene. [1] These were combined in multiplex with RNase P primers and probe. Real-time RT-PCR was performed using the SuperScript III One-Step RT-PCR System with Platinum Taq DNA Polymerase Kit (Invitrogen, Carlsbad, CA) on the Rotor-Gene Q instrument (Qiagen).

3.3. Comparator nucleic acid amplification tests

The Atila iAMP COVID-19 Detection Kit (Atila BioSystems, Mountain View, CA) is an internally-controlled, multiplex, real-time reverse transcriptase-isothermal amplification assay targeting the ORF1ab region and N gene in a single channel. Though the Atila iAMP assay received EUA, the reagents used in this study were obtained prior to authorization and were for research use only (RUO). Per the manufacturer’s instructions, 18 μL eluate was mixed with 2 μL of Sample Buffer C. Then, 15 μL of this mixture was added to 10 μL of reaction master mix, containing 5 μL of Primer Mix and 5 μL of Buffer Mix, for a total reaction volume of 25 μL. Isothermal amplification was performed on the CFX96 Deep Well Real-Time PCR Detection System (Bio-Rad Laboratories). The reaction was carried out for 30 s at 61 °C, and then fifty 1 min intervals at 61 °C with signal capture. Thresholds were set at 1000 for FAM (ORF1ab and/or N) and 200 for HEX (Internal Control, RNase P). Sample was considered positive if the FAM cycle threshold (Ct) was <45. SARS-CoV-2 negative samples with no Ct in the HEX channel were considered invalid.

The RealStar SARS-CoV-2 RT-PCR Kit 1.0 (Altona Diagnostics, Hamburg, Germany) is an internally-controlled multiplex, rRT-PCR that targets the lineage B betacoronavirus E gene, as well as the SARS-CoV-2-specific S gene. The Altona assay has also received EUA, though the reagents used here were obtained prior to authorization and were RUO. Per the manufacturer’s instructions, the RT-PCR master mix was composed of 5 μL of Master A, 15 μL of Master B and 1 μL of internal control, per reaction. Ten μL of eluate was added to 20 μL of master mix for a final volume of 30 μL. RT-PCR was performed on the CFX96 Deep Well Real-Time PCR Detection System (Bio-Rad Laboratories, Hercules, CA). Cycling conditions were as follows: hold for 20 min. at 55 °C, hold for 2 min. at 95 °C, then 45 cycles of 95 °C for 15 s, 55 °C for 45 s, and 72 °C for 15 s. Baseline setting was set to “Baseline Subtracted Curve Fit” and thresholds were set at 1000 for FAM (E), 1000 for Cy5 (S), and 200 for JOE (IC). SARS-CoV-2 E and S negative samples with no Ct in the JOE channel were considered invalid.

The CDC 2019-Novel Coronavirus (2019-nCoV) Real-Time RT-PCR Diagnostic Panel is composed of single-target, real-time RT-PCR assays targeting two regions of the N gene (N1 and N2) [6]. Primers were purchased from Integrated DNA Technologies (San Diego, CA) and hydrolysis probes were purchased from Biosearch Technologies (Petaluma, CA). Real-time RT-PCR was performed using the SuperScript III One-Step RT-PCR System with Platinum Taq DNA Polymerase Kit (Invitrogen) on the Rotor-Gene Q instrument (Qiagen). Each 26 μL reaction contained 10 μL of eluate, 12.5 μL of 2X Reaction Mix, 0.5 μL enzyme, 1 μL of 50 mM MgSO₄, and 2 μL of primer/probe mix. Cycling conditions were as follows: 52 °C for 15 min., 94 °C for 2 min., then 45 cycles of 94 °C for 15 s, 55 °C for 45 s, and 68 °C for 20 s. Detection was in the green channel (for both N1 and N2) with the threshold set at 0.05. Any sample with Ct<45 was considered positive.

3.4. Discrepancy analysis

Because reference testing was performed on freshly extracted nucleic acids and comparator testing was performed after at least one freeze-thaw of archived eluates, all discrepant samples (reference detected, comparator not detected) were re-extracted and those eluates tested without freeze-thaw. Total nucleic acids were extracted from 400 μL on the BioRobot EZ1 (Qiagen, Germantown, MD) using the EZ1 virus mini kit 2.0 according to the manufacturer’s recommendations, and eluted in 60 μL buffer AVE.

3.5. Statistical analysis

Positive percent agreement (PPA), negative percent agreement (NPA) and associated 95% confidence intervals (CI) were performed with the SHC EUA serving as the reference method. Cohen’s kappa coefficient of qualitative results (detected/non-detected) between the two assays with 95% CI was also calculated. Cohen’s kappa values greater than 0.81 were interpreted to indicate excellent agreement [7].

4. Results

To compare the performance of SARS-CoV-2 nucleic acid amplification tests, extracts from 80 nasopharyngeal swab specimens (30 detected, 50 not detected) originally tested using the SHC EUA were evaluated using the Atila iAMP COVID-19 Detection Kit, Altona RealStar SARS-CoV-2 RT-PCR Kit 1.0, and the N1 and N2 components of the CDC 2019-Novel Coronavirus (2019-nCoV) Real-Time RT-PCR Diagnostic Panel (Table 1 ). The reference assay positives included samples with virus loads that spanned clinically observed cycle threshold (Ct) values (median: 26.8 cycles, range: 17.7–38.7, interquartile range: 7.8).

Table 1.

Nucleic Acid Amplification Tests Evaluated in this Study.

Name	Gene Target(s)	Internal Control	Method
SHC EUA	E	RNase P	rRT-PCR
Atila	N, ORF1ab	RNase P	iAMP
Altona RealStar	E, S	Heterologous	rRT-PCR
CDC	N1	NA	rRT-PCR
	N2	NA	rRT-PCR

SHC EUA, Stanford Health Care Clinical Virology Laboratory Emergency Use Authorization assay; CDC, Centers for Disease Control and Prevention; E, envelope gene; N, nucleocapsid gene; ORF1ab, open reading frame 1ab region; S, spike gene; rRT-PCR, real-time, reverse transcription-PCR; iAMP, isothermal amplification.

The positive percent agreement of Atila with the reference assay was 82.8% (95% confidence interval (CI) 65.0 to 92.9) (Table 2 ). The kappa coefficient was 0.86 (95% CI 0.74 to 0.98), indicating excellent agreement. Five reference positive samples [36.8 (01), 38.1 (02), 36.3 (06), 37.2 (10), 38.7 (12); Ct (sample number)] were not detected by Atila. When the original samples were re-extracted and retested, two were detected [38.1 (02) and 36.3 (06)], while the three other samples remained negative. One sample was excluded from the analysis; the Atila internal control failed on both the original and re-extracted nucleic acids.

Table 2.

Comparison of the Stanford Health Care Assay with other Nucleic Acid Amplification Tests for detection of SARS-CoV-2 RNA in clinical samples.

		Atila iAmp N/ORF1ab
		Detected	Not Detected	Total
SHC EUA	Detected	24	5	29
	Not Detected	0	50	50
	Total	24	55	79

		Altona E gene
		Detected	Not Detected	Total
SHC EUA	Detected	26	4	30
	Not Detected	0	50	50
	Total	26	54	80

		Altona S gene
		Detected	Not Detected	Total
SHC EUA	Detected	26	4	30
	Not Detected	0	50	50
	Total	26	54	80

		CDC N1
		Detected	Not Detected	Total
SHC EUA	Detected	26	4	30
	Not Detected	0	50	50
	Total	26	54	80

		CDC N2
		Detected	Not Detected	Total
SHC EUA	Detected	27	3	30
	Not Detected	0	50	50
	Total	27	53	80

The positive percent agreements of the Altona E and S targets with the reference assay were both 86.7% (95% CI 69.7–95.3) (Table 2). The kappa coefficient was 0.89 (95 % CI 0.79 to 0.99), indicating excellent agreement. Four reference positive samples [36.8 (01), 38.1 (02), 36.3 (06), 38.7 (12)], overlapping those not detected by Atila, were also not detected by the Altona assay. Upon re-extraction and re-testing, two of these samples were detected by E and S [38.1 (02) and 38.7 (12)], one was detected by E only [36.8 (01)] and one was not detected by either [36.3 (06)].

Finally, the positive percent agreements of the CDC N1 and N2 targets with the reference assay were 86.7 % (95% CI 69.7–95.3) and 90.0 % (95 % CI 73.6–97.3), respectively (Table 2). The kappa coefficients were 0.89 (95 % CI 0.79 to 0.99) and 0.92 (95 % CI 0.83–1.0) indicating excellent agreement. Two reference positive samples [36.8 (01) and 38.7 (12)] were not detected by either N1 or N2, whereas two reference positive samples were detected only by N2 [36.3 (06), 37.2 (10)], and one reference positive sample was detected only by N1 [38.1 (02)]. After re-extraction and re-testing, all three N2 and three of four N1 discrepant samples were detected. Only one sample [38.7 (12)] was detected by N2 but not N1.

Negative percent agreement was 100 % (9 5% CI 91.5–100.0) for all comparisons.

5. Discussion

The SARS-CoV-2 nucleic acid amplification assays included in this analysis demonstrated similar performance when used to test nasopharyngeal swab specimens collected from persons under investigation for COVID-19 in Northern California. Discrepant results occurred exclusively in samples with low viral loads (Ct >35). Re-extraction from the original sample and re-testing of an eluate that had not undergone a freeze-thaw cycle resulted in improved SARS-CoV-2 detection, though a number of samples remained undetectable with these assays, even after re-extraction.

These results highlight subtle differences in sensitivity, the relative importance of which may vary based on the patient population tested and how frequently low viral load specimens are expected. Slightly reduced sensitivity was observed in the Atila iAMP assay; five reference positive samples were not detected in the initial experiments and three were still not detected after re-extraction. Isothermal reverse transcription amplification methods are typically at least as analytically sensitive as rRT-PCR, so the explanation for this difference is not readily apparent. Furthermore, Atila requires the highest nucleic acid eluate input volume (18 μL) of these tests and utilizes common SARS-CoV-2 gene targets, though the details of the oligos used in the assay are not disclosed. However, this isothermal method yields results approximately 1 h more rapidly than the rRT-PCR thermal cycling methods evaluated in this study. The decision to choose one assay over another in this setting should assess the balance between advantages in turnaround time and instrument or reagent availability, and the drawbacks of potentially reduced sensitivity. These include consideration of the underlying patient population being tested, as well as the possibly significant clinical and public health implications of missed COVID-19 cases.

This study provides valuable data using clinical specimens. However, this study is limited by its modest sample size and enrichment for positive samples. Additional studies performed in a prospective manner, with a larger sample size, are needed to evaluate existing tests, as well as those in currently under development.

In conclusion, these SARS-CoV-2 nucleic acid amplification tests demonstrated comparable performance using nasopharyngeal swab specimens from patients under investigation for COVID-19.

Funding

None.

CRediT authorship contribution statement

Philip L. Bulterys: Conceptualization, Methodology, Writing - original draft, Writing - review & editing. Natasha Garamani: Methodology, Investigation, Writing - review & editing. Bryan Stevens: Investigation. Malaya K. Sahoo: Investigation. ChunHong Huang: Investigation. Catherine A. Hogan: Writing - review & editing. James Zehnder: Writing - review & editing. Benjamin A. Pinsky: Conceptualization, Methodology, Supervision, Writing - original draft, Writing - review & editing.

Declaration of Competing Interest

The authors declare no conflicts of interest.