Testing the limits of multiplex respiratory virus assays for SARS-CoV-2 at high cycle threshold values: Comparative performance of cobas 6800/8800 SARS-CoV-2 & Influenza A/B, Xpert Xpress SARS-CoV-2/Flu/RSV, and cobas Liat SARS-CoV-2 & Influenza A/B

Abstract

Background:

Multiplex real-time RT-PCR assays for respiratory pathogens are valuable tools to optimize laboratory workflow and turnaround time. At a time when resurgence of influenza and respiratory syncytial virus (RSV) cases have been widely observed along with continued transmission of SARS-CoV-2, timely identification of all circulating respiratory viruses is crucial. This study evaluates the detection of low viral loads of SARS-CoV-2 by four multiplex molecular assays: Roche cobas 6800/8800 SARS-CoV-2 & Influenza A/B Test, Cepheid Xpert Xpress SARS-CoV-2/Flu/RSV, cobas Liat SARS-CoV-2 & Influenza A/B, and a laboratory-developed test (LDT).

Methods:

Retrospective upper respiratory tract specimens positive for various respiratory viruses at a range of cycle threshold (Ct) values (18–40) were tested by four multiplex assays. Positive and negative percent agreement (PPA and NPA) with validated RT-PCR assays were calculated.

Results:

A total of 82 samples were assessed, with discordant results observed in a portion of the samples (10/82, 12.2%) where Ct values were >33. The majority of the discordant results (6/10, 60%) were false negatives. Overall, PPA was 100% (58/58) for cobas 6800, 97.4% (38/39) for GeneXpert, 100% (17/17) for Liat, and 90.5% (57/63) for the LDT. PPA for the LDT increased to 92.1% after manual review of amplification curves.

Conclusions:

Commercial multiplex respiratory virus assays have good performance for samples with medium to high viral loads (Ct values <33). Laboratories should consider appropriate test result review and confirmation protocols to optimize sensitivity, and may consider reporting samples with additional interpretive comments when low viral loads are detected.

Introduction

Demand for respiratory virus testing was an unprecedented challenge during the COVID-19 pandemic. Testing volumes, reagent use, and demand for laboratory technologist labour has essentially doubled or tripled in most laboratories (1). Early in the pandemic, most existing SARS-CoV-2 assays did not include multiple targets, thus requiring samples to be tested on multiple platforms in order to report additional respiratory virus results. With the recent resurgence of influenza and RSV, along with continued spread of SARS-CoV-2, multiplex molecular assays cam streamline laboratory workflow, enhance testing capacity, and improve overall turnaround time (2–4).

A well-known limitation of multiplex assays is potential loss in analytical sensitivity (5,6), due to the need for primer redesign (primers for all targets must have similar annealing temperatures and product lengths) and potential out-competition for nucleotides in cases of co-infection. To ensure sensitivity for SARS-CoV-2 is not compromised as laboratories transition from singleplex SARS-CoV-2 assays to multiplex respiratory virus assays, this study evaluates the detection of low levels of SARS-CoV-2 by various multiplex molecular assays: Roche cobas 6800/8800 SARS-CoV-2 & Influenza A/B Test (“cobas 6800”), Cepheid Xpert Xpress SARS-CoV-2/Flu/RSV (“GeneXpert”), cobas SARS-CoV-2 & Influenza A/B on the Liat System (“Liat”), and a laboratory-developed test designed for detection of SARS-CoV-2, influenza A, influenza B, and RSV (“LDT” [laboratory-developed test]) (Table 1). Ease of use and turnaround times were also compared from the perspective of a clinical diagnostic laboratory. An understanding of the performance characteristics of multiplex respiratory virus assays is essential for patient care and public health surveillance, and will inform the testing strategies of laboratories which also must consider workflow, staffing resources, use of equipment and reagents, and establishing reporting cut-offs based on cycle threshold (Ct) values.

Table 1:

Key features of four multiplex respiratory virus molecular assays

	cobas 6800/8800 SARS-CoV-2 & Influenza A/B Test	Xpert Xpress SARS-CoV-2/Flu/RSV	cobas SARS-CoV-2 & Influenza A/B on Liat System	Laboratory-developed Test (LDT)
Targets  SARS-CoV-2	Open reading frame (Orf1a), envelope (E) gene	Envelope (E) gene, nucleocapsid (N2) gene using one channel	ORF1 a/b non-structural region, nucleocapsid protein gene	Envelope (E) gene (LightMix ModularDx SARS-CoV E-gene assay [TIB Molbiol])
Influenza A	Matrix 1 and 2 (M1/M2) gene	Matrix, PB2, PA gene	Matrix gene	Matrix gene (8)
Influenza B	Nuclear export protein (NEP), nonstructural protein 1 (NS1) gene	Matrix protein, non-structural protein gene	Non-structural protein gene	Hemagglutinin gene (8)
RSV	Not detected	Nucleocapsid gene	Not detected	Matrix gene (8)
Reported analytical sensitivity  SARS-CoV-2	59 genome copies equivalent/mL (0.008 TCID50/mL)	131 copies/mL	12 copies/mL (0.012 TCID50/mL)	106 copies/mL*
Influenza A	0.02–0.05 TCID50/mL	0.004–0.087 TCID50/mL	0.002–0.02 TCID50/mL	100 copies/mL
Influenza B	0.011–0.027 TCID50/mL	0.04 TCID50/mL	0.002–0.004 TCID50/mL	120 copies/mL
RSV	Not detected	0.22–0.43 TCID50/mL	Not detected	106 copies/mL
Instrument(s) required	cobas 6800 or 8800 (Roche)	GeneXpert Instrument System (can increase number of available modules)	cobas Liat Analyzer	Sample preparation: liquid handling instrument (Hamilton Vantage) Extraction: MagNA Pure 96 System (Roche) Amplification: LightCycler® 480 II (Roche)
Sample volume	600 μL	300 μL	200 μL	500 μL
No. of samples per run	96 (including controls)	1 per module (random access)	1 per instrument	96 (including controls)
Run time on instrument	3 hours	Positive: 25 min Negative: 36 min	20 min	Liquid handler: 15 min MP96: 90 min LC480: 90 min
Comments			No Ct values provided

^* Calculated from probit analysis of commercial reference material (quantified genomic RNA copies of SARS-CoV-2 [TIB Molbiol]; Influenza A, Influenza B, and RSV [Advanced Biotechnologies Inc.])

Materials and Methods

Retrospective upper respiratory tract specimens submitted to a clinical virology laboratory from February 2018 to May 2021 were included. Specimens had been stored at −70 °C and were previously characterized using a validated real-time RT-PCR singleplex assay as the SARS-CoV-2 reference method (LightMix ModularDx SARS-CoV E-gene assay [TIB Molbiol, Germany]). This assay was selected as the reference method because conversion of Ct values from the LightMix assay in our laboratory to quantitative SARS-CoV-2 levels in genomic RNA copies was previously described (7), allowing for a determination of SARS-CoV-2 viral load in each sample. Samples positive for respiratory viruses other than SARS-CoV-2 using pre-existing validated multiplex assays as reference methods for these targets were also included (influenza A, influenza B, or RSV as detected by Xpert Xpress Flu A/B/RSV test [Cepheid, USA]; adenovirus, human metapneumovirus, and parainfluenza 1-3 as detected by a laboratory-developed multiplex assay (8)). A convenience panel was selected to include different respiratory pathogens over a range of Ct values (18–40).

The panel of samples was tested by cobas 6800, GeneXpert, and Liat assays according to manufacturer instructions. For the LDT, 500 μL of sample underwent nucleic acid extraction by MagNA Pure 96 System (Roche) into 50 μL eluate, followed by RT-PCR on LightCycler 480 II (Roche) using the primers and conditions previously described (Table 1) (8). Medical laboratory technologists were surveyed to describe overall ease of use and hands-on processing time required for each multiplex assay.

Results

A total of 82 specimens were assessed, including 28 positive for SARS-CoV-2, 15 influenza A, 15 influenza B, 5 RSV, and 19 negative samples (which were positive for other respiratory pathogens including adenovirus, human metapneumovirus, and parainfluenza 1–3). Complete agreement between all multiplex assays was observed for 78.6% (22/28) of SARS-CoV-2 positive samples. Discordant results were detected for a portion of the SARS-CoV-2 samples (5/28, 17.9%), exclusively with Ct values >33, corresponding to a low SARS-CoV-2 viral load of <1.675 log₁₀ E gene RNA copies equivalent (Figure 1). The four assays demonstrated good general agreement for influenza A/B and RSV samples at the range of Ct values selected for this study (Figure 2).

Figure 1:

The performance of each multiplex assay when detecting SARS-CoV-2 RNA from upper respiratory tract samples. Black bullets indicate SARS-CoV-2 RNA was detected by the respective multiplex assay; grey bullets indicate SARS-CoV-2 RNA was not detected.

SARS-CoV-2 viral load in E gene RNA copies equivalent is calculated from the reference method (LightMix ModularDx SARS-CoV E-gene assay).

Figure 2:

The performance of each multiplex assay when detecting Influenza A, Influenza B, or RSV nucleic acid from upper respiratory tract samples.

Black bullets indicate the viral target was detected by the respective multiplex assay; grey bullets indicate the viral target was not detected. The cycle threshold (Ct) values displayed in this figure are from the selected reference method (Xpert Xpress Flu A/B/RSV test). Circles represent Influenza A; triangles represent Influenza B; squares represent RSV.

Cobas 6800 and Liat assays demonstrated the highest overall PPA to the reference method, for both SARS-CoV-2 and other respiratory virus targets (Table 2). GeneXpert and LDT were found to have somewhat decreased PPA for SARS-CoV-2 samples with Ct values ≥33. In contrast, the GeneXpert demonstrated the highest NPA, with all other multiplex assays experiencing at least one false positive result during this study (Table 3).

Table 2:

Positive percent agreement (PPA) and negative percent agreement (NPA) for four multiplex respiratory virus assays compared to the reference method

	PPA
	cobas 6800	GeneXpert	Liat	LDT
SARS-CoV-2 Ct value <33	100%, 22/22	100%, 9/9	100%, 2/2	95.5%, 21/22*
SARS-CoV-2 Ct value 33–40	100%, 6/6	80%, 4/5	100%, 5/5	33.3%, 2/6
All SARS-CoV-2	100%, 28/28	92.9%, 13/14	100%, 7/7	82.1%, 23/28
Influenza A	100%, 15/15	100%, 10/10	100%, 5/5	93.3%, 14/15
Influenza B	100%, 15/15	100%, 10/10	100%, 5/5	100%, 15/15
RSV	n/a	100%, 5/5	n/a	100%, 5/5
All Targets	100%, 58/58	97.4%, 38/39	100%, 17/17	90.5%, 57/63

	NPA
	cobas 6800	GeneXpert	Liat	LDT
All	95.8%, 23/24	100%, 16/16	92.9%, 13/14	81.8%, 9/11

^* Visual inspection revealed subtle amplification at Ct >35 for one sample, but was not detected by instrument software. This increases the LDT PPA to 100% (22/22) for SARS-CoV-2 samples with Ct values <33, or 92.1% (58/63) overall.

LDT = Laboratory-developed test

Table 3:

All discordant results with cycle threshold (Ct) values among the multiplex assays in this study

SARS-CoV-2
Reference test		cobas 6800		GeneXpert	LDT	Liat
E	Viral Load (log10 RNA copies/mL)	Orf1a	E	E & N2	E	Orf1 ab & N
32.94	1.693	34.76	33.98	35.2	NEG *	-
34.67	1.167	36.66	36.31	-	NEG	-
35.79	0.827	36.94	35.18	39.2	NEG	POS
36.27	0.681	37.44	35.48	NEG † , ‡	NEG ‡	POS
36.85	0.505	35.5	34.86	38.2	NEG	POS
NEG	NEG	40.79 §	NEG	NEG	NEG	-
NEG	NEG	NEG	NEG	-	36.89 ¶	-
NEG	NEG	NEG	NEG	NEG	-	POS
19.81	5.682	20.49	20.14	-	21.80 (plus Flu A detected with Ct 40.46)**	-

Influenza A
Reference Test		cobas 6800	GeneXpert		LDT	Liat
Flu A1	Flu A2	Flu A	Flu A1	Flu A2	Flu A	Flu A
33.1	33.3	31.62	29.9	31.2	NEG	POS

^* Visual inspection revealed amplification at Ct >35, but not detected by instrument

^† SARS-CoV-2 endpoint reported by GeneXpert was 1

^‡ Sample collected from patient 1 day earlier was positive for SARS-CoV-2 by reference method (Ct 32.44); likely false negative by GeneXpert and LDT

^§ Sample collected December 2018, prior to COVID-19 pandemic and was positive for parainfluenza 2 (Ct 25.3); likely false positive SARS-CoV-2 by cobas

^¶ Sample collected February 2020 and was positive for human metapneumovirus (Ct 29.21); likely false positive SARS-CoV-2 by LDT

^** Sample was highly positive for SARS-CoV-2 and no Influenza A was detected by reference method nor cobas; likely false positive Influenza A by LDT

LDT = Laboratory-developed test

Based on a survey of 10 medical laboratory technologists, the average hands-on processing time for each multiplex assay was: 15 minutes per 96 samples on cobas 6800, 1.6 minutes per sample on GeneXpert, 2 minutes per sample on Liat, and 20 minutes per 96 samples on LDT.

Discussion

There was complete positive concordance for all targets between the cobas 6800, GeneXpert, Liat, and LDT for samples with Ct values <33. Of the seven occurrences of a false negative result by any one assay, one (14.3%) would have undergone further investigation in our laboratory due to the visual appearance of a late amplification curve with the LDT. This increased the PPA from 90.5% to 92.1% for the LDT. Additionally, our laboratory has previously described the importance of reviewing endpoint values on GeneXpert before reporting respiratory virus results (9,10). Although the one false negative SARS-CoV-2 result by GeneXpert in this study was not observed to have an elevated endpoint, these findings emphasize the importance of manual result review by trained laboratory personnel to optimize RT-PCR diagnostic accuracy.

The remainder of the false negative samples had Ct values for SARS-CoV-2 in the range of 35–38, correlating to a SARS-CoV-2 viral load of only 1.067 log₁₀ E gene RNA copies equivalent or less. Samples with very low SARS-CoV-2 viral loads should be interpreted with appropriate clinical context, and may represent remnant viral RNA from remote infection, laboratory contamination/false positive, or more rarely, acute infection in the earliest phase of viral replication (11). The detection of minute amounts of SARS-CoV-2 viral RNA has potential for misinterpretation and unintended harm such as delayed medical procedures, exclusion from the workplace or school, or excessive repeat testing (12). Therefore, the clinical relevance of samples with very low SARS-CoV-2 viral loads are questionable, and the ability of molecular assays in clinical laboratories to detect these samples may be of less importance.

The reproducibility of results in this study for samples with Ct values <33, and correlation of high SARS-CoV-2 viral loads with disease severity (13,14), may help guide laboratories’ thresholds for reporting and interpretive comments. Samples with high Ct values may be reported with additional interpretive comments such as “Low positive” or “Indeterminate,” which can prompt clinicians to further investigate and correlate with the patient’s history and clinical status. This is not possible to implement with assays that do not provide Ct values or other quantitative analytical information (eg, Liat), and laboratories may need to consider additional testing on alternate RT-PCR assays for any positive samples in order to obtain Ct values and further characterize the result.

Of the four false positive results in this study, Ct values ranged from 36 to 40 for SARS-CoV-2 or influenza A. The false positive results may be due to possible cross-contamination during specimen handling or inherent issues in assay design/specificity, but again challenge the clinical and public health relevance of samples with high Ct values.

Limitations of this study include potential bias of the convenience panel, selected primarily from our laboratory’s pre-existing validated assays, which may have favoured performance of the LDT or GeneXpert. The findings of our study highlighted cobas 6800 and Liat as the tests with the greatest positive percent agreement, suggesting that sample bias, if any, was minimal. The false negative tests observed in this study may result from sampling bias inherent in samples with low viral loads, assay design/sensitivity issues, or degradation of target RNA after prolonged specimen storage; however, accessing retrospective, previously-characterized specimens was necessary in order to include a diverse set of respiratory pathogens representative of infections among our patient population over a range of Ct values. The SARS-CoV-2 viral load estimates in this study have diminished accuracy at high Ct values which fall below the lower limit of quantification (7) and are subject to variability due to the nature of the specimen type and varying collection techniques/quality. Sample size in this study was small due to the limited availability of test cartridges for each multiplex assay during the pandemic.

Adequate sample volume from retrospective specimens was also a significant challenge; in some cases, samples were not able to be tested by all four assays. Diluting the specimens to increase sample volume was not an option in this study, as the aim was to preserve the Ct value in order to evaluate the multiplex assays’ ability to detect the same sample with the same viral load. To address this limitation, we ensured each multiplex respiratory virus assay was evaluated using samples with a similar range of Ct values. We do acknowledge the PPA and NPA calculations in this study will be impacted by this method and therefore can only be used as rough estimates, particularly for the Liat which had the smallest sample size available in this study; a single false positive result decreased the Liat’s NPA more than other commercial assays. Furthermore, each multiplex assay had a variable number of targets (three for cobas 6800 and Liat, four for GeneXpert and LDT), and those samples positive for RSV could not be included in the PPA calculations for the assays lacking this target (cobas 6800 and Liat). Interestingly, our PPA findings (where cobas and Liat demonstrated the highest PPA for SARS-CoV-2) are consistent with the reported analytical sensitivities by the manufacturers of each assay (Table 1), which does add validity to our study’s findings.

Cobas 6800 and LDT are better suited for high-volume batched testing, whereas GeneXpert and Liat are advantageous for random access, urgent clinical testing. Clinical laboratories may have a need for both types of methods, and should choose to implement new assays in anticipation of a challenging upcoming respiratory virus season after careful consideration of analytical performance, relevance of Ct values, and reporting comments.

Funding Statement

A portion of the reagents for this study were provided in kind by Cepheid and Roche Canada. Cepheid and Roche were not involved in the study design nor with data analysis.

Contributors:

Conceptualization, MG Romney, N Matic, CF Lowe; Funding/reagents in-kind acquisition, MG Romney, N Matic; Investigation, T Lawson, G Ritchie; Formal Analysis, N Matic; Visualization, N Matic; Writing – Original Draft, N Matic; Writing – Review & Editing, CF Lowe, MG Romney.

Ethics Approval:

This study was approved by the University of British Columbia University-Industry Liaison Office, and was deemed exempt from full research ethics review by the Providence Health Care Research Ethics Board.

Informed Consent:

N/A

Registry and the Registration No. of the Study/Trial:

N/A

Funding:

A portion of the reagents for this study were provided in kind by Cepheid and Roche Canada. Cepheid and Roche were not involved in the study design nor with data analysis.

Disclosures:

NM reports honoraria related to speaker engagement outside the submitted work for Roche Molecular Systems, Inc. MGR reports travel and financial support for participating in a Roche Global Medical Advisory Board and meetings sponsored by HOLOGIC. All other authors report no relevant conflicts of interest.

Peer Review:

This manuscript has been peer reviewed.

Animal Studies:

N/A