Parallel Validation of Three Molecular Devices for Simultaneous Detection and Identification of Influenza A and B and Respiratory Syncytial Viruses

ABSTRACT

Rapid identification of respiratory pathogens, such as influenza virus A (FluA), influenza virus B (FluB), and respiratory syncytial virus (RSV), reduces unnecessary antimicrobial use and enhances infection control practice. We performed a comparative evaluation of three molecular methods: (i) the Aries Flu A/B & RSV, (ii) the Xpert Xpress Flu/RSV, and (iii) the Cobas Flu A/B & RSV assays. The clinical performances of the three methods were evaluated using 200 remnant nasopharyngeal swab (NPS) specimens against a combined reference standard. The limits of detection (LODs) were determined using FluA, FluB, and RSV control strains with known titers. The 95% LODs were between 1.702 and 0.0003 50% tissue culture infective dose (TCID₅₀), with no significant differences revealed among the three assays. Perfect qualitative detection agreement was obtained in the reproducibility study. The Cobas assay failed at the first run on 13 clinical specimens, resulting in an invalid rate of 6.5%. The sensitivities and specificities for all assays were 96.0 to 100.0% and 99.3 to 100% for all three viruses. For on-demand single-specimen and batched 12-specimen workflows, the test turnaround times were 115.5 and 128.8 min for the Aries assay (12 sample capacity), 34.2 and 44.2 min for the Xpress assay (16 sample capacity), and 21.0 and 254.4 min for the Cobas assay (one instrument), respectively. In summary, the Aries, Xpress, and Cobas Liat assays demonstrated excellent sensitivities and specificities for simultaneous detection and identification of FluA, FluB, and RSV from NPS specimens in cancer patients. Test turnaround time was significantly shorter on the Xpress when instrument scalability is unlimited.

INTRODUCTION

Respiratory viruses continue to be recognized as being among the most common causes of morbidity and mortality in the United States (1). Specifically, influenza virus A (FluA), influenza virus B (FluB), and respiratory syncytial virus (RSV) are among the leading causes of respiratory illness during the respiratory virus season, especially in children, elderly persons, and immunocompromised patients (2, 3). In addition, FluA/B and RSV are the dominant respiratory viral pathogens responsible for nosocomial transmission in the health care setting (3, 4). An accurate assessment of respiratory disease etiology allows for prompt patient care, including efficient antiviral treatment and effective patient cohorting and isolation, which, in turn, reduce the use of antibiotics and lower the cost burden (3, 4). Traditional laboratory tests for respiratory viruses, such as rapid antigen detection tests by enzyme immunoassays or direct florescence assays, suffer from insufficient test sensitivity (5, 6). Molecular diagnostic tests have gradually become the dominant test methods used in the laboratory for the detection and identification of respiratory viral pathogens. Rapid turnaround time, simultaneous detection of an array of pathogens, and superior sensitivity of the monoplex and multiplex molecular assays have the potential to decrease emergency department length of stay, reduce diagnostic testing, and shorten the duration of intravenous antibiotic administration (7–10).

There are several commercially available real-time reverse transcriptase PCR (RT-PCR) assays currently approved by the Food and Drug Administration (FDA) for the detection of single or multiple respiratory pathogens (11). They vary in test format (e.g., random access or batched), throughput, and numbers of pathogens covered (7–10, 12–19). Among them, the Aries Flu A/B & RSV (Luminex Corp., Austin, TX) (15), Xpert Xpress Flu/RSV (Cepheid, Sunnyvale, CA) (13), and Cobas Flu A/B & RSV (Roche Molecular Systems, Pleasanton, CA) (14) assays are integrated nucleic acid extraction-independent devices that have recently received FDA clearance for simultaneous detection and identification of FluA, FluB, and RSV in nasopharyngeal swabs. This study was conducted to evaluate in parallel the performances of the three assays for the detection and identification of FluA, FluB, and RSV in nasopharyngeal swab (NPS) specimens collected from cancer patients. In addition to accuracy and precision performance data, we also contrasted test turnaround time (TAT) and hands-on time (HOT) data for runs on individual and batched specimens.

MATERIALS AND METHODS

Clinical specimens and control strains.

A retrospective study was conducted on leftover NPS specimens collected from cancer patients during the 2015-2016 winter season submitted by clinicians for testing by the FilmArray respiratory panel (RP; BioFire, Salt Lake City, UT) testing. The first 50 positive specimens for FluA, FluB, and RSV and the first 50 Flu- and RSV-negative specimens were selected and included in the study. Each specimen was aliquoted into four vials and stored at −80°C until further testing. Three additional NPS specimens that were positive for FluA, FluB, or RSV were collected, divided into 9 aliquots of each, and stored at −80°C for reproducibility testing. FluA(H1), FluA(H1N1), FluA(H3), FluB, RSV-A, and RSV-B control strains (ZeptoMetrix, Buffalo, NY) were first diluted into viral transport medium (VTM). Each 10-fold-diluted viral strain was aliquoted into four vials and stored at −80°C for limit of detection (LOD) determination. An application for an exemption of collection or study of existing data was approved by the Memorial Sloan Kettering Cancer Center (MSKCC) institutional review board.

Luminex Aries Flu A/B & RSV assay.

The Luminex Aries Flu A/B & RSV assay is a “sample to answer” automated molecular test based on the MultiCode PCR technology with an extraction cartridge and target-specific fluorescent-labeled primers in a small attached master mix tube (Ready Mix) (15). A single cassette is used for a 200-μl NPS sample. Automated workflow includes nucleic acid extraction, multiplex MultiCode reverse transcriptase PCR (RT-PCR), and melting curve analysis for the detection of FluA, FluB, and RSV with a sample processing control (SPC) to monitor the reaction. When run on the instrument with two independent modules, the throughput ranged from 1 to 12 specimens, and the results were generated in 2 h (15, 19).

Cepheid Xpert Xpress Flu/RSV assay.

The Xpress assay is an optimized molecular test with integrated workflow, including optimized extraction, amplification, and detection, that take place in a disposable cartridge containing an SPC and a probe check control (PCC) (13). This assay is capable of detecting and differentiating FluA, FluB, and RSV from NPS simultaneously. Using a transfer pipette, approximately 300 μl of NPS sample was transferred into a sample chamber of the disposable cartridge. When run on the Cepheid GeneXpert XVI, the throughput ranged from 1 to 16 specimens, and the results were available within 32 min (13).

Roche Cobas Liat Flu/RSV assay.

The Roche Cobas Liat Flu/RSV assay is a fully automated multiplex real-time RT-PCR which performs sample purification, nucleic acid extraction and amplification, and detection and discrimination of FluA, FluB, and RSV in NPS specimens from patients with suspected respiratory tract infection (14). This is a single assay, and a 200-μl VTM specimen was added into a sealed tube containing SPC; all the steps are carried out in a tube. This benchtop analyzer is designed for point-of-care testing at bedside (14). When two units of the instrument were used, the throughput ranged from 1 to 2 specimens, and the results were ready in 20 min.

BioFire FilmArray RP assay.

The FilmArray RP is an automated nested multiplexed PCR assay that integrates sample preparation, nucleic acid amplification, detection, and analysis into a pouch, including two internal controls, which is designed for simultaneous qualitative detection and identification of 20 respiratory pathogens (17 viruses and 3 bacterial), including FluA(H1), FluA(H3), FluA(H1-2009), FluB, and RSV, from a single sample in about 60 min. One milliliter of supplied hydration solution was added to the pouch to rehydrate the reagents, and then 300 μl of the mixture was transferred (300 μl of NSP sample mixed with 500 μl of sample buffer) into the pouch by the sample-loading syringe. The pouch was placed in the FilmArray instrument, where two-step PCR was performed (12, 16, 18).

Data analysis.

The result (positive or negative) obtained by a majority of the four molecular devices (Aries, Xpress, Cobas, and FilmArray) was considered the reference result. Specimens with equally distributed results (two positives and two negatives) were sent to the Mayo Medical Laboratories and further tested using the Focus Diagnostics Simplexa Flu A/B & RSV assay (17) as the resolution method. Limits of detection (LODs) and reproducibility were determined by running selected specimens in triplicate. Hands-on-time (HOT) and test turnaround time (TAT) were determined by running clinical specimens in quintuplicate. The 95% confidence intervals of the sensitivities and specificities were calculated using the Wilson method, and a comparison of the two tests was conducted using a Kappa coefficient (http://epitools.ausvet.com.au). P values were calculated, and values of ≤0.05 were considered statistically significant.

RESULTS

A total of 3,436 NPS specimens were tested on the FilmArray system from 27 January 2016 to 20 April 2016 at the MSKCC, in which 242 (7.0%), 79 (2.3%), and 185 (5.4%) specimens were positive for FluA, FluB, and RSV, respectively. Among them, 251 (49.6%) specimens were from males, and 17 (3.3%), 59 (11.7%), and 430 (85.0%) specimens were from patients <4, 4 to 18, and >18 years old, respectively. A total of 200 NPS specimens, including the first 50 positives for FluA, FluB, and RSV (n = 150) and the first 50 negatives for FluA/FluB/RSV (n = 50), as determined by the FilmArray assay, were selected for the study. The Aries, the Xpress, and the Cobas assays detected 49, 50, and 50 FluA, 45, 44, and 46 FluB, and 48, 49, and 50 RSV cases, respectively (Table 1). Thirteen (6.5%) specimens yielded invalid results on the Cobas assay, and six (3.0%) specimens remained invalid after repeating. These six repeated invalid results were excluded from sensitivity and specificity analysis. No invalid results were observed in either the Aries or Xpress assay. In comparison to the combined reference, all three assays presented with satisfactory sensitivities and specificities over 95%, with Kappa values ranging from 0.96 to 1.00 (Table 1). One original FluB sample obtained negative results with the Aries and Xpert Xpress assays but generated a positive result from the Cobas assay, and it was confirmed by the Simplexa Flu A/B & RSV assay (Focus Diagnostics). The inconsistent results among the four assays are presented in Table 2 with resolution analysis.

TABLE 1.

Sensitivities and specificities of Aries, Xpress, and Cobas assays

Virus	Assay	Test result/reference standard result				Sensitivity (%) (95%CI)	Specificity (%) (95%CI)	Kappa (95%CI)
		+/+	+/−	−/+	−/−
FluA	Aries	48	1	2	143	96.0 (86.5–98.9)	99.3 (96.2–99.9)	0.96 (0.91–1.0)
	Xpress	50	0	0	144	100.0 (92.9–100)	100.0 (97.4–100)	1.0 (1.0–1.0)
	Cobas	50	0	0	144	100.0 (92.9–100)	100.0 (97.4–100)	1.0 (1.0–1.0)
FluB	Aries	44	1	1	148	97.8 (88.4–99.6)	99.3 (96.3–99.9)	0.97 (0.93–1.0)
	Xpress	44	0	1	149	97.8 (88.4–99.6)	100.0 (97.5–100)	0.99 (0.96–1.0)
	Cobas	45	1	0	148	100.0 (92.1–100)	99.3 (96.3–99.9)	0.99 (0.96–1.0)
RSV	Aries	48	0	1	145	98.0 (89.3–99.6)	100.0 (97.4–100)	0.99 (0.96–1.0)
	Xpress	49	0	0	145	100.0 (92.7–100)	100.0 (97.4–100)	1.0 (1.0–1.0)
	Cobas	49	1	0	144	100.0 (92.7–100)	99.3 (96.2–99.9)	0.99 (0.96–1.0)

TABLE 2.

Specimens with discrepant results among Aries, Xpress, and Cobas assays

No. of specimens detected	Target	Test result (CT value)a
		FilmArray	Aries	Xpress	Cobas	Final
2	FluA	+	−	+ (18.1–35.4)	+	+
1	FluB	+	−	+ (35.5)	+	+
1	RSV	+	−	+ (37.0)	+	+
1	FluB	+	+ (31.8)	−	+	+
1	RSV	−	−	−	+	−
1	FluB	−	−	−	+	−
2	FluB	+	−	−	−	−
1	FluB	+	+ (30.2)	−	−	−c
1	FluA	+b	+	+	+	+
1	FluB	+	+d	+	+	+

^aC_T, threshold cycle.

^bAlso positive for RSV.

^cConfirmed by the Simplexa Flu A/B & RSV assay (Focus Diagnostics).

^dAlso positive for FluA.

The LODs of the three assays were determined in VTM using 10-fold serially diluted FluA, FluB, and RSV control strains with known concentrations of virus. Two hundred microliters of each virus titer was added into the Aries cassette, whereas 200 μl of virus was brought to 300 μl by adding 100 μl of VTM for the Xpress assay detection. The 95% LODs of the Aries assay for FluA(H1), FluA(H1N1), FluA(H3), FluB, RSV-A, and RSV-B were 0.027, 0.003, 0.027, 0.240, 0.538, and 0.170 TCID₅₀/ml, respectively. The 95% LODs of the Xpress for FluA(H1), FluA(H1N1), FluA(H3), FluB, RSV-A, and RSV-B were 0.003, 0.003, 0.027, 0.024, 0.538, and 1.702 TCID₅₀/ml, respectively. The 95% LODs of the Cobas for FluA(H1), FluA(H1N1), FluA(H3), FluB, RSV-A, and RSV-B were 0.003, 0.0003, 0.027, 0.024, 0.054, and 0.170 TCID₅₀/ml, respectively. All three assays had lower LODs for FluA than for FluB and RSV. Because of the difference in starting volume required for each assay, the concentration of viral DNA used for LOD testing differed; however, the quantity of viral DNA used was consistent for all assays.

Three additional specimens positive for FluA, FluB, or RSV were run by three different staff members on three different days in triplicate using one machine and one lot number of reagents from each device. As presented in Table 3, all levels of positives for the different targets had percent agreements of 100% (95% confidence interval [CI], 70.1% to 100%) for the three devices. The coefficients of variation (CVs) of the Aries assay were 2.1%, 1.9%, and 2.2% for FluA, FluB, and RSV, while the Xpress assay had CVs of 2.7 to 2.9%, 2.6%, and 3.2% for FluA, FluB, and RSV, respectively (Table 3). The Aries and Xpress assays showed intratest CVs from 0.5 to 3.3% and 0.7 to 6.1% for the detection of the three targets. The CV values of the Cobas Liat assay were not available.

TABLE 3.

Intra- and interassay variabilities of Aries and Xpress assays^a

Device	Virus	Operator 1		Operator 2		Operator 3		Inter-CV (%)
		CT (mean ± SD)	Intra-CV (%)	CT (mean ± SD)	Intra-CV (%)	CT (mean ± SD)	Intra-CV (%)
Aries	FluA	24.0 ± 0.80	3.3	24.3 ± 0.46	1.9	23.9 ± 0.35	1.5	2.1
	FluB	26.8 ± 0.14	0.5	26.7 ± 0.14	0.5	27.3 ± 0.40	1.5	1.9
	RSV	24.8 ± 0.72	2.9	25.2 ± 0.82	3.3	25.1 ± 0.27	1.1	2.2
Xpressb	FluA1	19.4 ± 0.27	1.4	19.0 ± 0.38	2.0	18.3 ± 0.27	1.5	2.9
	FluA2	22.3 ± 0.50	2.3	21.8 ± 0.51	2.4	21.6 ± 0.78	3.6	2.7
	FluB	20.8 ± 0.93	4.5	20.4 ± 0.35	1.7	20.7 ± 0.21	1.0	2.6
	RSV	21.5 ± 0.15	0.7	21.3 ± 0.17	0.8	21.0 ± 1.28	6.1	3.2

^aC_T, threshold cycle; CV, coefficient of variation.

^bTwo channels (FluA1 and FluA2) are used to detect most influenza virus A strains.

Given the similar performances of these three devices, we focused on additional criteria, such as throughput volume and ease of use. Two workflows, the random-access single-specimen and the 12-specimen batch formats, which simulate routine and seasonal peak throughput volumes at MSKCC, respectively, were used to determine the HOTs and TATs. For single-sample testing, the HOTs were 1.0 ± 0.0 min for all three systems. When 12 specimens were batched and tested, the HOTs were 14.8 ± 1.6 min for the Aries, 11.0 ± 1.2 min for the Xpress, and 14.4 ± 0.5 min for the Cobas assays, with no significant differences found among the three devices. However, the TATs for the three systems were vastly different. The Cobas had the shortest TAT of 21.0 ± 0.0 min when the specimen was run in a random-access single-specimen format. However, using a single instrument to run a batch of 12 specimens required 254.4 ± 0.5 min, which was the longest TAT of the three systems. In contrast, when a GeneXpert XVI instrument (16 units) was used, the TAT for the 12-specimen batch was 44.2 ± 1.2 min, which was only 10 min longer than the single-specimen run (34.2 ± 0.0 min) and was the shortest TAT among the three systems. The Aries TATs for one and 12 specimens were 115.5 ± 0.0 and 128.8 ± 1.6 min, respectively (Table 4).

TABLE 4.

Comparison of workflows of Aries, Xpress, and Cobas assays^a

Assay	System scalability	Mean ± SD (min)a
		Individual HOT	12-sample/batch HOT	Individual TAT	12-sample/batch TAT
Aries	2 × 6 chambers	1.0 ± 0.0	14.8 ± 1.6	115.5 ± 0.0	128.8 ± 1.6
Xpress	16 chambers	1.0 ± 0.0	11.0 ± 1.2	34.2 ± 0.0	44.2 ± 1.2
Cobas	One chamber	1.0 ± 0.0	14.4 ± 0.5	21.0 ± 0.0	254.4 ± 0.5

^aHOT, hands-on time; TAT, turnaround time.

DISCUSSION

In this study, Aries Flu A/B & RSV, Xpress Flu/RSV, and Cobas Flu A/B & RSV assays showed satisfactory LOD, reproducibility, sensitivity, specificity, and overall agreement for the simultaneous detection and identification of FluA, FluB, and RSV in NPS specimens. These nucleic acid amplification-based devices with short hands-on time and turnaround time meet the requirements for speed and ease of use for point-of-care applications. Considering the increased levels of morbidity, mortality, and hospitalization associated with seasonal epidemics of influenza viruses and RSV, these assays, which offer rapid and accurate detection and identification, are likely to have an immediate impact on clinical decisions (20).

We paid close attention to the reliability and deliverability of the three devices during our parallel evaluation. While no invalid results occurred on either the Aries or Xpress assay, 13 (6.5%) specimens yielded invalid results for first-time testing by the Cobas assay. Among them, seven specimens yielded valid results after repeating. The remaining six specimens, which were negative for FluA, FluB, and RSV, remained invalid after repeating the test (data not shown). There were three and two invalid test results during the LOD and reproducibility studies, respectively. The invalid test results for the Cobas assay were the result of fluid relocation or internal process control failure (data not shown). Different levels of invalid results ranging from 0 to 3.3% were reported in the Cobas device previously (see https://www.accessdata.fda.gov/cdrh_docs/pdf15/K153544.pdf) (14, 21–23). Such invalid test results are problematic, because at the very least, these specimens will require retesting. Repeat testing on original or recollected specimens will increase expenses and test turnaround time and decrease the test efficiency.

Molecular assays differ in the number of targets covered, test throughput, hands-on time, the need for nucleic acid extraction, instrumentation, and performance (7–10, 12–19, 24). Both random-access and batched-testing platforms may be needed on the basis of routine and unexpected clinical microbiology practice needs. For a laboratory handling low to medium specimen volumes, the random-access platform is usually considered the mainstay for daily service, because it takes advantage of features, such as simple workflow and rapid turnaround time. The batched-testing platform is useful for unexpected increases, such as a pandemic influenza virus season, during which a batched high-throughput platform is needed. This was demonstrated during the 2009 pandemic season, where the batched system was used successfully to handle the large sample volumes (18, 24). Conversely, batched-testing paradigms reduce efficacy for point-of-care implementation; the random access and fast TAT of the Xpert Xpress and Cobas Liat assays make them better suited for point-of-care use. Rapid return of results would allow clinicians to make informed decisions with regard to antiviral therapy. In our study, the Xpress system possessed relatively shorter TAT at both random-access single-specimen and batched formats when an instrument with scalability to simultaneous run 16 units (GeneXpert XVI) was used. Workflow is dependent on instrument capacity, and we were limited in this study in testing the instrumentation available, which consisted of: one Aries two-module system (12-sample capacity), a GeneXpert XVI system (16-sample capacity), and one Roche Cobas Liat system (1-sample capacity). Thus, our results may not be representative of the workflow when performed on other platform configurations. Since Cepheid instruments can range from two to 48 modules, multiple Cobas instruments can be used, and because the Aries assay is also scalable, these systems are adaptable to the needs of a given laboratory. The costs of the various systems, configurations, and reagents will also vary between institutions and by the structure of contracts with various vendors (i.e., purchase versus reagent rental, etc.). Thus, in our institution, implementation of the Xpress system with relatively high scalability should nicely cover all daily timely antiviral prescribing, POC testing, and seasonal epidemic needs, with acceptable test turnaround time.