Evaluation and Implementation of FilmArray Version 1.7 for Improved Detection of Adenovirus Respiratory Tract Infection

Abstract

The BioFire FilmArray respiratory panel is a multiplex PCR technology capable of detecting a number of bacteria and viruses that cause respiratory tract infection. The assay is technically simple to perform and provides rapid results, making it an appealing option for physicians and laboratorians. The initial product released by BioFire (version 1.6) was reported to have poor sensitivity for adenovirus detection and was therefore of concern when testing immunocompromised patients. This study evaluates the redesigned FilmArray assay (version 1.7) for detection of adenovirus. In this evaluation, we performed both retrospective and prospective verification studies, as well as a detailed serotype analysis. We found that version 1.7 demonstrated improved adenovirus sensitivity. In retrospective studies, sensitivity improved from 66.6% to 90.5%, and in prospective studies, it improved from 42.7% to 83.3%. In addition, when 39 clinically relevant serotypes were tested, 8 were not detected by version 1.6 and only 1 was not detected by version 1.7. The limit of detection remained the same when tested against serotype 4 but improved by 2 log units for serotype 7. Lastly, turnaround time analyses showed that the FilmArray assay was completed 3 h and 9 min after collection, which was more than a 37-h improvement over the previous multiplex PCR assay performed in our laboratory.

INTRODUCTION

Clinical microbiology laboratories now have a number of diagnostic methods that can be employed for the diagnosis of respiratory viral infection. Traditional diagnostic methods, such as viral culture and direct fluorescent antibody (DFA) testing, which suffer from poor sensitivity and inconvenient workflows, have largely been replaced by rapid antigen testing (RAT) and molecular methods. The low sensitivity and specificity of RATs is well described, but their short turnaround time (TAT) and ease of use make them an appealing option (1, 2). Molecular methods offer superior sensitivity and specificity but can have extended turnaround times (3, 4). In addition, some multiplex molecular assays, such as those by Focus Diagnostics (Cypress, CA), Nanosphere (Northbrook, IL), and Hologic Gen-Probe (San Diego, CA), detect only a small number of pathogens. Currently there are only four FDA-approved molecular assays that detect more than 3 pathogens, the Luminex xTAG RVPv1 (Luminex Molecular Diagnostics, Austin, TX), the Luminex xTAG RVP fast (Luminex Molecular Diagnostics), the eSensor RVP (GenMark Dx, Carlsbad, CA), and the FilmArray RP (BioFire Diagnostics, Salt Lake City, UT).

Of these multiplex assays, only the BioFire FilmArray offers a simplified workflow that allows testing to be performed 24 h a day by laboratory staff. This can result in reduced hands-on time and improved TATs, as has been shown by Babady et al. and Xu et al. (5, 6). Although the FilmArray TATs are desirable, a side-by-side comparison to the three other multiplex platforms has shown that the analytical performance for certain pathogens is suboptimal (7). Of particular concern is a low sensitivity for adenovirus, which was shown in the Popowitch et al. study (7) to be 57.1%. These findings are consistent with other reports that have also shown low adenovirus sensitivity for the FilmArray assay (8, 9). Poor performance for adenovirus is of concern when testing immunocompromised patients, who are at greater risk for developing severe infection (10). As a result, laboratories are forced to either confirm all adenovirus-negative results with an alternative method or accept a high false-negative rate.

Given this limitation, BioFire revised the adenovirus portion of their assay to improve the detection of several serotypes that were missed by the original version (v1.6) of the assay (7). The goal of our study was to assess the performance of the revised FilmArray assay (v1.7) for the detection of adenovirus. Here, we assess versions 1.6 and 1.7 in a prospective comparison to a singleplex adenovirus assay validated to detect 39 clinically significant serotypes. In addition, we evaluate the abilities of both FilmArray versions to detect these serotypes, and we conduct limit-of-detection (LOD) experiments for two common respiratory tract-associated serotypes. Lastly, we conduct a workflow evaluation comparing the FilmArray with and without confirmatory adenovirus PCR to the Luminex xTAG RVP.

MATERIALS AND METHODS

Specimens.

Validation specimens included in this study were sputa, nasal aspirates, endotracheal tube aspirates, bronchoalveolar lavage fluid, and nasopharyngeal (NP) swabs collected using the Becton, Dickinson (BD) flocked-swab universal viral transport (UVT) collection kit (BD Diagnostics, Sparks, MD). The FilmArray is only FDA cleared for use on NP swabs. All other specimens constitute off-label use for which the assay was internally validated. This study was approved by the University of Texas Southwestern Medical Center Institutional Review Board. All patients were seen at Children's Medical Center of Dallas, TX, and were less than 18 years of age. Specimens included in the retrospective validation period were positive for adenovirus by either direct fluorescent antibody staining (Diagnostics Hybrids, Inc., Athens, OH) or by Luminex xTAG RVP PCR. All testing was performed according to the manufacturer's recommendations. Retrospectively tested specimens were stored at −80°C in UVT medium. Prospective analysis was performed during two periods. The first period (27 November 2013 to 5 April 2013) included 598 consecutive specimens (nasal aspirates/washes [n = 19], endotracheal aspirates [n = 12], bronchoalveolar lavage fluid [n = 11], sputum [n = 5], and nasopharyngeal swabs [n = 551]) that were tested by both FilmArray v1.6 and Focus Diagnostics adenovirus PCR. The second period (6 April 2013 to 10 April 2013) included 115 consecutive specimens (nasal aspirates/washes [n = 3], endotracheal aspirates [n = 4], bronchoalveolar lavage fluid [n = 2], sputum [n = 1], and nasopharyngeal swabs [n = 105]) that were tested by both the FilmArray v1.7 and the Focus Diagnostics adenovirus PCR.

Real-time PCR assays.

Qualitative and quantitative adenovirus PCRs were performed using analyte-specific reagents and the Integrated Cycler from Focus Diagnostics. Nucleic acid extraction for these assays was performed using the EasyMag Extractor system (bioMérieux). The assay was run as a four-step protocol: step 1, 97°C for 120 s; step 2, 97°C for 10 s; step 3, 60°C for 30 s; and step 4, repeat steps 2 and 3 40 times. Quantitative adenovirus PCR was performed using the above-described protocol with an Acrometrix OptiQuant ADV Plasma panel (ThermoFisher, Lenexa, KS) and reported as copies/ml. FilmArray testing was performed according to the manufacturer's recommendations. The performance of both the qualitative and quantitative adenovirus assays was internally validated. These assays were validated against results provided by a major reference laboratory in the United States. The sensitivity and specificity of both assays were 100%. The limit of detection for both assays was 125 copies/ml. The lower limit of reliable quantitation was 1,000 copies/ml.

Workflow analysis.

Hands-on-time calculations were generated by observing and timing 14 setups over 5 consecutive days. Turnaround time comparisons were calculated as the time between specimen collection and the time of final result entry in the laboratory information system (LIS). During the FilmArray v1.6 period, the assay was not considered complete until the singleplex adenovirus result was finalized.

Limit of detection and serotype testing.

Thirty-nine adenovirus serotypes purchased from the ATCC (Manassas, VA) were cultured and frozen at −80°C as working stocks. These working stocks were thawed and quantified as copies/ml using the assay described above. Each serotype was then diluted to an approximate concentration of 1.0 × 10⁷ copies/ml and was tested at this concentration to assess FilmArray v1.6 and v1.7. Only one limit-of-detection experiment was performed on serotypes 4 and 7, which are commonly associated with respiratory-virus infection. Each serotype was serially 10-fold diluted until assay results were negative (11).

Calculation of sensitivity and statistical analyses.

A combined gold standard method using DFA and xTAG was used for the retrospective calculation of sensitivity. Sensitivity in the retrospective study was calculated as the number of FilmArray positive results divided by the total number of specimens positive for adenovirus (by either DFA or xTAG). In the prospective portion of the study, sensitivity was calculated as the number of FilmArray positive results divided by the number of total positive specimens by the LDT.

Fisher's exact test was used to calculate the confidence interval for sensitivity, as well as to compare the sensitivities of the two tests performed.

RESULTS

Initial validation studies evaluated FilmArray v1.6 against 48 adenovirus-positive clinical specimens. Of these specimens, 19 were mixed viral infections (as determined by Luminex xTAG), which showed adenovirus plus infection with rhinovirus/enterovirus (n = 13), respiratory syncytial virus (RSV) (n = 1), parainfluenza virus (n = 2), RSV and rhinovirus/enterovirus (n = 1), rhinovirus/enterovirus and human metapneumovirus (n = 1), and RSV and influenza A virus (n = 1). FilmArray v1.6 sensitivity for adenovirus was determined to be 66.6% (95% confidence interval [CI], 0.52 to 0.80) for all specimens and 57.9% for mixed specimens (Table 1). Of these 48 specimens, only 10 had sufficient sample volume to be tested against FilmArray v1.7 for validation of the revised assay. An additional 11 positive specimens (obtained after the v1.6 studies had been completed) were included to supplement the evaluation of v1.7, for a total of 21 positive specimens. Seven of these specimens were mixed, all with rhinovirus/enterovirus. FilmArray v1.7 demonstrated significantly improved performance for adenovirus detection, with sensitivities of 90.5% (95% CI, 0.35 to 0.62) for all specimens and 100% among mixed specimens (Table 1). For a direct comparison, we evaluated 10 adenovirus-positive specimens that were previously undetected by FilmArray v1.6. Eight of these 10 specimens were positive with v1.7, for an 80% improvement within this subset. We conducted a limited validation of nonadenovirus analytes, as they were not changed from version 1.6 to 1.7. These studies indicated that performances for the detection of influenza A and B viruses, RSV, parainfluenza virus, coronavirus, Bordetella pertussis, human metapneumovirus, and rhinovirus/enterovirus were unchanged (data not shown).

Table 1.

Comparison of FilmArray version 1.6 and 1.7 adenovirus sensitivities

FilmArray version	Retrospective		Prospective
	No. positive	No. (%) detected by FilmArray (sensitivity) (P = 0.04)	Total no, tested	No. positive	No. (%) detected by FilmArray (sensitivity) (P = 0.20)
1.6	48	32 (66.6)	598	56	27 (48.2)
1.7	21	19 (90.5)	115	6	5 (83.3)

Given the inherent bias present in retrospective validation studies, we evaluated versions 1.6 and 1.7 in two prospective periods. Of 598 consecutive specimens collected during the first prospective period, a total of 56 (9.3%) were positive for adenovirus. FilmArray v1.6 detected 27 of these, for a sensitivity of 48.2% (Table 1). The second prospective period included 115 consecutive specimens, 6 (5.2%) of which were positive for adenovirus. FilmArray v1.7 detected 5 of the 6, for a sensitivity of 83.3% (Table 1).

Previous reports, as well as the package insert, for the FilmArray v1.6 assay suggested that there were certain adenovirus serotypes that the assay was unlikely to detect (7). To understand exactly which serotypes the improved assay could detect, we tested a standardized concentration (1 × 10⁷ copies/ml) of 39 clinically relevant adenovirus serotypes (Table 2). These analyses demonstrated improved detection of nearly all adenovirus serotypes. Version 1.6 failed to detect 8 of 39 serotypes, i.e., serotypes 6, 12, 20, 28, 31, 32, 36, and 40. Version 1.7 failed to detect only serotype 31.

Table 2.

Detection of adenovirus serotypes by FilmArray v1.6 and v1.7

Species	Serotype(s) testeda	Serotype(s) not detected
		v1.6	v1.7
A	12, 18, 31	12, 31	31
B1	3, 7, 16, 21
B2	11, 14, 34, 35
C	1, 2, 5, 6	6
D	8, 9, 13, 15, 17, 19, 20, 22–30, 32, 33, 36–39, 42	20, 28, 32, 36
E	4
F	40	40
G	None

^aAll serotypes were tested at a concentration of 1.0 × 10⁷ copies/ml.

In addition, previous reports suggested that FilmArray v1.6 had a high lower limit of detection for adenovirus (12). The limit-of-detection experiments in this study did not specify which serotype was used in their calculations but used representative clinical isolates. To understand how the limit of detection may have changed with version 1.7, we conducted testing of serial 10-fold dilutions of two serotypes (4 and 7) known to be associated with respiratory virus infection (11). FilmArray v1.6 and v1.7 had the same limit of detection for serotype 7 at 1,760 copies/ml. However, version 1.7 demonstrated an LOD that was 100-fold lower than that of version 1.6, at 125 and 12,500 copies/ml, respectively, for serotype 4.

Lastly, a workflow analysis was conducted to assess the impact of FilmArray on technologist resource utilization and test TAT. To calculate technologist hands-on time, we observed the processing of 14 total FilmArray runs over a 5-day period. The average time it took a technologist to completely process a specimen was 2 min and 54 s. Turnaround times (specimen collection to final result) achieved for respiratory virus testing were analyzed over three phases of implementation. The first was a retrospective pre-FilmArray evaluation of a month-long period using the Luminex xTAG RVP (March 2012). During this period, 409 specimens were processed at an average TAT of 40 h and 42 min. During the second period (March 2013), FilmArray v1.6 was employed. Due to low sensitivity, all adenovirus specimens were confirmed with a more sensitive singleplex assay (Focus Diagnostics). During this period, the turnaround time accounted for those specimens requiring confirmatory PCR. A total of 396 specimens were tested, with an average TAT of 23 h and 31 min. The final evaluation period was April 2013, when FilmArray v1.7 was implemented without confirmatory singleplex PCR. A total of 224 specimens were tested during this time, with an average TAT of 3 h and 9 min (data not shown). Importantly, adenovirus prevalences were 1.3% for March 2012 and 2.5% in 2013. These time periods were not statistically significantly different (P = 0.170).

DISCUSSION

To our knowledge, this is the first study to evaluate the performance of the revised FilmArray respiratory virus panel version 1.7. The original FilmArray v1.6 has been extensively studied, with varying results (6–9, 12). However, one common finding in these studies is that the FilmArray assay performed poorly for the detection of adenovirus. The revised version 1.7 includes a second adenovirus assay designed to improve upon the poor adenovirus detection of version 1.6. This poor performance puts microbiology laboratories in the difficult position of having to decide whether to confirm all negative results, accept a high false-negative rate, or implement a different assay. The cost of confirming adenovirus results is significant and may factor into the strategy for implementing FilmArray testing. The combined prevalence for our two prospective evaluation periods was approximately 10%. If 90% of specimens require confirmatory testing, the overall cost of respiratory testing would approximately double. This analysis considers reagent costs, as well as technologist time.

Here, we conducted retrospective and prospective evaluations of FilmArray performance, and we came to several important conclusions. First, we note the significant difference observed between prospective and retrospective analyses. For both versions 1.6 and 1.7, the retrospectively calculated sensitivity was significantly higher than the prospective evaluation. This is likely due in part to the use of a superior comparator method (Focus Diagnostics) in the prospective phase of the evaluation. The retrospective comparator (xTAG) has been shown to have low sensitivity for adenovirus detection (7). Second, we found that FilmArray v1.7 demonstrated significantly improved sensitivity for adenovirus compared to version 1.6. This is an important finding, because it is contrary to the body of literature that currently exists for FilmArray v1.6. Our results suggest that sensitivity is now somewhere between 83 and 91%, which supports the decision by the FDA to remove the adenovirus warning that had been present in the package insert for version 1.6.

Although our data showed that FilmArray v1.7 demonstrated improved performance for adenovirus detection, the singleplex LDT demonstrated superior performance. This is not surprising, as it is generally accepted that multiplex assays demonstrate poorer sensitivity than singleplex assays. Nonetheless, laboratories may miss some adenovirus-infected patients by using multiplex PCR methods for the detection of adenovirus. Patients at high risk for severe adenovirus infection may still require confirmatory testing in special circumstances.

The implications of improved adenovirus detection by FilmArray are significant. If laboratories can discontinue confirmatory testing, they can take full advantage of the simplified FilmArray workflow and rapid TAT. In our laboratory, for example, we are now able to perform respiratory virus testing 24 h a day, 7 days a week. As a result, we have now achieved a turnaround time of just over 3 h, which is a full 1.5 days earlier than what is possible with other multiplex assays that require batch testing.

In addition to clinical sample validation, we evaluated the performance of FilmArray in detecting 39 clinically significant adenovirus serotypes at concentrations of 1.0 × 10⁷ copies/ml. It should be mentioned that this concentration was selected because it was approximately 1 log unit above the LOD previously published by Couturier and colleagues (12). As expected, we found that version 1.7 detected a greater number serotypes than version 1.6, including all serotypes associated with respiratory virus infection. Interestingly, we showed that FilmArray v1.7 failed to detect serotype 31. According to the package insert, the assay should be able to detect this virus. One possible explanation for this discrepancy is that we tested the virus at a concentration below the LOD for that particular serotype. Following these experiments, we went on to calculate the LOD of each version for serotypes 4 and 7. Our data show that the FilmArray v1.7 LOD for both serotypes is significantly lower than that published for version 1.6 by Couturier et al. (12). However, we do not know what serotype was used in those LOD calculations, so it is difficult to know why our value differed so significantly. One likely explanation is that we tested different serotypes that simply have different FilmArray LODs.

This study has several limitations. First, the duration of our comparison periods for turnaround time (1 month) is relatively short. However, these periods included a large number of tests from which a reasonable average could be calculated. Second, these evaluations occurred during a time when overall respiratory virus disease activity was low. A limitation of the FilmArray assay is that it has low throughput, because only one test can be performed per instrument per hour. Thus, it remains to be seen if the TATs observed in this study can be achieved during periods of high-volume testing. Third, the prospective evaluation period included only 115 total specimens and 6 positives. The resulting sensitivity of 83.3% might have been different if more specimens had been tested. Fourth, we conducted LOD studies only for serotypes 4 and 7, and our results may not be generalizable to other serotypes. In addition, it was cost prohibitive to conduct these studies in replicate, and data were generated from a single experiment. Lastly, the prospective analyses in this study evaluated v1.6 and v1.7 during different time frames. This was a logistical necessity due to reagent availability.

We conclude that FilmArray v1.7 significantly improves the detection of adenovirus from respiratory specimens over the original version 1.6. This is likely due to an increased ability to detect a wider range of serotypes with a lower LOD than had been previously reported. This important finding suggests that laboratories can consider discontinuing costly and labor-intensive confirmatory tests, allowing them to take full advantage of the short FilmArray TAT.