Reproducibility of positive results for rare pathogens on the FilmArray GI Panel

Abstract

Though the FilmArray GI Panel has a reported aggregate specificity and reproducibility of >97% and > 99%, respectively, the reproducibility is less understood in clinical practice. We measured the reproducibility of positive results for low-prevalence pathogens. Samples with positive results for selected targets were repeated using a different FilmArray module. Overall, 331 of 373 (89%) results were reproducible. Giardia lamblia (57/57,100%), Cryptosporidium spp. (61/63, 97%), Cyclospora cayetanensis (34/35, 97%), Plesiomonas shigelloides (17/18,94%), and Rotavirus A (76/77,99%) were highly reproducible, while Adenovirus F40/41 (38/54, 70%), Vibrio spp. (8/10, 80%), V. cholerae (3/8, 37.5%), and Yersinia enterocolitica (36/50, 72%) were poorly reproducible. Review of 38 patients with nonreproducible results showed that 19 (50%) had evidence of gastroenteritis and only 6 (16%) had possible infection with the organism that showed a nonreproducible result. Higher false-positive rates with certain targets on FAGP emphasize the need for diagnostic stewardship.

1. Introduction

The microbiological assessment of patients with acute gastroenteritis has been revolutionized by the development of syndromic PCR assays with much greater sensitivity and efficiency than classic, culture-and microscopy-based tests. One of the most widely adopted has been the FilmArray GI Panel (FAGP; BioFire Diagnostics, Salt Lake City, UT), which is an on-demand, sample-to-answer, real-time PCR assay that targets 22 pathogens, including 13 bacteria, 5 viruses, and 4 parasites. The reported overall specificity of the assay is >97% per major validation studies by Buss et al. (2015) and Khare et al. (2014), and the overall reproducibility and negative percent agreement per the package insert are >99%. While these studies suggest high specificity and high reproducibility, in clinical practice, positive results in areas of low background prevalence for the specific organism–especially in patients with a nonspecific, atypical presentation where the pretest probability for any infectious etiology may be very low–are of uncertain value as they are highly likely to represent a false-positive, nonreproducible result. This was our anecdotal conclusion for several cases at our institution where the initial FAGP was positive for Y. enterocolitica but both culture and repeat FAGP were negative and the clinical course was not consistent with Y. enterocolitica infection. While false negatives are often considered to be major errors in diagnostic testing, the consequences of a false-positive result are not insignificant and include inappropriate treatment, which may lead to adverse drug effects and the selection of antimicrobial resistance, unnecessary public health investigations, and premature closure of the diagnostic workup (Burman and Reves, 2000; Graber et al., 2005).

At our institution, beginning in October 2016, 14 months after implementing FAGP for patient care and prompted by the cases of nonreproducible results for Y. enterocolitica, positive results for targets rarely positive by FAGP at our institution (Park et al., 2017), including Adenovirus F40/41, Cryptosporidium spp., C. cayetanensis, E. histolytica, G. lamblia, P. shigelloides, Rotavirus A, Vibrio spp., and Y. enterocolitica, were systematically assessed for reproducibility by repeating the assay from the same sample. The results of these repeat tests are reported here along with clinical adjudication to determine the possibility of false positivity in those with nonreproducible results.

2. Methods

2.1. Ethics

Per Stanford Institutional Review Board (IRB), this project constituted a quality improvement project and was exempt from IRB approval.

2.2. FilmArray GI Panel

Stools in Cary-Blair transport media were tested with the FilmArray GI Panel per the package insert. The laboratory began to offer the FAGP for clinical care in August 2015. From October 2016 through June 2018, tests positive for Adenovirus F40/41, Cryptosporidium spp., C. cayetanensis, E. histolytica, G. lamblia, P. shigelloides, Rotavirus A, Vibrio spp., and Y. enterocolitica were systematically retested by repeating the FAGP assay from the same sample on a different FilmArray instrument of the 4 available. No additional confirmatory testing with alternative methods was done. Nonreproducible results were reported as negative in the electronic medical record. Results for Clostridium difficile, enteroaggregative E. coli, enteropathogenic E. coli, and enterotoxigenic E. coli are not reported in the medical records at our institution per laboratory policy, as described previously (Hitchcock et al., 2018).

2.3. Chart review

Chart review was performed retrospectively by M.M.H. and C.A.H. when FAGP results for rare targets were not reproducible. Adjudicators were not blinded to these results. Reviews were performed independently, and a consensus determination between the reviewers was pursued in cases of initial disagreement. The final assessment was made by N.B. if no consensus by the primary reviewers could be reached. Age and sex of the patient and location of testing were collected. A determination of whether the patient had evidence of infectious gastroenteritis using a 4-point scale of definite, probable, possible, and unlikely was made based on evaluation of the available clinical data. This scale incorporates both the presence of a significant diarrheal illness consistent with the 2017 IDSA guidelines and the available diagnostic testing results from the institution (Shane et al., 2017), which included additional FAGP tests from different samples. A case was considered definite only if there was additional diagnostic evidence of infection aside from the nonreproducible target, such as a reproducible target (separate from the nonreproducible target) from the same paired FAGP tests or a positive result for the nonreproducible target by a different test, which could include a reproducibly positive FAGP result performed on a different sample. Example of a definite case would be a nonreproducible Adenovirus F40/41 result in a sample that has a reproducible result for Sapovirus (see patient ID no. 1 in Table 2). A case was considered a probable infection if there were no other test results aside from the nonreproducible target on FAGP suggestive of infection but no alternative cause of diarrhea could be plausibly identified. A case was considered a possible infection if there were no other test results suggestive of infection and an alternative cause of diarrhea was present but considered about as likely as infection with the nonreproducible FAGP target. A case was considered unlikely to be an infection if an alternative cause of diarrhea was considered to be more likely that an infection with the nonreproducible FAGP target. A secondary determination was made as to whether infection could be plausibly attributed to the nonreproducible target using the same scale, with cases considered definite only if there was additional evidence of the nonreproducible organism by an alternative testing method, which could include additional FAGP tests with a different sample collected during the same illness; probable if the case was clinically consistent with infection by the nonreproducible target and no other potential infectious causes were identified; possible if the case was less consistent with typical infection by the nonreproducible target but no other potential infectious causes were identified; and unlikely if the case was considered highly inconsistent with infection by the nonreproducible target or an alternative infectious cause was identified. All cases that were considered unlikely to represent clinical infectious gastroenteritis based on presenting symptoms were considered unlikely to be caused by the nonreproducible target (Shane et al., 2017). Basic descriptive statistics were calculated using SPSS v. 24 (IBM Analytics, Armonk, NY), but no hypothesis testing was performed.

Table 2.

Consensus assessment of the presence of clinical infection for patients with a nonreprodudble FAGP result.

Patient ID no.	Age (yrs)	Sex	Location	Initial FAGP result	Repeat FAGP result	Clinical adjudication for infectious gastroenteritis	Adjudication for infection due to nonreproducible result
1	32	F	Outpatient	Adenovirus, Sapovirus	Sapovirus	Definite	Possible
2	37	F	Inpatient (HD 3)	Adenovirus	Negative	Unlikely	Unlikely
3	13	M	Outpatient	Adenovirus	Negative	Probable	Probable
4	90	M	Inpatient (HD 3)	Adenovirus	Negative	Unlikely	Unlikely
5	77	M	Inpatient (HD 2)	Adenovirus	Negative	Unlikely	Unlikely
6	49	F	Inpatient (HD 1)	Yersinia enterocolitica	Negative	Unlikely	Unlikely
7	2	M	Outpatient	STEC, Yersinia enterocolitica	STEC	Definite	Unlikely
8	1	F	Inpatient (HD 2)	Adenovirus	Negative	Possible	Unlikely
9	12	F	Outpatient	Yersinia enterocolitica	Negative	Unlikely	Unlikely
10	65	F	Inpatient (HD 7)	Vibrio spp.	Negative	Unlikely	Unlikely
11	3	F	Outpatient	Adenovirus	Negative	Unlikely	Unlikely
12	13	M	Inpatient (HD 2)	Yersinia enterocolitica	Negative	Possible	Unlikely
13	14	M	In-patient (HD 11)	Cryptosporidium	Negative	Definite	Definite
14	64	M	Outpatient	Yersinia enterocolitica	Negative	Unlikely	Unlikely
15	54	M	Outpatient	Adenovirus	Negative	Unlikely	Unlikely
16	68	M	Outpatient	Yersinia enterocolitica	Negative	Possible	Possible
17	19	M	Outpatient	STEC, Yersinia enterocolitica	STEC	Definite	Unlikely
18	75	F	Outpatient	Yersinia enterocolitica	Negative	Unlikely	Unlikely
19	68	M	Outpatient	Yersinia enterocolitica	Negative	Unlikely	Unlikely
20	19	F	Outpatient	Yersinia enterocolitica	Negative	Possible	Possible
21	67	F	Outpatient	Salmonella spp., Yersinia enterocolitica	Salmonella spp.	Definite	Unlikely
22	21	M	Outpatient	Campylobacter spp., Plesiomonas shigelloides, Salmonella spp., Norovirus	Campylobacter spp., Salmonella spp., Norovirus	Definite	Possible
23	56	F	Outpatient	Adenovirus	Negative	Unlikely	Unlikely
24	36	F	Outpatient	Shigella spp., Sapovirus, Cryptosporidium	Shigella spp., Sapovirus	Definite	Unlikely
25	28	M	Outpatient	Cryptosporidium, Yersinia enterocolitica, Adenovirus	Cryptosporidium, Yersinia enterocolitica	Possible	Unlikely
26	26	F	Outpatient	Vibrio spp.	Negative	Unlikely	Unlikely
27	83	F	Outpatient	Yersinia enterocolitica	Negative	Unlikely	Unlikely
28	14	F	Inpatient (HD 1)	Yersinia enterocolitica	Negative	Possible	Unlikely
29	62	M	Outpatient	Vibrio cholerae	Negative	Probable	Unlikely
30	4	M	Inpatient (HD 3)	Adenovirus	Negative	Possible	Unlikely
31	35	F	Outpatient	Astrovirus, Vibrio cholerae	Astrovirus	Definite	Unlikely
32	68	F	Outpatient	Vibrio cholerae	Negative	Possible	Unlikely
33	58	F	Outpatient	Adenovirus	Negative	Unlikely	Unlikely
34	30	M	Outpatient	Adenovirus	Negative	Unlikely	Unlikely
35	52	M	In-patient (HD 9)	Vibrio cholerae	Negative	Unlikely	Unlikely
36	73	F	Outpatient	Yersinia enterocolitica	Negative	Possible	Unlikely
37	63	F	Outpatient	Vibrio cholerae	Negative	Unlikely	Unlikely
38	48	M	Inpatient (HD 5)	Adenovirus	Negative	Unlikely	Unlikely
39	44	M	Outpatient	Adenovirus	Negative	Unknown	Unknown
40	38	M	Outpatient	Adenovirus	Negative	Unknown	Unknown
41	57	M	Outpatient	Cyclospora cayetanensis	Negative	Unknown	Unknown
42	1	M	Outpatient	Rotavirus	Negative	Unknown	Unknown

STEC = Shiga-like toxin-producing E. coli (non-O157 strain); HD = hospital day number for those tested while inpatient.

Cells labeled “unknown” (patient IDs 39–42) represent cases where no clinical data from the time of testing beyond demographics and location were available for retrospective review and assessment.

The last 2 columns are intended to determine if the patient had an infectious cause of gastroenteritis and, if yes, whether it could be attributed to the nonreproducible FAGP result.

3. Results

Over the study period, a total of 5877 FAGP tests were performed and 1117 (19.0%) were positive for at least 1 target. Of these positive tests, 373 rare pathogens were detected in 367 tests. When these tests were repeated, 331 (89%) of the targets were reproducibly positive. Results by pathogen are shown in Table 1. The 42 (11%) nonreproducible results occurred in 42 (11%) separate tests in 42 unique patients. Thirty-four (81%) of the tests with a nonreproducible result were positive for only a single target on the initial test (Table 2). Reproducibility rates were particularly high for parasitic targets, especially G. lamblia, where 100% (57/57) of positive results were reproducible, but also Cryptosporidium spp. with 97% (61/63) and C. cayetanensis with 97% (34/35), and were also high for Rotavirus A (99%, 76/77) and P. shigelloides (94%, 17/18). There was only a single test positive for E. histolytica, and the repeated test was positive as well. Reproducibility rates were lower for Adenovirus F40/41 (70%, 38/54), Y. enterocolitica (72%, 36/50), and non-cholerae Vibrio spp. (80%, 8/10) and very poor for V. cholerae (37.5%, 3/8). Eight (50%) of the nonreproducible adenovirus results occurred in November 2017. Clinical assessment by retrospective chart review of the nonreproducible results is shown in Table 2, though 4 patients (ID numbers 39–42) had no data from the time of testing available for review. The average age of the patients was 41.4 years (range 1–90 years), and 22 (52%) of the patients were male. Thirty (71%) patients were tested in the outpatient setting, which includes the emergency department. Not including the 4 patients with missing clinical data, 19 of 38 (50%) patients were assessed as having clinical evidence of infectious gastroenteritis, of which 8 (21.1%) were considered definite, 2 (5.3%) were considered probable, and 9 (23.7%) were considered possible (Table 2); however, only 6 (15.4%) of 38 patients were thought to have clinical evidence of infection caused by the nonreproducible target, which was deemed definite in 1 (2.6%) patient, probable in 1 (2.6%) patient, and possible in 4 (10.5%) patients (Table 2). Of the 8 (21.1%) patients thought to have definite gastroenteritis, 7 (87.5%) had additional, reproducibly positive targets on the FAGP that likely explained the patients’ presentations; the only patient judged to have definite gastroenteritis caused by the nonreproducible target had hyper-IgM disorder and known chronic cryptosporidiosis with multiple positive, reproducible results for Cryptosporidium spp. on FAGP both before (50 days before) and after (20 days after) the nonreproducible result (Table 2). The 2 patients with probable gastroenteritis had an acute onset of symptoms consistent with an infectious gastroenteritis and no other testing performed; one was judged to be probably caused by a nonreproducible adenovirus (patient number 3), and the other was completely inconsistent with Vibrio cholerae infection by degree of symptoms and lack of travel history (patient number 29; Table 2). Three (33.3%) of the 9 patients judged to have possible gastroenteritis had organisms detected on other tests that may have caused the diarrheal symptoms, including C. difficile by GeneXpert C. diff./Epi tcdB PCR (Cepheid, Sunnyvale, CA), and plasma adenovirus PCR and HHV-6 PCR (laboratory-developed tests, Stanford Clinical Virology Laboratory), though each with a background of acute and/or chronic illness with a potential noninfectious etiology of diarrhea, including chemotherapy-induced mucositis with febrile neutropenia during therapy for acute lymphoblastic leukemia, gastrointestinal graft-versus-host disease following hematopoietic stem cell transplantation for acute myeloid leukemia, and chronic liver disease (Table 2; patient study numbers 28,12, and 8, respectively), and the remaining 6 (66.7%) had nonspecific symptoms. Of the 19 (50%) patients not thought to have infectious gastroenteritis, 7(18%) had alternative causes of acute diarrhea that were more likely than infection, generally a side effect from broad-spectrum antibiotic use or other medications, and the remaining 12 (32%) had chronic diarrhea that was inconsistent clinically with the nonreproducible target and was likely due to a noninfectious etiology such as irritable bowel syndrome.

Table 1.

Rare pathogens with nonreproducible results on the FilmArray GI Panel.

Pathogen	Reproducible result	Nonreproducible result	Total	Percent reproducible
Adenovirus F40/41	38	16	54	70.3
Cryptosporidium spp.	61	2	63	96.8
Cyclospora cayetanensis	34	1	35	97.1
Entamoeba histolytica	1	0	1	100
Giardia lamblia	57	0	57	100
Plesiomonas shigelloides	17	1	18	94.4
Rotavirus A	76	1	77	98.7
Vibrio spp. (non-cholerae)	8	2	10	80.0
Vibrio cholerae	3	5	8	37.5
Yersinia enterocolitica	36	14	50	72.0
Aggregate	331	42	373	88.7

4. Discussion

While multiplex PCR assays such as the FAGP have undoubtedly improved clinicians’ ability to make a rapid, specific diagnosis for patients with acute gastroenteritis and have improved laboratory workflow efficiency, the reproducibility of low-prevalence pathogens in clinical practice remains unknown (Buss et al., 2015; Khare et al., 2014). This study of reproducibility of positive results for these pathogens shows that most are reproducibly positive on repeat testing from the same stool sample, particularly for parasitic organisms, including Cryptosporidium spp., C. cayetanensis, and G. lamblia, as well as those for P. shigelloides and Rotavirus A, increasing confidence that these results are true positives and the test has high analytical specificity for these pathogens. Some results, particularly for Adenovirus F40/41, Vibrio spp., and Y. enterocolitica, were less reproducible, suggesting that there may be lower specificity for these targets. However, as 50% of the nonreproducible adenovirus results occurred in a single month, it is possible that these results are evidence of a deficient batch of FilmArray cartridges rather than systematic error associated with this assay. We reported a similar finding of clustered, unconfirmable Campylobacter spp. cases that resolved with change to a new batch of cartridges in a prior study (Park et al., 2017), so this may be a plausible explanation. When all adenovirus results from November 2017 are removed from the analysis (8 nonreproducible and 3 reproducible), the reproducibility rate rises from 70% to 81% (35/43). The nonreproducible Vibrio spp. and Y. enterocolitica results were distributed throughout the study period, suggesting a higher likelihood of intrinsic issues with detection of these pathogens, most likely related to low prevalence of these organisms. However, recent technical notes from BioFire reported false-positive results for Vibrio spp., including V. cholerae, and Y. enterocolitica, attributed to the presence of nonviable organisms, or nucleic acids derived from these organisms, in agar components of Cary-Blair media thought be due to the use of seaweed obtained from locations with contaminated water supplies (technical notes FLM1-PRT-0239 and FLM1-PRT-0250, BioFire Diagnostics). While false positives in a low prevalence setting is still the most likely explanation for the nonreproducible results for these organisms, the presence of low levels of nonviable organisms or nucleic acids near the limit of detection would also potentially explain nonreproducible results for Vibrio spp. and Y. enterocolitica. Of note, the 3 cases of V. cholerae with reproducible results in this study could have been affected by this phenomenon as well, as none of the cases were clinically consistent with cholera at the time of testing (Table 1). One patient had no travel exposure, suggesting this was an actual false-positive result. The other 2 cases are more difficult to interpret as both patients were returning travelers and both samples also contained other organisms (Campylobacter spp. in the first and both G. lamblia and Shigella spp. in the second), suggesting either true-positive results due to transient colonization with the organism (Weil et al., 2009), contamination with nonviable organisms or nucleic acids in the samples from the Cary-Blair media, or more typical causes of false-positive results due to nonspecific error in low-prevalence settings. It is also possible that other cases with reproducible results for non-cholerae Vibrio spp. and Y. enterocolitica included in this study were also consistently positive due to the presence of nucleic acids in Cary-Blair media. Half of nonreproducible positive results assessed in this study occurred in settings where the patient did not appear to have infectious gastroenteritis by retrospective chart review, reinforcing the importance of using the FAGP only in settings where there is a high pretest probability for an infectious gastroenteritis with at least 2 of the pathogens on the panel (Morgan et al., 2017).

There are several limitations to this study. First, reproducibility of positive FAGP results was used as a proxy for a true positive result. A nonreproducible, positive result could be a true positive if there was a low burden of organism present due to poor sample acquisition or sampling error, although Cary-Blair stool tends to be homogenous. No testing with reference methods was done as a comparison, though given the lower sensitivity of culture-based methods compared to PCR (Cybulski et al., 2018), it is unlikely that these would have helped discriminate between the causes of nonreproducibility. Testing with another molecular method might have helped resolve nonreproducible results. This was a single-center study, so it is possible that reproducibility may vary significantly between institutions. Despite over 18 months of data collection, 2 targets (E. histolytica and V. cholerae) still had <10 results, so studies with larger numbers are needed to investigate reproducibility of positive results for these pathogens. Chart review was done retrospectively and represents an inherently subjective determination of whether an infectious gastroenteritis was present given the often-limited documentation available.

5. Conclusion

In conclusion, positive results for rare pathogens on the FAGP were highly reproducible overall, especially for parasitic targets Cryptosporidium spp., C. cayetanensis, and G. lamblia, as were those for P. shigelloides and Rotavirus A, but poorly reproducible for Adenovirus F40/41, Vibrio spp., and especially V. cholerae and Y. enterocolitica, suggesting that positive results for these latter targets by FAGP require additional scrutiny. Repeat testing would eliminate a portion of these potential false positives. Ultimately, all positive results from any test should be interpreted in the clinical context, and this study suggests that these nonreproducible results may be more likely to occur in settings where there is a low pretest probability of infectious gastroenteritis, especially in patients with nonspecific presentations in low-prevalence settings, providing additional evidence for the need for diagnostic stewardship principles to guide the use of multiplex molecular assays (Hitchcock etal., 2018; Morgan et al., 2017).