Longitudinal Evaluation of Clinical Decision Support to Improve Influenza Vaccine Uptake in an Integrated Pediatric Health Care Delivery System, Houston, Texas

Abstract

Objective  Our study retrospectively evaluated the implementation of an influenza vaccine best practice alert (BPA) in an electronic medical record within an integrated pediatric health care delivery system.

Methods  An influenza BPA was implemented throughout a large pediatric health care delivery system in Houston, TX, to improve vaccine uptake. Outcomes were measured retrospectively over 3 years of BPA implementation and compared with a control year prior to BPA implementation. Primary outcomes were influenza vaccine uptake, distribution of influenza vaccines ordered by week, proportion of BPA displays ignored, and missed vaccination opportunities.

Results  Influenza vaccine uptake declined from the pre-BPA year (47.2%; 95% confidence interval [CI]: 47.0, 47.4) to the last study year (45.1%; 95% CI: 44.9, 45.2). BPA displays were increasingly ignored by clinical staff throughout the study years from 59.6% in 2014–2015 to 72.5% in 2016–2017. For providers, BPA displays were ignored less frequently each year from 53.4% in 2014–2015 to 51.4% in 2017–2017. Within the primary care outpatient group, the proportion of missed vaccination opportunities in sick visits decreased from 86.8% during the pre-BPA year to 81.0, 79.8, and 82.7% during the subsequent study years 2014–2015, 2015–2016, and 2016–2017, respectively.

Conclusion  Implementation of a widespread influenza BPA in an integrated pediatric health care delivery system did not produce meaningful increases in influenza vaccine uptake. Differences between clinical staff and providers on BPA use warrant further investigation.

Background and Significance

Influenza causes significant morbidity and mortality in the United States, with an estimated 12,000 to 80,000 deaths from influenza each year since 2010. 1 2 High-risk populations including young children, pregnant women, older adults, and those with chronic medical conditions are particularly susceptible to influenza infection and its complications. 3 Influenza vaccine remains the best available method to prevent influenza infection, and annual influenza vaccination for everyone 6 months of age and older is recommended by the Centers for Disease Control and Prevention (CDC) and the American Academy of Pediatrics (AAP). 3 4 Despite a universal recommendation, influenza vaccination rates continue to be suboptimal. For the 2017–2018 influenza season, estimates show that only 57.9% of children aged 6 months through 17 years were vaccinated, 5 which is far below the Healthy People 2020 goal of 80% coverage for this population. 6

Efforts to improve influenza vaccine uptake are necessary to achieve population level improvements among high-risk groups. Electronic medical record (EMR) systems are utilized in 76% of nonfederal acute care hospitals 7 and 94% of pediatric practices, 8 offering a readily available tool to assist providers in their efforts to increase influenza vaccination. 9 10 11 Many EMRs offer clinical decision support tools, such as best practice alerts (BPAs), which have demonstrated the potential to improve vaccination rates in children. 12 13 14 15 16 BPAs (i.e. pop-up reminders) are implemented to assist providers in improving medical care by addressing current care gaps. However, to our knowledge, no study to date has analyzed the long-term effects of a widespread influenza vaccine BPA in a large integrated pediatric health care delivery system such as Texas Children's Hospital.

Texas Children's Hospital offers outpatient primary through inpatient quaternary care for women and children in the greater Houston area. Moreover, the inpatient clinics participate in the U.S. News and World Report (USNWR) hospital rankings. Since 2007, USNWR has ranked pediatric hospitals based on hospital-reported questionnaire data. 17 Part of the USNWR-reported data includes influenza vaccinations for patients with selected conditions, such as ventilator dependence, cystic fibrosis, asthma, diabetes, and cancer on active chemotherapy. 17

Objective

An influenza BPA was developed and implemented throughout Texas Children's Hospital to increase influenza vaccine-related documentation and streamline USNWR vaccine-related reporting. The purpose of this study is to retrospectively evaluate the long-term impact of an influenza BPA on influenza vaccine uptake in Texas Children's Hospital.

Methods

Setting

Texas Children's Hospital includes a women and children's hospital that offers primary through quaternary care in multiple locations throughout a large metropolitan area. This study includes data from three areas of Texas Children's Hospital: (1) pediatric primary care practices, (2) outpatient pediatric subspecialty clinics, and (3) health plan primary care practices. The primary care practices, Texas Children's Pediatrics, are a network of more than 50 pediatric primary care practices employing more than 250 board-certified pediatricians. Pediatric outpatient subspecialty services are offered at multiple locations throughout the metropolitan area and include a comprehensive array of pediatric subspecialty care. Health plan primary care practices, the Center for Women and Children, are general practices that offer comprehensive care to children who are insured by the system-owned health plan insurance (Medicaid or CHIP option). Data from other locations—including mobile clinics; urgent care practices; inpatient, emergency department; and women's services—were excluded. Pediatric outpatient services henceforth refer to pediatric primary care practices, pediatric subspecialty services, and health plan primary care practices combined.

The Baylor College of Medicine Institutional Review Board approved this study.

Best Practice Alert Design and Workflow

We collaborated with colleagues in information services to develop an electronic BPA (i.e., a pop-up reminder) to display in the EMR (Epic Systems Corporation, Verona, Wisconsin, United States). Upon arrival of a patient to a participating location ( Fig. 1 ), the BPA displayed as a highlighted alert for all patients 6 months of age and older who had no documentation of an influenza vaccine for that season.

Fig. 1.

Best practice alert (BPA) flowchart. ∗See Table 1 .

Clinical staff (including registered nurses, licensed vocational nurses, and medical assistants) could acknowledge the BPA in one of three ways: order influenza vaccine, document influenza vaccine given elsewhere, or select an alternate response ( Table 1 ). If the clinical staff did not acknowledge the BPA or if they selected “Provider Review” from the alternate responses, then the BPA would display for the encounter provider. Providers included physicians, fellows, residents, nurse practitioners, and physician assistants. The influenza vaccine orders were associated with standing orders (SOs). SOs are written protocols approved by a physician that allow qualified health care providers to evaluate the need for vaccines, screen for contraindications, and administer appropriate vaccines. 18 Use of SO provided the potential to streamline influenza vaccine delivery and improve workflows within the practice. Although SOs were used in some areas prior to influenza BPA implementation, they were not used system-wide until the beginning of the 2014–2015 influenza season.

Table 1. BPA acknowledgment responses and subsequent effect.

Acknowledgment	Effect
Order influenza vaccine	Satisfies BPA for remainder of influenza season
Document influenza vaccine given outside of health care delivery system	Satisfies BPA for remainder of influenza season
History of anaphylaxis	Satisfies BPA for remainder of influenza season
Vaccine not available	Satisfies BPA for that encounter; alert continues to display in other encounters within the same day or at next visit
Provider review	Satisfies BPA for clinical staff and displays for provider in that encounter
Patient not present (i.e., conference or phone visit)	Satisfies BPA for that encounter; alert continues to display in other encounters within the same day or at next visit
Not eligible today (based on provider judgment)	Satisfies BPA for that encounter but displays again after 48 h
Not eligible this season (based on provider judgment)	Satisfies BPA for the remainder of influenza season
Decline for season	Satisfies BPA for the remainder of influenza season
Decline for today	BPA satisfied temporarily; displays again after 48 h

Abbreviation: BPA, best practice alert.

The BPA allowed clinical staff to document influenza vaccine receipt if it was given elsewhere. Parent or patient self-report of influenza vaccine receipt could be documented within the BPA; however, proof of vaccine administration was required for inclusion of the influenza vaccine date on the patient's immunization record. By documenting influenza vaccine given elsewhere, clinical staff and providers could satisfy the BPA and provide data on influenza vaccine uptake needed to satisfy USNWR reporting.

If influenza vaccine was not ordered or not documented as given elsewhere, providers or clinical staff could choose from an alternate response. The alternate responses for clinical staff included history of anaphylaxis, vaccine not available, provider review, and patient not present. In addition to these options, provider alternate responses also included not eligible today, decline for season, and decline for today. If the BPA was not addressed during the encounter, it continued to display for subsequent encounters until satisfied.

Best Practice Alert Deployment

The outpatient influenza BPA was active during the 2014–2015, 2015–2016, and 2016–2017 influenza seasons. The dates of BPA activity varied by year depending on local influenza vaccine availability, regional and local influenza activity, and peak influenza activity; dates by study year were 9/25/14–2/4/15, 10/7/15–3/1/16, and 9/7/16–3/1/17. The period 9/25/13–2/4/14 was used as baseline data to compare with the BPA intervention periods and is referred to as the pre-BPA year. Prior to implementation in study year 2014–2015, clinical staff were required to attend a small group, 45-minute educational session that provided information regarding the importance of influenza vaccination and training on BPA usage and workflow. All clinical staff members and providers received an electronic communication with instructions on how to use the BPA each year.

Outcome Measures and Data Analysis

Through collaboration with a database architect, we obtained aggregate data reports based on our eligibility criteria from our EMR. We examined differences in influenza vaccine uptake in pediatric outpatient services overall and by clinical outpatient subgroup between study years and compared with the pre-BPA year. We calculated vaccine uptake as the number of influenza vaccines ordered per total number of eligible patients. Eligible patients included those aged 6 months and older who visited a pediatric outpatient service for all visit types, sick or well, during each study year (9/25/14–2/4/15, 10/7/15–3/1/16, and 9/7/16–3/1/17) and for the pre-BPA, or baseline, year (9/25/13–2/4/14); each patient was counted once for each study year. Patients who became age-eligible (≥6 months of age) during the study year were included. For the purpose of this assessment, for children younger than 9 years who received two influenza vaccine doses, only the first dose was included. In a secondary analysis, we excluded children with medical contraindications and included those who had received vaccines elsewhere. For the pre-BPA year, using the same definition of an eligible patient as the post-BPA year, we obtained the total number of encounters for eligible patients, including those encounters that preceded or occurred on the date of influenza vaccination.

To determine the distribution of influenza vaccine uptake throughout the influenza season within pediatric outpatient services, we calculated the number of vaccines ordered each week per total number of vaccines ordered for each study year. To determine the frequency of occurrences in which the BPA displayed but was not acknowledged (i.e., ignored) by clinical staff or providers in pediatric outpatient services, we calculated the number of occurrences in which the BPA was ignored per total number of occurrences in which the BPA displayed.

Lastly, in primary care practices only, we calculated the number of encounters in which the BPA displayed and an influenza vaccine was not ordered per total encounters in which the BPA displayed to assess the proportion of missed vaccination opportunities by visit type (sick vs. well-child), using the CDC Pink Book definition of missed opportunities. 19 For the pre-BPA year, missed influenza vaccination opportunities were calculated as the number of eligible encounters in which a vaccine was not ordered per total eligible encounters.

All proportions were calculated overall and by clinical outpatient subgroup with 95% confidence intervals (CIs). Chi-square test was used to analyze trends over time as well as to compare each study year to the pre-BPA year. Risk differences were calculated to compare the difference in influenza vaccinations for the pre-BPA year compared with subsequent study years. Statistical analysis was performed using Stata version 15.1 (Stata Corp., College Station, Texas, United States).

Results

During the pre-BPA year, 276,411 patients eligible for influenza vaccine visited a pediatric outpatient service ( Table 2 ). The total number of eligible patients increased each study year with 298,214 patients in 2014–2015, 328,510 patients in 2015–2016, and 389,418 patients in 2016–2017. Within each clinical outpatient subgroup, eligible patients also increased each study year. Influenza vaccine uptake for pediatric outpatient services overall declined from the pre-BPA year compared with subsequent study years ( p  < 0.001). Influenza vaccine uptake was 47.2% during the pre-BPA year and 46.1% in study year 2014–2015 (risk difference compared with pre-BPA year: −1.13 [95% CI: −1.39, −0.87]; p  < 0.001), 45.6% in study year 2015–2016 (risk difference: −1.61 [95% CI: −1.87, −1.36]; p  < 0.001), and 45.1% in study year 2016–2017 (risk difference: −2.08 [95% CI: −2.32, −1.85]; p  < 0.001). The number of patients excluded due to medical contraindications was 610 in 2014–2015, 899 in 2015–2016, and 885 in 2016–2017. The number of influenza vaccines given elsewhere was 4,691 in 2014–2015, 5,938 in 2015–2016, and 4,518 in 2016–2017. After accounting for these exclusions and inclusions, influenza vaccine uptake was 47.7, 47.5, and 46.3% for the study years compared with the pre-BPA year ( p  < 0.001).

Table 2. Proportion of influenza vaccines ordered (%) by study year, compared with pre-BPA year and over study period.

	Pediatric outpatient services, overall				Pediatric subspecialty services				Primary care practices				Health plan primary care practices
	N	%	95% CI	p -Value a	N	%	95% CI	p -Value a	N	%	95% CI	p -Value	N	%	95% CI	p -Value
2013–2014 (pre-BPA)	276,411	47.2	47.0, 47.4	Reference	68,942	9.1	8.9, 9.4	Reference	224,154	54.7	54.5, 54.9	Reference	3,041	51.0	49.2, 52.8	Reference
2014–2015	298,214	46.1	45.9, 46.2	<0.001	75,167	9.7	9.5, 9.9	0.001	238,481	52.9	52.7, 53.1	<0.001	7,436	53.2	52.0, 54.3	0.039
2015–2016	328,510	45.6	45.4, 45.7	<0.001	92,395	8.6	8.4, 8.8	<0.001	249,551	53.7	53.5, 53.9	<0.001	13,705	56.8	56.0, 57.6	<0.001
2016–2017	389,418	45.1	44.9, 45.2	<0.001	115,182	9.0	8.8, 9.1	0.224	285,156	54.4	54.2, 54.6	0.039	18,405	54.4	53.7, 55.1	<0.001
p -Value for trend b				<0.001				<0.001				0.501				0.003

Abbreviation: BPA, best practice alert.

^a p -Value for vaccination uptake for study year as compared with pre-BPA year.

^b p -Value for vaccination uptake over entire study period.

The proportion of influenza vaccines ordered during week 1 of BPA activity increased from pre-BPA year (8.47%; 95% CI: 8.32–8.63) to study year 2014–2015 (10.45; 95% CI: 10.29, 10.61) and study year 2015–2016 (11.12%; 95% CI: 10.96, 11.28; Fig. 2 ). However, during the final study year 2016–2017, this proportion decreased (4.79; 95% CI: 4.70, 4.89). Overall, following the initial weeks of BPA activity, there were no appreciable differences in the proportion of influenza vaccines ordered during the remaining weeks.

Fig. 2.

Distribution of influenza vaccines ordered by week. BPA, best practice alert.

For clinical staff, the proportion of BPA displays ignored increased each study year from 59.6% (95% CI: 59.4, 59.8) in study year 2014–2015 to 67.2% (95% CI: 67.1, 67.4) in study year 2015–2016 and 72.5% (95% CI: 72.4, 72.6) in study year 2016–2017. Among providers, the proportion of BPA displays ignored decreased each study year from 53.4% (95% CI: 53.2, 53.6) to 52.5% (95% CI: 52.3, 52.7) and 51.4% (95% CI: 51.3, 51.6) for study years 2014–2015, 2015–2016, and 2016–2017, respectively.

Lastly, within primary care practices, the BPA significantly reduced missed vaccination opportunities in both sick and well-child visits ( p  < 0.001, Table 3 ). The proportion of missed vaccination opportunities in sick visits was 86.8% (95% CI: 86.6, 86.9) during the pre-BPA year and 81% (95% CI: 80.8, 81.2), 79.8% (95% CI: 79.6, 80.0), and 82.7% (95% CI: 82.5, 82.9) during subsequent study years. The decrease in the proportion of missed vaccination opportunities was more pronounced in well-child visits during the first two study years, 2014–2015 (20.1%, 95% CI: 19.9, 20.3) and 2015–2016 (23.6%, 95% CI: 23.3, 23.8), compared with the pre-BPA year (34.1%, 95% CI: 33.9, 34.4); this decrease was not as pronounced for well-child visits during study year 2016–2017 (30.2%; 95% CI: 30.3, 30.7).

Table 3. Missed influenza vaccination opportunities at primary care practices by visit type.

	Sick child visits				Well-child visits
	N	%	95% CI	p -Value	N	%	95% CI	p -Value
2013–2014	192,600	86.8	86.6, 86.9	Reference	147,188	34.1	33.9, 34.4	Reference
2014–2015	160,946	81.0	80.8, 81.2	<0.001	119,617	20.1	19.9, 20.3	<0.001
2015–2016	165,930	79.8	79.6, 80.0	<0.001	132,459	23.6	23.3, 23.8	<0.001
2016–2017	212,554	82.7	82.5, 82.9	<0.001	168,313	30.2	30.3, 30.7	<0.001
p -Value for trend				<0.001				<0.001

Discussion

To our knowledge, this is the first analysis of the long-term effects of an influenza vaccine BPA. The goal of implementing the influenza BPA was to improve vaccine uptake and documentation throughout Texas Children's Hospital. Overall, the BPA did not meaningfully improve influenza vaccine uptake. In the health plan primary care practices, minor improvements in influenza vaccine uptake were noted. In comparison, influenza vaccine uptake in pediatric outpatient subspecialty services was notably low in all years and failed to improve. Beyond initial differences, BPA implementation did not meaningfully alter influenza vaccine ordering practices. Moreover, a decrease in the proportion of influenza vaccine ordered in the initial weeks of study year 2016–2017 may be attributable to the early implementation date. BPA implementation occurred nearly 3 weeks earlier during the 2016–2017 study year than all other study years and pre-BPA year. While clinical staff increasingly ignored the BPA over consecutive years, this trend did not hold true for providers. This finding differs from previous research by Ledwich et al who found improved vaccination rates in a nurse-driven BPA process. 20 Although uptake did not increase in primary care practices, missed vaccination opportunities initially decreased in both sick and well-child visits and may have demonstrated an early response to the influenza BPA; however, these changes were not sustained over time, particularly among sick visits.

Although our influenza BPA did not result in meaningful increases in influenza vaccine uptake, BPAs and other clinical decision support tools in other settings have demonstrated success in improving vaccine delivery. For example, adding an influenza vaccination status field to an outpatient whiteboard, a clinic-wide electronic display that reveals patient location, time in location, and processes to be completed have shown improvement in vaccination uptake and delivery. 21 Previous efforts to utilize an influenza clinical alert included notable differences that may have contributed to their success, such as focus on a targeted patient population, engagement and education of a smaller provider and staff group, and implementation over a shorter time period. 14 15 20 Experts recommend several best practices related to the development, implementation, and maintenance of clinical decision support tools that may result in improved performance. 22 23 24 In comparing our BPA to these best practices, we identified areas in which we adhered to the best practice recommendations; however, in several areas, we failed to follow best practices which may have led to our suboptimal outcomes. Importantly, we believe simple and efficient clinical decision support tools work most effectively. 24 While our influenza BPA development team worked diligently on the presentation and content of the BPA, we speculate that our BPA was overly complex. Because the same influenza BPA was implemented throughout Texas Children's Hospital and was associated with SOs for vaccine administration, clinical staff and providers were required to know which influenza vaccine to order based on the patient's age and available vaccine types. Vaccine types varied widely throughout Texas Children's Hospital, as some locations utilized vaccine provided by the Vaccines for Children program while others utilized vaccine purchased by the hospital. Providers were required to select route, dose, and vaccine type (i.e., nasal vs. intramuscular [IM], 0.25 mL IM vs. 0.5 mL IM, trivalent vs. quadrivalent). Providers who routinely give influenza vaccine may have found that the influenza BPA simplified and shortened the vaccine-ordering process; however, this finding may not have been true for many providers and clinical staff less familiar with the influenza vaccine and the influenza BPA. Unfortunately, at the time of influenza BPA implementation, our EMR did not have the capability to predetermine the correct vaccine order for the user. In addition, our influenza BPA offered multiple alternate responses if the vaccine was not given. These responses were included to obtain additional data for USNWR reporting purposes; however, the appearance of multiple alternate responses may have seemed cumbersome and overly complex to users. Ultimately, the lack of simplicity and usability may have resulted in the high ignore rates by some clinical staff. Further investigation is warranted to understand the differences in BPA ignore rates between provider and clinical staff.

In addition to simplicity and usability, it is recommended that BPAs be integrated into the existing clinical workflow. 22 23 24 While we intended to integrate our influenza BPA into the existing workflow, we inadvertently altered it by utilizing SOs. While SOs existed within Texas Children's Hospital for many years prior to BPA implementation, they were often underutilized and not universally adopted. This change to the preexisting workflow may have led to wariness or frustrations about using the BPA. Given the low influenza vaccine uptake among subspecialty services, we surmise that some subspecialty providers and clinical staff may not have prioritized influenza vaccination in their clinic workflow or updated the clinic workflow to accommodate influenza vaccine ordering and delivery. Additionally, to prevent workflow disruptions, our BPA was not designed as a “hard stop.” We speculate that once users realized that the BPA could be ignored, and that ignoring it was potentially faster than to respond to or satisfy the BPA, lack of a “hard-stop” may have further added to underutilization of the influenza BPA.

Finally, BPAs should be prospectively implemented with planned interval quality reviews, user feedback, and education. Our design team received anecdotal feedback from users and attempted to respond to this feedback prior to restarting the BPA each year. However, system-wide, semiannual or annual formal reviews of BPA performance and utilization using a quality improvement framework (i.e., plan, do, study, and act cycles) were not routinely performed. This analysis, performed 3 years post-BPA implementation, is the first formal analysis of the influenza BPA. Furthermore, in such a large, complex institution, it was difficult to ensure effective ongoing education regarding the influenza BPA, despite the initial education provided, given the routine rate of clinical staff and provider turnover. Given the importance of administering annual influenza vaccine, the influenza BPA was presumed to be necessary and effective and restarted annually with the arrival of influenza vaccine. This retrospective review has demonstrated the importance of caution in making such presumptions regarding widespread implementation of clinical decision support tools.

We observed several limitations. First, due to the large sample size, many of the differences noted between the pre-BPA year and all study years reached statistical significance; however, we believe many of these differences were not clinically relevant. Second, our intervention was system-wide and did not include a control group; thus, we are unable to evaluate any background changes that may have occurred simultaneously. Data from the National Immunization Survey regarding influenza coverage in Texas revealed that there was a 2.3% decrease in influenza vaccination coverage for the pediatric population from 2016–2017 to 2017–2018, which may have contributed to the lack of increase in vaccine coverage found in our evaluation. 25 Third, we analyzed de-identified aggregate data and did not evaluate patients' demographics or vaccine status for individual patients over the consecutive study years. Fourth, our study assessed one pediatric-integrated health care delivery system and may not be generalizable to other pediatric populations. Fifth, Texas Children's Hospital experienced increases in patient volumes throughout the study years in all locations; the effect of these increases on outcomes is unknown. Sixth, the impact of staff turnover on adherence to BPA workflow is also unknown. Seventh, our study measured vaccines ordered not doses received. It was beyond the scope of this project to validate that doses were received; however, we believe failure to administer an ordered vaccine would have been rare. Finally, system-wide clinical staff and provider feedback has not been systematically collected post-BPA implementation to help inform why the BPA was ineffective.

Conclusion

Implementation of an influenza BPA at Texas Children's Hospital, a large pediatric integrated health care delivery system, including primary care and subspecialty services, did not meaningfully increase influenza vaccine uptake. Targeting a specific pediatric subpopulation (i.e., clinics serving patients in the USNWR target groups) may have increased engagement among providers. In addition, following best practices may have resulted in a BPA with more favorable outcomes.

Clinical Relevance Statement

Widespread implementation of a BPA for influenza vaccine did not meaningfully increase influenza vaccine uptake, suggesting that further development of clinical decision support tools is needed to increase the effectiveness of these interventions for vaccine uptake. Adherence to best practices may have resulted in a more effective BPA which could have improved outcomes.

Multiple Choice Questions

1. Which best describes individuals recommended to receive an annual influenza vaccine?

   1. Individuals with high-risk medical conditions.

   2. Children <18 years of age with high-risk medical conditions.

   3. Pregnant women.

   4. All persons ≥6 months of age.

   Correct Answer: The correct answer is option d.

2. Characteristics of an effective clinical decision support tool include which of the following?

   1. Focus on a large, diverse patient and provider population.

   2. Use of visually pleasing design.

   3. Allow expansive response options to satisfy the BPA.

   4. Incorporate user feedback at project intervals.

   Correct Answer: The correct answer is option d.