Effect of Provider Prompts on Adolescent Immunization Rates: A Randomized Trial

Abstract

Objective:

Adolescent immunization rates are suboptimal. Experts recommend provider prompts at health care visits to improve rates. We assessed the impact of either electronic health record (EHR) or nurse- or staff-initiated provider prompts on adolescent immunization rates.

Methods:

We conducted a randomized controlled trial, allocating practices in 1 of 2 practice-based research networks (PBRN) to provider prompts or standard-of-care control. Ten primary care practices participated, 5 intervention and 5 controls, each matched in pairs on urban, suburban, or rural location and practice type (pediatric or family medicine), from a PBRN in Greater Rochester, New York (GR-PBRN); and 12 practices, 6 intervention, 6 controls, similarly matched, from a national pediatric continuity clinic PBRN (CORNET). The study period was 1 year per practice, ranging from June 2011 to January 2013. Study participants were adolescents 11 to 17 years attending these 22 practices; random sample of chart reviews per practice for baseline and postintervention year to assess immunization rates (n = 7,040 total chart reviews for adolescents with >1 visit in a period). The intervention was an EHR prompt (4 GR-PBRN and 5 CORNET practice pairs) (alert) that appeared on providers’ computer screens at all office visits, indicating the specific immunizations that adolescents were recommended to receive. Staff prompts (1 GR-PBRN pair and 1 CORNET pair) in the form of a reminder sheet was placed on the provider’s desk in the exam room indicating the vaccines due. We compared immunization rates, stratified by PBRN, for routine vaccines (meningococcus, pertussis, human papillomavirus, influenza) at study beginning and end.

Results:

Intervention and control practices within each PBRN were similar at baseline for demographics and immunization rates. Immunization rates at the study end for adolescents who were behind on immunizations at study initiation were not significantly different for intervention versus control practices for any vaccine or combination of vaccines. Results were similar for each PBRN and also when only EHR-based prompts was assessed. For example, at study end, 3-dose human papillomavirus vaccination rates for GR-PBRN intervention versus control practices were 51% versus 53% (adjusted odds ratio 0.96; 95% confidence interval 0.64–1.34); CORNET intervention versus control rates were 50% versus 42% (adjusted odds ratio 1.06; 95% confidence interval 0.68–1.88).

Conclusions and Relevance::

In both a local and national setting, provider prompts failed to improve adolescent immunization rates. More rigorous practice-based changes are needed.

Since 2005, adolescent immunization delivery has become increasingly important but also more complex, the result of age-specific recommendations of vaccines to prevent tetanus, diphtheria, pertussis (Tdap), meningococcal (MCV4), and human papillomavirus (HPV) infections and universal recommendation of influenza vaccination for all persons, including teens.^1,2 Unfortunately, despite substantial research, publicity, and efforts by experts to optimize adolescent immunization delivery, immunization rates are suboptimal.^2,3 In 2012, 85% of adolescents aged 13 to 17 years had received Tdap, 74% had received MCV4, and ≈40% had received an influenza vaccination⁴; 54% of girls (and 21% of boys) aged 13 to 17 years had received >1 HPV vaccine, while 33% of girls and 7% of boys had received 3 doses.² Finally, some disparities exist, with minority adolescents having lower completion rates of HPV vaccine and lower rates of influenza vaccination.^5–8

Experts, including the Task Force on Community Preventive Services,^9,10 recommend that primary care practices use one or more strategies to optimize adolescent immunization delivery. One recommended strategy is health care provider prompts to reduce missed opportunities for vaccinations.^11,12 Provider prompts, or alerts, are produced by nurse/staff, or by the electronic health record (EHR) at the time of patient visits to remind health care providers to administer age-appropriate vaccinations. Few primary care practices use provider prompts for adolescent immunizations as a result of implementation barriers such as complexity and cost.¹³ Further, few studies have evaluated provider prompts’ effectiveness among adolescent populations, and those that have reported mixed results. One study showed no benefit of prompts for influenza vaccinations among children and adolescents with asthma,¹⁴ while a recent study in a hospital-based primary care system using a single EHR showed benefit for HPV vaccination initiation but not subsequent doses.¹⁵ Thus, although some experts recommend provider prompts for a variety of preventive measures,^16–20 little evidence supports their effectiveness for adolescent immunizations. Of note, earlier studies demonstrated that nurse/staff prompts had variable success in reducing missed opportunities for childhood vaccinations.^{11,12,21–24}

We conducted a randomized controlled trial in 2 practice-based research networks (PBRNs), an upstate New York network and a national network of pediatric continuity clinics, to evaluate the impact of provider prompts on adolescent immunization rates. We used community-based participatory research methods for practitioners to reach a consensus on the intervention to be studied, which converged on provider prompts, and then conducted the intervention in both PBRNs. We hypothesized that provider prompts would improve adolescent immunization rates.

Methods

Setting

The study was based in both a local and a national PBRN. The Greater Rochester PBRN (GR-PBRN)²⁵ consists of 85 primary care practices, including 44 pediatric and 14 family medicine practices serving >80% of all children in the Monroe County, New York, region, which has a population of 750,000. The national Continuity Clinic Research Network (CORNET) consists of 73 pediatric continuity clinics in 36 states serving over 683,000 children and adolescents; many are large hospital-based continuity clinics.

Study Design

Using concepts of diffusion theory outlined by Rogers,²⁶ we performed a 3-part mixed-methods study²⁷ to 1) determine a consensus-driven, practice-based strategy to improve adolescent immunization rates, 2) test the strategy in 2 PBRNs, and 3) evaluate practitioner perceptions of the interventions.

Selection of a Practice-based Intervention

We first conducted a mixed-methods mailed/online survey of primary care practices plus qualitative key informant interviews of a subset of practices in the 2 PBRNs to: 1) ascertain which of the recommended adolescent immunization strategies practitioners were using and would be interested in adopting (reported previously²⁸); 2) develop consensus about which specific intervention to evaluate on the basis of practitioner perception of the intervention’s feasibility, effectiveness, and sustainability; and 3) identify practices that were interested in participating in intervention evaluation. Two-thirds of GR-PBRN and three-quarters of CORNET practices selected provider immunization prompts delivered either by nurse/staff during patient visits, or delivered by EHR for study.

Evaluation of Provider Prompts

We conducted a randomized controlled trial, stratified by PBRN, to test the impact of provider prompts on increasing adolescent immunization rates. Intervention practices within each PBRN were matched with control practices in pairs by suburban, urban, or rural status and practice type (pediatrics or family medicine). The 12-month randomized controlled trial spanned June 6, 2011, to June 5, 2012 (GR-PBRN), and September 20, 2011, to January 30, 2013 (CORNET; intervention/control practice pairs had staggered starts over a 4-month period, but for each practice within an intervention/control pair of CORNET practices the study time period was the same).

Assessment of Practitioner Perceptions of the Intervention

After the intervention, we conducted a qualitative phone interview of one practitioner from all intervention practices to assess perceptions of feasibility, acceptability, and sustainability of the provider prompts. We used constructs from diffusion theory as the conceptual framework.²⁶ Two authors [SH, PV (GR-PBRN); SH, ND (CORNET)] performed, documented, and analyzed interviews.

The research subjects review board of the University of Rochester approved all 3 study components; the 14 CORNET sites required additional institutional review board approval. Parent and patient informed consents were not required because the intervention involved a practice-based intervention recommended for general use.⁹

Participants

Primary Care Practices and Randomization

Fourteen GR-PBRN and 15 CORNET practices agreed, before randomization, to participate in the randomized controlled trial intervention (Figure). Our power calculation called for 6 pairs (12 sites) per PBRN to be able to detect an increase of 10 percentage points (40% to 50%) in immunization rates with 80% power, an average of 160 patients per practice, a 2-tailed alpha of 5% and an intraclass correlation (ICC) of 0.01 (or 13 percentage points with ICC = 0.02); an ICC of 0.01 to 0.02 is typical for primary care trials. Within the GR-PBRN, we created practice pairs: 4 suburban pediatric, 1 rural family medicine, and 1 urban community health center. The community health center pair was excluded because the intervention practice could not implement the intervention, leaving 5 practice pairs. Within CORNET we created 5 urban pediatric pairs and 1 rural pediatric pair of practices. Using Stata 12.1, one author (AB) randomly assigned practices within each PBRN and practice pair to be an intervention or a standard of care control practice. No practice used provider prompts at baseline for adolescent immunizations.

Figure.

Flow diagram showing intervention (prompt) and control practices and subjects for chart reviews. Peds indicates pediatric practice; FM, family medicine practice; and CHC, community health center. *The CHC practice pair was excluded because the intervention practice was unable to perform prompts. °A random sample of adolescents who made a visit during the 12-month baseline or 12-month intervention period was selected for chart review; the same number of medical charts was reviewed for each time period (ie, n = 3,520 for the baseline period and n = 3,520 for the intervention period).

Subjects

The target population was all adolescents aged 11 to 17 years who were enrolled in a participating practice during the year before the intervention. For the qualitative postintervention survey, subjects were the practitioners.

Study Interventions

Identifying Adolescents Eligible for Immunizations or Preventive Visits

Adolescents were considered eligible for Tdap, MCV4, or HPV vaccine based on 2010 Advisory Committee on Immunization Practices guidelines:^5,29 Tdap if no prior Tdap or Td vaccination within 2 years (most practices used this time frame between Tdap and Td vaccines); MCV4 if no prior vaccination; HPV vaccine for girls [first HPV vaccination (HPV₁) if none prior, HPV₂ if >60 days from HPV₁, and HPV₃ if >24 weeks from HPV₁ and >12 weeks from HPV₂], and influenza vaccine if none received that season. We did not include catch-up vaccinations (eg, varicella vaccine³⁰).

EHR-based Prompts

We worked with EHR personnel to program EHRs to display a provider prompt (alert) on the initial screen that health care providers viewed upon opening each patient’s electronic medical chart. Although EHR prompts varied slightly in appearance, screen location, and method to document refusals, all prompts used the same algorithm and displayed a list of vaccines due at that visit. Prompts did not generally show prior vaccinations and did not include standing orders. We worked with 9 EHR vendors, 4 for the GR-PBRN and 5 for CORNET practices. EHRs were programmed to display prompts at all visits, not just preventive visits. In 3 of 4 EHR intervention practices from the GR-PBRN, the EHR prompt was only used for adolescent (not child) immunizations; in the remaining EHR practice, immunization prompts were activated for all ages, but results were analyzed only for adolescent immunizations. In CORNET’s EHR intervention practices, immunization prompts were turned on for all ages. Two CORNET intervention practices transitioned to EHRs during the study and performed nurse/staff prompts for several months until EHR prompts were implemented; these practices were allocated to the EHR group. For each intervention practice, we provided one or two 1-hour educational sessions to inform providers about EHR-based prompts. On the basis of participating practitioners’ preferences and practice patterns, providers could elect to follow or ignore prompts.

Nurse/Staff Prompts (1 Practice per PBRN)

At 2 intervention practices, practitioners preferred nurse/staff prompts because they lacked EHRs that could be programmed to deliver prompts. For these practices, we delivered 1 or 2 educational sessions to physicians and nurses/staff. We described the importance of immunizations and provided a nurse/staff protocol to: 1) review immunization records for every adolescent at each visit; 2) list immunizations due at each visit onto a sheet; and 3) display vaccine information statement forms.

Follow-up Telephone Calls

We conducted monthly telephone calls with intervention practices to assess progress, address concerns, and engage practitioners in group discussions and problem solving (eg, strategies during busy periods). We encouraged practices to select a small number of charts to review for missed opportunities as a process metric; of note, few practices complied as a result of limited time and staff to perform chart reviews. We used this “low-touch” approach on the basis of practitioner preferences and to optimize sustainability.

Controls

Adolescents in control practices received standard of care, which did not include prompts.

Postintervention Survey

For each intervention practice, a physician leader and an office manager were interviewed by phone by one investigator (SH), using a predefined outline of questions that assessed feasibility, acceptability, and sustainability of prompts. Respondents were sent the questions before the interview and were also asked open-ended questions to elaborate upon their answers.

Outcome and Process Measures

Patient Information, Missed Opportunities, and Immunization Rates

For each practice, we obtained the practice denominator at 2 points in time: at the start of the intervention to assess the 12-month baseline period, and 1 year later to assess the 12-month intervention period. The denominator for the baseline period was defined as adolescents who had >1 visit of any kind within the 12-month period and were 11 to 17 years at the start of the time period. The denominator for the intervention period was adolescents with >1 visit during the intervention period, regardless of visits during the baseline period. We then randomly selected 160 adolescents from each practice for standardized chart reviews, at both the start and end. Our research assistants reviewed GR-PBRN charts; for CORNET practices across 11 states, a provider or nurse champion at each site was trained to review the charts. All chart reviews were recorded on paper and sent to Rochester-based research staff for double data entry and analysis.

Measures

Main outcomes were receipt of recommended Tdap, MCV4, HPV_1,2,3, and influenza vaccines on or after the 11th birthday, as well as combinations of these vaccines. A secondary immunization outcome was time to vaccination in days since the 11th birthday. An additional outcome was missed opportunities, defined as visits in which an adolescent was due for a vaccination but did not receive the vaccine. Missed opportunities were further coded as refusals, deferred, other, or no notation in the chart.

Answers to the postintervention qualitative phone interviews were reviewed by 3 authors (SH, PV, CA), with interpretive differences resolved through discussion.

Statistical Analyses

The primary independent variable was group assignment (intervention vs control). We also performed an analysis limited to EHR practice pairs. All analyses were stratified by PBRN. We first used descriptive statistics to compare baseline provider/patient factors and immunization rates for intervention versus control practices. Because we had randomized practice pairs (intervention/control), we accounted for multiple levels of nonindependence and clustering (by intervention/control pairs, and then by practice). We used a multilevel mixed-effect logistic regression model with robust standard errors, with patients nested within practices, and covariates for pair assignment, study time period, intervention assignment, and an interaction between time and intervention assignment. The interaction term portrays how much more (or less) the intervention group’s odds of immunization increased compared to the control group’s odds of immunization. For time to event analyses, we used a stratified Cox model, stratifying on paired practices, and robust standard errors, with just the study time period, intervention assignment, and an interaction between the 2, within the model. We estimated hazard ratios for receipt of vaccinations. We used StataMP software, version 13.1.³¹

Results

Randomization and Baseline Characteristics

The Figure shows randomization and assessment processes. For the GR-PBRN, of 88 practices that responded to the survey, 43 practices expressed interest in being randomized to an intervention, and 31 were excluded before randomization (>50 miles from Rochester, New York; <500 adolescents; already using EHR prompts; or EHRs not programmable for prompts and not willing to use nurse prompts). The 12 remaining practices were eligible for randomization and were stratified by urban, suburban, or rural residents and pediatric or family medicine practices. They were subsequently randomly allocated within pairs to intervention or control groups. For CORNET, of 78 clinics that responded to the survey, 41 expressed interest in being randomized to an intervention, 12 were excluded for ineligibility (<500 adolescents, already using EHR prompts), 29 were eligible, and we selected 12 (6 pairs) that were geographically disparate for randomization. Table 1 shows practice and patient characteristics of intervention versus control groups stratified by PBRN. In the GR-PBRN group, intervention practices had a slightly higher percentage of publicly insured adolescents; in the CORNET group, intervention practices had a slightly higher percentage of white adolescent patients.

Table 1.

Baseline Practice and Patient Characteristics and Preventive Care Visit Rates by Randomization Group

Characteristic	GR-PBRN			CORNET
	Intervention	Control	P*	Intervention	Control	P*
Practice Characteristics
No. of practices	5	5		6	6
No. of pediatrics/family medicine practices	4/1	4/1	NA	6/0	6/0	NA
No. of practices by location			NA			NA
Urban	0	0		6	6
Suburban	4	4		0	0
Rural	1	1		0	0
Average size, mean (SD)†
All ages	7,125 (5,808)	9,125 (2,462)	.63	8,029 (4,249)	13,375 (8,499)	.59
11–18 y	2,600 (1,980)	4,450 (5,020)	.55	1,815 (747)	2,500 (1,870)	.87
Adolescent Patient Characteristics
From Baseline Chart Review
No. of patients	800	800		960	960
Patients by insurance type, n (%)			<.001			.44
Private	117 (15%)	159 (20)		589 (61)	605 (63)
Public	669 (84%)	605 (76)		334 (35)	279 (29)
None/military	14 (2%)	36 (5)		37 (4)	76 (8)
Female patients, n (%)	393 (49%)	399 (50)	.85	473 (49)	468 (49)	.86
Race, n (%)‡	NA	NA	NA			<.01
White, non-Hispanic				340 (35)	191 (20)
Black, non-Hispanic				366 (38)	345 (36)
Hispanic				105 (11)	186 (19)
Other, non-Hispanic				60 (6)	39 (4)
Missing				89 (9)	199 (21)
Patients with preventive care visit during study period, n (%)	626 (78)	639 (80)	.50	808 (84)	694 (72)	.11

GR-PBRN indicates Greater Rochester Practice-Based Research Network; CORNET, Continuity Clinic Research Network; and NA, not applicable.

^*P values from conditional logistics regression, conditioning on matched practices.

^†P value from paired t test.

^‡Race was missing in >75% of cases from the GR-PBRN (so is not presented here) and was also missing in 15% of cases from CORNET (recoded as “missing”).

Immunization Rates

Table 2 shows immunization rates at baseline and at the end of the study period, stratified by PBRN. Baseline immunization rates closely mirrored national rates.³² The adjusted odds ratio represents the ratio of the odds (postintervention period vs baseline immunization rates) for intervention versus control practices—that is, the change in immunization rates due to the intervention. Table 2 reveals 3 key findings. First, baseline immunization rates were high for 2 vaccines (Tdap and MCV4) and quite low for HPV and influenza vaccinations. Second, vaccination rates increased slightly from baseline to end of study period for both control and intervention practices (ie, secular trends). Third, the intervention did not appear to increase immunization rates in either PBRN for any of the individual vaccines or for combinations. That is, adjusted odds ratios were not significantly greater than 1.0.

Table 2.

Immunization Rates at Baseline and End of Study Period by Study Group Stratified by PBRN^*

Immunization	Baseline		End of Study Period		aOR† (95% CI)	P
	Intervention, n (%)	Control, n (%)	Intervention, n (%)	Control, n (%)
GR-PBRN
Tdap	760 (95)	746 (93)	783 (98)	767 (96)	1.44 (0.82–2.56)	.21
MCV4	709 (89)	701 (88)	741 (93)	728 (91)	1.15 (0.64–2.05)	.65
HPV1‡	251 (64)	268 (67)	253 (64)	286 (68)	0.92 (0.60–1.40)	.70
HPV2	229 (58)	249 (62)	234 (59)	261 (62)	1.01 (0.57–1.77)	.98
HPV3	199 (51)	205 (51)	201 (51)	221 (53)	0.93 (0.64–1.34)	.70
MCV4 and Tdap	678 (85)	665 (83)	734 (92)	704 (88)	1.39 (0.77–2.51)	.28
Seasonal influenza	252 (32)	243 (30)	279 (35)	282 (35)	0.93 (0.69–1.25)	.78
CORNET
Tdap	877 (91)	811 (84)	904 (94)	842 (88)	1.16 (0.68–1.99)	.59
MCV4	882 (92)	803 (84)	908 (95)	844 (88)	1.08 (0.82–1.41)	.59
HPV1‡	363 (77)	317 (68)	392 (82)	358 (75)	0.96 (0.59–1.56)	.87
HPV2	313 (66)	265 (57)	338 (71)	286 (60)	1.06 (0.68–1.66)	.79
HPV3	229 (48)	207 (44)	237 (50)	200 (42)	1.13 (0.68–1.88)	.64
MCV4 and Tdap	839 (87)	749 (78)	889 (93)	793 (83)	1.35 (0.92–1.97)	.13
Seasonal influenza	472 (49)	410 (43)	457 (48)	421 (44)	0.89 (0.69–1.16)	.54

PBRN indicates practice-based research network; aOR, adjusted odds ratio; CI, confidence interval; GR-PBRN, Greater Rochester Practice-Based Research Network; CORNET, Continuity Clinic Research Network; Tdap, tetanus toxoid, reduced diphtheria toxoid, and acellular pertussis vaccine; MCV4, meningococcal vaccine; and HPV, human papillomavirus vaccine (HPV₁, first vaccination in series, etc).

^*Based on 160 randomly selected chart reviews per practice at baseline and again at the end of the study. The GR-PBRN included 5 practice pairs; the CORNET PBRN included 6 practice pairs. Denominators are n = 800 (GR-PBRN) or n = 960 (CORNET) except for HPV vaccine, where the denominators are for female subjects only. Denominators of female subjects for HPV calculations are: GR-PBRN–baseline (intervention = 393, control = 399) and study end (intervention = 397, control = 420); CORNET–baseline (intervention = 473, control = 468) and study end (intervention = 478, control = 476).

^†aOR is odds of change in outcomes (immunization rates) for intervention versus odds of change in outcomes (immunization rates) for control practices, expressed as odds ratio. Odds ratios come from robust multilevel mixed-effect logistic regression models with covariates pair assignment, study time period, intervention assignment, and an interaction between time and intervention assignment.

^‡HPV assessed for female subjects only.

We also used survival analysis to assess time to HPV vaccination as a more sensitive measure of the intervention effect (Table 3). In both PBRNs, the intervention had no effect on the time from start of the intervention period to receiving any of the HPV vaccine doses or to being fully vaccinated, with one exception: in the GR-PBRN group, intervention subjects were 124% more likely to receive HPV₂ at any given time than adolescents from control practices (median of 54 days vs 79 days after HPV₁).

Table 3.

Hazard Ratios^π and 95% CI for Intervention vs. Control Female Subjects–Time From Eligibility for HPV (or Study Start for All Vaccinations Combined) to Receiving Missing HPV Vaccine Doses (or Receiving All 3 Recommended Doses)^*

	HR‡
Study Group	HPV1*	HPV2*	HPV3*	3 HPV Vaccines†
GR-PBRN	0.86 (0.52–1.43)	2.24 (1.21–4.16)	1.25 (0.61–2.53)	0.94 (0.60–1.49)
CORNET	0.85 (0.57–1.27)	1.24 (0.69–2.23)	1.32 (0.66–2.64)	1.21 (0.83–1.77)

HPV indicates human papillomavirus vaccine (HPV₁, first vaccination in series, etc); HR, hazard ratio; GR-PBRN, Greater Rochester Practice-Based Research Network; and CORNET, Continuity Clinic Research Network.

^*Time from eligibility to receipt of HPV vaccine dose or doses. Note that all female subjects were eligible for HPV₁ except for those who already had the first dose by the start of the study. Analyses for HPV₂ and HPV₃ receipt accounted for the recommended spacing intervals between doses.

^†Time from start of study period to being fully vaccinated with 3 doses of HPV vaccine.

^‡The hazard function is the chance that a subject will have an event at a specific time, given that the individual did not already have the event. The HR is the interaction term. For example, a HR of 2.24 (for HPV₂) means that the intervention group’s HR increased 124% over study periods compared to the control group’s HR. The second to the last column assesses the interval between HPV2 and HPV3 and is limited to adolescents who received HPV2. The last column assesses the time from the start of the intervention to completion of HPV series.

Missed Opportunities

Provider prompts were designed to improve immunization rates by reducing missed opportunities for immunizations. Table 4 shows the impact of the intervention on visits with missed opportunities (ie, visits as the unit of analysis) and on the percentage of adolescents who had a missed opportunity. With one exception, the intervention did not improve missed opportunity rates. Within the GR-PBRN, the intervention reduced missed opportunities for HPV vaccination by 18% (adjusted incident risk ratio 0.82; 95% confidence interval 0.72–0.94). We also examined simultaneous missed opportunities (ie, visits during which some but not all eligible vaccines were administered), and we found no differences between intervention and control groups (not shown in Table 4).

Table 4.

Missed Opportunities and Refusal Rates by Vaccine for Intervention Versus Control Practices During Baseline and Intervention Period by PBRN

	Baseline		Intervention Period
Characteristic	Intervention, n (%)	Control, n (%)	Intervention, n (%)	Control, n (%)	Incident Risk Ratio (95% CI)	P
GR-PBRN
Visits with missed opportunities during the intervention year*
HPV (girls)	184/552 (33)	166/528 (31)	165/606 (27)	173/567 (31)	0.82 (0.72–0.94)	.005
Influenza	194/527 (37)	179/514 (35)	192/538 (36)	188/516 (36)	0.92 (0.79–1.06)	.26
Any vaccine	552/584 (95)	535/568 (94)	598/659 (91)	558/616 (91)	1.01 (0.97–1.05)	.52
Patients with any missed opportunity during the intervention year	779 (98)	764 (96)	765 (96)	752 (94)	0.84 (0.41–1.72)	.64
Patients who refused any vaccine during the year†	51 (6)	59 (7)	121 (15)	91 (11)	1.64 (0.79–3.41)	.18
CORNET
Visits with missed opportunities during the intervention year*
HPV (girls)	163/632 (26)	215/749 (29)	131/472 (28)	172/577 (30)	1.00 (0.87–1.15)	.98
Influenza	218/549 (40)	273/707 (39)	174/415 (42)	211/556 (38)	1.00 (0.80–1.24)	.99
Any vaccine	556/678 (82)	753/821 (92)	388/503 (77)	591/634 (93)	0.99 (0.93–1.05)	.69
Patients with any missed opportunity during the intervention year	833 (87)	883 (93)	699 (78)	801 (87)	0.92 (0.51–1.66)	.77
Patients who refused any vaccine during the intervention year†	128 (13)	40 (4)	134 (14)	41 (4)	1.03 (0.36–2.97)	.96

PBRN indicates practice-based research network; CI, confidence interval; GR-PBRN, Greater Rochester Practice-Based Research Network; CORNET, Continuity Clinic Research Network; Tdap, tetanus toxoid, reduced diphtheria toxoid, and acellular pertussis vaccine; MCV4, meningococcal vaccine; and HPV, human papillomavirus vaccine.

^*Numerator is number of visits that involved a missed opportunity; denominator, total number of visits during the year when adolescents were eligible for a vaccine. The adjusted incidence-rate ratio is shown.

^†Refusal for any vaccine (MCV4, Tdap, HPV, or influenza).

Monthly Telephone Calls

Many practices found it challenging to participate on the monthly calls. For the GR-PBRN, we conducted 9 calls over the 12-month study period, and the participation rate among the 5 intervention practices was as follows: 5 sites represented (4 calls), 3 sites (4 calls), and 2 sites (1 call). For the CORNET PBRN (6 sites) we conducted 12 calls over the 12-month study period, and the participation rate was as follows: 6 sites (1 call), 5 sites (2 calls), 4 sites (1 call), 3 sites (4 calls), 2 sites (3 calls), and 1 site (1 call).

Refusals

Because provider prompts might have caused more discussion of vaccinations but also more parental refusals, we assessed parental refusal rates by medical record review (Table 4). Parent refusals were not different in intervention versus control groups.

Practitioner Perception of Intervention

Before analyzing data on intervention impact, we interviewed providers from the 11 intervention practices to assess perceptions of the intervention’s impact. Interestingly, providers from all 11 practices believed prompts were effective in reducing missed opportunities and improving immunization rates, and nearly all respondents wished to continue using prompts. However, most respondents also stated that providers ignored prompts when practices were particularly busy. Respondents reported varied practical barriers to prompts, including prompt errors (particularly early on for EHR practices as a result of incomplete vaccine records), added time for documenting refusals, time constraints for nurses for nurse-prompt practices, and lack of complete intervention acceptance by all practice members, including physicians and staff.

Discussion

In this randomized controlled trial conducted in 2 PBRNs, we found that provider prompts at office visits failed to improve adolescent immunization rates, despite the fact that practitioners selected provider prompts as the strategy most likely to be effective, feasible, and sustainable to raise adolescent immunization rates; that the prompts were generally performed by the EHR, ensuring that they were delivered; and that on regular conference calls and a postintervention phone interview, virtually all providers believed the prompts were effective in reducing missed opportunities and improving adolescent immunization rates.

Provider Prompts and Adolescent Immunizations

In 1996, we published results of a randomized trial in 2 urban practices to reduce missed opportunities for childhood vaccinations by using nurse/staff-generated paper prompts that listed immunizations needed at the time of office visits.²¹ The intervention reduced missed opportunities slightly, but not enough to raise immunization rates. The major barrier was that nurses/staff failed to place the prompts consistently on the paper medical charts. Over the past 15 years, the emergence of EHRs has revolutionized primary care,³³ and the concept of provider prompts/alerts has emerged to facilitate delivery of recommended services such as immunizations at the time of visits.^14–20,34 Many EHR systems are developing such alerts. Unfortunately, our current study, conducted in 21 practices across 2 PBRNs, had a similar outcome to the 1996 study: prompts for adolescent immunizations did not improve immunization rates, even within the EHR group. Although in our prior study the barrier was the delivery of the prompts, the barrier in the current study was that providers did not act on prompts sufficiently to increase immunization rates.

Improving Effectiveness of Provider Prompts

Provider prompts should represent low-hanging fruit for improving quality of care because they capitalize on patients already being in providers’ offices and because prompts can be programmed to reflect guideline-based care.^{15,16,18–20} A recent report showed that if missed opportunities for HPV vaccination were eliminated, HPV vaccination rates could rise to >90%.⁸ Another study of children with asthma found that if missed opportunities for influenza vaccination were eliminated, influenza vaccination rates would rise substantially.³⁵ The key to provider prompts is to ensure that providers adhere to them. Unfortunately, other studies have also found that prompts can fail to generate desired improvement in care as a result of providers not believing in the accuracy of the prompts, not heeding the prompts, forgetting the prompts, or feeling “prompt fatigue.”¹⁸ In our study, we did not notice any time trends in missed opportunities, suggesting that prompt fatigue may have been less important than providers simply ignoring prompts.

Because EHRs are increasingly implementing prompts, it is important to evaluate and improve their effectiveness in primary care settings. One potential strategy is a hard stop, which requires health care providers to act on prompts before being able to continue further use of the EHR. However, we suspect that few health systems are willing to incorporate hard stops for preventive prompts in order to retain hard stops for critical safety concerns and not hinder visit flow; indeed, the providers in our study specifically requested soft alerts that allowed providers to ignore prompts. A second potential strategy is to incorporate multiple prompts within the EHR pathway to alert nurses and providers multiple times (and perhaps in various ways) about the recommended preventive services. However, multiple prompts could become irritating to providers. A third potential strategy is to incorporate provider–patient communication strategies coupled with provider prompts in order to overcome parental (or provider) vaccine hesitancy. A fourth potential strategy is to incorporate quality-improvement based audit feedback and practice-improvement methods to make providers aware of missed opportunities for immunization. A recent study by Fiks et al¹⁵ noted that a clinician decision-support provider prompt resulted in an increase in HPV₁ vaccination rates (though no increase in HPV_2–3 rates) and shorter time to complete the HPV vaccine series in the prompt versus control group. The Fiks et al study was based in a single hospital-based network of practices all using the same EHR, and their intervention also included quarterly individual and practice-level performance feedback reports about missed opportunities and immunization rates. For our study, we elected to ask practices to self-monitor their performance, and we encouraged, but did not require, practices to self-audit charts each month to assess missed opportunities, thinking that this method was more scalable. Unfortunately, few practices complied with these self-audits. A fifth potential strategy is to use incentives, such as pay for performance, to increase adherence to immunization prompts; however, to our knowledge, no study has evaluated such a combined strategy. It is possible that multiple practice-improvement methods are needed, in addition to provider prompts, to help practitioners respond to immunization prompts.

Finally, multiple strategies beyond a successful prompt-based intervention are needed to improve adolescent immunization rates.^10,36 Although many of these strategies have had only limited field testing with respect to adolescent immunization delivery, they include patient reminder/recall, standing orders, audit feedback, and improved provider communication.

Strengths and Limitations

Study strengths include the use of a mixed methods study that incorporated qualitative interviews and a rigorous randomized controlled clinical trial, 2 independent PBRNs for generalizability, adequate sample size to detect even small effects of prompts, and a real-world intervention. Limitations include the limited number of practices (n = 24 practices before randomization) with an inability to control for some practice-based factors that might have affected missed opportunities or vaccination rates, the loss of one practice pair from the GR-PBRN, an inability to determine precisely the degree to which prompts resulted in increased discussions about immunizations, and high baseline meningococcal and Tdap immunization rates. Importantly, many practices had recently converted to EHRs, and it is possible that the overwhelming impact of using EHRs dampened immunization prompts’ effects. Further, in most cases the alerts were not modifiable; they simply appeared as standard prompts on the screen. Finally, we were unable to measure provider discussions with patients beyond assessing parent refusals or requests for delaying vaccinations. Nevertheless, in our study, refusal rates were similar for intervention and control practices, so we do not believe that parent refusals contributed substantially to the lack of benefit of the immunization prompts.

Conclusions

Although provider prompts are recommended to improve immunization rates, in this study performed in both a local and national PBRN, provider prompts failed to improve adolescent immunization rates and generally failed to reduce missed opportunities for immunization. More rigorous practice-based changes are needed to improve rates.