Centralized Reminder/Recall to Increase Influenza Vaccination Rates: A Two-State Pragmatic Randomized Trial

Abstract

Objective:

Centralized reminder/recall (C-R/R) by health departments using Immunization Information Systems (IIS) is more effective and cost-effective than practice-based approaches for increasing childhood vaccines but has not been studied for influenza vaccination. We assessed effectiveness and cost of C-R/R for increasing childhood influenza vaccination compared with usual care.

Methods:

Within Colorado (CO) and New York (NY), random samples of primary care practices (pediatric, family medicine, health center) were selected proportionate to where children are served-- 65 practices (N=54,353 children) in CO; 101 practices (N=65,777) in NY. We conducted 4-arm RCTs per state (1, 2 or 3 autodial reminders versus usual care), with randomization at the patient level within practices from 10/2016–1/2017.

Results:

In CO, the maximum absolute difference in receipt of ≥1 influenza vaccine was 1.7% between the 2 R/R group and control [adjusted risk ratio (ARR) of 1.06 (1.01, 1.10)]; other R/R arms did not differ significantly. In NY, ARRs for the study arms versus control varied from 1.05 (1.01, 1.10) for 3 R/R to 1.06 (1.01, 1.11) for 1–2 R/R groups and maximum absolute increase in vaccination was 0.6%. In time-to-event analyses, study arm was a significant predictor of vaccination in CO (p=.001) but not in NY. Costs/child randomized to one message were $.17 in CO and $.23 in NY.

Conclusions:

C-R/R for influenza vaccine using autodial had low-level effects on increasing influenza rates in 2 states. Given the feasibility and low cost of C-R/R in previous trials, its utility for influenza should be re-examined using different modalities.

Introduction

Seasonal influenza disease causes significant morbidity and mortality in the United States and globally.¹ Children contribute substantially to the public health burden of influenza because it causes high morbidity in young children^2,3 and also because children are a primary pathway through which infection is spread.¹ Therefore, the Advisory Committee for Immunization Practices (ACIP) recommends annual vaccination of all children ≥6 months of age.⁴ The Healthy People 2020 goal for influenza vaccination rates for children 6 months-17 years is above 80%⁵ however, current rates are only 59%.⁶

One evidence-based method for increasing vaccination rates is reminder/recall (R/R) messages, which can improve both routine childhood/adolescent and influenza vaccination rates.^7,8 The Task Force on Community Preventive Services recommends that practices use R/R for all vaccines⁹ however, less than 20% of practices conduct R/R due to a variety of barriers including time constraints, low staffing and staff turnover, lack of technical expertise, financial resources, and low confidence in the accuracy of patient immunization records.^10–13 Even when practices have access to needed technology and support for its use, studies suggest that only 14% to 21% of practices are willing to conduct R/R.¹⁴

Two previous trials have found that centralized R/R (C-R/R) by the state health department using the state immunization information system (IIS) was substantially more effective and cost-effective for increasing vaccination levels for routine childhood vaccines than approaches aimed at increasing practice-based R/R.^15,16 C-R/R has also been shown to be acceptable to most providers and parents.^17–19 However, evidence regarding the effectiveness of IIS-based C-R/R for increasing influenza vaccination rates among healthy children is limited. Influenza vaccination differs from childhood vaccination in a number of factors that could affect feasibility or effectiveness of C-R/R. These include the short time period during which vaccination needs to occur, frequent need for an additional visit for influenza vaccination (particularly for adolescents),²⁰ the involvement of community vaccinators (such as pharmacies) in influenza vaccination, a potentially high rate of parental hesitancy for influenza vaccine, and variation in vaccine supplies.

The objectives of the present study were to assess in a pragmatic randomized trial the effectiveness and cost of implementing IIS-based C-R/R of various intensity levels using autodialed messages as compared to usual care. The study was conducted within Colorado and New York in order to be able to contrast different IIS reporting laws, different experiences with C-R/R and different regulatory guidelines regarding community vaccinators. Our primary hypotheses were that patients who received the centralized R/R would have higher rates of receipt of ≥1 influenza vaccine and that the centralized approach would be cost-effective compared to usual care.

Methods

This study was approved by the Colorado Multiple IRB and by IRBs at University of California, Los Angeles, the New York State Department of Health and the Colorado Department of Public Health and the Environment. The intervention took place October, 2016 – January, 2017.

Colorado and New York State Immunization Information Systems and Policies

Colorado does not mandate that practices report immunizations into the Colorado IIS (CIIS). Despite this, at the time of the study, over 99% of children <6 years, 95% of children 6–10 years and 80% of adolescents 11–17 years had ≥2 immunizations in CIIS.¹⁸ CIIS receives patient demographic and vaccine event data through direct data entry into a Web-enabled application and through electronic transfers from practice Electronic Health Records, state vital records and insurers. Colorado pharmacists, public health departments, and visiting nursing services can vaccinate children. Roughly half of the major retail pharmacies and all public health entities submit vaccination data to CIIS. Finally, Colorado has been part of several large C-R/R studies by the health department for childhood vaccines.

The New York State IIS (NYSIIS) includes all of NY except for the 5 boroughs of New York City. New York City submits immunizations to an entirely different IIS, and, therefore, was not included. NYSIIS is populated from birth records from the New York State Vital Records. Unlike Colorado, NY mandates that all practices report children’s immunizations into NYSIIS, and 100% of children <6 years and 97% of adolescents have ≥2 immunizations in NYSIIS (based on census denominators). All public health entities and primary care offices report immunizations. Until 2018, pharmacies were unable to provide influenza vaccines to children in New York,^21,22 therefore almost all vaccination occurred within primary care sites. NYSIIS has once conducted C-R/R successfully using mailed letter reminders for human papillomavirus vaccine.²³

Study Populations

The overall sampling goal was to select a random sample of practices and children within practices generalizable to the overall population of each state, excluding New York City. We used a stratified two-stage cluster sampling approach, with practice as the primary sampling unit and rural/urban location as the strata. Once practices were randomly selected from the pool of all eligible practices, a random sample of children was selected from each practice and randomized to one of four treatment arms. Inclusion and exclusion criteria for practices and children are summarized in Figure 1.

Figure 1:

Consort Diagram in CO and NYS with Eligibility Criteria

Children 6 months to 17.9 years were included in both states and children who had received an influenza vaccine prior to the intervention were excluded from R/R. Patients with only demographic information from a birth certificate or the birth Hepatitis B dose were excluded as these patients likely were no longer residents of the state. In addition, based on previously developed criteria and required vaccinations for school, we also excluded adolescents 13–17.9 years with no immunizations since 9 years and children 6–12.9 years with no immunizations either on or after the 4^th birthday because of the likelihood that they were no longer in the state. Since both states require two MMR vaccines prior to entry into school, any child not having a second MMR at age four or older was judged unlikely to be in the state. We selected one index case randomly from families that had more than one child at the same phone number.

We conducted within practice randomization at the patient level with patients assigned to one of four arms: usual care; up to 1 auto-dial reminder; up to 2 auto-dial reminders; or up to 3 auto-dial reminders. Patients who became up-to-date with influenza vaccine were no longer sent R/R calls in future reminder rounds. In Colorado only, a sample of children who were not affiliated with a practice were included proportionate to the percentage of unaffiliated patients in CIIS, and all unaffiliated children in a given county were considered part of one “practice.” This was not done in NY since all patients were assigned a primary care practice based upon the most receipt of a non-influenza vaccination at a primary care practice. Randomization was conducted by the statistician in each state using SAS, and practice numbers were assigned to practices without practice identity being know until after randomization.

Centralized Reminder (C-R/R) Protocols

Each state used the same telephone company to perform autodial calls and used similar scripts in both English and Spanish. Both CIIS and NYSIIS have data fields for multiple phone numbers but designate one as the primary contact number, whether it is a mobile or landline; this primary contact field was used to populate the autodialer programs. Phone numbers are updated in the IIS automatically when an immunization service is given at a practice that has electronic data exchange capabilities with IIS; however phone numbers have to be manually updated when the practice does not have electronic data exchange with IIS. At the time of this study, the majority of immunization data was uploaded electronically into the IIS, 78% in CO and 70% in NY.

Messages included the name of the child’s primary care practice in NY; in CO, the child’s practice name was included if the practice agreed and, if not, or if the child was unaffiliated, the message included the name of the local health department. Monthly R/R calls began in October (CO) and in November (NY). Although the message stated the reminder was from the child’s provider or the health department, based upon the requirements of the autodialer company both states had to use an “800” number for the originating call rather than a caller identification (such as the practice’s phone number) that would have been familiar to the family. Messages were left if there was an answering machine. Patients could opt out of receiving further C-R/R by calling a toll-free number, pressing “9,” or sending an email address listed in the message. The first reminders were sent out 10/4 in CO and 11/9 in NY, with subsequent reminders occurring every 4–5 weeks after these dates.

Outcome Measures

The primary study outcome was documentation of ≥ 1 influenza vaccine within 6 months. Secondary measures included timeliness of vaccination and effectiveness within subgroups, including different age groups, practice specialty of the child’s provider, and whether the child had received an influenza vaccine in the prior year. We also assessed costs of delivering R/R from the perspective of a state public health department and calculated cost-effectiveness if centralized R/R was associated with significantly higher influenza vaccination.

Post-Hoc Survey

To better understand our results, we conducted an unplanned 8-question survey 6–7 months after the intervention had ended. We sent certified mail surveys to the office manager at each practice with instructions to have the lead physician, nurse or office manager, whoever was most knowledgeable about immunization delivery, complete the survey. The mailing included a $10 Amazon card (CO) or $10 bill (NY). We sent follow-up faxes, emails and phone calls for four weeks to non-responders. Survey questions assessed whether the practice sent any of their own reminders for influenza vaccine during the 2016–2017 season, explored potential reasons that some vaccinations might not have been reported to the IIS, (including cash-based influenza clinics within the practice that might not result in reporting of vaccinations to the IIS, or whether there was manual uploading to the IIS), and asked whether demand for the vaccine had been lower than usual this past season.

Statistical Analysis

The study was powered to detect an absolute difference of ≥2 percentage points in influenza vaccination rates at the end of the 6-month period in comparisons of control versus any intervention arm. We used mixed-effects multivariable logistic regression models with the primary outcome of receipt of ≥1 influenza vaccination adjusted for child’s age, practice type, urban/rural status, and influenza vaccine in the prior year as these factors have been shown to differentially affect immunizations and, potentially, response to our intervention. We used risk ratios to compare groups. We included a random effect for practice to adjust for clustering of patients within practices, and tested interactions between each predictor and study arm. In order to examine differences in timeliness of vaccination, we used Cox proportional hazards models, adjusted for clustering of patients within practices. To better visualize intervention impact on time to vaccination we also generated cumulative incidence curves and kernel-smoothed hazards plots. Finally, we included a post-hoc analysis assessing the major outcome eight weeks after starting the intervention rather than at the end of the season. All major analyses were intention to treat. All analyses were performed using SAS, version 9.4, SAS Institute, Inc.

We compared the cost of implementing centralized R/R for each intervention arm within each state within cost domains including: 1) consensus building and preliminary work, 2) software, 3) training, 4) collaboration, 5) implementation, and 6) reminder/recall messages. We examined costs from the perspective of the state public health department or IIS.

Results

Table 1 describes the characteristics of the participating practices and patients by state; comparisons of these factors between intervention and control arms for each state are included in Supplemental Table 1.

Table 1:

State, Practice and Patient Characteristics

Practice Information	Colorado n=65 n (%)	New York n=101	p-values
Practice type			<.001
Pediatric	23 (35)	78 (77)
Family Medicine	30 (46)	16 (16)
Community Health Center (CHC)	12 (18)	7 (7)
Practice location			.65
Rural practices	10 (15)	13 (13)
Urban practices	55 (85)	88 (87)
Inclusion of practice name on R/R	39 (60)	101 (100)	<.001
Patient Information	Colorado n= 54,353 n (%)	New York n=65,777 n (%)	p-values
Total unaffiliated patients	7,554 (14)	N/A
Age			<.001
6mo-<2y	4,418 (8)	5,396 (8)
2–5y	13,347 (24)	15,350 (23)
6–10y	14,517 (27)	18,143 (28)
11–17y	22,071 (41)	26,888 (41)
Total patients that opted-outa of R/R	333 (0.6)	366 (0.5)	0.20
Percentage of missing phone numbers	5,028 (9)	12,554 (19)	<.001

^aopted out by calling to have their name removed from the recall list or pressing 9 during autodial message.

Centralized R/R outcomes for influenza vaccine

As shown in Table 2, the maximum absolute difference between study arms in CO were between those who received 2 R/Rs vs usual care (absolute difference of 1.7%), which was statistically significant with an adjusted risk ratio (ARR) of 1.06 (1.01, 1,10); no significant differences were noted between the 1 R/R and usual care or 3 R/R and usual care arms. All three intervention arms were statistically significant in NY with ARRs ranging from 1.05 to 1.06. However, the maximum absolute difference between study and control arms was only 0.6% points. The cumulative incidence plots (Figure 2) show the total percent vaccinated for each study arm compared to controls (dashed line) by days from the start of the intervention in both states. Overall p-values for all study arms from a Cox proportional hazards model, accounting for clustering within clinic, were 0.001 in CO and 0.65 in NY. The smoothed hazard rate plot shows the estimated hazard rates over time for each study arm compared to controls (dashed line) in both states. Hazards show the largest difference between intervention and control arms around the peak vaccination time for each state, which occurred in the first several weeks following the start of the intervention in late fall. Table 3 demonstrates that ARRs were larger if the outcomes were assessed eight weeks after the intervention, again suggesting that the intervention resulted in earlier vaccination in intervention groups.

Table 2:

Vaccination Rates by Study Arm and Covariate Level, with Unadjusted and Adjusted Risk Ratios

	State	n by category	Vaccinated per category, n (%)	Unadjusted Risk Ratio	Adjusted Risk Ratio
Colorado	Age category
	6 months - 1.9 years	4,418	2,119 (48.0)	2.63 (2.42, 2.87)	1.97 (1.83, 2.11)
	2–5.9 years	13,347	3,652 (27.4)	1.50 (1.37, 1.64)	1.20 (1.14, 1.26)
	6–10.9 years	14,517	3,384 (23.3)	1.28 (1.21, 1.35)	1.15 (1.11, 1.20)
	11–17.5 years	22,071	4,024 (18.2)	-Reference-	-Reference-
	Practice type
	Family Medicine	10,915	2,319 (21.2)	-Reference-	-Reference-
	Pediatrics	28,847	8,102 (28.1)	1.32 (1.07, 1.64)	1.14 (0.99, 1.30)
	CHC/RHC	7,037	1,728 (24.6)	1.16 (0.91, 1.47)	1.05 (0.85, 1.29)
	Unaffiliated	7,554	1,030 (13.6)	0.64 (0.51, 0.81)	0.83 (0.72, 0.97)
	Rurality
	Urban	48,494	11,957 (24.7)	-Reference-	-Reference-
	Rural	5,859	1,222 (20.9)	0.85 (0.66, 1.08)	0.92 (0.79, 1.06)
	Prior year vaccination status
	Unvaccinated	39,385	5,124 (13.0)	-Reference-	-Reference-
	Vaccinated	14,968	8,055 (53.8)	4.14 (3.75, 4.57)	3.66 (3.31, 4.03)
	Study Arm
	Control	13,566	3,201 (23.6)	-Reference-	-Reference-
	1 Reminder	13,608	3,270 (24.0)	1.02 (0.97, 1.06)	1.02 (0.98, 1.06)
	2 Reminders	13,577	3,437 (25.3)	1.07 (1.03, 1.12)	1.06 (1.01, 1.10)
	3 Reminders	13,602	3,271 (24.0)	1.02 (0.98, 1.06)	1.01 (0.97, 1.05)
State	Category	n by category	Vaccinated per category, n (%)	Unadjusted Risk Ratio	Adjusted Risk Ratio
New York	Age category
	6 months - 1.9 years	5,396	2,540 (47.1)	2.73 (2.60, 2.86)	2.41 (2.29, 2.52)
	2–5.9 years	15,350	3,611 (23.5)	1.46 (1.40, 1.53)	1.23 (1.17, 1.28)
	6–10.9 years	18,143	3,201 (17.6)	1.12 (1.07, 1.17)	1.08 (1.03, 1.13)
	11–17.5 years	26,888	4262 (15.9)	-Reference-	-Reference-
	Practice type
	Family Medicine	5952	1,056 (17.7)	-Reference-	-Reference-
	Pediatrics	53,462	11,151 (22.1)	1.42 (0.94, 2.15)	1.17 (0.85, 1.63)
	CHC/RHC	6,363	1,407 (20.9)	1.62 (0.83, 3.16)	1.06 (0.66, 1.71)
	Rurality
	Downstate	31,483	5,727 (18.2)	-Reference-	-Reference-
	Upstate rural	5,745	928 (16.2)	0.83 (0.54, 1.29)	1.07 (0.75, 1.54)
	Upstate urban	28,550	6,959 (24.4)	1.65 (1.22, 2.23)	1.44 (1.16, 1.80)
	Prior year vaccination status
	Unvaccinated	48,275	4,947 (10.2)	-Reference-	-Reference-
	Vaccinated	17,502	8,667 (49.5)	4.08 (3.94, 4.23)	3.94 (3.81, 4.09)
	Study Arm
	Control	16,429	3,343 (20.3)	-Reference-	-Reference-
	1 Reminder	16,479	3,424 (20.8)	1.02 (0.98, 1.07)	1.06 (1.01, 1.11)
	2 Reminders	16,454	3,422 (20.8)	1.03 (0.98, 1.08)	1.06 (1.01, 1.11)
	3 Reminders	16,415	3,425 (20.9)	1.04 (0.99, 1.09)	1.05 (1.01, 1.10)

Random effect for practice included. The intervention effect did not differ by practice type (p-values for interaction terms in CO: p=.82, NY: p=.65), rurality (CO: p=.85, NY: p =.48) and did not differ by prior influenza year influenza vaccination (NY: p=.88) or by age category in NY (NY: p=.60). The intervention effect did differ slightly by age (CO: p=0.04) and prior year influenza vaccination in CO (CO: p= .03).

NY categorized into 3 regions due to differences in healthcare delivery systems by region. NY Upstate urban and Downstate are most comparable to CO urban; upstate rural and CO rural are most comparable.

Figure 2: Time-to-event analysis of flu vaccinations.

Cumulative incidence curves A and C above show the cumulative percent of study subjects vaccinated over time in CO and NY, with final vaccination rates among children not already vaccinated at the start of the study ranging, from 20–25%. Smoothed hazard plots B and D show the probability of unvaccinated subject receiving a vaccination at any given time through the study period. The peak vaccination time for both states occurred shortly after the start of the intervention in October, November, and early December, with instantaneous vaccination rates decreasing through the remainder of the season. Small differences were observed between study arms at this peak vaccination time. Overall p-values for study arm from a Cox proportional hazards model, accounting for clustering within cinic, were 0.001 in CO and 0.65 in NY.

Table 3:

Cost of Centralized R/R in CO and NY^*

	Colorado			New York
	1 Contact (n=13,608)	2 Contacts (n=13,577)	3 Contacts (n=13,602)	1 Contact (n=16,479)	2 Contacts (n=16,454)	3 Contacts (n=16,415)
Type of Cost
Personnel	$1,480	$1,566	$1,653	$2,895	$3,325	$3,760
Other	$864	$1,447	$1,976	$935	$1,844	$2,752
TOTAL COST (for total # patients in each column)	$2,344	$3,013	$3,629	$3,830	$5,169	$6,512
COST PER CHILD RANDOMIZED	$0.17	$0.22	$0.27	$0.23	$0.31	$0.40
Type of Activity
Consensus Building and Preliminary Work	$838	$838	$838	$936	$936	$936
Training	$51	$51	$51	$20	$20	$20
Software Costs	$646	$1,229	$1,759	$888	$1,774	$2,659
Collaboration	$538	$538	$538	$1,199	$1,199	$1,199
Implementation Meetings	$136	$203	$271	$87	$131	$175
Recall	$134	$153	$172	$699	$1,109	$1,523
TOTAL	$2,343	$3,012	$3,629	$3,829	$5,169	$6,512
COST PER CHILD RANDOMIZED	$0.17	$0.22	$0.27	$0.23	$0.31	$0.40

^*Due to rounding error to the nearest dollar across subcategories, the Total costs by intervention may be slightly different within Type of Cost versus Type of Activity. For interpretation purposes, the Type of Cost total costs were used within the manuscript text when referring to total cost by intervention.

In both states, we saw differences within both the usual care and intervention groups by age, practice type and prior vaccination receipt (Table 2). Influenza rates were highest among <2 years olds and lowest for adolescents. In unadjusted analyses in both states, children in pediatric practices achieved the highest rates among practice types, and influenza vaccine receipt in the year prior was highly correlated with receipt during the present season. After adjustment in the multivariable model, age and previous vaccination remained significant predictors of vaccination, but practice type did not, with the exception of patients in Colorado who were not affiliated with a practice.

Cost effectiveness of C-R/R for influenza vaccine

The cost per child randomized to 1, 2, and 3 R/R messages in Colorado was $0.17, $0.22 and $0.27, respectively, and in New York was $0.23, $0.31, and $0.40 (Table 4). In Colorado, the cost per additional shot received for 2 contacts vs. usual care was $29. In New York, when comparing the intervention arms to usual care, the cost per additional shot received was $48, $63, and $70 for the 1 contact, 2 contact, and 3 contact arms, respectively.

Table 4:

Rates of Vaccination Within 8 Weeks of the Start of the Intervention, by Study Arm with Unadjusted and Adjusted Risk Ratios

	State	n by category	Vaccinated per category, n (%)	Unadjusted Risk Ratio	Adjusted Risk Ratio
CO	Study Arm
	Control	13,566	2,130 (15.7)	-Reference-	-Reference-
	1 Reminder	13,608	2,242 (16.5)	1.05 (1.00, 1.10)	1.05 (1.00, 1.10)
	2 Reminders	13,577	2,362 (17.4)	1.11 (1.06, 1.16)	1.09 (1.04, 1.14)
	3 Reminders	13,602	2,255 (16.6)	1.06 (1.01, 1.10)	1.04 (1.00, 1.09)
State	Category	n by category	Vaccinated per category, n (%)	Unadjusted Risk Ratio	Adjusted Risk Ratio
NY	Study Arm
	Control	16,429	1,578 (9.6)	-Reference-	-Reference-
	1 Reminder	16,479	1,695 (10.3)	1.13 (1.06, 1.21)	1.17 (1.09, 1.25)
	2 Reminders	16,454	1,655 (10.1)	1.10 (1.03, 1.17)	1.14 (1.06, 1.22)
	3 Reminders	16,415	1,598 (9.7)	1.06 (0.99, 1.14)	1.08 (1.01, 1.15)

Random effect for practice included. The intervention effect did not differ by age category (CO: p=0.41, NY: p=.1333), practice type (p-values for interaction terms in CO: p=.96, NY: p=.07), rurality (CO: p=.65, NY: p =.47) or prior influenza year influenza vaccination (CO: p=0.28, NY: p=.53).

Post-hoc Survey Data

Survey response rates were 99% in CO and 74% in NY. Substantial percentages of the practices reported (1) sending R/R messages to some patients in their practices (40% CO; 49% NY), (2) at least one potential reason that vaccines might not have been uploaded to the IIS in all cases (75% CO; 67% NY), and (3) lower than usual demand for influenza vaccine (61% CO; 27% NY); yet, inclusion of these variables in the multivariable model did not alter our results (see Supplemental Table 2 for questions).

Discussion

Because of persistently low influenza vaccination rates, we sought to extend to childhood influenza vaccinations an evidence-based approach for centralized R/R that has been more effective than practice-based approaches for increasing vaccination rates in the childhood series. Our results show very small effects of the intervention in two contrasting states. We conducted post-hoc analyses to assess whether R/R being conducted by practices in addition to the C-R/R, problems with uploading influenza vaccination information into the IIS, or decreased demand for influenza vaccination during the 2016–2017 season could have diluted or masked a potential intervention effect. Although each of these issues were occurring in many practices within both states, their inclusion in analytic models did not change our results.

There are other possible reasons for why our findings were less robust that we cannot examine with our data. First, autodial calls may have been less effective during the 2016 election year when there were, potentially, many political autodialed messages being sent. It is also possible that autodialed methods, even in a non-election year, may be losing their value given the growth of autodialer calls overall. The fact that our autodialed calls would have been identified as coming from an “800” number rather than from the child’s provider may also have attenuated their effectiveness. This is an important potential factor to consider in the use of autodialer messages in future R/R efforts.

Second, influenza vaccines could have been underreported in the state IISs, especially in CO where reporting is not mandatory. In cases where practices upload manually or for cash-based influenza clinics, providers might consider uploading influenza vaccines to be less important than other vaccines since children need to be vaccinated every year regardless of previous vaccinations. Influenza vaccination levels in both CO and NY were, indeed, lower than National Immunization Survey-Flu (NIS-Flu) and Behavioral Risk Factor Surveillance System (BRFSS) combined data. Vaccination rates for the full flu season (including children vaccinated before our intervention began) in NY were 36% in our data compared with 64% for NY state (excluding NYC) from national NIS-Flu data⁶ for the 2016–2017 season; in CO, rates were 26–28% for the full season in our data compared with 63% from national data. National influenza rates derived from these surveys are based on self-report without verification and have been shown to overestimate rates (by as much as 17%).^22,24 However, if there was underreporting of influenza vaccinations to IISs in our study, this may have contributed to our negative results. Further, underreporting may have been a particular problem in CO because of the involvement of retail pharmacies in influenza delivery in CO (not all pharmacies report to CIIS). We chose to include a mandatory reporting state (NY) because of a priori concerns about underreporting. If underreporting of influenza vaccinations in IISs is occurring to a large extent, it may make the case for centralized R/R difficult to make and, therefore, limit its applicability for this vaccine.

A third possible explanation is the fact that there is so much media information regarding influenza vaccination during the influenza season. This may mitigate R/R effectiveness for influenza vaccine since awareness may not be much of a limiting step in getting vaccinated. Fourth, there are potential reasons why R/R may be becoming less effective for influenza in recent years than noted in earlier studies. Given media reports about the limited effectiveness of inactivated influenza vaccine during several seasons^25–27 and the withdrawal of live attenuated influenza vaccine for two seasons,^28,29 parents may feel influenza vaccination is not worthwhile. Conversely, they may have increased concerns about vaccine safety. Although vaccine hesitancy in general has been associated with influenza vaccine refusal,³⁰ there have been no national surveys of parents to assess how much hesitancy about influenza vaccines is contributing to low rates of vaccination.

Although R/R was shown in a recent systematic review to be effective in increasing influenza vaccination rates in children,⁷ it is worth noting that among the five pediatric studies included, only one study focused on healthy children rather than children with high risk conditions.³¹ This study found an absolute increase of 4.4% among children 6–23 months of age. The only trial of which we are aware that studied R/R for healthy children of all age groups, using text reminders, found an absolute difference of 3.7% in 2012.^32,33 Therefore, the effect size for R/R for influenza vaccine for healthy children, whether practice-based or centralized, may be modest compared with other childhood vaccines.

It is also important to note, however, that small increases in influenza rates may have substantial benefit at the population level. A recent modeling study demonstrated that a 5% increase in influenza vaccine coverage among children 6 months to 17 years could result in 282,000 fewer influenza infections and 1,440 fewer hospitalizations during a severe influenza season.³⁴ Modeling for smaller effect sizes has not been done. When one considers the costs associated with hospitalizations for influenza, even without including the many other costs associated with influenza illness, low cost interventions such as C-R/R, perhaps using different modalities, may be worthy of further consideration. In addition, shifting vaccination to earlier in the season may result in substantial benefit, depending on the timing of the onset of influenza disease.

Our data show expected trends in influenza delivery by age group and by vaccination in the prior year. Younger children nationally have the highest rates of vaccination and adolescents the lowest among pediatric populations³⁵ and other studies have shown that prior vaccination is a good predictor of influenza vaccination.³⁵ These trends provide some direction for interventions that may be needed to increase rates in different subgroups. Given the higher rates among 6-month- to 2-year-olds, probably related to their frequent health maintenance visits, efforts might better be focused on reducing missed opportunities at office visits rather than R/R. For children who were vaccinated in the prior year, provision of information about when the vaccine is available at vaccination sites may be sufficient. Rates of vaccination among school age and, especially, adolescents are very low, and their lack of response to R/R suggest approaches other than R/R, for example school-located vaccination, may be needed.^36,37

The cost of centralized R/R in this study was approximately $0.20 to $0.30 per child recalled. Our estimates are most similar to a previous study that calculated the mean cost of IIS-based C-R/R using an autodialed method as $0.53 per contact. Another study of C-R/R among adolescents reported $0.78 per adolescent recalled,³⁸ but this included a much smaller sample size and, therefore, did not benefit from economies of scale. Because the differences in vaccination rates were so small between study and control arms the cost-effectiveness estimates were much higher in this study than those previously reported for C-R/R.^15,16

There are important strengths and limitations to this study. It was a large, two-state trial with rigorous sampling methods to ensure generalizability of the childhood populations from the two states. Analyses were able to account for clustering within practices and for a variety of potential confounders. Randomization occurred at the level of the patient within each practice, which mitigated against confounding factors at the practice level. However, as discussed, our outcomes were determined based on data from the state IISs that likely were underreported, especially in CO. In addition, the intervention did not begin until November in NY which may have attenuated its effectiveness. Finally, in both states contact information in the state IISs may not have been up-to-date for some families.

Influenza vaccination rates for children have essentially plateaued over the past four years, and our study was not supportive of a large role for centralized IIS-based R/R in improving influenza vaccination coverage. However, given its prior success and relatively low cost in improving other childhood vaccine rates, effectiveness of centralized R/R for influenza vaccine should be reexamined during a non-election year and using different modalities other than auto-dialer messages. In addition, the completeness of reporting influenza vaccines into IISs deserves more study as incomplete reporting limits evaluation of IIS-based R/R methods. Our study shows the feasibility of centralized R/R, its low cost and its potential to shift receipt of influenza vaccine to earlier in the season. Its utility for recalling large numbers in the event of a pandemic may be much greater than we have demonstrated to date for seasonal influenza. Our data certainly underscore the fact that each vaccine presents different challenges, and nimbleness is required in matching evidence-based methods for increasing immunization rates to the specific challenges of each vaccine.

What’s New.

This large, two-state trial showed minimal but significant effects of centralized IIS-based autodialer reminder/recall on increasing influenza rates in both states. Given its low cost, this method should be explored using different reminder/recall modalities.