Effect of State Immunization Information System Based Reminder/Recall for Influenza Vaccinations: A Randomized Trial of Autodialer, Text, and Mailed Messages

Abstract

Objective:

To evaluate the effect of different modalities of centralized reminder/recall (C-R/R) (autodialer, text, mailed reminders) on increasing childhood influenza vaccination.

Study design:

Two simultaneous randomized clinical trials (RCTs) conducted 10/2017–4/1/2018 in New York State and Colorado. 61,931 children in New York (136 practices) and 23,845 children in Colorado (42 practices) were randomized to different C-R/R modalities—4 arms in NY (autodialer, text, mailed, no-reminder control) and 3 arms in Colorado (autodialer, mailed, no-reminder control). Message content was similar across modalities. Up to three reminders were sent for intervention arms. The main outcome measure was receipt of at least one influenza vaccine.

Results:

In New York, compared with the control arm (26.6%), post-intervention influenza vaccination rates in the autodialer arm (28.0%) were 1.4 percentage points higher; adjusted risk ratio (ARR) 1.06 (1.02, 1.10)], but rates for text (27.6%) and mail (26.8%) arms were not different from controls. In Colorado, compared with the control arm (29.9%), post-intervention influenza vaccination rates for autodialer (32.9%) and mail (31.5%) arms were 3.0 percentage points and 1.6 percentage points higher, respectively [ARRs 1.08 (1.03, 1.12) and 1.06 (1.02, 1.10), respectively]. Compared with the control arm, the incremental cost-per-additional vaccine delivered was $20 (New York) and $16 (Colorado) for autodialer messages.

Conclusions:

Centralized reminder/recall for childhood influenza vaccine was most effective via autodialer, less effective via mail, and not effective via text messages. The impact of each modality was modest. Compared with no reminders, the incremental cost-per-additional vaccine delivered was also modest for autodialer messages.

Trial registration

ClinicalTrials.gov: NCT03294473 and NCT03246100

Seasonal influenza causes substantial illnesses, hospitalizations, ambulatory visits, and deaths throughout the U.S among both children and adults.^1,2 Despite longstanding national recommendations for influenza vaccination of all children over 6 months of age and Healthy People 2020 goals for >80% childhood influenza vaccination rates,³ national influenza vaccination rates using the 2017–2018 National Immunization Survey (which does not include provider record checks for influenza vaccine), were only 58% for children 6 months to 17 years of age.⁴

Patient reminder/recall (R/R) can raise influenza vaccination coverage.^5,6 Most published studies of R/R for influenza vaccination involved practice-based reminders, and many of these targeted high-risk subgroups of patients and not an entire population of children.⁵ The Task Force on Community Preventive Services recommends sending patients R/R messages for any vaccine,⁷ yet only 20–33% of primary care practices send any R/R messages for any vaccinations including influenza vaccination.^8–10 A frequently cited barrier is limited resources.^9,11,12

Some experts have begun focusing upon centralized R/R (C-R/R) as a potential scalable R/R strategy. Centralized systems such as state immunization information systems (IISs) or health systems can use economies of scale to send reminders to patients. Several studies of IIS-based C-R/R have noted improvements in routine childhood vaccination rates,^13–15 but few investigated the impact of C-R/R for raising influenza vaccination rates among an entire population of children. The few prior studies using C-R/R for influenza vaccine have focused upon children with chronic diseases,⁸ subsets of the population,^16,17 senior adults,^18,19 pregnant women,²⁰ or small numbers of practices.^20,21 High vaccine hesitancy²² for influenza vaccination,^23–25 potential confusion among parents about locations for influenza vaccinations²⁶ (ie, practices, schools, pharmacies), and general barriers for vaccination^27,28 may reduce the impact of C-R/R for influenza vaccination for the entire child population. On the other hand, even a small increase in vaccination rates, if scaled up widely, may reduce influenza-related vaccine-related morbidity.²⁹

Patient R/R messages can use a variety of modalities- e.g., phone, mail, and text.⁵ Phone reminders can involve live person calls or autodialers; autodialers are more scalable.

Methods

This study evaluated: (1) the effectiveness of C-R/R upon influenza vaccination of an entire population of children, (2) the relative effectiveness of autodialer (phone), text message, or mailed C-R/R, and (3) the cost-effectiveness of different reminder modalities. We simultaneously conducted two multi-arm RCTs of IIS-based C-R/R across New York (NY) and Colorado (CO). We hypothesized a priori that C-R/R would raise influenza vaccination rates overall, but did not have a priori hypotheses about relative effectiveness by reminder modality.

The Institutional Review Boards (IRB) at UCLA, New York State Department of Health, University of Colorado, and the Colorado Department of Public Health and the Environment all approved this study. The C-R/R messages were sent between October 2017 and December 31, 2017 (Figure 1; available at www.jpeds.com)). In NY, we evaluated a 4-arm RCT (autodialer, text, mail, or no-C-R/R control arm). In CO, we evaluated a 3-arm RCT (autodialer, mail, or no-C-R/R control arm), without a text message arm due to state limitations for sending text messages without consent from recipients.

Figure 1:

Consort Diagram plus practice and patient eligibility criteria

These two states had different IIS reporting laws, compliance regulations for community vaccinators, and involvement with C-R/R. NY had mandatory IIS reporting for vaccines given to persons <19 years of age; CO did not. CO allowed vaccination of children in pharmacies; NY did not until late in the study data collection. CO had performed IIS-based C-R/R for routine childhood vaccinations; NY had not.

In NY, the setting included 57 counties (2.3 million children) outside of the five New York City boroughs. We excluded New York City which has a different immunization registry. In CO the setting included ten urban counties encompassing Denver Metro (total child population of about 660,000) plus several urban counties in northern and southern Colorado.

New York State and Colorado Immunization Information Systems and Policies

In both states, practices routinely sent vaccination and demographic information to the state IIS via electronic transfers from practice EHRs or via direct data entry into a web-enabled application.

New York State’s IIS (NYSIIS) and the New York City IIS exchange immunization updates found on children who have a home address in the other’s jurisdiction. New York mandates that all vaccinators send vaccination data to the IIS. CDC has a standard method for state IISs to report on completeness of IIS data, by comparing the number of unique individuals contained in the IISs with census denominators. In NY the number of children <6 years with ≥2 vaccination records in NYSIIS was about equal to the actual census count of children <6 years, and the number of adolescents who had ≥2 vaccination records in NYSIIS was 97% of the census count of adolescents. Prior to January 25, 2018, NY pharmacies could only administer adult influenza vaccines. Starting 1/25/2018 (after all reminders had been sent) NY began allowing pharmacies to vaccinate children ages ≥2 without a physician order; however few children were vaccinated at pharmacies. NYSIIS had performed an HPV C-R/R mailed reminder study³⁰ but no prior C-R/R studies for influenza vaccine.

Colorado’s IIS (CIIS), did not have mandated reporting, yet >99% of <6 year-olds, 95% of 6 to10 year-olds, and 80% of 11 to 17 year-olds had at least two immunization records in CIIS. In CO, pharmacists, local public health agencies, and primary care practices vaccinate children. About half of pharmacies and all local health clinics submitted vaccination data to CIIS, and pharmacies can vaccinate children of any age (although they rarely vaccinate children less than 2 years of age). CIIS had a previous history of C-R/R for childhood vaccination^13–15 but not for influenza vaccine.

Study Populations

Using a stratified, two-stage cluster sampling approach, with practice as the primary sampling unit and rural/urban location as the strata, we randomly selected practices from the pool of all eligible practices in each IIS, sampling practices proportional to practice size. We then randomly selected children per practice and then within each practice randomly assigned them to autodialer message, text message (NY only), mailed message, or no-reminder groups. In NY we allocated 30% of subjects to autodialer, text message, or no-reminder arms and a smaller number (10%, due to costs) to the mailed C-R/R arm. In CO, a smaller overall sample size was selected, and patients were assigned evenly between the three arms. A statistician for each state performed the randomizations using SAS version 9.4. The sample sizes in both states were sufficient to provide >80% power to detect improvements of 2.5 percentage points in each intervention arm versus the control arm. This conservatively assumes a 50% control arm vaccination rate, a 2-fold Bonferroni correction in CO and a 3-fold correction in NY for the multiple intervention arms, and an overall significance level of 0.05.

Figure 1 outlines practice and patient inclusion/exclusion criteria. We included children 6 months to 17.9 years old, grouped siblings by common phone or address, and randomly selected one child per family in families with ≥1 child. We excluded children who had been vaccinated prior to 10/18/2017 in NY or 10/2/2017 in CO (start of the reminder fieldwork).

Centralized Reminder (C-R/R) Fieldwork

Message Content (Figure 3; available at www.jpeds.com):

Online Figure.

Autodialer and text message content in the two states.

*Note: Colorado did not include “state health department” in scripts.

the content of the message was similar in both states and across all intervention modalities. In NY, irrespective of modality (autodialer, text, mail), messages stated that the message was from the NY State Health Department and named the primary care practice. In CO, messages stated that the reminder was on behalf of the primary care practice and also included practice names. Reminders in both states had a different educational message each month.

Phone Messages:

A commercial telephone company (http://www.teletask.com) sent autodialer calls for both states, in English and Spanish (recipients pressed “2” for Spanish). We used the phone number listed in the IIS as the primary contact number, whether cell or landline. Based upon state preferences and also requirements of the telephone company, both states used an “800” number rather than the practice’s phone number; thus, the families using caller identification methods did not see the child’s primary care practice’s phone number. Voicemails were left if a call went unanswered. Recipients could opt out of receiving further calls by calling a toll-free number, or pressing “9”.

Text messages:

In NY, Teletask also sent text messages, in English and Spanish. Recipients could reply, “SPAN” to receive the text message in Spanish or “STOP” to opt out.

Mailed Messages:

We sent postcard messages in English and Spanish. In NY, to comply with privacy concerns, the postcard was folded and sealed with the main message (including the primary care practice name and office phone number) on the interior, and in CO a regular postcard was used. Recipients could call or email to be removed from the reminder list.

Protocols:

Up to three messages were sent to eligible patients approximately every 4 to 6 weeks. Patients who received an influenza vaccine according to the IIS were removed from the calling list between reminder rounds, and phone numbers and addresses were updated if they were updated in the IIS.

Outcome Measures

The primary study outcome was IIS-based documentation of ≥ 1 influenza vaccine within six months of the start of the study (October 2017 through March 31, 2018 [Figure 1]). We assessed vaccinations from the IISs after April 1 (to allow for data to be uploaded into the ISSs.) Secondary outcomes included costs of the intervention stratified by personnel and other related costs, and cost-effectiveness. We estimated the total cost by summing the costs related to: consensus building and preliminary work; training; software costs; collaboration; implementation meetings; and reminders. We used the viewpoint of the state IIS when calculating cost effectiveness measures.

Statistical Analyses

The primary analysis compared the effectiveness of IIS-based C-R/R, sent by autodialer, text (NY only), or mail, compared with no-reminder control, in increasing receipt of ≥1 influenza vaccine. We used generalized linear mixed modeling to assess the impact of C-R/R on receipt of ≥1 influenza vaccine. We used Poisson regression with a robust variance estimator to obtain risk ratios, and we adjusted for covariates including patient age, receipt of ≥1 influenza vaccine in the prior 2 years and type of practice. These covariates have been noted in prior studies to affect vaccination rates and could affect the response to our intervention. Possible within-practice correlation was accounted for using a random effect for practice. We also tested interactions between each predictor and study arm to determine whether there were differential intervention effects by covariates. We employed intention-to-treat analyses and used version 9.4, SAS Institute, Inc.

In addition, we performed two additional analyses to test whether the intervention resulted in earlier receipt of influenza vaccine; this is relevant because optimal immunity from the vaccine is achieved a couple of weeks following vaccination and also influenza can arrive in December in some seasons. We repeated the unadjusted and adjusted Poisson regression to obtain risk ratios for vaccination but now with the endpoint being December 1 rather than March 31. Second, we used time-to-event analyses to assess differences in timeliness of vaccination by study arm, adjusted for clustering of patients within practices.

Cost effectiveness analyses took into account personnel time to plan and send reminders in addition to cost of autodialer, text, and mail messaging. We considered the cost differential to carry out C-R/R for autodial, text (NY only), and mail vs no IIS-based C-R/R reminder. The total cost for each randomized arm and state was the sum of the cost activities related to personnel and other expenses relevant to a state IIS. We reported the cost per child randomized within each arm and state as well as the incremental cost per additional vaccine delivered for each active arm versus the control arm.

Results

Enrollment occurred from September 2017 to April 2019. Table 1 shows patient and practice characteristics by state. There were large numbers of practices and children across the practice types, geographic regions, study arms, and age groups. Very few children in either state had opted out of the IIS or had missing phone numbers.

Table 1:

Characteristics of study practices and patients in each state (intervention and control subjects combined).

Characteristics		New York n (%)	Colorado n (%)
Practices
Number		136	42
Practice Type	Pediatric	95 (69.9)	19 (45.2)
	Family Medicine	31 (22.8)	19 (45.2)
	Community Health Center (CHC)	10 (7.4)	4 (9.5)
Geography	New York
	-Downstate (mostly urban)	57 (41.9)
	-Upstate urban	22 (16.2)
	-Upstate rural	57 (41.9)
	Colorado
	-Metro (Denver and surrounds)		21 (50.0)
	-North of Denver		12 (28.6)
	-South of Denver		9 (21.4)
Children
Number	All study arms	61931	23,845
Study Arm	Autodialer	19003 (30.7)	7,910 (33.2)
	Mail	4779 (7.7)	8,007 (33.6)
	Text	19090 (30.8)	N/A
	Control	19059 (30.8)	7,928 (33.2)
Age Group	6 months to <2 years	3967 (6.4)	1,952 (8.2)
	2–5 years	13620 (22.0)	5,810 (24.4)
	6–10 years	18728 (30.2)	6,534 (27.4)
	11–17 years	25616 (41.4)	9,549 (40.0)
Vaccinated in last 2 years	Vaccinated in 2015–2016 or 2016–2017 season	24957 (40.3)	10,457 (43.9)
Miscellaneous	Total patients that opted-out* of R/R	1199(1.9)	NA
	Missing phone numbers	129(0.2)	910 (3.8)

^*Opted-out by calling to have their name removed from the recall list or pressing 9 during autodial message (this was only possible in New York).

Influenza Vaccination Rates

New York:

Influenza vaccination rates (Table 2) were slightly higher (by 1.4% points) in the autodialer arm than in the control arm (p=0.007) but not statistically different in the text or mail arms versus the control arm: autodialer-28.0%, text-27.6%, mail-26.8%, control-26.6%. In adjusted models, the probability of vaccination was significantly increased in the autodialer arm compared with the control arm [ARR = 1.6, 95% CI (1.02, 1.10)], but not for the text or mail arms. The intervention effect differed (not shown in tables) by age (interaction effect p=.04), but not practice type (p=.83), or prior vaccination (p=.07). Children 6 months to 2 years were most likely to have received the vaccine and those <11 years were more likely than those ≥11 years to be vaccinated.

Table 2:

Vaccination rates at the end of the study and unadjusted and adjusted risk ratios for influenza vaccination by patient characteristics and study arm.

	Category	n by category	Vaccinated per category, (%)	Unadjusted Risk Ratio	Adjusted Risk Ratio
New York
Age category	6 months – 1.9 years	3967	52.3	2.40 [2.28, 2.53]	2.28 [2.16, 2.40]
	2 – 5.9 years	13620	30.8	1.41 [1.35, 1.47]	1.09 [1.05, 1.14]
	6 – 10.9 years	18728	25.8	1.17 [1.12, 1.21]	1.10 [1.06, 1.14]
	11 – 17.9 years	25616	22.4	-Reference-	-Reference-
Practice type	Family Medicine	7273	20.9	-Reference-	-Reference-
	Pediatrics	49390	28.1	1.38 [1.17, 1.62]	1.17 [1.04, 1.33]
	CHC/RHC	5268	23.5	1.20 [0.90, 1.59]	0.98 [0.80, 1.20]
Rurality	Downstate	27456	26.9	-Reference-	-Reference-
	Upstate rural	8540	24.8	0.82 [0.67, 0.99]	0.93 [0.82, 1.07]
	Upstate urban	25935	29.1	1.02 [0.88, 1.17]	1.05 [0.95, 1.16]
Vaccinated in last 2 years	Unvaccinated	36974	10.6	-Reference-	-Reference-
	Vaccinated	24957	51.3	4.68 [4.51, 4.85]	4.27 [4.13, 4.40]
Study Arm	Autodialer	19003	28.0	1.06 [1.02, 1.10]	1.06 [1.02, 1.10]
	Text	19090	27.6	1.04 [1.00, 1.08]	1.03 [0.99, 1.07]
	Mail	4779	26.8	1.01 [0.95, 1.07]	1.00 [0.94, 1.06]
	Usual care	19059	26.6	-Reference-	-Reference-
Colorado
Age category	6 months – 1.9 years	1,952	59.5	2.43 (2.24, 2.63)	2.15 (2.02, 2.30)
	2 – 5.9 years	5,810	35.9	1.47 (1.32, 1.63)	1.16 (1.10, 1.23)
	6 – 10.9 years	6,534	29.3	1.20 (1.11, 1.30)	1.15 (1.10, 1.20)
	11 – 17.9 years	9,549	24.5	-Reference-	-Reference-
Practice type	Family Medicine	6,451	27.7	-Reference-	-Reference-
	Pediatrics	15,071	33.0	1.19 (0.99, 1.44)	1.17 (1.06, 1.28)
	CHC/RHC	2,323	31.6	1.14 (0.92, 1.42)	1.17 (1.02, 1.34)
Rurality	Metro	11,083	35.3	-Reference-	-Reference-
	North	5,431	30.5	0.86 (0.72, 1.03)	1.02 (0.92, 1.13)
	South	7,331	26.3	0.75 (0.63, 0.88)	0.88 (0.79, 0.99)
Vaccinated in last 2 years	Unvaccinated	13,388	13.8	-Reference-	-Reference-
	Vaccinated	10,457	54.0	3.91 (3.56, 4.29)	3.71 (3.40, 4.05)
Study Arm	Autodialer	7,910	32.9	1.10 (1.05, 1.15)	1.08 (1.03, 1.12)
	Mail	8,007	31.5	1.05 (1.01, 1.10)	1.06 (1.02, 1.10)
	Usual care	7,928	29.9	-Reference-	-Reference-

vaccination by patient characteristics and study arm.

Overall Interaction p-values:

New York: Age * study arm p=0.03; Practice type * study arm p=0.92; Region * study arm p=0.34; Prior 2yr vaccination * study arm p= 0.10

Colorado: Age * study arm p=0.83; Practice type * study arm p=0.01; Region * study arm p=0.63; Prior 2yr vaccination * study arm p= 0.85

The costs per child randomized were $0.28 for autodialer, $0.24 for text and $1.76 for mail (Table 3). Compared with the control arm, the incremental cost per additional vaccine delivered was $20 for autodialer and $24 for text arms, but with no observed statistically significant difference in vaccination rates between text and control arms. The incremental cost for the mail arm was $869 (more costly but not more effective than the control arm).

Table 3:

Cost of centralized reminder-recall

Type of Cost	New York				Colorado
	Autodialer (n=19,003)	Text (n=19,090)	Mail (n=4,779)	Control (n=19,059)	Autodialer (n=7,910)	Mail (n=8,007)	Control (n=7,928)
Personnel	$2,473	$2,213	$2,133	$0	$1,683	$1,670	$0
Other	$2,753	$2,395	$6,265	$0	$2,195	$12,131	$0
TOTAL	$5,226	$4,608	$8,398	$0	$3,878	$13,801	$0
Cost per child randomized	$0.28	$0.24	$1.76	$0	$0.49	$1.72	$0
Percent vaccinated	28.0%	27.6%	26.8%	26.6%	32.9%	31.5%	29.9%
Incremental cost per additional vaccination	$20	$24	$879	-	$16	$108	-

Colorado:

Vaccination rates were higher in the autodialer arm (by 3.0 percentage points, P < .001 in adjusted model) and the mail arm (by 1.6 percentage points, p=0.01 in adjusted model) than the control arm (autodialer-32.9%, mail-31.5%, control-29.9%). In adjusted models, the probability of vaccination was significantly increased compared with the control arm [ARR (95%CI) for autodialer 1.08 (1.03, 1.12), and for mail 1.06 (1.02, 1.10)]. The effect of the intervention did not differ by age (interaction effect p=.86), practice type (p=.27), region (p=.37) or prior vaccination (p=.70). Children <11 years (compared with older subjects) and children who received an influenza vaccine in the prior year (compared with those who did not) were more likely to receive a vaccine during the study season.

The costs per child randomized were $0.49 for autodialer and $1.72 for mail (Table 3). Compared with the control arm, the incremental cost per additional vaccine delivered was $16 for autodialer and $108 for mail arms.

Both States:

We performed two post-hoc analyses to assess whether the intervention improved the timeliness of influenza vaccination. Table 4 (available at www.jpeds.com) shows unadjusted and adjusted risk ratios vaccination by 8 weeks after the start of the study in each state. Compared with the findings for end-of-season vaccination, findings were identical in NY but showed a slightly greater impact in CO. Figure 2 shows time-to-event analyses including Kaplan-Meier curves and smoothed hazard functions to assess differences in timeliness of vaccination by study arm. In NY, the autodialer arm had higher likelihood of vaccination in the first 8 weeks than the control arm. In CO both the autodialer and mailed arms had higher vaccination rates in the first 8 weeks than the control arm.

Figure 2:

Time-to-event analysis of influenza vaccinations

Cumulative incidence curves A and C above show the cumulative percent of study subjects vaccinated over time in NY (p=0.004) and CO (p<0.001), with final vaccination rates among children not already vaccinated at the start of the study ranging from 26–33%. Smoothed hazard plots B and D show the probability of an unvaccinated subject receiving a vaccination at any given time through the study period. The peak vaccination time for both states occurred shortly after the start of the intervention in October, November, and early December, with instantaneous vaccination rates decreasing through the remainder of the season apart from a small uptick around late January. Differences between study arms were observed primarily during peak vaccination time in the first 8 weeks of the study; at the end of 8 weeks adjusted RR in NY for autodial, text, and mail were 1.06 [1.01, 1.11], 1.04 [0.99, 1.09], 0.90 [0.83, 0.98] respectively and in CO for autodial and mail arms were 1.13 (1.08, 1.19) and 1.10 (1.05, 1.15) respectively.

Finally, we performed a post-hoc analysis utilizing cell phone-scrubbing software to identify landline vs. cell numbers; this did not change the study findings.

Discussion

Our study tested IIS-based C-R/R for influenza vaccine across large populations and also in comparing different modalities of C-R/R (autodialer, text messaging, and mailed reminders versus controls). We found that IIS-based C-R/R messages sent by autodialer calls had a modest impact on raising influenza vaccination coverage in both NY and CO, and that the impact of autodialer calls in both states occurred largely during the first 8 weeks after the initial call. IIS-based C-R/R sent by text message did not raise influenza vaccination rates in NY. Mailed C-R/R messages had no effect in NY and had a small effect in CO, again primarily during the first 8 weeks.

A Cochrane systematic review noted that R/R was effective in increasing influenza vaccination rates in children.⁵ However, only one pediatric study focused on healthy children rather than children with high risk conditions, and this study noted only a 4.4% point improvement for children 6 to 23 months of age.¹⁶ One trial of text reminders for healthy children of all age groups conducted in 2012 across four New York City practices noted an absolute difference of 3.7% points between text message and no-reminder control arms.³¹ Thus, the expected effect size for C-R/R for influenza vaccine for the entire child population, whether practice-based or centralized, might be small. Importantly, a recent CDC modeling study of adult plus child influenza vaccination noted that a 5% point improvement in vaccination rates would substantially reduce population-wide influenza morbidity.²⁹ If an intervention could be scaled up widely, even an impact of 2 to 3 percentage points would have a significant public health benefit.

We evaluated 3 common R/R modalities. Autodialer R/Rs can be scaled-up widely and are low-cost. Text messages might have greater sense of urgency or importance than phone calls and are scalable and low-cost. Mailed R/Rs might be perceived as more important and can remain in patients’ homes as a continuous reminder. One prior study noted that mailed reminders were more effective though also more costly than autodialer reminders;³² however, that study focused only on low-income adolescents. Because mailed reminders are more costly, they are less scalable than phone or text reminders.

Unlike some other studies,^21,31,33 text messages had no benefit in NY whereas autodialer reminders had a small benefit. Perhaps frequent texting in everyday life diminished the impact of text message reminders. In addition, leaving messages on answering machines might cause autodialer messages to have greater impact than text messages. More study is needed to compare the relative impact of different reminder modalities.

For any R/R to be effective, phone or mail contact information must be accurate. The accuracy of IIS-based contact information is unclear, and the relative accuracy of phone numbers versus mailing addresses is unknown; both depend on data uploads and corrections from immunization providers. Our trials were pragmatic because we did not improve on contact information but rather used existing IIS-based data. Optimizing IIS-based contact information might improve the impact of IIS reminders.

One important issue that has not been well studied is whether R/R, either practice-based or C-R/R, might be less effective today than in the past. The Cochrane review includes R/R studies since 1974, and did not formally test time-trends in effectiveness of R/R. We speculate that there may be some current challenges to IIS-based C-R/R for influenza vaccination. Because of state requests and also requirements of the telephone company, our autodialer calls emanated from an 800 number rather than from the child’s pediatric practice phone number. Some parents might automatically ignore or delete calls from unfamiliar sources. It is also possible that the flood of telemarketing calls,³⁴ now experienced by many individuals, might diminish the potential impact of health-related autodialer reminders. One report noted that there were 3.4 billion robocalls made during one month in 2018, representing >10 calls per US resident that month.³⁵ Our autodialer calls might have been ignored by many parents due to these concerns.

Underreporting to IISs of childhood influenza vaccination might limit our ability to observe the full impact of the intervention. Underreporting by practices might occur if they do not upload electronic data directly to the IIS or offer cash-only flu vaccine clinics without entering the vaccine into the electronic medical record. Our vaccination coverage was markedly lower than levels reported by the National Immunization Survey and Behavioral Risk Factor Surveillance System (BRFSS) combined data² which showed >60% coverage in both states (based on self-report without verification for influenza vaccine) but which might overestimate rates.^36,37 One reason is that we excluded children who had been vaccinated prior to the start of the intervention (October 18 in NY, October 2 in CO). Also, we included both NY, a mandatory reporting state, and CO, a non-mandatory reporting state in which many children receive flu vaccination in pharmacies (where IIS underreporting may be more common). Yet the impact of IIS-based C-R/R was similar in both states. Unfortunately, we cannot quantify the degree of underreporting or its impact on our study. Linkages with insurance/Medicaid databases or billing exchanges, and encouraging providers to use CMS meaningful use standards might improve reporting to state IISs.

Another factor that may have limited the impact of our intervention is influenza vaccine hesitancy due to parent concerns about sub-optimal influenza vaccine effectiveness, vaccine safety, or general vaccine hesitancy. Vaccine hesitancy for influenza is well described^23,25,38 although the prevalence is unknown in either these states or nationally. More intensive Interventions are needed to address vaccine hesitancy.

As expected, we found that younger children and those previously vaccinated were more likely to be vaccinated during our study, but the effect of the intervention didn’t vary consistently by these two factors. For older children, and those not previously vaccinated, practice-based interventions or other options such as school-located influenza vaccinations²⁶ or vaccination in community settings such as pharmacies⁹ should be considered.

Most of the effect of the autodialer and (for CO) mail reminders occurred during the first 8 weeks; for these study arms the intervention raised overall vaccination rates and also shifted vaccinations earlier. This might be beneficial because the benefit of the vaccine is optimal >2 weeks after delivery, and because the onset of influenza epidemics is variable across years. Also, the unexpected bump in vaccination rates around January for all groups (including controls) may have been due to media reports of influenza disease.^39-41

The cost of centralized R/R in this study was approximately $0.28 to $0.49 per child reminded by autodialer or text. Our estimates are most similar to a previous study that calculated the mean cost of IIS-based C-R/R using an autodialed method as $0.53 per contact.¹⁴ A study of C-R/R for a relatively small adolescent population reported $0.78 per adolescent sent a reminder.⁴² The incremental cost per additional vaccine delivered comparing the autodialer arms to the control arm were relatively consistent across both NY and CO, i.e., an estimated additional cost of $20 (NY) or $16 (CO) to deliver an additional influenza vaccine in the autodialer arm. Our cost-effectiveness finding is similar to the cost per child for any preschool immunization for children 19–35 months of age using IIS-based C-R/R.¹⁴ These cost-effectiveness estimates could help decision makers who face competing implementation alternatives and fixed budgets. Some health systems or practices, particularly those receiving additional reimbursement for influenza vaccination rates, might consider such costs worthwhile.

Our study has both strengths and limitations. The use of pragmatic trials across 2 states enhance generalizability. The study design (randomization within practices and large numbers of practices and patients) enhances internal validity by allowing us to control for multiple potential confounders. Limitations include an inability to use practice telephone numbers that might have been recognized by parents in the autodialer or text message arms, a potential underreporting to the IISs of updated contact information and influenza vaccination that would blunt our ability to detect an intervention effect, and data from only two states. In addition, some other R/R studies^31,43,44 have employed more reminder messages than the three we sent; we limited to three reminders based upon prior input from state leaders and parents. We received very few calls or letters from parents objecting to the reminders, although a small number opted out of future autodialer, text, or mail reminders in NY (1%, 2%, and <1%) or CO (3%, 1%, and 1%). Of note, there were no adverse outcomes requiring reporting to the IRBs.

In conclusion, our pragmatic trials lend support to centralized reminder-recall for influenza vaccination among children, particularly using autodialer or mailed reminders although autodialer reminders are more cost-effective. Our findings do not support text message reminders for IIS-based C-R/R. Because of its scalability and potential low cost, and evidence that even small improvements in influenza vaccination rates at the population level could prevent substantial morbidity, IIS-based C-R/R remains a viable option for raising influenza vaccination rates. However, more intensive interventions are needed to increase substantially the U.S.’s rates of childhood influenza vaccination.