Centralized Reminder/Recall for Human Papillomavirus Vaccination: Findings From Two States—A Randomized Clinical Trial

Abstract

Purpose:

Centralized reminder/recall (C-R/R) using Immunization Information Systems has been effective in increasing childhood immunization rates. Previously, C-R/R using autodialer for human papillomavirus (HPV) vaccine did not raise rates. We assessed C-R/R for HPV vaccine using other modalities and focused on younger adolescents.

Methods:

We conducted a three-arm pragmatic RCT in randomly sampled primary care practices in Colorado (n = 88) and New York (n = 136), proportionate to where adolescents received care. We randomized, within practices, adolescents aged 11-14 years who had not completed the HPV vaccination series to receive C-R/R using different modalities (Colorado: autodialer, mail, or control; New York: autodialer, text, or control). Up to two reminders were sent in intervention arms for each dose needed between 2/2017 and 12/2018.

Results:

In Colorado, no significant differences were found for series initiation (31.3% control, 31.1% autodial, 31.8% mail), with slight improvement for series completion in the autodialer arm (29.7% control, 31.1% autodialer, p = .04) but not the mail arm (30.9%, p = .06). No significant differences were found in New York for series initiation (24.1% for all arms) or completion (17.1% control, 16.9% autodial, 17.9% text). Adjusted analyses showed higher completion rates for the autodialer arm in Colorado but not for other arms. In Colorado, C-R/R reduced time to series completion by around 2 months. Cost per adolescent was $1.81 for mail; under $.40 for all other modalities.

Conclusions:

C-R/R has less benefit for raising HPV vaccination rates than other studies have noted for childhood immunizations, although it may quicken series completion at little cost.

Human papillomavirus (HPV) is responsible for an estimated 33,700 new cancer cases [1] and nearly 4,000 cancer-related deaths in the U.S. annually [2]. HPV infection causes the majority of cervical, anal, and oropharyngeal cancers and large proportions of vulvar, vaginal, and penile cancers [3-6]. Existing HPV vaccines are highly effective at reducing new infections, but are underutilized [7]. Although routinely recommended by the Advisory Committee on Immunization Practices for all adolescents beginning at age 11 years [8], in 2019 only 54% of US adolescents 13–17 years of age were up-to-date for HPV vaccinations and only 71% had received an initial vaccination [9].

Reminder/recall (R/R), in which parents or adolescents are reminded about vaccinations that are coming due (reminder) or past due (recall), is recommended by the Task Force on Community Preventive Services based on strong evidence of effectiveness in increasing vaccination rates [10] and was found to be a highly successful method for increasing adolescent vaccines in a recent Cochrane review [11]. R/R can be conducted by individual practices or in a more centralized manner for many practices or for entire geographic areas. Centralized approaches to R/R (C-R/R) can be initiated by state public health departments utilizing a state or regional Immunization Information System (IIS), or immunization registry to identify adolescents missing recommended vaccines. Several studies have demonstrated that C-R/R is more effective and cost-effective than practice-based R/R for increasing childhood vaccine series rates [12-14]. In addition, IISs across the country report using C-R/R for many vaccines, albeit usually without rigorous, controlled evaluation of effectiveness, for vaccines beyond the routine childhood vaccine series [15,16].

Our research group previously conducted a pragmatic randomized controlled trial to assess the effectiveness of C-R/R for increasing HPV vaccination rates using an autodialer to make automated telephone calls, for adolescents aged 11–17 years in Colorado and New York [17]. Results of this trial were disappointing, with only minimal effects in Colorado and no effects in New York. However, we hypothesized that focusing the intervention on younger adolescents, who may be less likely to have refused the vaccine previously, and using multiple outreach modalities (mail and text message rather than autodialer) might result in effect sizes more in line with those seen for routine childhood vaccines [12-14] and for HPV vaccine R/R done by practices or health systems [18-24]. The objective of the current multistate pragmatic RCT was to test the effectiveness of IIS-based C-R/R for increasing HPV vaccination among adolescents aged 11–14 years due or past due for a dose of HPV when sent via mail, autodialer, and text message. In addition, we compared cost associated with the different methodologies.

Methods

This study was approved by the Institutional Review Boards at UCLA, New York State Department of Health, University of Colorado, and the Colorado Department of Public Health and Environment.

Setting and population

In Colorado, we targeted practices in eight urban counties along the Front Range. Rural counties were not included because of low numbers of rural practices and because vaccination delivery is substantially different in rural areas of Colorado. Denver County was excluded because many practices there were participating in other local projects aimed at increasing HPV vaccination rates. In New York, all 57 counties outside of New York City were targeted. New York City was excluded due to use of a different IIS than New York State. New York counties were grouped into three sections (Upstate Urban, Upstate Rural, Downstate), and representativeness of the sampled practices was evaluated.

Pediatric, Family Medicine, and Community/Rural Health Center practices were randomly selected from IIS data by the study team as the primary sampling unit. In Colorado, practices were randomly selected to be proportionally representative by practice type of where adolescents were seen for primary care. In New York, practice sampling was stratified by practice type, and practices were selected from each stratum proportional to the size of the practice. Practices participating in other C-R/R projects were excluded. Patients aged 11-14 years who, according to the IIS, had not completed their HPV vaccination series at baseline were randomly selected within each practice, proportional to practice size as determined by the number of adolescents 11–14 years at the site. For families identified by phone number in the IIS, only one sibling was included. Owing to potential issues of patients not being inactivated within the IIS, we excluded adolescents who did not meet certain vaccination record criteria. Patient and practice eligibility and exclusion criteria are outlined in Figure 1.

Figure 1.

Consort Diagram plus practice and patient eligibility criteria.

Colorado and New York IIS

In both states, practices routinely send vaccination and demographic information to the state IIS via electronic transfer from practice EHRs or via direct data entry into a web-enabled application. Colorado’s IIS (CIIS) does not mandate immunization reporting, yet >99% of <6-year-olds, 95% of 6- to 10-year-olds, and 80% of 11- to 17-year-olds had at least two immunization records in CIIS. New York State’s IIS (NYSIIS) does mandate that all vaccinators send child vaccination data to NYSIIS. The number of children <6 years of age with ≥2 immunization administrations in NYSIIS was approximately equal to the official US Census count of children <6 years, and the number of adolescents with ≥2 immunization administrations in NYSIIS was 97% of the Census count of adolescents in New York State outside of New York City. Contact information in both IISs can be updated by the child’s primary care office (generally at the time of vaccination), but neither IIS identified a systematic approach for updating contact information without vaccination administration. Both CIIS and NYSIIS participated in previous population-level C-R/R trials, including one on HPV vaccine [12-14,17,24,25].

Intervention

Within each practice, selected patients were randomized 1:1:1 to either the control group (usual care, no C-R/R) or one of the two intervention groups (C-R/R via different modalities). In Colorado, C-R/R was delivered by mail or autodialer call; in New York, it was delivered by SMS text message or autodialer call. Text message C-R/R was not possible in Colorado due to regulatory issues; mailed C-R/R for HPV had been tested previously in New York. [24].

C-R/Rs were sent every two months between February 2017 and December 2018. Patients in the intervention groups received up to two reminders for their initiation and completion doses of the HPV series, if they were due for a vaccine at the time of the C-R/R event. Following the HPV vaccination schedule recommended by ACIP in 2017 [26] C-R/Rs were sent to adolescents 11–14 years of age who either had not initiated the series or had not received a second dose six months post-receipt of the initiation dose. For example, a previously unvaccinated patient who received a message in 02/2017 could receive the first dose and subsequently receive a recall message regarding a second dose six months later. Vaccine receipt was tracked via IIS data pulls conducted one week prior to each C-R/R event. Outcomes (HPV vaccinations within the IISs) were assessed via a final data pull in March 2019.

Message content was similar across all C-R/R modalities and developed using the Health Belief Model framework [27] (Appendix Figure A1). Messages informed parents/guardians that their child was due for an HPV vaccine dose and to contact their provider or health department for further information. Messages were kept at an eighth-grade reading level and did not mention any other vaccinations. Messages were addressed from the State Health Department and included the adolescent’s practice information which was on file with the IIS. In Colorado, providers could decline to have their practice name included on the messages prior to project launch, but could not keep their adolescents from reminded; those adolescents would receive R/R messages with only the health department’s information. In New York, practices could choose not to participate and not have their adolescents included in the trial altogether; however, if they participated they could not opt out of including their name on messages. Although we could not know language preference at the time of sending, messages were sent in both English and Spanish and included ways for parents to opt out of future C-R/R via phone, email, and text. A single vendor (Teletask; www.teletask.com) was used for sending autodialer and text messages. As our analysis framework was intention-to-treat, we included randomized patients in the text arm regardless if their phone on record at the IIS was a cell phone or land line. Teletask provided reports on message delivery success. The Colorado study team mailed postcards, and postcards returned with incorrect addresses were collected. Both were used to calculate reach.

Outcome measures

The primary study dual outcomes were IIS-based documentation of HPV vaccine initiation or completion within the study timeframe (February 2017 through March 2019). We assessed vaccinations from the IISs three months after the final C-R/R event to allow for vaccination to occur and for immunization data to be uploaded into the IIS.

We assessed a number of secondary outcomes. Since HPV vaccination is most effective if completed before first sexual contact, we examined if C-R/R was associated with a higher rate of HPV series completion before age 13. Also, because the second dose is recommended sometime between 6 and 12 months after the first dose, we sought to determine if C-R/R shortened the time between initiation and completion. We also assessed the primary outcomes in relevant subgroups including age (11-12 vs. 13–14), gender, and practice type (pediatrics, family medicine, community/rural health center), as these subgroups could have varying impact from the intervention. As practice reporting into IIS is variable based on what data practices collect, we assume that gender is sex-at-birth, but we could not confirm this. We estimated intervention costs by summing the costs related to: consensus building and preliminary work (e.g., meetings with local physician groups and public health to determine feasibility, efforts to tell providers of the intervention); training; software; collaboration; implementation meetings; and reminders. We used the viewpoint of the state IIS when calculating cost measures. Activities that were deemed to have been done only for research purposes, and so would not be done by an IIS, were not included.

Data analysis

The primary analysis compared the effectiveness of IIS-based C-R/R, sent by autodialer and mail (Colorado) or autodialer and text (New York), compared with no-C-R/R, in increasing initiation and completion of the HPV vaccine series. The study was powered to detect an absolute difference of 2% in series completion between the intervention and control arms. We used generalized linear mixed modeling to assess the impact of C-R/R on receipt of HPV vaccine with a random intercept for practice to account for correlation between patients seen at the same location. A log-binomial model was used to obtain adjusted and unadjusted risk ratios [28]; in New York, a failure of convergence of the log-binomial model was addressed by instead applying a Poisson model with a robust variance estimator [29]. Adjusted models controlled for covariates including patient age, gender, and physician specialty. These covariates have been noted previously to affect vaccination rates and could affect the response to our intervention [9]. We also tested interactions between each covariate and study arm to determine whether there were differential intervention effects. Significance level was set a priori to p = .05. We employed intention-to-treat analyses. All analysis was conducted using SAS, version 9.4 (SAS Institute, Cary, NC).

Secondary analyses included Cox proportional hazards regression on age at vaccination to assess differences by study arm within each state, with standard error estimates adjusted for clustering of patients within practices by using a robust covariance matrix estimator or using bootstrap resampling by practice. Patients not initiating or completing the vaccine series by the conclusion of the study were treated as censored. The exclusion of patients who had initiated or completed prior to the start of the study was accounted for using age at study start as the left-truncation time. Time-to-event analyses were performed in R using the ‘survival’ package [30].

Cost analyses accounted for personnel time and resources to plan and send reminders, and varying costs associated with autodialer, text, and mail C-R/R. Costs were categorized as start-up or implementation. We reported total cost, the cost per adolescent randomized, and cost per HPV vaccine administered within each arm. Cost-effectiveness using the cost per additional vaccine received were planned if clinical findings indicated a significant signal.

Results

A total of 32,283 adolescents from 88 practices in Colorado and 37,003 adolescents from 136 practices in New York were included in the study. Patient and practice characteristics are shown in Table 1.

Table 1.

Characteristics of primary care practices and adolescents

Characteristics of practices and patients	Colorado N (%)	New York N (%)
Practice characteristics
# recruited for the study	88	136
Practice type
Pediatric	42 (48%)	95 (70%)
Family medicine	34 (39%)	31 (23%)
Community health center	12 (14%)	10 (7%)
Practice location
Downstate	N/Aa	57 (42%)
Upstate rural		22 (16%)
Upstate urban		57 (42%)
Inclusion of practice name on R/R	83 (94%)	N/Ab
Patient characteristics	32,283	37,003
Total patients
H to <13y	22,360 (69%)	27,193 (73.5%)
13 to <15y	9,923 (31%)	9,810 (26.5%)
Gender
Female	15,486 (48%)	17,721 (48%)
Male	16,797 (52%)	19,282 (52%)
Total patients who opted-out of R/Rc	168 (1%)	617 (1.7%)
Percentage of missing phone numbersc	1,879 (6%)	84 (.2%)

^aAll practices in Colorado were located in urban counties.

^bPractices in New York were not allowed to opt out including the practice name.

^cThese patients were still included in the intention-to-treat analysis. Those who opted out did not receive further R/R.

Effectiveness of C-R/R for HPV vaccination

Both initiation and completion rates for all adolescents in the study population were low, as adolescents who had previously completed the series were not included in the study. In Colorado, 13% of autodialer calls were deemed undeliverable and 19% of postcards were returned with an incorrect address. In New York, 20% of autodialer calls and 3% of text messages were deemed undeliverable. Figure 2 shows initiation and completion rates by state at the end of the trial. Initiation rates were identical across all three study arms for both states (Colorado: 31.3% control vs. 31.1% autodialer, p = .69; vs. 31.8% mail, p = .31; New York: 24.1% control vs. 24.1% autodialer, p = .45; vs. 24.1% text, p = .91). In Colorado, there was a small difference in series completion for the autodialer arm (29.7% control vs. 31.1% autodialer; p = .04), but not for the mail arm (30.9%, p = .06). There was no significant difference in rates for completion across study arms for New York (17.1% control vs. 16.9% autodialer, p = .27; vs. 17.9% text, p = .14).

Figure 2.

(A) Vaccine series initiation/completion rates in CO at the end of study; *Indicates statistically significant difference, p = 0.04. (B) Vaccine series initiation/completion rates in NY at the end of study.

Table 2 shows the results of a multivariable mixed model, both unadjusted and adjusted. While the log-binomial model results are presented for Colorado, convergence failed for New York; thus, New York results reflect the modified Poisson model. Colorado rates for initiation did not show any differences between intervention and control arms, but completion rates were slightly higher for the autodialer intervention arm compared to control arm (risk ratio 1.04; 95% confidence interval: 1.004–1.06). New York showed no differences between either intervention arms or control arm for both initiation and completion.

Table 2.

Multivariable mixed model showing unadjusted and adjusted risk ratios for HPV vaccine series initiation and completion by study group (control vs. autodial vs. mail/text)

Unadjusted and adjusted risk ratios for vaccine initiation and completion by study arm
Study group	Colorado			New York
	%	Unadjusted risk ratio	Adjusted risk ratioa	%	Unadjusted risk ratio	Adjusted risk ratioa
HPV vaccine initiation
Control	31.3	reference	reference	24.1	Ref	Ref
Autodial	31.1	.99 (.94–1.04)	.99 (.94–1.04)	24.1	.97 (.92–1.01)	.97 (.92–1.01)
Mail	31.8	1.02 (.97–1.07)	1.02 (.98–1.07)	N/A
Text	N/A			24.1	1.00 (.96–1.05)	1.00 (.96–1.05)
HPV vaccine completion
Control	29.7	reference	reference	17.1	Ref	Ref
Autodial	31.1	1.04 (1.004–1.09)	1.04 (1.003–1.09)	16.9	.95 (.90–1.00)	.95 (.90–1.01)
Mail	30.9	1.04 (.998–1.08)	1.04 (.999–1.08)	N/A
Text	N/A			17.9	1.06 (1.00–1.11)	1.05 (1.00–1.11)

^aBold font indicates statistically significant finding.

Table 3 shows series initiation and completion rates within the specified age, gender, and practice type subgroups. In adjusted analyses, there was a lower rate of initiation among 13- to 14.9-year-olds compared with 11- to 12.9-year-olds in both states, and a lower rate of completion among older adolescents in Colorado. Males had a significantly lower HPV series completion rate in both states, and a lower series initiation rate in Colorado. However, in both states, the intervention effect did not differ significantly by any subgroups as evidenced by the lack of significant estimates for interaction terms.

Table 3.

HPV vaccine series initiation and completion rates by select patient and practice characteristics

Patient and practice features		Vaccine initiation			Vaccine completion
		N (initiation)	%	Adjusted RR (95% CI)a	N (completion)	%	Adjusted RR (95% CI)a
Colorado
Age (years)	11.0–12.9	14,599	34.6	Ref	22,360	32.6	Ref
	13.0–14.9	5,920	23.6	.72 (.69–.76)	9,923	25.9	.84 (.81–.87)
Gender	Female	9,718	32.3	Ref	15,486	31.6	Ref
	Male	10,801	30.7	.95 (.92–.99)	16,797	29.6	.94 (.91–.97)
Practice type	Fam. Med	3,849	30.2	Ref	5,955	27.5	Ref
	Pediatrics	14,880	31.0	1.06 (.93–1.21)	22,975	31.0	1.11 (.94–1.31)
	CHC/RHC	1,790	37.4	1.20 (.99–1.45)	3,353	32.8	1.16 (.92–1.47)
New York
Age (years)	11.0–12.9	22,118	25.6	Ref	27,193	17.5	Ref
	13.0–14.9	7,612	19.6	.83 (.79, .87)	9,810	16.9	1.00 (.96, 1.06)
Gender	Female	14,070	24.1	Ref	17,721	18.0	Ref
	Male	15,660	24.1	1.01 (.97, 1.05)	19,282	16.7	.93 (.89, .97)
Practice type	Fam. Med	3,262	21.2	Ref	4,210	16.9	Ref
	Pediatrics	24,814	24.4	1.17 (.91, 1.51)	30,417	17.0	.97 (.72, 1.33)
	CHC/RHC	1,654	24.8	1.53 (.99, 2.34)	2,376	22.3	1.65 (.97, 2.78)

CI = confidence interval; CHC = Community Health Center; RHC = Rural Health Center; RR = risk ratio.

There were no significant differences by study arm.

^aThe adjusted estimates are adjusted for all other covariates in the table, plus study arm. Bold font indicates statistically significant finding.

Time-to-event analyses showed no significant effects on initiation in either state. In Colorado, Cox proportional hazards regression analysis showed that both treatments significantly increased the rate of completion relative to control (p = .02 for each intervention arm vs. control). The estimated median age at completion was accelerated by approximately 2 months in each of the intervention arms (14.04 years in control, 13.89 years in the autodialer arm and 13.85 years in the mail arm). In NY, there was no significant difference in rate of completion between the study arms. Among those who initiated the HPV series during the study, the lower quartile of the time-to-completion distribution was estimated to be accelerated relative to control by 1.45 months for mail (p = .03) and 1.61 months for autodial (p < .01); no significant differences in the median were observed (estimated to be approximately 1.1 years post-first dose for each treatment arm). In New York, no significant acceleration was observed.

Cost

Costs for the intervention can be seen in Appendix Table A1. Autodialer and text total cost were lower than mail; total cost for mail (Colorado only) was $19,717; total cost for autodialer was $3,717 in Colorado and $3,421 in New York; total cost for text (New York only) was $3,331. While upfront costs were relatively comparable, mail intervention costs were far greater than both text and autodialer intervention costs. Cost per child in Colorado was $0.35 for autodialer and $1.83 for mail. In New York, cost per child was $0.27 for autodialer and $0.28 for text. In Colorado, cost per vaccine received was $0.68 for autodialer and $3.58 for mail. In New York, cost per vaccine received was $1.22 for autodialer and $1.18 for text. Because there was no clear evidence for C-R/R effectiveness, cost-effectiveness, looking at differential costs for outcomes observed, was not computed.

Discussion

This three-armed pragmatic RCT aimed to expand on our previous study of autodialer C-R/R [17] by studying mail and text message C-R/R modality and also by focusing on a younger age cohort for which we hypothesized we might have a larger impact. Results of the present study were similar to the previous study [17] with only small positive impact on HPV vaccine completion rates observed in the autodialer arm of one state. We observed no increases in vaccination rates from mail or text message R/R. With absolute differences in vaccination rates under 2% for all arms, the evidence for the role of IIS-initiated C-R/R in increasing either HPV vaccine initiation or completion is not compelling. It did not appear that expanding outreach modalities for HPV C-R/R or targeting the younger age group among adolescents resulted in the kind of vaccination increases that have been found for IIS-based C-R/R for childhood vaccines [12-14].

Mail C-R/R was unsurprisingly more expensive than automated methods (text and autodial), as postage and printing costs keep mailed C-R/R prices higher and do not allow for economies of scale enabled for by automated options. However, due to regulatory and legal restrictions, some IISs are not able to perform C-R/R using an automated approach [15,16], so mailed reminders might be the only choice in these settings. Costs were comparable to those in our previous studies [12,13,17,25].

Although largely negative, our findings are an important addition to the literature, given the amount of effort currently being devoted to C-R/R efforts for HPV. For example, a recent national survey and interviews of IIS managers demonstrated that HPV was the second most common target for C-R/R efforts by health departments [15,16]. While there are many factors for IISs to consider when choosing which vaccines to target, and differences in other states’ populations could produce different results, these findings along with findings from our previous study suggest there should be skepticism about the effectiveness of C-R/R for increasing HPV rates on a population level. All things being equal, IIS managers may wish to better target their limited resources. Limited findings from Colorado suggest that if resources are spent on HPV C-R/R, focusing them on completion doses rather than initiation doses may help in speeding the rate of completion by up to 2 months. Others have found similar benefits for completion for family-focused interventions [31]. However, this may not be generalizable outside of Colorado, as evidenced by the lack of effect in New York.

Potential reasons for our largely negative findings include the possibility that participating practices conducted R/R for HPV vaccine, although several studies indicate that overall, practice-initiated R/R is not common [32,33]. Incomplete immunization information (unreported HPV doses) within the IIS is certainly possible, which would bias estimated rates downward, although this would be expected to occur equally across treatment groups. Another aspect of IIS data quality [15,16] is outdated adolescent contact information within the IISs; this could play a bigger role in C-R/R effectiveness for adolescents who, compared to younger children, attend fewer primary care visits where they might receive vaccines and have contact information updated in the IIS. While we reported the low rate of undeliverable contact across all modality, there is no way of knowing that those R/R messages that were delivered reached the intended person as we were not able to follow up with individuals. “Correct” autodialer and text outreach simply indicated that the R/R attempt reached a working number. Likewise, postcards that were delivered did not necessarily reach the intended person.

Our results put to rest one hypothesis as to the lack of effectiveness found in our previous HPV trial, that C-R/R delivered by autodialer may be losing its effectiveness because of the overload of autodialer messaging and spam calls, as seen in other industry reports [34,35]. In the present study, effectiveness did not vary substantially by modality. C-R/R by mail or text was not found to be effective, despite evidence showing usefulness of both in non-IIS initiated R/R in select populations in New York [20,22-24,36].

Recent literature suggests that hesitancy about the HPV vaccine may be the largest barrier to vaccination and the success of C-R/R [37]. While there is some disagreement around the definition of “hesitancy” in the literature [38-40], reminding parents may not be effective if parents have significant underlying concerns about the vaccine. Studies have shown that the reasons for HPV vaccine hesitancy include lack of a provider recommendation, misinformation about the vaccine, and concerns over vaccine safety or efficacy [41-44]. A recent survey in a nationally representative sample of US parents demonstrated an overall hesitancy rate of about 23% for HPV, with safety being a major source of concern. In that survey, being “hesitant” was highly correlated with previously having refused HPV vaccine due to concerns, suggesting that hesitancy is a major contributing cause to lower vaccination rates [37]. Given the population for this trial included only those adolescents who have not yet completed the HPV vaccine series, we feel that high current rates of vaccine hesitancy for HPV vaccine might be one reason for the lack of impact of our intervention. However, this hypothesis is not testable with our current trial. Future studies on R/R for other routine adolescent vaccines may help tease out any negative effect HPV vaccine hesitancy may have in C-R/R success.

This study has several strengths, including a large sample size and strong study design using within-practice randomization, which helps control for local factors affecting HPV vaccination rates and improved power to detect differences. Limitations include potential generalizability issues since the two included states may not well represent other states. The fact that all adolescents in the study had a primary care provider, according to IIS data, also somewhat limits generalizability since our intervention may have been targeting adolescents who were already getting recommendations for HPV vaccine and not following them. Inclusion of sociodemographic factors such as race/ethnicity and income was not possible given the data available in the IIS, but including these could allow for more targeted interventions. Future C-R/R studies might consider testing messages framed in different ways (e.g., using behavioral economic principles) to assess whether message content affects the impact of C-R/R.

In conclusion, we found that C-R/R done by an IIS for HPV vaccine may not be as promising of an intervention as it is for other vaccine series. The low cost and relative ease of conducting C-R/R still makes it attractive, but careful evaluation is needed to understand its vaccine-specific value.

IMPLICATIONS AND CONTRIBUTION.

Centralized reminder/recall (C-R/R) for HPV vaccine sent via autodialer, text, or mail from state Immunization Information Systems did not raise HPV vaccination rates. However, there is some indication that C-R/R may help hasten slightly the age at completion of the HPV series.