Supporting and Sustaining Centralized Reminder/Recall for Immunizations: Qualitative Insights from Stakeholders

Abstract

Centralized reminder/recall (C-R/R) is an evidence-based strategy for increasing vaccination rates that uses a population-level database such as a state immunization information system (IIS) to send notifications across large geographic areas. IISs are usually based in state public health departments, which could initiate C-R/R. While C-R/R is a promising strategy, the factors influencing its initiation and sustainment are not clear. Utilizing qualitative content analysis methodology and interviews with key stakeholders involved in or knowledgeable about C-R/R, we examined the characteristics of these initiatives and factors influencing their success. We identified and spoke with managers and senior leaders across IISs, health plans, health systems, pharmaceutical companies, and advocacy organizations and focused especially on C-R/R activities within IISs. Several considerations were determined important to C-R/R success: decision-making, stakeholder buy-in, partnerships, funding, data and technology, evaluation, and message content. Salient barriers were costs and lack of funding, poor contact data quality (i.e. telephone number, home address), and messaging that is either overly broad or too specific. Pertinent facilitators of C-R/R included notifying health providers in advance of an initiative, conducting a rigorous post-reminder/recall evaluation, and engaging a range of partners. Partnerships were important to stakeholders for multiple reasons including technical assistance, resource sharing, and sharing of best practices. Overall, our results illustrate the many opportunities to advance C-R/R through further collaboration within and across public health departments and potentially via public-private partnerships.

1. Introduction

Centralized reminder/recall (C-R/R) is an evidence-based strategy for increasing vaccination rates [1, 2]. It involves notifying individuals when vaccinations are due (reminders) or late (recalls), and enables messages to be sent via mail, auto-dial call, text, or other method across large geographic areas and to be targeted and tailored to the population. C-R/R relies on a population-level database such as an immunization information system (IIS), which compiles statewide or regional data on vaccinations administered by participating providers [3, 4]. IISs currently operate in all U.S. states, the District of Columbia, five U.S. cities, and several U.S. territories [5]. In 2017, 95% of children under six years old, 79% of adolescents between 11 and 17 years old, and 51% of adults 19 and older had a record in their state or city IIS [6].

Studies indicate that reminder/recall utilizing an IIS may be preferred and more sustainable when compared to other methods of reminder/recall [7-11]. Pediatric trials have shown IIS-based efforts are more effective and cost-effective at improving population-level childhood vaccination rates than incentivizing provider-based initiatives [7, 8, 12]. Likewise, at least one IIS-based reminder/recall effort has shown success at increasing vaccination rates among adults aged 65 and older, and at a reasonable cost [13].

Although C-R/R is a promising method for increasing vaccination rates, its uptake across public health departments has not been reported in the literature. Moreover, barriers to initiating C-R/R are not entirely clear. Researchers have recognized challenges associated with C-R/R, including poor data quality, high infrastructure costs, and legal and regulatory obstacles [14, 15]. In addition, some providers have expressed concerns about C-R/R such as potential negative responses from patients [16, 17]. But overall, C-R/R tends to be supported by providers [11, 16-18] as well as families and patients themselves [19-21]. As such, the barriers to C-R/R appear to be complex and not well understood.

In this study, we sought to describe the range of factors influencing C-R/R success, as experienced by those who operate or oversee the initiatives. Our research questions centered on: 1) how decisions to implement C-R/R are made and by whom, 2) how C-R/R is typically structured, and 3) the barriers to and facilitators of implementing and sustaining C-R/R. Our primary analytical focus was IIS-based C-R/R within public health departments, though we also interviewed health insurance plans and health systems, including accountable care organizations (ACOs); pharmaceutical companies (which have programs to assist in C-R/R); and advocacy organizations focusing upon immunizations or public health to supplement our core data from IISs and to glean best practices from other segments of healthcare.

2. Methods

This study involved qualitative interviews with key stakeholders knowledgeable about C-R/R. Our research was guided by qualitative content analysis methodology, a systematic yet flexible technique capable of offering rich insight into C-R/R sustainability [22]. Human subjects protection approval was granted by the Colorado Multiple Institutional Review Board as an expedited review with a waiver of written consent.

2.1. Sample and Recruitment

2.1.1. IIS Interviewees

Our team utilized a purposeful sampling strategy informed by a previously conducted survey of IIS managers across the U.S. and territories [23] and a subsequent screening questionnaire. First, we utilized the survey results to determine which IISs had conducted C-R/R (62%). Next, we emailed the screener questionnaire to an IIS manager at each of those IISs asking them to describe their C-R/R efforts, including the number, type, timing, and funding of the initiatives. Finally, screener responses facilitated the recruitment of a diverse sample of IIS managers from states with one of the following: 1) sustained C-R/R, meaning one or more current C-R/R projects with plans for continuation; 2) sporadic C-R/R, meaning one or more current C-R/R projects but no clear plans for continuation; and unsustained C-R/R, meaning one or more past C-R/R projects but no current C-R/R or plans for future C-R/R. We ensured broad representation of C-R/R efforts across vaccine type and funding source.

2.1.2. Non-IIS Interviewees

We purposefully recruited stakeholders at organizations other than IISs to gain insight into C-R/R activities outside public health departments and to learn about their activities and interests in potential public-private partnerships for C-R/R initiatives. Specifically, we contacted managers or senior leaders at health insurance plans and health systems, including ACOs (which together we refer to as “health plans and systems”); pharmaceutical companies; and advocacy organizations. These organizations and individuals were chosen based on their involvement in or knowledge of C-R/R and partnerships with public health departments for C-R/R, as determined through publicly available documentation and the research team’s knowledge.

Prospective interviewees were recruited through a combination of phone calls and emails, up to three total, using a standardized script. We provided them with a one-page information sheet describing the study and obtained verbal consent. Gift cards of either $50 or $100 were offered, with the higher amount being given to physicians and executives. At the conclusion of each interview, we asked stakeholders if they were aware of other organizations or individuals who met our inclusion criteria, facilitating a snowball sample.

2.2. Data Collection

Interviews were semi-structured and conducted by phone, and lasted 30 to 60 minutes. Discussion topics included interviewee involvement in C-R/R; decision-making related to C-R/R; details about past, present, or planned C-R/R; barriers to and facilitators of C-R/R; desired levels of C-R/R cost and effectiveness; and C-R/R evaluation efforts. Interviews were audio-recorded and transcribed. Interviewing continued until we reached thematic saturation [24].

2.3. Data Analysis

Interview data were analyzed using content analysis methodology [22]. Transcripts were uploaded into ATLAS.ti version 8.4.15 (Scientific Software Development, GmbH, Berlin). The first three interviews were inductively coded independently by the first and second authors, who met to resolve coding discrepancies through discussion and consensus. The result was a scheme of more than 50 codes pertaining to C-R/R sustainability which was used by the second author to independently code subsequent interviews. A handful of new codes were added when no predefined code was applicable. Overall, twenty percent of interviews were coded by both researchers to ensure consistency of coding over time. Concordant processes of coding and memoing on codes enabled the elaboration of codes and clustering of codes into categories. Data within and across codes and categories were analyzed to develop themes.

3. Results

Overall, 24 organizations were recruited, and individuals at 23 of them completed an interview. One organization did not respond. Ten interviews were conducted with representatives of an IIS, including 5 with sustained C-R/R, 3 with sporadic C-R/R, and 2 with unsustained C-R/R; 8 with health plans or health systems, all which had conducted at least one C-R/R initiative; 2 with pharmaceutical companies; and 3 with advocacy organizations (Table 1). In five instances, stakeholders preferred to be accompanied by a colleague or group of colleagues. We first describe the characteristics of the C-R/R initiatives in our sample, followed by factors influencing sustainability within IISs. Our results conclude with an assessment of the similarities and differences between IIS-based C-R/R and health plan- or system-based C-R/R.

Table 1:

Participant Characteristics

Interviewee Type	Sub-grouping	Have Done C-R/R at Organization	# People on Interview
IIS	Sustained C-R/R	Yes	1
	Sustained C-R/R	Yes	1
	Sustained C-R/R	Yes	2
	Sustained C-R/R	Yes	4
	Sustained C-R/R	Yes	2
	Sporadic C-R/R	Yes	1
	Sporadic C-R/R	Yes	1
	Sporadic C-R/R	Yes	2
	No current efforts (unsustained)	Yes	2
	No current efforts (unsustained)	Yes	1
Non-IIS	Health System	Yes	1
	Health System	Yes	1
	Health Plan	Yes	1
	Health Plan	Yes	1
	Health Plan	Yes	1
	Health Plan	Yes	1
	ACO	Yes	1
	ACO	Yes	1
	Pharmaceutical	No	1
	Pharmaceutical	No	1
	Policy	No	1
	Policy	No	1
	Policy	No	1

3.1. Overview of IIS-Based C-R/R

Our IIS interviews elicited information on C-R/R program characteristics, specifically target vaccines, modalities, message content, funding sources, and reported effectiveness and cost, as described below.

3.1.1. Target Vaccines

C-R/R notifications were sent most often for childhood and adolescent vaccines, including some focusing exclusively on the Human Papillomavirus (HPV) vaccine. However, several IISs reported conducting C-R/R for adult vaccines, influenza, and localized disease outbreaks. Vaccines of focus were selected because of perceived community needs related to low vaccination rates, or specific funding opportunities.

3.1.2. Modalities

Multiple C-R/R modalities existed, but the most common were postcards and letters, which had been used by a majority of IISs. Other modalities included text message and autodialer calls, though these were less common due to perceived legal issues. Such issues centered on the interpretations of the Telephone Consumer Protection Act (TCPA), a law limiting the circumstances under which autodialer calls or text messages can be made [25].

3.1.3. Message Content

C-R/R content and wording varied, but notifications typically included a health department name without an individual health provider or practice name. Some IISs omitted information about specific vaccines, while others included the name of the needed vaccines. In addition, some notifications included information to help recipients act upon the notice. Message content varied by modality, with text messages being more general due to space constraints. Also, patient privacy concerns shaped the content of some messages, including those on postcards where information was widely visible.

3.1.4. Funding Sources

The Centers for Disease Control and Prevention (CDC) was the most common C-R/R funder among our sample. CDC funding included prescriptive grants and supplements, which were typically competitive and not offered consistently, and creative use of existing CDC contractual agreements. Also, some IISs were part of the CDC’s Sentinel Sites program, which provides extra funding for IIS projects including C-R/R evaluation [26]. State funding for C-R/R varied across states, though many IISs reported receiving no or few such funds. Other funding sources were pharmaceutical companies and regional and state coalitions, which provided C-R/R resources such as postcards, postage, and contracts for text messages or autodialer calls. When these other resources were used, the IIS still provided staff to conduct the C-R/R.

3.1.5. Reported Effectiveness and Costs

IISs reported differences in effectiveness across C-R/R efforts, ranging from not at all to highly effective. Most reported moderate effects, with immunization rate increases around 5 to 10%. Many interviewees found this to be an appropriate target range, while others felt that any vaccination rate increase would be worthwhile. However, expectations for effectiveness varied based on the vaccine type and modality. Project costs reportedly varied based on modality and project scope, though specific cost information was not provided. Likewise, most interviewees did not offer details on what reasonable C-R/R costs would be. Overall, C-R/R was seen as beneficial and worthwhile.

3.2. Factors Influencing C-R/R Sustainability within IISs

Interviewees discussed a variety of factors that positively or adversely influenced C-R/R at their IIS. These fell into seven categories: decision-making, stakeholder buy-in, partnerships, funding, data and technology, evaluation, and message content (Figure 1). Legal concerns pertaining to patient privacy and contact modality and language fell within these domains.

Figure 1:

Factors Influencing C-R/R Sustainability within IISs

3.2.1. Decision-making: Driven by Motivations to Increase Vaccination Rates but Constrained by Limited Funds and Competing Priorities.

Not surprisingly, IISs were motivated by the promise of increasing vaccination rates using C-R/R, and many saw it as integral to their mission. At times these motivations centered on a specific geographic region or immunization type. However, C-R/R decision-making was often shaped by funding opportunities and competing priorities. External funding was reportedly pivotal to developing IIS-based C-R/R efforts, and in some instances appeared to be the primary catalyst. Further, IISs noted the challenge of prioritizing C-R/R in relation to other vaccine-related efforts to get the most “bang for your buck.” For example, one IIS stated: “When we're trying to prioritize, do we think we're going to get more improvement in rates from a centralized reminder/recall, or... some other way... It's always a prioritizing issue too since we never have enough funding.” C-R/R decision-making often involved a joint process among IIS Managers and senior leaders, such as Division Directors or Bureau Chiefs.

3.2.2. Stakeholder Buy-In

Widespread Internal Support Exists for C-R/R.

IISs that sustain C-R/R efforts noted organizational support for C-R/R at multiple levels, including senior leadership. Also, support for C-R/R was noted at the IIS Manager or Immunization Program Manager level. In some instances, staff had been in their positions for multiple years, and this longevity was thought to foster an ongoing C-R/R portfolio. As one long-standing IIS manager noted: “Well I think [my colleague] and I have been here the whole time, so I think both of us kind of continue to make sure that we want to do [C-R/R] and that it happens.”

A Limited Number of Patients and Parents Resist C-R/R, but This Rarely Impacts Continuation. Despite general patient and parent support for C-R/R, IISs received occasional negative feedback from message recipients, and in rare cases, issues were elevated to other governmental agencies such as the attorney general. Concerns usually centered on privacy questions such as how immunization information had been obtained and whether an opt-out option was available (and indeed, several IISs reported established procedures for opting out). A couple of IISs reported pushback specific to the HPV vaccine. One IIS reported that a C-R/R effort for adult vaccines had been halted “after a month or two of very negative pushback.” However, most IISs did not encounter such disruptive responses. Nonetheless, IISs tend to be conservative with their messaging strategies to avoid upsetting the public.

Providers Are Typically Supportive of C-R/R, Particularly When Notified in Advance of an Initiative.

Overall, IISs experienced a high level of provider support for C-R/R. However, a few IISs witnessed negative responses when providers were “caught off guard.” For example, one IIS stated: “We did not reach out to the physicians… and let them know that this was something we were doing. So when their patients were calling them telling them that they'd gotten a text saying that they needed a vaccine, what vaccine did they need, the physician was like ‘What text? What are you doing?’” Some IISs mitigated this possibility by notifying physicians in their region in advance.

3.2.3. Partnerships

IISs Benefit from a Range of Collaborative Relationships Related to C-R/R.

Many IISs viewed external partnerships as important. These partnerships were primarily with pharmaceutical companies, but also with provider groups, health plans and systems, and universities. The reasons for these partnerships included technical assistance, sharing of best practices, and research and evaluation (full list in Table 2). As an example of a pharmaceutical industry partnership, one IIS stated: “[The company is] open to providing feedback and bringing in best practices… I mean they've been really great.” Another IIS discussed an emergent partnership with a health plan: “We are working to establish relationships with payers for the different insurers in [our state]. We would like to see them take on some of this reminder/recall function as well, and we would be happy to collaborate with them, to assist them in doing it.” Some partnerships also included funding arrangements, as discussed below.

Table 2:

Types of C-R/R Partnerships among IISs

Public Partnerships (within or between IISs and Health Departments)
Staff and resource sharing for outreach within health departments (such as a mutually purchased text message platform) Assistance with funding via shared grant writing or other mechanisms Coalitions across mutually focused health groups for advocacy, securing funding, and providing personnel Data sharing between different IISs with shared populations Sharing best practices and approaches via conferences and member organizations (such as the American Immunization Registry Association)
Public-Private Partnerships (between IISs and Other Healthcare Organizations)
Funding for materials (postcards or autodial or text message contracts) Enhanced data sharing Collaborative C-R/R including: 1) joint C-R/R effort with area providers, and 2)“dividing and conquering” of outreach between IIS and area providers Contracted implementation Evaluation services Technical assistance Aid in onboarding immunization providers for data quality improvement

Broader Coalitions and Committees Are Important to C-R/R.

Some IISs noted the importance of inter-organizational coalitions and committees as a means for information exchange, resource sharing, and general guidance. Examples included statewide immunization and cancer coalitions, a statewide immunization advisory committee, and organizations such as the American Immunization Registry Association and Association of Immunization Managers. These organizations provided resources to help further C-R/R projects in their communities. For example, one IIS noted: “[The coalition] rallied with us because [HPV-related cancer] is something that could be preventable.” A handful of IISs expressed an interest in having even greater partnership among stakeholders, such as providers, to ensure collaborative goals are met.

3.2.4. Funding

External Funding is Instrumental, and Some IISs Discontinue C-R/R for Lack of Funding.

Most IISs in our sample emphasized the importance of external funding, as internal or state funding was inconsistent. While some IISs were able to maintain their C-R/R when external funding ceased, others were not. For example, one IIS expressed: “We don't have anything happening currently, but we have in the past… We did suspend those a couple of years ago. The amount of staff time involved in preparing those files was just prohibitive.” A few IISs were not experiencing funding difficulties currently but were concerned about the future. For example, one IIS noted: “Our entire operation is entirely CDC funded… The issue you run into with doing it through grant funding is… Is this one of the first things up for being on the chopping block?”

Mailing and Staffing Costs Are Major Challenges to C-R/R.

Several IISs described mailing and staffing costs as the most burdensome. Regarding mailing costs, some found them to be prohibitive without external funding. For example, one IIS noted: “When it's not sponsor-paid-for by a vaccine manufacturer, that cost for mailing is extremely high. It's not something without [a sponsor] that the immunization program would ever have been able to take on.” Likewise, staffing expenses were noted as particularly challenging, since C-R/R can be time-consuming. As one IIS noted: “We at one point had a staff member [whose] sole job was to process those return postcards, to correct that address, eight hours a day, five days a week.” Reportedly, even text-based C-R/R can be laborious because of programming costs and technological limitations.

Startup Costs Are High but Lay the Groundwork for Future C-R/R Initiatives.

Many IISs shouldered high C-R/R startup costs. In particular, the cost of information technology infrastructure was noted as a substantial barrier. However, after the initial investment, subsequent C-R/R initiatives were reportedly more feasible and cost effective. For example, one IIS stated: “We did have the initial cost of building the text message system. And now… I think the infrastructure that we built for text message can be leveraged for email… It doesn't really cost us a lot right now. It cost us a considerable amount to build the capacity to do it, but we did that a few years ago with supplemental funding from CDC.” In other words, initial infrastructure can be challenging to build, but once established may make C-R/R more feasible.

3.2.5. Data and Technology

IISs Face Major Challenges Maintaining Updated Contact Information.

A notable challenge for IISs is maintaining updated contact information. Inaccurate information can prevent reminders from reaching the intended recipient or result in duplicate notifications. Incorrect or missing addresses and phone numbers were reported as the most common such problem. For example, one IIS noted: “The biggest ongoing, well biggest issue… is data quality… the data issue largely is with addresses.” Another IIS stated “This issue of getting the mobile phone numbers is perhaps the biggest challenge that people are having.” These data issues reportedly add to material and personnel costs and reduce overall return on investment (ROI). Also, duplicate records within the IIS were noted to increase work associated with C-R/R. Further, several IISs reported challenges updating their death records, which resulted in reminders being sent to deceased patients. Even an IIS that obtained electronic vital statistics records reported problems since it did not receive information on deaths occurring out of state.

Immunization Data Are Improving but Still Pose Challenges.

The implementation of Health Level Seven or “HL7” connections between electronic health records (EHRs) and IISs was reported to improve immunization data quality. As one IIS stated: “That was a big move forward for us… This movement to reporting from EHRs. Now it really takes a burden off of the providers. They basically just have to document immunizations in their own EHR, and that data flows in real time.” Indeed, practice interface onboarding was reported as a priority. However, some IISs still experienced difficulty with outdated immunization data and noted this can lead to unwarranted reminders and negative patient feedback. Further, only certain states mandate providers to enter immunization data into their state’s registry, and reportedly this lack of a mandate can reduce data quality, especially among those without EHR-IIS connectivity. One IIS noted that regardless of mandates, providers may delay the input of immunization data, resulting in inaccurate records in the short term.

Technology Can both Improve and Be a Barrier to C-R/R.

Recent advancements in IIS capabilities were reported to facilitate C-R/R. While built-in algorithms for determining missing vaccines, known as forecasters, are a requirement for C-R/R and are common, the increasing complexity of forecasters themselves and outputs they create have reduced workload and increased C-R/R specificity. Several interviewees reported their forecasters facilitating the automation of processes such as creating personalized C-R/R letters or text messages, thereby reducing staff time required for C-R/R. As one IIS noted: “We also have a lot of features that allow us to both pull the cohorts that we need to, from the application itself, and then to pull all the reports, mailing labels, and a lot of different pieces of the mailing directly out of the system. So with those, it simplifies a lot of the work.” Interviewees also described the usefulness of complex algorithms, such as those which narrow a C-R/R effort to a specific subgroup in the event of a disease outbreak. And as new functionalities emerge, they are disseminated rapidly, aided by the existence of larger vendor-supported systems.

Despite technological advancements, some IISs described challenges related to technology use such as a glitch with a texting system that led to duplicate reminders, problems with a scanner that tracked returned mail using barcodes, an autodialer call originating from an out-of-state phone number that prompted questions about the call’s legitimacy, and an electronic records system that was not updating. Further, IISs often encountered legal barriers to electronic forms of C-R/R such as text messaging. Such challenges usually centered on the TCPA and associated state laws, though exact details of the barriers were not typically known by interviewees, as they were usually handled by the IIS’s legal department. While some IISs were able to navigate these barriers, others were not. For example, one IIS stated: “Our legal definitely said “no” to any phone calls, text messages, or emails… They said “no,” and so we really just took what we could and settled with the postcards.” However, on the whole, technology and automation seem to have furthered the IISs ability to perform C-R/R.

3.2.6. Evaluation: A Key to C-R/R Sustainability, but Challenging to Conduct and Rigor Varies.

Most IISs reported some type of evaluation effort, though they ranged in scope and rigor. Some simply monitored vaccination rates. Others tracked process measures, primarily contact information accuracy rates, in addition to vaccination rates. A handful of IISs conducted sophisticated controlled evaluations, some which were externally supported. For example, one IIS obtained a federal grant for evaluation work as part of being a CDC Sentinel Site, while another partnered with a university. Importantly, many IISs noted the difficulties of measuring the true impact of a C-R/R initiative. For example: “It's always also difficult to separate out what impact has come from reminder-recall and what impact is from other projects we have going on, and media campaigns, things like that. That's always been the thing with reminder-recall. I think it is hard to evaluate.” All interviewees agreed that evaluation was crucial for justifying the continuation of their C-R/R programs.

3.2.7. Message Content

C-R/R Messages without Specific, Actionable Information Raise Questions about Usefulness.

Some interviewees expressed challenges associated with C-R/R notifications that lacked certain details. One such concern involved the absence of vaccine information. For example, an IIS that did not list the needed vaccines on a reminder stated: “Most of the calls we got from parents were parents wondering which specific vaccines their adolescent needed.” In addition, numerous IISs encountered challenges associated with reminders that were not “actionable,” meaning that no detailed next steps were included such as contact information for a provider or clinic. For example, one IIS noted: “It was a very general message, basically aimed at parents saying ‘We know you have a child who is due for a flu vaccine. Please take your child to the doctor’… And it was from the Health Department; it was not associated with a doctor’s office. And that may have been why the effect was small.”

C-R/R Messages with Overly Specific Information Raise Legal and Public Relations Concerns.

While specific reminder information may be useful, it reportedly comes with challenges. Some IISs faced difficulty obtaining organizational or legal approval to include vaccine names on reminders, for fear of a perceived conflict of interest related to promotion of a pharmaceutical product. At other times, IISs omitted information about needed vaccines to protect patient privacy in case the message was intercepted, particularly when messages were sent via postcard. For example, one IIS stated: “A postcard was sent out. It was more of a general message. We didn't say anything like ‘according to our records’… We said, ‘there are three vaccines that are recommended at the 11-year annual visit, which are DTaP, HPV, and meningococcal.’” In addition, some IISs omitted patient name information to avoid sending reminders to the wrong person or a deceased person. The language on messages frequently needed to be approved by the IIS’s legal team, who sometimes required IIS managers to use more generic wording.

3.3. Health Plan and System C-R/R Practices

Our health plan and system interviews provided insight into C-R/R outside of IISs, and contribute to the knowledge base on C-R/R sustainability. We discovered a range of C-R/R efforts among those organizations, with some differences when compared to IIS-based reminder/recall (Table 3). Areas of public-private collaboration were also identified.

Table 3:

Differentiating Features of Health Plan and System C-R/R

Category	Feature(s)
Decision-Making	Incorporates HEDIS measures in decision-making; places more emphasis on ROI.
Stakeholder Buy-In	Appears to garner leadership support especially when there is an expected increase in HEDIS measure performance.
Partnerships	May involve partnerships with pharmaceutical companies (like IISs), though it appears to be less common.
Funding	Often relies on self-funding; may encounter fewer staffing and resource issues; may involve higher costs if call centers are used; can include patient incentives.
Data and Technology	Involves increased access to automated, sophisticated outreach technology.
Evaluation	Tends to tie evaluation to HEDIS metrics.
Message Content	Appears to include “actionable” reminder information more often.
Legal Issues	Tends to receive consent for contacting patients during enrollment, thus avoiding legal issues concerning the right to contact that health departments might face.

Health plans and systems who conducted C-R/R did so for a variety of populations and vaccines, and efforts varied greatly in scope. Key differences of health plan and system C-R/R, when compared to IISs, included an emphasis on Healthcare Effectiveness Data and Information Set (HEDIS) performance measures, which were a motivator and basis for evaluation; less apparent support from pharmaceutical companies; more common self-funding; fewer apparent staffing and resource problems; the use of costly call centers in some cases; the occasional use of patient monetary incentives; greater access to automated outreach technology, with links to EHRs enabling very specific C-R/R efforts; more frequent inclusion of “actionable” reminder messages; and less salient legal issues, reportedly because of mechanisms for building patient consent for contact into their operations.

Like IISs, health plans and systems experienced limited patient pushback, but it primarily came from general frustration with the organization’s frequent communication rather than the C-R/R message itself. Health plans and systems were mostly open to the idea of partnerships and funding relationships with IISs. For example, a few had supplemented their own immunization information with IIS data for C-R/R, and some were receptive to the potential idea of funding large-scale C-R/R driven by IISs. However, concerns existed about the logistics of funding projects which could involve patients other than their own, as well as administrative burdens associated with payment and data sharing arrangements.

4. Discussion

This study is among the first to explore both the initiation and sustainability of C-R/R among IISs and other key stakeholders. Our findings align generally with the literature on public health program sustainability [27, 28] and complement existing knowledge of C-R/R sustainability [15], suggesting there are several important issues to consider for C-R/R success. These include decision-making, stakeholder buy-in, partnerships, funding, data and technology, evaluation, and message content. Furthermore, these issues were slightly different for IISs when compared to non-IISs; it is plausible that each group can learn from the other and potentially partner with one another.

Notably, our results identified several barriers to C-R/R. Although IISs have made substantial progress in achieving C-R/R success, challenges remain. Salient issues were costs and lack of funding, poor contact data quality (i.e., addresses or phone numbers), and messaging that is overly broad or too specific. Our results confirm that C-R/R costs can be burdensome and that funding is often inconsistent [15], though costs can become less of a barrier over time. Also, we found that IISs grapple with data quality issues, as described previously [14] and that these challenges exist even when processes and systems are in place for data transfer. In addition, our results confirm challenges related to the specificity of reminder/recall messages. Like past studies, we found that specific, “actionable” information such as provider contact information is preferred [17], though the logistics of including this information are not always straightforward. Similarly, some IISs faced legal or public relations barriers to the inclusion of a specific vaccine or patient name, a finding which sheds light on why reminders may lack useful information despite recommended practices [14, 15].

In contrast, several factors emerged as facilitators of C-R/R success. Most pertinent were notifying providers in advance of C-R/R, conducting a rigorous evaluation, and engaging a range of partners. While our results confirm a high level of support from patients and providers [11, 16-21], they also shed light on the importance of notifying providers before a C-R/R effort. Provider notification was used by several IISs, while some who did not notify providers received negative feedback or inquiries. Also important to C-R/R sustainability is the presence of program evaluation. Not surprisingly, our results confirm that favorable results may be used to justify C-R/R program continuation or proliferation [27]. Increased evaluation support through funding or partnership can help dictate best approaches to C-R/R and bolster its effectiveness and cost-effectiveness.

In addition, partnerships were particularly important for C-R/R sustainability, and ranged from collaborations with health plans and systems to universities and pharmaceutical companies. While the importance of public-private partnerships to achieve population-level health is well recognized [29], our results provide insight into the current and potential collaborations specific to C-R/R. One potential approach to achieve maximum C-R/R initiation and sustainability could involve health plans and pharmaceutical groups offering or bolstering support to health departments or IISs for postcards, autodialer calls, or staff resources. These entities would conduct C-R/R for entire states or regions. IISs likely have the most consolidated immunization records, compared to health plans or other healthcare entities, and they have the ability to reach entire regional or state populations given that many IISs receive birth records via vital statistics data [14]. However, barriers to receiving timely immunization information vary by IIS, especially as it relates to mandatory reporting among providers. Depending on state policies and laws, providers may either have voluntary or mandatory reporting of immunization data to an IIS which can affect data quality across IISs.

Given these barriers, another possible approach could involve private groups such as health plans and systems partnering formally with IISs to ensure receipt of consolidated patient-level data [29]. This could be achieved by increasing bi-directional data exchanges between EHRs and IISs. For example, health plans or health systems likely have better patient contact information such as addresses and telephone numbers which could improve the reach of C-R/R. This approach, at least in concept, was received positively by several health plans and systems interviewed.

Importantly, this study has several strengths and limitations. A key strength is that this is among the first to explore this topic in depth, particularly from a qualitative perspective. We had a high response rate among those who we recruited. We also made efforts to recruit a variety of IISs and non-IISs to elicit a diverse array of perspectives. However, it is important to note that our results are not generalizable to a broader range of organizations beyond our sample. While not a limitation, per se, given the qualitative methods utilized, our sample included a subset of the organizations who are likely conducting C-R/R. In addition, our data may not reflect the range of perspectives within organizations since we conducted only one interview with each organization, and typically with one interviewee. Similarly, our data do not reflect the perspectives of relevant stakeholders external to these organizations such as patients or caregivers.

5. Conclusions

Overall, our study explored in detail C-R/R efforts conducted by IISs and other healthcare stakeholders. IISs noted some barriers and also highlighted facilitators of implementation and sustainability that could serve as lessons learned for any IIS replicating or tailoring these methods of increasing population-level immunization rates. In addition, IISs can learn from successful private-sector C-R/R practices such as those presented in this study. Importantly, the potential for public-private collaboration should be explored.