Using Intervention Mapping to Develop an Efficacious Multicomponent Systems-Based Intervention to Increase Human Papillomavirus (HPV) Vaccination in a Large Urban Pediatric Clinic Network

BACKGROUND AND INTRODUCTION

Human papillomavirus (HPV) infection is a preventable yet common cause of morbidity and mortality with more than 14 million new cases per year in the U.S.¹ HPV causes nearly all cervical cancers and many vaginal, vulvar, anal/rectum, penile, and oropharyngeal cancers.^2,3 Approximately 1 in 4 people in the U.S. are currently infected with HPV, and nearly 80% of people will develop HPV during their lifetime.¹

The HPV vaccine can prevent HPV infections that cause HPV-associated cancers. The Advisory Committee on Immunization Practices (ACIP) recommends initiation of the HPV vaccination series during early adolescence between ages 11 and12 years for both males and females.⁴ Despite safety of the vaccine⁵, endorsement from professional organizations such as ACIP⁴, and a Healthy People 2020 HPV vaccine series completion goal of 80%,⁶ HPV vaccine uptake rates remain suboptimal. At the time of this study, national HPV vaccination initiation rates for ages 13 and 17 years were 53.5% and 65.4%, respectively.⁷

Though evidence-based strategies have been developed to increase uptake of other vaccines,⁸ interventions aiming to increase HPV vaccine use have demonstrated limited impact.^9,10 Evidence-based strategies to increase HPV vaccination rates include provider assessment and feedback, provider cues, patient reminders, and delivering bundled vaccine recommendations, but using any one of these strategies in isolation is less effective than applying multiple strategies.⁸ Theory- and evidence-based interventions can optimize effectiveness for individual and organizational change.¹¹ The Adolescent Vaccination Program (AVP) is a theory- and evidence-based multilevel and multicomponent HPV vaccination intervention comprising sequential rollouts of system-level strategies. A recent quasi experimental study of the AVP, conducted in a large urban southwestern pediatric clinical network, demonstrated its success in significantly increasing HPV vaccination initiation and completion rates over a 3-year period (p ≤ 0.05).¹²

Intervention Mapping (IM) is a systematic approach to planning theory- and evidence-based health promotion interventions.¹³ A recent systematic review demonstrated significant increase in the uptake of disease prevention behaviors associated with IM-based interventions when compared to placebo control groups.¹⁴ IM has been used to develop interventions for preventing cancer, including skin,¹⁵ lung,¹⁶ breast,¹⁷ and cervical^17–20 cancers. A recent study reported the use of IM to develop an HPV vaccination behavioral education intervention for parents of Hispanic adolescents.²¹ However, few applications of IM have been reported in the context of developing a multicomponent intervention that have comprised vaccination strategies targeting clinics, providers, and parents.²¹ The purpose of this paper is to describe the application of IM in the development of the theory- and empirically based AVP to increase HPV vaccination rates.

METHODS

Intervention Mapping

IM is a stepped framework to guide the development of behavioral change interventions that enable developers to systematically apply social and behavioral science theories.²² The 6 steps of IM (Table 1) are to: 1) assess needs and develop a logic model of the problem, 2) develop matrices of behavioral change objectives for the program, 3) identify theory-based methods and practical applications to be applied in the program, 4) produce program components and materials, 5) plan for program adoption, implementation, and sustainability, and 6) plan for evaluation.¹³ This project was approved by The University of Texas Health Science Center at Houston Institutional Review Board (HSCSPH-14–0725).

TABLE 1:

Intervention Mapping (IM) Steps with Associated Tasks and AVP Deliverables

Yr	IM Steps	IM Tasks	Development Deliverables
1	1. Assess need and develop a logic model of the problem	• Establish and work with a planning group. • Describe the context for the intervention, including the population, setting, and community. • Conduct a needs assessment to create a logic model of the problem.	• Clinic Provider Advisory Group • Literature review – evidence table • PRECEDE Model
	2. Develop matrices of change objectives and a logic model of change	• State expected outcomes for behavior and environment. • Specify performance objectives for behavioral and environmental outcomes. • Select determinants for behavioral and environmental outcomes. • Create a logic model of change. • Construct matrices of change objectives.	• Matrices for provider outcome behaviors comprising 8 performance objectives and 65 learning objectives. • Conceptual model for the AVP (model of change).
	3. Identify theory-based methods and practical applications for program design	• Generate program themes, components, scope, and sequence. • Choose theory- and evidence-based methods to create change. • Select or design practical applications to deliver change methods.	• AVP design document comprising specifications including: content, design features, functionality, language, logistics of use and implementation in the clinic, orientation needs, and evaluation specifications.
2	4. Produce program components and materials	• Refine program structure and organization. • Prepare plans for program materials. • Draft messages, materials, and protocols. • Pretest, refine, and produce materials.	• AVP Champion webinars (n=4). • AVP Champion Binders for each clinic. • Provider assessment and feedback reports. • Provider continuing education module. • EHR Best Practice Advisory for HPV vaccination. • Pediatric Wellness Registry for patient HPV vaccination reminders. • Pilot test protocols and results:  • Manual of Procedures.  • AVP component feasibility testing in advisory clinics.
	5. Plan for program adoption, implementation, and sustainability	• Identify potential program implementers. • State outcomes and performance objectives for implementation. • Construct matrices of change objectives for implementation. • Design implementation interventions.	• Processes and channels for deployment of AVP strategies. • Matrix of key stakeholders / gatekeepers for implementation.
	6. Plan for evaluation	• Write effect and process evaluation questions. • Develop indicators and measures for assessment. • Specify evaluation design.	• Evaluation Design Manual of Procedures, including:  • Study hypotheses and protocols.  • Measures for assessment.  • Baseline and follow-up surveys for physicians and clinic staff.

The Development Timeline

Completion of the IM development process encompassed 2 years of activities (Table 1). The first 6 months of Year 1 involved development of the logic model of the problem (IM Step 1), defining program outcomes and objectives and matrices of change (IM step 2), and instituting the vaccination champion component to advocate for and mediate the implementation of the AVP within the clinic sites (as an advocate and mediator for the AVP). The remaining 6 months of Year 1 involved program planning, developing the AVP design document (IM Step 3), and initial rollout of the assessment and feedback strategies. Year 2 involved completing the full program prototype, including development cycles for each component followed by a formative evaluation with pilot testing of components (IM step 4). Plans for implementation and evaluation (IM Steps 5 and 6) were consolidated during the period of AVP formative testing and were implemented from 2015 through 2018.

IM STEP 1: LOGIC MODEL OF THE PROBLEM

Step 1 comprised the following: establishing a planning group; conducting a needs assessment informed by the PRECEDE (Predisposing, Reinforcing and Enabling Constructs in Educational Diagnosis and Evaluation) planning model that outlines the factors associated with the problem; defining the context of the intervention in terms of population, setting, and community; and starting to implement program goals.¹³

Task 1.1: Establish and work with a planning group

Pediatric clinic population and setting.

The AVP development involved collaboration with a large urban pediatric clinic network in the southwestern United States. The network comprised 51 pediatric practices in 5 counties (encompassing over 220 physicians and over 800 staff members), serving an estimated 20% of the pediatric population in these counties. Clinics varied in size, staff composition, patient demographics, and rates of initiation of HPV vaccination. Most clinics (97%) were certified by the National Committee for Quality Assurance (NCQA) and, where eligible, were NCQA-recognized medical homes. Five clinics were located in underserved areas and provided pediatric medical services for families who would otherwise receive limited or no health care due to low family income or lack of health insurance. The network participated in Healthcare Effectiveness Data and Information Set (HEDIS) accreditation.²³ Patient demographics comprised white (59%), Hispanic (23%), African-American (13%), and Asian (5%). Most patients (73%) had commercial insurance; the rest had Medicaid (17%), Children’s Health Insurance Plan (CHIP) (6%), or no insurance (4%). Approximately 25% were eligible for the Vaccines for Children (VFC) program.

Stakeholder Advisory Committee.

The IM process recommends identification of key stakeholders, including experts, community members, potential implementers, leaders, and members of the population of interest, to form a planning group that guides intervention development.¹³ The AVP stakeholder advisory committee (SAC) comprised 3 researchers in HPV and cancer prevention, behavioral science, and intervention development, 3 pediatricians, 1 pediatric information technologist, 1 data analyst, and leaders of the network, including the chief medical officer (CMO). The CMO identified a core of 6 advisory clinics with diverse geographic locations and mixed patient demographics and insurance payer base (private vs. public) to enable broad access to “frontline” providers for formative assessments of the AVP components prior to implementation. The CMO, an administrator, and the project team held regular biweekly in-person meetings through the entire IM process to: plan and design components (including review content, assess functionality, flow, and the “look and feel” of AVP components); develop plans for seamless implementation (rollout) without disruption to standard operating procedures; and plan for evaluation activities.

Task 1.2: Conduct a needs assessment to create a logic model of the problem

The needs assessment identified clinic-, provider- and parent-level barriers to HPV vaccination to inform a logic model for HPV vaccination. This comprised the following: 1) rates of HPV vaccination among adolescents in the pediatric network compared to national rates; 2) perceived barriers, attitudes, and practices regarding clinic organization and provider-related factors impacting HPV vaccination; 3) perceived barriers, attitudes, beliefs, and needs regarding HPV vaccine among parents of adolescents in the pediatric network, and 4) current national best practices regarding HPV vaccine promotion and strategies for incorporating HPV vaccination best practices into clinical settings. Quantitative and qualitative methods included literature review, analysis of cumulative vaccination data from the electronic health record (EHR), interviews with clinic leaders, focus groups with providers and staff in the 6 advisory clinics, and surveys with providers and staff across the network.

Literature review

Conducted in 2014 in collaboration with a research librarian, the literature review provided background on 1) current rates and burden of HPV infection, and 2) evidence-based strategies to increase vaccination rates and the clinical, behavioral, and psychosocial factors associated with their implementation. Inclusion criteria comprised articles published in peer-reviewed journals, including review articles and surveys as well as practice guidelines. Abstracts, poster presentations, and editorial publications were excluded. Electronic publication databases comprised PubMed, EMBASE, and MEDLINE. The Community Preventive Services Task Force’s Community Guide⁸ provided a systematic review of the evidence of effectiveness of health promotion strategies that was foundational for this study. Strategies included provider assessment and feedback (A&F), provider cues, provider communication strategies (e.g., bundled messaging), and patient reminders. Evidence tables were developed for expert review. The literature review provided information on national immunization recommendations to prevent HPV, system factors in clinic settings that facilitate provider recommendations for HPV vaccination initiation, and physician-level factors affecting parent decision making to accept HPV vaccination recommendations. Critical findings that informed the AVP are provided in Table 2.

Table 2.

Literature review key findings

Key Finding	Qualifications
System factors are a major determinant of receiving HPV vaccination.	Primary parental determinants of HPV vaccination initiation among adolescents were talking with a doctor, having enough time to discuss the vaccine, having a healthcare provider recommend it, and having a healthcare visit in the past year.25–28 Parents express a strong preference to receive information about HPV vaccination directly from trusted healthcare providers.29–31
The research on provider attitudes and practices describes several sources of provider hesitancy to recommend or discuss the HPV vaccine with parents.	Common sources of provider hesitancy include providers’ “perception that younger adolescents are less at risk of HPV so vaccination can be delayed,” providers’ perceptions of parental hesitancy and ambivalence, misunderstanding parental barriers to vaccination,22,32 and limited time with patients.33–38 Furthermore, delaying discussion of HPV vaccination leads to missed opportunities39 because younger adolescents (11–14 years) are 3 times more likely to attend preventive visits than older adolescents.40
Physician recommendation remains an important determinant in parents’ decision to vaccinate their child.29,41–43	The CDC estimates that HPV vaccination initiation would reach over 90% if providers’ recommendations for HPV vaccination were similar to their recommendations for other adolescent vaccines.5 Commensurate with this is that providers convey vaccine recommendations consistent with evidence-based guidelines; provide accurate, evidence-based information about HPV and HPV vaccine; reassure patients of high vaccine safety due to ongoing postlicensure safety surveillance; and reinforce the message that the HPV vaccine is recommended despite not being required for school.

Analysis of cumulative vaccination data from the electronic medical record

HPV vaccine initiation and completion rates were assessed for all patients ages 11–17 seeking care at the 51-clinic pediatric network from January 1 through December 31, 2013. Among 92,735 patients over the 12-month period, overall HPV vaccine initiation was 49.4%. HPV vaccine initiation among girls was 54.0%, and among boys the rate was 44.9%. Overall completion rate for the HPV vaccine series was 24.2%. The completion rate was 30.3% among girls and 18.3% among boys. These rates fall far below the Healthy People 2020 goal of 80%. In the recommended 11- to 12-year age group, overall HPV vaccine initiation was 39.1%. For girls ages 11–12, HPV vaccine initiation was 42.0%; for boys it was 36.3%. Additionally, 44% of physicians had an HPV vaccine uptake rate less than 50%; 22% had an uptake rate less than 40%; and over 7% had an uptake rate less than 30%. In contrast, vaccination rates for tetanus, diphtheria, and pertussis (Tdap) and meningococcal vaccine (MCV) exceeded 91%.

Interviews with clinic leaders in 5 advisory clinics

Site leaders and practice managers and 14 clinic leaders were interviewed at 5 of the advisory clinics. Interviews were recorded and transcribed. Information was obtained on leadership roles and responsibilities, clinic workflow, current vaccine practices and protocols to adjust to changes in vaccine recommendations, barriers to HPV vaccination, and suggested strategies to increase HPV vaccination. The network’s expectation was for clinics to adhere to national standards (ie, ACIP), but there was significant variation in delivery of HPV recommendations between clinics and between physicians and medical assistants (MAs).

Focus groups with providers and staff in the 6 advisory clinics

In-person focus groups were conducted with 78 staff members within the advisory clinics. Group size ranged from 9 to 18. Participants included 22 pediatricians, 15 MAs, 14 nurses, 10 certified medical assistants (CMAs), 6 front desk/reception staff, 5 practice managers, and 1 assistant director, clinical supervisor, physician assistant, X-ray technician, triage worker, and referral specialist. Each focus group was recorded and thematically analyzed. Focus group findings informed the AVP development and encompassed themes related to how the vaccine was introduced, provider barriers to recommending the vaccine, and parental barriers to the vaccine (Table 3).

Table 3.

Focus group key findings

#	Category	Findings
1	Introducing the HPV vaccine	Providers tended to differentiate HPV vaccine from other vaccines recommended at the 11- to 12-year-old visit, presenting tetanus and diphtheria (Tdap) and meningococcal vaccine (MCV) as required for school but framing HPV as optional, either consciously or subconsciously. Most providers appeared reluctant to pursue the topic of HPV vaccination if the parent was hesitant or resistant, especially for younger children. Practices varied on whether the physician or clinical support staff first introduced the HPV vaccine and whether the parent received the Vaccine Information Statement (VIS) at the beginning or the end of the visit.
2	Provider barriers	Providers, particularly MAs, stated their own concerns as insufficient knowledge about HPV and HPV-related diseases, the perception that there was no immediate need to vaccinate younger adolescents, and not understanding the rationale for HPV vaccination starting at age 11.
3	Parental barriers	Providers stated that the most frequent concerns expressed by parents were not knowing or understanding the diseases the HPV vaccine prevents, wanting to wait until the child was older (child not having sex), wanting to wait until more was known about the long-term effects (vaccine was too new), and wanting to think about it or discuss it with their spouse.

Surveys with providers and staff across the network

Online surveys were distributed to all clinical staff at each clinic in the network (n=51) to assess baseline perceptions of HPV vaccine. The 30-minute surveys were completed by clinical staff (nurses, physician assistants, and MAs) (n=375; response rate: 88.7%), practice managers (n=45; response rate: 90%), and physicians (n=134; response rate: 59.7%). The survey focused on physician experiences with the HPV vaccine and addressed organization and patient barriers that they encounter when vaccinating adolescents. The surveys comprised items with fixed format response options including 4-point Likert scales with varied response options (Strongly Agree/Strongly Disagree; Not at all a barrier/A major barrier, etc.).²⁴ Providers were asked to select responses most representative of their experience. Analysis by the project team determined that lower initiation rates were mainly associated with physician concerns about parents’ negative perceptions about the HPV vaccine, the vaccine’s safety, its efficacy, and the financial burden the vaccine places on patients²⁴ (Figure 1).

Figure 1.

Logic model of the problem: Health care provider determinants of provider behaviors and parent outcomes

Task 1.3: Describe the context for the intervention including the population, setting, and community

The AVP was developed for implementation in primary care clinics within a large pediatric network (previously described). The heterogeneity offered across the 51 clinics (size, location, time within the network) and the patient population (demographics, insurance status) provided an excellent test-bed for development. The priority environmental focus was the clinical organization and the provider. A parent-facing educational program is described elsewhere. Community sentiment regarding vaccination in general, and HPV vaccination in particular, were acknowledged as important environmental influences in vaccine decision making (Figure 1). However, broader community influencers, while important, were outside the scope of the project.

Task 1.4: State program goals

The goal of the AVP was to use a multicomponent strategy to enable clinics to meet national metrics for HPV vaccination initiation and completion. This entailed enabling clinicians, providers, and staff members to adopt and implement evidence-based strategies to increase HPV vaccination. Respective organizational, provider/staff, and patient goals for the AVP included the following:

1. Primary care pediatric clinics that adopt and implement the AVP will demonstrate a significant increase in HPV vaccination initiation and completion rates in the clinic during implementation compared to rates prior to implementation.

2. Providers and staff who adopt and implement AVP-related behaviors within their clinic will demonstrate a significant increase in their patients’ HPV vaccination initiation and completion rates during implementation compared to rates prior to implementation.

3. Children who attend clinics implementing the AVP will be more likely to receive the HPV vaccination after implementation than they were prior to implementation of the AVP.

IM STEP 2: PROGRAM OUTCOMES AND OBJECTIVES – LOGIC MODEL OF CHANGE

Step 2 comprised the following: identification of expected outcomes, performance objectives, and determinants of the behavior and environment; the development of matrices of change objectives; and the construction of a logic model of change for the AVP.¹³ This step enabled the triangulation of data obtained in Step 1 (from theory, empirical findings, and participant involvement) to inform a logic model of change.

Task 2.1: State expected outcomes for behavior and environment

Expected Behavioral Outcomes.

The AVP was designed to positively impact the adoption and implementation of evidence-based strategies to increase HPV vaccination rates in primary care pediatric clinics. The expected behavioral outcome was that pediatricians will vaccinate eligible patients against HPV in accordance with ACIP guidelines. Targeted health and quality-of-life outcomes included impact on health status (decreased sexually transmitted infection [STI] and cancer incidence, reduced hospitalizations), functional status (increased future productive days at work, enhanced functioning and relationships), and long-term impacts (reduced societal cost of years of life lost, medical care, and long-term care costs) (Figure 2).

Figure 2:

Logic Model of Change for AVP

Expected Environmental Outcomes.

The AVP was designed to enable clinics to adopt evidence-based strategies (AVP champions, assessment and feedback, provider education, provider cues, and patient reminders) as usual practice.

Task 2.2: Specify performance objectives for health-promoting behavior and environmental outcomes

Performance objectives (PO) for adoption and implementation of evidence-based HPV vaccination strategies in the AVP

Performance objectives comprised the following: collaborate with the clinic’s champions on immunization status updates and strategies regarding HPV vaccination (PO.1); review quarterly assessment and feedback reports for HPV vaccination (PO.2); coordinate with clinical support staff to ensure that consistent messaging is delivered to patients regarding HPV vaccination (PO.3); check vaccine eligibility (Forecaster database) at every encounter to identify if patient is eligible for vaccination (PO.4); deliver strong recommendation for HPV vaccination to all eligible patients at time of visit (PO.5a); bundle the HPV vaccine recommendation with other vaccines when the patient is due for other vaccinations at the same visit (PO.5b); determine specific patient/parent concern if they express vaccine hesitancy (PO.6); communicate tailored messages to address specific patient/parent concerns (PO.7); and remind patients to schedule follow-up HPV vaccine dose(s) before leaving the office (PO.8) (Figure 2).

Task 2.3: Select determinants for behavioral and environmental outcomes

Findings from the empirical literature, relevant theory (ie, Social Cognitive Theory,⁴⁴ Theory of Reasoned Action, ⁴⁵ Health Belief Model⁴⁶) and prior formative research (Task 1.2 above) informed selection of behavioral determinants. These comprised knowledge, self-efficacy, outcome expectations, skills, and normative beliefs as important and changeable for providers to perform AVP-related performance objectives (Table 4).

Table 4. Health care provider matrix of performance objectives, determinants, and change objectives.

Behavioral outcome: Physicians will vaccinate eligible patients against HPV in accordance with ACIP guidelines

Performance Objectives	Behavioral Determinants*
	Knowledge	Skills and Self-efficacy	Outcome Expectations	Normative Beliefs
PO.1. Collaborate with clinic’s champion on AVP strategies to promote HPV vaccination	K.1a: Identify the designated AVP champions in his/her clinic K.1b: Describe the role of AVP champions as mediators for strategic rollout of 4 strategies* in clinics	SSE.1a: Demonstrate ability to collaborate with clinic champion on strategies* to promote HPV vaccination SSE.1b: Express confidence in ability to collaborate with clinic champion on strategies* to promote HPV vaccination	OE.1: Expect that collaborating with clinic’s champion on HPV vaccine promotion strategies will improve personal and clinic-level HPV vaccination rates	NB.1: Recognize that vaccinating all eligible patients against HPV in accordance with ACIP guidelines is part of the network’s expectation for optimal physician performance
PO.2. Review quarterly assessment and feedback report for HPV vaccination	K.2a: Identify when and how A&F reports will be delivered K.2b: Describe the content of the A&F reports K.2c: Recognize that A&F is one of the most effective strategies to promote vaccination	SSE.2a: Demonstrate ability to state personal and clinic-level vaccine rates from a quarterly A&F report SSE.2b: Demonstrate ability to state personal HPV vaccination goal for the following quarter SSE.2c: Express confidence in ability to interpret A&F reports	OE.2a: Expect that reviewing A&F reports will allow his/her clinic and staff to track progress toward HPV vaccination goals OE.2b: Expect that reviewing A&F reports will improve personal and clinic-level HPV vaccination rates	NB.2: Recognize that vaccinating all eligible patients against HPV in accordance with ACIP guidelines is part of the clinic network’s expectation for optimal physician performance
PO.3. Coordinate with clinical support staff to ensure consistent messaging is delivered to patients regarding HPV vaccination	K.3a: Describe the difference between consistent and inconsistent messaging about HPV vaccination in a clinic setting K.3b: Recognize that inconsistent messaging about HPV vaccination occurs in clinics K.3c: Recognize that inconsistent messaging about HPV vaccination can lead to parental vaccine hesitancy	SSE.3a: Demonstrate ability to communicate with clinical staff about consistent HPV vaccine messaging** SSE.3b: Express confidence in ability to communicate with clinical staff about consistent HPV vaccine messaging**	OE.3a: Expect that delivering consistent messaging from all clinical staff to patients will reduce patient/parent resistance OE.3b: Expect that delivering consistent messaging from all clinical staff to patients will improve personal and clinic-level HPV vaccination rates	NB.3: Recognize that vaccinating all eligible patients against HPV in accordance with ACIP guidelines is part of the clinic network’s expectation for optimal physician performance
PO.4. Check forecaster at every encounter to identify if patient is eligible for vaccination	K.4a: List ACIP eligibility criteria for HPV vaccination in adolescents K.4b. Describe functions of Forecaster K.4c: Recognize that patients are less likely to come back to doctor after age 13	SSE.4a: Demonstrate ability to check forecaster in a timely manner to determine patient vaccinations status SSE.4b: Express confidence in ability to check Forecaster in a timely manner	OE.4a: Expect that identifying eligible patients at every encounter will reduce missed opportunities to vaccinate OE.4b: Expect that identifying eligible patients at every encounter will improve personal and clinic-level HPV vaccination rates	NB.4: Recognize that vaccinating all eligible patients against HPV in accordance with ACIP guidelines is part of the clinic network’s expectation for optimal physician performance
PO.5. Deliver strong recommendation for HPV vaccination to all eligible patients at time of visit	K.5a: Recognize HPV vaccination is an effective cancer prevention tool K.5b: Recognize HPV vaccination is safe and recommended by medical organizations with the same strength as other adolescent vaccines K.5c: Describe components of a strong HPV vaccine recommendation	SSE.5.1: Demonstrate ability to deliver strong HPV vaccine recommendation SSE.5.2: Express confidence in ability to deliver strong HPV vaccine recommendation	OE.5a: Expect that delivering a strong HPV vaccine recommendation to all eligible patients will reduce patient/parent resistance OE.5b: Expect that delivering a strong HPV vaccine recommendation to all eligible patients will improve the likelihood of patients initiating the HPV vaccine series	NB.5a: Recognize that vaccinating all eligible patients against HPV in accordance with ACIP guidelines is part of the clinic network’s expectation for optimal physician performance NB.5b: Recognize that HPV vaccination is widely endorsed by medical organizations and other physicians as a safe and effective cancer prevention tool NB.5c: Recognize that physicians tend to overestimate the level of hesitancy parents have about the HPV vaccine
PO.5a. Bundle HPV vaccine recommendation with other vaccines when patient is due for other vaccinations at the same visit	K.5a: Describe the difference between a bundled HPV recommendation and one that singles out HPV from other vaccines K.5b: Describe key elements of a bundled HPV vaccine recommendation	SSE.5a.1: Demonstrate ability to deliver bundled HPV vaccine recommendation SSE.5a.2: Express confidence in ability to deliver bundled HPV vaccine recommendation	OE.5a: Expect that bundling HPV vaccine recommendation with other vaccinations will reduce patient/parent resistance OE.5b: Expect that bundling HPV vaccine recommendation with other vaccinations will improve the likelihood of patients initiating the HPV vaccine series	NB.5a: Recognize that vaccinating all eligible patients against HPV in accordance with ACIP guidelines is part of the clinic network’s expectation for optimal physician performance
PO.6. Determine specific patient/parent concern if they express vaccine hesitancy	K.6a: List common parental concerns related to vaccine hesitancy	SSE.6a: Demonstrate ability to identify specific patient/parent concerns related to HPV vaccination SSE.6b: Express confidence in ability to identify specific patient/parent concerns	OE.6a: Expect that identifying specific patient/parent concerns will save time in vaccine discussions OE.6b: Expect that identifying specific patient/parent concerns will improve the likelihood of patients initiating the HPV vaccine series	NB.6: Recognize that vaccinating all eligible patients against HPV in accordance with ACIP guidelines is part of the clinic network’s expectation for optimal physician performance
PO.7. Communicate tailored message to address specific patient/parent concern	K.7: Describe key talking points to address common parental concerns related to vaccine hesitancy	SSE.7a: Demonstrate ability to deliver tailored message to patients/parents about their specific vaccination concerns SSE.7b: Express confidence in ability to deliver tailored message to patients/parents about their specific vaccination concerns	OE.7a: Expect that delivering tailored messages to address patient/parent vaccination concerns will save time in vaccine discussions OE.7b: Expect that delivering tailored messages to address patient/parent vaccination concerns will improve the likelihood of patients initiating the HPV vaccine series	NB.7: Recognize that vaccinating all eligible patients against HPV in accordance with ACIP guidelines is part of the clinic network’s expectation for optimal physician performance
PO.8. Remind patients to schedule 2nd and 3rd HPV vaccine dose before leaving the office	K.8a: Describe CDC’s recommended dosing schedule for the HPV vaccine series K.8b: Recognize the importance of series completion for optimal cancer prevention benefits	SSE.8a: Demonstrate ability to remind patients to schedule 2nd and 3rd vaccine dose before leaving the office SSE.8b: Express confidence in ability to remind patients to schedule 2nd and 3rd HPV vaccine dose before leaving the office	OE.8: Expect that reminding patients to schedule their 2nd and 3rd HPV vaccine dose will improve the likelihood of patients completing the HPV vaccine series	NB.8: Recognize that vaccinating all eligible patients against HPV in accordance with ACIP guidelines is part of the clinic network’s expectation for optimal physician performance

^*Strategies include: Immunization Champions, Assessment and Feedback (A&F), CME, and Provider Cues.

^**Communication with clinical staff should include ensuring they: present HPV vaccination with the same importance as other vaccines, bundle the introduction of HPV vaccine with other vaccines when appropriate, and understand that physicians are looking to increase their vaccination rates and thus expect to vaccinate all eligible patients against HPV when they come in for a visit.

Task 2.4: Construct matrices of change objectives

Matrices were developed that cross-referenced behavioral performance objectives with psychosocial determinants to produce change objectives (Table 4). The resulting cells of each matrix contained change objectives. Change objectives described the criteria for which a specific determinant (eg, self-efficacy) could positively influence a specific performance objective.

Task 2.5: Create a logic model of change

The resultant logic model provided an encapsulated understanding of the functional components required by the AVP to impact the provider behaviors (Figure 2).

IM STEP 3: PROGRAM PLAN

Step 3 comprised the following: the generation of the AVP’s scope and sequence, the choice of theory- and evidence-based methods, and the design of practical applications to deliver change methods. Step 3 tasks were informed by evidence tables constructed in Step 1 and from the research team’s collective academic and clinical experience. Regular planning group meetings and brainstorming informed the AVP plan.

Task 3.1: Generate program themes, components, scope, and sequence

The theoretical framework for the AVP is based in Social Cognitive Theory,⁴⁴ Theory of Reasoned Action,⁴⁵ the Health Belief Model, ⁴⁶ clinical guidelines for HPV vaccination,⁴ and empirical evidence drawn from the review of literature on evidence-based strategies to increase HPV vaccination rates. The development challenge was to meet both the clinic provider and staff needs in a format for easy institutionalization within clinics. Components. Intervention components comprised the following: immunization champions, A&F reports, provider online continuing education (CE), EHR provider cues, and parent vaccination reminders (Figure 3). Design documents and schematics were produced by the project team, reviewed by stakeholders, and piloted with providers in situ in advisory clinics prior to implementation (detailed in task 4.4 below). Scope. AVP scope was defined by evidence-based strategies shown to be efficacious in increasing HPV vaccination rates in clinic settings. Provider interviews (described previously) and observation of clinic workflow suggested the scope and sequence of the AVP functions and rollout (Figure 3) and is described in detail in step 4 below. Theme. The AVP was designed as a sequential rollout of strategies with minimal disruption to clinic flow. The title Adolescent Vaccination Program (AVP) was initially a working title during development. Despite having broader connotations beyond HPV, the name stuck during field testing.

Figure 3.

AVP: System rollout of evidence-based strategies into network clinics

Task 3.2: Choose theory- and evidence-based change methods

Individual Behaviors

Theoretically and empirically based methods varied for each AVP component. Methods included assessment of HPV vaccination behaviors, feedback on HPV vaccination rates, reinforcement for behavioral successes, goalsetting to address improvement in HPV vaccination rates, advance organizers and cues for real-time alerts to instigate HPV vaccinations, self-monitoring of HPV vaccination behaviors, facilitation and linkage to skills training, and technical support as needed (Table 5). The project team selected methods based on empirical evidence for their use to impact the target determinants (exemplified in Tables 1 and 2).¹³

Table 5. Example of methods and practical applications used in AVP to impact determinants for vaccination behavior.

Behavioral outcome: Physicians will vaccinate eligible patients against HPV in accordance with ACIP guidelines

Performance objective: PO2. Review quarterly assessment and feedback (A&F) report for HPV vaccination

# Objective	Method	Practical Application
KNOWLEDGE
K.2a: Identify when and how A&F reports will be delivered K.2b: Describe the content of the A&F reports K.2c: Recognize that A&F is one of the most effective strategies to promote vaccination	Skill building and guided practice Chunking Tailoring Feedback Consciousness raising	Champions receive one champion binder to hold A&F reports, newsletters, and information about webinars for providers to access. Champions send logs recording each provider’s receipt of the A&F report back to the project team. Champions educate providers about the effectiveness of A&F reports. Providers engage in CEs, which provide education regarding the effectiveness of A&F reports.
SKILLS AND SELF-EFFICACY
SSE.2a: Demonstrate ability to state personal and clinic-level vaccine rates from a quarterly A&F report SSE.2b: Demonstrate ability to state personal HPV vaccination goal for the following quarter SSE.2c: Express confidence in ability to interpret A&F reports	Elaboration Reinforcement Goal setting Tailoring	A&F reports colorful images, graphs, and tailored reports to display information about clinic- and provider-level vaccination rates. Clinic-level reports provide vaccination rates for each provider in the clinic and the clinic’s rate in comparison to all other network clinics. Provider-level reports include messaging and badges to encourage providers who have met ACIP vaccine and initiation and completion goals to continue their strong work in cancer prevention. Provider-level reports inform providers when they have not met their vaccination prevention and completion goals and provide the number of additional vaccination initiations and completions needed to meet ACIP goals in the coming quarter.
OUTCOME EXPECTATIONS
OE.2a: Expect that reviewing A&F reports will allow his/her clinic and staff to track progress toward HPV vaccination goals OE.2b: Expect that reviewing A&F reports will improve personal and clinic-level HPV vaccination rates	Self-assessment	Quarterly A&F reports are stored in the champion binder for tracking provider and clinic vaccination rate progress from quarter to quarter.
NORMATIVE BELIEFS
NB.2: Recognize that vaccinating all eligible patients against HPV in accordance with ACIP guidelines is part of the clinic network’s expectation for optimal physician performance	Persuasive communication Information about others’ approval	Physician newsletters from the clinic network director provides messaging regarding the network’s vaccination initiation and completion goals.

Clinic Environment

Quality-of-Care Measures. Published quality-of-care measures for clinical practice were consulted to determine context of use for the AVP. The AVP was aligned with the Healthy People 2020 Guideline and HEDIS benchmarks of 80% vaccination for eligible patients. HEDIS metrics for quality of care have been adopted as best-practice standards for U.S. clinics.²³ Clinic Task Analysis. Task analysis was conducted in each of the participating clinics to examine data flow within the clinic, provider decision making, interaction points between the patient and provider or clinic staff, and interaction with the EHR. This identified logical opportunities for adoption and implementation of evidence-based strategies (Figure 4).

FIGURE 4.

Clinic task analysis flow.

Task 3.3: Select or design practical applications to deliver change methods

Practical applications were selected to operationalize the theory-based change methods in ways that fit the population and setting. The AVP was designed for easy adoption by clinic providers and staff. The champions provided an acknowledged point of contact, an “embedded” advocate for the AVP, and a mediator for delivery of AVP strategy rollout. Clinic information technology was used to provide online CE training (through HealthStream, the online portal for provider education within the clinical network), provider cues for HPV vaccination eligibility (through Epic), and patient reminder notifications (through MyChart). This is discussed further in Task 5.3.

IM STEP 4: PROGRAM PRODUCTION

Step 4 comprised refinement of the AVP’s structure and organization, planning for program materials, drafting of messages and materials, and pretesting, refinement, and production of materials.¹³

Task 4.1: Refine program structure and organization

Evidence-based provider-level strategies, previously described in the empirical literature (step 1), informed the development and adaptation of AVP component strategies. The AVP included an implementation strategy (AVP champions embedded in each clinic) and 4 evidence-based interventions (goal-based A&F, provider education, provider reminders, and tailored patient reminders) that provide strong evidence when used in combination. A description of each strategy and its implementation are described below.

AVP champions

Immunization champions are an implementation strategy. They serve as advocates of the AVP and as mediators for rollout of evidence-based strategies. They distribute A&F reports to physicians (physician report) and clinic staff (nurses, physician assistants, and MAs) and clinic managers (clinic level report), promote CE completion, and announce implementation of provider reminders. Two AVP champions were selected per clinic and typically comprised 1 site leader or physician and 1 clinical supervisor or clinic staff member. Champion recruitment comprised an email sent from the clinic network’s CMO requesting AVP champions be instituted. Champions participated in four 30-minute lunchtime webinar trainings that occurred prior to each strategy rollout. Webinars comprised the following: 1) an overview of project goals and objectives; 2) evidence-based strategies; 3) how to implement and monitor intervention strategies; 4) resources and technical support from the project team; and 5) Q&A. Webinars were conducted live and recorded for later use. AnyMeeting, an online platform for webinar delivery, was used to host the webinars. The same physician who narrated the provider education modules recorded the narration for champion webinars. Champions received a binder to store resources to assist AVP implementation. The binders included an overview of the AVP, contact lists of the project team, a directory of all champions within the clinic, an introduction from the CMO, A&F reports from each quarter, printed webinars (including PowerPoint slides), fact sheets, information about future webinars and initiatives, and resources (Qlikview tutorial and CDC HPV tip sheet for health care providers, promotional flyers, and tracking forms).

Assessment and Feedback reports

A&F reports were designed for physicians, clinic managers, and clinic staff to evaluate their past and current vaccine rates (Figure 5). Reports provided to clinic staff and practice managers contained clinic-level data (clinic vaccination rates) while reports provided to physicians also contained personalized information on vaccination performance and vaccination goals. Content of the physician reports was particularly informed by CDC’s Assessment, Feedback, Incentives, and Xchange (AFIX) program strategies for improving HPV vaccination. SAC feedback guided iterative development of the report including data presented, layout, colors, and messaging. A&F reports comprised the following: 1) vaccine trends (Tdap, MCV, HPV) across the network clinics, 2) quarterly vaccination rates for each clinic, and 3) quarterly vaccination rates for each provider. Metrics included percentage of eligible patients who have ever received vaccines for Tdap, MCV, or HPV, and percentage of patients who have completed the HPV vaccine series. Also included were tailored text summaries for each provider comprising either a target goal (ie, “To meet the national goal of 80% HPV vaccination over the next year, you need to initiate at LEAST ___ patients per quarter”) or a reinforcement if the provider reached 80% HPV series initiation, 60% series completion, or both (ie, “WOW! Thank you for your OUTSTANDING work in Cancer Prevention! Keep up the good work!”). Providers who initiated or completed the HPV series equal to or above these goals also received a badge of recognition. Clinics meeting the 80% initiation criterion also received a badge stating: “All doctors >80% HPV Series Initiation.” Qlikview, an application within the network’s EHR system, was used to generate and refine monthly data by the project team statistician, who translated this into graphic displays for inclusion in the quarterly A&F reports. The team delivered the reports to clinic champions, who distributed them to each physician within their clinic in February, May, August, and November.

Figure 5.

Assessment and feedback reports

Provider education

A comprehensive online continuing education (CE) for doctors (continuing medical education, CME) and nurses (continuing nursing education, CNE) was developed for network pediatricians, nurses, and clinical staff (Figure 6). CE objectives were to: 1) inform providers about emerging HPV vaccination guidelines and new initiatives being implemented by the network, and 2) provide skills to help providers engage with and motivate patients/parents to adhere to vaccination schedules. Content comprised the following: 1) ethical principles in HPV vaccination; 2) about HPV; 3) latest guidelines on the HPV vaccination; 4) evidence-based strategies for increasing HPV vaccination; and 5) recommended communication strategies (e.g., assertive bundled recommendations) and rolling with resistance when parents are vaccine-hesitant. A network physician provided voice narration. The finalized CE was reviewed by the SAC and accepted by the accreditation board of the network for credit approval. Clinic network leadership approved the HPV training module for 1 hour of ethics CE credit. A medical ethicist collaborated with the team to incorporate ethics principles (e.g., the principle of justice encompasses the need to recommend HPV vaccination equally and universally to all eligible patients). Ethics credit provided further incentive. Provider CE was implemented in the form of a self-paced CE module delivered through HealthStream, an online content management system.

Figure 6.

Provider continuing education

Provider reminders

Provider behavioral cues comprised Best Practice Advisory (BPA) alerts to enable providers and staff to easily identify age-eligible patients due or overdue for HPV vaccination. The BPAs were developed in collaboration with the clinic network’s EHR team, including the physician developer of pre-existing asthma and flu BPAs, and informed by CDC guidelines (Figure 7). The algorithm for the alert system comprised the following: 1) alert for first HPV vaccine (HPV-1) if patient is a female 12–26 years of age or male 12–21 years of age AND has no prior HPV vaccination; 2) alert for second HPV dose (HPV-2) if patient is female or male 12–26 years of age AND received HPV-1 before 15 years of age AND 6 months or more have passed since HPV-1 vaccine OR patient is female or male 12–26 years of age AND received HPV-1 at 15 years of age or older AND 1 month or more has passed since HPV-1; and 3) alert for third dose of HPV vaccine (HPV-3) if patient is female or male 12–26 years of age AND received HPV-1 at 15 years of age or older AND 4 months or more have passed since HPV-2 vaccine. While ACIP recommends routine HPV vaccine initiation beginning at age 11, the network preferred to commence the BPA alerts beginning at age 12. BPA alerts commenced at age 12 because HPV vaccination was already considered standard care for the 11-year-old visit, when the vaccine is included in the order set. The BPAs were added to the Epic system and modified to reflect updates in CDC guidelines, most notably in 2017 when the 2-dose schedule for adolescents under 16 years of age was released. During clinical encounters with a patient who is due or overdue for HPV vaccination, an alert appears in the patient’s EHR, prompting the provider to initiate HPV vaccination. The BPA system sends alerts during both well-child and sick visits. Alerts contain a link to order the vaccine and multiple response options for case records: done, ordered, patient declined, patient not eligible, discussed, or not addressed.

Figure 7.

Provider reminders

Patient reminders

The American Academy of Pediatrics (AAP) reminder and recall systems guidelines informed the development of reminder messages for parents with vaccine-eligible children (Figure 8). Messages were developed to remind parents to Initiate the HPV vaccine and to schedule 2^nd and, as appropriate, 3^rd doses. Messages followed existing formatting standards used by the network and were reviewed for content by the SAC before being incorporated into an automatic messaging system. Patients who were identified as 10 years and 11 months of age through 17 years and vaccine eligible were flagged to receive targeted reminders. This was done using an existing Pediatric Wellness Registry integrated within the network’s Epic and MyChart systems. Automated messages were sent 1 month before the child’s HPV vaccine due date. Parents were able to see their upcoming due date for their child’s HPV immunization on the Preventive Care page in MyChart, the patient-facing application of the Epic EHR.

FIGURE 8.

Patient reminders

Task 4.2: Prepare plans for program materials

AVP design documents provided a blueprint of the functional specifications and rollout sequence of each strategy (Figure 3). Project team conference calls and face-to-face meetings provided iterative review and feedback on the design. Design documents described content, design features, functionality, language, logistics of use and implementation in the clinic, orientation needs, and evaluation specifications. The SAC had few concerns about the use of the AVP within the clinics, recommending only minor modifications to layout, clarity of content, and ease of access for minimal disruption to clinic services.

Task 4.3: Draft messages, materials, and protocols

Program drafting followed a stepped sequence. Each component draft built upon the iterative review of previous developmental drafts, allowing multiple reviews. Strategies were developed for deployment using pre-existing delivery platforms: CE provider education on narrated PowerPoint slides on HealthStream, Epic cues as programming logic for inclusion in the Epic EHR, and patient reminders as text statements formatted for insertion into MyChart email announcements (discussed further in Task 5.3).

Task 4.4: Pretest, refine, and produce materials

Each AVP strategy prototype was pretested and refined through an in-house review and, as amenable, a feasibility pilot test in the 6 advisory clinics.

AVP review by the SAC.

The AVP project team and SAC previewed AVP content and function for consistency with clinic mission and professional protocols, and for anomalies such as logical inconsistencies, illegibility, or unappealing format. Review was conducted in regular meetings or via e-mail. Feedback was collated and approved by the project directors prior to feasibility testing.

AVP component feasibility testing in advisory clinics.

Advisory clinics followed protocols to select an AVP champion and rolled out provider assessment and feedback over a 2-week period. Champions completed logs recording any problems encountered. One champion was interviewed at each of the 6 stakeholder clinics. Champions expressed satisfaction with the process and their role in distributing and tracking the assessment and feedback reports, and the champions noted that physicians liked the reports and were interested in comparing their rates with others. Champion recommendations led to protocol adjustments to deliver reports at the beginning of the month prior to monthly meetings and to provide a 2-week window to return Distribution Logs.

IM STEP 5: IMPLEMENTATION PLAN

Step 5 comprised the description of potential program implementers, defining the outcomes and performance objectives for implementation, constructing matrices of change objectives for implementation, and designing implementation interventions.¹³

Task 5.1: Identify potential program implementers

The AVP was designed for use by pediatric primary care clinic providers and staff. Potential adopters included the director of the pediatric network, clinic directors, providers (pediatricians), and clinic managers.

Task 5.2: State outcomes and performance objectives for implementation

Performance objectives for adoption were brainstormed by the project team with consideration of the decision-makers in the network and informed by the IM framework¹³ and characteristics for diffusion of innovation.⁴⁷ Outcomes included that implementers would recognize a need for the AVP and its relative advantage and would make a formal commitment to use information technology (IT). Steps for implementation included that the clinic network director would assess the need for a program to initiate strategies to increase HPV vaccination, review the AVP and its components and note objectives and relative advantages for its adoption, obtain feedback from clinic staff on potential barriers to/advantages of adopting the AVP, and agree to trial the AVP components.

Task 5.3: Construct matrices of change objectives for implementation & Task 5.4: Design implementation interventions

Critical opportunities for AVP strategy implementation within the clinics were identified using clinic task analysis (previously described). This also helped identify existing IT channels by which to deploy the strategies (Table 6). Matrices categorized objectives for the network CMO to implement the AVP across the network and for the champions to implement the AVP within their clinic (Table 7). The AVP is more likely implemented if it is minimally disruptive to clinic activities or clinic overhead. Advantages of the AVP include its provision of resources and protocols with established feasibility and a requirement of only a single investment of resource (mainly staff time commitment) to set up an A&F report structure, CE program access, Epic cuing setup, and parent reminder message blasts. This upfront commitment is offset by significant increases in HPV vaccination rates that approach HEDIS benchmarks.

Table 6.

Processes and channels for deployment of AVP strategies

HPV	IMPLEMENTATION CHANNELS
STRATEGIES	PROMOTION	DEPLOYMENT (including IT platforms)
Immunization champions	Champions were notified of webinar trainings by e-mail.	• Live training webinars (30 min) using AnyMeeting preceded each strategy rollout. System enabling session recording and attendance tracking. • Champion binders and materials were mailed via the network mail system.
Assessment and feedback (A&F) report	Champions notified clinic personnel in regular meetings.	• Reports were generated after accessing data via a Qlikview portal within the EHR. • Printed reports were delivered to champions via the network mail service. Champions distributed reports in clinic meeting or mailboxes at their discretion. • Champions kept an additional copy of each A&F report in their binder and kept a distribution log for tracking.
Provider continued education	CE promotion by: • Champions in regular meetings; • Promotional flyer included with each A&F report; • Network CMO in monthly newsletters.	The CE was deployed online through HealthStream, the network’s online learning system, and was accessible throughout the study.
Provider Reminders	Champions notified clinic personnel in regular meetings.	Provider reminder best-practice alerts (BPAs) were included in the Epic EHR.
Parent Reminder	Champions were updated in webinar and notified clinics in regular meetings.	MyChart, a patient facing component of the Epic EHR, sent automated content and messaging to all patients listed in an existing Wellness Registry (updated nightly) that identified vaccine- and age-eligible patients. Quarterly reports tracked the number of reminders sent.

Table 7.

Matrix of key stakeholders/gatekeepers for implementation

Performance Objectives	Implementation Stakeholder
	Network CMO				Clinic Champion
AVP Strategy To facilitate Implementation of …	Knowledge	Skills and Self-efficacy	Outcome Expectations	Normative Beliefs	Knowledge	Skills and Self-efficacy	Outcome Expectations	Normative Beliefs
Assessment and Feedback (A&F) Reports	K.CMO.A&F: Recognize the content of A&F reports, their source data, and optimal implementation.	SSE.CMO.A&F: Demonstrate ability to promote A&F reports across the network via regional meetings and monthly newsletters.	OE.CMO.A&F: Expect that A&F reports will lead to increased awareness of individual-, clinic-, and network-level vaccine rates.	NB.CMO.A&F: Recognize that A&F reports can move the network’s culture toward prioritizing HPV vaccination.	K.Champ.A&F: Recognize the content of A&F reports, their source data, and optimal implementation in the clinic.	SSE.Champ.A&F: Demonstrate ability to promote and distribute A&F reports via clinic meetings and/or report distribution.	OE.Champ.A&F: Expect that A&F reports will lead to increased awareness of individual-, clinic-, and network-level vaccine rates.	NB.Champ.A&F: Recognize that A&F reports can move the clinic’s culture toward prioritizing HPV vaccination.
Continued Education (CE)	K.CMO.A&F: Recognize the content of CE, source data, and optimal implementation in the network.	SSE.CMO.A&F: Demonstrate ability to promote the CE reports across the network via regional meetings and monthly newsletters.	OE.CMO.A&F: Expect that CE will lead to increased HPV-related knowledge and skills of network personnel.	NB.CMO.A&F: Recognize that CE can increase the network’s collective awareness and skills for HPV vaccination.	K.Champ.A&F: Recognize the content of CE, source data, and optimal implementation in the clinic.	SSE.Champ.A&F: Demonstrate ability to promote the CE via clinic meetings and/or flyer distribution and ensure linkage.	OE.Champ.A&F: Expect that CE will lead to increased HPV- related knowledge and skills of clinic personnel.	NB.Champ.A&F: Recognize that CE can increase the clinics’ collective awareness and skills for HPV vaccination.
Provider Cues (PCs)	K.CMO.A&F: Recognize the content of PCs, their logic, source data, and optimal implementation in the network.	SSE.CMO.A&F: Demonstrate ability to promote PCs across the network via regional meetings and monthly newsletters and facilitate inclusion in EHR..	OE.CMO.A&F: Expect that PCs will lead to decreased missed opportunities for HPV vaccination across the network.	NB.CMO.A&F: Recognize that PCs are a component of optimal network performance to meet national HPV vaccination guidelines.	K.Champ.A&F: Recognize the content of PCs, source data, and optimal implementation in the clinic.	SSE.Champ.A&F: Demonstrate ability to promote via clinic meetings and/or notice distribution.	OE.Champ.A&F: Expect that PCs will lead to decreased missed opportunities for HPV vaccination within the clinic.	NB.Champ.A&F: Recognize that PCs are a component of optimal clinic performance to meet national HPV vaccination guidelines.
Parent Reminders (PRs)	K.CMO.A&F: Recognize the content of PRs, their source data, layout, and optimal implementation in the network.	SSE.CMO.A&F: Demonstrate ability to promote PRs across the network via regional meetings and monthly newsletters and facilitate inclusion in MyChart.	OE.CMO.A&F: Expect that PRs will lead to increased appointments for HPV vaccination across the network.	NB.CMO.A&F: Recognize that PRs are a component of optimal network performance to meet national HPV vaccination guidelines.	K.Champ.A&F: Recognize the content of PRs, source data, and optimal implementation in the clinic.	SSE.Champ.A&F: Demonstrate ability to promote PRs via clinic meetings and/or notice distribution.	OE.Champ.A&F: Expect that PRs will lead to increased appointments for HPV vaccination in the clinic.	NB.Champ.A&F: Recognize that PRs are a component of optimal network performance to meet national HPV vaccination guidelines.

IM STEP 6: EVALUATION PLAN

Step 6 comprised effect and process evaluation questions, developing indicators and measures of assessment, and specifying an evaluation design.

Task 6.1: Write effect and process evaluation questions

The primary question to be addressed in planning the evaluation of the AVP was: Does the use of the AVP within a primary prevention pediatric clinical network over a 3-year period increase HPV vaccine initiation and completion rates? Stated as an alternative testable empirical hypothesis: A clinical network that uses the AVP in the context of their usual clinic operations over a 3-year period will demonstrate significantly higher rates of HPV vaccine initiation and completion compared to rates prior to AVP implementation. Planned process evaluation questions included assessment of factors that mediate the success of the AVP as well as facilitating its implementation. These include intervention exposure (number of A&F reports received, number of providers and staff completing the CE); impact on patient-provider communication (change to a bundled vaccine recommendation); application of provider cues within the EHR; and institution of patient reminders.

Task 6.2: Develop indicators and measures for assessment

Evaluation of the AVP focused on collection of centralized data on vaccination initiation and completion of the HPV vaccine measured as a binary variable (yes/no). Initiation was defined as receiving at least 1 dose of the HPV vaccine. Completion was defined as receiving 3 doses in years 2014 and 2015 and as receiving 2 or 3 doses, depending on age at initiation, for 2016–2017. This dosage change corresponded with the updated guideline that went into effect in October 2016. Quarterly rates were calculated at physician and clinic levels, and annual rates were calculated for all clinics combined. AVP data were compared to state-level data from the National Immunization Survey (NIS)-Teen for the years 2014, 2015, and 2016.⁴⁸ An age group reported by NIS-Teen (13–17 years) was the primary comparison with the network in order to evaluate the effect of secular trends.

Planned process measures to assess implementation fidelity were specified for each strategy. These included a champion attendance log (to indicate attendance at webinars), a provider signoff sheet (to indicate receipt of assessment and feedback reports by providers), a back-end data base (to record CE use by providers and clinic staff), test results from the network IT (to confirm accuracy and ongoing functionality of EHR-based cues), and reports on number of vaccination reminders sent to parents of vaccine-eligible youth. Plans also included records of any reported refusal to adopt strategies or barriers to implementation whether organizational or logistic.

Task 6.3: Specify evaluation design

The evaluation design for the AVP was an ecological single-group pre-/post-test evaluation within the 51-clinic network. A randomized design could not be implemented without contamination across study conditions. Further, the funding mechanism focused primarily on delivery of services and evaluation, and secondarily on research. The systems-based rollout of the AVP components took place within all 51 clinics simultaneously. Providers and staff in each clinic were invited to complete the baseline survey prior to AVP rollout and again at the end of the evaluation period. Cumulated vaccination rate data were assessed at baseline and quarterly throughout the project in order to give feedback to the physicians and clinics on their A&F reports. Clinics then rolled out the AVP strategies according to a sequenced timeline. Primary analysis involved comparisons of changes in vaccination rates from baseline through 4 years using logistic regression. Limitations of the evaluation design are those of internal validity because a quasi experimental design has no randomization or comparison group. Although this design can establish a trend, it cannot definitively attribute results to the AVP alone. However, it is noteworthy that the AVP was associated with significant increases in HPV vaccination initiation and completion rates even after considering state-level secular trends based on the NIS-Teen.¹²

DISCUSSION

The AVP is a successful HPV vaccination program designed to address the need identified in the Community Guide for implementation of evidence-based strategies to increase HPV vaccination initiation and completion rates and to increase rates to be commensurate with those of Tdap and MCV, targeting HEDIS criteria of 80%. It is also responsive to the Healthy People 2020 objective to increase the proportion of persons receiving HPV vaccination.⁶

The IM framework was used to design the AVP due to its potential utility in developing multilevel systems-based approaches. Advantages of the framework include the imposition of a systematic approach; thoroughness in detailing needs and solutions informed by theoretical and empirical literature; encouraging critical thinking regarding implementation, evaluation, and dissemination; and ensuring that priority populations were consulted throughout. The IM framework is built to accommodate the use of theories designed to inform development of behavioral change interventions (eg, Social Cognitive Theory,⁴⁴ Theory of Reasoned Action,⁴⁵ Health Belief Model⁴⁶) as well as those designed to inform the development and packaging of implementation strategies that facilitate the use of interventions within clinics (eg, Diffusion of Innovation⁴⁷. Consolidated Framework for Implementation Research).^48,49

The resulting components of AVP are theory- and evidence-based, packaged into a product that can be integrated into an existing clinic network’s workflow and technology system. Though many of the components found in this study have been used previously, they are independently insufficient; combining them with other evidence-based components and a novel parent educational app reflects the next generation of interventions to increase HPV vaccine rates.

Other interventions focused on HPV vaccine rates have been effective to a varying extent. Though many interventions have tested the efficacy and effectiveness of one component or one level of an HPV intervention,⁹ this is the first intervention to our knowledge to use IM to develop a successful multilevel intervention focused on increasing HPV vaccine rates in a pediatric clinic setting. To date, there is a lack of ubiquitous adoption and implementation of evidence-based practices. The current study targets the provider and clinic levels to influence behavioral and system-level changes.

The current study has several strengths. First, a comprehensive team of experts and potential participants on the provider level developed the intervention using dynamic feedback from those who could benefit most from the intervention’s components. Second, the intervention received strong “buy-in” from the participating pediatric network, which provides health care for a significant portion of the city’s pediatric care population. Third, the intervention was developed to address needs at organizational, provider, and patient levels, which contribute to a higher likelihood of behavior change than focusing on one level alone.^9,50

Findings need interpretation in light of study limitations. The generalizability of the AVP is unknown because it was developed with the participation of one pediatric network and limited to a single geographic urban area. However, by utilizing one of the largest pediatric networks in the U.S., including 51 clinics of various size, this study helps build evidence of feasibility and acceptability across diverse clinic settings. Further, while the evaluation of the AVP suggests success in terms of increasing HPV vaccine rates, the relative efficacy and impact of each individual component on the outcomes of our intervention are unclear.

The AVP development presented here represents one case study application for a systems-based intervention in a clinical context. In this capacity, it provides a guide for future development in analogous domains, populations, and applications. However, in practice the degree of fidelity to IM core processes varied with each development task. The formality of posing questions, brainstorming answers, reviewing findings from published research, accessing and using theory, identifying and addressing the need for new research, and formulating the working list of answers varied among components of A&F, CE, cues, and reminders dependent on project constraints and existing practices. Also, the evaluation plan was limited to the period of study and did not include an assessment of sustainability in the long term. IM was conducive to providing an innovative multicomponent approach to implementing evidence-based strategies within primary care pediatric clinics. By providing evidence-based tools and procedures for identifying and assisting clinics to increase HPV vaccination rates, this study contributes to the national goal.

CONCLUSION

Limited impact of current interventions to increase use of HPV vaccine among adolescents represents a missed opportunity to prevent multiple types of HPV-related cancer. IM provided a framework to develop a multilevel, multicomponent intervention aimed at clinic system, providers, and parents to promote implementation of evidence-based strategies to increase HPV vaccine uptake and completion among adolescents ages 11–17. The AVP’s feasibility for clinic use and efficacy in increasing HPV vaccination in a large pediatric clinic network in the southwestern United States is testament to the utility of IM as a framework for development of systems-based interventions.

Future directions

Future directions for the AVP include determining overall initiation and completion outcomes and testing dissemination and implementation for use among other clinic networks. The AVP is currently being expanded to a smaller pediatric clinic network in the same state. Of note, members of the original clinic network in this study have requested assistance with permanent adoption of AVP components for sustainability within the clinics. This interest in the AVP suggests that broader dissemination is indicated.