Coaching primary care clinics for HPV vaccination quality improvement: Comparing in-person and webinar implementation

Providers who received quality improvement coaching reported more positive attitudes and self-efficacy toward HPV vaccination. In-person coaching reached more providers per clinic than webinar coaching.

Abstract

State health departments commonly use quality improvement coaching as an implementation strategy for improving low human papillomavirus (HPV) vaccination coverage, but such coaching can be resource intensive. To explore opportunities for improving efficiency, we compared in-person and webinar delivery of coaching sessions on implementation outcomes, including reach, acceptability, and delivery cost. In 2015, we randomly assigned 148 high-volume primary care clinics in Illinois, Michigan, and Washington State to receive either in-person or webinar coaching. Coaching sessions lasted about 1 hr and used our Immunization Report Card to facilitate assessment and feedback. Clinics served over 213,000 patients ages 11–17. We used provider surveys and delivery cost assessment to collect implementation data. This report is focused exclusively on the implementation aspects of the intervention. More providers attended in-person than webinar coaching sessions (mean 9 vs. 5 providers per clinic, respectively, p = .004). More providers shared the Immunization Report Card at clinic staff meetings in the in-person than webinar arm (49% vs. 20%; p = .029). In both arms, providers’ belief that their clinics’ HPV vaccination coverage was too low increased, as did their self-efficacy to help their clinics improve (p < .05). Providers rated coaching sessions in the two arms equally highly on acceptability. Delivery cost per clinic was $733 for in-person coaching versus $461 for webinar coaching. In-person and webinar coaching were well received and yielded improvements in provider beliefs and self-efficacy regarding HPV vaccine quality improvement. In summary, in-person coaching cost more than webinar coaching per clinic reached, but reached more providers. Further implementation research is needed to understand how and for whom webinar coaching may be appropriate.

Implications.

Practice: Public health practitioners who deliver immunization quality improvement programs may be able to reach more providers through in-person coaching than webinar coaching.

Policy: Public health organizations that sponsor immunization quality improvement programs should allocate funding for costs incurred by in-person delivery.

Research: Future research should identify ways for increasing the reach of webinar coaching so as to improve the effectiveness of this highly acceptable, low-cost alternative to in-person coaching.

BACKGROUND

Human papillomavirus (HPV) vaccine is licensed to prevent several HPV-related cancers [1]. Routine vaccination at age 11 or 12 years has been recommended by the Advisory Committee on Immunization Practices (ACIP) since 2006 for females and since 2011 for males [2]. In 2016, ACIP updated HPV vaccine recommendations to include a two-dose schedule for adolescents initiating the vaccination series before their 15th birthday (0, 6–12 month schedule) [3]. Three doses remain recommended for those who initiate the vaccination series at age 15 or older (0, 1–2, 6 month schedule) [3]. Despite effectiveness and national recommendations for routine administration of HPV vaccine, many age-eligible adolescents in the USA are not vaccinated according to schedule. As of 2016, less than two thirds (60%) of 13- to 17-year olds in the USA had initiated HPV vaccination [4]. This coverage is considerably lower than for other vaccines in the adolescent platform: tetanus, diphtheria, and pertussis (Tdap; 88%) and meningococcal conjugate (82%) [4].

In the USA, state policies have contributed to high rates of Tdap and meningitis vaccination as most states require these vaccines for school entry [5]. However, only Rhode Island, Virginia, and Washington DC have implemented such policies for HPV vaccination [6]. Political concerns, including more limited support for HPV vaccine school-entry requirements among parents [7] and healthcare providers [8], have been barriers to the enactment of these laws. Thus, successful delivery of HPV vaccine depends to a large extent on healthcare providers’ recommendation practices. A key reason that adolescents are not up-to-date with HPV vaccination is that many healthcare providers do not consistently recommend that adolescents complete the series by age 13 [9,10]. Provider-level barriers to HPV vaccination underscore the need for immunization quality improvement (QI) interventions that can decrease missed opportunities in primary care clinics [11].

One evidence-based approach to immunization QI is the Centers for Disease Control and Prevention’s (CDC) Assessment, Feedback, Incentives and eXchange (AFIX) program [12]. AFIX uses strategies, including assessment and feedback, in the context of brief, in-person coaching sessions. In these sessions, an immunization QI coach from a state or regional health department evaluates a clinic’s vaccination coverage levels and delivers training to clinic providers on immunization best practices. Based on data that show that AFIX increases early childhood vaccination coverage [13], the CDC recommends that each year health departments deliver AFIX to at least one quarter of providers of federally funded vaccines [12]. The AFIX program’s existing infrastructure and success raise the possibility that HPV vaccine-focused AFIX coaching could increase coverage among adolescents. In a pilot trial, we conducted with 91 primary care clinics in North Carolina and found that, at 5-month follow-up, AFIX coaching modestly improved HPV vaccination coverage among adolescents, ages 11–12, compared to control clinics [14]. Although many health departments have begun using a modified version of AFIX to address the underuse of adolescent vaccines, including HPV vaccine, the use of AFIX to improve adolescent vaccination is understudied. Rigorous evaluation of an HPV vaccine-focused AFIX, in states with a range of capacity to deliver immunization QI coaching, is needed to inform national practice.

To address this gap, we developed HPV vaccine-focused QI coaching tools, in partnership with immunization specialists in health departments in Illinois, Michigan, and Washington State. These tools included suggested scripts for the AFIX coaches who deliver the QI sessions, as well as materials such as an Immunization Report Card and an Action Plan (materials are available online at www.hpviq.org). In 2015, our three health department partners incorporated the tools into their routine adolescent AFIX programs. Together, we conducted a randomized controlled trial with 224 primary care clinics to evaluate the effectiveness of HPV vaccine-focused QI coaching (or “HPV AFIX”) for raising HPV vaccination coverage; clinics received the intervention via traditional, in-person coaching or interactive webinar. We found that in-person coaching achieved modest, but consistent improvements across time (6- and 12-month postintervention) and states for HPV vaccine initiation (≥1 dose) among adolescent ages 11–12, compared to no-coaching control (Gilkey, Calo, Leeman et al., in preparation). Webinar coaching also achieved statistically significant improvements but these were smaller than in-person coaching and inconsistent across time points and states. To better understand how delivery mode may affect the implementation of HPV AFIX, we conducted a process evaluation that compared in-person coaching sessions to those delivered via interactive webinar in terms of reach, acceptability, perceptions about HPV vaccine QI, adoption of recommended QI strategies, barriers to implementing QI strategies, and delivery cost.

METHODS

Clinic recruitment and assignment to trial arm

The trial included primary care clinics specializing in pediatric and family medicine. To ensure high reach, we included clinics having at least 500 patients, ages 11–17, with active records in the Illinois, Michigan, and Washington immunization information systems (IIS). In March 2015, we randomized these 925 clinics, using a 1:1:1 ratio within state, to receive either in-person HPV AFIX coaching, webinar HPV AFIX coaching, or no coaching (control). Recruitment to the study started with the first clinic in each list of study arms and alternated between arms in order to conduct in-person and webinar coaching over the same time period. In total, 76 clinics received in-person coaching, 72 clinics received webinar coaching, and 76 clinics were in the control arm. Study clinics served over 312,000 patients, ages 11–17, with active records in the IIS. This report examines the 148 clinics, with 213,006 age-eligible patients, assigned to the in-person and webinar intervention arms. We do not report further on clinics in the control arm because we did not collect implementation data from these clinics. Fifty-four percent of clinics contacted to participate in the in-person arm enrolled in the trial (21 clinics were ineligible, 20 did not respond, 24 declined or cancelled), and 49% of clinics contacted to participate in the webinar arm enrolled (17 clinics were ineligible, 37 did not respond, 22 declined or cancelled). This report is focused exclusively on the implementation aspects of the intervention.

Intervention

From April to October 2015, immunization QI coaches from the state health departments delivered one in-person or webinar HPV AFIX coaching session to each intervention clinic. Prior to each session, the QI coach used the state IIS to generate a one-page Report Card showing the clinic’s current vaccination coverage levels for three vaccines routinely delivered to adolescents: HPV, meningococcal, and Tdap vaccines. Coaching consisted of a single session, designed to be ~60 min in length, in which a QI coach met with providers (i.e., physicians, nurse practitioners, clinic managers, other vaccine providers, and clinic staff) to discuss the clinic’s immunization coverage levels. The QI coach encouraged the attendance of all of the clinic’s providers and staff, offering up to 1 hr of continuing medical education (CME) credit as an incentive for participation. The QI coach began each session by sharing the clinic’s immunization coverage assessment. Specifically, the QI coach used the Report Card to show providers how their clinic’s HPV vaccination coverage compared to their Tdap and meningococcal vaccination coverage levels, which are typically higher than HPV vaccination. This assessment gave providers a baseline they could use to measure the success of their immunization QI activities.

The QI coach then provided information about immunization best practices, including strategies to decrease missed opportunities, and evaluated the clinic’s potential areas for improvement from a list of 18 immunization best practices developed by the CDC [15]. The QI coach also discussed HPV vaccination guidelines and the importance of the vaccine providers’ role in making a strong recommendation for HPV vaccination. The QI coach asked providers to set a goal to improve their HPV vaccination coverage by delivering the first dose of HPV vaccine to at least 10% of their clinic’s 11- to 12-year-old patients over the next 6 months. The QI coach then worked with providers to identify one or two QI strategies from a one-page Action Plan to address challenges specific to their clinic to reach the HPV vaccination goal. At 3 and 6 months after the session, the QI coach sent the clinic an updated Report Card so that providers could assess their progress. These follow-up reports, along with a library of email coaching materials that we also developed with our state partners, were designed to support QI coaches to stay in touch with providers and maintain their engagement.

In-person and webinar coaching were designed to deliver the same content. Webinar coaching used an interactive software platform, such as GoToMeeting, that allowed the QI coach to interact with providers in real time, similar to in-person coaching. The software included a screen-sharing function that allowed the QI coach to show materials to coaching session attendees. QI coaches used a structured slide set with talking points for their webinars and most also used them for the in-person coaching. The trial was registered in clinicaltrials.gov (NCT02370459) and was ruled exempt by the University of North Carolina Institutional Review Board.

Implementation data collection procedure

We collected data concurrent with the implementation of HPV vaccine QI coaching from provider surveys and delivery cost assessment.

Provider surveys

Providers at each clinic completed three online surveys: a precoaching survey prior to the coaching session, a postcoaching survey directly after the coaching session, and a follow-up survey 6 months after receiving coaching. Although we encouraged the same person to respond all surveys, not all clinics had the same provider completing the three surveys. Response rates were 88%, 85%, and 62%, respectively. Completing the postcoaching survey allowed providers to claim CME credit. Coaches sent up to five reminders, by email and phone, to nonrespondents. Prior to fielding the surveys, we cognitively tested our instruments with two physicians and two nurses. Surveys are available online at www.hpviq.org.

Delivery cost

To assess the costs that state health departments incurred delivering HPV AFIX coaching, we examined weekly time logs, salary data, and nonstaffing expenses. State partners logged their time spent on intervention activities (e.g., Report Card preparation, travel, QI session delivery) via a weekly online survey throughout a 58-week trial period. Compliance for completion of the weekly time logs was 100%, with state partners reporting 16% of time logs retrospectively 1–4 weeks after being due. We assessed current salary data, including fringe benefits, for all state partners who participated in the study using publicly available state employee salary information or requesting these data from state departments of human resources. Nonstaffing expenses (i.e., mileage reimbursement, lodging, air travel, meals, and webinar-hosting fees) came from purchase orders and invoice records kept by the research coordinator using an expense tracking log. We excluded costs incurred for research purposes (e.g., sending survey reminders). Because we were interested in assessing HPV AFIX coaching costs to state health departments, we did not collect costs incurred by providers, patients, or parents.

Measures

Reach

We recorded the number and type of providers who attended coaching sessions, as well as those who claimed CME credit. We also assessed with whom providers shared the clinic’s Report Card and how they shared it via the follow-up survey (i.e., one-on-one conversations, announcements at a staff meeting, via email, or displaying the report in the office).

Acceptability

The postcoaching survey assessed overall satisfaction with items that measured the extent to which attendees found QI coaching sessions easy to understand, convenient, helpful, and well-facilitated. The 5-point response scale ranged from “strongly disagree” to “strongly agree.” The postcoaching survey also assessed the importance of individual session components, including review of the clinic’s adolescent vaccination coverage levels, discussion of provider recommendation of HPV vaccine, goal setting to improve HPV vaccination coverage, and selection of immunization QI strategies. The 5-point response scale ranged from “not important” to “extremely important.” A final post-coaching survey item assessed whether providers would, if given the choice, have preferred receiving the coaching via in-person or webinar delivery.

Beliefs and self-efficacy

Pre- and post-coaching surveys assessed beliefs that HPV vaccination QI was an important goal, that HPV vaccination coverage is low, that the clinic can improve HPV vaccination coverage, and that the provider can help lead those efforts (self-efficacy). The surveys also assessed providers’ belief that the immunization QI coaching and tools improved HPV vaccination coverage in their clinics. The 5-point response scale ranged from “strongly disagree” to “strongly agree.”

Adoption of QI strategies

The follow-up survey assessed which QI strategies, if any, clinics tried, as a result of the coaching session, to improve their HPV vaccine coverage. Strategies were as follows: encouraging providers to routinely recommend vaccination, reviewing CDC guidelines with staff, training front office staff on how to schedule vaccination appointments, signing standing orders, using reminder/recall systems, and providing educational materials to parents and patients.

Barriers to implementing QI strategies

The follow-up survey assessed the biggest barrier clinics faced to improving HPV vaccination coverage: staff turnover, limited staff time to implement QI activities, lack of support from leadership, limited stock of HPV vaccine, and time constraints during patient visits.

Analyses

We compared study arms on clinic characteristics using chi-square tests and analyses of variance. We used t tests to compare clinics in the in-person and webinar arms on mean scores related to acceptability, beliefs, and self-efficacy. These analyses accounted for provider clustering by clinic and state using the svy command [16]. Statistical tests were two-tailed with a critical alpha of .05. We also compared in-person and webinar arms within each state, adjusting for clinic level clustering of providers; we used a critical alpha of .01 to correct for multiple comparisons. After applying this correction, we did not observe significant differences between study arms within states (results not shown). We analyzed survey data using Stata 13.0 (College Station, TX).

To assess the cost to state health departments of delivering HPV vaccine QI coaching sessions, we used micro costing methods. This approach involves separately identifying, measuring, and valuing the resources utilized in the intervention [17]. Using 2015 and 2016 salaries, we calculated personnel costs by multiplying the time spent on each intervention activity by the adjusted salary per hour of the staff. We calculated intervention cost per clinic by dividing the total intervention cost for each study arm by the number of clinics in that arm. We calculated intervention cost per provider by dividing the total intervention cost for each study arm by the average number of providers per clinic who attended the QI coaching session. We analyzed cost data using Excel.

RESULTS

Overall, 165 providers completed the precoaching survey, 229 completed the postcoaching survey, and 102 completed the 6-month follow-up survey (Table 1). Across the three surveys, the majority of providers practiced in family medicine clinics (57%–68%). Most respondents were nurses (31%–46%) or clinic managers (18%–32%); physicians constituted a smaller proportion (5%–10%) of survey respondents. Many respondents had worked in their current role for ten or more years (39%–56%). Over half (61%) of those who participated in the coaching sessions and responded to the evaluation survey claimed the CME credit we offered, including 78% of participating physicians (14/18) and 65% of nurses (68/104). Claiming CME credit did not vary by delivery mode.

Table 1 |.

Respondent characteristics

	Surveys
	Pre-coaching	Post-coaching	Follow-up
	n = 165	n = 229	n = 102
	n (%)	n (%)	n (%)
Practice location
Illinois	52 (32)	58 (25)	45 (44)
Michigan	37 (22)	74 (32)	21 (21)
Washington	76 (46)	97 (42)	36 (35)
Practice specialty
Pediatrics	59 (36)	73 (32)	44 (43)
Family medicine	106 (64)	156 (68)	57 (57)
Role
Physicians	9 (5)	18 (8)	10 (10)
Nurses	56 (34)	104 (46)	32 (31)
Other vaccine providers	15 (9)	20 (9)	5 (5)
Clinic managers	47 (28)	42 (18)	33 (32)
Other staff roles	3 (2)	1 (<1)	0 (0)
Other staff	35 (21)	43 (19)	22 (22)
Years in role
<5	65 (39)	75 (33)	19 (19)
5–9	36 (22)	55 (24)	26 (25)
≥10	64 (39)	99 (43)	57 (56)
Claimed CME credit	n/a	140 (61)	n/a

Response rates for precoaching, postcoaching, and follow-up surveys were 88%, 85%, and 62%, respectively (response defined as at least 1 respondent per clinic). n/a not applicable.

Reach

In-person HPV coaching reached more providers than webinar coaching. An average of nine providers (range: 1–33) attended in-person sessions, whereas five providers (range: 1–26) attended webinar sessions (p = .004). The majority of providers (87%) reported sharing the Report Card with others in their clinics. They reported sharing the Report Card with vaccine providers (69%), clinic managers (52%), or other providers (17%). Report Card sharing occurred through in-person, one-on-one conversations (63%), announcements at a clinic’s staff meeting (39%), email communications (18%), or displaying the report in the clinic (10%). More providers in the in-person coaching arm shared the Report Card at a clinic’s staff meeting compared with those in the webinar arm (49% vs. 20%, respectively; p = .029). No other differences between arms were observed for sharing the Report Card.

Impact on beliefs, self-efficacy, and immunization QI strategies

Both in-person and webinar coaching improved providers’ beliefs about HPV vaccine QI and self-efficacy. More specifically, pre- and post-coaching comparisons demonstrated that coaching increased providers’ beliefs that their clinic’s HPV vaccination coverage was low, their belief that the clinic could improve coverage, and their self-efficacy to help lead those QI efforts (all pre/postcomparisons p < .05; Fig. 1). At 6-month follow-up, over half of respondents reported believing that implementing the QI activities improved their clinic’s HPV vaccine coverage levels (54% somewhat or strongly agreed). We did not find evidence to suggest that beliefs and self-efficacy differed across delivery modes.

Fig 1.

| Changes in providers’ beliefs and self-efficacy about immunization QI before and after receiving an HPV QI coaching session. Error bars show standard errors.

Providers in the in-person and webinar arms reported similarly on clinics’ implementation of immunization QI activities. At 6-month follow-up, the majority of clinics reported having conducted efforts to encourage providers to routinely recommend HPV vaccine (95%), provide educational materials to parents and patients (84%), and review CDC guidelines for HPV vaccination with staff (66%). Fewer than half of the clinics trained front office staff on how to schedule appointments (49%), used reminder/recall systems (47%), or signed standing orders (34%) for HPV vaccination. Most providers listed limited staff time (31%), time constraints during patient visits (33%), or staff turnover (19%) as the biggest barrier to implementing QI activities. Few providers reported lack of support from leadership (7%) or limited stock of HPV vaccine (5%) as the major barrier. Perceived barriers did not differ by arm.

Acceptability

Providers rated both in-person and webinar HPV QI coaching and tools highly on acceptability (Table 2). These measures included ease of understanding (mean = 4.5), convenience (mean = 4.3), helpfulness (mean = 4.1), and facilitation (mean = 4.4). Providers also rated highly the importance of individual intervention components such as using the Report Card to review clinic’s immunization coverage rates (mean = 4.3), discussing provider recommendation of HPV vaccine (mean = 4.3), setting a 6-month goal to improve HPV vaccination levels (mean = 3.9), and selecting immunization QI strategies (mean = 4.1). Providers in the in-person and webinar arms gave the same high scores on these measures of satisfaction and importance. In terms of intervention delivery mode, slightly more providers (p = .01) who received in-person coaching preferred this mode (85%), compared to those who received and preferred webinar coaching (79%).

Table 2 |.

Mean (SD) acceptability scores by coaching delivery mode

	In-person	Webinar
Overalla
Ease of understanding	4.5 (1.0)	4.6 (1.0)
Convenience	4.2 (1.1)	4.4 (1.0)
Helpfulness	4.0 (1.1)	4.3 (0.9)
Facilitation	4.3 (1.2)	4.5 (1.1)
Specific componentsb
Review immunization rates	4.3 (0.8)	4.3 (0.8)
Discuss provider recommendation	4.3 (0.8)	4.4 (0.7)
Set 6-month goal	4.0 (1.0)	3.8 (0.8)
Select quality improvement strategies	4.1 (0.9)	4.1 (0.8)

In-person and webinar arms had comparable acceptability scores (all comparisons p > .05). Comparisons adjusted for clustering by state and clinic.

^aItems assessed with a 5-point response scale ranging from “strongly disagree” to “strongly agree.”

^bItems assessed with a 5-point response scale ranging from “not important” to “extremely important.”

Delivery costs

The time spent per clinic to deliver the full intervention was 12.6 hr in the in-person study arm, including 1.1 hr to conduct the QI session, and 9.0 hr in the webinar arm, including 0.9 hr to conduct the QI session (Table 3). In the in-person arm, staff time traveling to coaching sessions was the most time-consuming activity (3.6 hr/clinic). In both arms, the most time consuming of the remaining activities were follow-up with clinics (3.3 hr/clinic), session preparation (2.7–2.8 hr/clinic), and recruitment and scheduling (2.0 hr/clinic).

Table 3 |.

Average staff time spent and cost per HPV AFIX coaching session

	Time spent		Delivery cost
	In-person hours (%)	Webinar hours (%)	In-person $ (%)	Webinar $ (%)
Staffing
Clinic recruitment and scheduling	2.0 (16)	2.0 (22)	117 (16)	117 (25)
Session preparation	2.7 (21)	2.8 (31)	129 (18)	136 (30)
Coaching session	1.1 (9)	0.9 (10)	65 (9)	56 (12)
Travel to/from session	3.6 (29)	n/a	233 (32)	n/a
Follow-up with clinics	3.3 (26)	3.3 (37)	141 (19)	141 (31)
Travel
Mileage and meals	n/a	n/a	20 (3)	n/a
Lodging and airfare	n/a	n/a	28 (4)	n/a
Webinar license	n/a	n/a	n/a	11 (2)
Total	12.6	9.0	733	461

n/a not applicable.

The intervention cost per clinic was much higher for in-person than webinar coaching ($733 vs. $461), but intervention cost per provider was lower for in-person than webinar coaching ($81 vs. $92, respectively). In the in-person arm, staff time traveling accounted for almost one third (32%; $233 per clinic) of the cost. Recruitment and scheduling accounted $117 per clinic, and state personnel averaged four attempts to reach and schedule clinics, which did not differ by arm. The per clinic cost associated with session preparation and follow-up was $129 (18%) and $141 (19%), respectively, for in-person coaching, and $136 (30%) and $141 (31%), respectively, for webinar coaching. Mileage, meals, lodging, and airfare accounted for 7% ($48 per clinic) of the in-person arm cost, and webinar licensing comprised 2% ($11 per clinic) of the webinar-delivered cost.

DISCUSSION

Using implementation data from a large, multistate trial with primary care clinics, this process evaluation compared in-person HPV vaccine QI coaching sessions to those delivered via interactive webinar, finding relative advantages of each. In-person delivery reached more providers and staff as more attended coaching sessions and more shared the Report Card at their clinics when compared to webinar delivery. Providers rated both arms highly in terms of acceptability and showed similar, positive changes in provider beliefs about key immunization QI activities. Webinar coaching, however, cost about one third less per clinic than in-person coaching, although it cost more per provider in the coaching session. Thus, state health departments seeking to incorporate webinar-delivered coaching to their AFIX programs must weigh the cost-saving advantage of webinars against more limited reach, particularly given that the webinar coaching produced small, inconsistent improvements in HPV vaccination coverage.

Overall, clinic participation was high across study arms. By using CME incentives and making a concerted effort to reach multiple providers per clinic, we were able to draw seven providers to QI coaching sessions on average, compared to typically one provider per clinic in our pilot trial [18]. However, in-person delivery outperformed webinar delivery on reach in the current trial with nine versus five providers participating on average, respectively. In-person participants were also more likely to share the Report Card in staff meetings. Differences in provider participation may be explained by several factors. First, in the webinar arm, providers may have had limited access to resources (e.g., multiple computers, conference space) needed to join the webinar without disrupting workflow or to accommodate a larger number of providers at the same time. On the other hand, in-person delivery may have elicited a more collegial environment or seemed more in line with usual norms for attending face-to-face staff meetings, resulting in higher participation [19]. Receiving printed copies of the Report Card, which may have occurred with in-person but not webinar delivery, may have similarly facilitated dissemination of this document to colleagues. Whatever the case, to increase reach, future efforts may wish to explore the use of recorded webinars as a way to extend reach, so that providers who are not able to join the in-person session may still benefit from the content shared.

Six months after participating in the coaching session, providers reported engaging in a variety of immunization QI activities. Almost every provider in both arms reported encouraging other clinic providers to routinely recommend HPV vaccine. A central component of the intervention was to discuss the importance of provider recommendation, and our Report Card offered providers examples on how to make a strong recommendation for HPV vaccination. This QI strategy was prioritized in our intervention as over 70% of adolescents who receive a recommendation initiate HPV vaccination [20]. However, implementation of other practice-based approaches was modest with, for example, less than half of providers reporting that their clinics used reminder/recall systems or standing orders. These two approaches are associated with increased immunization coverage among young children and are recommended by the Community Preventive Services Task Force [13]. Use of reminder/recall systems, in particular, was found to improve HPV vaccine uptake in a recent systematic review [21]. Despite the evidence, it was not a surprise to find that reminder/recall systems were not widely implemented as these systems require staff resources that not all clinics can support [22]. Furthermore, half of our sample reported that the main barriers for implementing QI activities were lack of staff time and high staff turnover. One approach to overcome these barriers is the use of a collaborative centralized reminder/recall system between health departments and clinics. A recent randomized trial with 576 primary care practices showed that a collaborative centralized reminder/recall system was more effective and more cost-effective than a practice-based system [23]. In addition, most providers reported they preferred that the health department, rather than their practice, conduct reminder/recall notifications centrally [24], showing this approach is also highly accepted among providers.

The two arms achieved very similar results in terms of acceptability and improving providers’ perceptions about HPV vaccine QI efforts. This finding is in line with our pilot intervention [18] and prior studies [19] showing satisfaction levels with online-delivered training as high as in-person delivery. Although webinar can introduce some challenges to QI coaches and providers in terms of how they interact with each other, providers rated coaches highly in the way they facilitated sessions. Our intervention materials provided coaches with structured techniques to engage providers throughout the session (e.g., a didactic PowerPoint presentation that guided discussions between QI coaches and providers and contained prompts to facilitate conversations) so, whatever the delivery mode, the session was designed to build rapport and facilitate two-way discussions. Both arms also achieved statistically significant improvements in provider beliefs and self-efficacy to help raise clinic’s HPV vaccine coverage. Taken together, these findings suggest that webinar delivery could substantially increase providers’ positive perceptions towards engaging in QI activities without adversely affecting their satisfaction with QI coaching, although data on effectiveness are needed.

Per clinic, we found that webinar coaching cost a third less than in-person coaching due to reduced travel costs. However, per provider, webinar coaching cost slightly more compared to in-person coaching due to higher provider participation in the in-person arm. Webinar coaching could be a lower-cost delivery mode if state immunization QI programs increase the number of providers who attend this type of sessions and webinars raise vaccination coverage. The lower costs of webinar coaching could extend the reach of immunization QI programs in many ways. In our trial, for example, we could have delivered HPV vaccine QI coaching to an additional 45 primary care clinics had we used the funds spent on in-person coaching for webinar coaching instead. Webinar delivery also eliminates other key challenge state health department personnel faces when delivering in-person coaching sessions, including the amount of time they spend traveling to clinics. Travel accounted for almost one third (29%) of the time reported and most of the time and cost difference between the two delivery modes. In addition, in states with large geographic areas, like Illinois, Michigan, and Washington, travel can pose a major inconvenience, so clinics in rural or isolated areas may be especially well-served by webinar delivery. Even though webinar delivery took less time, it is important to note that both delivery modes took a substantial commitment in terms of recruitment and scheduling of clinics. Delivering email coaching (i.e., 3- and 6-month follow-up Report Cards) also took a lot of time, and might be made more efficient through the use of automated email systems [25]. Trial vaccination coverage analyses are pending; webinar would need to be as effective as in-person coaching for the cost-savings to have practical value to programs.

In terms of strengths, our trial employed a strong research design, including random allocation of clinics and a large sample of clinics. We also worked with a range of state health departments, so we obtained implementation data that could resemble the capacities to deliver immunization QI coaching in many states. In our trial, contrary to our pilot in which one person delivered all coaching sessions, several QI coaches delivered in-person and webinar coaching, which improves the generalizability of our findings. Limitations include the use of self-reported measures to assess process measures; future studies should consider using fidelity monitoring to assess intervention delivery (whether or not QI coaches delivered coaching sessions as intended) and adoption (whether or not providers participated and implemented QI activities as reported). Our micro costing method focused on intervention delivery so as to compare delivery modes; we excluded factors (e.g., overhead expenses, provider costs) that would be needed to calculate overall program costs or to conduct a cost-effectiveness analysis. Additional research is needed to understand how our findings generalize to clinics with lower patient volumes, particularly given that small clinics may have limited resources to implement immunization QI activities or reduced capacity to support webinar delivery. We focused on medium-to-large clinics as the CDC prioritized them for receiving AFIX because they can have a larger impact on vaccination rates [12]. Lastly, future research should explore the relationship between implementation processes (e.g., reach, acceptability, perceptions about HPV vaccine QI, barriers to implementing QI strategies) and vaccination rates.

CONCLUSIONS

The CDC is currently investing resources in strengthening AFIX for HPV vaccination, and our trial implementation findings may inform these efforts to maximize the effective use and integration of QI coaching strategies into existing state immunization programs. Webinar technology could be a promising tool for delivering the AFIX model, but more research is needed to overcome the more limited reach of this approach and to understand the impact of webinar coaching on vaccination coverage. State health departments should focus on how to overcome the shortcomings of this method, while capitalizing on the cost and time savings and broad acceptability of this approach to HPV vaccine QI efforts. Forthcoming effectiveness data will be important for understanding how and for whom webinar delivery of HPV vaccine QI coaching and tools is appropriate.

Funding:

The Robert Wood Johnson Foundation provided a research grant that funded the trial (grant no. 71272). Authors’ time was supported by training and career development grants from the National Cancer Institute: R25CA116339 and K22CA186979. Funders played no role in (a) study design; (b) the collection, analysis, and interpretation of data; (c) the writing of the report; or (d) the decision to submit the manuscript for publication. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Cancer Institute or the Robert Wood Johnson Foundation.

Compliance with Ethical Standards

Primary Data: The findings reported have not been previously published, and this manuscript is not being simultaneously submitted elsewhere. The authors presented some preliminary data from this work at the 37th Annual Meeting and Scientific Sessions of the Society of Behavioral Medicine. The authors have full control of all primary data and agree to allow the journal to review the data if requested.

Informed Consent: The trial was ruled exempt by the University of North Carolina Institutional Review Board, including the collection of informed consent from participants.

Statement of Human Rights: All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Statement on the Welfare of Animals: This article does not contain any studies with animals performed by any of the authors.

Conflict of Interest: Brewer has received commercial research grants from Merck and Pfizer and is a consultant/advisory board member for Merck. He is chair of the National HPV Vaccination Roundtable. The other authors have no financial disclosures or potential conflicts of interest to report.