Two Pilots of an Implementation Science Training Program for HIV Prevention and Care Service Providers: Lessons Learned and Future Directions

Introduction

Since 2019, multiple agencies within the U.S. Department of Health and Human Services (HHS) have made significant investments in the Ending the HIV Epidemic (EHE) in the U.S. Initiative with the goal of reducing new HIV infections by 90% by 2030¹. The National Institutes of Health (NIH) has funded implementation research (IR) projects that promote partnerships between academic institutions, and state and local organizations to jointly translate findings to improve implementation of HIV testing, prevention, treatment, and response services for priority populations in EHE priority jurisdictions^1–4. While IS studies the methods to promote the integration of evidence-based interventions (EBI) into healthcare/public health practice and policy, IR is the scientific study of the use of strategies to adopt and integrate evidence-based health interventions into clinical and community settings to improve individual outcomes and benefit population health⁵. Successful development, implementation, and dissemination of such strategies necessitate strong partnerships with those implementing interventions in health care institutions, public health departments and/or community-based organization settings (hereinafter referred to as implementers). The research-to-practice gap not only comprises a deficit in knowledge of effective implementation strategies but also reflects the reality that there are still too few HIV implementers with training to conduct and consume findings from IR-related studies. As such, training is needed to increase implementers’ understanding of IS methods so they can fully utilize and inform IR studies that will produce findings relevant to their needs, context, and the populations they serve. In this commentary, we describe two pilots of an IS training tailored for implementers, their implementation, and lessons learned to enhance future trainings.

The Importance of Implementer-focused IS Trainings

As dissemination and implementation (D&I) research continues to grow and grants for D&I research are increasingly awarded, institutions have begun developing frameworks for training in this sphere⁶. IS training programs have been shown to improve comfort, competency, and academic productivity regarding IS concepts and frameworks^7–10. To date, most trainings have focused on grounding investigators in the field of IS and comparatively few have been designed for individuals responsible for EBI implementation. Juckett and colleagues conducted a scoping review to identify and describe features of training interventions that build implementation practice capacity across researchers and practitioners¹¹. Among the 14 studies they identified, nine were conducted in the US, and only five of these were solely targeted for practitioners or practitioner-in-training. This disparity of training type emphasizes that there is minimal research evaluating how to enhance implementer capacity in IS and even less resources for implementers in research partnerships.

Trainings that target implementers’ knowledge of implementation strategy selection and development are needed to enhance their ability to implement EBIs within their particular context¹². Evaluations of existing training have found that trainees continue to use their teachings and discuss what they learned after a training period, increasing their knowledge and their capacity to effectively implement new interventions.¹³ IS training provides implementers with the tools to identify determinants specific to their context, select strategies, address equity, and evaluate their implementation¹⁴.

Development of Training Curriculum

The Implementation Science Coordination Initiative (ISCI) is an NIH-funded coordination center with the goals of supporting high-quality IR and advancing IR through training and mentorship for researchers and implementation partners¹⁵. The Sociostructural Implementation Science Coordination Initiative (SISCI) is a separately funded coordinating hub for five R01 studies trialing implementation strategies for structural interventions that address social and structural determinants of health (SSDoH) contributing to HIV inequities^16–18. As part of their shared aims to increase the knowledge and use of implementation science, ISCI and SISCI developed and piloted an implementation science training curriculum tailored to implementers: Practitioner-Oriented IS Education (POISE) and the Implementation Practitioner Training (IPT), respectively.

To inform development of the training curriculum we conducted a needs assessment to understand practitioner level of IS knowledge, areas of interest in IS, potential applications of IS in their work, and preferred training modalities that met their needs and work/time constraints. The needs assessment was completed by 40 participants working in CBOs, HIV clinics, health departments, and national HIV-focused organizations and showed that 95% of participants were interested in knowing more about IS to carry out their EHE efforts, specifically: assessing and addressing implementation barriers and facilitators, how to integrate IS into practice, and selection and adaptation of implementation strategies and outcomes. Most participants indicated interest in applying findings and/or engaging in IS to improve the implementation of existing programs or services, better evaluate the implementation of their programs, and adapt programs to other populations or new settings. Their preferred training modalities were multimodal, including a combination of live webinars, hands-on workshops, videos and readings, and group trainings/consultations. We received varied suggestions on the ideal duration of a training program which depended on implementers’ ability to take time for training from the daily activities.

Based on this feedback and informed by prior research^11,19 we developed a training curriculum (Figure 1.) designed for implementers that is relevant to their context, needs-driven, based on learning theories^20,21, and interweaves social structural determinants of health. This curriculum was used for both the POISE and IPT trainings. We sought additional input on the curriculum from ISCI’s Community and Implementation Practice Committee (CIPC) which is composed of individuals working in HIV prevention and care services organizations, health department managers, and other regional and national HIV- organizations who provide technical capacity building in HIV.

Figure 1.

Curriculum Overview

To facilitate translation of IS concepts into real-world settings we developed a case study of an HIV service organization that aimed to implement a comprehensive Pre-Exposure Prophylaxis (PrEP) program (see Appendix A). The case study consists of an HIV/AIDS service organization that had three sites throughout a county: a Federally Qualified Health Center, an HIV prevention community-based organization (CBO), and a mobile clinic that provides acute harm reduction and acute care services for persons who use drugs. The case study provides a shared experience for participants who worked in diverse HIV service settings. Facilitators incorporated examples from the case study into their didactic presentations, and participants used the case study to complete IS related worksheets to identify implementation determinants and social determinants of health (SDOH), implementation strategies, and implementation outcomes (see Appendix B).

Participant Recruitment

Recruitment of training participants differed for each training course, and application invitations were sent out at different times. For IPT, eligible participants were implementation partners of investigators in the IS/SSDoH fellows training or partners of investigators funded through RFA-AI-21–024. For POISE, participants were recruited through an open invitation distributed to Centers for Disease Control and Prevention - Division of HIV Prevention and Health Resources and Services Administration – Ryan White HIV Program representatives to disseminate the application with their grantees, and The National Association of State and Territories AIDS Directors (NASTAD) who shared it with their membership. Both IPT and POISE applications described the overall goal of the course, namely, to increase their understanding of implementation science and how it could be applied to real-world practice. For POISE, applicants were additionally asked to describe the level of institutional support to participate in the training program, and to select an HIV intervention/program of interest to their organization to apply the IS knowledge from the training.

Variations in Format Between POISE and the IPT

We tested the same curriculum using two different approaches in training duration, delivery, and coaching to apply knowledge to a local project (see Figure 1). POISE was delivered through a combination of in-person and virtual sessions over 5 months and the participants applied the knowledge gained to a project of their choosing. The IPT was entirely delivered through 6 virtual sessions over 10 weeks solely using the case study.

POISE commenced with a virtual session introducing basic concepts in IS presented by a faculty member experienced in IS. A week later, ISCI hosted participants to a 1.5 day in-person workshop in Chicago guided by an expert facilitator with ample experience in capacity and technical assistance for HIV service providers using IS tools. Participants received didactic training on the main IS components (e.g. determinants, implementation strategies, and implementation outcomes) presented by ISCI faculty experienced in IS and applied the IS frameworks to the case study using the worksheets in small group break-out sessions.

In the subsequent four months after the in-person session, participants completed the same worksheets but for the intervention they selected from their organization. We hosted a monthly learning lab per IS component to discuss progress on their worksheets and provide further support as needed. Additional coaching sessions were offered with IS experts upon participant request. The training program concluded with two half-day virtual meetings where participants presented their project-specific implementation research logic model (IRLM).

The IPT consisted of a virtual version of the same didactic sessions and application of the case study similar POISE. Training sessions (1.5 hours) were held every other week over the course of three months. The case study worksheets that applied IS frameworks to the case study were completed as assignments in between sessions and discussed at the beginning of the following session. In contrast to POISE where participants applied IS knowledge to their local projects, IPT participants were offered an optional one-on-one consultation to assist them in developing an IRLM for a project related to their work.

Lessons Learned

A total of 4 participants were accepted for the IPT training and 12 for POISE. Of the 16 participants selected for either IPT or POISE, 3 participants (2 from IPT and 1 from POISE) were unable to complete the full training due to work-related conflicts. All who completed the training worked at academic health centers, health departments, or community-based organizations (CBOs), and held mainly director, program manager, or team lead roles in their organizations. For POISE, where additional participant demographic information was collected, 75% of participants identified as part of an underrepresented minority²², 67% of participants identified as female, and 33% identified as male.

Among participants who completed IPT or POISE, all 13 participants responded to the pre-post evaluation surveys to assess changes in IS knowledge using a 5-point Likert scale (from Strongly Disagree to Strongly Agree), and 11 responded (2 from IPT and 9 from POISE) to the post-training open-ended questions regarding contribution of the training and recommended improvements. Given the similarity in content and training objectives and the small sample size of IPT respondents, we conducted a pooled analysis of IPT and POISE programs of the open-ended questions assessing training satisfaction and recommendations for improvements to the training.

Table 1 presents the median pre- and post-training ratings and median post-pre rating differences in understating of IS by training program and combined programs. Overall, after the training, all IPT and POISE participants reported an increased understanding of what IS, implementation determinants, implementation strategies, implementation outcomes are, and how to identify each of these. The greatest increase in understanding (i.e. from neutral to strongly agree) was reported for understanding what IS is, and how determinants are linked to implementation strategies.

Table 1.

Median Pre-Post Differences Rating in Understanding of IS, by Training Program. The evaluations were conducted using a 5-point Likert scale (5 = Strongly Agree, 4 = Agree, 3 = Neutral, 2 = Disagree, 1 = Strongly Disagree).

	IPT (N=2)			POISE (N=11)			IPT and POISE Combined (N=13)
Implementation Science Knowledge	Pre-Training Median	Post-Training Median	Post-Pne Difference	Pre-Training Median	Post-Training Median	Post-Pre Difference	Pre-Training Median	Post-Training Median	Post-Pre Difference
I understand what Implementation Science is	3.5	5	1.5	3	5	2	3	5	2
I understand the difference between Implementation Science and Implementation Research	3	4	1	3	4	1	3	4	1
I understand what determinants are	4.5	4.5	0	4	5	1	4	5	1
I understand how and why we use determinant frameworks	4	4	0	4	5	1	4	5	1
I can identify barriers and facilitators to delivering HIV services in my organization	4.5	4.5	0	4	5	1	4	5	1
I understand what implementation strategies are	4	4.5	0.5	4	5	1	4	5	1
I understand how determinants are linked to the strategies to address them	4	4	0	3	5	2	3	5	2
I understand what implementation outcomes are	4	5	1	3	5	2	4	5	1
I can describe examples of outcomes	4	4.5	0.5	4	5	1	4	5	1
I can distinguish between implementation and clinical outcomes	3.5	4	0.5	3	5	2	3	4	1

Qualitative analyses of combined post-survey open-ended questions assessing training satisfaction and recommended improvements to the trainings, identified the following aspects of the training to be most useful: learning from and access to people experienced in IS, use of the case study to serve as examples in the training, access to IS frameworks/tools and use of the IRLM to clearly illustrate how the IS frameworks come together, and IS concepts highly applicable to their work. Three participants expressed that the case study was a useful tool to understand IS concepts. The case study provided participants with a common frame of reference when trying to complete group activities. Given that the use of a case study was a new experience, and key to completing the worksheets, one participant suggested we provide a more extensive description of the case study and how to extract the information needed to complete the worksheets.

When asked what should be changed to improve the training, multiple participants recommended more one-on-one time with the instructors to receive additional feedback on their particular assignments and help clarify areas of confusion. Two participants recommended more time allotted to training and assignments. One participant expressed wanting more repetition and practice of concepts, another would have liked more self-paced interactive modules to re-enforce lessons between assignments, and another would have liked more examples to prepare them for completing the worksheets.

When we asked participants what they would like from future training courses, suggestions ranged from: how to effectively adapt implementation strategies in the context of rapidly evolving public health crises (e.g. emerging infectious diseases); learn about how IS can speed up the implementation process for getting services to those most impacted by HIV; common barriers and facilitators; an example of a program using implementation science frameworks and how the program was sustained over time; and program evaluation.

Results from the post-evaluation survey of the in-person session that was unique to POISE participants (n=12) indicate that the in-person meeting helped build a sense of community among participants, facilitated opportunities for brainstorming, and generated insightful conversations about IS. Seven participants indicated that the small group breakouts to identify determinants, strategies, and implementation outcomes using the case study and worksheets was of great value, and some highlighted that the individualized TA provided by instructors and the facilitator during the breakouts were very helpful. All POISE participants were familiar with the challenges of introducing change into their environments and had difficulty visualizing the practicality of IS. It was not until we discussed implementation outcomes and the process of adjusting strategies based on organizational needs that they were able to appreciate the use of IS concepts and potential applications to their work.

Despite consensus on the relevance and usefulness of the training, the evaluation also revealed contrasting preferences regarding the format of the training. Three participants reported that they enjoyed the interactive nature of the training and learning about other participants’ projects. Conversely, two other participants indicated a preference for more self-paced work or individualized time with instructors and found that the group engagement was not as helpful. This feedback underscored the importance of accommodating diverse learning styles when working with a large cohort of practitioners.

Results from the evaluation surveys that are unique to the IPT participant mentioned that understanding social determinants of health, choosing strategies based on the implementation determinants, and connecting IS to their own CBO was among the most useful parts of the training. We believe that the participants mentioned these aspects of the training since it connected the lessons to their real-world setting and helped them apply the lessons to their organization. Both participants that finished the training indicated liking the “dialogue” they participated in during the didactic presentations, time for questions, and reflections/review of the previous session before starting the next didactic session. Due to our training being held virtually, we had limited opportunities to build deeper rapport and allow for brainstorming like the POISE program. The time we provided for “dialogue” and reflection, we believe helped facilitate community and brainstorming. All participants showed difficulty in initially linking implementation strategies and implementation outcomes to address determinants. It is possible that the difficulty is driven by the complex and inaccessible framing of IS concepts that has typically been used in IS, but which we worked to reframe in common language. At the end of the IPT, both participants asked for additional sessions, wanting to continue discussing IS and applying it to their workplace.

Future Directions

The post-session feedback from each training session for POISE indicated a desire for balance between foundational knowledge and practical application as well as a need for individualized support. Incorporating more real-world examples can clarify complex concepts. By providing more opportunities to integrate practical applications throughout the curriculum, participants can apply concepts to their own projects/organizations and receive personalized feedback and interaction with the instructors. Furthermore, an additional virtual learning lab will be added for future POISE cohorts. Ultimately, extending the training duration and optimizing the balance between in-person and virtual training can alleviate information overload and permit participants to fully absorb content and apply their learning effectively in their respective contexts.

While participants were initially encouraged to complete the full training program, only two of the four IPT participants were able to do so due to work commitments. Despite the small sample size, we found it important to reflect IPT participant’s experience and changes in understating related to IS. However, pre-post changes in IS understanding from IPT participants should be interpreted with caution as it precludes the ability to meaningfully interpret the differences. Moving forward, IPT will make training expectations more explicit by requesting a written statement of how the participants’ organization will support them to complete the training like POISE’s initial application. Based on participant feedback, future IPT sessions will be expanded to six 2-hour sessions to allow for additional time to review the previous sessions and worksheets, ask questions during didactic presentations, and complete in-session work (i.e., concept-related worksheets). IPT sessions will be held every 3 weeks to supply time to complete worksheets. Finally, because the HIV case-study proved to be helpful for establishing familiarity with the curriculum and key concepts, we will dedicate more time to engaging participants in its content and establish a thorough understanding of the case study at the start of the training.

IS for non-specialists can be difficult to grasp, as indicated by our participants, and more time is needed to solidify IS concepts. Future introduction of these concepts and frameworks will need to be simplified, with further time spent illustrating how they can be applied in real-world settings. We share our experience of these pilot training programs so that other implementation researchers and implementers may borrow from and/or adapt this curriculum for their own use. While researchers often focus on training other researchers, it is increasingly important that we train the partners with whom researchers work. Strong academic-community partnerships require that partners can equally engage in defining and selecting research questions, interventions to implement, and outcomes of interest, among other aspects of the research²³. Training implementers is critical to making this possible.

Future research and evaluation should also examine whether practitioner acceptability of training modality or curricular differences varies according to practitioner role (e.g., HIV tester, chief development officer, physician). This would enable trainers to identify any adaptations that may be needed according to the type of work in which the practioner is most engaged. Ideally, practitioners would be in mid- or higher-level positions where they have enough institutional backing to implement change. Extending evaluations in this area would also trainers to identify the appropriateness of training modalities and curricula according to practitioner role and whether extra attention is needed to recruit practitioners from clinics and practitioner roles that are not being reached. Finally, dissemination of training applications should be tested and evaluated to ensure that trainers are reaching the appropriate audience, to assess their reach, and to determine whether changes need to be made to reach further practitioners. Regarding POISE and the IPT, we believe we can increase participation by broadly disseminating the application and sharing testamonies of what current participants found beneficial, and creating new training modalities such as conference workshops that introduce IS, so that they may later choose to take one of our more extensive trainings.

Some limitations should be noted regarding the findings we describe in this commentary. First, the sample size from the IPT training from which we gleaned lessons is extremely small, making it difficult to identify trends. Second, these findings are only taken from the first year of training delivery for both POISE and the IPT. As such, there is still much to learn. We believe despite these limitations that the lessons we have identified will prove helpful for others seeking to offer IS training for implementers in real-world settings.

Conclusion

Increasingly, researchers in the IS field recognize that collaborations with implementers are necessary to conduct research that can be sustainably and equitably implemented after a study ends^24–26. While a majority of federally funded implementation studies include some form of collaboration with implementers, engagement with implementers as partners is often relegated to consultation and is more common in formative research than in implementation trials²⁷. Including implementers in the development or selection of strategies and the identification of outcomes of interest increases their level of engagement beyond that of mere consultation. This type of engagement requires implementers to have a deeper understanding of IS, its benefits, and how to effectively utilize IS to enhance their work. Ongoing development, refinement, and scale up of training for implementers is needed to make previously mentioned requirements a reality and further research is needed to identify long-term impacts of such trainings. In this commentary, we offer examples of two IS training programs that were piloted with HIV practitioners and intervention implementers. Each training course proved to be effective at increasing participants’ knowledge and ability to apply key IS concepts and lessons learned can help inform other training modalities that meet implementers needs.