Evaluating an Implementation Science Training Program: Impact on Professional Research and Practice

Abstract

Real-world application and implementation of evidence-based practice continue to be a challenge across multiple sectors, including behavioral health settings. Providing the opportunity for future researchers and practitioners to gain capacity and knowledge through structured experiential learning in implementation science is critical to closing the research to practice gap. The Institute for Translational Research Education in Adolescent Drug Abuse (ITRE) is a graduate certificate program that offers specific coursework, a large-scale service-learning project based in the community, and mentorship related to implementation science research and practice. The purpose of this evaluation was to examine, from the perspective of ITRE scholars, the perceived impact on the development of professional research and practice skills once graduated from the ITRE program. Fifty-eight semi-structured interviews across five cohorts were selected randomly for in-depth thematic analysis (n = 58). Results suggest that the ITRE provides a unique approach grounded in implementation science for building robust and transferable skills for future researchers and practitioners working in a variety of behavioral healthcare settings.

Introduction

The development and implementation of evidence-based practices (EBPs) over the last three decades has significantly increased in the field of behavioral health (Carvalho et al., 2013; Fixsen et al., 2009). However, a large gap still exists between research and practice, and this may be associated with a lack of attention to and comprehension of the implementation process of EBPs (Proctor et al., 2009). Therefore, it is important for researchers and practitioners in behavioral healthcare to possess competent knowledge and skills rooted in implementation science and practice (e.g., knowledge translation) as the means for closing the research to practice gap (Baldwin et al., 2017). The current study aims to examine the impact of a structured, integrated model designed to enhance professional development among interdisciplinary graduate students and practitioners in implementation science research and practice.

Implementation Science Research and Practice: Professional Development

In the last decade, multiple theories, frameworks, models, and strategies in the field of implementation science have been developed to combat issues associated with implementation and outcomes (Nilsen, 2015; Vroom & Massey, In Press). However, the intricacies of the implementation process, practice, utilizing complex theory and frameworks, and conducting implementation science research in community settings have proven to be difficult and require a specific set of skills and knowledge (Schultes et al., 2021). Given these realizations, the field has acknowledged the critical need to train healthcare researchers and practitioners in a comprehensive set of skills in implementation science research and practice to improve the implementation and outcomes of evidence-based health services (Ramaswamy et al., 2019).

Implementation science research is the study of methods, approaches, and strategies that assist with the adoption, use, and sustainability of EBPs in routine practice (Bauer et al., 2015; Kendall & Beidas, 2014). Implementation practice can be conceptualized as knowledgeable individuals utilizing implementation theory, frameworks, and activities, informed by research, to assist with the integration, implementation, and maintenance of evidence-based programs or approaches (Vroom & Massey, In Press). In recent years, implementation science-oriented training, professional development opportunities, and graduate-level curriculum (i.e., coursework and/or graduate certificate programs) have been developed for future and current researchers and practitioners (Ramaswamy et al., 2019; Schultes et al., 2021). However, many of these professional development opportunities are targeting individuals who are interested in pursuing or are currently employed in research, government, and/or academic positions and focus less on persons who are looking to apply this knowledge in practice (Ramaswamy et al., 2019).

Ramaswamy et al. (2019) noted that to translate research into practice, it is important to provide training in implementation science research and practice as well as recognize the important relationship between researchers, practitioners, and the community. Competency-based training in implementation science has become important to advance the field in terms of both research and practice. Emerging out of medical education field, competency-based education is “an approach to preparing physicians for practice that is fundamentally oriented to graduate outcome abilities and organized around competencies derived from an analysis of societal and patient needs. It de-emphasizes time-based training and promises greater accountability, flexibility, and learner-centredness” (Frank et al., 2010, p. 636). This particular learning model emphasizes the need for trainees to demonstrate proficiency of knowledge and skills acquired in a specific subject matter (Brightwell & Grant, 2013).

Competency-based training is viewed within the field of implementation science as a mechanism that can facilitate the establishment of more unified and rigorous standards against which professional growth and development can be measured (Leung, 2002; Thacker & Brownson, 2008). Competencies assist in providing structure and standardization, which may be beneficial for evaluating trainee growth (Padek et al., 2015) as well as potentially providing the opportunity to acquire expertise and direct experience that is appealing to current and future employers. The opportunity to set clear and concise expectations for implementation science researchers and practitioners is seen as particularly promising given the relative infancy and interdisciplinary nature of the implementation science field (Padek et al., 2015; Schultes et al., 2021).

To capitalize and translate skills and knowledge obtained during structured, professional development opportunities, it may be beneficial to offer a guided space to apply these newly acquired competencies in real time within community-based settings. Service learning is one such approach used to prepare the workforce through experiential learning that requires trainees to engage in community-based projects where they utilize and master skills learned in the classroom while simultaneously meeting the needs of the community (Sabo et al., 2015). Leveraging service learning as a guiding framework for competency-based training may assist with creating an intentional structure to support optimal learning within an immersive experience (McKinnon et al., 2017).

Service learning is meant to provide participants with an opportunity to make an impact on the community and be impacted by the experience while learning leadership skills (Fredericks, 2003). Engaging in a project within the community may provide opportunities for participants to become more involved citizens of their community, learn how to collaborate with others, and gain greater cultural awareness of others’ beliefs and values (Fredericks, 2003; McKay-Jackson, 2014). Cited frequently within public health, service learning is considered an ideal vehicle for addressing social justice issues and health disparities because it requires extensive collaboration between academic institutions, students, and the community (Coombs et al., 2019). Service learning can be used as a mechanism to facilitate community–university partnerships and can assist with integrating curriculum and learning into a reciprocally advantageous relationship between academia and the community (Williamson et al., 2016). In addition, mentorship during the service-learning experience also may be critical as research has shown mentoring to have a significant impact on career trajectory and longevity, enhancement of research and practice skills and knowledge, and collaboration and communication abilities (Abedin et al., 2012; Young et al., 2015).

Current Study

Current literature shows a lack of evidence on effective strategies for measuring the outcomes of programs linking education and applied practice (Moore et al., 2018; Sabo et al., 2015). For example, Baldwin et al. (2017) suggest evaluations of mentoring programs may be limited in their indicators for measuring programmatic success as many appear to emphasize academic productivity as the primary measure of success. There also seems to be a lack of research addressing best practices to evaluate the mentorship experience and the complexities of the university–community partnerships within service-learning opportunities (Williamson et al., 2016). In addition, there is limited research surrounding stakeholder feedback related to capacity-building initiatives in implementation science (Moore et al., 2018; Tabak et al., 2017). The current study aims to examine the unique perspectives of scholars who participated in the Institute for Translational Research Education in Adolescent Drug Abuse (ITRE) graduate certificate program on specific skills and knowledge gained that they can and/or are utilizing in applied research and practice settings in behavioral healthcare. The evaluation questions of the study were as follows: (1) How does service learning influence scholars’ professional development in implementation science research and practice?; and (2) From the participants’ perspective, what most effectively facilitates professional development in implementation science research and practice?

Methods

Institute for Translational Research Education in Adolescent Drug Abuse (ITRE)

The Institute for Translational Research Education in Adolescent Drug Abuse (ITRE) at the University of South Florida (USF) and Northern Arizona University (NAU) is a federally funded graduate certificate program that aims to develop innovative, applied research skills among its graduate certificate participants (referred to as “scholars” in the ITRE context). The ITRE is housed within the Department of Child and Family Studies at USF and within the Center for Health Equity Research at NAU. Coursework is completed and graduate certificates are awarded through USF. Scholars include graduate students from various disciplines as well as current researchers and professionals (i.e., non-degree seeking individuals) working in community settings. Participants include scholars from multiple health and social science disciplines such as anthropology, behavioral health, clinical psychology, school psychology, criminal justice, epidemiology, nursing, public health, occupational therapy, rehabilitation and mental health, and social work. The ITRE primarily admits doctoral and master’s students, and on occasion, exceptional undergraduates. In addition, the ITRE also welcomes non-degree seeking researchers or professionals who have already earned a bachelor’s, master’s, and/or doctoral degree and are currently working within the behavioral health field.

To the author’s knowledge, the ITRE is the only targeted graduate education program with an integrated, team-based service-learning component coupled with structured coursework grounded in implementation science and community-based participatory research and evaluation (CBPRE) (Davis & D’Lima, 2020). The program is completed over four consecutive semesters and requires scholars to complete 15 credits hours: three online courses (three credits per course) and three service-learning courses (two credits per course). The three online courses provide scholars with a foundation in implementation science theory and research, translational research methodologies, CBPRE, and adolescent substance use. The service-learning courses, completed over three consecutive semesters, involve collaboration and project development with their community partner, Institutional Review Board submission, data collection and analysis, dissemination of project results to their community partner, and a presentation at a national conference. The service-learning courses also provide in-person seminars and workshops related to manuscript and conference presentation development, research methods, and resource sharing among scholar teams. Service-learning teams typically consist of two to three scholars per team depending on the size of the cohort.

In addition, a variety of mentors including academic, peer, community members, and national experts are made available to scholars to provide assistance in conducting their service-learning projects. Each scholar team is assigned an academic mentor, a peer mentor, and a community partner contact at the start of their projects to assist in the management and supervision of the service-learning projects. National experts provide the ITRE with expertise in all ITRE-related coursework and research initiatives. In this multi-modal structured approach, the ITRE aims to improve scholars’ competencies in translational and implementation science research and practice in community settings. A more thorough overview of the ITRE model and its components can be found in prior publications (c.f., Baldwin et al., 2017; Burton et al., 2016).

Setting and Procedures

The ITRE has trained over 90 students in Florida and Arizona using a tailored model of instruction since its inception in 2013. This qualitative study emerged from a larger ongoing evaluation being conducted of the ITRE and will focus solely on the first five cohorts of students (2013–2019). The ITRE was primarily located at USF until NAU was added as a site to the training grant’s renewal starting in 2016 with cohort 4. Qualitative interviews were used to elicit scholar perceptions of the ITRE experience, including coursework, career outlook, and service-learning experiences. Qualitative data were also used to assess how former scholars have applied the skills learned in their current pursuits (e.g., academic and/or employment).

Participants

Participants consisted of scholars from cohorts 1–5 (2013–2019), from both USF and NAU sites, which had completed the program (i.e., exit interviews), and those who were 2-year post-graduation (i.e., alumni follow-up interviews). More specifically, the current study included USF scholar exit and alumni interviews from cohorts 1–5 and NAU scholar exit and alumni interviews from cohorts 4 and 5. The objective of the exit interviews conducted with all scholars at the end of the program was to obtain their feedback, explore their overall experiences in their service-learning projects and ITRE, and discuss their future career/education goals. The alumni interviews, which took place 1- and 2-year post-graduation, looked to explore former scholars’ current educational/vocational pursuits and how the ITRE influenced these pursuits. For the purpose of this study, only the 2-year post-graduation alumni interviews were used for analysis to examine long-term impact.

A purposive sampling technique (Bernard & Ryan, 2010) was used to recruit scholars for interviews. Inclusion criteria included scholars who have fully completed their service-learning projects and have been awarded their graduate certificate from USF. Participant recruitment involved directly emailing the scholars to request their participation. Over 100 exit and alumni interviews have been conducted since 2013. Cohorts 1–5 varied in size from 10 to 15 scholars per cohort. Thirty-four exit interviews and 24 alumni interviews (2-year post-graduation) were randomly selected using a random numbers table for a total sample size of 58 interviews (N = 58). The number of interviews per subgroup was selected based on prior research suggesting the identification of unique issues/themes (theme saturation) asymptotes between 8 and 12 interviews on a particular domain/phenomenon within a group of knowledgeable informants (Fusch & Ness, 2015; Guest et al., 2006). Interview transcripts were stripped of their identifying cohort numbers, refiled, and then randomly selected to avoid the potential bias of a cohort effect. This study received approval from the USF’s Institutional Review Board. Participation was voluntary and all participants were sent an informed consent form and were asked to verbally consent to participate prior to the interviews taking place. No incentives were given to participants for their participation.

Data Collection

Qualitative interviews took place over the phone, were audio recorded with participant consent, and were conducted by trained research assistants (RAs). Semi-structured interview guides were used for both exit and alumni interviews to elicit scholar perceptions of the ITRE and the service-learning experience (see Online Appendix 1 for sample interview questions). The same exit and alumni interview guides were used for all scholars in cohorts one through five to increase the reliability of the data (Guest et al., 2012). Interview guides were developed by the Principal Investigator (PI) for the purposes of the grant’s evaluation. On average, exit interviews lasted approximately 35–40 min and alumni interviews lasted approximately 20–25 min.

Data Analysis

Interviews were transcribed by an independent transcription service and were reviewed for accuracy against the recordings by two trained RAs. Applied thematic analysis (Guest et al., 2012) was used to create a preliminary codebook using an inductive approach to identify a priori and emergent codes. The preliminary codebook included a brief definition of the code, a full definition, and examples. The codes were created based on a thorough review of several interview transcripts selected for inclusion in this study, codes that correspond with specific questions asked within the interview guides, and the study’s research questions. Interviews were reviewed by two trained RAs and the PI (i.e., the research team). Codes were then created by the entire research team. The first transcript of each subgroup (i.e., exit and alumni) was coded independently by the two RAs using the preliminary codebook. The research team met to discuss inconsistencies and assess inter-coder agreement using subjective agreement (Guest et al., 2012). The codebook was amended accordingly, and the remaining transcripts were coded by the two RAs.

Inter-coder reliability was assessed using percent agreement (Guest et al., 2012) by randomly selecting four transcripts (i.e., two transcripts per subgroup). The percent agreement between the two coders was 72%, which is considered an adequate level of agreement (Bernard & Ryan, 2010; Stemler, 2004). Once coding was completed, the two RAs compiled all codes into matrices representing a priori and emergent themes. The three members of the research team independently reviewed all codes from the 58 interviews. The research team then met frequently to compare and contrast independent analyses until unanimity was reached to finalize overarching themes. Data were organized and analyzed in ATLAS.ti v.8.1, a qualitative data software program (Atlas.ti. Berlin: ATLAS.ti Scientific Software Development GmbH, 2020).

Results

Four overarching themes, as well as six sub-themes, emerged from the data that speak to the utility of the service-learning model and how it enhanced professional development in implementation science research and practice among participating scholars. Overarching themes included: (1) implementation science application; (2) research skills (i.e., hard and soft skills); (3) vocational impact; and (4) mentorship. Few differences were found between alumni and exit interviewees. Therefore, results from both alumni and exit interviews will be presented concurrently unless otherwise specified.

Implementation Science Application

The emergent theme of implementation science application can be defined as scholars utilizing knowledge and skills gained from the ITRE related to implementation science and translational research in their current pursuits (i.e., education and/or employment). Many scholars discussed an increased awareness obtained from the service-learning experience and coursework relating to the importance of bridging the gap between research and practice. Scholars expressed interest in incorporating implementation science theory, frameworks, and models into their current and future work to increase their capacity to affect change in community-based settings.

Exit Scholar 57: “I want to do research that actually can help bridge the gap between practice and policy… and translational research is all about that… I would never have known what translational research was before this Institute and so I now have the passion for that too, and really, when I do research that can have an impact on policy and practice and specifically state what those implications for behavioral health would be in my research, so it was clear and just do that kind of thing as a part of my career.”

Many scholars echoed the above statement and went on to describe how the ITRE impacted their future approach to conducting research and/or their clinical practice, including their theses/dissertations and “using an evidence-based approach in everything they do” in clinical practice.

Scholars also were able to conceptualize different areas of implementation science research and practice that are relevant for community-based organizations including evidence-based approaches, evaluation procedures, collaborations with stakeholders, fidelity, training, and buy-in.

Alumni Scholar 14: “…it also gave me insight into the importance of implementation and evaluation. We want to go ahead and create these fantastic programs but are they being implemented with fidelity? Are we evaluating the work that we’re doing? How do we know it’s going well? Are we wasting money? So, all of those things came out of the Institute.”

Scholars discussed how the ITRE experience allowed them to view research through an entirely new lens. Instead of experiencing research in a traditional format (e.g., lab, randomized controlled trial, etc.), where the researcher tends to be disconnected from the population of interest, the ITRE experience allowed the scholars to interact not only with the community partners but also incorporate them into the decision-making/research process. The scholars commented positively on the service-learning projects incorporating CBPRE methods as well as developing their ability to see and communicate translational issues in their daily work. One scholar discussed how they gained skills in communicating with diverse audiences, including policy, academic, and community levels as a result of the ITRE and their service-learning experience:

Alumni Scholar 20: “ I have a huge ability to understand local practices, and what people are doing here, and through my work, I understand local policies, and how those two things don’t work well together a lot of the time, but through the Institute, I was able to learn how to better articulate, how all of these things intersect, and I have the ability to discuss that with different audiences, so it’s almost like speaking several different languages. I speak a science researcher’s language, probably the poorest, to be honest, and then I speak a policy language and practitioner language. So, I think the Institute helps me bridge those three.”

Given the program’s interdisciplinary nature, scholars discussed how implementation science could be applied to multiple health and social service disciplines including occupational therapy, nursing, public health, social work, rehabilitation and mental health counseling, anthropology, and education.

Alumni Scholar 4: “…what I do on a daily basis relates to the Institute because I’m running an educational series that’s focusing on health – healthcare workers so doctors, nurses, health educators, veterinarians, anyone that really needs continuing education. We’re basically trying to decrease the gap in the implementation of programs that work and increasing people’s awareness of these programs and informing them about the situation, and then also giving them things that they can do.”

Research Skills

Scholars discussed multiple skills and professional development opportunities acquired from the ITRE in relation to conducting implementation science research and practice as well as translational research. To assist with organization, the research team created the sub-themes of hard and soft skills related to research skills. Hard skills can be defined as technical skills related to research methods, writing manuscripts, and publication. Soft skills can be defined as adaptive skills related to collaboration and communication within scholar teams and between teams and community partners as well as dissemination of study results to community partners and conference presentations.

Hard Skills

Scholars discussed the opportunity to learn and utilize qualitative and quantitative methodologies firsthand. The ITRE requires scholars to work directly with their community partner(s) to develop a project and study design that is applicable for that specific community agency, develop measures and data collection procedures (i.e., surveys and/or interview protocols), and analyze the data. Scholars reported the coursework, the “hands-on, real-world” experience, and the guidance provided by mentors during this process were an integral part of their success.

Exit Scholar 25: “This method, the research and stuff, that really strengthened because I felt like now that I know more about it and I was able to actually work hands-on from the very beginning - because we have to figure out, ‘Okay, what type of research are we going to do? How we’re going to get our results? How we’re going to analyze the data?’ I am able to apply that with the terms I have now. I’m able to do this. I’m, “Oh hey. I have knowledge in that.”

Alumni Scholar 12: “I realized at that time, doing the research, doing the IRB, conducting research and the skillset… I was lacking a lot of skills. That’s why it was a great starting point, but I realized that I needed to take it a step further, to really hone in on those skills.”

Scholars also discussed the importance of having the opportunity to write a manuscript and to be published in a peer-reviewed journal. The timeframe scholars are allotted to conduct a full-scale research study and disseminate their findings (i.e., four consecutive semesters), while simultaneously writing a manuscript, was reported by scholars to be challenging. Despite these challenges, scholars expressed how they still gained valuable insights in writing manuscripts including authorship and the writing and editing process. Many mentioned that although they did not publish their ITRE projects, the skills gained in the program assisted with other manuscripts pursued post-graduation.

Exit Scholar 50: “Well, definitely, they’ve made me a better writer. We’re used to reading articles and reading how they were written and the formatting. I feel like my writing has gotten so much stronger and my understanding of how to describe methods and results. I think that’s really helpful.”

Soft Skills

The field experience offered by the service-learning component facilitated the development and/or refinement of various non-technical or “soft” skills such as adaptability, communication and collaboration, as well as creative problem solving. Scholars were asked to comment on their experiences with conducting their service-learning projects with their community partners. Many scholars discussed navigating the establishment of communication with their community partners as essential for project success, especially with the recruitment of study participants and data collection. Scholars discussed that they were able to observe the inner workings of real-world organizations serving communities. In addition, scholars noted organizational realities such as leadership turnover and how ever-changing internal policy can impact the buy-in and motivation to participate and assist with research being conducted by an external entity.

Exit Scholar 46: “I think there was also a component of like maybe initially there wasn’t really buy-in from them [community partner]. I think that they felt like, ‘Oh, we did that in the past, I guess we got to do it again,’ rather than them really being invested in it and wanting it.”

Scholars working with vulnerable populations such as foster care youth, Native American youth and families, homeless youth, and Hispanic farmworkers and their families noted that building trust and rapport with these communities was critical to the implementation of their projects. Some scholars attended community events and trainings, religious services, and frequently visited the agencies to become a familiar presence to youth and their families.

Exit Scholar 51: “Us going every single week, we were a familiar face to the kids, the kids knew us. We were familiar faces to the parents. The parents would see us when they would pick up their kids. I think that was vital.”

In addition, many scholars mentioned the importance of having an advocate at the community agency and they learned firsthand that having the buy-in of the community partner and their agency is critical to project development and implementation. Scholars reported that not all community partners and agencies are well-versed in research (e.g., methodology) and its importance. Scholars discussed learning how to balance this concern to facilitate a successful project.

Exit Scholar 42: “I, personally, keep seeing actions, how priority is – for each team could be different than – for an organization. I also got an opportunity to startup a dialogue so we could reach a negotiating point so that we would get to a topic that was relevant for both parties concerned, meaning the research team and the community partners. So I think that negotiating skill and making sure that it works out for both parties was a good thing to learn…”

Alumni Scholar 14: “It gave me more insight on the necessity for community-based participatory research. It also gave me more insight on how difficult it is to engage in that research because those schedules, and timing and misalignment of goals.”

The scholars discussed having to determine ways to communicate and collaborate efficiently within their team. Due to the interdisciplinary nature of the program and inclusion of both doctoral, masters, and on occasion, undergraduate students, different strengths of team members were perceived as both a positive and challenging aspect of working together. Scholars mentioned experiencing an imbalanced or disproportionate workload or having to serve in a mentorship or leadership capacity on their team if they were more experienced or farther along in their educational career (i.e., doctoral student). However, they also discussed the ability to learn new skills from their teammates, how to collaborate effectively, and actively listen.

Exit Scholar 46: “Yes, in that I had a lot more experience than she did in the field but in conducting research as well so in our methodologies too. So, it felt like rather than a mentor or the supervisor or somebody else sort of giving her that guidance, I was giving it, which was sort of inevitable when you’re working closely with somebody and trying to get a project done.”

Exit Scholar 11: “I really enjoyed the team I was a part of. I thought that we did a fairly good job of problem solving, communicating with each other and working through whatever issues presented themselves to actually get to the end goal.”

Scholars discussed the importance of disseminating the results of their project with community partners and agencies. Many noted that being able to see change within the agency as a direct result of their project, even if considered minimal, was very rewarding and helped solidify their career path of conducting research or working in community settings.

Exit 1 Scholar: “When we gave our results to the community agency, they were so happy. They were like, ‘...We didn’t know any of this. We’re going to totally redesign our training manual.’ That’s what I’m talking about because I have felt so isolated in some of the other works that I’ve done and I’ve felt very pointless. This was helpful to them.... that’s the kind of change I want to have because that’s how you change the world. It sounds corny but it’s true. I don’t want to sit here and think of a research I do that has no relevance to the community anymore. So those are the directions that it’s taking me.”

The ITRE provides its scholars the opportunity to present their results at the USF-sponsored Annual Research & Policy Conference on Child, Adolescent, and Young Adult Behavioral Health in Tampa, FL. Scholars are additionally provided a structured practice environment to refine presentations before presenting. Scholars discussed how this was an invaluable experience and gave them the confidence to submit abstracts and present at other health-related conferences (e.g., American Public Health Association, American Psychological Association, and American Evaluation Association), held the scholars accountable for finishing their projects and meeting their deadlines, and was a critical line added to their curriculum vitae.

Alumni Scholar 17: “Aside from my two defenses for my thesis work, I haven’t done any presentations. I have been starting to look at different conferences where I could submit abstracts to. I would say that had it not been for the Institute, I wouldn’t have the confidence to go and present at a conference on my own.”

Vocational Impact

Vocational impact can be defined as the perceived influence of the ITRE training program on scholar educational or career pursuits and successes. Vocational impact can be broken down into four sub-themes including: (1) coursework; (2) research interests; (3) career outlook; and (4) qualifications and opportunities.

Coursework

Scholars discussed their appreciation for gaining exposure to concepts offered in the coursework. This was viewed as essential for learning basic theories, frameworks, and concepts underpinning the mechanics of translational research and implementation science research that are conducted in community settings. Scholars reflected on the complementary nature of the coursework to their service-learning projects. Many cited the curriculum was helpful in learning techniques to use in the community and provided structured deadlines to encourage accountability. The sequenced nature of the assignments to support each phase of their project from conceptualization to manuscript preparation and submission was noted as being essential for project success.

Exit Scholar 4: “The way the classes were set up - what we were doing in the actual classes…was really translated into what we needed to do for our project with our agency. I really thought that how it was sequenced was very well thought out and well planned because it seems the course of the classes seem to really coordinate with what we were expected to be doing with the service-learning component.”

Research Interests

Scholars reported how the experience of conducting applied research in a community setting further helped them to solidify their research interests as well as narrow the populations of focus that they would like to work within their careers. Some scholars also reported developing unanticipated or emergent interests in adolescent behavioral health and implementation science and translational research.

Alumni Scholar 7: “It helped to open my eyes to different research techniques and methods that I wasn’t familiar with before. Implementation trials and translational research was a new area for me, I didn’t know a whole lot about it before joining the Institute and that’s actually where my dissertation research is headed, is actually looking at the implementation hub and innovative model of healthcare. It actually helped to shape my dissertation work which has been fantastic.”

Career Outlook

Scholars discussed how the exposure to the field of adolescent behavioral health and how community-based organizations operate served to increase scholar interest in research, evaluation, and community-based work. Their confidence to conduct research and present their findings improved and subsequently helped provide scholars with direction for future educational and career pursuits. Scholars also mentioned the notion of wanting their current and future research and evaluation efforts to benefit community stakeholders and how they would like to contribute to their discipline.

Exit Scholar 46: “I think that it’s actually more significantly influenced the directionality of my career path than I had originally anticipated. I think that the exposure, again, to the clients in detox and sort of the in-depth exploration of what was happening and being able to connect that to the theory through this manuscript process and just like the culminating results of the project really helped me to identify more clearly where I want to contribute to the field.”

Qualifications and Opportunities

The collective structure of the ITRE, that is—both the coursework and service-learning component, were noted by many scholars as giving them a competitive edge for pursuing career goals and other educational opportunities not given elsewhere in their degree programs. Specifically, exposure to concepts in implementation science research and practice was viewed as unique and particularly advantageous in distinguishing themselves from others while seeking employment and/or educational pursuits.

Exit Scholar 32: “The skills that you learned in the Institute really make you more qualified for the types of jobs that we would like to have coming out of a master’s program. That work experience that they want you to have, the Institute gives you that work experience within an academic environment. There’s no substitute for that and you can list that as actual work experience and it’s just -it makes you more qualified for the things that we would like to do.”

Not all scholars participating in the ITRE expressed their intent to pursue a career in research. These particular scholars continued to view their gained research experience as highly valuable for showing prospective employers that they are well-rounded, therefore, setting them apart for job opportunities. The experience of working with different and vulnerable populations, as well as skills gained overall, supported scholars in taking steps to advance their careers.

Alumni Scholar 20: “My degrees are all really general. I have my MPH, which is in health education and then the DrPH is in leadership, so there’s really nothing that has behavioral health on my education resume, except for the Institute. That actually is a really big bonus. I can reference the classes that I’ve taken that were very specific, and specific to my future employers and community partners, and how that is relevant to legislators.”

Alumni Scholar 6: “It’s definitely giving me more insight into how we approach things from a service perspective. If I had not done the research side of it, I would just be an intervention provider with very little insight into why the intervention they’re provided and how they are provided and how that was happening but because of being a part of the Institute, I have more understanding for how it came about…”

Scholars shared how networking through the ITRE led to several educational and career opportunities including field placements, working with community partners in their master’s thesis and dissertation research, continuing their research with community partners post-ITRE, obtaining recommendations when applying for continued educational pursuits, and employment.

Alumni 13: “On the other side, professionally, it’s been really great to come out of a clinical social work program with research experience. I think that sets me apart from other candidates when applying for jobs or just kind of deciding how will my career moving forward. Having both of those types of trainings helped me…”

Alumni 19: “I’m very, very grateful for the relationships that I developed as a result of my participation in the Institute because it definitely helped me in terms of completing my doctorate much quicker because I had those connections and that support.”

Mentorship

Scholars discussed the importance of the multi-tiered mentorship approach that provided varying levels of support throughout the lifespan of their project. From receiving targeted support to informal guidance, mentorship appeared to be a critical component influencing the level of success of their service-learning projects, and scholars looked to their mentors as role models. Scholars reported receiving guidance in developing what can be considered the hard skills in research such as, how to formulate research questions, develop data collection protocols, complete the IRB process, run data analyses, and disseminate their results through presentations and manuscript submissions. Scholars also felt they learned soft, adaptive skills from their mentors, such as understanding how to best collect data from different populations of focus including those considered to be vulnerable or underserved (e.g., foster care or Native American youth). Mentorship allowed for the activation of different CBPRE methods such as assistance in navigating the inclusion of the community partner in activities such as the formulation of the project, feedback and approval of data collection measures (i.e., surveys or interview guides), continuous updates on data collection and analysis, and incorporating community partners into disseminating the results of the projects (i.e., national presentations and manuscripts).

Exit Scholar 32: “The amount of time and investment that she [mentor] puts into each student is incredible. [They are] an incredible mentor. [They are] very, very good at being able to assess what your strengths are and play to those and help you work on some of the things that you need to get stronger at. [They are] really good at facilitating just tense situations that sometimes arise between – in research projects with community partners, especially the number of community partners we’re working with. [They], in particular, [have] been so invaluable for our project because of who [they are] in the community…”

In addition, some ITRE mentors have extensive histories in conducting research in community settings and have established working relationships with key community contacts. Scholars mentioned how this increased their capacity to network and engage with the community themselves as well as shape their projects. Scholars also noted how their mentors were able to provide oversight of the scholar team and ensure smooth team dynamics as well as help them learn how to problem solve and troubleshoot issues that arose with the project and community partners. A few scholars also shared how their mentors influenced their career outlook and educational pursuits by advising the scholars in their professional development goals and providing additional opportunities to be a part of research and/or community-based work outside of the ITRE.

Discussion

The purpose of this study was to examine the impact of an integrated, sequenced training model focused on professional development in implementation science research and practice through a collection of activities, including coursework, service learning, and mentorship. Currently, there are gaps in the literature related to the evaluation of research and practice-based training programs including stakeholder feedback and measurement of capacity building in implementation science (Baldwin et al., 2017; Moore et al., 2018). This study aimed to add to the literature surrounding stakeholder feedback on programmatic outcomes in a competency-based implementation science graduate certificate program.

Results of the current study suggest the ITRE’s structured design provided scholars the guided space to solidify implementation science knowledge and skills (i.e., competencies) gained in the classroom through real-time application in their service-learning projects. Previous research focused on implementation science training competencies has noted the importance and helpfulness of training programs in developing competencies related to research methodology and implementation science knowledge and skills among trainees (Schultes et al., 2021). The organization of the current training effort, that integrates coursework and service-learning experience, allows for the development of ‘competencies in action,’ and may alleviate the need to depend on implementation science competencies that must be learned through on-the-job or self-guided experiences (Schultes et al., 2021). For critical competencies such as the utilization of implementation science theory and frameworks, there is a need for a comprehensive set of skills where students not only understand the theory and research, but also build the skills necessary to collaborate and develop partnerships in behavioral healthcare (Schultes et al., 2021).

Graduate students are rarely provided with opportunities to realistically apply implementation science theory and frameworks in applied research settings. The results of this evaluation suggest that applying a service-learning approach in conjunction with a didactic component of implementation science training enhances the development and sustainability of general research skills, implementation science research and practice knowledge, and capacity to engage effectively in community-based work. Through practical application to and in the community, and by understanding local issues, implementation becomes less about doing to, and more about doing with communities, which is ultimately the core of successful implementation science in practice.

Research literature has discussed the importance of making research relevant for not only trainees, but the communities in which the research is conducted (Tabak et al., 2017). Scholars noted the ITRE offered opportunities to create unique partnerships that allowed the community partners to engage in and contribute to the project from its inception and to conduct meaningful research and evaluation projects that led to real change within the community agencies. The service-learning experience for these scholars enabled them to better bridge the gap between research and practice and established opportunities for negotiation and the use of CBPRE methodologies. Past literature has stated the importance of developing competencies that can assist with increased and targeted collaboration, which may lead to more relevant research/evaluation results for the end-user (i.e., community stakeholder) (Tabak et al., 2017). This also corresponds with research stating that students may be more interested in training material that targets “practical problems” as opposed to general education competencies such as literature reviews (Ullrich et al., 2017).

Criticisms of competency-based training have claimed that fulfilling a professional role, such as a behavioral health researcher or practitioner, is more than the sum of the individual tasks that are completed in training. That is, competency-based training alone is considered limited in its capacity to account for “higher cognitive skills” and lacks attention to learning that occurs within collaborative settings (Brightwell & Grant, 2013). However, integrating service learning into the training process, as seen in the ITRE, may allow for the addition of structure while also permitting the development and practice of those intangible, non-technical “soft” skills that are better taught and learned outside the classroom. For example, Metz et al. (2021) emphasize the importance of researchers and implementation support practitioners interacting with service providers, such as in communicating study findings without overly technical jargon that is common in academic settings. CBPRE methodologies also lend themselves well to the development of soft skills given the emphasis on co-learning, capacity-building, as well as conducting research for the mutual benefit of all partners (Israel et al., 2018).

Mentorship was identified as a critical aspect of the ITRE program that fostered significant development among scholars in research/evaluation methods, professionalism, and most importantly, the ability to engage stakeholders and build relationships within the community agencies. Having active, invested mentorship available over time may assist in bolstering self-confidence among scholars to engage community partners and create sustainable buy-in that is essential to project success. Research has shown that strong connections created between mentor and trainee can positively influence future or long-term collaborations, linkage to employment or grant opportunities, and publication (Brownson et al., 2017; Luke et al., 2016). This may be in contrast to other training opportunities, which struggle to build competencies in stakeholder engagement and relationship building and may be the result of limited mentorship and/or utilizing a shortened workshop technique to deliver materials (Moore et al., 2018). Implementation science in practice requires that researchers, EBP purveyors, and implementation facilitators have insights regarding the lived community experience as well as soft and hard research skills to enable the effective initiation, integration, and sustainment of EBPs. A model training approach that assists with building these capacities may be particularly valuable in furthering the successful implementation of new programs in community behavioral health settings.

The field of implementation science comprises perspectives from multiple disciplines (Schultes et al., 2021). The results of the current study give evidence to the importance of interdisciplinary collaboration, with scholars expressing the benefits of being able to conduct research with mentors, scholars, and community members coming from different disciplines and cultures other than their own. Interdisciplinary collaboration may allow for further growth and opportunity to expand the scope of the field of implementation science research and practice beyond a singular discipline and encourage the breakdown of silos often seen within healthcare. In turn, this may serve to benefit communities with diverse needs.

Long-term maintenance and application of knowledge and skills gained through training initiatives is a key objective for any discipline. However, within the scope of implementation science research and practice, little attention has been given to assessing the long-term sustainability of knowledge and skills gained through such initiatives (Park et al., 2018). Findings from this study suggest that the unique blended model of team-based service-learning and structured coursework grounded in implementation science and CBPRE is an effective model for the delivery of implementation science training. Scholars continue to apply knowledge and skills gained two years after completion of the program, with many reporting how the ITRE shifted how they approach their work or education, be it from a research or service perspective. In addition, alumni scholars reported greater attention to detail in their current pursuits with how behavioral health services are implemented and evaluated in community settings. This aligns with the emerging translation research agenda called for by implementation science experts to address scale-up and sustainability efforts of evidence-based services (Proctor et al., 2015).

The study’s results also suggest some features of the program that may be critical for its effective operation and sustainability. One such feature is the presence of well-qualified and committed mentors who are familiar with implementation science and who can work well in community settings. A second feature is undoubtedly engaged community partners who are willing to work with graduate students in codeveloping implementation research projects. Finally, as implementation science is a rapidly developing field, curriculum developers must be able to adjust the curricula as the field matures and integrate training with the practical issues arising from the community.

Strengths and Limitations

The strengths of this study should be noted. The study was able to analyze data from five cohorts of students as well as students who were 2-year post-graduate, which allowed for a 2-year follow-up evaluation for each cohort to assess the sustainability of implementation science research and practice skills gained. In addition, the number of participants per subgroup was selected based on the research literature stating 12 or more interviews are recommended to reach sufficient data and theme saturation (≈ 92%) on a particular domain/phenomenon within a group of knowledgeable informants. The large sample size allowed for greater saturation of themes, breadth of data, and in-depth understanding (Bernard & Ryan, 2010; Guest et al., 2006).

Limitations also should be noted. The ITRE is a unique training opportunity. Therefore, the evaluation was limited to one specific professional development opportunity and may not be generalizable to other training initiatives or graduate programs. Differences between USF and NAU sites also were not assessed. Future evaluation efforts would benefit from examination of different study sites to provide a better understanding of site influence on professional development. In addition, more USF scholars were represented in this sample than NAU scholars. NAU did not become a site with the ITRE until 2016 beginning with cohort 4. Future evaluations will include more NAU scholars as data become available. This study only included interviews from scholars who completed the graduate certificate program and does not include the perceptions of individuals that did not complete the program fully. Future research may benefit from incorporating the perspectives of non-completers to gain better understanding of why they did not complete the program and if receiving only some of the program still resulted in professional development gains. In addition, due to the way the transcripts were deidentified, it was not possible to examine cohort effects. The study aimed to examine the data as a whole; however, it may be beneficial to evaluate cohort effects in the future. Lastly, this study only included post-program and follow-up interviews. Future evaluation efforts should also include pre/post data to acquire a baseline estimate of knowledge and skills gained from the program over time.

Implications for Behavioral Health

Evaluating the impact of service-learning programs, such as the ITRE, which are specific to implementation science can assist with building an evidence base focused on best practice approaches for developing the professional capacity of researchers and practitioners across multiple disciplines and settings. As varying types and levels of implementation science training programs continue to emerge, understanding the components and mechanisms across programs can help inform optimal education and training methods. Results of this evaluation, along with previous evaluations of the ITRE (Baldwin et al., 2017; Young et al., 2015), suggest other indicators can be used to denote programmatic success, including examining factors that contribute to professional development as a motivation for participating in the program. While the primary goal of the ITRE was aimed at improving research/practitioner competence among ITRE scholars, future research might assess the impact of the ITRE on the partnering community agencies specifically to assess stakeholder perspectives of the impact of the program on organization functioning, service delivery, and/or outcomes. Overall, evaluating service-learning programs designed to improve the professional capacity of researchers and practitioners in implementation science is essential for ensuring the development of a competent workforce that can assist in bridging the academic-community gap.

Data Availability

The datasets generated during and/or analyzed during the current study are not publicly available due to participant confidentiality considerations. Aggregate data are available from the corresponding author on reasonable request.