Translating evidence-based interventions from research to practice: challenges and lessons learned

Abstract

Despite the increasing popularity of translation research, few studies have described the process and challenges involved in implementing a translation study. The main objective was to determine whether a multi-component group behavioral intervention could be successfully translated from an academic setting into the community health system of federally qualified health centers (FQHCs) funded by the Health Resources and Services Administration (HRSA) in Miami, NY, and NJ. Key challenges and “lessons learned” from the dissemination and implementation process for the SMART/EST (Stress Management And Relaxation Training/Emotional Supportive Therapy) Women’s Project (SWP) III in low-resource primary care settings are described. The Reach Effectiveness Adoption Implementation Maintenance (RE-AIM) model served as the theoretical framework for the translation of the study. This study outlines several essential factors related to Glasgow’s RE-AIM model that need to be considered in order to accomplish successful translation of evidence-based interventions from traditional academia to “real-world” community health center settings.

Until recently, most health research consisted of formulating innovative approaches to the prevention and/or control of a medical condition; preparing (successful) applications for federal support; demonstrating the efficacy of an untested procedure, compound, or strategy; and publishing one’s findings in one or more scientific journals. Today, this still may be “necessary,” but “not sufficient” to warrant the support of the major health research funding agencies. As Russell Glasgow’s Reach Effectiveness Adoption Implementation Maintenance (RE-AIM) model [1, 2] so cogently outlines, addressing the “effectiveness” and “sustainability” of clinical findings may be equally or even more important than the findings themselves, if research is to have a meaningful impact on national and global public health. Thus, the scientific community has taken on the challenge of demonstrating the utility of research findings through “translational” research, which examines methods to successfully implement and disseminate academic research findings into clinical and public health service settings. This paper will share with the reader some of the key challenges and “lessons learned” from experiences of this research team with the dissemination and implementation process for a multi-component group behavioral intervention in low-resource primary care settings.

The SMART/EST (Stress Management And Relaxation Training/Emotional Supportive Therapy) Women’s Project (SWP) is an evidence-based behavioral intervention designed to enhance the overall quality of life and health status of culturally diverse, disadvantaged women living with HIV/AIDS [3, 4]. The intervention was initially developed in 1996 as a collaborative effort by the research teams of the University of Miami Miller School of Medicine (UMMSM) in Miami, FL, and Clinical Directors Network (CDN), a primary care practice-based research network (PBRN) in NY/NJ. The SMART/EST intervention consisted of 10 weekly, two-hour group cognitive-behavioral stress management, plus expressive-supportive therapy (CBSM+) intervention sessions followed by 6 additional weekly group healthy lifestyles promotion program, for a total of 16 sessions. Several publications from three federally funded studies conducted by UMMSM and CDN over a 10-year period confirmed the evidence base of the program [3–15].

The main objective of “Translating Effective Health Behavior Strategies into Practice for HIV Positive Women (SWP III)” was to determine whether the intervention could be successfully translated from an academic setting into the community health system of federally qualified health centers (FQHCs) funded by the Health Resources and Services Administration (HRSA) in Miami, NY, and NJ. The RE-AIM model, which served as the theoretical framework for the translation of the study, consists of five components critical to the effective translation of interventions from academic to community health-care settings: reach, effectiveness, adoption, implementation, and maintenance. For the purpose of this study, Reach is the number, proportion, and representativeness of those electing to participate in the intervention. Effectiveness is the intervention’s impact on specific outcomes. Adoption is the number, proportion, and representativeness of community health centers (CHCs) willing to integrate the intervention into their health-care program. Implementation is fidelity to elements of the intervention and the consistency of delivery. Maintenance is the degree to which the intervention program is institutionalized as an element of routine “standard of care” service within CHCs. These components, operationalized below, primarily address the external validity of the translation and can also be used to evaluate clinical and organizational variables related to the translation process itself. Below, we describe the major challenges faced in each phase of applying the RE-AIM model during the translation of SMART/EST III to community health-care settings (CHCs), such as FQHCs.

REACH

The reach component of the RE-AIM model refers to the number, proportion, and representativeness of individuals and organizations willing to participate in the program. In SWP III, this involved recruiting women of differing ethnic/linguistic backgrounds in numbers proportional to their representation in the patient base of the participating community health-care centers. Due to sample size and power considerations, the study groups were conducted in English only. However, after the study period concluded, the CHC staff members elected to offer the intervention to monolingual Spanish and Creole speakers.

Community health centers were contacted (n = 12) and invited to participate; all agreed to participate, and of those, four clinics were selected with an additional “backup” CHC for the NY/NJ site. CHCs were selected based upon their HIV-infected female patient census and its representativeness of the ethnic distribution within the local population. As such, all selected CHCs met the criteria for the operationalized definition of reach. The selection and engagement of the participating community health centers and associated health-care providers were a primary challenge during the early months of the project. One of the most pressing obstacles to achieving participation in the translation project was determining how the FQHCs would obtain financial support to continue the project, if successful, after research funding ended. Most FQHCs operate on limited and strictly defined budgets based, in part, on the specific productivity quotas of their health-care clinicians. As such, clinicians may not have the discretionary time that would allow them to spend extensive periods of non-billable time in training or conducting non-billable interventions. Given that financial support was not an available incentive for these CHCs, administrators had to be convinced that participation in the translation project would ultimately be cost-effective for the CHCs, enabling clinicians to administer essential HIV care to patients in a manualized group intervention format, which would improve the efficiency of service provision without sacrificing quality of care. To achieve this, evidence that the intervention would promote positive health outcomes and increase quality of life for their HIV patients by improving medication adherence, promoting positive health behaviors, and increasing emotional well-being was described in detail to CHC administrators and clinicians. In other words, we had to “sell” the project, including the element of translation, to CHC administration and staff, “courting” the selected CHCs and discussing the project aims with CHC administrative and clinical leadership.

An important challenge during this phase of the process was highlighting the benefits of translation research to medical directors with limited research training. In Miami, several CHC administrators and staff shared their reluctance to collaborate with an academic institution on yet “another research study” in which investigators typically “parachute” into their environment, recruit their patients, and depart once the study is complete. In their experience, researchers often required CHC staff complete substantial paperwork that they had neither the time nor the resources to provide, adding to CHC staff resistance to participate in research. In NY/NJ, where CHCs have long-standing experience participating in clinical and translational research as part of a PBRN, concerns persist over the impact of the study on clinician productivity and workflow barriers created by group visit scheduling. Substantial time and effort was devoted to explaining the purpose of the translation study in order to obtain “buy-in” and engagement from CHC staff and administration. It was emphasized that CHC staff would receive valuable professional development, in the form of training and clinical supervision in delivering new clinical behavioral interventions that would enhance their ability to provide effective and cost-efficient services to their patients. Promotion of the intervention and translation process by energetic, dedicated, and highly enthusiastic project staff was immensely helpful to achieve this end.

Administration buy-in facilitated the allocation of important CHC resources, such as space for the intervention, protected staff time for training and implementation, access to patients’ electronic health records (for those patients who consented to participate in the study), and CHC-wide recruitment opportunities. However, clinician engagement and buy-in were also critical components to beginning the study. The initial training (a four-day intensive workshop) was used to generate enthusiasm among the CHC staff to conduct the intervention. After completing the training workshop, CHC staff participated in 8 months of real-time “on-the-job” training and clinical supervision by the research staff as they gradually gained experience in conducting the intervention. Throughout the training process, CHC staff were encouraged to provide feedback and to take ownership of the intervention, such as input on selection of space, time of day, refreshments, incentives for participants during the intervention, and certificates for graduates following successful completion of the intervention. In addition, CHC staff were encouraged to give a name to the intervention at their site. The simple act of naming the intervention at their respective sites and of sharing decision-making with the research staff promoted ownership of the intervention. Certificates of participation and training level were used to increase motivation among trainees as they progressed from “workshop attendee” to “co-leader” to “group leader.” The research staff witnessed the transformation of reticent or resistant CHC staff at the beginning of the training process into enthusiastic supporters of the intervention by the end of their 8-month “internship” period.

In addition to CHC administration and providers, community buy-in and advocacy also functioned as an important catalyst to the translation process. The research team marketed the SMART/EST intervention to community stakeholders, advisory boards, and other community-based organizations representative of the target populations. These efforts proved successful as community stakeholders and leaders functioned as advocates for the intervention and for the successful implementation of the translation. These community leaders championed the intervention to CHC administration, to CHC health-care providers, and to other members of the target community, thereby augmenting receptivity and adoption of the intervention. A list of the most prominent site concerns and their rank order in terms of importance to the CHCs is shown in Table 1.

Table 1.

Order of CHC site concerns

Site concerns	Rank
How will the facility obtain financial support to continue the project? How can reimbursement be obtained for services offered?	1
What is the benefit to participants and staff of translation research; is it not just another study?	5
How can the intervention be integrated into the clinic subculture?	4
How can the intervention be fluid and adapted to the requirements of the CHC environment, e.g., space, staff hierarchy, crisis management, CHC policies?	3
How can CHC staff be supported financially for training to lead intervention groups and follow participants?	2
Can the intervention program be offered in multiple languages?	6
How can trained CHC staff mentor their peers to conduct the intervention?	7

EFFECTIVENESS

In Glasgow’s RE-AIM model, the term effectiveness refers to the impact of the intervention on specified outcomes. Effectiveness was operationalized using the principal study hypothesis posited that the CHC staff-led intervention groups (experimental group) would achieve improvement of health status (depression, HIV viral load) and quality of life as effectively as the research project staff-led intervention groups (control group). Additionally, it included the hypothesis that the CHC staff not only would learn to conduct the intervention as effectively as the research project staff but also that they would learn the skills necessary to train others to conduct the intervention, a critical component for sustainability of the intervention, given the level of staff turnover in FQHCs. Study outcomes affirmed the effectiveness of the translation process as detailed in previous publications [4, 13, 14]; the CHC staff were capable of delivering and implementing the intervention as well as, or better than, the research project staff who had originally provided it for both primary clinical outcomes of depression, medication adherence, and HIV viral load [14] as well as behavioral outcomes [13]. The CHC staff had valuable clinical experience with the target population and already had an established and continuing relationship with the CHC patients in the groups.

Testing study hypotheses relied on the active participation and collaboration of the CHCs. Fidelity of implementation of the intervention by CHC staff depended on the identification of qualified staff and on the quality of training by the research team. Identification of qualified staff to participate proved challenging, as the research staff had to rely on the staff selected by CHC administration. Some staff selected did not have an “ideal” background for delivery of the intervention, as the original research therapists were all doctoral-level (PhD or PsyD) psychologists and required additional training. In other cases, training and certification were provided at the peer or co-leader level rather than at the primary facilitator level. To maintain CHC administration engagement in the translation process, criteria for objective measures of progress and staff performance as well as successful completion of training were developed in collaboration with CHC supervisors. Feedback regarding staff performance on the project was provided while communicating respect and sensitivity for the CHC’s own performance evaluation strategies. One indicator of successful collaboration with respect to training guidelines and staff performance criteria was that the study method of performance evaluation was integrated into the standard CHC staff performance evaluations.

ADOPTION

The term adoption in the RE-AIM model represents the number, proportion, and representativeness of community health centers that are willing to incorporate the intervention into their health-care delivery program. Of the five CHCs participating in the study, four were able to adopt the program. One clinic, due to a change in leadership and fiscal issues, was unable to operationalize and integrate the program into their ongoing clinical services. An additional clinic was added, which adopted the program successfully, clinic number 5. An important challenge encountered in facilitating the adoption of SMART/EST into mainstream service delivery at the CHCs was the integration of the intervention into the changing subculture of each CHC. CHCs operate in a changing regulatory environment with changing policies from HRSA and CMS (Medicaid) to meet the demands of the community and comply with complex requirements for managing quality and cost. The implementation of the intervention was at times threatened by CHC staff turnover, revised roles and responsibilities of staff identified to carry out the intervention, and changes in administrative and supervisory staff originally designated to participate in the project. These changes in the structure of some of the CHCs necessitated considerable flexibility and patience by the research staff. For example, the team was frequently required to re-initiate training without advance notice, accept sudden changes in timing and or location of groups scheduled months in advance, or re-initiate the mobilization of ownership and investment among new CHC supervisors asked to participate midway through the project. Successful management of these challenges and maintenance of the intervention fidelity depended on the understanding and acceptance by the Miami and NY/NJ research teams of the reality that CHCs are characterized by fluid changes in frontline staff roles and responsibilities that require constant flexibility. The research team was repeatedly reminded that for implementation of the intervention to be successful, the team needed to adapt rather than force the CHCs to adhere to the study timeline and training schedule. Integration into the CHC subculture demanded a perceptual shift by the research team: It was we and not they that needed to be flexible. Adaptation also included sensitivity to what appeared to be “minutia” related to the implementation of the intervention, including acceptance of the use of CHC-selected space for the intervention, use of CHC policy on crisis protocol, respect for CHC hierarchy of command, and adherence to CHC policies related to patient care and use of site facilities. In short, successful integration, adaptation, and implementation required understanding and acceptance that the CHCs “called the shots.” Although this flexibility at times challenged ensuring fidelity to the study protocol, the research teams in Miami and NY/NJ also gained an appreciation of the “real-world” complexities and obstacles faced by staff that work in CHCs in the adoption process.

IMPLEMENTATION

In the RE-AIM model, implementation refers to the fidelity to the various elements of an intervention, including consistency of delivery. Implementation was assessed by review of audio recordings of sessions and therapist fidelity checklists (see Appendix 1, Example therapist checklist). To ensure fidelity, audiotaped sessions were used to generate fidelity checklists, which were reviewed by CHC facilitators and research staff together, and feedback and coaching were provided. Overall fidelity to the intervention was approximately 70 % using the checklists. However, the audio recordings revealed that the checklists, while useful, did not capture the continuum of all the sessions, which allowed the facilitators to provide elements of previous sessions to roll over to future sessions. Fidelity was thus likely higher than this estimate.

Research staff had to make special efforts to ensure that these activities were completed. Collecting process data such as fidelity checklists may be seen as particularly burdensome by CHC staff in service settings. In addition, for the translated intervention to maintain fidelity, project strategies must ensure an adequate and sustainable training resource for qualified CHC staff. In SWP III, the major challenge was transferring program ownership to CHC staff while maintaining program fidelity. This was accomplished by incorporating the “train-the-trainer” strategy, whereby each trainee not only learned the skills necessary to conduct the program but also learned how to train others to become group leaders. Given the level of CHC staff turnover, incorporating training skills into the format of the training program was essential to creating a “reservoir” of skilled talent available to meet the staffing needs in each CHC and to implementing a sustainability plan to add to the reservoir. Specifically, after the CHC staff completed a four-day training workshop, research staff served as the group intervention leaders with community health center site staff serving as co-leaders for the first two treatment cohorts. During the third and fourth cohorts, CHC staff assumed the role of group intervention leaders and research staff served as co-leaders, providing continuing on-site supervision and support to CHC group leaders. Following a performance review and certification by project staff trainers, CHC staff conducted all subsequent intervention groups, with CHC group leaders continuing to train other CHC staff as co-leaders who would ultimately lead groups themselves.

MAINTENANCE

Maintenance is the extent to which a program becomes institutionalized as part of routine standard of care service delivery within the CHC. Of all of the RE-AIM components, the “proof of the pudding” of successful translation lies in whether the program will be sustained after withdrawal of developmental financial support. Maintenance was operationalized within the hypothesis that community health center leadership in at least three of the four participating sites would elect to continue the intervention by the end of the grant period. Currently, at least two of the four participating sites continue to provide the intervention. A key factor that resulted in continuation of the intervention at those two sites was the successful development and implementation of a reimbursement model for the intervention, which relied primarily on Ryan White Title IIIb funding for staff to provide the intervention, as opposed to generating revenue from billable encounters. Although there are several internal factors of central importance to whether the program continued after the end of the study, the primary concern of CHC leadership was whether reimbursement for services offered could be obtained from external sources. This issue was resolved by securing funding from Ryan White based upon the relevance of SWP III to Ryan White objectives.

Sustainability of the program was facilitated by attention to factors believed essential to its maintenance. In addition to a viable reimbursement model, CHC administration and staff commitment to continuing the program was critical for its sustainability. One important factor that facilitated staff and administrative commitment to continuing the program was the time and effort invested by the CHC in the training of personnel to deliver the intervention. Positive reviews and enthusiasm by participants in the intervention also contributed to CHC commitment. Several participant graduates of the intervention expressed a desire to mentor their peers and to contribute to new cycles of the program by appearing as guest speakers or peer facilitators. CHC staff trained in the provision of the intervention also reported interest in delivering the intervention to other clinical populations beyond HIV+ women, such as men living with HIV and transgender persons. While this speaks to the enthusiasm of the CHC staff to continue to provide and broaden the intervention, adapting the intervention to new populations may represent a significant departure from the population for which the intervention was developed and tested and presents a challenge to external validity. This further dissemination represents a positive step in the direction of program ownership, similar to naming the program, and should be determined by the CHC leadership who are responsible to their communities. This issue notwithstanding, it was clear that staff and participant enthusiasm to continue to provide the intervention fueled CHC administrative commitment to integrate the program into their standard of care.

Although funding was a primary concern of CHC leadership, the high turnover of CHC staff remained a challenge to sustainability of the program. The ongoing training process for new facilitators was essential to maintain the integrity and fidelity of the intervention. Integration of the SWP intervention into regular service programming at the participating CHCs required continued implementation of the “train-the-trainer model” applied during the initial translation of the intervention. CHC personnel trained in the intervention applied the training methods they had learned with newly selected CHC staff interested in learning and implementing the intervention. The research team made training resources available to participating CHCs, such as training manuals, training videos, training certification criteria, etc., to facilitate on-going training efforts. Training new, interested CHC providers ensured a reservoir of facilitators available to provide the intervention to a variety of populations. For example, one Miami site continues to deliver the intervention in Spanish for monolingual female patients; this site also used the train-the-trainer model to train an additional male provider who is currently providing the intervention to a group of male patients. A second Miami site trained a Creole-speaking provider to provide the intervention for monolingual Haitian patients. Finally, the research team also continued to function as a “sounding board” to guide training efforts at the participating CHCs after the study ended and to provide additional albeit reduced guidance. Finally, the provision of training skills also provided the potential to expand the program to other CHCs, using experienced group leaders as trainers of staff from other CHCs in their region. A summary of each of the RE-AIM components with associated outcome data is presented in Table 2.

Table 2.

RE-AIM components and associated outcomes

RE-AIM component	Description	Outcome
Reach	The number, proportion, and representativeness of those electing to participate in the intervention.	Community health centers (n = 12) contacted; all agreed to participate. Four clinics were selected with an additional “backup” for the CHC NY/NJ site. “Buy-in” and ownership were demonstrated by the CHC staff enthusiasm, willingness to participate in the translation (e.g., naming the intervention, developing new target groups/languages/adaptations for the intervention) and by the degree to which they became the principal advocates for the intervention to CHC administration and to their patients.
Effectiveness	The impact of the intervention on specific outcomes.	CHC staff delivered and implemented the intervention as well as, or better than, the research project staff who had originally provided it for both primary clinical outcomes of depression, medication adherence, and HIV viral load [14] as well as behavioral outcomes [13]. Effectiveness of delivery was measured by performance evaluation checklists that assessed comprehensiveness of content covered in each session, as well as facilitator skills. Audio tapes of CHC-led groups were evaluated by the research team on a biweekly basis.
Adoption	The number, proportion, and representativeness of community health centers (CHCs) willing to integrate the intervention into their health-care program.	Five CHCs participated; 4 were able to adopt the program. One clinic was unable to operationalize and integrate the program into their ongoing clinical services. An additional clinic was added, which adopted the program successfully (clinic 5). Flexibility and adaptability of the research team were evaluated by the degree of willingness to adjust intervention schedules and other logistics associated with the program in a manner that favored the CHC subculture. CHC/study team meetings were used to review the procedural fit and relevance of the intervention within the CHC setting, encouraging considerable flexibility in problem-solving among the entire team. Integration of the intervention into existing services was evaluated by objective evidence that the intervention was being included in their mainstream services (i.e., observing clinic schedule of events and programs offered to the patients during and after the translation was complete).
Implementation	Fidelity to elements of the intervention and the consistency of delivery.	Overall fidelity to the intervention was approximately 70 % using the checklists. Audio recordings revealed that the checklists, while useful, did not capture the continuum of all the sessions, which allowed the facilitators to provide elements of previous sessions to roll over to future sessions. Fidelity was likely higher than 70 % and was measured by periodic evaluation of performance checklists, content checklists collected at each session, and review of audiotapes of each intervention session.
Maintenance	The degree to which the intervention program is institutionalized as an element of routine “standard of care” service within CHCs.	Currently, at least 2 of the 4 participating sites continue to provide the intervention. Sustainable staffing was measured by the amount of staff involved in the translation that continue to 1) Provide the intervention 2) Train other providers in the CHC in provision of the intervention Evidence of development of a reimbursement model was provided by reports from the CHC administration of success in obtaining reimbursement for the intervention (e.g., Ryan White funding). Staff costs associated with training were CHC personnel salary covering attendance at the 2-day workshop and subsequent hours spent in supervision. Actual hours spent co-leading/leading treatment groups were covered by existing commitment to provision of CHC services. Change in leadership and staffing problems resulting in poor enrollment contributed to failure to maintain the intervention at the 1 discontinued site. Funding for the study and training activities was obtained from a grant from the CDC.

Limitations of this study are primarily related to the small number of CHCs involved, all of which were “inner-city,” restricting its generalizability to other settings, e.g., rural communities. As mentioned earlier, power and sample size considerations limited the study population to English speakers. Although four of the five sites continued the intervention at study termination, only two of the four CHCs had maintained the intervention at 2 years post study completion (one of the five CHCs was terminated in the initial stages of the study due to a lack of support for the study resulting from a change of senior CHC leadership). Although only HIV-seropositive women were included in this study, two of the sites adapted the intervention to HIV-seropositive men and monolingual (Creole, Spanish) patients following study termination. Additionally, as the resources available through Ryan White funding may have enhanced the continued provision of the intervention, CHCs serving other patient populations who do not receive Ryan White funding may be less willing to support the necessary staff commitment.

In summary, this study outlines several essential factors related to Glasgow’s RE-AIM model that need to be considered in order to accomplish successful translation of evidence-based interventions from traditional academia to real-world CHC settings. In terms of the reach component of the RE-AIM model, committing the significant time and energy to develop buy-in and commitment from CHC administration and staff and the transfer of ownership from the research team to the CHCs were crucial for the translation process. The sites that demonstrated the most evidence for buy-in and ownership of the intervention were the sites that incorporated the intervention as part of their routine service delivery. As noted in our earlier publication [14], transferring ownership without sacrificing “core” elements of the intervention (e.g., cognitive behavioral skills, group format, session content) in the health-care context required extensive negotiation between the research and CHC leadership and staff. Effectiveness was demonstrated by the ability of CHC staff to achieve study outcomes equal or superior to research staff. Successful adoption required a perceptual shift by the research team involving an understanding that the research team needed to be flexible and adapt to the CHC subculture, rather than requiring the CHCs to follow pre-determined procedures and timelines. This involved an ongoing willingness by the research team and CHC staff to collectively improvise, innovate, and problem-solve to integrate the program into the existing CHC service delivery system. The research team’s ability to adapt to the CHC subculture was evident in the degree to which they were willing to make decisions associated with the logistics of the intervention that favored the CHC, rather than the research team. Successful integration of the intervention into the CHC subculture in turn resulted in greater adoption by the sites. Implementation was accomplished by the use of a sustainable training model (i.e., train-the-trainer) that could be replicated by CHC staff on an ongoing basis. Finally, the maintenance component of the model required the development of a sustainable staffing, finance, and reimbursement model for the services provided. The sites that adopted the intervention were the ones that were successful in developing a viable reimbursement model for the translated intervention. Although the sites did not incur any direct cost associated with training or implementing the project, the considerable time spent in training the providers in the intervention and implementing the project represented a potential loss of profit (i.e., billable service hours) for CHC administration. Therefore, it was very important to demonstrate to the CHCs that the initial time spent in training their providers would pay off by augmenting the quality of care in a manner that could be sustainable with the development of a reimbursement model.

Given the large number of CHCs funded by HRSA, which in 2012 included nearly 1200 grantees and over 7000 delivery sites who provided primary care to 21.1 million patients, with nearly 115,000 HIV+ patients [16], the potential impact of disseminating and implementing this effective, evidence-based intervention across all HRSA-funded sites would be quite significant, for both patients and staff. Both the NIH and the CDC have recognized the importance of translation research, and increasing numbers of funding opportunities have targeted translation, implementation, and dissemination research to facilitate the uptake of research findings into clinical settings. Future translation studies should include a reimbursement strategy, enabling the health-care system to rapidly convert effective interventions to ensure the sustainability of new clinical services.

Appendix 1

Example therapist checklist