Translating HIV Interventions into Practice: Community-based Organizations’ Experiences with the Diffusion of Effective Behavioral Interventions (DEBIs)

Abstract

Efficacious behavioral interventions developed to address the spread of HIV/STIs are currently being disseminated in the USA through a national diffusion program (DEBI) spearheaded by the Centers for Disease Control and Prevention (CDC). Understanding how interventions are translated to real world settings is necessary to further scientific knowledge of this process and to facilitate future translation efforts in public health. Prior studies have begun to elucidate how agencies translate behavioral interventions into practice, but further work is needed. Guided by the ADAPT framework, we examined agencies’ assessment, preparation, and implementation of interventions. Our qualitative interview-based study focused on six community-based agencies in California (United States) funded to implement three group-level HIV interventions. Findings showed considerable variation in the extent to which agencies engaged in assessment and broad-based preparation and in the ease with which agencies implemented the interventions. The findings provide insight into the process that agencies undergo in the translation of effective behavioural interventions and illustrate how agencies can inform logic models that guide translation. We also identify relevant dimensions of existing models, including the ADAPT framework and Roger’s (1995 and Roger’s (2005) diffusion of innovations in organizations, that have value for agencies that are translating research to practice.

Efficacious behavioral programs developed to address the spread of HIV and STIs in the United States are currently being disseminated through a national diffusion program (DEBI) spearheaded by the Centers for Disease Control (CDC) (Collins, Harshbarger, Sawyer, & Hamdallah, 2006; www.effectiveinterventions.org).

Prior studies have begun to elucidate how agencies translate behavioral interventions into practice, but further work is needed. It is essential that systematic examination of adoption and implementation of programs in real world settings is conducted to advance the science of dissemination (Dworkin, Pinto, Hunter, Rapkin, & Remien, 2008), to provide guidance for HIV prevention technology transfer (Veniegas, Kao, Rosales, & Arellanes, 2009), and to contribute to increasing the effectiveness of interventions in practice settings (Jemmott, Jemmott, Fong, & Morales, 2010).

The increasing emphasis on delivery of evidence-based programs in HIV prevention provides an opportunity to observe the process of translation. The majority of state health departments and CBOs funded through the CDC came under the mandate of delivering evidence-based behavioral interventions (EBIs) (McKleroy, Galbraith, Cummings, Jones, Harshbarger, Collins et al., 2006). However, given the strong grassroots history of HIV prevention, this new approach required a cultural shift for community-based organizations. Instead of developing programs internally, agencies must select from among the available interventions and, if needed, modify the program for delivery in their community. Further, agencies must have, or develop, the expertise to successfully implement the intervention. While there is a strong rationale for evidence-based practice (Institute of Medicine, 2003), research that examines translation is needed to strengthen behavioral translation models and to build our knowledge of factors that facilitate effective translation of programs.

Most health-related advances require some level of adaptation as they move from research to practice (Elliott, O’Loughlin, Robinson, Eyles, Cameron, Harvey et al., 2003). Behavioral programs pose unique challenges because they address the client’s individual needs under particular circumstances (Glasgow, Lichtenstein, & Marcus, 2003); thus they are likely to require greater adaptation than a typical vaccine program or medical interventions where dosage and administration can be clearly specified for general use (Glasgow et al.). In this regard, behavioral programs may require adaptation in response to expressed needs of the local community or of specific clients, or to fit into an existing organizational structure. It is increasingly clear that adaptation occurs when proven interventions are incorporated into practice (McKleroy et al., 2006; Rebchook, Kegeles, Huebner, & the TRIP Research Team, 2006; Veniegas et al., 2009) and that there is a need for appropriate models that provide systematic guidance for the translation of behavioral interventions. The current study draws on one of the few available guides, the ADAPT framework (McKleroy et al., 2006), to examine the translation of group-level EBIs in six community-based agencies. This qualitative investigation is not aimed at hypothesis testing; our goal is to discover how agencies are going about translating EBIs, the extent to which agency activity corresponds to the proposed framework, and what the findings suggest for future translation models.

The ADAPT framework is a logic model describing the steps involved in the adoption, adaptation, and implementation of a behavioral intervention (McKleroy et al., 2006; see Table 1). The framework draws heavily on the innovative-decision process explicated in Rogers’s diffusion of innovations theory (Rogers, 1995, 2005). This process has five steps: knowledge, persuasion, decision, implementation, and confirmation. Research on the innovative-decision process has focused primarily on how individuals, rather than organizations, adopt innovations. There is some evidence of stages (or steps) in the innovative-decision process, but there is no universal evidence for each step (Rogers, 2005). Data suggest that passing through the innovative-decision process takes time, with the time factor reflecting years, rather than weeks or months. Individuals tend to pass through a knowledge and persuasion step, but not all engage in a trial phase prior to implementation (Rogers, 2005). The ADAPT framework was designed to provide a comprehensive view of the steps involved in organizations’ adoption of innovative programs. Below we outline the ADAPT framework, which was developed in part to prepare for the national diffusion of EBIs, and review the available literature on adoption of EBIs.

Table 1.

Comparison of steps in ADAPT framework, Diffusion of innovations theory, and “in practice”.

ADAPT Framework	Diffusion-Innovation Processa	In Practice
Assess: Assess target population, potential EBIs, agency capacity, stakeholders, “goodness of fit”	Knowledge Stage Persuasion Stage	Informal assessments of target population; frequently no assessment of agency capacity or of stakeholders
Select: Decision to adopt, adapt, or select another intervention; determine needed changes in intervention, develop logic model and timeline, plan and make necessary changes	Decision Stage	All agencies selected an EBI, but few systematically determined necessary modifications at this juncture
Prepare: Carrying out adaptations, organizational preparation, pre-testing	Implementation Stage	Emphasis was on organizational preparation; adaptation sometimes non-systematic, pre-testing infrequent
Pilot: Implementation plan, pilot test adapted intervention	Implementation Stage	Implementation plan; pilot informal; implementation often viewed as pilot
Implement: Implement adapted intervention including (a) process monitoring, evaluation, (b) quality assurance, (c) outcome monitoring and evaluation	Implementation Stage	Variable success in implementing the EBIs, variable quality assurance, minimal evaluation and monitoring

^aFrom E. M. Rogers’ Diffusion of Innovations (2005).

According to the ADAPT framework, the adoption process has three primary phases: assessment, preparation, and implementation. The assessment phase includes assessing the target population, the stakeholders, organizational capacity, and the available interventions. The assessment of available interventions should include gaining an understanding of the core elements of a program, which are the essential features that are thought to be responsible for the intervention’s outcomes. The assessment process should lead to the selection of an intervention that is a good fit for the agency (i.e., the organization has the capacity to deliver the program) and for the target population. To date, there is limited information about the assessment and selection of EBIs or about how the EBIs fit with the culture of the organization adopting these programs. Past work has shown that provision of resources (e.g., technical assistance manuals, staff training), such as that provided in the DEBI process, facilitates adoption of EBIs (Kelly, Somlai, DiFranceisco, Otto-Salaj, McAuliffe, Hackl et al. 2000).

The preparation phase includes organizational preparedness, such as hiring and training staff or finding suitable locations for intervention delivery. It also entails adapting and pre-testing the intervention with the target population. Given the interrelationship between adaptations, program fidelity, and program outcomes (Blakely, Mayer, Gottschalk, Schmitt, Davidson, Roitman et al., 1987), it is important to understand what adaptations are made and why agencies make these changes in the evidence-based interventions they adopt. Prior research indicates that adaptations are made for a number of reasons including: to meet the target community’s needs, to increase ownership of the program, to expand the program to address additional issues, and/or to simplify a complex program (Rogers, 1995). Additionally, agencies sometimes make adaptations because they lack knowledge about a program, have limited capacity, or require their own specific changes (McKleroy et al., 2006). Research on adaptations of EBIs provides evidence of significant deletions of program core elements, substantial changes or additions in activities, and changes in the size of intervention groups (McKleroy et al., 2006; Rebchook et al., 2006; Veniegas et al., 2009). In a recent study of organizations that had adopted Focus on Kids, a group-level EBI, one core element involving group composition was rarely implemented, two other core elements were implemented by less than half of the organizations, and no organization included all eight core elements (Galbraith, Stanton, Boekeloo, King, Desmond, Howard et al, 2009). The majority of research on adaptations is limited to several DEBIs (i.e., Mpowerment, Focus on Kids, VOICES, Popular Opinion Leaders).

According to the ADAPT framework, as the agency moves from preparation to implementation, it either develops an implementation plan for the adapted intervention or conducts a pilot test of the adapted intervention or of its components. In the final phase, the adapted intervention is implemented. The present study provides an opportunity to examine these latter steps in the model in greater detail. In addition to proposing the phases of translation and outlining steps within phases, the ADAPT framework also suggests that there are feedback loops in the process; agencies may revisit earlier steps if they encounter difficulties and need to reconsider prior actions or decisions. Finally, the framework includes both process and outcome monitoring evaluation conducted throughout the three phases.

The present study builds on prior work in the adoption of innovations by focusing on agencies funded to deliver three group-level EBIs. We employed qualitative methods to address prior limitations of research on the innovative-decision process (Rogers, 2005) and the limited data on implementation of programs (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005). This qualitative study examined the experiences of agencies that were involved in the initial cycle of funding for DEBIs by the CDC. The ADAPT framework, which was developed about the time these agencies were funded, provides a lens through which their experiences can be viewed.

Method

Six agencies in California that were directly funded by the CDC to implement three different DEBIs targeting high-risk populations were invited to participate. We focused on group-level DEBIs because of their wide availability and high levels of adoption. We conducted an in-depth structured interview with two types of key informants: Executive Directors (or Program Managers) and staff responsible for implementing the program (Implementers). Face-to-face interviews were conducted between June 2006 and November 2007. Agencies received a stipend for participating in the project to compensate for time and inconvenience.

The study team, including interviewers, developed, pilot tested, and revised instruments prior to fielding the study. One interviewer was a senior scientist with a doctorate, and all others had masters degrees in a related field and were experienced trainers. Interviewers completed a 2-hour training on the study instruments prior to going into the field. The instruments were pilot tested with an Executive Director and an Implementer from two community-based agencies, not part of our sample, that conduct group-level DEBIs.

Interview Content

Structured interviews were designed to reflect the steps identified in the ADAPT framework (McKleroy et al., 2006), although the study is not intended as a test of the model. The interview addressed (1) assessment (of the target population, the intervention, stakeholders, and the organization) (e.g., Before selecting [program] what formal and/or informal assessment activities did you conduct?), (2) selection of the intervention (e.g., Tell me a little about why your agency chose [program]?), (3) adaptation (of the intervention, organization/preparation, and pre-testing) (e.g., Tell me about what changes, if any, your agency made to prepare for implementation.), (4) pilot testing, and (5) implementation (pre-testing, implementation of adapted intervention) (e.g., What is working well in implementing the intervention?). The full interview guide is available from the authors. The Executive Director and Implementer interviews differed somewhat in emphasis: The ED interview focused more on funding decisions, application procedures, and administrative issues, and the Implementer interview focused more on delivery of the intervention.

Recruitment and Procedures

With approval from our institutional review board we contacted the Executive Director/Program Manager (ED, PM) at six agencies funded by the CDC to implement DEBIs, to assess interest in participation. Agencies were chosen to represent both Northern and Southern California and to include three different group-level DEBIs. No Executive Directors declined to participate. A brief telephone screener familiarized EDs with the overall goals of the study and scheduled interviews. Face-to-face interviews were conducted first with ED/PM, and then with staff implementing the study; interviews were scheduled at a mutually agreeable time. One implementer declined to participate; another implementer was selected and interviewed at this agency. Each participant provided written informed consent prior to participation. Interviews averaged 80 minutes in length and were audiotape recorded and transcribed. Transcripts were checked by staff for accuracy.

Coding

Prior to developing a coding scheme, multiple members of the study team read all completed interviews. The team then developed a coding scheme that reflected the ADAPT framework and could examine the extent to which this framework applied to the participating agencies. A number of additional codes were developed to reflect other important themes emerging from the interviews. The current analyses focus on the congruence between the ADAPT model and community-based organizations’ practices.

Two members of the study team coded each interview using the coding criteria. At key points, a senior investigator met with coders and reviewed coding decisions. Throughout the process, coders met, compared codes, discussed disagreements in coding, and came to consensus on final codes (100% agreement). Codes were entered into NVIVO software program.

Results

The six agencies funded to deliver group-level EBIs in California that participated in the present study were engaged in implementing one of the following EBIs: Healthy Relationships (Kalichman, Rompa, Cage, DiFonzo, Simpson, Austin et al., 2001), Safety Counts (Hershberger, Wood, & Fisher, 2003; Rhodes, Wood, & Hershberger, 2000), or Many Men Many Voices (Kelly, St Lawrence, Hood, & Brasfield, 1989; Wilton, Herbst, Coury-Doninger, Painter, English, Alvarez et al., 2009). Our sample included two agencies delivering each of the interventions. All three EBIs are multi-session group-level interventions that focus on high-risk populations. Healthy Relationships is a five-session small-group intervention focused on men and women living with HIV/AIDS. Safety Counts targets active injection drug users and crack cocaine users and has seven sessions that include both group and individual components. Many Men Many Voices is a six-session group intervention that targets gay men of color.

Characteristics of Agencies and Agency Staff

Agencies

Three participating agencies were located in Northern California and three were in Southern California. Agencies varied considerably in size, ranging from smaller agencies that served 2,500 clients at several sites to an agency that served 22,000 clients at over 20 sites (see Table 2). We included agencies that provided multiple services and those that focused exclusively on HIV/AIDS prevention and care services. The populations served by the agencies also were varied and covered the range of populations that reflect the high-risk groups for HIV/AIDS.

Table 2.

Characteristics of Community-Based Organizations (N = 6).

Geographic Region n (%)	Agency Type n (%)	Percent of Funding devoted to HIV/AIDS Prevention n (%)	Agency Size (number of sites) n (%)	Number of Clients Served
Northern Ca 3 (50)	HIV/AIDS Care and Prevention Only 2 (33)	≤20% 1 (16)	1–2 Sites 1 (16)	≤5000 2 (33)
Southern Ca 3 (50)	Multi-service 4 (67)	21–40% 3 (50) ≤41% 2 (33)	3–5 Sites 3 (50) ≤6 sites 2 (33)	5001–10,000 2 (33) ≤10,001 2 (33)

Percents do not add to 100 due to rounding.

Agency Staff

Executive Directors

Six Executive Directors/Program Managers participated in the study. The EDs/PMs’ experience in public health ranged from 6 to over 20 years. EDs/PMs had been in their current position for at least 1 year; two had less than 5 years at their current agency, while four had 6-10 years experience at their agency. Five out of the six EDs/PMs held masters of arts degrees and one had a bachelor of arts degree.

Implementers

Nine implementers from six agencies completed interviews. Implementers had between 4 and 10 years experience working in public health. They had been employed at their current agency for various lengths of time, ranging from 6 months to 8 years. Two implementers held bachelors degrees and the others had public health-related training and practical experience. On average, implementers had two and a half years experience at their agency.

Mapping agency activity onto the ADAPT framework

As described earlier, the ADAPT framework suggests three phases that correspond to five action steps: assess, select, prepare, pilot, and implement. Our analyses focused on the extent to which agencies engaged in these steps as they sought funding for and carried out the interventions (see Table 1). Below, we present agency activities relevant to each step in the framework and provide representative quotes from Executive Directors (designated by ED) and Implementers (designated by I) (indented and in italics).

Assess

The initial step of the ADAPT framework focuses on assessing factors that impact the decision to select an evidence-based behavioral intervention (EBI). This step includes assessing the available DEBIs, identifying and understanding the target population, assessing the ‘goodness of fit’ between the intervention and the target population’s needs, identifying stakeholders and their needs, identifying potential collaborating organizations, and identifying other organizational factors related to resources and experience. The data below reflect the assessment process for participating agencies. Overall, this process was more limited than might be ideal. Agency activity focused to some degree on elements related to organizational capacity and financial considerations; less attention was paid to “goodness of fit” and to stakeholders.

Target population

Assessment of the target population was typically informal and based on existing knowledge of the community. No agency conducted a formal assessment of the target population to assist in the process of intervention selection.

About 70% of the population that come to [agency] for services is Latino…. (I) …all of our programs are focused around [type of client] and their families. (I)

Interventions

Assessment of the interventions focused on whether the intervention had been developed for the population that the agency was to target.

It was a natural fit into the population we serve. And a large amount of our population are injection drug users. It was an easy fit. They’re not that many DEBIs that apply to injection drug users, so there wasn’t much of a choice. (ED)

Few agencies examined the overall fit of the intervention. In general, agencies did not look closely at the intervention structure and content at this stage.

Organizational factors

Agencies focused on organizational factors to a greater extent than on other dimensions identified in the framework, but even this activity was limited in scope. Interestingly, this frequently involved a process of elimination focused on choosing an EBI that shared similarities to other programs the agency had conducted.

So it was more a matter of elimination as opposed to “YES, we know how to do this stuff.” Because, in fact, it represented a totally new way of doing things for us and for the agencies we’re working with, and it probably wasn’t the best match. (ED)

I don’t want to be so crass as if it was just a process of elimination, but in a way we looked at all the things they [CDC] were going to fund, and we said, OK, what can we not do, and crossed those off right away. Then there were a few that were left . . . And, then we had a dialogue in relation to our organizational capacity was as far as what made sense, and we ended up with [intervention]. (ED)

Organizational capacity is closely linked to financial stability. For some agencies, assessment focused primarily on financial considerations; an agency’s need for resources to continue to support programs and staff played a significant role in seeking DEBI funding.

We were part of a [name] demonstration project. And, that was coming to an end, so we were actively looking to how to not lay people off. (ED)

And it seemed like our only option was to do it in this format, which was through, applying through the DEBIs. . . . Because previously we had been doing much more of a traditional outreach model. And, not done a structured program. (ED)

Stakeholders

Finally, assessment of stakeholders was usually done after the fact, if at all. Only one agency reported formally meeting with stakeholders. Other agencies appeared to make decisions internally, without soliciting input from others.

When the decision was made [to fund the intervention] and we obtained our first on-line staff. … We actually had a Community Advisory Board meeting and we invited the community, more or less our target population. (ED)

Select

Selection or adoption of an EBI is the second step in the ADAPT framework. All agencies in our sample selected an EBI. As is evident above, the factors that contributed to selection frequently were at variance with the framework. In particular, although organizational factors weighed heavily, there was often a lack of consideration of the ‘fit’ between the EBI and organizational capacity. In at least one case, the agency staff did not understand the nature of the selected intervention until personnel attended EBI specific trainings. Initially, some agencies were unfamiliar with the components of a given EBI, even when they had selected it for use.

And it was a while before the training, not everybody went to trainings right away, so it took a series of months for everybody at all the sites to actually go through all the trainings. And only once you go through trainings did we understand, did they really understand what this intervention was, and what we were gonna be expected to do. (ED)

Prepare

Following selection and funding decisions, agencies began preparation for implementing the EBI. According to the framework, preparation includes adaptation of the intervention, organizational preparation, and pre-testing the intervention with the target population.

Organizational preparation

All agencies engaged in some degree of organizational preparation specific to the EBI. Not unexpectedly, this often focused on hiring new staff. Staff turnover made building organizational capacity difficult and interfered with implementation. We observed looping back to an earlier step in the process; that is, agencies that lost staff were forced to return to organizational preparation, sometimes more than once in order to rehire and retrain staff.

Well, there wasn’t a lot of structural changes…. It was more hiring new staff…I think for us it was the challenge of starting up a new intervention. (ED)

We were in the process of implementation, within the first two months of finally getting our feet wet and starting our groups, one of the individuals resigned and moved…. so we were kind of hindered a bit…The next outreach worker also left our agency…It took us a long time to get a replacement. (ED)

Adaptation

Adaptation of the intervention also took place at this juncture and is considered part of preparation. Adaptation should include careful attention to core elements (Kelly, Heckman, Stevenson, Williams, Ertl, Hays et al. 2000) and documentation of adaptations is recommended (McKleroy et al., 2006). All but one agency made some adaptations to the EBI. At the time the agencies in our study were funded, the primary advice concerning adaptations was that adding or deleting sessions violated core elements and that changes that contradicted any core elements were also unacceptable.

The following adaptations were reported: additions or deletions, translation into another language, integration into other services, and non-systematic adaptations. Additions and deletions to interventions were done to accommodate the needs and desires of the target population. For example, one agency added a session because participants wanted an opportunity to discuss issues with each other in an informal way. Another agency deleted a session because they felt that a shorter intervention met the needs of the population. Although additions to programs are no longer viewed as failing to adhere to core elements, deletions of critical portions of the intervention are.

Integration of the EBI into other services offered by the agency was an important aspect of making the EBI available to clients. In the view of one implementer, the evidence-based intervention must be delivered in the context of other services.

The reason that I think that we work well is that we have other supportive services in-house. And, I don’t know if all the [other funded] agencies have other supportive services as well. If they’re not, then it will be a big challenge. [Intervention name] is not a stand-alone program. You need to have other things in place. (I)

In at least one case, adaptation appeared to be unsystematic and it was not clear how the adaptations influenced core elements. One agency had plans for adapting a multi-session program and conducting it as a retreat.

We are considering doing it in retreat form. (I)

One agency added a session and another reported no need to make adaptations.

[additional content was added] to the sessions…. We talked about stigma, homophobia, internalized homophobia, discrimination. All the Latino factors. (ED)

We didn’t change the intervention at all. We followed it strictly by the manual. .. The manual was so perfect … There wouldn’t be any problems with implementation. (ED)

Pre-testing

Another element of preparation includes pre-testing all or part of an intervention, prior to implementation. Pre-testing aims to ensure the cultural appropriateness of the intervention, and may include review of material by knowledgeable community members, and focus groups soliciting feedback on readability, compatibility, or attractiveness of the intervention. Only one agency in our study reported conducting a pre-test.

We had a focus group of about ten people before we did the intervention and we basically had a set of questions and information that we showed them to see how well they would receive the information. And then after that we did some tweaking of what we were going to do at the time with our intervention. And then we did …. a pilot of our intervention. (I)

Pilot/Implementation Plan

According to the ADAPT framework, the pretesting should be followed by a pilot. A pilot test involves a preliminary examination of the adopted/adapted intervention to determine feasibility and assess potential of the intervention to achieve desired outcomes. In the present study pilot testing was an informal process involving trial runs with the agency’s program staff (house staff) or adjustment of the EBI after the first or second round of implementation. In this regard, a number of agencies considered the first round of implementation as a pilot. Two reported running pilot tests.

We had a group of around six or seven individuals who came frequently once a week…and people were actually um very involved in it. …We piloted the entire series. (I)

Well, because of our bumpy start with the intervention, I looked at a lot of that as being a pilot. (ED)

The next step in the translation process would be the development of an implementation plan, but in many cases, agencies had already begun this activity. Agencies had identified the target population and had informal knowledge about the group’s risk behaviors. Less attention was given to outcomes because most agencies did not have a mechanism for evaluation. As noted elsewhere, the study sample included agencies that had both internal and cross-agency referral systems, but these focused primarily on recruitment rather than on identifying resources for clients who have needs beyond those addressed in the EBI.

Implement

The final phase of the model is implementation of the adapted intervention. Ease of implementation varied considerably by agency, with some quickly getting the intervention rolling while other agencies struggled to get the program launched.

Um, I think it’s gone really well. I think the trick with the intervention is really being able to adapt it and to tailor it. I think if we stuck to the curriculum very strictly it would be tough. But that there is room, I think, to aim it towards, towards the population, and I, I really feel like [the clients] are getting stuff out of it. (I)

And so we feel very limited because … this is the enrollment session for the program, and so there’s just this clash of, it’s like a culture clash to me around the culture of research and the culture of the street. And the reality of these people’s lives. So, it just doesn’t seem like a very successful way to begin that relationship with them. (ED)

Um, it was kind of a process of trial and error of how, um, a lot of the, a lot of pieces of the intervention were new to us and new to the staff. Especially like for the social…. What technique works best when introducing this to the client, on inviting people and friends in. It took us a while to kind of figure that one out and find something that was effective. (ED)

Process monitoring and evaluation are included in the ADAPT framework, and may take place at many stages. Typically, agencies funded to implement DEBIs are not required to do formal outcome evaluations, but some agencies engaged in activities to better understand how the program was received. Further, some agencies may have requirements from other funders that mandate evaluation. Most evaluation was focused on process and was often related to acceptability. One participant noted that his/her agency had an evaluation department, but this did not appear to be the norm. Some study participants focused on personal perceptions or feelings that the program was working, but there were no formal outcome evaluations.

We’re evaluating in the sense of the basic three forms that we have. We have a pre and a post test and [a specific form that another agency requires]. Those are the three forms that I know of that are components of it [evaluation]…The concern, the question that we have is, How do you evaluate the data we provide? What does it mean? And, we don’t have a system in place for that [evaluation]. (I)

…I know the evaluation form, but I sometimes don’t believe in this form. Because the people don’t say all… When, after the session, we talk to the clients, we ask them how they feel, what they think about the session. Most clients say that it’s good, it’s helpful. Most of the clients disclose their sexual orientation, their HIV status in the process. But this is not the reason [I think it is successful]. The reason is lower stress. Some people told us, “I feel good, no more stress.” Maybe they don’t or won’t disclose their status, but they don’t have stress anymore. And, it’s good. This is the reason that this intervention is working. (I) (Although defining stress and improving coping skills are core elements of this DEBI, it is not a formal outcome for the intervention.)

…I think there are probably [some behavior changes]. It’s just a guess. I can’t say that the intervention is effective. I can’t say that it’s not. (I)

Discussion

The ADAPT framework, a logic model proposed by the Centers for Disease Control and Prevention (CDC) was used to frame our qualitative investigation of community-based agencies funded to implement three group-level DEBIs in California. This framework is a heuristic model that will evolve as research findings emerge (McKleroy, 2006). In the following sections we discuss the theoretical and practical implications of our findings.

Theoretical Implications

Our investigation provides new information of theoretical importance to the field of translation. Based on our findings we outline what was learned from agencies that can inform future translation models, what aspects of the current framework appear to be valuable, and we also discuss how future models may benefit from incorporation of Rogers’s newer model of translation in organizations.

How agency experiences can inform the ADAPT framework

Data from our investigation suggest that translation logic models should be designed to explicitly address issues of agency buy-in, balancing adaptation and reinvention during implementation, provide greater detail on implementation, and de-emphasize pilot testing. Not surprisingly, staff buy-in, or openness, to the innovation was critical to agency experience with a DEBI, suggesting that translation models should explicitly address these stakeholders (Israel, Schulz, Parker, & Becker, 1998). Prior work indicates that receptiveness to innovation is facilitated when a program originates within an organization (Rogers, 2005), but the DEBI process by its very nature prohibits this from taking place. Further, adoption of an EBI is frequently in response to a mandate, or what Rogers (2005) calls an authority innovative decision, that may be met with resistance by staff and management. Our findings suggest that buy-in at multiple levels within an agency (e.g., front line staff, supervisors) can reduce barriers to implementation, and this should be reflected in the translation model.

Diffusion of innovations research shows that opportunity to significantly adapt innovations facilitates translation because agency staff develops a sense of ownership (Rogers, 2005). This is a particularly difficult challenge with evidence-based interventions where adaptations must be balanced against a standard (i.e., fidelity to the original program). This has been a topic of considerable discussion throughout the DEBI process, with various sets of guidelines being provided to agencies as to how much programs can be adapted during translation. Future models that provide explicit guidance regarding the balance between adaptation and reinvention (i.e., adaptations substantial enough that that they can be considered a new program) would be beneficial to the field.

We found that agencies seldom pre-tested or pilot tested EBIs prior to implementation, although the ADAPT framework calls for these activities. In circumstances in which agencies are working with proven interventions, provided with training, and have access to detailed intervention manuals, it may not be necessary to conduct formal pilot testing. A revised framework that takes this into account would be a better reflection of the reality of the translation process.

The ADAPT framework does not provide specific guidance on implementing an innovative program, such as detailed breakdown of the steps in implementation or how to manage client recruitment and retention. Some logic models provide more step-by-step detail on implementation (e.g., MATCH; see Simons-Morton, Greene, & Gottlieb, 1995); translation models would be of greater value in practice if they included such detail because organizations that are participating in the DEBI process often have minimal experience with implementing EBIs. In particular, a model that includes a focus on recruitment and retention of hard-to-reach populations would contribute to agencies’ ability to implement group-level EBIs that were the focus of our study.

How the ADAPT framework can inform agencies

Based on our findings and on the significant body of research demonstrating the value of logic models to program planning and implementation (Green & Kreuter, 2005), we believe that some elements of the ADAPT framework are highly relevant to agencies adopting EBIs. Specifically, the assessment phase, the conceptualization of adaptation as taking place across multiple phases of translation, and the recognition of the need to revisit earlier translation activity (i.e., the feedback loop) throughout the translation process are valuable components of the current model.

The assessment phase, that point at which agencies examine the available interventions and determine goodness of fit, is a critical element of translation. A de-emphasis on assessment, as was found in our study, may contribute to agency challenges in implementing EBIs. In recognition of this, the CDC has spearheaded the development of a course offered through the National Network of STD/HIV Prevention Training Centers to assist agencies in selecting an EBI that is a good fit. Attention to ‘fit’ should help agencies choose a program that will be appropriate for the populations they serve and that their staff can successfully execute. Refinements of the ADAPT framework should retain a focus on assessment, because activity at the early stages impacts later aspects of translation.

According to the ADAPT framework, adaptations take place at multiple points during translation, as was observed in our investigation. It is important that logic models reflect this reality, rather than artificially confining adaptation to one specific point in the process as some models do.

The ‘feedback loops’ built into the ADAPT model are clearly reflective of the reality that agencies faced in implementing EBIs. Numerous agencies went through the process of getting started, having to stop and regroup, and then restarting the implementation process (e.g., staff turnover required a new hiring and training process). Revisions of ADAPT should continue to include attention to this aspect of translation.

Incorporating Organizational Models of Translation

In the prior sections we outline the elements of ADAPT framework that are not reflective of the current diffusion process, as well as noting aspects of the model that should be retained in future revisions. Additionally, Rogers’s more recent model of innovation in organizations provides a valuable theoretical perspective (2005) that can contribute to future translation models directed at EBIs.

Research on organizations demonstrates that both innovations and the organizations that adopt an innovation will likely change (Rogers, 2005); in the case of DEBIs, there should be an expectation that the evidence-based programs will be adapted, but also that the agency itself will be transformed in some way. Rogers’s organizational model proposes that agency changes that are made in order to accommodate the innovation (i.e., restructuring) take place during implementation. Future frameworks should explicitly address the need for organizational change when innovations are adopted. Research on diffusion in organizations also identifies a phase during which agency personnel gain comfort and ownership of the innovation (e.g., referred to as clarifying). This work demonstrates that it takes time for agency staff to accept an innovation, and that attempts to accelerate this process can lead to rejection of the innovation. Thus, the element of time, which is not explicitly addressed in ADAPT, should be included in future translation models. At a minimum, these two elements of Roger’s organizational model should be considered for inclusion in future translation models developed to understand diffusion of EBIs.

In summary, we recommend revisions to the ADAPT framework or the development of a new translation model that includes attention to agency buy-in, the time needed to facilitate buy-in, and the recognition that adoption of an innovation will likely lead to agency change. Additionally, a revised model should address the balance between adaptation and reinvention, while retaining the current acknowledgment that adaptation may take place throughout the translation process. Further, we recommend a more detailed outline of the implementation process. Future models should continue to include aspects of ADAPT and other logic models that have been shown to have value in practice, including assessment and the need for feedback loops that take the agency back to earlier steps when barriers are encountered.

Practical Implications

In addition to providing data of theoretical relevance, our findings also have practical implications. Specifically, the findings suggest some possible changes in funding approaches, as well as identifying factors that should facilitate translation for agencies adopting EBIs.

Our findings suggest two areas in which the translation process may benefit from modifications in funding practices. First, DEBI-specific training is critical to good translation, and is required by the CDC. It is important that training continue to be available to agencies as the DEBI process unfolds. It is recommended that the CDC consider a second training, a ‘booster’ of sorts, to ensure better fidelity to the adopted program. This type of training may also facilitate the progression of EBIs to the point of sustainability, thus ensuring that prevention programs continue over time.

Second, as noted earlier, agencies sometimes were ill informed about the EBI that they had selected because the assessment process was truncated. A change in funding strategies has potential to improve this (also see Gandelman, DeSantis, & Rietmeijer, 2006; McKleroy et al., 2006). For example, a two-phase funding mechanism could be instituted in which agencies would initially be funded to conduct a needs assessment and select an EBI. Once this process was completed, agencies could apply for a second phase of funding to adapt and implement the selected EBI. The separation of translation activity into two discrete steps might also facilitate more systematic adaptation in the translation process. If funding opportunities expressly built in support (e.g., time, resources) for assessment and possible adaptation, agencies would be in a better position to systematically adapt the intervention.

Finally, our findings point to several sets of factors that enhanced the translation of interventions into practice. It was clear that training and supervision are critical to success, and that appropriately trained staff and low staff turnover also facilitate implementation. Agencies that had strong networks and those that had integrated programs were more successful with recruitment and retention of clients for the EBI process.

Limitations

Qualitative investigations typically focus on a small sample with the goal of providing in- depth information; in this study our goal was to enhance our understanding of the translation process through the lens of the ADAPT framework. Although we met these goals, there are limits to our work. Our study examined the experiences of six agencies in California that were funded to deliver one of three group-level EBIs that target high-risk populations. As with most qualitative research, the findings may not generalize to other geographical areas or other types of EBIs (e.g., community-level interventions). Our data are based on interviews with Executive Directors and implementers, two important positions within agencies. However, we do not include the clients’ perspective. Social desirability may have influenced participants’ reports, although our findings suggest that many participants willingly discussed challenges with EBIs.

Conclusions

The findings from this study provide insight into the process that community-based agencies undergo in adopting group-level EBIs and illustrate how agency experiences can inform logic models that guide the translation process. We also identified relevant dimensions of existing models, including the ADAPT framework and Roger’s diffusion of innovations in organizations, that have value for agencies that are translating EBIs to practice. The theoretical and practical implications of our findings contribute to our understanding of translation to practice in the context of group-level DEBIs.