Challenges in Replicating Interventions

Abstract

Purpose

This paper describes and reflects on an effort to document, through a set of six interventions, the process of adapting effective youth risk behavior interventions for new settings. It provides insights into how this might best be accomplished. It discusses six studies funded by NIH starting in 1999. The studies were funded in response to a Request for Applications [RFA] to replicate HIV prevention interventions for youth. Researchers were to select an HIV risk reduction intervention program shown to be effective in one adolescent population and to replicate it in a new community or different adolescent population. This was to be done while systematically documenting those processes and aspects of the intervention hypothesized to be critical to the development of community-based, culturally sensitive programs. The replication was to assess the variations necessary to gain cooperation, implement a locally feasible and meaningful intervention, and evaluate the outcomes in the new setting.

Methods

This paper lays out the rationale for this initiative and describes the goals and the approaches to adaptation of the funded researchers.

Results

The paper discusses issues relevant to all interventions, those unique to replication and to these replications in particular. It then reflects on the processes and the consequences of the adaptations. It does not address the further challenges in taking a successful intervention “to scale.”

Conclusions

Replications of effective interventions face all of the challenges of implementation design plus additional challenges of balancing fidelity to the original intervention and sensitivity to the needs of new populations.

Introduction

Interventions that are successful in promoting behavior change are those that are theory-based and specifically aimed at that behavior [1]. They are also hypothesized to be ″community-based,″ [2] and ″culturally sensitive,″ reflecting the needs, values, and cultures of the targeted individuals

Yet there is also the expectation that interventions should be replicable and able to be “scaled up” for much wider use [3,4). Issues relating to the tensions between specificity and generalizability seem to occur across a wide range of innovations [5]. ″Replication″ implies fidelity to the original, while ″community-based and culturally sensitive″ implies variation. Continuing creation of new approaches may be wasteful of scarce resources, but ‘cookbook” replications are also fraught with hazard. Therein is a challenge.

Despite consensus regarding the need for “core” elements in interventions (6), it is often unclear which elements [in design, methods of instruction, or curriculum content] are essential. A 2001 study examined a number of sexuality education interventions and found design [not curricular content] commonalities in the successful ones. Those included having a strong theoretical base, training the teachers well, delivering a clear message, providing accurate information, addressing social pressures, using teaching methods that involve students, and lasting a sufficient amount of time [7]. Yet one study of the adaptation process of Stanton’s Focus on Kids program found thirty versions of the original intervention, but few were evaluated. [8]. A nationally representative 1998 survey of the implementation of prevention programs in schools found that the quality of implementation is, in general, poor [9] and that the average prevention program included only 71% of recommended content and used half of the best-practices methods recommended.

Two more recent studies [10,11] examined curriculum content, and included several of the studies discussed in this paper Shortening programs, eliminating the condom demonstrations or implementing community-designed studies in school classrooms had a deleterious effect on program effectiveness. The common curricular contents of successful programs included: how to talk with parents, basic information about risk and avoidance of risk, personal and peer values about sexual behavior, perceptions of abstinence, condom use and sexual behaviors, and clear messages about values. Issues relating to assuring fidelity of implementation are also important in programs and in replications, though are not specifically addressed here. See [12].

Here we document and reflect on an effort to develop, through six separate intervention studies, experience with the process of adapting effective adolescent HIV prevention interventions into new settings, either of populations [e.g. age, ethnicity, race] or locations [e.g. urban, rural, school type, after school program]. The work began in December of 1998 when the NIH issued a Request for Applications [RFA] to replicate community based HIV interventions for youth. Researchers were to select an HIV intervention program that had been shown to be effective in one adolescent population and to replicate it in a new community or population in the U.S. This was to be done while documenting those processes and aspects of the intervention hypothesized to be critical to the implementation of community-based, culturally sensitive programs. The chosen intervention must have been reported or accepted for publication in a refereed journal, and have demonstrated statistically significant behavior change related to at least one of these risk factors: sexual initiation, numbers or types of partners, the frequency of sexual intercourse and/or the consistent and correct use of condoms.

Here we lay out the background for this initiative, and describe the goals and the approaches to adaptation of the grantees. We discuss issues relevant to all interventions, those unique to replication and to these replications in particular. We then reflect on the processes and the consequences of the adaptations.

The theoretical basis for the need to vary interventions to achieve cultural sensitivity and community acceptance is widely accepted, but has been systematically developed and articulated less frequently [13]. The formal theories most often used in the design of intervention programs usually contain a rationale as to why a given approach is expected to work but may not address issues relating to a particular cultural context. If the potential for behavior change is influenced not only by intra-individual psychological factors but also by characteristics of the social and cultural context, such theories may provide an incomplete basis for the design of effective interventions [14). Calls for such an approach are common as are those that call for tests outside of randomized controlled trials, more relevant to “real world” conditions. These calls continue. [15,16]. On the other hand, from a marketing perspective, there are some suggestions that culture-based arguments are effective in producing initial, cursory responses, but that such arguments fail to be persuasive when the individual has time to deliberate on a decision. [17]

The Replication Projects

The six studies selected by NIH peer review for funding all provided evidence in their applications of community acceptance and willingness to cooperate in the intervention selected by the Principal Investigator. Each study took a unique approach to the problems inherent in adaptation. Table 1 provides a quick reference point and thumbnail sketches of the six projects follow.

Table 1.

Original Program Target and Replication Target Audiences

								Changes in:
	OT age	RT age	OT locatior	RT location	OT race/e	RT race/e	Program	duration	location
Stanton	MS	MS	inner city	rural	AA	white	Focus on Kids		x
Tortolero	HS	HS	school	alternative school	mix	mix	Safer Choices	x
Morrison	MS	MS	inner city	inner city	AA	mix	Focus on Kids		x
Jemmott	MS	MS	inner city	inner city	AA	Hispanic	Making a Difference
Borawski	MS	MS	inner city	suburban school	AA	mix	Be Proud Be Responsible	x	x
Zimmerma	MS	MS	inner city	rural school	mix	white	Reducing the Risk		x

Key:

MS = Middle School, ages 11-13

HS = High School, ages 14-18

AA= Africa American

Borawski replicated Jemmott’s Be Proud! Be Responsible! (BPBR) among older adolescents in five urban and suburban high schools in the middle west. [18]. The primary results paper from this study is in preparation.

Jemmott replicated his Making a Difference abstinence-based curriculum and his Making Proud Choices safer sex curriculum, with 6^th and 7^th-grade African American and Latino youth. [19] The primary results paper from this study is in preparation.

Morrison replicated Focus on Kids, [20] in the ethnically mixed Seattle, Washington area. This study compared the intervention to an attention control group that focused on career exploration. The primary results paper from this intervention is in press [21]

Stanton replicated Focus on Kids. [20] Originally developed for inner city, African American youth, the program was replicated among white, rural youth. This study used a three-part trial comparing two approaches to intervention replication. The primary outcome papers from this study are [22,23].

Tortolero replicated Safer Choices, [24,25,26] a school-based intervention in alternative schools. The primary results paper is in preparation.

Zimmerman replicated Reducing the Risk, an intervention originally designed for pregnancy prevention [27] among adolescents in rural areas and small towns in an economically depressed Appalachian region. The primary results paper is [28}

General Issues in Program Implementation

Although this group of studies was initiated to address challenges relevant to program adaptation, each had also to address the common concerns of intervention programs. These issues have been extensively discussed (see, specifically, 6,10 and the papers from each investigator about their programs]. The issues varied across locations. Briefly, the components are: community support, program fidelity and staff selection and training.

A variety of different tools were used to establish community and school support for their programs. For example, Dr. Zimmerman’s team partnered with local health departments, which were already teaching in area high schools. . As required by state law, for one project, in the event of a positive STD finding, adolescents - and not parents -- should be notified. School board members felt very strongly that parents should be notified. While this was not legally possible, it took a considerable amount of effort to assure school board members that the research team would work with adolescents to ensure they had the needed treatment.

A second element, critical for evaluation accuracy, is monitoring to be sure that the program is implemented as designed (e.g., program fidelity]. Each project was able to do this. The projects utilized well-tested field methods in this process.

A third linked element is staff selection and training. Experienced facilitators may be accustomed to having a voice in the design and implementation of programs and to customizing the interventions based upon their understanding of the audience. Although the original programs replicated in this initiative all used facilitators hired by the research groups, some of the replication studies trained on-site teachers and nurses to deliver the curriculum.

Challenges in the Adaptation Process

There are excellent articles on the technology transfer processes necessary to get a proven product into the field. See the special issue of the Journal of AIDS Education and Prevention in 2000 [volume 12, supplement A]. However, those studies looked for “fit” between the original program and the new venue, rather than changing the intervention to match the community. One [6] did call for a careful examination of the “core elements” of a program when replicating it, though the author did not suggest what those core elements might be [for that, see 11]. There are many reasons to look for the “best fitting” intervention rather than working on adaptation but there are challenges in defining “fit.” [See 29l. Here we discuss the processes followed by these NIH funded researchers.

Geography and existing community structures

Adolescents’ lives differ depending on the type of area in which they live. Rural and suburban youth are much more geographically dispersed than are urban youth. This dispersion complicates efforts to conduct programs outside the regular school day. Without transportation, students may not be able to participate in after school or weekend programs when the venue is far from home. Some programs provided transportation for youth.

Geography can also affect adolescents’ social lives. In rural areas where there are no local parks or malls at which to hang out, much of the local activity occurs at school. References to the mall or to local hangouts may be alien to suburban or rural youth, and, references to football games to urban youth. This can have implications for program effects because less diffusion of new knowledge and values occurs when students do not interact outside the program setting.

Programs in new places may differ in their needs for timing, number, and length of program sessions. They need to fit in to ongoing organizational structures and schedules. One project developed a more intense program model to counteract frequent student absences and high drop-out rates. Modifying the curriculum to run in 45-minute sessions, twice weekly, for eight to ten weeks increased the likelihood of exposing students to the whole program. Another project created alternative length versions of the curriculum to accommodate the schedules of summer programs and other community activities. All attempted to maintain content coverage and ensure that all core program elements were covered consistently with dose equivalence.

Community Cultures and Support

Obtaining local support for a program is crucial to successful implementation, even when local stakeholders have provided theoretical support for a known program contingent upon funding, as happened with the Stanton program. While the purpose of the replication projects was to adapt interventions in culturally sensitive ways, the very fact of approaching a new community with an HIV prevention intervention and the concomitant funding set up the potential for “mismatch”. Several of these replication projects engaged their communities’ support without difficulty, while others did encounter problems. Three types of mismatch occurred.

First, not all communities were open to HIV prevention for youth, although all had agreed in advance to participate in the research. This was less the case in alternative schools, where administrators and teachers understood the need for HIV prevention programming among their students and welcomed the research team than in a suburban setting. Areas where schools and community agencies were not already open to and/or providing adolescent sexuality education posed a challenge for investigators, however. One project worked hard to engage their community, but was unable to utilize a number of program elements from the chosen intervention. [22]

A second issue was the openness of communities to the advice of “outside experts.” For several projects, the original program’s proven effectiveness was an important selling point. Many schools were happy to receive pre-developed, tested curriculum materials, and school administrators valued both the provision of HIV prevention programming for their students as well as training for their teachers and nurses. However, in some rural settings the fact that the intervention had been designed by “outsiders for outsiders” continued to be a significant deterrent to its utilization.

A third issue was existing or changing community priorities. In one location, the community was just not interested or excited about the program. They felt that other needs were more pressing and relevant to their youth, most specifically, issues of alcohol use.

Community Cultures and Program Content

Clearly, using family or communication examples that clash with cultural norms is counterproductive. Different norms exist around socializing, dating, and sexual behavior among different ethnic and cultural groups. Concepts and practices related to family relations, religion, and work also may differ across and within communities. These differences all invite some degree of program adaptation. Community and youth advisory boards can be essential in determining program areas where change may be necessary - as well as in assuring buy-in to the program. The projects took different approaches to adapting their programs, and the degree and levels of adaptation varied. Each approach is summarized below, and examples of the specific changes made are described.

Approaches

Dr. Jemmott’s team adapted his own curricula to Latino culture. They took into account how Latino culture supports HIV/STD protective and risk behavior. They sought to retain the core elements of the curricula and to incorporate Latino community and adolescent input throughout the process. This was accomplished through extensive review of existing research, preliminary studies of the relation of cultural values to sexual behavior, the use of focus groups with adolescents and parents of adolescents, pilot tests of the interventions, and debriefing of pilot intervention facilitators and adolescent participants.

To develop the cultural adaptation of her own program, Dr. Stanton’s team utilized community input. This team was attempting to contrast two versions of the replication [one of which was to remain “true” to the original intervention except for minor alterations and the other was a more open process to change. However, because many communities were not comfortable with some of the “standard” elements, a number of activities, particularly those dealing with condoms, were removed from the unchanged program, perhaps contributing to the weakened intervention effect in this new setting.

In adapting Safer Choices for an alternative school population, Dr. Tortolero’s team utilized intervention mapping, [30] which considers the theoretical foundations most likely to influence behavior change and the likely variation needed to address cultural or other relevant issues for the new population. They found that involving members of the original development team provided important insights on which elements of the program should be retained or revised for the new population.

In adapting the ETR-developed program to meet the needs of rural teens, Dr. Zimmerman and his team reviewed the literature and assessed each element of the curriculum. Those elements deemed essential included behavior-related knowledge, self-efficacy, personalization of risk and adverse effects, and peer norms. Input from the program designer was invaluable and helped the team remain focused on important content issues when developing new modes of presentation.

Dr. Morrison’s team was committed to keeping all elements of the Stanton program. They utilized focus groups, community input, and project staff experience to change only those elements that were inappropriate or unworkable. Feedback from the original project staff was valuable in supporting team decisions to retain elements intact and was supportive of most changes.

Finally, Dr. Borawski’s team decided prior to the study that they would not change the Jemmott curriculum in order to examine its transferability to students from different backgrounds. Because the community was then engaged in a heated debate on sex education many schools were reluctant to include one of the more extensive condom discussions in the curriculum. Removing this one activity allowed the team to retain all other activities involving discussion, demonstration and role playing with condoms.

Modifications in the Programs

There were a number of areas in which investigators made program changes to improve consonance with community cultures. Each program had unique needs, which included:

Preferred communication styles

One project removed labels from the source program which suggested that indirect ways of communicating were generally incorrect or inferior. For traditional Asian Americans and Asian immigrants, less direct styles of communication are the norm and appropriate.

Machismo

One project framed HIV risk and protective behavior through an initial activity that introduced participants to attitudes and behaviors that were consistent with the many positive aspects of Latino cultural values. For example, participants were introduced to the positive aspects of the cultural norm of machismo. Such behaviors included the responsibility to protect one’s family and partner. Thus, condom use was viewed as a behavior that was consistent with this dominant value. Throughout the curriculum, preventive behavior was framed in a manner that was consistent with these values.

Family Composition & Family Structure

One program modified a family tree activity, which was designed to get youth buy-in by creating a family system with which youth participants could identify. Their adaptation was based on two types of family systems, one in which parents were born in the United States and one in which parents were immigrants. Many details were then supplied by the group, which allowed this multi-cultural audience to invent a family reflective of their own community norms.

Community Culture and Program Implementation

Cultural differences among target populations also affected the strategies used in implementing programs. For example, one project team working in a multicultural setting learned that youth were not correctly describing the program to their parents, so they hired consent gatherers, fluent in the parents’ language, to explain the program. These staff members were also helpful in tracking down families for follow up screening.

In another project, research staff as well as facilitators and project assistants who worked on previous research studies with the investigative team were provided with an overview of cultural characteristics and norms for communication common among Latinos. In addition, it was necessary to ensure that key staff were bilingual. This was especially important in recruitment and retention follow-up efforts.

One project included African immigrant youth, a number of whom were Muslim. To include these teens, project staff needed to learn about cultural and religious practices of their families and incorporate their perspective into the intervention. This included adjusting the schedule to accommodate religious holidays and not serving meat (e.g., pepperoni pizza) as a snack.

Adapting Programs to Address Emergent Needs

Another challenge that led to changes in the original programs was the discovery of specific problems within the new target populations. In many cases, programs were adapted to address those needs. Examples include:

Addressing relationship power issues

In its needs assessment, one project learned that many girls dated older men and the age gap made it more difficult to negotiate sexual relationships and condom use; a high prevalence of dating violence existed; and the language their population used to describe sexual relations and the opposite sex underscored a level of disrespect and depersonalization associated with sexual relationships. To address these issues, new strategies were added on recognizing healthy and unhealthy relationships, and an additional lesson was included on avoiding or exiting high-risk situations.

Addressing future-orientation, self-efficacy, and attitudes

One project found a number of additional self-esteem related risk factors among their population, so they developed a video, “Making Choices” to address determinants such as self-efficacy, power in relationships, future orientation, perceived risk, and attitudes by using modeling and increasing the opportunity for vicarious learning experiences.

Addressing social/sexual experiences & HIV knowledge

One project’s baseline data suggested that its suburban population was less sexually experienced and knowledgeable than those in the original trial, although of the same chronological age. The investigators adapted the curricula by incorporating more basic information about puberty, adolescent development, and sexuality and by incorporating more scripted role-plays, which are more appropriate for a less sexually experienced population. Another population was much more experienced with a wider range of risk behaviors, than the original group, so the curriculum was broadened.

Narrowing or expanding the intervention focus

One project’s original intervention also focused on risks other than HIV, including substance use, drug sales, and gang involvement, but these were not major issues for the study participants. So those portions of the curriculum were removed. Another project faced the opposite issue. Their study population had many more risk factors than addressed in the original intervention and required additional lessons to meet their needs. In a third, the community did not feel that alcohol abuse, an issue of great concern, was adequately addressed in the source intervention, and that other sexual risks were over-emphasized. So the adapted intervention included alcohol use issues.

Conclusions

Many elements must come together for a prevention program to be replicated and to produce behavior change in a different environment. In replicating their programs, investigators encountered both structural and cultural challenges. Some changes made to the original programs were theory-driven, to bring the examples closer to home, such as in vignettes selecting names of youth from the local community or making references to single as well as two parent families. Some changes were driven by local constraints and realities, within the theoretical bounds of the intervention. Some were driven by the discovery of unmet needs in the target population. All investigators, even those who had originally developed the program being replicated, struggled with the process of adaptation. There is some suggestion in the literature on program innovation that “additions” to a program do not eviscerate its effectiveness, but that “adaptations” or “alterations” may [31].

When a curriculum has a number of elements, combined in a variety of ways, it becomes difficult to evaluate which elements are actually essential to the success of a program. The “core” curricular elements [e.g. a condom use demonstration] are not and perhaps could not be presented or evaluated as stand-alone interventions. Measurement of such elements in the absence of multiple trials with random assignment to varying conditions is close to impossible. Thus, the impact of the various specific changes made in these interventions may range from negligible to serious. However, only one project, so far, has found measurable behavioral change. That project, Dr. Zimmerman’s replication of Reducing the Risk, shared many of the same features as the other programs - community and facilitator support, adaptation to the study population, and measured fidelity to the core elements of the original program. Some of the projects are still analyzing their outcome data. Three of the programs [Stanton, Morrison, Borawski] have not found significant changes in behavior, although Stanton’s program did find positive changes in perceptions and intentions. Borawski’s and Morrison’s results have not been published, so these findings are as yet tentative.

There may be a number of issues beyond the effectiveness of the program that need to be considered. For example, moving a program that was successful in a population with very high rates of sexual initiation and that found a strong impact on condom use, [20] to another, Seattle, where sexual initiation rates were much lower, resulted in no measurable impact of the program on condom use. The proportion of youth who initiated sex in either intervention or control groups was simply too small for a statistically significant change. The program was replicated, but upon reflection the focus of the original intervention program had little program relevance in the new situation. The Tortolero program, which had to ramp up its focus on risk reduction, is another case where the original match was less than ideal, but changes were possible. This reflects back on the need to improve the match of program with audience. It would seem that cultural sensitivity may have some influence at the margins, but that intervention delivery, training of facilitators and inclusion of essential curricular components continues to be more powerful influences upon resulting behavior change.

In the best of situations, risk reduction interventions have small positive impact on sustained behavior change. [cf 32]. Research has shown fairly clearly the curricular and programmatic elements necessary for such change. It is not clear whether studies isolating single curricular elements would be able to determine the “essential” ones for such change, given the interactions of one with another and with other aspects of the target audience.

Furthermore, the larger, relatively immutable, social context influences individual behaviors. In the US, media and popular culture may play a role. [26, 33, 34]. An ecological perspective builds from the individual through social relationships to living conditions, neighborhoods, institutions and policies [35] Researchers have listed a range of these extra-individual factors relevant to HIV/AIDS, including but not limited to neighborhood deprivation, the acquisition of social capital, and peoples’ social and sexual networks [see 36,37]. This issue has been carefully discussed in the arena of early childcare [38]. One could argue for the need for HIV/AIDS researchers and program people to develop a broader perspective. We as health providers, educators and researchers must consider how reasonable is it to expect cognitive individual-level interventions to have large and lasting effects. Nonetheless it is essential to continue to work producing small but widespread change over time, as these can build to normative shifts. Consider the changes in seat belt use or smoking.