Exploring Implementation and Fidelity of Evidence-Based Behavioral Interventions for HIV Prevention: Lessons Learned From the Focus on Kids Diffusion Case Study

Abstract

Evidence-based interventions (EBIs) are used in public health to prevent HIV infection among youth and other groups. EBIs include core elements, features that are thought to be responsible for the efficacy of interventions. The authors evaluate experiences of organizations that adopted an HIV-prevention EBI, Focus on Kids (FOK), and their fidelity to the intervention’s eight core elements. A cross-sectional telephone survey was administered to 34 staff members from organizations that had previously implemented FOK. Questions assessed how the organization adhered to, adapted, dropped, or altered the intervention. None of the organizations implemented all eight core elements. This study underscores the importance for HIV intervention researchers to clearly identify and describe core elements. More effort is needed to reflect the constraints practitioners face in nonresearch settings. To ensure intervention effectiveness, additional research and technical assistance are needed to help organizations implement HIV prevention EBIs with fidelity.

New estimates suggest that 15% to 30% of all HIV cases occur among individuals younger than 25 years, in particular, African American and Latino youth (Morris et al., 2006). Evidence-based interventions (EBIs) for HIV prevention have been increasingly identified by both federal and private agencies and disseminated to frontline providers as a method of keeping youth healthy. Several agencies have cataloged efficacious HIV prevention EBIs for youth (Card, 2001; Centers for Disease Control and Prevention [CDC], 2006; Kirby, Laris, & Rolleri, 2006). Although the criteria agencies use to determine efficacy of the EBI vary in rigor, their purpose is to aid practitioners in selecting programs for replication.

Some agencies have further facilitated the translation of efficacious programs by packaging, training, and offering capacity building opportunities related to HIV prevention EBIs to service providers. The CDC has a three-phase process for identifying, packaging, and disseminating EBIs (Collins, Harshbarger, Sawyer, & Hamdallah, 2006; Eke, Neumann, Wilkes, & Jones, 2006; Lyles, Crepaz, Herbst, Kay, & HIV/AIDS Prevention Research Synthesis Team, 2006). Other programs that package and disseminate HIV prevention interventions include Program Archive on Sexuality, Health, and Adolescence (Card, 2001) and the Resource Center for Adolescent Pregnancy Prevention (Education, Training, and Research Associates [ETR], 2006).

The interest in identifying and disseminating HIV prevention EBIs for young people reflects the recognition of both the intensive resources (time and finances) required to develop and evaluate these interventions and the importance of disseminating those that have demonstrated efficacy in changing behaviors to reduce HIV infection. Yet questions remain about how and to what extent EBIs are adapted when community or state organizations adopt and implement these interventions. Kelly et al. (2000) note the following:

In mechanical engineering, it is customary to test a prototype with a sample of consumers and to study carefully how they use the product, including uses that were not specifically intended. By determining variations in how consumers will use the product, it is possible to specify the limits within which the product will work and then to reengineer the prototype so that it will work well in the real world (as opposed to a product development laboratory). Similarly, it is important to study how behavioral HIV risk reduction interventions will be used by community providers of HIV prevention services so that interventions can be designed to remain effective under real world conditions. (p. 96)

In this article, we describe the extent to which Focus on Kids (FOK), an efficacious HIV prevention program, was changed when adopted by organizations and implemented in community settings with different target audiences. FOK has been identified as an EBI and has been disseminated both nationally and internationally.

BACKGROUND ON FOCUS ON KIDS

Development and Evaluation of the FOK Intervention

FOK was developed with funding from the National Institute of Mental Health and the National Institute of Child Health and Human Development; funding for the project began in 1990 and continued through 2002. The intervention was developed, implemented, and evaluated in Baltimore, Maryland, for use with urban, low-income African American adolescents in their early to mid-teens.

FOK is based on protection motivation theory (PMT), which is organized around the processes of threat appraisal and coping appraisal, postulated to be responses to possible threats (R. W. Rogers, 1983). The threat is the perception of the negative outcomes that may occur if one engages in a high-risk behavior. Threat appraisal, or consideration of engaging in the risk behavior, is mediated by a balance between perceived severity of and vulnerability to the adverse consequences of engaging in the risk behavior and the intrinsic and extrinsic rewards that accompany the behavior (personal pleasure and social approval, respectively). Coping appraisal, or consideration of the protective behavior, is negotiated by balancing the response efficacy (perceived likelihood that the protective behavior will reduce negative consequences) and self-efficacy (belief that the individual can complete the protective behavior) with the response cost (barriers) of completing the protective behavior. Cultural and family influences affect the constructs in both processes. These two processes combine to form protection motivation, the intention to respond to a potential threat with either a risky or protective behavior (R. W. Rogers, 1983).

The results of the research evaluating FOK are discussed in depth elsewhere (Stanton et al., 1996). Briefly, the percentage of youth reporting condom use during last episode of intercourse was significantly greater among sexually active youth who received the intervention compared with control youth (85% vs. 61%; p < .05) at 6 months postintervention. By 12 months, the intervention effects had decayed and there were no group differences in condom use. The pattern of improved behavior among intervention youth at 6 months with a subsequent regression to baseline behavior at 12 months was apparent within various demographic subgroups (boys, older youth, and sexually experienced youth). In a later study, the addition of a 2-hr, video-based parental intervention broadened the intervention impact at 24 months postintervention.Youth receiving parental intervention and FOK compared with youth who received only FOK were less likely to use marijuana or other illicit drugs (18.3% vs. 26.8% and 1.4% vs. 5.6%, respectively; p < .05) and were more likely to ask sexual partners about past condom use (77.9% vs. 64.9%; p < .05; Stanton et al., 2004).

Program Dissemination

FOK was identified by CDC’s Prevention Research Synthesis project (PRS; CDC, 2006) as an intervention with evidence of effectiveness. Subsequently, ETR, a national publisher of educational curricula, began identifying curricula having credible evidence of reducing HIV risk behaviors among youth (ETR, 2006). FOK met both the program and evaluation criteria required by ETR. These criteria include being a theory-based educational program for middle and high school age youth demonstrated to be efficacious in reducing health-related risky behavior. The intervention also must use multiple learning activities and strategies that are factually correct, are of sufficient duration, and provide opportunities for building skills.

In keeping with diffusion theory (E. M. Rogers, 1995), ETR prepared FOK for national dissemination by working with the program developers to create a facilitators’ guide and materials for training trainers and master trainers. These products were used in national training workshops conducted by ETR and the developers.

Core Elements

Components of innovations can be divided into two categories: core elements or discretionary (optional) components. Core elements are required elements that embody the theory and internal logic of the intervention and most likely produce the intervention’s main effects (Kelly et al., 2000; McKleroy et al., 2006). Discretionary components can be deleted or changed without having an impact on the desired outcome (Center for Substance Abuse Prevention [CSAP], 2002). Identifying the core components of effective programs is repeatedly cited as a fundamental step in balancing fidelity and reinvention (CSAP, 2002; Kelly et al., 2000). FOK along with the parental companion intervention Informed Parents and Children Together (Stanton et al., 2004; Wu et al. 2003), has also been identified as an intervention with evidence of effectiveness by CDC’s PRS Project (CDC, 2006; Lyles et al., 2007). As a result, CDC’s Division of HIV/AIDS Prevention is preparing FOK and the supplementary parental intervention for further dissemination. For this dissemination effort, CDC and the developers collaboratively identified eight core elements for FOK: (a) delivering intervention to youth in community-based settings; (b) using two skilled facilitators to implement the youth group sessions; (c) using friendship groups to strengthen peer support; (d) using culturally appropriate interactive activities that have been proven to be effective learning strategies, such as games, role plays, and community projects, to help youth capture the important lessons of the constructs in the theory; (e) including a family tree to contextualize and personalize abstract concepts, such as decision making and risk assessment; (f) enabling participants to learn and practice a decision-making model, for example, SODA (stop, options, decision, action); (g) training participants in assertive communication and refusal skills specifically related to negotiation of abstinence or safer sex behaviors; and (h) teaching youth proper condom use skills.

RELATIONSHIP TO DIFFUSION THEORY

In the current study, we examined the process of adaptation/reinvention that occurred during the dissemination of FOK. Reinvention and adaptation, terms that are often used interchangeably, emerged from diffusion theory (E. M. Rogers, 1995). Diffusion theory explores the process of an idea, program, or practice that is perceived as new and the channels through which it is communicated over time among members of a social system (E. M. Rogers, 1995). Rogers defined reinvention as “the degree to which an innovation is changed or modified by a user in the process of its adoption and implementation” (E. M. Rogers, 1995, p. 17). McKleroy et al. (2006) narrowed the definition of adaptation as follows:

The process of modifying an intervention without competing with or contradicting its core elements or internal logic. An intervention is modified to fit the cultural context in which the intervention will take place, individual determinants of risk behaviors of the target population, and the unique circumstances of the agency and other stakeholders, but the core elements and internal logic are not changed. (p. 62)

McKleroy et al. (2006) further distinguish adaptation from reinvention, defining reinvention as the process of changing an intervention without maintaining fidelity to the core elements. For the purpose of this article, adaptation and reinvention are viewed along a continuum of fidelity to the core elements. The term modifications is used to include any types of revisions to the implementation logistics or to the curriculum content, such as deletions (activities were not completed at all), changes (activities were modified from the outlined procedures), and additions (new activities were added).

METHOD

In the fall of 2002, a cross-sectional telephone survey was conducted with service providers working in organizations that had implemented the FOK curriculum. The research was determined to be exempt by the Institutional Review Board at University of Maryland School of Medicine and University of Maryland, College Park.

Selection of Participants and Inclusion Criteria

Using snowball sampling (Babbie, 1982), health educators who participated in the FOK trainings assisted in identifying other organizations that had been trained in or that had implemented the FOK curriculum. Organizations known to have been trained or to have implemented FOK were contacted. Eligibility to be a study participant included responsibility for conducting FOK in the past year, knowledge of the rationale behind curriculum changes, and competence in English (FOK groups did not have to be conducted in English provided that the manager was able to use English to answer the survey). A minimum of 7 of the 53 activities that form the FOK curriculum and operationalize the content core elements (vide infra) must have been implemented. A subset of these activities make up each of the content core elements discussed below. The researcher struggled with the number of activities from the curriculum an organization needed to complete to be a part of the study. Because many of the activities are components of other curricula and are quite popular, it was not desirable to set the limit too low because only completing one or two activities could not be equated with adopting the curriculum. Setting the criteria too high would cause a potential loss of a large group of adopters who only used a small number of the activities from the curriculum. A minimum of 7 activities from the curriculum was decided as a sufficient number to distinguish between those organizations not necessarily adopting the curriculum and still low enough to determine the full range of adopters. To ensure all organizations were equally weighted, only one individual from each organization was interviewed one time. Participants who had used multiple versions of the FOK were asked to answer the survey for the last version they implemented to increase recall clarity.

Measures

The Reinvention Measurement Instrument (RMI) was developed to assess the nature of and extent to which modifications occurred during diffusion of the FOK program. The RMI had 113 items and was developed through a review of similar existing instruments, assistance from a panel of experts, and a pilot test. The RMI included sections on (a) demographic characteristics of the new target audiences, (b) characteristics of implementing organizations, (c) modifications to implementation logistics, (d) modifications to curriculum activities (deletions, additions, and changes to the activities), and (e) reasons cited for modifications. Question format included multiple-choice questions, 5-point Likert-type scales, and open-ended questions.

The RMI measured fidelity to the individual activities. For each activity, the following were assessed: whether the activity was (a) conducted as written in the curriculum, (b) conducted as written in the curriculum but in a different order, (c) not conducted at all, or (d) conducted with changes. If an activity was changed, the interviewer asked the participants to describe the changes. Descriptions of new activities were collected. The participants were also asked, from a list of prepared reasons, to attribute all reasons for modifications (deletions, additions, and changes to the activities) that were made. The list of reasons was developed by means of a literature review and input from an expert panel about common reasons for reinvention, for example, to simplify, to expand to another risk behavior, to increase ownership/make more suitable for new target audience, to meet needs of the organization requiring the change, to update or modernize, and to adapt to time constraints (Kelly et al., 2000; E. M. Rogers, 1995). If other was chosen, the participant was asked to further explain the reason for the change. Test–retest reliability for the RMI was conducted with eight of the participants. All core element variables had kappa scores of .7 or above.

Data Collection

Survey administration was via the telephone by one interviewer who had been part of the original development and evaluation of FOK (JG). Before the interview, a copy of the survey was sent to participants to help them prepare for the interview and provide responses. Participants were asked to have their entire curriculum and other materials gathered for the interview. Prompts were included in the interviewer script to clarify questions and to facilitate respondents’ recall. Descriptions of all curriculum activities were available if respondents needed help in recalling the activities.

Interviews took an average of 45 min to complete. All interviews were taped. Participants were compensated for their time with a $10.00 gift certificate and a copy of the video Protect Your Child from AIDS (an HIV prevention video that targets parents of adolescents), which had been shown to be efficacious at broadening and sustaining the effects of the FOK program (Stanton et al., 2004; Wu et al., 2003).

Analysis

Core element variables were constructed from items in the RMI pertaining to each curriculum activity or implementation logistic that made up the core elements. Activities that were changed were evaluated by one of the authors (JG) to determine if they continued to capture the previously identified intent of the activity or construct of the theory. Several examples of changes to the parent role-play activity illustrate how this process was completed. In the original curriculum the parent role-play activity was scored as operationalizing self-efficacy, response efficacy, and response costs. One organization changed the parent role-play activity to a discussion on parent versus youth communication styles. This change did not operationalize the original constructs (self-efficacy, response efficacy, and response costs) that the parent role play was assigned and therefore was scored as not keeping fidelity to this activity. The decision to score the revised activity this way was based on a belief that a conversation about different communication styles did not give youth the opportunity to practice the skill nor the confidence that such a conversation was possible (response efficacy), that they could do it (self-efficacy), and exemplify the problems that might occur during the conversation (response costs). Another organization in a juvenile justice facility changed the role-play activity to talking to their probation officers instead of parents, to tailor it for participants. This group was given credit for operationalizing the PMT constructs (response efficacy, self-efficacy, and response costs) because the youth seemed to receive the same skills from this altered activity. To reduce bias, the author evaluating the changes did not know the identity of the organization changing activities and while scoring relied solely on the description of the changed activity. Decision rule charts were kept on decisions made for changed activities to ensure consistency of scoring across surveys. Those organizations that did not conduct one or more of the activities that formed a core element were considered not to have implemented the core element with fidelity. Similarly, those organizations that changed the implementation of one or more of the activities, so that it no longer captured the intent of the original activity, were also considered not to have implemented the core element with fidelity. Frequency distributions of fidelity to each core element and changes, deletions, and additions of activities and attributed reasons for modification were computed. Activities most frequently deleted or changed across organizations were determined, types of new activities were examined, and the most frequent reasons cited for modifications were summarized.

RESULTS

Sample Characteristics

Snowball sampling was used to identify 247 persons who had participated in the FOK training or delivered FOK to youth. We attempted to contact all 247 persons; 153 (61.9%) were successfully reached through phone, e-mail, or fax. Among these 153 individuals, 43 (28%) were determined eligible for the study and were invited to participate in the survey. Of the 43 eligible persons, 34 (79%) agreed to be interviewed.

All persons interviewed had implemented FOK in the last year and had firsthand knowledge of implementation procedures, curriculum modifications, and rationale for the modifications. Participants worked in organizations located in 5 countries: United States (27 organizations from the District of Columbia, California, Iowa, Louisiana, Maryland, Michigan, Nevada, North Carolina, South Carolina, Virginia, Washington, and West Virginia), Bahamas (3), Mexico (1), Trinidad and Tobago (3), and Vietnam (1). One program was implemented along the border of the United States and Mexico and is therefore counted twice. Although organizations in China and Namibia used FOK as a template to create new curricula, service providers in China were not fluent in English, and service providers in Namibia were unable to discern which activities were drawn from FOK.

Characteristics of the Target Audience

Of the 34 participating organizations, 22 (65%) worked with urban youth, 15 (44%) worked with rural youth, and 3 (9%) worked with suburban youth (several organizations targeted youth in more than 1 region and, therefore, the sum is greater than 34). Sixteen (47%) organizations targeted primarily African American youth, 11 (32%) targeted youth from multiple ethnic groups, 5 (15%) targeted Caribbean youth of African descent, 1 (3%) targeted Vietnamese youth, and 1 (3%) targeted European American youth. The youth ranged in age from 8 to 24 years.

General Modifications

The curriculum was translated into five languages (Vietnamese, Spanish, Chinese, and two local African languages). Eleven (32%) implementing organizations changed the FOK name (e.g., Able to Achieve, Teen for New Futures, and Teen Chatters). Innovative and varied modifications included using satellite television broadcasts to reach youth in rural settings, adding a parental component, and adding HIV testing.

Implementation of Core Elements

No sites implemented all eight core elements with fidelity. The mean number of core elements that were implemented with fidelity across organizations was 4.26 (SD = 2.23) with a range of 0 to 7. Six agencies implemented seven of the eight core elements with fidelity. Three organizations did not implement any core elements with fidelity.

Core Element One: Delivering Intervention to Youth in Community-Based Settings

FOK was originally conducted in recreation centers and other community-based settings to capture youth at the highest risk who may be chronically truant from school and who do not access services at health clinics. Of the 34 organizations that participated in the survey, 14 (41%) implemented the intervention in recreation centers, churches, or community centers and kept fidelity to the core element (Table 1). Of these 20 (58%) implemented FOK in institutional settings, for example, schools, health departments, group homes, and juvenile detention centers. One program used satellite television hook-up (the facilitators were in one location and the youth were in several different rural community locations). The 5 (15%) organizations that implemented FOK in both community and institutional settings, such as schools, were given credit for implementing this core element with fidelity. Another rationale for the community settings was a concern that many institutions, such as schools, might have difficulty in disseminating FOK in the classroom due to condom activities and long length. However, it should be noted that comparison of agencies that implemented the program in a community setting compared with those that implemented the program in an institutional setting found no significant differences in fidelity to the remaining core elements.

Table 1.

Number and Percentage of Focus on Kids Implementing Organizations That Kept Fidelity to Core Elements (N = 34)

Core Element	Fidelity to Core Element, n (%)	Dropped Core Element, n (%)
Core element one: Delivering intervention to youth in  community-based settings	14 (41.2)	20 (58.8)
Core element two: Using two skilled facilitators to  implement the youth group sessions	24 (70.6)	10 (29.4)
Core element three: Using friendship groups to strengthen  peer support	2 (5.9)	32 (94.1)
Core element four: Using culturally appropriate interactive  activities that have been proven to be effective learning  strategies, such as games, role plays, and community  projects, to help youth capture the important lessons of  the constructs in the theory	20 (58.8)	14 (41.2)
Core element five: Including a family tree to contextualize  and personalize abstract concepts, such as decision  making and risk assessment	27 (79.4)	7 (20.6)
Core element six: Enabling participants to learn and practice  a decision-making model, such as SODA (stop, options,  decision, action)	26 (76.5)	8 (23.5)
Core element seven: Training participants in assertive  communication and refusal skills specifically related to  negotiation of abstinence or safer sex behaviors	14 (41.2)	20 (58.8)
Core element eight: Teaching youth proper condom  use skills	21 (61.8)	13 (38.2)

Core Element Two: Using Two Skilled Facilitators to Implement the Youth Group Sessions

As originally evaluated, the FOK intervention was implemented with two facilitators per group. This strategy allowed facilitators to model communication, negotiation, and refusal skills for the youth. It also allowed one facilitator to deal with individual issues of a group member without disrupting the group. Although the core element was use of two facilitators, three organizations used more than two facilitators (between 3 and 5) and were considered to be implementing with fidelity because they still met the intent of the core element. Of the participating organizations, 24 (71%) retained this core element by using two or more facilitators and 10 (29%) used one facilitator.

Core Element Three: Using Friendship Groups to Strengthen Peer Support

The original FOK was implemented with naturally formed friendship groups. The rationale for using friendship groups was to influence peer norms positively. The developers posited that groups of friends who went through the program together could support each other in developing new skills and could also facilitate peer norms supporting healthy behaviors. Only two (6%) organizations specifically used naturally formed friendship groups. In the remaining 32 (94%) organizations, youth were not necessarily members of natural friendship groups, but were familiar with each other, including 8 (23%) organizations that administered FOK to classrooms of students.

Core Element Four: Using Culturally Appropriate Interactive Activities Proven as Effective Learning Strategies to Help Youth Capture the Important Constructs in the Theory

FOK included interactive games and activities, such as role plays and community projects, to help the youth capture the important lessons of the constructs in the theory. It was believed that youth would learn quicker if they enjoyed what they were doing and had fun doing it. A large number of practice exercises were included because of the belief that the more youth practice new skills the more developed the skills become. Organizations that implemented 75% of the activities in a manner that still captured the constructs of the PMT were determined to have kept fidelity to core element four (for a full discussion on how organizations were scored, see Galbraith, 2004). Of the total, 20 (59%) organizations implemented core element four with fidelity.

Table 2 shows the top five most frequently deleted activities, the number and percentage of organizations that deleted the activity, description of the activity, most frequently cited reason for deleting the activity, and number of programs citing the reason. The five most frequently deleted activities were three field assignment activities, a community project, and a puberty video.

Table 2.

Number and Percentage of Focus on Kids Implementing Organizations That Deleted Commonly Deleted Activities, Most Common Reason for Deleting Activity, and Number and Percentage of Organizations Citing This Reason for Deletion (N = 34)

Activity	Description of Activity	n (%)	Most Common Reason Cited for Deleting	n (%)
Additional  field  assignment	Youth are assigned additional  outside learning activities  that were not completed at  the end of Session 3.  Assignments include  discussing with a parent  what they have learned in  the program.	28 (82.4)	Time constraints	14 (50.0)
Review of  field  assignments	Youth share with the class  what they have learned  completing the field  assignments.	19 (55.9)	Time constraints	14 (73.7)
Field  assignments	Youth are instructed to  complete outside learning  activities, including a  condom hunt, parent  interview, and calling a  hotline.	17 (50.0)	Time constraints  and to make  more suitable  for target  audiences	10 (58.8)
Community  projects	Youth choose and implement a  community project to teach  others what they have  learned about HIV  prevention.	16 (47.1)	Time constraints	11 (68.8)
Video	Youth watch a video, What  Kids Want to Know About  Sex and Growing Up, about  puberty. A discussion is led  about the changes that  happen in the body during  puberty.	16 (47.1)	No resources	11 (68.8)

Table 3 shows the top five activities most frequently cited as changed, the descriptions of the activity and the most frequently cited change to the activity, the number of organizations that changed the activity, and the most frequent reason cited for change. Changes were considered when the activity was not conducted according to the steps outlined in the curriculum, the activity was moved to another session, the content was changed, or when the storyline of a vignette was changed (just changing names mentioned in the vignette was not considered a change). Activities that were reported as partially completed were also considered changes to the curriculum. Substitution of materials (i.e., a blackboard vs. newsprint) was not considered a change. Frequently changed activities included one designed to give youth a more realistic idea of how many of their peers were participating in risk behaviors, the family tree described in detail next, a factual lesson on contraceptives, an activity about preventing date rape, and an activity that helped youth assess the level of risk for HIV that various behaviors pose.

Table 3.

Number and Percentage of Focus on Kids Implementing Organizations That Changed Commonly Changed Activities, Most Common Reason for Changing Activity, and Number and Percentage of Organizations Citing This Reason for Change (N = 34)

Activity	Description	n (%)	Most Common Reason for Making Change	n (%)
M-n-M game	Original: Youth are asked to  estimate percentage of youth  engaged in various risk  behaviors. Youth are then  given the actual percentage.  Reasons for this difference  are discussed. Change: Used  local statistics	20 (58.8)	Make more  suitable for  target audience	16 (80.0)
Family tree	Original: Youth are given a  skeleton of a family tree and  asked to create the  circumstances of and  relationships between the  family members. Change:  Changed the names and story  to fit culture of youth	15 (44.1)	Make more  suitable for  target audience	10 (66.7)
Contraception  lesson	Original: Various forms of birth  control and their positive and  negative aspects are  discussed. Change: Added  additional methods	16 (47.1)	Update	7 (43.8)
Sex, a  decision  for two	Original: This activity uses a  story about date rape to  illustrate the terrible results  that can occur by  miscommunications. Myths  and facts of acquaintance  rape are discussed, along  with how miscommunication  can be avoided. Change:  Included alcohol/drugs or  date rape drug in story	14 (41.2)	Make more  suitable for  target audience	4 (28.6)
How risky  is it?	Original: Youth determine level  of risk for HIV on different  behaviors and are given  appropriate information  about risk for each behavior.  Change: Dropped or added  behaviors	11 (32.4)	Make more  suitable for  target audience	8 (72.7)

Core Element Five: Including A Family Tree to Contextualize and Personalize Abstract Concepts, Such as Decision Making and Risk Assessment

One unique activity of FOK is the family tree. In this activity, youth are given a skeleton of a family tree and asked to create the circumstances of and relationships between the family members. The three objectives for this activity specify that youth will be able to (a) explain that decision making occurs in a social context, (b) talk about their lives without disclosing personal information, and (c) understand that decisions made while they are young can have an impact on their future. The characters in the family are used throughout the curriculum to put decision making into a personal context.

The original family tree was culturally appropriate for an urban, African American target audience (i.e., the family was reflective of the majority of families of the youth whom the original research targeted, e.g., multigenerational, headed by a single mother, and the characters had names that were common in the community). Therefore, group leaders were given specific instructions in the facilitators’ manual and during training on modifying activity for their target audience. This guidance on how to modify the activity seemed to assist implementers in successfully adapting the activities for their own target audience without losing the objectives of the activity. A total of 27 (79%) organizations implemented the family tree with fidelity to the original activity and 15 (44%) made changes to the family tree to make the activity more appropriate to their target audience. All of these changes were deemed to be in keeping with the intent of the activity. Seven (21%) organizations eliminated the family tree activity.

Core Element Six: Enabling Participants to Learn and Practice a Decision-Making Model

Another core component of FOK that is incorporated throughout the curriculum is the SODA decision-making model. Each of the four steps is taught in a separate interactive activity by using scenarios with characters from the family tree. To implement this core element with fidelity, organizations must have completed all four activities. Twenty-six (77%) organizations implemented core element six with fidelity. Eight (23%) made changes or dropped activities, and as a result, the core element was not captured.

Core Element Seven: Training Participants in Assertive Communication and Refusal Skills Specifically Related to Negotiation of Abstinence or Safer Sex Behaviors

Youth are taught through interactive activities about effective listening and verbal and nonverbal communication. A role play allows youth to practice all the acquired communication skills. Four activities constitute this core element. All four activities had to be completed to implement core element seven with fidelity. Fourteen (41%) organizations conducted all of these activities with fidelity.

Core Element Eight: Teaching Youth Proper Condom Use Skills

FOK was designed to foster positive attitudes and norms toward consistent condom use for sexually active youth and to provide the appropriate instructions for condom use. The activities that constructed core element eight were a condom demonstration and a condom race, which allowed youth to practice using a condom. If either of the activities was not implemented or the implementation form of either of the activities was changed and no longer captured the intent of the original activity (e.g., condoms were just shown and youth did not have an opportunity to practice putting a condom on a model), the organization was not considered to be implementing the core element with fidelity. A total of 21 (62%) organizations implemented core element eight with fidelity and 13 (38%) made changes or dropped activities, and thus the intent of the core element was not captured.

New Activities

Of the total, 18 (53%) organizations added new activities, and the mean number of activities added was 3.85 (SD = 5.87). The topics of the new activities included alcohol and drug use prevention (7 organizations), sexual abuse and harassment (7), building healthy relationships (7), additional information about HIV/STD (4), cultural diversity (1), prevention of cigarette smoking (1), and HIV testing (1). The most frequent reasons cited for including additional activities were to expand to another problem and to make the activity more suitable for the target audience.

Rationale for Modifications

The most frequent reason cited across all activities among all organizations for modifications was time constraints (on average organizations cited this as a reason for changing or deleting 13 of the 53 activities in FOK). The second most frequent reason was making the activity more suitable for the target audience (reason for changing or deleting 8 of the 53 activities on average), followed by narrowing in on a topic (2 of the 53) and making modifications because of organizational policies (2 of the 53). Organizational policies requiring the change was most often cited as a reason for modifying the activities that gave information about condoms and contraception methods.

DISCUSSION

Overall fidelity to the core elements was modest. A minority of organizations implemented with fidelity two of the implementation core elements, delivery to youth in community setting (41%) and use of friendship groups (6%). A third implementation core element, using two facilitators, was more likely to be implemented with fidelity (71%). A majority of organizations implemented with fidelity four of the five intervention content core elements: activities convey PMT constructs (59%), family tree (79%), decision making (77%), and condom use skills (61%). However, the core element for communication skills was implemented with fidelity by only 41% of organizations.

Existing guidelines for balancing adaptation and fidelity include identifying the core elements of EBIs (CSAP, 2002; Kelly et al., 2000; McKleroy et al., 2006). When the FOK program was disseminated, the core elements had not been clearly described. The modest percentage of organizations in the study that implemented more than 75% of core elements with fidelity demonstrates the importance of identifying and clearly describing core elements before large-scale dissemination.

Extension Activities

The four most frequently deleted activities were all three field assignment activities and the community project, all of which could be viewed as extension activities designed for outside the group time. The finding that extension activities are frequently dropped is important for curriculum developers. Whenever possible, extension activities should not be used exclusively to develop core knowledge and skills; rather, these core components must be included during group time. Extension activities may be used to reinforce key messages. However, extension activities should be used sparingly as they seem to be challenging for frontline providers to implement.

Advanced Technology

The fifth most commonly dropped activity was the puberty video, which resulted from a lack of resources (video cost was $59.99). The video was an entertaining method of providing much factual information in a short amount of time. However, adolescent service providers are often under budgetary constraints limiting purchase of videotapes.

Another disadvantage of using a video is difficulty in modifying it for new audiences. Many cross-cultural appropriateness and transferability problems can arise when relying on a video developed for a specific community. This problem limits the generalizability to other target audiences, thus increasing the likelihood of the video being dropped and the information lost. A final problem is that videos become outdated with time, perhaps affecting their long-term effectiveness. Other advanced technologies, for example, PowerPoint presentations and CD-ROMs, present similar challenges.

In two adaptations of FOK, limitations of the video were addressed by replacing it with a short skit with characters from the family tree. The script was a part of the curriculum, and group leaders performed it. Although some organizations might have resources for videos, having a low-cost alternative might improve fidelity.

Commonly Changed Activities

The three most commonly modified activities were the M-n-M game, the family tree, and the contraception lesson. Suggestions and resources were provided in the curriculum for updating and modifying these activities. “Sex, a decision for two” and “How risky is it?” were also frequently modified to be more appropriate for new target audiences; however, no modification guidance was provided. Future editions of FOK should include modification guidance for these activities.

Potential Limitations of the Study

The study had several limitations. First, all modifications were based on self-report. Some subjectivity could have existed on participants’ interpretation of whether the activity was done exactly as it was written in the curriculum. Prompts were written into the survey; however, it was impossible to eliminate all subjectivity. There is also the possibility of bias due to the issue of social desirability. Participants were assured there were no right or wrong answers. Nineteen (56%) of those interviewed were trained by the interviewer and could have felt awkward reporting changes. However, we do not believe that this potential bias influenced our results as fidelity scores were equivalent regardless of whether or not the participant was trained by the interviewer.

The small sample size and its lack of representativeness were further limitations. Analyses of subgroups were not possible. It would have been interesting to look at the subgroup of organizations that had not been trained to determine if there were differences with this group. However, it was impossible to reach most of those who purchased the curriculum directly from ETR without training. Only three such organizations were identified, posing too small a group to explore differences. Furthermore, adaptation to only one EBI, FOK, was explored. It is possible that other EBIs would be adapted differently.

Finally, the concept of core elements itself poses a limitation. Core elements are those features of an intervention thought to be responsible for the behavior change. However, expensive and complex research designs would be necessary to determine with confidence the components of an intervention responsible for significant behavior change. As with many other EBIs, in-depth research on the core elements has not been conducted for FOK. Although we evaluate implementation of core elements in this study, we cannot determine if these components truly were responsible for the behavior change seen in the original research study.

Notwithstanding methodological limitations, this exploratory study offers important insights health educators and others working in the field can use in conducting future studies on the dissemination and adoption of research-based prevention programs.

Recommendations for Health Education Practice

The current study demonstrates the importance of identifying the core elements of the intervention during program dissemination. Harshbarger, Simmons, Collins, Sloop, and Cuelo (2006) found higher rates of fidelity to core elements in their follow-up study of the VOICES/VOCES, an HIV prevention EBI that was disseminated with established core elements. Elliott and Mihalic (2004) argue that fidelity is possible only when a substantial effort is made to build local capacity before program implementation. They endorse high-quality training and technical assistance for practitioners in the field. Our findings also suggest that such training should provide practitioners with clear explanations of the theory and rationale behind the intervention, including the mechanisms and core elements of the curriculum thought to be responsible for behavior change. Practitioners should receive guidance on modifying activities to fit the needs of new circumstances while still maintaining the intent of the activity. Whenever possible, practitioners should be encouraged to incorporate process and outcome evaluation into their implementation plan. Finally, practitioners must better understand that making substantive changes or deletions to core elements could mean the intervention may no longer yield the desired behavioral outcomes.

Although experts agree that core elements are essential for implementation fidelity, a systematic process for identification of core elements for EBIs remains undeveloped. Kelly et al. (2000) suggest a three-step process for identifying core elements of an intervention. The first is to look at the behavioral science theory. Kelly et al. state that “theories emphasize the critical role played by constructs such as information, attitudes, beliefs, intention to change, expectation about outcomes, and perceived self efficacy as determinants of behavior change that reduce risk” (p. 90). The constructs of the theory are translated into activities. These constructs are thought to be directly responsible for risk reduction and therefore the activities that operationalize these constructs should be considered core elements (Kelly et al., 2000). A second method of assessing core elements is to gain extensive experience with the intervention and feedback from participants and experienced program staff about what activities were most effective (Kelly et al., 2000). Consultation with the program developers is a necessary step to this process (CSAP, 2002). A final method for assessing core elements of interventions, that have been underutilized to date, is through controlled experiments. Presently there is limited research in this area.

Mowbray, Holter, Teague, and Bybee (2003) also suggest three methods for developing core elements or what they describe as fidelity criteria: (a) drawing from a model or characteristics that are similar across programs that have evidence of efficacy (e.g., Kirby et al., 2006, list of 17 characteristics of effective programs), (b) using expert opinion from querying experts or from the literature, and (c) conducting qualitative research of opinions about what works from users of the programs.

The Division of Reproductive Health at CDC identifies core elements by assessing three categories: content, pedagogy, and implementation. Content core elements are the essential elements of what is being taught by the intervention, which is believed to change risk behaviors. Pedagogical core elements are the essential elements of how the intervention content is taught. Finally, implementation core elements are the essential logistical characteristics of an intervention that set up a favorable learning environment (ETR & CDC, in press).

Our study suggests a possible approach to substantiating core elements. If process measures expose a core element that is consistently dropped because of the difficulty imposed by implementing it in the field, a less rigorous outcomes evaluation could be conducted to assess if the intervention remains efficacious without fidelity to the specific core element. An example from FOK is the low fidelity (6%) to implementing the intervention with groups of friends. A subsequent FOK evaluation formed groups with youth who used common community centers, the approached used by many implementing organizations. The intervention remained efficacious (Wu et al., 2003), thus providing an empirical basis for expanding the core element beyond friendship groups to also include venue-based groups. Although such an evaluation takes resources, the approach is less resource intensive than a full-scale study of all the core elements.

Even with established core elements and capacity building, real world constraints may not always allow implementation with fidelity. Dusenbury and Hansen (2004) argue that program adaptation is necessary to meet local needs. They encourage curriculum developers to simplify and redesign programs to make them appealing to practitioners and applicable to the needs of communities. Health educators face several restrictions when conducting HIV behavioral interventions for adolescents, including political and community concerns and time constraints. Developers need to better understand and address these constraints to ensure greater fidelity when frontline providers implement their interventions. The results of this study demonstrate the need for curriculum developers and disseminators to understand the environment in which HIV prevention curricula are used. Time constraints were repeatedly cited as a reason for core elements being dropped. Developers need to understand how and why implementers modify interventions to conduct rigorous outcomes evaluation on variations of the intervention that address implementation challenges (e.g., shorter versions of intervention). This type of evaluation could be used to determine if adapted interventions that are easier to implement remain efficacious in changing risk behavior.

Further research is needed to better understand the processes by which frontline providers adopt, adapt, and implement EBIs. More qualitative research is needed that allows a comprehensive view of the experiences of the service provider. Programs that provide training and capacity building for community-based organizations need to work closely with service providers to determine how to best meet their needs (Collins et al., 2006). Research is needed to delineate the relationship between fidelity to core elements and outcome efficacy. Research is also needed to look at the relationship between fidelity and outcomes with the following variables: characteristics of implementing organization, methods of implementation, and amount of training and technical assistance implementing organizations receive. Studies are also needed to determine how adaptation affects the intervention’s sustainability. Such research could aid in developing a model of adaptation that describes factors that facilitate achieving positive behavioral outcomes and sustainability.