Implementing Evidence-Based Programs: A Four-Step Protocol for Assuring Replication With Fidelity

Abstract

Health care providers are challenged to replicate evidence-based programs in their communities. These programs may be adapted to fit new communities, but the key components must be delivered with fidelity. This article describes a four-step fidelity assurance protocol developed by the Hawai‘i’s Healthy Aging Partnership as it adapted and replicated evidence-based health promotion programs for Hawai‘i’s older adults. The four steps are the following: (a) deconstruct the program into its components and prepare a step-by-step plan for program replication; (b) identify agencies ready to replicate the program, and sponsor excellent training to local staff who will deliver and coordinate it; (c) monitor the fidelity of program delivery using standardized checklists; and (d) track participant outcomes to assure achievement of expected outcomes. The protocol is illustrated with examples from Hawai‘i’s Healthy Aging Partnership’s experience replicating EnhanceFitness, a senior exercise program. This protocol is transferrable to other communities wanting to adapt and replicate evidence-based, public health programs.

INTRODUCTION

Nationally, more than 50% of older adults have one or more chronic conditions (Centers for Disease Control and Prevention & Merck Company Foundation, 2007), and few meet the recommendations for physical activity (Kruger, Carlson, & Buchner, 2007). Chronic conditions are associated with functional disabilities, poor quality of life for older adults, and economic stress for family caregivers, health care systems, and government (Agency for Healthcare Research and Quality, 2009). Although chronic conditions are rarely cured completely, research consistently validates the benefits of a healthy lifestyle. Several health promotion programs have been proven, though controlled trials, to effectively enhance fitness, increase self-management skills, and decrease health care costs (Altpeter, Bryant, Schneider, & Whitelaw, 2006).

Public health providers are being asked to replicate programs proven to work in other communities, rather than trying to “reinvent the wheel” (Brownson, Baker, Leet, Gillespie, & True, 2011). Since 2003, for example, the U.S. Administration on Aging (AoA) has promoted the replication of evidence-based health promotion programs for older adults in new communities, supporting collaborations among agencies in local networks of planning and coordinating agencies, elder care service providers, health care professionals, and academics (AoA, 2010). This nationwide effort, cosponsored by the National Council on Aging (NCOA), has made significant progress in expanding opportunities for older adults to participate in evidence-based health promotion programs, such as Stanford’s Chronic Disease Self-Management Program® (CDSMP; to empower people to take control of their health), EnhanceFitness® (to promote strength, balance, and flexibility), A Matter of Balance® (to reduce risk of falling), and Healthy IDEAS® (to control depression; AoA, 2011). For example, CDSMP is now offered to older adults in 46 states and Puerto Rico, and an online version of the program is also available nationwide (NCOA, 2011).

Successful replication of an evidence-based program implies that the content and quality of the original program are maintained, so that the outcomes of the replication are as good as those found in the controlled testing of the original program. States were assisted by the NCOA’s Center for Healthy Aging in replicating evidence-based health promotion programs for older adults, recommending the use of the RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) framework for planning and evaluation (Bryant, Altpeter, & Whitelaw, 2006; Glasgow, Lichtenstein, & Marcus, 2003; Green & Glasgow, 2006) and the Track Change Tool adapted by the NCOA (n.d.) from Peterson (2003).

Local networks, however, were responsible for the nitty-gritty work of assuring that programs were replicated with fidelity, yet adapted to “fit” their communities. In Hawai‘i, this job fell to the Hawai‘i Healthy Aging Partnership (HHAP). Formed in 2003, HHAP is dedicated to increasing access to health promotion programs among Hawai‘i seniors with chronic conditions (Tomioka, Braun, Compton, & Tanoue, 2012; Tomioka, Sugihara, & Braun, 2012). HHAP members include professionals from government offices for aging and public health, elder care agencies, and the university. The partnership worked together to train its members in evidence-based programming (2003–2005) and to secure federal and local grants to improve elder health by replicating evidence-based health promotion programs (2006 to present). The purpose of this article is to share the four-step protocol used by HHAP as we replicated evidence-based programs under this AoA/NCOA initiative. We use our experience with replicating EnhanceFitness to illustrate the steps of the protocol. In line with the journal’s mission, this article demonstrates how assessment, planning, and evaluation techniques can be applied by and for practitioners to strengthen their ability to successfully adapt, replicate, and evaluate evidence-based programs.

EnhanceFitness is a group exercise program developed by University of Washington and Senior Services of Seattle to increase the strength, flexibility, and balance of older adults through structured, 1-hour, group exercise sessions of stretching, low-impact aerobics, balance, and strength training (Senior Services, 2007). Classes are held three times a week. They are led by Certified Fitness Instructors who have received specialized training in EnhancedFitness. These Instructors are trained and monitored by Master Trainers, who have received advanced training from Senior Services. Several studies have demonstrated the effectiveness of EnhanceFitness in improving physical performance of older adults in different communities and ethnic groups in the continental United States (Belza et al., 2006; Belza, Snyder, Thompson, LoGerfo, 2010; Tomioka, Sugihara, et al., 2012; Wallace et al., 1998). HHAP needed to adapt the program to Hawai‘i, where two thirds of older adults are of Asian and/or Pacific Islander ancestry (Hawai‘i Executive Office on Aging, 2006; U.S. Census Bureau, 2010).

METHOD: THE FOUR-STEP PROTOCOL

HHAP’s protocol include four steps: (a) deconstruct the program into its components, and prepare a step-by-step plan for program replication; (b) identify agencies ready to replicate the program, and sponsor excellent training to local staff who will deliver and coordinate it; (c) monitor the fidelity of program delivery using standardized checklists; and (d) track program participant outcomes to assure achievement of expected outcomes. As shown in Table 1, different tools (described below) are used in each step of the protocol.

TABLE 1.

Four-Step Protocol and the Tools Used in Each Step

Step	Tools
1. Deconstruct the program into its components, and prepare a step-by-step plan for program replication.	Track Change Tool Adaptation Traffic Light Implementation plan
2. Identify agencies ready to replicate the program, and sponsor excellent training to local staff that will deliver and coordinate it.	Readiness interview Mastery of knowledge, skills, and confidence to deliver program (pretest and posttest)
3. Monitor the fidelity of program delivery using standardized checklists.	Fidelity checklists
4. Track program participant outcomes to assure achievement of expected outcomes.	Participant health status (pretest and posttest) Participant satisfaction with program (posttest)

Step 1: Deconstruct Program and Prepare Implementation Plan

The Track Change tool prompts adapters to deconstruct the evidence-based program into domains, for example, marketing, recruiting enrollees, identifying staff, training, scheduling, implementation, and evaluation (NCOA, n.d.; Peterson, 2003). Adapters note how each aspect of the program was carried out in its original offering (based on the program’s implementation guide) and then give a detailed description about how each aspect would be implemented in the new community. This tool assures that adapters thoroughly plan the replication and identify possible discrepancies in implementation compared with the original program. Adding “due dates” and identifying the “responsible party” for each activity transform the Track Change Tool into a plan for action.

HHAP then used the “Adaptation Traffic Light” to judge which elements of the evidence-based program could be modified (Centers for Disease Control and Prevention, Division of Reproductive Health, & ETR Associates, n.d.). Red-light changes are those that cannot be made, such as substantially shortening the program or deleting activities. Green-light changes, such as creating a local name for the program, can be freely made. Yellow-light changes are those that can be made with caution, because there is a chance that they could decrease program effectiveness. One of the activities in the replication plan is to review potential yellow-light changes with the original developers of the evidence-based program.

Step 2: Identify Agency Readiness and Sponsor Training

We conducted a semistructured interview to assess the readiness of organizations wanting to replicate EnhanceFitness in their communities. Eight interview questions were developed from the “Self-Assessing Readiness for Implementing Evidence-based Health Promotion and Self-management Programs” tool (NCOA Center for Healthy Aging, 2007). Items prompted organizational representatives to consider positive and negative factors associated with implementing the program, including support from the organization’s management, availability of organizational resources, accessibility of technical assistance, support from other partners, and commitment to replicating the program with fidelity and using the evaluation forms.

Adapters, potential instructors, and coordinators must complete training in the evidence-based program. Given Hawai‘i’s distance from the continental United States, HHAP initially brought trainers from Senior Services of Seattle to Hawai‘i. HHAP conducted pre–post tests of trainees to gauge their knowledge, skills, and confidence to implement EnhanceFitness. Our six-item questionnaires were based on those developed by Senior Services of Seattle but were adapted and pre-tested in Hawai‘i before being used to evaluate our initial EnhanceFitness training. Responses were analyzed by paired t tests.

Step 3: Monitor Fidelity

Fidelity monitoring refers to the strategies to assess the degree of adherence to delivering the program, as well as the adapter’s competence in delivering the program (Frank, Coviak, Healy, Belza, & Casado, 2008). This step is critical in ensuring true replication of the program. HHAP members engaged in ongoing discussion about the recommended frequency of monitoring new Instructors, and this needed to be balanced against the availability of Master Trainers willing to monitor program fidelity.

Master Trainers were trained by Senior Services to use a nine-page fidelity-monitoring tool. This tool prompts Master Trainers to observe eight areas of fidelity (record keeping, overall instruction, warm-up, aerobics, cooldown, strength training, balance, and stretch) and to rate the Instructor’s delivery using three categories: above standard, meets standards, and needs improvement. Monitoring was done during the first month, fourth month, and every 4 months thereafter.

Step 4: Track Participant Outcomes

Evidence-based programs often provide tools that adapters can use to evaluate their programs. Senior Services provides such tools for EnhanceFitness, including forms for registration, physician clearance, Fitness Checks, and satisfaction (Senior Services, 2007). Fitness Checks are conducted at baseline and every 4 months. These include the following: (a) “chair stands,” in which we recorded the number of times the participant can move from sitting and standing in 30 seconds; (b) “arm curls,” in which we recorded the number of times the participant can lift a weight in 30 seconds; and (c) the “up-and-go,” in which we measured the number of seconds it took the participant to stand, travel 8 feet, go around a cone, return to the chair, and be reseated. Satisfaction with the program and confidence to exercise regularly (from 1 = not at all to 10 = totally) are measured at 4 months and annually thereafter.

Essential to this step is establishing mechanisms to regularly review data with program Instructors and coordinators. HHAP held monthly meetings, with representatives of the state, the four counties, the evaluators, and various service providers. Progress toward recruitment objectives was presented at every meeting, whereas findings from fidelity monitoring and over-time changes in participant health status were shared quarterly. Because we are part of a network of states replicating evidence-based programs, we also compared our findings against national averages.

FINDINGS: ILLUSTRATING THE PROTOCOL

Step 1: Deconstruct Program and Prepare Implementation Plan

HHAP identified how major EnhanceFitness domains were originally implemented. Our EnhanceFitness team discussed how HHAP could replicate each element within a domain or if HHAP wanted to adapt it. Then each desired change was judged against the traffic light, and the final decision about replication was made.

HHAP made several green-light (allowable) changes (Table 2). For example, because two thirds of Hawai‘i’s older adults are Asian and Pacific Islander, the team added an array of Asian and Pacific Islander ethnic groups to the registration form. In the original study, participants were recruited through senior centers. However, HHAP EnhanceFitness team wanted to offer the program at nutrition sites and recreation centers in their counties. Although EnhanceFitness has marketing materials, HHAP developed its own to reflect local culture and faces. EnhanceFitness requires music at specific tempos for specific phases of the class (e.g., warm-up, aerobic, stretching, cooldown) and provides sample music. HHAP felt that some of the music might not be attractive to our participants, so the team selected familiar music that met tempo requirements (e.g., Hawaiian music for the cooldown).

TABLE 2.

Step 1: EnhanceFitness® Elements Examination Results

Domains	As Originally Implemented	Issues	Traffic Light	As We Will Implement It
Participants	Older adults; majority were White	Hawai‘i population is two-thirds Asian and Pacific Islander.	Green	Decided to target older adults with any ethnic backgrounds
Recruitment	Senior Center newsletter and the center bulletin board	Coordinators go to nutrition sites to promote the program by using PowerPoint and distributing flyers.	Green	Providers offered EnhanceFitness to the individuals who come to nutrition sites
Marketing	Name of the program “EnhanceFitness”; brochure illustrates people from Seattle	Name OK, but materials do not feature local culture and faces.	Green	Developed local materials and recruitment presentation
Stereo/music	A variety of music at different tempo (revolutions per minute), ranging from 108 to 124 for warm-up and cooldown, to 120 to 132 for aerobics	Some of the music may not attract Hawai‘i older adults.	Green	Used same tempo but used more locally well-known music
Room setting	Indoor exercise environment with good lighting, a wooden floor	Our facilities have concrete floors and trade wind ventilation.	Yellow	Senior Services approved holding the classes in these facilities but advised monitoring of safety
Equipment	Armless, hardback chairs	Our facilities had only folding chairs, and we did not have funds to purchase new chairs.	Yellow	Senior Services cautioned that instructors cue participant posture and safety regularly during chair exercises.
Shoes	Exercise shoes	Hawai‘i older adults usually wear zori (flip-flops) or sandals; some do not own athletic shoes.	Red	Senior Services strongly recommended that we require participants to wear close-toed athletic shoes.

HHAP identified two yellow-light changes. In Seattle, EnhanceFitness is offered in indoor, temperature- controlled, exercise rooms with a carpet-free wooden floor and straight-back chairs. However, most community-based facilities in Hawai‘i are built with concrete floors with trade wind ventilation and equipped only with folding chairs. Senior Services agreed that Hawai‘i classes could be held in these facilities but encouraged HHAP to purchase straight-back chairs when funds allowed. They recommended that Instructors cue posture regularly when participants use the folding chairs, and caution them to put their weight in the middle of the chair in order to reduce risk of tipping.

From the EnhanceFitness training and manual, it was clear that red-light changes would include shortening the duration of the exercise class, eliminating phases of exercise (e.g., the warm up), and adjusting the tempo of the music to which each phase of exercise was to be led. HHAP also asked if participants could wear zori (flipflops) or sandals, as this is the most popular footwear in Hawai‘i. Senior Services deemed this a “red-light” change and strongly recommended that we require participants to wear close-toed athletic shoes to maximize safety. HHAP agreed to find resources for older adults who might need assistance with shoe purchase.

Step 2: Identify Agency Readiness and Sponsor Training

Readiness interviews were conducted after training in but prior to agency implementation of EnhanceFitness. Findings helped identify potential barriers to implementation. For example, two agencies reported difficulty with resources (e.g., they could not find qualifying Instructors in their communities) and were worried that they could only offer the program two, rather than three, times a week. HHAP members worked with these agencies to identify ways to overcome barriers. In the end, however, the two agencies reporting significant barriers were unable to sustain the program.

The initial EnhanceFitness training was held in June 2007. Seven Certified Fitness Instructors completed the EnhanceFitness Instructor training, and four completed EnhanceFitness Master Trainer training. Because Master Trainers can train other Certified Fitness Instructors in EnhanceFitness and monitor fidelity, this reduced our dependence on Senior Services. From our pre–post questionnaires, we could see that Hawai‘i trainees made significant improvements in knowledge (t = −4.95, p = .001), skills (to lead chair exercise, t = −7.29, p < .001; to track attendance, t = −5.16, p < .001), and confidence (to answer questions, t = −4.38, p < .001; to gather performance data, t = −4.72, p < .001; to help participants, t = −2.69, p = .025).

Step 3: Monitor Fidelity

As noted, fidelity monitoring was conducted by the Master Trainer during the first month and then quarterly, following checklists provided by Senior Services (2007). Initial findings suggested that new Instructors scored best on balance, cooldown, and record keeping and least well on stretching and strength training. Onsite assistance was provided by Master Trainers to help Instructors increase fidelity of delivery.

Because instructors leave, EnhanceFitness training has to be offered every year or two. In all, 31 Hawai‘i providers have been certified (7 Master Trainers and 24 Instructors). However, only three Master Trainers are currently active, and only 8 trained Instructors are offering classes regularly. Others were not interested in offering classes regularly (n = 9), were not interested in partnering with HHAP (n = 1), found full-time jobs (n = 2), retired (n = 1), moved away (n = 4), or resided in the community that stopped offering EnhanceFitness (n = 3).

Step 4: Track Participant Outcomes

Initially, HHAP implemented EnhanceFitness classes three times per week every week at four sites in Hawai‘i. EnhanceFitness classes were very attractive to Hawai‘i older adults, and new sites were opened in 2008, 2009, and 2010. By December 2011, eight different classes were offered at seven sites, and all sites have waiting lists. Characteristics of Hawai‘i participants (n = 331) are presented in Table 3, along with data from all sites (provided by Senior Services). Comparatively, greater percentages of Hawai‘i participants are Asian and/or Pacific Islander. They also appear to be older, with greater percentages of participants with diabetes and hypertension. A large amount of missing data in the national sample (>30%), however, precluded statistical testing of these differences.

TABLE 3.

Participants’ Demographic Characteristics, 2007–2011

Characteristics	Hawai‘i (N = 331), n (%)	National (N = 24,691), n (%)
Age, years	Mean = 77.7, SD = 8.48	Mean = 75.0, SD = 9.98
Female gender	301 (90.9)	17,245 (83.7)
Ethnicity
Caucasian	79 (24.0)	11,852 (48.0)
African American	0 (0)	2,555 (10.3)
Hispanic	1 (<1.0)	712 (2.9)
Asian	205 (62.3)	889 (3.6)
Native Hawai‘ian/Pacific Islander	22 (6.7)	72 (<1.0)
American Indian/Native American	1 (<1.0)	303 (1.2)
Multiracial	18 (5.5)	248 (1.9)
Other	3 (1.0)	130 (<1.0)
Missing	2 (<1.0)	7,930 (32.1)
Chronic conditions
Diabetes	79 (23.9)	2,445 (13.4)
Heart diseases	51 (15.4)	1,829 (10.0)
Hypertension	141 (42.6)	5,098 (28.0)
Arthritis	130 (39.3)	6,003 (32.9)

By December 2010, 330 Hawai‘i program participants successfully completed baseline Fitness Checks during the first week of class, and 249 (75%) participants completed the 4-month Fitness Checks (Table 4). Our findings showed significant increases in number of chair stands (from 11.49 to 13.86; p < .001) and arm curls (from 10.76 to 12.77; p < .001) and reduced time for the up-and-go test (from 8.57 to 7.66 seconds; p < .001). In comparing our findings with national data, we found that Hawai‘i participants made similar improvements in the three performance measures.

TABLE 4.

Baseline and 4-Month Fitness Check Results, 2007–2011 (Hawai‘i n = 249; National n = 1,845)

	Mean			t	df	p
	Baseline (SD)	4-Month (SD)	Change (SD)
Chair stand (repetitions)
Hawai‘i	11.49 (3.72)	13.86 (3.93)	−2.38 (3.39)	−11.03	247	.001
National	12.3	13.6
Arm curl (repetitions)
Hawai‘i	10.76 (3.48)	12.77 (3.83)	−2.00 (3.82)	−8.26	247	.001
National	17.0	18.2
Up-and-go (seconds)
Hawai‘i	8.57 (3.68)	7.66 (3.36)	0.91 (1.90)	7.56	248	.001
National	8.9	8.1
Falls (number)
Hawai‘i	0.18 (0.52)	0.18 (0.67)	0.008 (0.84)	0.11	119	.913
National	Not available

Because the satisfaction survey was administered at the 4-month Fitness Checks and annually thereafter and was anonymous, 482 forms were available for analysis (with 139 participants completing more than one survey). Program participants were very satisfied with EnhanceFitness and Instructors (M = 9.45, SD = 0.97), willing to continue the exercise learned from EnhanceFitness (M = 9.16, SD = 1.48), and very confident that they could continue to exercise regularly (M = 9.10, SD = 1.44).

DISCUSSION AND CONCLUSION

Evidence-based health promotion programs are developed in research settings, and replicating them in real-world settings can be challenging (Glasgow et al., 2003; Green & Glasgow, 2006). Having a standard protocol or guideline for adapting evidence-based programs to new communities can be very useful. HHAP developed and used a four-step fidelity protocol to adapt evidence-based programs in Hawai‘i, home to a culturally diverse population. This article illustrated how the protocol was used to ensure fidelity for each area, along with associated standard tools, to adapt a senior exercise program, EnhanceFitness.

Adaptation process began with examination of original program and how it could be implemented locally. Although our modifications to the original program were minimal, we believe that they were crucial to the success of our replication because they gave the program a “Hawai‘i feel.” This finding also illustrates that program adaptors may not have to make large changes in evidence-based programs to tailor them to their own communities. The deconstruction process also helped us develop and follow a detailed implementation and evaluation plan.

Evidence-based program adaptation requires organizational commitment, and organizations must be “ready” to change. Initially, we used the readiness tool for assessment and tried to nurture “unready” agencies rather than dissuade them from trying to replicate the program. “Unready” agency representatives were invited to training with “ready” agency representatives, and did well at the training. The positive results from pre–post training assessment made HHAP believe that they were ready to adopt the program. In reality, however, they faced too many organizational and logistical barriers to sustain the evidence-based program.

We believe that our attention to fidelity led to our seeing similar improvement in participant upper- and lower-body strength and balance as the original study and in other replication sites nationwide. However, we had difficultly retaining Instructors and Master Trainers. Program replicators should plan regular trainings for the evidence-based program to assure an adequate pool of providers.

Successful replication of a program should yield outcomes as good as those found in the controlled testing of the original program. Thus, it is critical to collect outcome data, using the forms and following the protocol established by the original program. Then, your findings can be compared against those in the literature. Some evidence-based programs (e.g., EnhanceFitness) offer the opportunity to send data to a data center, and this allows for cross-site comparison. Because Hawai‘i’s Fitness Checks findings were positive and similar to published results of EnhanceFitness effectiveness (and the national database), we feel confident that our replication sites are delivering the program with high fidelity, but more research is needed to correlate fidelity and outcome data.

There are several advantages to using this four-step protocol. First, it helped HHAP make EnhanceFitness more attractive to Hawai‘i older adults without jeopardizing its key, change-producing components. Second, following the protocol helped assure that the Hawai‘i Instructors were teaching classes with a high level of fidelity. Finally, we saw our diligence pay off when our participants realized expected improvements on their Fitness Checks. HHAP also applied this protocol in its replication of CDSMP, and the protocol worked similarly well with this program (Tomioka, Braun, et al., 2012). Replicating evidence-based programs requires attention to the four task areas in the protocol, and this four-step-protocol can be used to guide replication of other evidence-based programs.

Since conceptualizing and implementing our translation of EnhanceFitenss in Hawai‘i, researchers associated with program replication efforts in other states have published on similar work. An article by Frank et al. (2008) examined five aspects of fidelity in program replication—fidelity in design, training, delivery, receipt, and enactment. They used this framework to describe how four different health promotion programs assured fidelity and discussed key “lessons” for other communities. Many of their “lessons” resonated with us. For example, we agree with Frank et al.’s prohibition on modifying the key behavior change components of the original intervention. We also agree with the need to engage the local community when planning to replicate an intervention, and the need to simplify data collection for practice settings. Finally, we agree to the need to maximize program fidelity through excellent training, by integrating fidelity monitoring into regular agency practice, and by providing trainers and participants with findings from fidelity checks and participant outcomes.

Additionally, we learned that organizations have to commit time and staff resources to carry out this protocol. High resource commitment enhances the translation process and makes the program more effective and efficient. We learned that agencies should not be encouraged to replicate a program if they identify replication barriers related to organizational commitment, resources, and logistics. We learned the importance of training multiple providers per community, as some will relocate, retire, and so on. Adapters should anticipate loss of champions and build in ways to continually develop new ones.

Finally, we realized that organizations must participate in ongoing discussion as they apply the four steps and sustain the replicated program in their community. We recommend building a strong partnership for replication that includes program staff and administrators, evaluators, policy makers, and funders. The partnership should develop a good relationship with someone from the program you want to replicate, who can review any modifications you want to make, share program and evaluation tools, and provide access to summary data from other sites against which you can compare your outcomes.

To conclude, we described a four-step fidelity assurance protocol for replicating evidence-based programs. The protocol was illustrated with examples from Hawai‘i’s experience replicating EnhanceFitness, a senior group exercise program. We believe this protocol is transferrable to other communities that want to adapt and replicate evidence-based, public health programs.