Treatment Fidelity among Family Health Promoters Delivering a Physical Activity and Nutrition Intervention to Immigrant and Refugee Families

Introduction

Treatment fidelity refers to the “methodological strategies used to monitor and enhance the reliability and validity of behavioral interventions” (Bellg et al., 2004). Treatment fidelity greatly impacts both the internal and external validity of intervention research, improves statistical power, and promotes early detection as well as correction of experimental error (Kazdin, 2003; Moncher & Prinz, 1991; Perepletchikova & Kazdin, 2005; Perepletchikova, Treat, & Kazdin, 2007). Without careful attention to treatment fidelity, inferences regarding treatment success, generalizability, and utility are difficult to assess in behavioral research (Bellg et al., 2004; Borrelli et al., 2005). Additionally, careful consideration and monitoring of treatment fidelity has been linked to better treatment outcomes (Durlak & DuPre, 2008; Elliott & Mihalic, 2004; Moncher & Prinz, 1991) likely due to reduction of unintended error (Borrelli, 2011). Complex behavioral interventions are most susceptible to error in assessment and delivery of treatment when there is a greater number of treatment components, use of multiple materials, large intervention staff, and longer duration of sessions (Miller & Rollnick, 2014; Perepletchikova & Kazdin, 2005). Therefore, in order to design and disseminate clinically meaningful interventions, attention to treatment fidelity in behavioral research should be a priority.

The National Institutes of Health and the American Heart Association established the Behavior Change Consortium (BCC) Treatment Fidelity Workgroup to help advance the definition, methodology, and measurement of intervention fidelity in the field of behavior change. The workgroup developed a framework to conceptualize, assess, and monitor treatment fidelity in behavior change clinical trials (Bellg et al., 2004; Borrelli et al., 2005; Braveman et al., 2011; Klesges, Estabrooks, Dzewaltowski, Bull, & Glasgow, 2005; Orleans, 2005; Prochaska, 2005). This framework was trialed among 15 studies that examined two theories of behavior change, or the effectiveness of one theory across multiple health behaviors such as diet, exercise, or smoking. Best practice recommendations derived from these studies were outlined by the BCC. Five areas of treatment fidelity emphasized by the BCC include the study design, training of study interventionists, delivery of the intervention, receipt of treatment, and enactment of treatment skills.

The study design category comprises factors to be considered when designing the study as well as practices intended to ensure replication and hypothesis testing (Borrelli, 2011; Borrelli et al., 2005; Klesges et al., 2005). Recommended strategies consist of ensuring consistent treatment dose (e.g., same frequency and duration of intervention contact), plan for implementation setbacks, and basing the intervention on a specific theoretical framework of behavior change. The training category focuses on ensuring adequate training of study interventionists and maintenance of treatment delivery throughout the project. Standardized training, establishing performance criteria, and continued skill building throughout the study are examples of strategies specific to the training of study interventionists (Borrelli et al., 2005; Gearing et al., 2011; Glasgow, Klesges, Dzewaltowski, Bull, & Estabrooks, 2004). Delivery of treatment consists of issues related to monitoring in order to administer the intervention as intended. Methods to assess treatment delivery focus on adherence to the intervention protocol and monitoring for non-specific treatment effects. Treatment receipt consists of strategies that enhance participants’ understanding of the intervention and ability to perform behavioral skills in real-life settings (Bellg et al., 2004). Finally, treatment enactment refers to the ongoing assessment of participants’ behavior outside of sessions (Borrelli et al., 2005; Resnick, Bellg, et al., 2005).

Unfortunately, treatment fidelity, in general, remains under-reported in behavioral health intervention research, and is greatly lacking in studies with underserved populations. Borelli and colleagues (2011) conducted a meta-analysis evaluating the use of the BCC treatment fidelity strategies utilized in health behavior research over the past 10 years. They found 54% of studies failed to utilize any treatment fidelity monitoring, and only 15.5% of studies adhered to the BCC recommendations. Fewer examples exist regarding the implementation of the BCC model for fidelity monitoring in health disparities research (Glasgow, 2008). Given the need to improve the health of underserved populations (Beadle & Graham, 2011; Braveman et al., 2011; LaVeist, Gaskin, & Richard, 2011), the need to maximize the utility of research findings by creating sustainable, applicable practices becomes increasingly important. An additional layer of complexity (e.g., issues of language and culture compounded by social, economic, and political vulnerability) is added to the design of effective interventions to promote positive health behaviors among immigrant and refugee populations. Fidelity monitoring with interventions among these populations is also poorly understood (Renzaho, Mellor, Boulton, & Swinburn, 2010), emphasizing the need for documented fidelity procedures. In addition, understanding the rationale and methods for adapting fidelity measures among underserved populations would help promote the development of pragmatic approaches for intervention studies (Glasgow, 2013).

The purpose of this paper is to describe the fidelity-monitoring plan for an intervention trial designed using a community-based participatory research (CBPR) approach to improve physical activity and dietary quality among immigrant and refugee families. Important considerations and challenges related to fidelity monitoring and implementing a community-based participatory intervention are also addressed.

Methods

Overview of the Healthy Immigrant Families (HIF) Project

CBPR principles encourage collaborative investigation between community members and academic partners through an equitable relationship during all phases of research design and implementation (Horowitz, Robinson, & Seifer, 2009; Israel, Schulz, Parker, & Becker, 1998; Shalowitz et al., 2009). This research approach is particularly well suited to intervention work, as it empowers communities, promotes understanding of culturally pertinent issues, and targets multi-faceted barriers to health (Minkler, 2005; Wallerstein & Duran, 2006; Wells & Jones, 2009).

The current intervention builds upon a successful CPBR partnership consisting of dedicated community and academic partners Rochester Healthy Community Partnership (RHCP). The mission of RHCP is to promote health and well-being among the Rochester, Minnesota community through CBPR, education, and civic engagement to achieve health equity (www.rochesterhealthy.org). Since its inception in 2004, RHCP has become productive and experienced at deploying data-driven assessments and interventions with and for immigrant and refugee populations (Wieland et al., 2011; Wieland et al., 2012).

RHCP’s Healthy Immigrant Families (HIF) is an that intervention consists of a series of home-based, family mentoring and education sessions aimed at increasing physical activity and healthy eating, delivered by trained Family Health Promoters (FHPs) from each of the participating immigrant and refugee communities (i.e., Hispanic, Somali, Sudanese). The rationale for addressing physical activity and dietary quality was derived from a community needs assessment among immigrant and refugee adults in Rochester, Minnesota. Therefore, this project was designed to evaluate the efficacy of a behavioral intervention designed to promote physical activity and healthy eating among 160 individuals, including both adults and adolescents, randomly assigned to receive the intervention either immediately or after a one year delay (Insert Figure 1 here). Primary outcomes included changes in physical activity,(measured by accelerometer), and dietary quality,(measured by 24-hour dietary recall), that was assessed for up to two years after implementation. Secondary outcomes included changes in body mass index, blood pressure, waist-hip ratio, health-related quality of life, and theory-based measures of physical activity and healthy eating. The intervention was based on Social Learning Theory and therefore theory based measures of nutrition and physical activity assessed level of support, motivational readiness and self-efficacy for healthy eating and physical activity level.

Figure 1.

Study Design and Intervention Schedule

Participatory intervention development was achieved by the creation of eight workgroups, each charged with completing specific tasks related to content, dissemination, recruitment, community capacity, etc. Each workgroup met regularly (e.g., bi-weekly), and included participation by both community and academic partners. Existing manuals for physical activity and healthy eating (i.e., Brownell-LEARN manual, the Mayo Clinic Diet, Active Living Every Day) further informed the intervention content, as well as the results of 16 focus groups conducted among 127 participants (both adolescents and adults) from each of the participating communities (Tiedje et al., 2014; Wieland et al., 2013). The resulting products included an intervention manual containing 12 modules pertaining to physical activity and healthy eating, delivered in the homes of the participating families by FHPs (Insert Table 1 here).

Table 1.

Intervention Topics Delivered by Family Health Promoters

Module Title	Session Objectives
Fruits and Vegetables	Describe vitamins found in fruit and vegetables Identify specific examples of vitamins and their functions Know recommended daily intake of fruits and vegetables Become familiar with nutrition label content
Beverages	Increase awareness of hidden sugars in beverages and their impact on health Increase understanding of beverage portion sizes Identify nutrients in various beverages
Fats	Identify differences between saturated and unsaturated fats Increase knowledge of hidden fats and impact of high-fat diets Promote awareness of alternative cooking methods that include healthier fats Understand fat content in various foods
Snacks	Increase ability to identify healthy snack options Reinforce familiarity with food labels
Portion Control	Educate participants about healthy portion sizes Understand the impact of caloric intake on weight Understand differences between adult and child portion sizes
Smart Shopper	Increase strategies for healthy food shopping Reinforce knowledge and familiarity with food labels
Increasing Physical Activity	Promote education about benefits of physical activity Increase knowledge regarding recommended amount of physical activity in children and adults Provide strategies to incorporate physical activity into daily life
Muscle Strength and Flexibility	Educate participants on basic stretching exercises Teach basic strength training exercises using resistance bands Encourage benefits of increasing muscle strength and flexibility
Screen Time	Provide recommendations regarding recommended amount of screen time Discuss relationship between physical activity, nutrition, and screen time Offer strategies for reducing screen time and promote alternative activities
Overcoming Barriers to Physical Activity	Problem solve barriers to physical activity Identify sources for social support and encouragement related to physical activity Practice goal setting
Exercise/Food/Work/Live Balance	Review relationship between sleep, stress, and quality of life Discuss strategies for stress management
Celebrating Accomplishments	Review lessons relevant to the family Encourage attendance of future follow-up visits Celebrate progress and accomplishments

The FHPs delivered the intervention during 13 home visits with participating families over a 6-month period. The first visit was for introductions and orientation to the intervention. Home visits 2-13 were used to sequentially deliver each of the 12 physical activity and healthy eating modules. At least one adult and one child were required to be present at each visit. Each home visit was 30 to 90 minutes in length. Each session consisted of educational material as well as structured activities that the FHP could apply flexibly, depending on the needs of the family. At each visit, FHPs monitored progress on family goals, provided social support and mentoring to problem-solve barriers, and set realistic future goals. Motivational Interviewing (MI) techniques were incorporated to address participants’ ambivalence and/or resistance to changing physical activity or dietary patterns (Rosengren, Hartzler, Baer, Wells, & Dunn, 2008). After completion of the 13 home visits, FHPs initiated follow-up phone calls to each participating family every two weeks, for up to 12 phone calls over the next 6 months. The purpose of these follow-ups was to monitor progress towards goals, and re-enforce points related to each of the 12 modules. Participating families received small incentives relating to the intervention modules (e.g., water bottles, resistance bands, athletic balls) in addition to food vouchers totaling $300. The intervention phase of this study has now been completed.

Treatment Fidelity Related to Treatment Design

Treatment fidelity related to design focuses on the methodological processes that ensure the study adequately assesses the proposed hypotheses in relation to a theoretical framework. (Borrelli, 2011) (Insert Table 2 here). Adherence to behavior change principles was provided by experts in behavior change interventions (e.g., Clinical Psychologists, Doctorate level Nurses) as well as MI to ensure that the intervention manual and FHP training was consistent with these theoretical frameworks. Selection of FHPs included recommendations from community members, ensuring cultural and linguistic congruence with the populations in which they work. Participants received a fixed number of pre-determined sessions, and the same information was covered per topic for each of the families in the intervention. Sessions were conducted in the native language of the participants. (Insert Table 3 here). Protocol deviations were manually recorded by the FHPs after the completion of each session using a checklist form (yes/no);brief summaries of the deviations were included. This information was entered into a database for data analysis. Protocol deviations were considered to be: sessions under 30 minutes or over 90 minutes; sessions that did not have a child and an adult enrolled in the study present; missed sessions or session content; and sessions inconsistent with the spirit of MI (e.g., failure to assess motivation/confidence; identification of barriers and solutions; goal settings)

Table 2.

Treatment Fidelity Checklist for the Nutrition and Physical Activity Intervention

Treatment Design

Fixed number of sessions Frequency of session is pre-determined FHPs use scripted manuals Record deviations from protocol (e.g., off topic, over/under time) Same information is covered per topic for each of the families FHPs were selected based on community member recommendations FHPs received specialized training on CPBR and MI FHP received specialized training in working with families

Training Providers

Discuss training schedule All FHPs met a priori performance criteria before delivering the intervention All FHPs trained together, using standardized materials Training took into account different levels of past training for all FHPs Used structured practice and role-playing Used standardized patients FHPs were observed with pilot participants (audio tape, live), engaged in role play, and were provided feedback Engaged in FHP-identified problem solving and debriefing Provided FHPs with written test of MI skills pre and post training Observed FHPs deliver intervention with trained actors Continued observation in real time setting with mock participants Monthly supervision FHPs rated on adherence to intervention protocol using standardized form

Delivery of Treatment

Used standardized manual FHPs reported protocol deviations Used self-report of FHPs to determine protocol violation Maintained a standardized treatment protocol

Receipt of Treatment

FHPs asked participants if they had questions and if they understood the material FHPs utilized elicit-provide-elicit technique to determine participant understanding Used MI techniques to determine participants’ level of understanding Participants set goals at the end of every session and were queried as to their abilities to meet goals in between sessions Assessed for motivation and confidence Used MI strategies to increase self-efficacy for change

Enactment of Treatment Skills

Used accelerometers to electronically monitor behavior Had follow up discussion/calls and counseling with participants Reviewed ongoing use of skills with participants Used self-report of participants to engage in problem solving

FHP-family health promoter

CBPR-community-based participatory research

MI-motivational interviewing

Table 3.

Training Schedule

Module 1: Orientation to the Institution

Module 2: Introduction to Rochester Healthy Community Partnership (RHCP) & Healthy Immigrant Families (HIF) History of RHCP Development of HIF Study Community-Academic Partners Project Overview Study Timeline What to Expect

Module 3: Orientation to community-based participatory research (CBPR) Definition of research Principles of CBPR Working with diverse communities Cross-cultural communication Strategies for resolving conflict

Module 4: Protecting Human Subjects Role of the Institutional Review Board (IRB) Human Subjects Protection Training Application of principles through case studies/discussion Examination/Certification

Module 5: Working with Families Defining the role of the Family Health Promoter Basic etiquette when in the home Communicating with families Building therapeutic relationships with families Establishing professional boundaries Asking for help/Process for referral

Module 6: Social Cognitive Learning Theory Theory Basics Explaining behavior change Applying theory to real-life examples Group didactics/discussion

Module 7: Motivational Interviewing Basic principles Application to health behavior Role play/didactics

Module 8: Basics of Nutrition & Physical Activity: The LEARN Manual

Module 9: Applied Nutrition & Physical Activity Seminars provided by nutritionists and physical activity trainers

Module 10: Family Intervention Intervention overview Frequency, timing and structure of home visits Review of each of the 12 intervention modules

Module 11: Working with Data How to provide documentation Practicing data entry Auditing and quality checks

Treatment Fidelity Related to Training

Training of FHPs was designed to ensure satisfactory delivery of the intervention to study participants. Trainings were tailored to account for different backgrounds and past training experiences of the FHPs. For instance, half of the FHP’s did not have previous experience in research, behavioral interventions, nutrition, or physical activity education. All FHPs were required to complete all training modules prior to intervention delivery, and meet a priori performance criteria (i.e., pilot tested the intervention with test families while being observed by staff), prior to intervention delivery with participating families. Additionally, all FHPs completed a written test of MI skills pre and post training using the Video Assessment of Simulated Encounters-Revised (VASE-R; Rosengren et al., 2008) to ensure acquisition of MI skills.

Three of the four FHPs were trained together; however, an additional staff member from the Hispanic community was hired to account for the high Hispanic participation rate. While not ideal, this FHP received accelerated training using the same format of the first three FHPs. Training was able to be accelerated because this individual had extensive medical training and previous expertise working in research. All trainings utilized the same instructors, and sessions incorporated structured practice and role-plays. Unfortunately, recordings of actual intervention sessions could not be conducted due to time and cost constraints associated with transcription and translation. Ongoing training was conducted using live observation with standardized patients as well as trouble shooting, problem solving, and debriefing difficult sessions. All FHPs were provided with documented feedback immediately following each observed interaction by supervisors and subject-matter experts. FHPs were rated on use of MI skills, such as: asking for permission; use of open-ended questions; determining importance of the topic to the family; assessing family confidence for behavior change; skills used to build the family’s confidence; rolling with resistance; use of evocative statements; addressing barriers; providing solutions; and goal setting. Raters assessed if the skill was completed during the session or not. A total of 64 sessions were assessed for delivery of intervention protocol as well as use of MI skills. The overall FHP simulated session adherence rate was 84.3%. These post-supervision sessions were initially conducted bi-weekly, then monthly, and finally, bi-monthly, prior to 12 month assessment. Immediate access to supervisors and subject-matter experts, (e.g. exercise physiologists and dieticians), was available to the FHPs to discuss difficult cases or scenarios that developed during the intervention. Likewise, community leaders from each group (i.e., Hispanic, Somali, Sudanese) were available to discuss issues of cultural sensitivity, tailoring, and logistical concerns during the intervention.

Treatment Fidelity Related to Delivery of the Intervention

Treatment fidelity pertaining to treatment delivery includes ensuring that the content and dose are consistent as well as adherence to the manual (Bellg et al., 2004). Utilization of a standardized treatment manual that was accessible to all staff was emphasized during supervision. FHPs were encouraged to report and record any protocol deviation. The inability to record intervention sessions prevented assessment of intervention delivery in the field. Instead, FHPs were encouraged to immediately document each family encounter using a session checklist that highlighted the main teaching points for that module. The FHP’s completed documentation for 97.67% of all sessions (294/301). On average, visits lasted a total of 50.7 minutes (SD=12.2, range 20-90 minutes); a majority of visits included at least one adult (90.8%) and one child (88.1%). Self-ratings of adherence to session components ranged from 82.0-89.0% (Insert Table 4 here). The FHP’s recorded difficulties in maintaining protocol integrity, which primarily included family’s decision to decline participation in activities and goal setting for modules they felt capable of doing already (e.g., healthy grocery shopping), and children unable to participate in the counseling session due to after school activities. Further, several sessions were rescheduled due to conflicting schedules with family members. The FHP’s addressed these issues proactively by contacting families ahead of time regarding sessions, reminding them of the need for family members to be present, and maintained flexible work schedules in order to accommodate the needs of each family (e.g. visiting families in the evenings and weekends, when necessary).

Table 4.

FHP Self-Report Fidelity Form

Item	Percent Completed at each Session
Assessed existing knowledge, motivation, confidence & current behaviors about the new topic	89.1%
Shared new content	87.1%
Conducted the activity	84.7%
Identified barriers and solutions	85.5%
Goals were set by the famiy	82.4%

Treatment Fidelity Related to Receipt

Treatment receipt ensures that participants received and understood the treatment provided (Borrelli et al., 2005). All participants in the study were asked by FHPs if they had questions regarding session content and if they understood the material. FHPs used MI techniques to determine participants’ level of understanding, such as the MI technique “Elicit-Provide-Elicit”(e.g., “What do you know about [topic to be covered]; here is some additional information to consider…; what do you think?”). Additionally, participants were asked to set goals at the end of every session and queried as to their abilities to meet goals in between sessions, in addition to their motivation and confidence to carry out goals.

Treatment Fidelity Related to Enactment

Enactment of treatment pertains to the assessment and monitoring of participant behavior outside of the intervention (Resnick, Inguito, et al., 2005). Each session in the intervention was designed to assess the family’s progress towards goals (e.g., problem-solve barriers, encourage progress towards behavior change, address motivation/confidence). Further, intervention sessions took place in the home; therefore, FHPs witnessed enactment of treatment in a real-world setting (e.g., types of foods and beverages present in the home). FHPs reviewed ongoing use of skills and progress towards goals with participants during follow-up telephone calls.

Discussion

Treatment fidelity is an important source of variation affecting the credibility and utility of intervention research. Little is known about fidelity of interventions among underserved populations. Development and implementation of a treatment fidelity plan, especially among underserved populations, requires careful conceptualization of study needs in conjunction with what is feasible in the population. The current study utilized a complex treatment protocol designed to help families improve their diet as well as increase physical activity. Both community and academic partners informed the development of this intervention and the manner in which it was delivered to participating families. Treatment fidelity was incorporated into the planning and design of this intervention, using recommendations set forth by the BCC, to ensure consistent intervention delivery. Additionally, adaptations to these recommendations were necessary to accommodate the needs of the community. This longitudinal community “co-creation” of fidelity mechanics and measures provided an important platform for local and cultural sensitivity to study procedures. The shared strengths approach between community and academic expertise is an example of CBPR principles impacting these facets of study design and implementation.

Several unique challenges related to treatment fidelity arose in this study, such as working with families as opposed to individuals; heavy reliance on the self-report of FHPs regarding adherence to the protocol; the risk for social desirability bias among participants reporting treatment fidelity related to receipt; and the need to hire an additional staff member once initial protocol training was complete. Additionally, several staff members from various disciplines (e.g., psychology, nursing, medicine, community leaders with cultural expertise) participated in about 250 hours of interventionist training and monitoring, which the cost in FTE alone was great. This may not be feasible in other settings. All sessions were conducted in the native language of the family, which was based upon the request of the community as English may not be regularly spoken in the home. In addition, the many families indicated a preference to avoid video or audio recording sessions. Further, sessions could not be recorded due to time and cost of translation and transcription, difficulties recording audio in the family’s home, and community member’s discomfort with audio/video recording. Language differences among the three different priority populations arose when developing the intervention manual, which required additional time and preparation. Finally, flexibility in scheduling as well as persistence in contacting family members was necessary in order to ensure visits were attended. The willingness of the FHP to adjust their schedule, attend visits in the homes of each family, and organize consistent follow-up was necessary in order to ensure participation.

Future Research Directions

Few studies have reported fidelity planning for behavioral health research targeting immigrant and refugee populations (Glasgow, 2008). Several of the efforts outlined in this paper therefore address an important gap in the literature and offer strategies for measuring treatment fidelity in similar contexts and populations. Application of treatment fidelity remains an important topic in behavioral health research as it encourages dissemination of effective interventions, opportunities to correct ineffective interventions, and understanding of how to replicate these techniques.

Future studies conducting behavioral intervention research for underserved populations should consider treatment fidelity in the design, implementation, and evaluation of intervention effectiveness. Research focusing on the development of cost effective training methods for study interventionists, creative strategies for monitoring adherence, and feasible measures for participant understanding would improve measures of intervention fidelity in studies involving heterogenous communities. The BCC guidelines employ flexible methods that can be used to accommodate the needs of the community within a CBPR framework while maintaining intervention integrity. Further, treatment fidelity monitoring offers several additional benefits such as the opportunity to provide interventionists with positive feedback, support, and supplementary training throughout the course of the study. Additionally, utilization of treatment fidelity strategies can assist with identification of protocol violations, ensure the intervention is delivered as intended, and can be replicated in future studies. Finally, through fidelity monitoring, researchers are able to identify whether community needs are being met by the intervention, and adjust the intervention based on this feedback. Ultimately, intervention fidelity recommendations set by the BCC are important to underserved populations as they identify effective treatment components and how to disseminate these interventions in order to maximize their reach.