Development of an Innovative Process Evaluation Approach for the Families Improving Together (FIT) for Weight Loss Trial in African American Adolescents

Abstract

This study demonstrates how a multi-theoretical, multilevel process evaluation was used to assess implementation of Families Improving Together (FIT) for weight loss intervention. FIT is a randomized controlled trial evaluating a culturally tailored, motivational plus family-based program on weight loss in African American adolescents and their parents. Social Cognitive, Self Determination, Family Systems theories and cultural tailoring principles guided the conceptualization of essential elements across individual/family, facilitator, and group levels. Data collection included an observational rating tool, attendance records, and a validated psychosocial measure. Results. Attendance records (0=absent, 1=present, criteria=≥70%) indicated that 71.5% of families attended each session. The survey (1=false, 6=true, criteria=≥4.5) indicated that participants perceived a positive group climate (M=5.16, SD=.69). A trained evaluator reported that facilitator dose delivered (0=no, 1=yes, criteria=≥75%) was high (99.6%), and fidelity (1=none to 4=all, criteria=≥3) was adequate at facilitator (M=3.63, SD=.41) and group levels (M=3.35, SD=.49). Five cultural topics were raised by participants related to eating (n=3) and physical activity (n=2) behaviors and were integrated as part of the final curriculum. Discussion. Results identify areas for program improvement related to delivery of multi-theoretical and cultural tailoring elements. Findings may inform future strategies for implementing effective weight loss programs for ethnic minority families.

1. Introduction

Effective weight loss interventions for underserved (low income, ethnic minority) youth are urgently needed to address health disparities in obesity (Ogden, Carroll, Kit, & Flegal, 2012). However, few randomized controlled trials have specifically targeted underserved adolescents (Kitzman-Ulrich, et al., 2011; Wilfley, et al., 2007; Wilson, 2009; Wilson, Alia, Kitzman-Ulrich, & Resnicow, 2013), and those that have, demonstrated only marginal or no effects on weight loss (Barr-Anderson, Adams-Wynn, DiSantis, & Kumanyika, 2013; Whitlock, Williams, Gold, Smith, & Shipman, 2005). Furthermore, attrition rates are higher among underserved populations with previous reviews reporting attrition rates as high as 70% among minority populations in comparison to an average range of 12-18% among non-minority youth (Budd, et al., 2007; Jelalian, et al., 2008; White, et al., 2004). Thus, there is a need for interventions that address culturally relevant barriers to participation and weight loss experienced by underserved, African American youth. Process evaluation, which measures the extent to which an intervention is delivered or planned (Moore, et al., 2014), may enhance the effectiveness of weight loss interventions that are tailored to meet the needs of African American families (Seo & Sa, 2010; Wilson, 2009). The purpose of the present article is to demonstrate how process evaluation is used to examine the implementation of the Families Improving Together (FIT) for Weight Loss Trial, a culturally-tailored, family-based weight loss program for African American adolescents and their parents. Specifically, this study utilizes a multilevel process evaluation approach to assess program reach, dose, and fidelity to provide formative feedback for improvement of the FIT program.

1.1 Need for Conducting Process Evaluations of Youth Weight Loss Programs

Conducting comprehensive process evaluations may increase the effectiveness of weight loss interventions for minority youth. Process evaluation may support the implementation of weight loss interventions by monitoring components such as reach (proportion of intended audience that participates in the intervention), dose (extent to which program content is addressed and received), fidelity (extent to which the intervention conforms to theoretical elements and is delivered as planned), and context (aspects of the intervention that may influence implementation or study outcomes) (Saunders, Evans, & Joshi, 2005). By assessing these components in the present FIT trial, the process evaluation may ensure adherence to cultural and theoretical elements for promoting weight loss and provide guidance for ongoing program improvement. Importantly, assessing process components across multiple levels may be important given that systems related to individual/family participants, facilitators, and group dynamics may influence implementation (Durlak & DuPre, 2008).

Despite the benefits of process evaluation, it remains an underutilized component of weight loss trials. A recent study revealed that only 40% of childhood obesity interventions have reported using process evaluation to assess program implementation, and those that do often only focus on one type of evaluation such as participant satisfaction or attendance (Branscum, Sharma, Wang, Wilson, & Rojas-Guyler, 2013). Thus, more studies are needed which demonstrate how process evaluation may be used to understand implementation of childhood obesity programs, especially in underserved and ethnic minority populations.

1.2 The Families Improving Together (FIT) for Weight Loss Trial as a Novel Approach to Weight Loss

This paper describes a multilevel, theory-based process evaluation for Project FIT. The FIT trial integrates constructs from Social Cognitive Theory (SCT)(Bandura, 1986, 2000), Self-Determination Theory (SDT)(Ryan & Deci, 2000), and Family Systems Theory (FST)(Baumrind, 1966) into a culturally tailored weight loss program for underserved African American families (Figure 1). Specifically, FIT targets behavioral skill building (e.g., self-monitoring, goal setting, skill mastery) and positive parenting strategies (e.g., shared decision making, positive communication, social support, parental monitoring) in the context of a positive social climate intervention. In this study, a positive social climate is defined as one that promotes autonomy and belongingness and, as a result, builds intrinsic motivation for lifestyle change. Cultural strategies (Kreuter, Lukwago, Bucholtz, Clark, & Sanders-Thompson, 2003; Resnicow, Braithwaite, Ahluwalia, & Dilorio, 2001; Wilson, 2009) are also embedded in Project FIT to increase retention and to overcome culturally relevant barriers (e.g., time constraints, competing priorities, multiple caregiving roles, etc.) to weight loss.

Figure 1.

Multi-level Conceptualization of Theoretical Essential Elements

Growing evidence suggests that interventions which integrate parenting, motivational and behavioral constructs and address cultural issues are likely to have greater success in producing weight loss outcomes for youth (Barr-Anderson, et al., 2013; Kitzman-Ulrich, et al., 2011; Kitzman-Ulrich, et al., 2010; Kitzmann, et al., 2010; Whitlock, et al., 2005; Wilson, 2009; Wilson, et al., 2013). Notably, a preliminary study to the FIT trial demonstrated the feasibility of integrating motivational and parenting components in the context of weight loss program for overweight African American and Caucasian adolescents. Results revealed significant improvements in energy intake and weight related outcomes in adolescents participating in the intervention as compared to a health education condition (Kitzman-Ulrich, et al., 2011). The preliminary study provides an important example of how to implement a social climate-based weight loss program for families, but the current study expands on this work by adding more formative work to developing the cultural tailoring of the intervention. In particular, implementation of programs that integrate multiple theoretical approaches may be challenging (Durlak & DuPre, 2008; Saunders, et al., 2005), and process evaluations which assess complex interventions could enhance program implementation. The present study thus describes how a novel approach to formative process evaluation is used to improve the delivery of a culturally tailored family-based weight loss intervention in the FIT trial during the face-to-face group-based part of the intervention (see Wilson, et al., Under Review for overview of study design).

1.3 Evaluating Social Climate-based Health Promotion Interventions

The process evaluation for the FIT intervention builds on a growing literature that has documented the implementation of positive social climate health promotion interventions (Branscum, et al., 2013; Davis, et al., 2000; Reynolds, et al., 2000; Robbins, Pfeiffer, Wesolek, & Lo, 2014; Wilson, et al., 2009; Wilson, et al., 2011). The Active by Choice Today (ACT) Trial was a school-based randomized controlled trial testing the efficacy of a positive social climate (student choice in activities, cooperative planning, and teacher and peer support) and behavioral skill building (group goal setting etc.) after-school intervention on increasing physical activity (PA) in underserved adolescents (Wilson, et al., 2009; Wilson, et al., 2008; Wilson, et al., 2011). Results of the process evaluation demonstrated high dose and fidelity, suggesting that both motivational and behavioral elements for promoting PA were successfully implemented during the school-based program (Wilson, et al., 2009).

Another randomized trial, Guys Only Activity for Life (GOAL), was guided by principles from the Health Promotion Model (perceived benefits, enjoyment, perceived barriers, PA self-efficacy, social support, role models, norms) and SDT (competence, autonomy, relatedness) to increase physical activity among 6^th and 7^th grade, multiethnic boys (Robbins, et al., 2014). The GOAL intervention included an after-school PA club designed and a nurse-delivered motivational interviewing session. Findings revealed high fidelity for delivering the PA club program. However, number of minutes of PA provided during sessions did not meet dose criteria. The authors report that this may have been because greater instruction time was needed than planned for supporting behavioral skill development among overweight/obese participants.

Taken together, these previous studies have important implications for weight loss programs. Elements related to the group climate as well as facilitator delivery of theoretical constructs may influence implementation; this suggests that process evaluations should assess delivery across multiple systems to fully understand implementation. The FIT process evaluation expands on this past research by using a multilevel perspective to assess implementation of a positive family and group climate intervention targeting individual behavioral skills for weight loss in overweight African American youth and their parents.

1.4 Integrating a Culturally Tailored Approach to Weight Loss

The assessment of implementing a culturally tailored weight loss intervention within the context of a positive social climate is a novel contribution of the FIT process evaluation. Several studies have assessed dose delivered of cultural elements as part of obesity prevention programs for African American preschoolers (Fitzgibbon, et al., 2011), teenage girls (Story, et al., 2003), and adults (Campbell, Resnicow, Carr, Wang, & Williams, 2007; Resnicow, et al., 2004). For example, the Hip-Hop to Health Jr. program incorporated cultural adaptations, such as targeting culturally relevant foods, music and dance, in the context of a teacher delivered, obesity prevention program for African American preschoolers (Fitzgibbon, et al., 2011). Results from the process evaluation revealed that teachers implemented nutrition and exercise lessons, suggesting that cultural components included in the curriculum were implemented, though there was no direct assessment of fidelity to cultural elements. Another study, Body and Soul, was a dietary intervention for adults conducted through African American churches (Campbell, et al., 2007; Resnicow, et al., 2004). The intervention incorporated a variety of cultural targeting strategies, including pastor support, development of a cookbook that included church members’ recipes, and church wide nutrition promotion activities. Process evaluation results demonstrated adequate dose delivered of program components and high participant satisfaction with program materials.

Collectively, these studies demonstrate that culturally tailored components can be successfully implemented within the context of health promotion programs. However, they do not describe the in-depth use of a process evaluation tool for assessing the implementation and effectiveness of the cultural strategies. For example, results do not describe the mechanisms through which components addressed cultural specific barriers to weight loss (e.g., whether cookbooks helped to overcome barriers to reducing caloric dense ethnic foods) (Resnicow, et al., 2004). The FIT process evaluation expands on this past research to describe the extent to which cultural components lead to discussions for addressing unique cultural issues and barriers. In this way, the FIT process evaluation assesses the use of cultural strategies which may shape ongoing program improvement. Further, results may more fully illustrate the process for delivering a culturally relevant intervention and could provide guidance to future weight loss programs.

Thus, the purpose of the present study is to demonstrate how process evaluation is used to assess implementation of, and to provide formative feedback for, a culturally tailored, motivational plus family-based weight loss program. Process evaluation results related to reach, dose and fidelity are examined across individual/family, facilitator, and group levels to understand adherence to the FIT theoretical framework and implementation effectiveness.

2. Methods

2.1 Study Design

The Families Improving Together (FIT) for Weight Loss Trial is a randomized controlled weight loss trial evaluating the efficacy of a culturally-tailored, motivational plus family weight loss (M+FWL) versus a comprehensive health education program on decreasing z-score body mass index (BMI) among African American adolescents and their caregivers (Wilson, et al., Under Review). The trial utilizes a cohort design, with 13 cohorts run across 5 years (2013-2017). Each cohort participates in a 16-week intervention comprised of two phases: 1) an 8-week face-to-face group program and 2) an 8-week online program. For each phase of the program, parent-adolescent dyads are randomized to either the M+FWL program or a comprehensive health education program. The focus of this manuscript is on the process evaluation for the M+FWL face-to-face group sessions. Sessions, which were held once a week for an hour and a half, were led by certified intervention facilitators. Data from cohorts 1-2 are summarized to illustrate how this multi-level approach to process evaluation may be used to assess program implementation of theory-driven program elements across facilitator, group and family levels. Results are used in a formative manner to provide guidance for improving implementation. The study was approved by the University of South Carolina Institutional Review Board.

2.2 Participants

Participants were recruited through a variety of methods. The Principal Investigator and Project leaders developed relationships with local organizations, including churches, healthcare providers, libraries, and school districts. These partnerships provided ongoing community connections to support recruitment. Radio advertisements were also aired on culturally-targeted radio stations, and flyers were posted at community centers. Local festivals and events that were attended by a broad demographic of families are attended by trained recruitment staff.

Families were eligible to participate if: 1) they have an African American adolescent between the ages of 11-16, 2) the adolescent is overweight or obese, defined as having a BMI ≥85^th and <99^th percentile for age and sex, 3) at least one parent or caregiver living in the household with the adolescent is willing to participate, and 4) the family has internet access. Exclusion criteria include presence of a medical or psychiatric condition that would interfere with changing physical activity or dietary behaviors, taking medication that could interfere with weight loss or concurrent participation in a weight loss program.

2.3 Motivational Plus Family Weight Loss (M+FWL) Intervention Description

The essential elements for the M+FWL intervention are listed in Table 1. The essential elements were guided by the FIT theoretical framework and are conceptualized across individual/family, facilitator, and group levels (Figure 1). The M+FWL intervention integrates constructs from SCT (Bandura, 1986, 2000), SDT (Ryan & Deci, 2000), and FST (Baumrind, 1966) to target family communication (active listening, descriptive praise, push-pull language, problem solving), positive parenting skills (autonomy support, social support, limit setting) and behavioral skills (self-regulation, self-monitoring, relapse prevention) while promoting manageable goals related to weight loss and calorie reduction. Increases in motivation, behavioral skills, and positive parenting strategies are hypothesized to mediate the effects of the program on weight related outcomes in adolescents and their parents. Cultural strategies (Kreuter, et al., 2003; Wilson, 2009) are integrated throughout the program to address culture specific barriers to weight loss.

Table 1.

FIT Group Session Essential Elements

Theory	Essential Element	Element Description by Level
		Individual/Family	Facilitator	Group
SCT	Self-Monitoring	Parents and adolescents monitor their caloric intake, energy expenditure, and weight, using a tool of their choice.	Facilitators review, provide weekly feedback, and provide choice on self-monitoring logs/goals.	Families share and provide one another with feedback on self-monitoring goals.
SCT	Goal Setting	Parents and adolescents set specific weight-loss goals together weekly, including intake, expenditure, and sedentary behavior goals.	Facilitators review, provide weekly feedback, and provide choice on goals.	Families share and provide one another with feedback on goal-setting.
SCT	Self-Regulation Skills	Parents and adolescents learn to identify personal barriers, substitute healthier alternatives, and provide positive reinforcements.	Facilitators help participants identify barriers, substitute healthier alternatives, and provide positive reinforcement.	Families share and help one another identify barriers, substitute healthier alternatives, and provide positive reinforcement.
SDT & FST	Communication Skills	Parents and adolescents use positive communication strategies, including reflective listening, problem-solving, and shared decision-making, to discuss weight-loss behaviors.	Facilitators model and provide families with feedback and coaching on positive communication strategies.	Families model positive communication strategies when providing one another with feedback and support.
FST	Parental Monitoring and Limit Setting	Parents monitor and track adolescent self-monitoring and goals, set limits with adolescents around weight loss behaviors, and monitor implementation of family rules and rewards for adhering to weight-loss behaviors.	Facilitators review and provide weekly feedback on parental monitoring and limit-setting.	Parents problem-solve and provide support to other parents on parental monitoring and limit-setting.
SCT, SDT, & FST	Social Support	Adolescents use strategies for eliciting social support for weight-loss behaviors from parents. Parents provide adolescents with social support for weight-loss behaviors.	Facilitators provide adolescents with strategies for eliciting social support and parents with strategies for giving social support.	Families provide each other with social support.
SDT	Autonomy Support	Adolescents have choices and are provided with opportunities to give input. Parents seek input from adolescents and negotiate rules and behavior changes together. Families engage in shared decision-making.	Facilitators model and give families opportunities to provide input.	Families engage in group decision-making.
SCT	Self-Efficacy	Adolescents and parents have opportunities to practice and successfully master weight loss strategies.	Facilitators provide opportunities for hands-on, skills-based practice in managing weight-loss behaviors.	Families increase self-efficacy through observational learning.
SDT	Motivation	Families provide input and build confidence in changing weight-loss behaviors.	Facilitators create a climate which fosters autonomy, competence, and belongingness.	Families feel a sense of belongingness other families.
Cultural Tailoring	Adaptation to Cultural Issues	Families develop action plans for resolving cultural barriers to weight loss and parenting skill development as appropriate.	Facilitators respond to cultural topics and promote an environment that is welcoming of discussion of cultural issues.	Families share and discuss cultural topics related to weight loss and parenting skill development with one another as they arise.

At the individual/family level, families are encouraged to develop manageable goals for weight loss and caloric reduction. Participants are provided their BMI and associated recommended calorie levels during week 2 of the intervention. Specifically, parents and teens are provided with a recommended daily caloric intake range depending on their BMI and age. Participants then use these values to develop long and short term goals related to weight loss or maintenance, calories in and calories out. Project FIT targets five health behaviors related to caloric intake/expenditure: 1) increasing fruit and vegetable intake, 2) decreasing fast food and junk food intake, 3) decreasing sugar sweetened beverages, 4) increasing physical activity, and 5) decreasing screen time. National recommendations are promoted throughout the program and are used as a benchmark for setting weekly calorie goals. To support the attainment of weekly goals, families are encouraged to participate in activities related to behavioral and family skill building both in session and at home. These activities include weekly behavioral skill building activities (e.g., self-monitoring) and family bonding activities, which are take-home activities that support the development of positive parenting skills and communication (e.g., setting a family health goal, cooking a meal together, etc.). Importantly, families are provided with choice on methods for monitoring and completing weekly goals. For example, families are shown paper-pencil (e.g., tracking forms) and electronic methods (e.g., My Fitness Pal) for self-monitoring calories or energy expenditure and are supported in finding a tool of their choosing. Additional tools provided to the families to support behavioral and family skill building include a workbook that includes pages for each weekly session, a Calorie King journal, and a pedometer.

At the facilitator level, facilitators (lead and co-facilitator) are responsible for implementing the intervention curriculum. The lead facilitator is responsible for delivery of key content and adherence to program essential elements and for modeling positive communication skills for the families. Individualized feedback sessions are implemented weekly, and focus on youth's progress with his/her weight and caloric goals, are conducted by both the lead and co-facilitator prior to the group session. During these sessions, teens and their caregivers are provided the opportunity to receive one-on-one feedback on the previous week's behavioral goals. Facilitators provide feedback to participants on their weekly progress and assist with goal revision and action plan development. Facilitators also provide feedback on development of family skills through review of the weekly family bonding activity and through reinforcing use of positive family skills (e.g., positive communication, social support) related to working on weight loss goals. Sessions last approximately 15 minutes and are held either before or after session each week.

The use of a group structure is foundational to FIT; in line with SDT, a positive social climate atmosphere can help facilitate motivation and learning for all families. To promote positive interactions and sharing at the group level, families set ground rules during week 1 of the intervention which include being respectful and engaging in active listening. The purpose of the ground rules is to encourage modeling of positive communication among group participants, and to promote a positive social climate where participants feel respected and supported. Each session begins with group feedback (lasting approximately 15 minutes) where families are able to share successes and discuss challenges related to weight loss and skill building as a group. The group feedback is intended to encourage social support and positive modeling and communication skills among group participants. Break-out activities are also incorporated into the weekly sessions. During the break-out activities, parents and teens have the opportunity to discuss and practice family skills separately and share strategies with one another for implementing the various skills.

Program materials and the intervention curriculum were designed by experts in cultural tailoring to be relevant for African American families. The FIT group-based intervention utilizes both surface- and deep-structure cultural strategies (Kreuter, et al., 2003; Wilson, 2009). Surface-level methods include peripheral tailoring of intervention materials. Workbook pages and take home activities are designed using culturally relevant images and examples (e.g., pictures representing African American families). Deep-structure methods include sociocultural values that are incorporated into the intervention through the use of culturally relevant discussions to support the participants in addressing barriers. Preliminary discussion points were included in the facilitator's guides for cohort 1 that included themes such as identifying foods with special meaning, discussion of emotional eating, pillar syndrome (i.e., role of women as caregivers for the family) and hairstyle during physical activity. These themes in addition to new ones identified as part of the process evaluation (described below) were integrated for the final set of cultural themes for the final intervention curriculum.

2.4 Facilitator Training

All facilitators undergo extensive training that includes both didactic and hands-on, role-play components. Facilitators receive training on advanced behavioral skills related to weight loss and positive parenting and family communication strategies. Facilitators also receive extensive training on motivational interviewing techniques and learn how to promote and maintain a positive social climate while targeting development of behavioral and family skills. Furthermore, facilitators are trained in cultural competency skills to allow for targeting of content delivery to meet needs of group members. To further assist with implementation, facilitators are provided a facilitator's guide for each session. The guides include session objectives, key content, activities, and prompts for each session. Key content was identified for each session to ensure delivery of theoretical elements. In the event that families are absent at a given session, facilitators conduct make-up sessions with participants. The make-up sessions were included to support dose delivered of program content. Facilitators call absent families within a day to set up the make-up session, which may take place either in person or over the phone depending on the participant's preference. During the make-up sessions, facilitators deliver key content (identified using a checklist) to participants as outlined in the facilitator's guide.

2.5 Methods for Process Evaluation Demonstration

A comprehensive and systematic approach (Saunders, et al., 2005) was used to develop the Project FIT process evaluation and related process evaluation questions. This paper focuses on process evaluation questions related to reach, dose and fidelity at facilitator and group levels (Table 2). Although the FIT process evaluation is designed for both formative and summative purposes, the main focus of the paper is on the formative process from cohorts 1-2 of the multi cohort trial. Results were summarized at the conclusion of both cohorts and shared with project investigators to inform ongoing program improvement. In addition, findings were incorporated into the facilitator training. Reflections based on the data and implications for program improvement are described in the discussion section.

Table 2.

Process Evaluation Questions

Process Component	Process Question
Fidelity	1) To what extent did the facilitators implement behavioral, motivational and parenting skill building essential elements as intended?
	2) To what extent does the group climate reflect behavioral, motivational and parenting skill essential elements as intended?
	3) To what extent did facilitators utilize cultural targeting strategies as intended?
Dose Delivered	1) To what extent did facilitators provide individualized feedback to participants?
	2) To what extent did facilitators provide group feedback?
	3) To what extent were intended curriculum content covered?
	4) To what extent did the facilitators explain the take home activities?
Dose Received	1) To what extent were present families engaged in intervention activities during group sessions?
	2) How fun and enjoyable did participants feel the program was?
Reach	1) How many participants were present at each session?
	2) What percentage of families enrolled were present at each session?

2.5.1 Attendance Tracking

Attendance forms are completed at the beginning of every intervention session by intervention staff to assess program reach. A priori criteria are set at ≥70%. To calculate attendance, participants are assigned a score based on their attendance for each week of the program (0=neither parent nor adolescent attended, 1=adolescent and/or parent attended). Participants self-reported reasons for absences were also recorded. Families who complete a make-up session prior to the next session (within 7-days of content delivery) are also assigned a score using the above rating scale. The average number and percentage of families per weekly session are calculated using the average frequency and proportion of families in attendance per week. The average number of sessions attended per family is calculated by taking the mean number of sessions attended across the 8-weeks for all families per group per cohort. The number of families completing ≥70% of the sessions is calculated by summing the percentage of families completing 6 or more of the 8 (≥75%) sessions. Scores are calculated without and with inclusion of make-up attendance data.

2.5.2 Observation Checklist

Observational data are collected by a trained, independent process evaluator using systematic observation. The observation checklist is designed to assess fidelity to essential elements in the M+FWL intervention at facilitator and group levels (Table 1; see electronic supplement for the complete checklist). Dose delivered of project components is also assessed. The process evaluator(s) attends all 8-weeks of the one and a half hour program.

Questions pertaining to dose are rated using a binary response (0=no, 1=yes). Each item represents a specific program component. The a priori goal was for program components to be delivered ≥75% of the time. The rating form includes 15-items pertaining to delivery of general program components, intervention specific program components, and behavioral and family skills identified in the facilitator's guide as key content. The a priori goal was for program components to be delivered ≥75% of the time.

Questions pertaining to fidelity are rated on a 4-point likert scale ranging from 1=none to 4=all. A priori criteria are set at ≥3 for fidelity. The observation checklist includes subscales that assess facilitator implementation of essential elements: behavioral skills (13-items), facilitator communication with participants (4-items), facilitator modeling positive communication (10-items), social support (6-items), autonomy support (3-items), self-efficacy (4-items), and session content (3-items). The tool also includes subscales related to fidelity to theoretical elements at the group level, including behavioral skills (4-items), communication skills (8-items), and group climate (4-items). Summary scores for each of the subscales are created by taking the mean of the items for each week, and then averaging across the 8-week intervention for an average fidelity score.

A formative process was used to develop a long term plan for assessing fidelity to cultural elements. As assessment tool was developed and refined during cohort 1 to more accurately capture delivery of cultural elements. The final assessment tool, which was put into use during cohort 2, includes both binary and descriptive items for identifying and describing cultural topics/issues as they arise during session. For each topic/issue identified, the following are rated: brought up by facilitator or participant, facilitator responded (yes=1, no=0) and group engaged in discussion related to the topic/issue (yes=1, no=0). Responses for each item are summed for each week of the intervention and a percentage value is created by identifying the percentage of responses where the facilitator responded and the group engaged in discussion. An overall percentage score is created for each cohort to provide descriptive ratings of facilitator responsiveness and group discussion. Results of delivery of cultural elements are summarized in the current manuscript for cohort 2 only given that the assessment tool was used in a formative way in cohort 1 and then was integrated into the final intervention curriculum.

2.5.3 Group Climate Survey

The Relatedness to Others in Physical Activity Scale (ROPAS) (Wilson & Bengoechea, 2010) was used to assess participants’ perceptions of the group climate. Participants responded to 6-items assessing respondents’ perception of meaningful connection and belongingness to other group members. Items are rated on a 6-pt likert scale and a priori criteria for summary scores are set at ≥4.5. An example item includes, “I have developed a close bond with others.” Surveys are completed anonymously by participants at weeks 4 and 8 of the intervention in the M+FWL group. Adolescents and parents complete surveys separately. Surveys were conducted with families who were present at the session. On the days that the surveys were distributed, average attendance ranged from 50.0% to 71.4%. Out of the families in attendance, 100% completed the survey. Summary scores are calculated by averaging responses both within each cohort and across cohorts. Specifically, responses for both weeks 4 and 8 are averaged to provide a mean rating of group climate for each cohort overall; ratings are also used to provide an average rating across cohorts. Reliability and validity for the instrument have been previously demonstrated in a sample of young adults (Wilson & Bengoechea, 2010). Reliability in the present study was high (α=.84).

2.6 Training of External Evaluators

Training for the observation checklist consists of three phases. In the first phase, candidates attend the intervention facilitator training. This training takes place across two, four-hour sessions. Trainings include an overview of the project, didactic training related to intervention implementation of essential elements and program components, and hands-on practice for delivering program content. The second phase of the training includes a didactic portion that is specific to the FIT process evaluation procedures. The didactic session lasts two hours and includes a review of process evaluation, an overview of the FIT process evaluation purpose and methods, and an in-depth review of the observation checklist. During the third phase, candidates rate an example session via voice recording. Ratings are then reviewed in person and compared with a gold-standard rating.

To assist with reliability of ratings for the observational checklist, evaluators are provided a rater dictionary. The rater dictionary provides detailed criteria for ratings each of the fidelity and dose items. For example, the item „facilitators provide feedback on use of self-monitoring tools’ is assessed with 1=Facilitators do not provide feedback on use of self-monitoring tools and 4=Facilitators provide feedback on use of self-monitoring tools as opportunities arise 90-100% of the time. Reliability of observational ratings was established in cohort 1 using percent agreement. Results indicated high reliability for dose items (100%) and acceptable reliability for fidelity subscales, with an average agreement of 92.4%.

2.7 Certification of Evaluators

Candidates complete two certification tasks for the observation checklist. The first task is to rate a second example session via voice recording. Candidates’ scores are compared to a criterion measure that is established by process evaluation leadership. In order to complete the first phase of certification, candidates must earn scores that are within at least 80% agreement or higher of the criterion. An agreement for dose items is defined as a perfect match between candidate's codes and the criterion codes. Agreement for fidelity items is defined as a match between candidates’ codes and criterion codes within 1 (+/−) rating on the 1-4 scale. In other words, a rating of 3 and 4 on the likert scale for fidelity ratings is considered a match, whereas a rating of 2 and 4 is not considered a match.

Candidates who pass the first phase of certification are invited to participate in the second phase of certification. In this phase, candidates rate the first week of each group session at the beginning of every cohort. A gold standard rater also attends the session with the candidate. Both the candidate and gold standard rate the session in person. To complete certification, the candidate must earn an agreement score with the gold standard that is at least 80% or higher; the same criteria listed above are used to establish percent agreement.

3. Results

3.1 Reach

A total of 39 parent-adolescent dyads were randomized to receive either the M+FWL intervention (n=19) or the comprehensive health education program (n=20). The current study focuses on the participants randomized to the M+FWL intervention. Results of program reach are displayed in Table 3. Results indicated that overall 71.5% of the families were in attendance on average when make-up sessions were included in calculations; this value met the goal of ≥70%. Furthermore, 63.2% of families attended 6 or more sessions, with an average of 5.8±1.8 sessions attended per family.¹ Reasons for absences include scheduling conflicts (e.g., school conflicts, work conflict), transportation issues (e.g., not able to get a ride, unable to pay for bus transportation), family emergencies (e.g., death in the family, participant recovering from surgery) and reasons unknown.

Table 3.

Participant Attendance in the Motivational Plus Family Weight Loss Intervention

	Cohort 1	Cohort 2	Total
Total Parent-Teen Dyads (N)	7	12	19
Make-up Not Included
Average Families per Session, N (%)	4.5 (64.3%)	7.5 (62.5%)	12 (63.2%)
Attended ≥6 Sessions, N (%)	4 (57.1%)	6 (50.0%)	10 (52.6%)
Average Number of Sessions Attended per Family, M (SD)	5.1 (1.9)	5.0 (2.3)	5.1 (2.1)
Make-up Included
Average Families per Session, N (%)	5.1 (73.2%)	8.5 (70.8%)	13.6 (71.5%)
Attended ≥6 Sessions, N (%)	5 (71.4%)	7 (58.3%)	12 (63.2%)
Average Number of Sessions Attended per Family, M (SD)	5.9 (1.3)	5.7 (2.3)	5.8 (1.8)

Note: A total of 39 parent-adolescent dyads enrolled in the program overall. Values above are presented for the Motivational Plus Family Weight Loss intervention program only. In cohort 1, both parent and teen were present for each dyad for each session 96.9% of the time. In cohort 2, both parent and teen were present for each dyad for each session 99% of the time.

3.2 Dose

3.2.1 Dose Delivered – Facilitator Delivery of Program Components

Results from external observation checklist indicated that 14 out of the 15 dose items were delivered 100% of the time. The one exception was that family skills (which are specific skills identified for each weekly session, e.g., push-pull language, social support) were discussed during session 78.6% of the time during cohort 2. Findings indicate that dose delivered exceeded the a priori goal of ≥75% for all program components.

3.2.2. Dose Received – Group Climate

A total of 47 surveys were completed across cohort 1 (n=19) and cohort 2 (n=28). The average overall rating for perceived group climate was 5.16 (SD=.69), with parents rating (M=5.28, SD=.74) and adolescents rating (M=5.04, SD=.63) both above the a priori rating of ≥4.5. Adequate dose received was also achieved for each cohort separately, with ratings ≥4.5 for both cohort 1 (M=5.09, SD=.79) and cohort 2 (M=5.21, SD=.63).

3.3 Fidelity

3.3.1 Facilitator and Group Fidelity to Motivational, Behavioral and Family Elements

Results of facilitator and group fidelity to motivational, behavioral, and family elements are presented in Table 4. Results revealed adequate facilitator delivery of essential elements (M=3.63, SD=0.41). Ratings were only slightly lower for delivery of behavioral skills (M=3.34, SD=0.46) and social support (M=3.32, SD=0.41) as compared to other variables. Fidelity at the group level was also adequate overall (M=3.35, SD=.49). Notably, ratings were only slightly lower for behavioral skills (M=3.16, SD=0.62) than for communication (M=3.47, SD=0.30) and group climate (M=3.40, SD=0.47). These findings suggest adequate adherence to FIT theoretical essential elements at facilitator and group levels.

Table 4.

Fidelity of M+FWL^a Intervention Implementation, M(SD)^b

	Cohort 1	Cohort 2	Totalc
Facilitator Level
Behavioral Skills	3.32 (.59)	3.36 (.33)	3.34 (.46)
Facilitator Communication with Participants	3.69 (.35)	3.91 (.13)	3.80 (.28)
Facilitator Modeling Positive Communication	3.88 (.10)	3.92 (.15)	3.90 (.13)
Social Support	3.02 (.35)	3.63 (.19)	3.32 (.41)
Autonomy Support	3.54 (.40)	3.79 (.17)	3.67 (.32)
Self-Efficacy	3.18 (.49)	3.69 (.35)	3.45 (.48)
Session Content	3.96 (.12)	3.92 (.15)	3.94 (.13)
Average Facilitator Level Fidelity	3.51 (0.48)	3.74 (.29)	3.63 (.41)
Group Level
Behavioral Skills	3.48 (.69)	2.88 (.42)	3.16 (.62)
Communication Skills	3.56 (.32)	3.39 (.26)	3.47 (.30)
Group Climate	3.69 (.27)	3.10 (.45)	3.40 (.47)
Average Group Level Fidelity	3.58 (.44)	3.12 (.43)	3.35 (.49)

^aMotivational plus Family Weight Loss

^bItems rated on a 1-4 pt likert scale, 1=none and 4=all. Fidelity criteria are set at a rating of ≥3.

^cTotal calculated by averaging across cohorts.

3.3.2 Fidelity to Cultural Elements

Results of coding for cultural elements for cohort 2 indicated that two main themes were raised across the 8-week sessions with an average of 0.63 topics raised per session. Themes were related to eating (n=3) and physical activity (n=2) behaviors. Eating topics included making kids clean their plate: “[Participant described] a cultural-generational need to make kids finish their plate because of the historical influence/fact that parents have spent their money.” Two other examples were related to eating behaviors and church; for example, a parent participant raised the challenge of eating after church: “Dad shared that family [eats a lot] after church.” Another participant identified eating fried chicken at church as a barrier to maintaining weight loss skills, and suggested that they would try to drink a shake before church to overcome this challenge. Participants also shared that church is a place where they engage in physical activity. One parent participant noted that they engage in activities with their chorus group: “[Participant] mentioned using church as a resource for physical activity/praise dancing.” All topics (100%) were raised by participants (vs. facilitators) and facilitators responded 87.5% of the time.

4. Discussion

The present study described how process evaluation was used to assess implementation of the Families Improving Together (FIT) for weight loss randomized controlled trial. Motivational, behavioral, and family theoretical elements and cultural tailoring principles guided the assessment of reach, dose and fidelity across facilitator and group levels. Results from cohorts 1-2 provide important formative insight for program improvement. Evaluation results indicated that although adequate program reach was obtained overall, attendance was somewhat challenging across the two cohorts and make up sessions were helpful in terms of increasing reach and dose. Facilitator dose delivered and fidelity to theoretical elements met implementation criteria, suggesting complete and acceptable delivery of the intervention at the facilitator level. Results at the group level demonstrated adequate dose received. Group fidelity was also adequate, though findings indicate that adhering to theoretical elements at the group level was slightly more challenging than at the facilitator level. Assessment of cultural elements revealed participants raised a range of topics/issues related to weight loss and skill development. These results were used to inform refinement of program materials to improve facilitator delivery of cultural strategies. This study furthers understanding of challenges related to attendance among underserved families, and suggests that flexibility in intervention delivery through integrating make up sessions may support relationship development and increase reach and dose delivered. Furthermore, results highlight the importance of examining implementation at facilitator and group levels. Additionally, the assessment of cultural tailoring is novel and may inform future evaluations of culturally-targeted interventions. The framework used in this study may have important implications for the implementation of complex weight loss interventions in real world settings.

Results from the present study indicate that make up sessions improved program reach and may be a method for increasing reach in underserved populations that have barriers. Previous research that has found lower attendance rates among low income, minority populations in general (Befort, et al., 2008; Budd, et al., 2007; Jensen, Aylward, & Steele, 2012; Kumanyika & Obarzanek, 2003; Resnicow, et al., 2000). Resnicow and colleagues (2005) found that attendance rates ranged from 57-66% in a 6-week culturally tailored, behavioral group intervention for overweight African American adolescent females. Another study demonstrated that monthly household income was a predictor of session attendance in a 10-week family-based behavioral weight management program; specifically, attendance rates were 64.5% of average but results indicated that lower income families attended significantly fewer sessions (Jensen, et al., 2012). Findings from this study indicated that families missed sessions for a variety of reasons, including transportation issues, family emergencies and scheduling conflicts and that make up sessions were well received as a feasible option for overcoming barriers. To reduce barriers to participation experienced among low income, minority populations, Project FIT utilized a number of strategies, including a brief, face-to-face group based program and make-up sessions conducted in the event that families were absent. In addition, a 2-week run in period allowed for those with barriers to drop prior to participation in the intervention. Reminder calls, door prizes, and the availability of child care were also used to increase attendance. A lesson learned from this research is that using flexible strategies designed to reduce barriers experienced by underserved populations may support the development of positive relationships between research staff and participating families. For example, make-up sessions provide families with an opportunity to receive program materials in the event that they are unable to make it to session and this may foster a stronger bond between facilitators and participants. Consistent with past research, the FIT trial integrates addressing socio-cultural facilitators and barriers to program participation among African American families (Kumanyika, et al., 2007).

This study examined facilitator delivery of multi-theoretical program components related to building a positive social climate for weight loss. Results demonstrated high facilitator dose delivered, suggesting complete delivery of program components. Further, results indicated that adequate fidelity was achieved at the facilitator level. It is important that facilitators provide support related to behavioral skills in the context of a positive social climate that promotes choice and support, given that it is the integration of these factors that leads to the development of nurturing environments for promoting weight loss (Biglan, Flay, Embry, & Sandler, 2012; Ryan & Deci, 2000). However, previous research suggests that implementing both behavioral and motivational elements may be challenging, especially when working with overweight/obese youth who may need additional support for developing weight loss skills (Robbins, et al., 2014). This suggests that facilitators may need additional support for navigating the implementation of SDT, FST, and SDT elements. Using the results from this process evaluation, the training for Project FIT facilitators was enhanced to include role plays where facilitators have the opportunity to practice using choice and support while providing feedback on the use of behavioral and family skills.

The assessment of dose and fidelity at the group level is novel and sheds light into the complexities related to building a positive social climate for weight loss. Conceptually, a positive social climate is one where group members reinforce one another for engaging in positive behaviors and provide support for reducing barriers (Biglan, et al., 2012). Such a context promotes an environment where members feel accepted and a sense of belonging; this builds intrinsic motivation for successful lifestyle change (Ryan & Deci, 2000). In the context of the FIT trial, a positive group climate is defined as one where group members encourage one another, provide support for overcoming barriers to developing behavioral and family skills, use positive communication strategies, and share personal stories related to weight loss. Results from the process evaluation indicated participants’ perception of the group climate was positive, suggesting adequate dose received at the group level. Fidelity at the group level was also adequate overall, although ratings decreased from cohort 1 to cohort 2. Coaching and booster trainings are now implemented weekly to indicate problems with fidelity.

Overall, the study shows that the group fidelity ratings were somewhat more complex and can be achieved with high fidelity although not quite at the level of the facilitator level fidelity ratings. Complex interventions (i.e., those involving a number of theoretical components) interact with and change in response to the context in which they are implemented (Hawe, Shiell, & Riley, 2004; Shiell, Hawe, & Gold, 2008). As a result, like in the FIT trial, delivery of program essential elements should be monitored using a systems approach that captures how implementation may differ depending on how the intervention interacts with contextual influences. In Project FIT, assessing implementation at both facilitator and group levels revealed that fidelity to essential elements at the group level may be more challenging that at the facilitator level and indicate a need for supporting adherence to theoretical elements within the group context. Further, the examination of reach at the family level provides an added layer of understanding contextual influences on implementation. Supporting implementation at the group level is a challenge given that approaches used at the facilitator level may not be appropriate. Alternatives for supporting fidelity at the group level may include incorporating additional opportunities for group interactions (e.g., include more games) and increased reinforcement of social support among group members. The Project FIT process evaluation will allow for documentation of strategies for supporting the group climate, which may provide guidance to future group-based weight loss programs. To our knowledge, this is one of the first studies to assess fidelity to theoretical elements for developing a positive social climate at the group level.

Findings may also aid in the identification of strategies for implementing culturally-tailored weight loss interventions. The approach to cultural targeting at the group level used in FIT is consistent with a socio-cultural („deep-structures’) approach where health issues are integrated in the context of broad social and cultural values (Kreuter, et al., 2003; Wilson, 2009). Process evaluation is used to assess implementation of cultural elements in terms of the number and quality of topics raised. Though there is no basis for comparison available, it was surprising that only 5 topics were raised and all were raised by participants. This may suggest that facilitators did not use the prompts for cultural tailoring in the facilitators’ guide. Thus, one strategy for increasing fidelity to cultural elements may be to provide additional support related to utilizing cultural prompts. In response, findings from the present study and previous literature were used to revise facilitators’ guides to include additional cultural themes. Specifically, we integrated cultural themes into a final set of comprehensive issues that addressed both diet related traditions and issues related to sedentary behavior and physical activity in African American youth. For example, the final guides address how foods with special meaning fit within a healthy lifestyle, how to prevent relapse during special occasions (e.g., church meals, family pot lucks, holidays), how to address the high prevalence rate of sedentary behavior in ethnic minority youth, discussion of emotional eating, pillar syndrome (i.e., role of women as caregivers for the family) and hairstyle during physical activity.

Results from the fidelity to cultural tailoring demonstrate how process evaluation may be used in a formative manner to enhance the delivery of culturally relevant and evidence-based intervention strategies. Notably, several prompts related to integrating culturally relevant foods (i.e., foods with special meaning) and food at special events (e.g., church buffets) within a healthy lifestyle were added given that cultural food was a frequent topic raised across weekly sessions. This is consistent with past qualitative research which indicates that food traditions are a barrier to weight loss among African American women (Allicock, et al., 2012). Importantly, these results only provide a snapshot of delivery of cultural tailoring elements. Continued evaluation may provide a more rich understanding of the processes for implementing culturally tailored interventions. Although previous health promotion (Budd, et al., 2007; Story, et al., 2003; Wilson, et al., 2011) and weight loss programs (Kitzman-Ulrich, et al., 2011; Wilson, 2009) for youth have integrated culturally relevant intervention strategies, to our knowledge this is the first study to assess delivery of cultural elements for youth weight loss directly. Findings may guide the development of future studies aimed at providing a culturally-appropriate context for weight loss and healthy lifestyle change among minority families.

Overall this paper provides an approach to process evaluation that may support the implementation of climate-based weight loss programs. As programs become increasingly more complex (Hawe, Shiell, & Riley, 2009; Rimer & Glanz, 2005), they become more challenging to implement. Technical assistance is an important tool that can be used to maintain quality implementation through individualized coaching (Wandersman, Chien, & Katz, 2012). In the FIT trial, results from the process evaluation were used in a formative manner to provide ongoing feedback and weekly coaching to facilitators. A similar approach could be used to provide ongoing feedback to providers interested in implementing the FIT trial in real world contexts, such as community centers or health clinics. The FIT process evaluation will also help to identify which essential elements are most strongly linked with weight loss in underserved youth. Results from the FIT process evaluation may be used at the conclusion of the study to examine the mechanisms through which intervention elements impact weight related outcomes. This could inform translation of the program by indicating which elements should be implemented with high fidelity and which may be adapted depending on context and organizational capacity (Durlak & DuPre, 2008). Previous health promotion programs for youth have conducted similar evaluations for PA (Saunders, Ward, Felton, Dowda, & Pate, 2006). The FIT trial will expand the literature by examining the relationship between implementation and program outcomes in the context of a weight loss trial for underserved African American families.

The study has some limitations. The results of the process evaluation were useful for informing program improvements within the FIT trial, but generalizability to other studies is limited given that data were only presented from two cohorts of a positive climate-based approach. Additionally, though a rigorous approach was used to develop items for the observational measure future research is needed to validate that these constructs are predictive of intervention outcomes. Despite these limitations, this is one of the first studies to include a multilevel process evaluation approach for assessing program implementation at facilitator and group levels. The assessment of cultural tailoring is also novel, and may inform the development of culturally relevant intervention strategies.

5. Lessons Learned

Several lessons may be gleaned from the FIT process evaluation. First, the use of a collaborative, systematic planning process was helpful in translating theoretical elements into process evaluation components. Though the process was time consuming, it was useful for developing the process evaluation methods as well as guiding the intervention planning process. Second, the rigorous certification process for external evaluators resulted in high reliability of observational assessments. The development of a detailed rater dictionary and weekly meetings with the process evaluation team were also critical for maintaining high reliability. Third, assessing culture was a challenge and requires highly trained evaluators. Advanced training related to identifying and describing cultural topics/issues may be important for future studies. Finally, given the complex nature of this study, utilizing the data for formative use in a timely fashion was challenging. An important next step for future studies is to identify innovative methods (e.g., use of electronic rating forms) that could be used to navigate this type of program evaluation.

6. Conclusions

This study demonstrated how process evaluation is used to assess implementation of the Families Improving Together (FIT) for Weight Loss Trial. A multilevel, theory-based process evaluation was used to assess delivery of multi-theoretical elements across individual/family, facilitator and group levels. Findings from this study underscore the challenges associated with attendance among minority families and suggest that make-up sessions may be a useful strategy for increasing reach. Complete and acceptable delivery was achieved at the facilitator level. Participant perceptions of the group climate were positive which suggests adequate dose received; however, maintaining fidelity at the group level was somewhat more challenging. Future research is needed to identify strategies for maintaining a positive group climate while targeting the development of behavioral and family skills. The assessment of cultural elements is a novel aspect of this evaluation and may be used to improve delivery of deep-structure cultural strategies related to weight loss. Overall, this study makes an important contribution to research on weight loss interventions for minority families by identifying strengths and potential areas for improvement across multiple systems. The process evaluation framework used in Project FIT may support implementation of future weight loss interventions for underserved families who experience unique challenges to participation and weight loss. Specifically, data collected through this evaluation may inform translation of evidence-based practices by providing feedback for program improvement and uncovering the mechanisms linking program implementation with weight loss outcomes.

Highlights.

• We present a multilevel, theory driven process evaluation of a weight loss program for African American families.

• Make-up sessions and flexible program components may help overcome barriers to attendance experienced by low income, ethnic minority populations.

• Implementing behavioral and motivational program elements may require regular coaching for program facilitators.

• Fidelity at the group level is distinct from fidelity at the facilitator level.

• We provide a model for evaluating implementation of cultural program elements and insights may be used to develop culturally relevant intervention strategies.

Biography

Kassandra A. Alia, M.A. is a doctoral student in the Clinical-Community Psychology Ph.D. program at the University of South Carolina. Ms. Alia is also the Process Evaluation Coordinator for the Families Improving Together (FIT) for Weight Loss Trial and is the lead author on this manuscript.

Dawn K. Wilson, Ph.D. is a Professor of Psychology at the University of South Carolina and is the Principle Investigator on the FIT Trial. Dr. Wilson recently served as the President for the Society of Behavioral Medicine and was awarded the Senior Scientist Award from Division 38 for Health Psychology for the American Psychological Association.

Tyler McDaniel, M.A., is a doctoral student in the Clinical-Community Psychology Ph.D. program at the University of South Carolina. Mr. McDaniel is a research assistant with the FIT Trial and is a leader on the process evaluation component for the trial.

Sara M. St. George, Ph.D., is a post-doctoral fellow at the University of Miami in the Department of Public Health Sciences. Ms. St. George was instrumental in developing the presented process evaluation plan. She was recently awarded the Outstanding Dissertation Award from the Society of Behavioral Medicine.

Heather Kitzman-Ulrich, Ph.D., is an Assistant Professor at the University of North Texas in the Department of Behavioral & Community Health. Dr. Kitzman-Ulrich is a co-investigator on the FIT Trial and was instrumental in developing the process evaluation methods presented in the current manuscript.

Kelsey Smith, M.Ed. is a doctoral student in the Clinical-Community Psychology Ph.D. program at the University of South Carolina. Ms. Smith has served as a process evaluator for the project and provided critical support in the development of this manuscript.

VaShawn Heatley is the Intervention Coordinator for the FIT trial. As the Intervention Coordinator, Ms. Heatley has been intimately involved in the process evaluation and has provided critical oversight in the weekly coaching sessions provided to project facilitators.

Courtney Wise is a graduate student in Chemistry at the University of South Carolina. Ms. Wise is a research assistant on the FIT trial and provided critical support in the data analyses presented in the current manuscript.