Scaling a behavioral weight‐loss intervention for individuals with serious mental illness using the enhanced replicating effective programs framework: A preconditions phase proof‐of‐concept study

Abstract

Background

Given the obesity's high prevalence among individuals with serious mental illness (SMI), translating weight‐loss interventions with demonstrated effectiveness is needed. This study describes the initial translation phase of such an intervention using the Enhanced Replicating Effective Programs (REP) Framework for delivery by mental health program staff.

Methods

The Achieving Healthy Lifestyles in Psychiatric Rehabilitation (Achieving Healthy Lifestyles in Psychiatric Rehabilitation) trial intervention was preliminarily adapted to create the ACHIEVE‐Dissemination (ACHIEVE‐D) curriculum. A treatment‐only study was conducted to rapidly evaluate the curriculum using a mixed‐methods approach including surveys and focus groups. A study coach delivered an abbreviated curriculum to individuals with SMI from a single psychiatric program. Among all participants with SMI (n = 17), outcomes were attendance and satisfaction; 14 participated in a focus group. The program staff observed curriculum delivery and participated in a focus group (n = 3).

Results

Overall, 23 group sessions were delivered. Median attendance was 78.6% across participants with SMI; 92.9% would recommend ACHIEVE‐D to others. The staff found the curriculum acceptable, particularly its structured nature, inclusion of weight management and exercise, and integrated goal setting and tracking. These improvements recommended by participants and/or staff were to assess participant readiness‐to‐change prior to enrollment, change the frequency of weigh‐ins, and train staff coaches on anticipated challenges (e.g., exercise engagement, weight fluctuations).

Conclusions

During this first REP phase, individuals with SMI and program staff were satisfied with ACHIEVE‐D. Additional refinements will aid future implementation and improve participant experience.

This study is an example of an application of the pre‐conditions phase of the Enhanced Replicating Effective Programs (REP) Framework, specifically within the community mental health setting. We used the Systems Engineering Initiative for Patient Safety (SEIPS) Model to leverage experience from the original trial team members to identify implementation barriers and solutions that were then integrated in a preliminary strategy for adapting the evidence‐based practice. We describe implementation challenges unique to the community mental health setting and identify solutions to these challenges by engaging participants with serious mental illness and mental health center staff.

1. INTRODUCTION

Among people with serious mental illness (SMI), the prevalence of obesity is significantly elevated relative to the general population, ¹ , ² , ³ and correspondingly, this population has a high prevalence of other cardiometabolic risk factors. ⁴ , ⁵ , ⁶ , ⁷ For example, one study found that 29% of men and 60% of women with SMI had obesity, which was higher than the prevalence of obesity among matched samples without SMI (men without SMI: 18%; women without SMI 29%). ¹ People with SMI have documented excess mortality due to heart disease, cerebrovascular disease, and diabetes mellitus. ⁸ , ⁹ , ¹⁰ Research has found that adults with schizophrenia were more than 3.5 times as likely to die compared adults in the general population during the same follow‐up period, and cardiovascular disease had the highest mortality rate among adults with schizophrenia (approximately 400 per 100,000 person‐years). ⁹

Several factors may contribute to the high prevalence of obesity among the SMI population. Many individuals with SMI take at least one long‐term psychotropic medication that potentially promotes weight gain and metabolic abnormalities, such as hyperglycemia and dyslipidemia. ¹¹ , ¹² , ¹³ However, these medications may not be modifiable for many patients managing SMI. Research has also found that individuals with SMI have low socioeconomic status (SES) with disparities in educational and employment outcomes as well as income, ¹⁴ and low SES has been associated with increased risk of obesity through mechanisms such as increased psychosocial stress, food insecurity and neighborhood environments unconducive to physical activity. ¹⁵ Lifestyle behaviors, including diet and physical activity, are typically poor in this population, ¹⁶ , ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ but represent a modifiable factor that could reduce cardiometabolic disease risk.

In general populations, behavioral weight‐loss interventions have been demonstrated to improve cardiometabolic risk factors, such as reducing blood pressure and glucose, as well as improving lipid profile. ²² Individuals with SMI critically need behavioral interventions promoting a healthy diet, physical activity and weight loss, but interventions should be adapted for this population given prevalent challenges related to memory impairment and limited executive function. ²² , ²³ Low mood, stress and lack of support have been identified as barriers to exercise participation in this population. ²⁴ However, a recent systematic review identified key components of effective behavioral weight‐loss interventions among persons with SMI, including regular contact, tools to support behavior change and tailored materials. ²⁵

The Achieving Healthy Lifestyles in Psychiatric Rehabilitation (ACHIEVE) randomized controlled trial (RCT) tested a behavioral weight‐loss intervention for persons with SMI and demonstrated clinically significant weight loss at 18 months. ²⁶ , ²⁷ The ACHIEVE RCT took place in psychiatric rehabilitation programs (PRPs), which provide community‐based comprehensive rehabilitation and recovery services, support and promote successful community integration, as well as facilitate the use of community resources. ²⁸ The PRP setting also offers resources to address food insecurity and provide safe spaces for exercise. In this RCT, trained interventionists led group and individual weight‐management sessions as well as group exercise sessions in PRPs. The ACHIEVE intervention included all aforementioned key intervention components ²⁵ by providing frequent contacts, opportunities for group interaction and social support, goal setting and self‐negotiation, problem solving, and examples of new behavioral options. ²⁶ In particular, the intervention focused on repeating six key messages: avoid sugary drinks, avoid junk food, eat five servings of fruits/vegetables per day, portion control, smart snacking habits, and regular physical activity. This messaging approach was based upon prior successful didactic interventions among individuals with schizophrenia that emphasized learning a few, specific and narrow skills repeatedly, breaking material into small units, repetition of content, and rehearsing behavioral skills. ²⁹ Given that the ACHIEVE intervention is effective and includes components key for success, there is an urgent need to translate ACHIEVE to similar settings and populations as those included in the trial, which could be accomplished by enabling the PRP staff to deliver the sessions. ³⁰

The Enhanced replicating effective programs (REP) Framework is a model for rapidly translating evidence‐based interventions into community settings. ³¹ It includes four stages: pre‐conditions, pre‐implementation, implementation, and maintenance/evolution. In the pre‐condition phase, REP focuses on the identification of need, effective interventions, and implementation barriers, as well as drafting an intervention package. In the pre‐implementation phase, REP focuses on obtaining stakeholder input to adapt the intervention and implementation strategies for community and pilot testing. The implementation phase focuses on training, support, evaluation and refinement, while the maintenance/evolution phase focuses on supporting sustained implementation and dissemination.

In this article, a description is first provided of the REP pre‐conditions process of translating the ACHIEVE intervention to a curriculum appropriate for staff delivery within community PRP settings (phase 1). Then, applying the Obesity‐Related Behavioral Intervention Trials (ORBIT) model of using treatment‐only design to rapidly evaluate a behavioral intervention, ³² this curriculum was preliminarily tested among persons with SMI to determine acceptability to individuals with SMI, feasibility and acceptability to PRP staff, as well as potential implementation barriers perceived by individuals with SMI and PRP staff (phase 2).

2. MATERIALS AND METHODS

2.1. Phase 1 Activities: REP pre‐conditions process for translating Achieving Healthy Lifestyles in Psychiatric Rehabilitation (ACHIEVE) intervention

The REP pre‐conditions phase includes identification of need and effective intervention, identification of barriers, and drafting an intervention package. Table 1 provides an overview of the elements of the REP pre‐conditions phase and corresponding activities. To expand this evidence‐based intervention to similar settings and populations as those in the trial, an adaptation of the ACHIEVE intervention was planned with the goal of enabling the PRP staff to deliver sessions rather than research team interventionists. ³⁰ The GUIDED checklist for intervention development is provided in Supplemental Materials 1. ³³

TABLE 1.

Replicating Effective Programs (REP) pre‐conditions phase elements* & corresponding activities by study phase.

	Phase 1: Draft curriculum process	Phase 2: Proof‐of‐concept study
Identify need:	• High prevalence of obesity & obesity‐related morbidity among persons with SMI	‐‐
Identify effective intervention:	• Behavioral weight‐loss intervention from ACHIEVE randomized controlled trial	‐‐
Identify barriers:	• ACHIEVE trial team	• Participants with SMI • Psychiatric rehabilitation program staff
Draft package intervention:	• Design the ACHIEVE‐Dissemination curriculum	• Preliminarily test the curriculum

Note: *Pre‐conditions phase elements from the Enhanced Replicating Effective Programs Framework. ³¹

Abbreviations: ACHIEVE—Achieving Healthy Lifestyles in Psychiatric Rehabilitation trial, SMI—serious mental illness.

Researchers typically identify a high‐burden condition as the need and an intervention previously tested in a RCT to address this condition. ³¹ People with SMI have a high prevalence of obesity, which contributes to cardiometabolic disease and excess mortality. ¹ , ² , ³ , ⁴ , ⁵ , ⁶ , ⁷ , ⁸ , ⁹ , ¹⁰ Therefore, obesity treatment is a need among this population. The behavioral weight‐loss intervention tested in the ACHIEVE RCT was effective among this population in PRP settings. ²⁷ Therefore, ACHIEVE is an effective intervention to treat obesity among individuals with SMI.

Barriers to implementation are identified by researchers and representatives from practice sites. ³¹ Barriers were identified by researchers during phase 1, while phase 2 identified barriers among participants with SMI and PRP staff (see below). A 2‐h discussion group with 5 research team members from the ACHIEVE trial was conducted to reflect upon challenges experienced during this study, identify anticipated challenges for PRP staff who would be delivering the future curriculum, and brainstorm strategies to address the challenges. These ACHIEVE research team members (study interventionists, managers and coordinators) offered perspectives of challenges across the 10 PRPs that participated in this trial. As the ACHIEVE trial ended in 2011, it was infeasible to recruit participants with SMI from the original study. A current study team member with expertise in human factors engineering, who was not involved in the original trial, facilitated this discussion. Using the Systems Engineering Initiative for Patient Safety (SEIPS) Model, ³⁴ the research team's perceived work system factors, including both facilitators and barriers, that influenced ACHIEVE implementation were examined (Figure 1). Key challenges were identified (bold items in Figure 1) along with corresponding strategies to address these barriers.

FIGURE 1.

Factors Influencing ACHIEVE‐D Implementation by psychiatric rehabilitation program (PRP) Staff & Strategies to Address These Challenges. This figure displays the application of findings to the Systems Engineering Initiative for Patient Safety (SEIPS) Model ³⁴ from Achieving Healthy Lifestyles in Psychiatric Rehabilitation (ACHIEVE) trial research team members regarding perceived challenges for the ACHIEVE‐D curriculum across different PRPs along with potential strategies to address these challenges. Items identified in bold font indicate key challenges to address along with their related potential solutions as indicated by numbers within brackets. All potential strategies to address challenges are listed in numeric order in the gray‐shaded box.

Based on the need, effective intervention and barriers, a draft package is created by intervention developers. ³¹ Using the REP framework and common steps reported across adaptation frameworks, ³¹ , ³⁵ the study team's intervention developers (GJJ, ATD, SG) adapted the ACHIEVE trial intervention to a draft curriculum called “ACHIEVE‐Dissemination” (ACHIEVE‐D) for the PRP setting. The ACHIEVE‐D curriculum retained the simplified messaging, repetition of key program elements, and dividing information into small components from the ACHIEVE trial, as this tailoring addresses deficits in memory and executive function that are common among individuals with SMI and was effective in the previous RCT. The curriculum has one lesson each week that is repeated three times over the week. As the curriculum was designed for the PRP setting, it leverages PRP resources to address food insecurity and provide safe spaces for exercise similar to the ACHIEVE RCT. Another overarching principle of the curriculum design was to fit into the current PRP staff workflow and regular client programming that PRPs are already providing. To address this issue, group sessions that addressed both weight management and exercise were created along with facilitator guides, to‐do lists, scripts and videos to assist delivery. Table 2 provides a comparison of the key features of ACHIEVE and ACHIEVE‐D.

TABLE 2.

Comparison of Achieving Healthy Lifestyles in Psychiatric Rehabilitation (ACHIEVE) intervention in the randomized controlled trial (RCT) to the ACHIEVE‐D curriculum adapted for delivery by psychiatric rehabilitation program (PRP) staff.

	ACHIEVE intervention	ACHIEVE‐D curriculum
Duration	18 months	6 months
Group sessions	2 group types: Weight management—45‐min class once a week with weekly weigh‐in Group exercise—moderate intensity, 50‐min class three times a week	1 group type: • Multi‐purpose group weight management and exercise—45‐min class three times a week with weigh‐ins (17 min of weight management; 20 min of moderate intensity exercise; 8 min for weigh in)
Individual sessions	15‐min weight management visit once a month	None
Total time	~795 min per month	~540 min per month
Delivery modality of components	Weight management: Led by trained interventionist Exercise: Led by trained interventionist or video‐assisted	Weight management: Led by PRP staff member assisted with short videos and facilitator guide Exercise: Video‐assisted
Facilitators	Trained interventionist	Trained PRP staff member and peer leader
Behavioral messages	6 messages: Weight loss success Avoiding sugar drinks Avoiding junk food Eat smart portions Eat more fruits & vegetables Smart snacking	6 messages: Weight loss success No sugar drinks No junk food Eat smart portions Eat more vegetables Putting it all together
Goal	10‐Lbs weight loss in 18 months (tailored to individual)	5‐Lbs weight loss in 6 months (tailored to individual)

Abbreviation: ACHIEVE, Achieving Healthy Lifestyles in Psychiatric Rehabilitation trial.

2.2. Phase 2 Activities: Proof‐of‐concept study design

The National Institutes of Health and ORBIT consortium has recommended the ORBIT model, which uses a treatment‐only design to rapidly evaluate a behavioral intervention. ³² A 2‐month non‐randomized proof‐of‐concept study tested the ACHIEVE‐D curriculum at one PRP in Maryland (registered on clinicaltrials.gov, NCT03999892). Sample size for proof‐of‐concept studies within the ORBIT model is typically small and selected from accessible subjects, as the goal is to efficiently determine whether the treatment merits more rigorous and costly testing. ³² Therefore, the study aimed to have at least 10 individuals with SMI and at least 3 staff observers participate, as we anticipated this goal would be feasible and likely to meet the study goals. The Johns Hopkins University School of Medicine Institutional Review Board approved this study (IRB00194122).

A trained health coach from the study team delivered the first 2 months of the 6‐month curriculum (modules on “weight loss success” and “no sugar drinks”) to a group of individuals with SMI from one PRP. Psychiatric rehabilitation program staff observed curriculum delivery by attending the group session led by the study team coach (PRP staff had no interaction with participants with SMI). Supplemental Figure 1 displays study activities (July‐August 2019). The CONSORT checklist for pilot or feasibility trials for this study is provided in Supplemental Materials 2.

A mixed‐methods approach was used for evaluation. Several types of data were gathered from participants with SMI: 1) attendance at each session to determine engagement, 2) survey to assess satisfaction, and 3) a focus group to determine acceptability and identify barriers. A focus group with PRP staff observers was also conducted to identify barriers as well as determine feasibility and acceptability. Video recordings were used to determine coach fidelity to the curriculum and identify challenges in delivery. All data were collected onsite at the PRP by study team members. Additional information about these data sources and analyses is described below.

2.3. Phase 2 Activities: Participants with SMI—Data collection & Analysis

For individuals with SMI, the study recruited participants through announcements at routine program meetings and flyers. Individuals were eligible if they were aged 18 years or older, expected to receive care at the PRP for at least 6 months, were able to attend the ACHIEVE‐D classes 3 days per week, had body mass index (BMI) ≥ 25 kg/m², and were interested in losing weight. People with underlying medical conditions for which dietary change, physical activity, and/or weight loss may be contraindicated or require medical supervision (e.g., lung disease, angina, pregnancy) were excluded. The 2018 Physical Activity Readiness Questionnaire (2018 PAR‐Q+) and the American College of Sports Medicine Exercise Pre‐participation Health Screening recommendations were used to identify individuals in need of medical clearance prior to participation in moderate exercise. ³⁶ Baseline data including demographics as well as measured height, weight, and waist circumference were collected.

To evaluate the acceptability of ACHIEVE‐D to individuals with SMI, the study examined attendance and satisfaction with the curriculum. Session attendance was recorded by the study team at each group session and median attendance was calculated. At study end (2‐month follow up), participants completed an adapted 12‐item satisfaction questionnaire, ³⁷ indicating their agreement to a statement with higher scores indicating greater satisfaction (1—disagree strongly to 6—agree strongly) (Supplemental Materials 3). Mean responses for each item were calculated. A 90‐min focus group was conducted in‐person at the study end (n = 14). A trained moderator facilitated the group using a semi‐structured guide (Supplemental Materials 4) and an assistant moderator recorded notes. The moderator inquired about satisfaction with the curriculum, feedback on format, understandability of content, and likability of intervention materials, as well as challenges they experienced. The group was audio‐recorded and transcribed verbatim by a professional transcription service, and transcripts were checked for accuracy by a study team member. A thematic analysis approach was used to understand the experiences across participants. ³⁸ Two study team members completed a 6‐step process: data familiarization, initial code generation, search for themes (i.e., repeated patterns), review themes, define/name themes, and produce a report. Themes were discussed among the entire team of investigators.

2.4. Phase 2 Activities: psychiatric rehabilitation program staff participants—Data collection & Analysis

The study recruited staff participants from the program through announcements at staff meetings and at the recommendation of the program director. Individuals were eligible if they had observed at least 2 sessions of ACHIEVE‐D curriculum delivery. The study aimed to determine the feasibility and acceptability of ACHIEVE‐D to PRP staff. At the end of the study, a 90‐min focus group was conducted in‐person (n = 3). A trained moderator facilitated the group using a semi‐structured guide (Supplemental Materials 5) and an assistant moderator recorded notes. The moderator explored the perceived feasibility of delivering such a curriculum, potential barriers, as well as satisfaction with its content and format. Similar to above, the group was audio‐recorded and transcribed verbatim. Using thematic analysis, ³⁵ two study team members identified meaningful segments within responses and assigned codes, which were grouped into themes and discussed among the entire team of investigators.

2.5. Phase 2 Activities: Video recordings—Data collection & Analysis

Each class session was video‐recorded. Study investigators adapted the fidelity tool used in the ACHIEVE RCT to grade coach fidelity to the ACHIEVE‐D curriculum (Supplemental Materials 6). In brief, the fidelity tool assesses the extent and timeliness to which the coach implemented core components of the ACHIEVE‐D curriculum and used a counseling style consistent with motivational interviewing techniques (e.g., open‐ended questions, affirmations). ³⁹ The tool has several segments that mirror those of the session curriculum: “before the session” (2 items), “group exercise” (7 items), “weigh‐in” (4 items), “focus on weight loss” (6 items), “role model video” (5 items), and “commit to high impact behavior” (7 items). Each item was scored as absent (0)/present ² or on a three‐point scale of absent (0), below expectations, ¹ or meets expectations, ² as appropriate. After using a consensus process to create a standard for grading sessions and a scoring guide, this tool was used to grade all session videos for fidelity. One of the senior study investigators (KAG) reviewed all grading assignments.

For each session, individual item scores were summed to determine cumulative scores for each segment and the entire session (total maximum score per session = 62). The average score for each fidelity item was calculated across all recorded sessions as well as the percentage of sessions where each item met expectations (score = 2). The average score for each segment and for the entire session was also calculated.

3. RESULTS

Results from the phase 2 proof‐of‐concept study are presented below (all phase 1 activities are described in the Methods section). Overall, the heath coach delivered three classes per week for 8 weeks for a total of 23 sessions (one session was canceled). Seventeen individuals with SMI and 3 PRP staff members were study participants. On average, 9.1 participants with SMI attended each group session (SD 2.0) (range: 6–13 participants).

3.1. Participants with serious mental illness

Of the 17 participants with SMI, mean age was 42.5 years (SD 9.1), 47.1% were women, 47.0% were non‐Hispanic white, and mean BMI was 37.9 kg/m² (SD 9.8) (Table 3). Median attendance at groups was 78.6% (range: 8.7%–100%) over the 8‐week period. At the end of the study, 16 participants completed the satisfaction survey. Participants indicated strong agreement with the statements: “I am perfectly satisfied with the ACHIEVE coaching I have been receiving” (mean score 5.8 (SD 0.5)) and “I have an extraordinary amount of confidence in the ACHIEVE coach I have been seeing” (mean score 5.4 (SD 0.9)). Participants strongly disagreed with the statement: “There are some weight issues that I feel my ACHIEVE coach has not given enough attention to” (mean score 1.8 (SD 1.7)). Supplemental Materials 7 contains the additional satisfaction items. Table 4 displays an overview of the themes and subthemes identified from the focus group, in which 14 individuals with SMI participated.

TABLE 3.

Baseline characteristics of participants with serious mental illness.

	(n = 17)
Mean age in years (SD)	42.5 (9.1)
Women, %	47.1%
Race/Ethnicity
Non‐hispanic white	47.0%
Non‐hispanic black	41.2%
Other a	11.8%
Never married, %	82.4%
Education, %
Less than high school	35.3%
High school graduate or equivalent	35.3%
Beyond high school	29.4%
Receiving disability, %	88.2%
Living in a residential program, %	88.2%
Medical history, %
Hypertension	35.3%
Diabetes mellitus	17.6%
High cholesterol	47.1%
Mean BMI in kg/m2 (SD)	37.9 (9.8)
Weight status, %
Overweight	17.7%
Class I obesity	35.3%
Class II obesity	23.5%
Class III obesity	23.5%
Mean waist circumference in cm (SD)	120.9 (16.9)
Elevated waist circumference b , %	88.2%

^a Other race/ethnicity included Hispanic Black (n = 1) and Non‐Hispanic Other (n = 1).

^b Elevated waist circumference defined as >102 cm in men and >88 cm in women ²¹ .

Abbreviations: BMI, body mass index.

TABLE 4.

Summary of findings from focus groups with participants with serious mental illness (SMI) and staff.

Population	Theme	Subtheme(s)
Clients with SMI	Satisfaction with the ACHIEVE‐D curriculum	Most indicated they would want to continue the program and would recommend the program to others Multiple individuals indicated that the program is ideal for those who recognize that they need to make changes and are ready to make changes
	Feedback on the ACHIEVE‐D curriculum and its components	Reported challenges with ACHIEVE‐D Too much emphasis on unhealthy foods/drinks without discussing healthier options Difficulty with maintaining dietary changes Some found the exercise portion to be overwhelming or too intense Some may appreciate more variation with the types of exercise Very few exercised outside of class A small minority found the tracker and weight graph to be confusing Common frustration with weight fluctuations—this led some to stop attending Some had issues with making behavioral changes due to stress, outside influences, etc.
		Reported facilitators with ACHIEVE‐D Most found the program to be very educational—improved understanding of what foods/drinks to avoid General appreciation for the exercise portion Coach was very supportive and provided necessary encouragement All appreciated having their peers in the class with them Some found the repetition of material to be helpful and appreciated the consistency Most liked the role model videos All appreciated the posters/visuals
Staff	Acceptability of the ACHIEVE‐D curriculum	Observed barriers with ACHIEVE‐D Curriculum may be too simplistic for higher functioning clients Some clients may not be willing to fully engage in the exercise component, either during or outside of the group sessions Tracking weight loss may trigger anxiety when weight loss is not achieved
		Observed facilitators for ACHIEVE‐D Consistency of topics and repetition viewed as beneficial for clients Exercise seen as a positive way to support clients in their goals Goal setting and tracking weight‐loss progress perceived as motivating for some clients
	Perceived feasibility of implementing ACHIEVE‐D by a PRP staff member	Perceived barriers for PRP staff as ACHIEVE‐D coach Lack of buy‐in or perceived self‐efficacy to serve in the health coach role Limited time to prepare for group sessions in their current role Challenges in sustaining client engagement in the program
		Proposed solutions for PRP staff as ACHIEVE‐D coach Opportunity to volunteer for the role rather than be assigned Dedicated time to devote to ACHIEVE‐D training and group session preparation Allow staff to serve as co‐facilitators so the onus of intervention delivery does not fall on one person Delivering the intervention in a “closed group” format to promote consistency and group cohesion

Abbreviations: PRP, psychiatric rehabilitation program; SMI, serious mental illness.

3.1.1. Theme 1: Satisfaction with the ACHIEVE‐D curriculum

Most participants with SMI who took part in the focus group expressed interest in continuing the program (n = 12, 85.7%), with all but one participant agreeing that they would recommend the program to others (n = 13, 92.9%). The general consensus was that ACHIEVE‐D is best suited for those who are prepared to make lifestyle changes, and future participants who know this expectation would be highly satisfied with the curriculum. Multiple participants made remarks such as, “folks need to know that they need to do changes outside of the classroom and they need to be ready to really make those changes.”

3.1.2. Theme 2: Feedback on the ACHIEVE‐D curriculum and its components

We identified two subthemes—reported challenges and reported facilitators with ACHIEVE‐D. When asked about the ACHIEVE‐D curriculum, the majority concluded that ACHIEVE‐D was informative, with several remarking that the weight management discussions provided a greater understanding of foods and drinks to abstain from and strategies such as portion control. In terms of the program format, all participants appreciated the group‐based format and indicated that completing the class with other PRP clients was beneficial. As one participant shared, “to see my peers looking like theyre getting something out of [the program] made me want to come more.” Some favored the repetition and consistency of messages within the curriculum, whereas others disliked it. Several participants noted challenges with sustaining behavioral changes, pointing to stress, personal experiences, or temptations to indulge from others as a limiting factor. One participant described such a challenge as, “my roommate, he dont like to really bake stuff… we always fry things. So I want to stop frying‐‐ eating fried foods, but I can never do it because he dont cook nothing else and I dont know how to cook a lot.” Additionally, multiple participants stated that the coach was incredibly supportive and effective in their role as an educator. For example, one participant stated, “When I thought I couldnt do certain things in the group, she [the coach] helped me be prepared for whatever that come my way. And I knew and she knew that I could do it.”

Participants provided feedback on specific components of the ACHIEVE‐D curriculum. Most participants shared positive responses to the role model videos, and all participants liked the posters/visuals used throughout the program. Despite largely positive feedback, the focus group yielded mixed findings on certain aspects. Although numerous consumers appreciated education about which foods and drinks to avoid, others found that there was too much emphasis on unhealthy options and too little attention given to healthy alternatives. While most participants indicated that the exercise component was helpful, some found it to be too strenuous and only two participants reported exercising outside the class. A few found the behavior tracker and weight graph to be confusing. Finally, some participants reported that weight fluctuations, noted during weigh‐ins, led to frustration and decreased motivation to attend groups. One such participant stated, “I [initially attended] all the time for a whole month straight. But then after a while, I stopped coming because I kept losing and gaining weight and it didnt seem like it was helping me too much.”

3.2. Staff participants

Three PRP staff took part in the focus group; all were women and counselors at the PRP. Table 4 displays an overview of themes and subthemes identified from the focus group.

3.2.1. Theme 3: Acceptability of the ACHIEVE‐D curriculum

We identified two subthemes—observed barriers and observed facilitators with ACHIEVE‐D. All of the staff found the ACHIEVE‐D curriculum to be acceptable for the PRP setting, particularly in terms of the structured nature of its content, its focus on both weight management and exercise, and the ways in which the program integrated goal setting and tracking of weight loss progress. One staff stated, “I thought it [the curriculum] was very well structured. The way she [the coach] went about talking about what she was teaching them… she kind of talked about something in a different way each time she talked about it, so it was like they were still learning the same things but they were learning it a little different every time.” The staff did note that some clients may find the curriculum too simplistic or may not be willing to engage in exercise. The staff also reflected that tracking weight might trigger anxiety in some clients when weight loss is not achieved. One staff described, “When there isnt progress, a lot of our clients do experience anxiety. They do have mood issues, so sometimes when they didn't see the results they wanted that was a little unfavorable for them.”

3.2.2. Theme 4: Perceived feasibility of implementing ACHIEVE‐D by a psychiatric rehabilitation program staff member

The first subtheme was perceived barriers for PRP staff as ACHIEVE‐D coaches. Opinions varied regarding whether ACHIEVE‐D could be successfully implemented by a PRP staff member. The staff recognized the potential benefits of knowing their clients histories and the strategies that tend to work best for motivating and redirecting them. However, they also felt that their existing relationships with clients could negatively affect their effectiveness as an ACHIEVE‐D coach, particularly if they were promoting physical health rather than mental health. One staff stated, “They [the clients] don't take us seriously.” Notably, when asked about implementing ACHIEVE‐D at their own sites, they were reluctant to become ACHIEVE‐D coaches themselves. Several factors contributed to this reluctance, including limited self‐efficacy as a lifestyle coach and in using motivational interviewing, particularly in a group setting. One staff member described, “Its a little bit easier for us to use it [motivational interviewing] with our clients, as opposed to a group of clients that come in for a group that were teaching.” The staff also anticipated that keeping clients engaged in the ACHIEVE‐D program, particularly in terms of the exercise component, would be a key implementation barrier. Some staff witnessed clients who tried to avoid the exercise component or did not want to make plans to exercise on their own. By integrating ACHIEVE‐D coaching into existing workflows, staff noted that preparing for group sessions was already an expectation for their role. However, the realities of competing demands often meant that preparing for groups was not always prioritized. As one staff member reflected, “I feel like Im never caught up on work. So, I definitely do kind of prioritize appointments, treatment plans, billing over my groups that I teach, as awful as it sounds, I dont really prepare much.”

The second subtheme was perceived solutions for PRP staff as ACHIEVE‐D coaches. The staff offered several potential solutions to facilitate PRP staff acting as ACHIEVE‐D coaches. First, they recommended that staff should have dedicated time to devote to both training and preparation for ACHIEVE‐D sessions. To increase buy‐in for the coach role, the staff felt it was important for the person to volunteer, and if desired, allow two staff to work as co‐facilitators to share responsibility for curriculum delivery. As one staff member said, “if it's only one person doing it [leading ACHIEVE], then that puts a lot of pressure on that one person because if they not here then it's like everythings falling apart and thats just added stress… So it would be nice to have two people, separate people doing it, just so it could feel like they have a support system.” Lastly, the staff suggested that the curriculum be delivered to the same group of clients who join every week to promote consistency and group cohesion.

3.3. Coach curriculum fidelity via video recordings

Complete video data were available for 15 sessions, partial data for 5 sessions, and 3 sessions had no video due to technical issues. Across recorded sessions, coach fidelity to ACHIEVE‐D was high as the delivery of most items was at the expectation in ≥75% of sessions (22 of 31 items (71.0%)). Table 5 displays the average score for each item across all recorded sessions. Time management was challenging for the later session segments, where the coach met expectations in only 23.5% of sessions for the weigh‐in, 5.3% for the focus on weight loss, never (0%) for the role model video, and 23.5% for the commit to high impact behavior. Scores for several items in the final segment, “commit to high impact behavior,” were low (Table 5). Supplemental Materials 8 contains a summary of video data by sessions throughout each week of the study, and average scores were relatively similar across sessions.

TABLE 5.

Health coach fidelity to ACHIEVE‐D: Average scores and percentage of sessions at expectation across all video‐recorded sessions during a 2‐month period.

Items	Average Score	Sessions at Expectation
Segment 1: Before the session	3.8 out of 4 possible
Organizes appropriate supplies and learning materials before participants arrive to class. a	1.9	95.0%
Sets up exercise video in DVD player before participants arrive to class. a	1.9	94.4%
Segment 2: Group exercise	12.5 out of 14 possible
Cues participant toward mastery of movements and routines.	2.0	94.1%
Provides affirmation for specific positive behaviors.	1.9	91.7%
Encourages full participation, as appropriate.	2.0	100%
Exercises at appropriate intensity with the participants. a	2.0	100%
Reminds participants of the importance of a cool down. a	1.9	94.1%
Reminds participants to have water after exercise. a	0.8	41.2%
Manages time appropriately during this section. a	1.9	94.1%
Segment 3: Weigh‐in	6.2 out of 8 possible
Uses open‐ended questions, supportive language, and appropriate discretion during weigh in.	1.9	94.1%
Aligns high impact behavior with weight change.	1.8	88.2%
Records weight on the graph. a	2.0	100%
Manages time appropriately during this section. a	0.5	23.5%
Segment 4: Focus on weight loss	9.9 out of 12 possible
Reviews agenda (use visual and/or verbal cues). a	2.0	100%
Ensures session remains focused on main topic as per leader's guide.	1.8	84.2%
Uses appropriate materials, props and visual aids. a	2.0	100%
Provides evidenced‐based weight‐related information.	2.0	100%
Limits discussion to ACHIEVE‐D weight loss approaches.	2.0	100%
Manages time appropriately during this section. a	0.1	5.3%
Segment 5: Role model video	7.9 out of 10 possible
Uses role model video. a	1.9	94.7%
Orients participants to the role model video and lesson topic. a	2.0	100%
Encourages participant responses to video.	2.0	100%
Affirms participants' correct responses to questions and/or comments about healthy behavior change as it relates to the video. a	2.0	100%
Manages time appropriately during this section. a	0	0%
Segment 6: Commit to high impact behavior	7.8 out of 14 possible
Summarizes the session (uses visual and/or verbal cues). a	1.3	63.2%
Encourages participants to set high impact behavioral goals.	2.0	100%
Invites participants to identify how and/or when they will try out their plan.	1.4	58.8%
Invites participants to record high impact behavioral goals, as appropriate.	1.6	82.4%
Offers encouragement that participants can be successful when they follow the program. a	0.1	5.9%
Encourages attendance at upcoming classes. a	0.9	47.1%
Manages time appropriately during this section. a	0.5	23.5%
TOTAL ACROSS ALL SEGMENTS (entire session)	48.1 out of 62 possible

^a Items scored as absent (0) or present. ² All other items scored on 3‐point scale of absent (0), below expectation, ¹ or meets expectation. ²

4. DISCUSSION

In this study, the processes for translating the ACHIEVE intervention into the ACHIEVE‐D curriculum adapted for PRP staff delivery are described. The approach was informed by the REP framework, and the results represent the pre‐condition phase. A rapid evaluation of this curriculum was completed by applying the ORBIT model approach, ³² which identified additional refinements needed prior to the REP pre‐implementation and implementation phases. In general, participants with SMI were satisfied with their experience and the PRP staff found the ACHIEVE‐D curriculum acceptable for their clients.

A critical first step in the process was the application of the SEIPS Model to inform the initial adaptation. ³⁴ Challenges experienced across PRP sites during the ACHIEVE trial were identified along with strategies to address these challenges by engaging research team members. As a result, the study team was able to quickly create a version of the ACHIEVE‐D curriculum that was generally perceived as acceptable to staff and participants with SMI as suggested by the results in this study. One of the strengths of this study is leveraging the experience of the trial team members, which may be a unique application of the REP pre‐conditions phase in identifying implementation barriers. ³¹

While engaging with the ACHIEVE trial team members was key, the study team also sought to identify barriers with two stakeholder groups: PRP clients with SMI and PRP staff. These two groups identified additional barriers as well as potential modifications to improve the implementation of the ACHIEVE‐D curriculum, which is another strength in this study. Lessons learned from individuals with SMI and PRP staff are described below.

Overall, participants with SMI were highly satisfied with the ACHIEVE‐D curriculum and coach. Given that participants suggested that ACHIEVE‐D would be best suited for individuals who are ready to make lifestyle changes to lose weight, future studies should include an orientation to ACHIEVE‐D recommended lifestyle changes and assessment of readiness to change for participants. This change would align with general recommendations for evaluating readiness to change prior to initiating obesity treatment in clinical settings ²¹ ; however, it would also likely limit the number of program participants (e.g., people not ready to make lifestyle changes or people not interested in weight loss). Future research may need to consider designing and testing interventions to increase interest in weight loss and readiness to make the necessary lifestyle changes among individuals with SMI—who could then be transitioned to the ACHIEVE‐D program. Another challenge identified was the frustration with weight fluctuations (participants' decreasing and increasing of weight during the study), which contributed to non‐participation in groups. Several strategies could be considered to mitigate this issue, such as changing the frequency of weigh‐ins and including training sessions for PRP staff on this issue. Obesity treatment guidelines recommend weighing at least once a week during weight loss, ²¹ which support this change. As participants only took part in 2 months of a 6‐month curriculum, several of the other challenges reported would likely be addressed by participating in the complete curriculum. For example, participants noted a lack of emphasis on healthy food alternatives and a lack of variety in the exercise class; however, the complete curriculum includes modules on healthy eating habits (i.e., Eat More Vegetables) and will use a variety of exercise videos.

Psychiatric rehabilitation program staff also identified challenges within the ACHIEVE‐D curriculum for staff delivery, particularly related to implementation. With regard to the curriculum content, staff also raised concerns about potential negative effects of the weigh‐ins three times a week, which may be mitigated by changing the frequency of this activity and training staff on how to respond to participants' weight fluctuations. The staff also noted that engaging clients in exercise may be challenging. To address this challenge, future studies should provide training to future ACHIEVE‐D staff coaches on best practices for group exercise that include participant engagement—the existing relationships that staff have with clients may also help address this challenge. Several key implementation barriers and solutions were discussed with regard to PRP staff serving as an ACHIEVE‐D coach. Therefore, working with PRP leadership to identify staff interested in this area may likely be key in recruiting a successful ACHIEVE‐D coach. The study team will also need to discuss with PRP leadership the need to protect staff coaches' time for training and preparation for this content. Protected time may help address staff challenges related to competing work demands. Staff training may increase self‐efficacy in delivering physical health content in ACHIEVE‐D rather than typical mental health group content. Having a structured facilitator guide would also likely help support ACHIEVE‐D staff coaches in this role. A similar approach has been used in the Diabetes Prevention Program, which has been disseminated by the Centers for Disease Control and Prevention. ⁴⁰ Finally, future studies should promote ACHIEVE‐D to be delivered as a “closed group” where the same clients participate each week over the 6‐month curriculum, which will improve both staff coach and client experience and may improve outcomes.

Video‐recording the group sessions enabled objective identification of successes and challenges for the ACHIEVE‐D coach. The study found that overall coach fidelity to the ACHIEVE‐D curriculum was high for most items. Of note, time management was a challenge for the coach, which began during the weigh‐in portion of the session. The weigh‐in process took longer than anticipated as the coach had several tasks of recording the weight, plotting it on a graph, and checking in with each participant about their lifestyle behaviors since their last group session. As a result, the coach then had less time to devote to the subsequent portions of the curriculum (i.e., “focus on weight loss,” “role model video,” and “commit to high impact behavior” segments). Another contributing factor in the time management issue may be the group size, as some group sessions had more than 10 attendees. A large group size could limit the coach's ability to tailor advice and mitigate some participants' concerns about weight fluctuations during the weigh‐in process. Future studies may need to modify the weigh‐in portion in several ways to address this challenge: 1) aim for no more than 10 participants per group, 2) decrease frequency of weigh‐ins to once a week per participant, 3) have an assistant to the coach record and graph weight, and/or 4) shift some weigh‐ins to the end of class when groups are larger than recommended.

This study has several limitations. This study was a small and occurred at a single PRP—this design is appropriate for rapid evaluation of the behavioral intervention per the ORBIT model but limits the generalizability of our findings. During the proof‐of‐concept study, PRP staff only observed the delivery of the curriculum and did not deliver the actual intervention. Examining weight loss as an outcome was outside the scope of this study given the abbreviated curriculum. Future studies will need to determine whether ACHIEVE‐D results in similar weight benefits seen in the ACHIEVE trial, and a detailed description of the final intervention should be provided at that time using the template for intervention description and replication checklist. ⁴¹ Individuals with SMI were compensated for data collection activities, which may have influenced some to participate rather than a desire to lose weight. We also did not characterize the SMI subtype among participants; however, disease severity was high as most reported disability status and living in a residential program. For some sessions, the group size exceeded 10 participants who may have negatively impacted coach performance as well as participant experience. Finally, a formal qualitative analysis method was not used when interpreting the focus groups.

This study demonstrates the use of the REP framework for the beginning stages of translating the RCT‐tested ACHIEVE intervention to the ACHIEVE‐D curriculum appropriate for delivery by mental health center staff. This approach enabled the adaptation of the curriculum to meet the needs of clients with SMI as well as PRP staff and will facilitate further refinement of the curriculum for use during the REP framework's pre‐implementation and implementation phases.

CONFLICT OF INTEREST STATEMENT

KAG serves as the medical director for the American Board of Obesity Medicine, has a research grant from Novo Nordisk, and is a paid consultant to Eli Lilly. All other authors declare that they have no competing interests.

Supporting information

Supporting Information S1

Gudzune KA, Jerome GJ, Dalcin AT, et al. Scaling a behavioral weight‐loss intervention for individuals with serious mental illness using the enhanced replicating effective programs framework: a preconditions phase proof‐of‐concept study. Obes Sci Pract. 2023;9(6):618‐630. 10.1002/osp4.690