Effectiveness of “run-ins” at predicting adherence in a behavioral weight loss efficacy trial

Abstract

There is limited research on whether run-in procedures predict participant adherence during behavioral efficacy trials. This study examined whether information from behavioral run-ins (food diary completion, questionnaire completion, and staff interview) predict intervention adherence, trial retention, and trial outcomes in a behavioral weight loss trial. Using run-in data, trial staff predicted which participants would have high, moderate, or low trial adherence. Participants with predicted high or moderate adherence were randomized. Results showed that predicted high adherers had better intervention adherence (session attendance and completion of self-monitoring records) and superior trial outcomes (i.e. weight loss). Run-in data did not predict trial retention. Results suggest that run-ins may be effective at identifying participants adherent to intervention protocols, thereby enhancing internal validity of behavioral efficacy trials.

Introduction

Trial run-in procedures involve potential participants completing specific tasks prior to randomization to determine likelihood of intervention adherence and trial retention. Run-ins are often used in efficacy (vs. effectiveness) trials wherein new interventions are being evaluated and internal validity is paramount. In pharmaceutical trials, run-ins commonly involve assessing intervention adherence to a placebo pill over an extended period-of-time ([1]). In behavioral trials, run-ins often involve assessing intervention adherence via completion of self-monitoring records (e.g., daily monitoring of food/exercise) [2, 3]. In the pharmaceutical literature, there is mixed evidence regarding the utility of run-ins. Some studies demonstrate that run-ins improve intervention adherence and trial retention [4], thereby increasing statistical power. However, other pharmaceutical studies raise questions regarding the utility of such procedures, and the impact of these procedures on the generalizability of treatment effects [4–6]. The utility of run-ins on intervention adherence and outcomes in behavioral trials is less-studied and results are mixed. Only one study examined the impact of run-ins on adherence.[3] Results showed that run-in procedures had no effect on adherence to treatment attendance.[3] Two studies examined the link between run-in adherence and trial outcomes; one study showed that diary run-in completeness was associated with better weight loss outcomes in a behavioral weight loss trial [3] whereas another study demonstrated no effect of run-ins on weight loss [7]. To our knowledge, no studies have examined the impact of run-ins on other aspects of behavioral trial adherence such as intervention adherence and trial retention. Further, only one study examined whether behavioral clinical trial run-ins reduce generalizability, and results showed that younger individuals may be disproportionally excluded from behavioral trials due to non-compliance with run-in procedures, thereby reducing generalizability of trial results to this population.[2]

The present study is the first to examine the effects of behavioral run-in procedures on intervention adherence, trial retention, and trial outcomes in a behavioral efficacy trial. To determine whether behavioral run-ins impact participant selection and generalizability of results, we also examined demographic differences among individuals predicted to have high, moderate, or low adherence. Further, we hypothesized that individuals randomized and predicted to have high adherence based on behavioral run-in data would have better intervention adherence, trial retention and superior weight loss outcomes.

Methods

This study was part of a randomized controlled efficacy trial examining the effects of peer e-coaching for behavioral weight loss [8]. The focus of the present study was on the behavioral run-in; specifically, whether pre-randomization behavioral run-in data predicted intervention adherence, trial retention, and trial outcomes.

Trial participants were recruited via newspaper advertisements, listservs, and mailings. Initial eligibility was assessed via phone screen. For full inclusion criteria, see Leahey et al., 2020 [8]. Following a phone screening, participants attended an orientation session where study procedures were provided, including information on the behavioral run-in. Participants were informed that the run-in would provide a sense of what it is like to participate in the trial, identify whether the trial is a “good fit” for them at this time and whether they are a “good fit” for the commitment involved in a trial, and address any questions/concerns they may have. Participants were aware that, after the run-in, they could decide whether to participate and, similarly, research staff would discuss whether they were an appropriate fit for the behavioral efficacy trial.

Following orientation, participants began run-in procedures. The behavioral run-in included three components: (1) self-monitoring food/drink for one week, (2) completing the Block Food Frequency Questionnaire (FFQ) at home and returning it at the run-in visit, and (3) participating in a behavioral interview with study staff at the run-in visit. The behavioral interview was consistent with previous, semi-structured behavioral interviews [9, 10], with the goal of exploring whether trial participation is appropriate, or a good “fit,” for the participant at this point in their life, and answering any questions they may have.[10] Specifically, during the one-on-one behavioral interview, research staff (1) reviewed the study purpose and procedures; (2) discussed participants’ experiences with dietary monitoring; (3) underscored the importance of commitment when joining a trial; (4) gave participants the opportunity to express any hesitations regarding trial involvement; and (5) asked participants whether they foresee any barriers to participation, inquiring specifically regarding prolonged travel, schedule change, childcare, transportation, and any planned relocation. Behavioral interview duration was approximately 15-minutes. Research staff conducting behavioral interviews were trained by the PI and a master’s level interventionist with relevant experience. Training involved orientation and discussion of the behavioral run-in procedures, observing behavioral interviews between experienced staff members and participants, and conducting interviews in the presence of the PI or experienced staff member with feedback. Once training was complete, staff conducted behavioral interviews independently.

Following the behavioral run-in, study staff met as a team to discuss run-in information, including number of food diary days completed, completion of the FFQ, and participant’s behavioral interview responses (e.g., concerns with monitoring, scheduling conflicts). Guidelines were given for coding likely adherence as high, moderate, or low. Guidelines for high adherence included monitoring ≥6 days of meals/snacks (out of a possible 7), FFQ completion, and no concerns regarding adherence to treatment components (e.g., self-monitoring, session attendance) or trial protocol (e.g., assessments). Moderate adherence guidelines included FFQ completion, 5 monitoring days, and/or possible treatment barriers (e.g., lengthy travel time to treatment sessions). Low adherence guidelines were ≤4 monitoring days, not completing the FFQ, and/or major concerns with treatment components (e.g., averse to monitoring) or major barriers (e.g., relocation). Given that there were more participants interested in the study than could be accommodated, only predicted high and moderate adherent participants were randomized. Remaining individuals were given referrals to community weight loss programs.

Measures.

Participants received $25 for completing assessments at 3-, 6-, and 9-months, and $50 for completing the 12-month assessment.

Demographics & baseline body mass index (BMI).

Sex, age, race, ethnicity, and education were collected. Weight and height were measured via digital scale and wall-mounted stadiometer, respectively; BMI was calculated (kg/m²).

Intervention adherence.

Number of intervention sessions attended (out of a possible 18) and number of weekly self-monitoring records completed (out of a possible 52) were used to measure intervention adherence.

Trial retention.

Number of assessment sessions attended (out of a possible 5) and whether the participant completed the primary endpoint visit (month 12) were used to measure trial retention.

Primary trial outcome: Percent weight loss.

Percent weight loss was calculated as follows: [(baseline weight – post-treatment weight) / baseline weight]*100.

Statistical analyses.

Between group differences for continuous or dichotomous variables were examined using ANOVAs or chi-square tests, respectively. Associations were examined using Pearson correlations. Treatment arm was a covariate. Effect sizes are reported as Cohen’s d or r. Participants who prematurely discontinued treatment (N=5) due to serious medical issues (e.g., cancer diagnosis) were not included in analyses given that their non-adherence was due to unforeseen medical issues and could not be accurately accounted for by behavioral run-in predictions.

Results

Behavioral run-in adherence classifications & participant baseline characteristics.

Among individuals who completed behavioral run-in procedures and provided demographic data (N=348, 77.6% female, 51.7±5.6 years, 25.6% racial/ethnic minority, 58.3% college-educated, BMI=34.8±3.3 kg/m²), there were no significant differences in those predicted to have high, moderate, or low adherence on age, racial/ethnic minority status, education, or baseline BMI (p’s≥.11). However, a greater percentage of males (vs. females) were predicted to have high adherence (78.2% vs. 60.0%, r=.16, p=.003). Among individuals invited to participate and randomized (N=273, 75% female, 27% minority, 52±6 years, 35±3kg/m²), 81.7% (N=223) were predicted to have high adherence and 18.3% (N=50) moderate adherence. The two groups did not differ on any demographic characteristic except sex; men were more likely than women to be identified as high adherers (90.0% vs. 72.6%, p=.01).

Behavioral run-in classifications, intervention adherence, trial retention, and primary trial outcome.

Behavioral run-in classifications predicted intervention adherence (Table 1). Predicted high adherers attended more treatment sessions than moderate adherers (14.8±4.0 vs. 12.4±5.5, d=0.50, p<.001). Similarly, predicted high adherers completed more self-monitoring records during treatment (27.7±15.2 vs. 21.1±15.3, d=0.43, p=.005) and lost more weight (9.0±8.4 vs. 6.1±8.5, d=0.34, p=.03). Trial retention, however, did not differ between predicted high vs. moderate adherers (Table 1).

Table 1.

Adherence by behavioral run-in prediction.

	Predicted High Adherence (N=223)	Predicted Moderate Adherence (N=50)	p-value
Intervention adherence
Intervention sessions attended (out of a possible 18), M±SD	14.8±4.0	12.4±5.5	<.001
Self-monitoring diaries completed (out of a possible 52), M±SD	27.7±15.2	21.1±15.3	.005
Trial retention
Assessment sessions completed (out of a possible 5), M±SD	4.6±0.9	4.6±0.9	.70
%age of participants who completed the primary endpoint assessment	95.1	98.0	.70
Trial outcome
% weight loss, M±SD	9.0±8.4	6.1±8.5	.03

Discussion

This study showed that behavioral run-ins involving self-monitoring, questionnaire completion, and behavioral interview are effective at predicting adherence in a behavioral efficacy trial. Those expected to be more adherent based on run-in information had better intervention adherence and lost more weight. Interestingly, run-in information did not predict trial retention, likely due to high staff effort (reminders, phone calls) and incentives for assessment completion.

Behavioral run-in procedures used herein did not differentially eliminate younger individuals or individuals with lower education or of racial/ethnic minority status [2, 5]; however, women were predicted to be less adherent than men. Women often have more caregiver roles [11–13], which may make self-monitoring challenging or attendance at treatment and assessment sessions more difficult. Further, men do not often enroll in weight loss trials [e.g., 8, 9, 10], thus, the men that did enroll may have been especially motivated to prioritize compliance with run-in procedures and attendance at study meetings.

To our knowledge this is the only study that has examined whether predicted trial adherence based on behavioral run-in data actually predicts adherence during a behavioral efficacy trial. Further, this trial examined three important factors: intervention adherence, trial retention, and outcomes in a large efficacy trial. Objective measures were also used. Results from this study suggest that run-in procedures may be effective at predicting intervention adherence and outcomes and thus enhancing the internal validity of behavioral efficacy trials.