Empiric validation of a process for behavior change

Abstract

Most behavior change trials focus on outcomes rather than deconstructing how those outcomes related to programmatic theoretical underpinnings and intervention components. In this report, the process of change is compared for three evidence-based programs’ that shared theories, intervention elements and potential mediating variables. Each investigation was a randomized trial that assessed pre- and post- intervention variables using survey constructs with established reliability. Each also used mediation analyses to define relationships. The findings were combined using a pattern matching approach. Surprisingly, knowledge was a significant mediator in each program (a and b path effects [p<0.01]). Norms, perceived control abilities, and self-monitoring were confirmed in at least two studies (p<0.01 for each). Replication of findings across studies with a common design but varied populations provides a robust validation of the theory and processes of an effective intervention. Combined findings also demonstrate a means to substantiate process aspects and theoretical models to advance understanding of behavior change.

INTRODUCTION

Knowledge about how healthy lifestyles can prevent illness has outpaced effective means to assist people in living health lives [1]. That limitation is not due to lack of research, and contributions have come from diverse disciplines, such as health promotion, behavioral economics, psychology, decision analysis, and sociology. However, progress may have been constrained due to a focus on outcomes rather than deconstructing how change occurred [2–4]. Our group has studied a series of behavior change interventions that use a shared format and theoretical structure. All had documented efficacy in randomized controlled trials [5–10], and for each, specific program components were related to behavioral endpoints [11–13]. Comparing their observed patterns of findings presents a means to assess theories of behavior change and validate an effective intervention format.

PROGRAM THEORY, DESIGN, AND DECONSTRUCTION

Conceptual theory

When designing an intervention, the theoretical foundations determine variables to target to achieve the desired results. Aspects of three theories informed our interventions: the Health Belief Model (HBM) [14], the Theory of Planned Behavior (TPB) [15], and Social Cognitive Theory (SCT) [16]. Together, they comprised the interventions’ conceptual theory [17]. The HBM suggests that a person integrates benefits of a behavior with barriers and the severity and susceptibility to adverse effects of not performing it. Where the HBM describes a balance sheet of perceived threat and net benefit, the TPB begins to integrate norms of what others do. The TPB also adds the concept of self-efficacy or one’s belief that he or she can accomplish a specific task. Although not often mentioned, both models have a role for knowledge, as it is integral to the constructs of severity, susceptibility, and peer norms. In addition, each individual brings other factors that influence outcomes, such as cultural background, psychological state and traits, and prior experiences.

SCT extends beyond the individual and adds a dynamic reciprocal interaction between an individual and the environment, including directly observing the actions of others. The interaction is typically represented by a triadic relationship among personal factors (such as thoughts, outcome expectancies, skills, and self-efficacy); the environment (e.g., cues and vicarious experiences); and the behavioral outcomes of interest. SCT also includes the importance of goals, with monitoring progress toward those goals affecting self-efficacy and subsequent behavior [18].

Action theory

The action theory defines how the conceptual theory’s targeted variables will be altered [17]. All of our programs shared the same underlying action theory or intervention processes: (1) peer-led direct instruction to affect knowledge, abilities, and norms and (2) explicit goal setting and monitoring at an individual and team level.

Peer-led direct instruction

Using peers to facilitate sessions has several advantages. Peer leaders have intrinsic credibility and ability to tailor the discussion with relevant examples [19]. They also promote a sense of shared norms and social support [20]. In other settings, peers have been shown to be as effective as professionally delivered formats [21]. Although a potential downside is the need to recruit and train peer leaders, each program used a scripted format that minimized training and even allowed that role to be rotated among team members.

The scripted curriculum required a design to maximize learning, and direct instruction provided those guidelines [22]. In general, direct instruction follows a specific sequence [23]. First is a statement about learners’ end objectives or what they will be able to do following the lesson. The second is to establish content relevance and/or how the lesson builds on preceding material. Next is providing information and having learners practice, reformulate, and personalize it. Lastly, learning is reviewed, and goals are set and disclosed, with additional motivation coming from the session’s momentum, self-efficacy of the practice, and feedback when reporting on progress. Although primarily designed as pedagogy for younger learners, direct instruction also aligns well with principles of adult learning [24, 25]. Experienced educators often dislike direct instruction’s more prescriptive format, as their styles differ, and they value their personal creativity as educators and students’ discovering the content, rather than upfront objectives. However, for our peer leaders, the limited preparation and easy-to-follow format were ideal.

Individual and team goal setting and self-monitoring

Goal setting is a component of many behavior change programs, with its positive impact relating to social disclosure enhancing commitment, receiving feedback and enhancing self-efficacy [26]. For each of program, goals were set each session and monitored at an individual and team level.

A focus for our interventions was formation of teams, groups of four to seven members who work through the curricula together. We use the term team rather than group, as teams are characterized by a sense of shared commitment and mutual accountability [27, 28]. With a team, the goal-reward structure can be individual, shared, or both [29]. Team level rewards promote mutual accountability and cooperation; however, they may reduce personal responsibility. Having both levels operating simultaneously can augment productivity [30], and that hybrid system was used with our programs’ teams.

Mediation analysis to combine action and conceptual theories

Mediation analysis is a means to assess the hypothetical relationships across intervention processes, targeted variables, and outcomes. It provides an explicit check on whether a program affected the intermediate variables and whether the purported theoretical relationship of those variables to outcomes holds.

A basic mediation model is shown in Fig. 1, where the intervention (X) is presumed to affect the mediator (M), and that path (a) represents a test of the action theory. The mediator affects the outcome or dependent variable (Y), and that mediation path (b) represents a test of the conceptual theory [31]. In an idealized program, both the a and b paths are confirmed. That is, the program changed the mediator, and that change related to the outcome. However, that ideal situation does not always apply. An intervention might affect the mediating variables in the absence of significant effects on outcomes, a finding that suggests that the conceptual theory was faulty. Conversely, outcomes could be altered without a change in the presumed mediating variables, indicating that change was influenced by other unmeasured intermediate variables. Mediation analysis informs intervention design, and its application is becoming more frequent in understanding interventions [32, 33].

Fig 1.

Basic mediation model

A strength of our studies was application of mediation analyses to similar underlying behavior change models. Figure 2 depicts the programs’ components and theoretic structure. Comparing the pattern of which paths were confirmed across studies provides information about the validity of their structure.

Fig 2.

Integrated action and conceptual model components. Some components modified from Fishbein M, Cappella JN. J Commun. 2006;56:S1-S17 (with permission)

Lacking in Fig. 2 are elements of the built environment and external cues that enhance behavior change, aspects that are prominent components of ecological models [34]. However, robust environmental components were absent from our programs to avoid diffusion to the control condition. Accordingly, environmental features have a limited role in our interventions’ construction or theoretical underpinnings. Adding those elements would be expected to enhance the programs’ efficacy.

OVERVIEW OF THREE EVIDENCE-BASED PROGRAMS

The initial behavior change intervention was entitled Athletes Training and Learning to Avoid Steroids (ATLAS) [5]. Funded by the National Institute on Drug Abuse (NIDA), this prospective randomized trial evaluated a sport team-centered drug prevention/health promotion program for high school football players. The scripted curriculum was designed to be delivered during the sport season and integrated into a team’s usual practice activities. The coach facilitated the program, with the entire team meeting as a large group, partitioned into six-member squads, with one individual per squad designated as the student leader. The program was formatted as a playbook with explicit lesson plans and a programmed text for the coach and squad leaders, while other team members used corresponding workbooks. The majority of the ten, 60-min weekly sessions was peer-led, as the squads of student athletes worked through activities to alter their knowledge, skills, and attitudes about anabolic steroid and other performance-enhancing substance use.

That general structure was replicated in the NIDA-funded Athletes Targeting Healthy Exercise and Nutrition Alternatives (ATHENA) program, with the curriculum content modified to address risk and protective factors relevant to young women’s use of harmful body-shaping actions [6]. For both programs, from the athletes’ perspective, they were about improving sport performance, while the hidden curriculum was deterring harmful actions and enhancing life skills. Both ATLAS and ATHENA achieved favorable immediate and 1-year behavioral outcomes, and they are listed on the National Registry of Evidence-based Programs and Practices (http://nrepp.samhsa.gov/). In addition, the program effects appeared durable as at 1 to 3 years following high school graduation, ATHENA intervention participants reported significantly less tobacco use, marijuana and alcohol use, plus improved nutrition and body image, compared to control group graduates [7].

In response to the National Institutes of Health’s request for new models of behavior change, the team-centered, peer-led format was applied to an occupation with natural teams, firefighting. Fire stations generally are staffed by three shifts, each working 24-h, followed by 48-h off duty. Each four to eight member shift is a team of coworkers. For the Promoting Healthy Living: Alternative Models’ Effects (PHLAME) study, fire stations within fire departments were randomized to (1) a team program, (2) to a one-on-one motivational interviewing, or (3) to a control condition [8]. The team-centered format used the same educational principles and theoretical basis as ATLAS and ATHENA. A team of coworkers (a shift at a station) met once a week for 12, 45-min scripted peer-led sessions, with one team member using a team leader manual and others workbooks. The team lessons included content related to health promotion (e.g., nutrition, physical activity, sleep) and safety (such as lifting techniques and joint protection). The team curriculum achieved immediate (1 year) [8] and longer-term favorable behavioral changes [9, 10] and was cost effective in reducing injuries [35].

Despite differences in participant characteristics, targeted behaviors, and curricula specifics, the three programs shared similar structures and showed efficacy in randomized controlled trials. Each program’s process and outcomes were deconstructed using mediation analysis. Repeated application of the same programmatic model across groups is unique, and we hypothesized that comparing patterns of confirmed action and theoretic constructs across studies could provide a robust validation of a model for behavior change.

METHODS

Detailed program descriptions and mediation methods have been published for all three programs [5–13].

Participants

One thousand five hundred six football players from 31 high schools participated in ATLAS, and 1668 adolescent female athletes from 40 sport teams at 18 high schools participated in ATHENA. Their average age was 15.9 and 15.3, for boys and girls, respectively. Overall, 85 % were White. For the PHLAME program, 397 firefighters participated in the control and team conditions, with a mean age at enrollment of 41 years; 93 % were male and 91% were White.

Design

For ATLAS, high schools were randomized to conditions after matching on win/loss records and socioeconomic status. Girls sport teams have fewer athletes than football teams, and with ATHENA, schools were paired based on size and student demographics. One school per pair was randomly assigned to the experimental condition and the other to control. An offer to participate in ATHENA was extended to all girls’ sports teams at experimental schools. After a team agreed to participate, the same sport team at the paired control school was recruited for participation. For PHLAME, 5 fire departments in the Pacific Northwest participated, and stations within departments underwent balanced randomization to the team-centered intervention (15 stations) and control (17 stations) condition. Students and their parents or guardians and firefighters provided written informed consent. The Institutional Review Board of the Oregon Health and Science University approved each study.

Measures

Firefighters were assessed by self-report questionnaires, dietary instruments, and physiological testing before beginning the program and 1-year later. For football players, pre-season and immediate post-season surveys were used. For female athletes, because sports from all three seasons were represented, the longer-term assessment varied from 9 to 12 months. For each program, specifics of the assessment constructs and their reliabilities are available in the original publications [5, 6, 8, 11–13].

Analysis

Mediation analysis was conducted for each of the individual studies [31]. Analyses controlled for any initial differences between conditions by including those measures as predictors of corresponding follow-up measures. Separate single mediator models and multiple mediator models were assessed with versions of Mplus [36], controlling for hierarchical clustering [37]. Full-information maximum likelihood was employed to include all available data. Analyses controlled for pretest differences between conditions by including pretest measures as predictors of corresponding posttest measures. The significance of mediated effects was further assessed with the PRODCLIN program [38]. For clarity and consistency, multiple mediator model findings are presented here. We combined the studies using a pattern matching approach [39], which is a process of examining a theory within different contexts and with different participant groups to validate the underlying hypotheses.

RESULTS

Table 1 summarizes selected mediation findings of the ATLAS, ATHENA, and PHLAME studies, and those results are overlaid on the action and conceptual model in Figure 3. Paths that are statistically significant are bold, with the width increasing depending on whether paths were significant in one, two, or three studies.

Table 1.

Mediation findings of the ATLAS, ATHENA, and PHLAME studies

Study	Hypothesized mediator	a Path (path estimate [standard error])	b Path (path estimate [standard error])	Outcome
ATLAS	Knowledge anabolic steroid (AS) effects	2.34 (0.23)***	−0.02 (0.01)***	Intentions to use AS
	Reasons for using AS	0.54 (0.23)***	−0.02 (0.01)**	Intentions to use AS
	Perceived susceptibility to AS effects	0.44 (0.11)***	−0.05 (.01)***	Intentions to use AS
	Team as information source	0.56 (0.06)***	−0.03 (.03)	Intentions to use AS
	Peer tolerance to drug use	−0.17 (0.10)*	0.03 (0.02)*	Intentions to use AS
	Ability turn down drug offer	0.28 (0.07)***	−0.17 (0.02)***	Intentions to use AS
	Multiple mediator model χ2 (36) = 59.817**, CFI = 1.00, RMSEA = 0.021
ATHENA	Knowledge diet pills, diuretics, vomiting	0.27 (0.07)***	0.07 (0.02)**	Outcome expectancy for harmful effects
	Norms society and coach concerning weight	−0.22 (0.06)***	0.23 (0.03)***	Intentions to use drugs or vomiting to lose weight
	Mood management	0.20 (0.07)**	−0.05 (0.02)**	Intentions to use drugs or vomiting to lose weight
	Self-efficacy	0.34 (0.02)**	−0.08 (0.02)***	Intentions to use drugs or vomiting to lose weight
	Multiple mediator model χ2 (209) = 706.474***, CFI = 0.911, RMSEA = 0.038, SRMR = 0.062
PHLAME	Knowledge benefits fruits and vegetables	0.31 (0.08)***	0.65 (0.13)***	Fruit consumption
	Knowledge exercise benefits	0.06 (0.09)	0.10 (0.05)*	Self-reported physical activity
	Self-monitoring food intake	0.22 (0.14)	0.12 (06)*	Fruit consumption
	Coworker dietary norms	0.20 (0.08)*	0.46 (0.12)***	Vegetable consumption
	Exercise coworker support	−14 (0.09)	0.14 (0.05)**	Self-reported physical activity
	Multiple mediator model χ2 (82) = 136.914***, CFI = 0.950, RMSEA = 0.04, SRMR = 0.048

*p<0.05; **p<0.01; ***p<0.001

Fig 3.

Validated action and conceptual model. Solid lines are confirmed paths, with the width relating to the number of studies confirming the path. The associated numbers indicate the studies in which path was confirmed. Path coefficients (standard errors) and significance levels are shown in Table 1

The ATLAS program achieved a reduction in new anabolic steroid (AS) use, and the primary outcome was intentions to use performance-enhancing AS. Significant mediators were perceived severity and susceptibility for AS effects, knowledge of AS effects, ability to turn down drug offers, reasons for using and not using AS, perceived coach tolerance of AS use, team as an information source, and beliefs in the media. The ATHENA intervention had intentions toward harmful body-shaping actions as an outcome. For that program, changes in knowledge, norms, mood management, outcome expectancies for risky behaviors, and self-efficacy were associated with healthy changes in intentions, which in turn, were associated with behaviors. The strongest mediator of the ATHENA effects was the construct measuring perceived norms from the coach and magazine advertisements.

PHLAME’s outcomes related to both improved nutrition behaviors and regular physical activity. As shown in the figure, potential intermediate variables were in the domains of knowledge, perceived control or self-monitoring, and norms. Although the conceptual theory distal to those variables was confirmed, the intervention did not successfully change all the potential mediators, as the path from the intervention to self-monitoring was not confirmed. The mediation models fit the overall findings well for each program (Table 1) (fit is acceptable when CFI >0.95, RMSEA <0.06, SRMR <0.05 [40]).

As shown in Fig. 3, each path was significantly substantiated by at least one study, with the majority of paths significant in two or more trials. Similar findings in different populations and contexts suggest robust confirmation of the underlying process and theoretic structure.

CONCLUSIONS

Evolving more effective behavior change interventions is dependent on discerning their underlying mechanisms. Yet, few studies relate intervention outcomes back to their theoretical and process structure [2–4]. We compared the pattern of mediation findings across prospective intervention trials with different populations and behavioral outcomes to validate conceptual and action theory components and demonstrate a methodology to respond to the challenge of identifying effective processes of behavior change.

We combined the studies using a pattern matching approach [39], which is a process of replicating research with different participant groups. A more elaborate meta-analytical approach was not used in this research because only three studies were involved, and the studies differed in many respects, such as sample characteristics, construct measures, and outcomes [31, 41, 42]. As a result, we focus on the pattern of results across studies rather than combining the studies quantitatively.

Unexpectedly, knowledge emerged as a consistent mediator of behavior change. For each study, increasing understanding (e.g., risks of using anabolic steroids, knowledge about risks of diet pills, and reasons for eating fruits and vegetables) was a mediator of positive behavioral outcomes. This differs from the common wisdom that knowledge-based programs do not impact actions [43–45]. For some programs where baseline knowledge is high, such as consequences of tobacco and alcohol use, there may be little room for improvement [46]. However, as seen in marketing, certain information can have the ability to persuade [47]. In a meta-analysis of intervention studies with family caregivers, those that combined content education with social support and other tactics produced the largest effects [48]. And as mentioned in the discussion of the programs’ theoretical structure, knowledge may inform attitudes and could serve as a prerequisite for change. For example, knowing what constitutes a serving size might facilitate eating five servings of fruits and vegetables a day. Because we measured mediators and outcomes at only one time point after the intervention, we cannot differentiate whether knowledge was a prerequisite or itself instrumental. However, its importance was striking, especially considering that it generally is considered to have little impact.

A measure of team cohesion does not appear in the model, partly because the appropriate index is challenging. Cohesion needs to be related to specific tasks [49] and may vary as teams work through weekly goals toward a final objective. A recent study looking at four dimensions of cohesion found it had no impact on behavioral outcomes [50], and as those researchers pointed out, current measures for team characteristics relating to behavior change lack reliable indices of competition and cooperation.

Our findings across three evidence-based interventions with a common design but varied populations and curriculum, generally validated their theoretical underpinnings and program components’ ability to impact mediating variables. The combined results demonstrate a process for substantiating design features and theoretical models to advance the science of behavior change.