Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 Mar 1.
Published in final edited form as: Adm Policy Ment Health. 2019 Mar;46(2):209–219. doi: 10.1007/s10488-018-0907-5

The Role of Youth Trainee-Trainer Alliance and Involvement in School-Based Prevention: A Moderated-Mediation Model of Student Gatekeeper Suicide Prevention Training

Christine M Wienke Totura 1,*, Christa D Labouliere 2, Kim Gryglewicz 3, Marc S Karver 4
PMCID: PMC6494619  NIHMSID: NIHMS1020239  PMID: 30460591

Abstract

This study examined the roles that youth involvement and youth trainee-trainer alliance play in school mental health prevention within the context of youth suicide gatekeeper training. Measures included youth involvement in programming, trainee-trainer alliance, and intentions to refer at-risk youth at pre- and post-training. A moderated-mediation design was used to examine associations among these factors. Results show alliance mediating pre- and post-training referral intentions, and involvement moderating the relationship between alliance and post-training intentions. On average, trainee intentions improved from pre- to post-training, but trainees reporting high alliance endorsed higher post-training referral intentions regardless of involvement level. Low alliance resulted in lower than average post-training referral intentions, even with active involvement in programming, and those with both low alliance and involvement showed the lowest post-training referral intentions. Given these findings, fostering the youth trainee-trainer relationship may be an avenue to optimize prevention program effectiveness.

Keywords: trainer alliance, youth trainee involvement, community prevention, suicide prevention, school-based mental health

Even when prevention programs are promising, simply adopting them into community settings does not guarantee effectiveness (Ajzen, 1985). The success of prevention programming depends upon several factors, including supportiveness of the implementation climate, the extent to which program trainers engage trainees, and how well the program influences participants’ attitudes and behaviors (Ajzen, 1985; Hagger, Chan, Protogerou, & Chatzisarantis, 2016). The degree to which prevention trainers develop supportive relationships, or alliances, with trainees also has implications for how engaged they will be and the skills and knowledge they gain (McCormick, O’Connor, Cappella, & McClowry, 2013). It is well-established in education and clinical psychology that a strong alliance between youth and their teachers or therapists results in increased youth involvement in the learning process and knowledge/skill transfer (Karver et al., 2008; Kim & Capella, 2016; Labouliere, Reyes, Shirk, & Karver, 2017); however, the role of youth trainee-trainer alliance and youth involvement in school-based prevention programming has received little attention.

Although the prevention literature has not yet fully examined its role, alliance is one of the most important predictors of successful mental health treatment outcomes with youth, beyond dosage or specific techniques used (Bickman et al., 2012; Cummings et al., 2013; Shirk, Karver, & Brown, 2011). Integral to the Transtheoretical Model of Change as a “helping relationships” indicator (Prochaska, DiClemente, & Norcross, 1992), therapeutic alliance is a common process factor which focuses upon the collaborative relationship between a therapist and client (Karver et al., 2008; Shirk et al., 2011). The therapeutic alliance literature suggest that, regardless of the therapist’s particular skillset or adherence to a programmatic model, even the most promising interventions will fail if a therapist cannot successfully form a relationship with their client (Horvath, Del Re, Flückiger, & Symonds, 2011). Alliance is believed to positively influence youth outcomes by increasing youth involvement in treatment (Shirk et al., 2011), attendance (Berry, Gregg, Lobban, & Barrowclough, 2016), and satisfaction (Weisz, Doss, & Hawley, 2005). Since alliance plays such a critical role in clinical interventions, it stands to reason that alliance between teacher trainers and youth trainees (i.e., students), a similar relationship where the helper cannot successfully pass on information if the recipient is not receptive, may also influence training receptiveness and outcomes in prevention programs.

In contrast to the dearth of research on alliance, prevention research has long identified trainee responsiveness as a key variable in the success of prevention programs (Berkel, Mauricio, Schoenfelder, & Sandler, 2011). Responsiveness has been conceptualized as a multifaceted construct that includes trainee involvement, satisfaction, attendance, retention, and program dosage. Each of these constructs has demonstrated associations with program outcomes (Garvey, Julion, Fogg, Kratovil, & Gross, 2006; Whittaker, & Cowley, 2012); however, attendance, retention, and dosage have been the most measured indicators of trainee responsiveness (Berkel et al., 2011). This is unfortunate, as research shows that the mere amount of time that someone is present in training does not necessarily predict outcomes; rather, it is the quality of active participation in the training that matters most (Fredricks, Bohnert, & Burdette, 2014; Nix, Bierman, McMahon, & The Conduct Problems Prevention Research Group, 2009). Many overlapping terms (e.g., responsiveness, engagement, involvement, participation) are used interchangeably in the clinical, education, and prevention literature, creating a great deal of conceptual confusion. For the sake of clarity, we utilize the term “involvement,” as it captures the active features of responsiveness that appear to matter most (e.g., verbal and behavioral participation of trainees, attempting newly-taught skills, etc.; Castro-Blanco, Karver, & Chiechi, 2010). Trainee involvement in training may be critical because it can affect the comprehensiveness of material experienced by the trainee. For example, a trainer may not progress to more advanced material if trainees do not indicate that they understand or if they appear disinterested in the content. In addition, individuals who do not participate in activities or discussion may not learn as much as those who do, thereby impacting successful training outcomes (Berkel et al., 2011; Low, Van Ryzin, Brown, Smith, & Haggerty, 2014; Xia, Taylor, & de Guzman, 2017).

Despite its known importance to successful program outcomes, few studies have assessed the specific mechanisms by which trainee involvement is related to prevention outcomes (Durlak & DuPre, 2008; Low et al., 2014; Xia et al., 2017). Based on the theoretical model proposed by Durlak and DuPre (2008) and the Interactive Systems Framework for Dissemination and Implementation (ISF; Wandersman et al. 2008) supporting program implementation as a highly contextualized activity integrating individual, program, and organizational components, Berkel and colleagues proposed a conceptual model wherein trainee involvement is a mediator of program effects, such that, when programs are implemented with quality, the level of trainee activity in the program is the mechanism by which delivery contributes to positive outcomes (Berkel et al., 2011). A recent review of prevention programs has alternatively suggested that trainee involvement serves as a moderator of program effects, yet there is a dearth of empirical work to substantiate this (Sandler et al., 2014). Similarly, although the construct has not yet been tested in prevention, alliance is established in clinical literature as a mechanism for enhancing involvement, a mediator by which program benefits are realized (Bickman et al., 2012; Karver et al., 2008). While it is likely that trainers possessing the skill to develop positive working relationships stand a better chance at encouraging trainee involvement, existing research (e.g., Johansson & Hoglend, 2007) is not clear on the mechanism by which trainee alliance and involvement together influence prevention program outcomes. Theories undergirding implementation effectiveness indicate that factors such as alliance and involvement are interactional, yet researchers note that their direct, separate relationships with program or treatment outcomes are often what is assessed (e.g., Durlak & DuPre, 2008; Shirk et al., 2011), providing further support for the need to elucidate the relationship between alliance and involvement.

In this study, we seek to address this gap in the prevention literature and explore the mediator and moderator effects of the alliance between student trainees and health education teacher trainers and youth trainee involvement on prevention program outcomes in the context of suicide prevention programming. Suicide prevention is an important area for research, as these efforts are notorious for stigma and limited community participation based on the sensitive nature of the topic (Totura, Kutash, Labouliere, & Karver, 2017). Suicide is the 2nd leading cause of death for adolescents, making it a significant public health concern for community prevention (Centers for Disease Control & Prevention, 2016). Moreover, a large proportion of at-risk youth are in need of mental health services, but most are unable to seek out helping professionals and instead confide in peers, who typically do not know how to identify warning signs of suicide or respond to their needs (Bruffaerts et al., 2011; Kalafat & Elias, 1994; Labouliere, Tarquini, Totura, Kutash, & Karver, 2015). School-based gatekeeper suicide prevention training provides adolescent trainees with the knowledge and skills to assist at-risk peers by learning how to identify warning signs of suicide and referring peers to trained mental health specialists for help. Unfortunately, the opportunity to capture the learned suicide prevention behaviors of trainees is often limited, since school resource and time constraints hinder the implementation of evaluation designs that include the longer follow-up periods needed to assess changes in behavior (i.e., typically several months or years beyond the immediate training assessment period). As such, in this study, trainees’ behavioral intentions to utilize prevention skills were assessed as the program outcome, as intentions are a close proxy to actual suicide prevention referral behaviors (Azjen, 1985; Brown Hangartner, Totura, Labouliere, Gryglewicz, & Karver, 2018). Building off previous clinical research on alliance (Bickman et al., 2012; Karver et al., 2008; Shirk et al, 2011) and the theoretical models guiding conceptualization of involvement in prevention efforts (see Berkel et al., 2011; Durlak & DuPre, 2008; Wandersman et al., 2008), we hypothesized that stronger alliance between student trainees and teacher trainers will be related to greater youth trainee involvement in suicide prevention programming, thereby predicting improvements in referral intentions from pre- to -post training. The conceptual mechanisms suggested by this hypothesis were empirically tested by comparing moderation, mediation, and mixed moderation-mediation models.

METHOD

Participants

Participants were 418 adolescents (47.5% male, 52.3% female) from two high schools in a large, diverse, urban school district in the southwest. Approximately 51.6% of the sample was Latino/a, 20.5% White or Caucasian, 13.5% two or more races, 5.3% Native American or American Indian, 4.3% Black or African American, 1.4% Asian or Asian-American, and 3.4% another racial or ethnic group. Adolescents in the 9th grade comprised 83.7% of the trainee sample, with the remaining distributed across 10th (7.4%), 11th (4.8%), and 12th (4.1%) grades.

Trainees received suicide prevention training from one of six trainers, all of whom were the trainees’ existing health education teachers. Four trainers (66.7%) were Latino/a, one was Caucasian (16.7%), and one chose not to identify their racial or ethnic identity (16.6%); two were male (33.3%) and four female (66.7%). Three trainers held a master’s degree (50%), while the others held bachelors’ degrees. On average, teacher trainers had worked in the field for approximately 12 years (SD=12.37; range = <1 to 27 years), and were employed in their current position for 9.6 years (SD=10.15; range = <1 to 27 years).

Measures

Planned Behavior and Implementation Questionnaire (PBIQ).

The PBIQ is a study-designed questionnaire (Totura, Tarquini, Naoom, Karver, & Kutash, 2008), based on the Theory of Planned Behavior (Ajzen, 1985; 2017) and theoretical factors associated with successful implementation (Fixsen, Naoom, Blase, & Friedman, 2005). The questionnaire contained 12 pretest items and 47 posttest items, including youths’ suicide prevention knowledge, attitudes about suicide prevention, intentions to engage in referral behaviors, confidence in suicide prevention skills and knowledge (pre- and post-training), involvement in training, alliance with the health education teacher trainer, perception of classmate participation, and fidelity of training implementation (post-training only). In this study, only the alliance, involvement, and intentions subscales were used. Alliance was assessed by one item at post-training (e.g., “I like the teacher who taught the Jason Foundation program”), as the emotional bond component of alliance is considered the core component of alliance in youth (Bickman et al., 2012; Karver et al., 2008; Karver et al., in press). Intentions to refer was assessed by three items (e.g., “If I suspect that someone I know is suicidal, I intend to refer him/her to a responsible adult”) at pre- and post-training (αpre=0.84; αpost=0.75), and involvement by three items (e.g., “I actively participated in the Jason Foundation program discussions”) at post-training (α=0.72). All items utilized a 7-point Likert scale, ranging from strongly disagree (1) to strongly agree (7), and mean scores were created for each subscale. Items from the PBIQ have been successfully used in other adolescent suicide prevention studies (Labouliere et al., 2015) and have adequate psychometrics (Totura, Figueroa, Wharton, & Marsiglia, 2015).

Procedures

Programming.

As the Jason Foundation (JF) “A Promise for Tomorrow” gatekeeper prevention program was part of the required health curriculum, parental permission for youth participation was obtained via waiver of signed informed consent methods (i.e., parents were sent an informational letter about the study and could “opt-out” their teen from participation). Secondary data analysis of de-identified evaluation data was approved by both the school board and university Institutional Review Board (IRB).

The JF program promotes awareness of the problem of youth suicide, provides student trainees with the knowledge and resources to interact with at-risk youth, and encourages referral behaviors. Training sessions were conducted by six health education teachers who were supervised by trained school mental health personnel during health classes in two urban public high schools. Training content was delivered over three classes (each session lasting 40 minutes, for a total training time of 2 hours) using a variety of teaching methods, including soliciting student questions, facilitating inter-student discussions on warning signs and resources, and working in groups on interactive exercises; when possible, trainers also utilized skill-based role plays. Within each training group (health class), there were an average of 20 youth participants (SD=4.32; range = 13 to 29 participants). Training sessions were administered with high fidelity: 95% of student-reports and 87% of observer-reports confirmed adherence to a training protocol using a structured checklist of program core components. Surveys were administered to students immediately before and after receiving the curriculum. Responses were de-identified and matched using an identification number.

Analyses.

Regression-based moderation and mediation analyses were run in SPSS 25 (IBM, Inc.; Chicago, IL, USA), using the highly-validated PROCESS macro (Hayes, 2013; 2018). Within-trainer, within-school, and trainer-within-school clusterings were examined to determine whether these clustering variables warranted inclusion in analyses. Interclass coefficient estimates in the unconditional models were close to zero (<.00001, representing less than a tenth of a percent variance accounted); therefore, the use of mixed-effects models with trainer, school, or trainer-within-school clustering variables was unnecessary and non-nested results are presented for the sake of parsimony. In separate models, alliance and involvement were tested as mediators of the relationship between pre- and post-training intentions to refer, using 95% confidence intervals calculated via 5,000 bootstrapped samples. Subsequent moderated-mediation analyses were guided by the result of initial mediation testing (Hayes, 2013; 2018). All models were run using both standardized and unstandardized metrics. As relationships between variables and conclusions did not change between standardized and unstandardized models, the unstandardized coefficients are reported for ease of interpretation (Hayes, 2013; 2018).

RESULTS

Table 1 presents descriptive statistics for all variables. Hierarchical regression analyses demonstrating predictive validity of both alliance and involvement on post-training intentions to refer (controlling for baseline intentions) are presented in Table 2.

Table 1.

Descriptive statistics and correlations.

Descriptive Statistics
 Correlations
Max / Min Mean (SD) 1 2 3 4
1. Intentions (pre) 1 / 7 5.55 (1.35) 1 .60*** .32*** .25***
2. Intentions (post) 1 / 7 5.83 (1.16) 1 .39*** .29***
3. Alliance (post) 1 / 7 5.70 (1.44) 1 .39***
4. Involvement (post) 1 / 7 4.64 (1.52) 1
Open in a new tab

Note: N=418.

***

Denotes correlations significant at p < .001 level.

Table 2.

Alliance and involvement moderation and mediation analyses.

Model 1 R R2 F df    p
Involvement .249 .062 27.495 1, 416    .0001
Post-Training Intentions to Refer .614 .377 125.736 2, 415    .0001
Coefficient (SE) t p LLCI / ULCI
Involvement
 Constant 3.085 (0.305) 10.115 .0001 2.485 / 3.685
 Pre-Training Intention to Refer 0.280 (0.054) 5.244 .0001 0.175 / 0.386
Post-Training Intention to Refer
 Constant 2.641 (0.212) 12.480 .0001 2.225 / 3.057
 Involvement 0.110 (0.031) 3.622 .0001 0.051 / 0.170
 Pre-Training Intention to Refer 0.482 (0.034) 14.051 .0001 0.415 / 0.550
Model 2 R R2 F df p
Post-Training Intention to Refer .622 .387 87.072 3, 414 .0001
Coefficient (SE) t p LLCI / ULCI
Post-Training Intention to Refer
 Constant 1.593 (0.463) 3.439 .001 0.682 / 2.504
 Involvement 0.362 (0.104) 3.492 .001 0.158 / 0.566
 Pre-Training Intentions to Refer 0.865 (0.087) 7.900 .0001 0.514 / 0.855
 Pre-Training Intentions to Refer x  Involvement Interaction −0.047 (0.019) −2.538 .01 −0.084 / −0.011
Model 3 R R2 F df p
Alliance .320 .102 47.373 1, 416 .0001
Post-Training Intentions to Refer .633 .401 138.834 2, 415 .0001
Coefficient (SE) t p LLCI / ULCI
Alliance
 Constant 3.808 (0.283) 13.457 .0001 3.252 / 4.364
 Pre-Training Intention to Refer 0.342 (0.050) 6.883 .0001 0.244 / 0.439
Post-Training Intention to Refer
 Constant 2.310 (0.223) 10.369 .0001 1.872 / 2.748
 Alliance 0.176 (0.032) 5.472 .0001 0.113 / 0.240
 Pre-Training Intention to Refer 0.453 (0.034) 13.163 .0001 0.385 / 0.521
Model 4    R R2 F  df p
Post-Training Intention to Refer    .651 .424 101.559  3, 414 .0001
Coefficient (SE) t p LLCI / ULCI
Post-Training Intention to Refer
Constant 0.411 (0.515) 0.797 .43 −0.602 / 1.423
Alliance 0.535 (0.094) 5.715 .0001 0.351 / 0.720
Pre-Training Intentions to Refer 0.846 (0.102) 8.273 .0001 0.645 / 1.047
Pre-Training Intentions to Refer x  Alliance Interaction −0.072 (0.018) −4.072 .0001 −0.107 / −0.037
Open in a new tab

Note: N=418.

Involvement as a mediator versus moderator.

Student involvement significantly mediated the relationship between pre- and post-training intentions to refer at-risk youth (Model 1, Z=2.93, p < .001). Youth with higher pre-training intentions also had higher post-training intentions (t417=14.05, p < .001). Pre-training intentions significantly predicted involvement in training (t417=5.24, p < .001), which subsequently was positively associated with post-training intentions (t417=3.62, p < .001), even beyond the effect of pre-training values.

However, this test suggested partial mediation by involvement since, once introduced to the model, the direct effects of pre-to-post intentions did not reduce to non-significance (Zhao, Lynch, & Chen, 2010). Therefore, an alternate model was examined which supported involvement as a moderator of the relationship between pre- and post-training intentions (Model 2, F3, 414=87.07, p < .001; ΔR2=.01). Lower levels of involvement reduced pre- to post-training intentions gains, particularly for those with lower involvement (−1SD: b=0.54, SE=0.04; t417=13.32, p < .001) compared with higher involvement (+1SD: b=0.39, SE=0.05; t417=8.04, p < .001). Trainees with high pre-training intention to refer maintained high intentions post-training, regardless of their level of involvement; however, trainees with low pre-training intentions who were highly involved reported significantly higher post-training intentions than those who were less involved (F3, 414=43.20, p < .001).

Alliance as a mediator versus moderator.

Alliance with the trainer mediated the relationship between pre- and post-training intentions to refer at-risk youth (Model 3, Z=4.29, p < .001). Youth with higher pre-training intentions also had higher post-training intentions (t417=13.16, p < .001). Pre-training intentions significantly predicted alliance (t417=6.88, p < .001), which subsequently was positively associated with post-training intentions (t417=5.47, p < .001), even beyond the effect of pre-training values. The alliance-intentions relationship was also partially mediated; therefore, an additional model was run that supported alliance as a moderator of the relationship between pre- and post-training intentions (Model 4, F3,414=101.56, p < .001; ΔR2=.23). Lower levels of alliance reduced pre- to post-training intentions gains, particularly for those with lower alliance (−1SD: b=0.54, SE=0.04; t417=13.54, p < .001) compared with higher alliance (+1SD: b=0.34, SE=0.04; t417=7.81, p < .001). Much like with involvement, trainees with high pre-training intentions to refer maintained high intentions at post-training, regardless of their level of trainee-trainer alliance, whereas trainees with low pre-training intentions who reported high trainee-trainer alliance showed significantly higher post-training intentions than those who reported low alliance (F3, 414=46.64, p < .001).

The relationship between alliance and involvement.

Based on results of Models 1–4, hybrid moderated-mediation models were tested. Models testing alliance and involvement as dual moderators of the pre- to post-training intentions relationship (Model 5, F5,412=0.53, p =0.47), alliance and involvement as serial mediators of the pre- to post-training intentions relationship (Model 6, F3,414=1.97, p =0.05), and alliance as a moderator and involvement as a mediator (Model 7, F3,414=1.30, p =0.19) were not supported. Finally, a model involving alliance as a mediator and involvement as a moderator was tested (Model 8), and this model provided the best fit with the data (see Table 3 and Figure 1). Involvement moderated the relationship between alliance and post-training intentions (t417=−2.39, p < .05), but did not moderate pre-training intentions and alliance (t421=−1.21, p = .23). Intentions to refer significantly increased from pre- to post-training for most participants (Low Alliance-Low involvement: t83=−1.99, p < .05; High Alliance-Low Involvement: t93=−1.97, p < .05; High Alliance-High Involvement: t186=−4.37, p < .001), with the exception of those reporting low alliance and high involvement who remained stable (t52=−1.18, p < .24). Those with low alliance and low involvement had significantly lower post-training intentions than all other participants (F3,414=16.31, p < .001; see Figure 2).

Table 3.

Alliance and involvement moderated-mediation analysis.

Final Selected Model R R2 F df p
Alliance .454 .206 35.830 3, 414 .0001
Post-Training Intentions to Refer .644 .415 73.114 4, 413 .0001
Coefficient (SE) t p LLCI / ULCI
Alliance
   Constant 2.139 (0.656) 3.261 .01 0.859 3.429
   Pre-Training Intentions to Refer 0.391 (0.123) 3.185 .01 0.150 / 0.632
Post-Training Intentions to Refer
   Constant 1.180 (0.485) 2.433 .02 0.227 / 2.133
   Pre-Training Intentions to Refer 0.436 (0.035) 12.603 .0001 0.368 / 0.504
   Alliance 0.338 (0.084) 4.014 .001 0.172 / 0.503
   Involvement 0.342 (0.121) 2.824 .005 0.104 / 0.579
   Alliance x Involvement Interaction −0.047 (0.020) −2.389 .02 −−0.086 / −0.008
   Pre-Training Intentions to Refer x
   Involvement Interaction		 −0.032 (0.026) −1.212 .23 −0.084 / 0.020
Open in a new tab

Note: N=418.

Figure 1.

Figure 1

Open in a new tab

Path estimates (and standard errors) for the model in which the relationship between pre- and post-training intentions to refer are mediated by alliance and the relationship between alliance and post-training intentions is moderated by student involvement in training. *p < .05; **p < .01; ***p < .001.

Figure 2.

Figure 2

Open in a new tab

Pre- and post-training intentions to refer an at-risk peer, where the relationship between pre- and post-training intentions are mediated by alliance and the relationship between alliance and post-training intentions is moderated by involvement in training.

A slope analysis of the moderated-mediation effects found that youth reporting high levels of alliance showed significantly different patterns of intentions to refer than youth reporting low levels of alliance (Pre: F3,414=9.78, p < .001; Post: F3,414=16.31, p < .001). When standardized scores were calculated to compare youth trainees’ ratings to the average score for each time point, trainees reporting high alliance endorsed pre-training and post-training intentions that were slightly above average, regardless of reported level of involvement. Alternatively, trainees with low levels of alliance but high reported involvement showed pre-training intentions similar to the average that decreased at post-training. Lastly, trainees with both low levels of alliance and involvement reported significantly lower intentions to refer at both pre- and post-training, resulting in post-training intentions that were lower than any other group (approximately 0.5–0.8 standard deviations below other groups).

DISCUSSION

The present study assessed the relationship between youth trainee involvement and youth trainee-health education teacher trainer alliance in explaining changes in prevention program outcomes, namely students’ intentions to utilize learned skills following participation in youth gatekeeper suicide prevention programming. To our knowledge, this study is the first to examine the role that involvement and alliance jointly contribute to intentions to change referral behavior in prevention science, particularly suicide prevention. Based on findings from the clinical literature (Karver et al., 2008; Shirk et al., 2011), it was expected that positive alliance between trainer (health education teacher) and trainee (student), as well as greater involvement in programming, would contribute to improvements in intentions to utilize learned skills and knowledge from gatekeeper sessions (i.e., ability to refer at-risk peers). However, the exact process through which these associations would occur as part of a prevention program was not clear, and therefore several models were tested to determine the best fit of these factors in explaining changes in program outcomes.

Results indicate that, similar to the clinical treatment literature, alliance acts as a mediator of prevention program changes. In turn, level of trainee involvement functions as a moderator of the association between alliance and post-training outcomes. Specifically, student trainees who report high alliance and involvement had the greatest intentions to use their suicide prevention knowledge and skills following training participation. These trainees may see value in the training and become encouraged to actively participate as a result of positive modeling and relationship development with their trainer. Thus, it can be assumed that greater involvement coupled with alliance potentially translates to greater skill acquisition and willingness to utilize these skills.

Similarly, participants reporting low involvement and high alliance also reported high intentions to use their knowledge and skills, suggesting that bonding with an influential trainer may mitigate against a lack of active participation. Yet among these participants, despite reported intentions to refer, it remains unknown whether effective skill and knowledge acquisition can be achieved with minimal trainee involvement in practicing, discussing, or brainstorming prevention strategies. Since alliance is measured at the same time-point as involvement and intentions (i.e., post-training), parsing directionality of alliance effects is difficult. Researchers have found that changes in alliance over time have variable impact on clinical treatment outcomes (Hogue, Dauber, Stambaugh, Cecero, & Liddle, 2006; Fluckiger, Del Re, Wampold, Symonds, & Horvath, 2012); therefore, it is quite possible that greater perceived gains from training could contribute to greater satisfaction with the training experience, and therefore more positive perceptions of trainers.

Notably, participants with both the least involvement and lowest alliance had the poorest intentions to refer, which was true at both pre and post-training. Perhaps, these participants were not given adequate information to engage them in the topic, or they may have simply “checked out” of program activities. Intriguingly, participants with high levels of involvement but low alliance with their trainer made pre- to post training improvement in intentions to refer, but nevertheless reported lower than average intentions to refer at post-training. These participants may have had high motivation when starting the program, but potentially became disappointed when they could not relate to their trainer or the messages conveyed by the trainer may have resulted in negative perceptions of the trainer and training. This inability to connect meaningfully with the main individual modeling normative beliefs about the importance of the prevention program has tremendous implications for trainees’ intentions to use what they learned. A recent review indicated that intentions to perform and lack of environmental constraints are necessary and sufficient prerequisites for behavior change when motivation is present (Michie, van Stralen, & West, 2011). Consistent with this notion and the present findings, it is possible that alliance is a necessary and sufficient “environmental” component for implementation, while involvement alone is necessary but not sufficient. These findings underscore the importance of fostering trainee-trainer alliance in prevention work. Moreover, given that alliance occurs as a function of characteristics that both trainers and trainees bring to the prevention context, it may be that trainers have a more difficult time developing relationships with trainees they perceive as low in involvement. Conversely, participants may have a more difficult time developing an alliance with trainers perceived as less engaged in training or with low capacity to conduct them. Trainer behaviors such as active listening, incorporation of trainees’ experiences into sessions, being responsive to training needs/learning styles, and demonstrating interest in the program and its trainees can improve alliance development (Karver et al., in press; Shirk et al., 2011). Much like measures of fidelity and dosage, gatekeeper training programs would be well-informed by recurrent assessment of alliance over time so that trainers can continue to tailor their approaches and ensure youth trainees are engaged. These are areas worthy of future study, especially given that much work remains to be conducted on how to improve youth alliances even in the treatment literature (Murphy & Hutton, 2018).

While the findings from this study help clarify the role of alliance and involvement in prevention work, they also introduce further questions. As only participant report was used, it is unclear whether trainer or observer report of trainee involvement would strengthen study methodology. Second, while the psychometrics of the PBIQ are strong, the items were created for the purposes of this study, and only one item measured alliance. Consensus in the adult treatment literature suggests that alliance is generally comprised of three components: 1) emotional or affective bond, 2) goal consensus between therapist and client, and 3) task collaboration (Bordin, 1979). However, most studies conducted with youth have failed to validate this model. Empirical studies and meta-analyses alike indicate that the affective/emotional bond component of alliance (i.e., bonding and likeability) is what most promotes therapeutic work with youth (Karver, et al., in press; Shirk et al., 2011), and that if the client likes the therapist, they tend to rate all other aspects of the alliance similarly. A recent study by Bickman et al. (2012) used item response theory and determined that a single alliance item was sufficient to determine youth alliance in therapy. However, it is also possible that the youth therapy alliance literature may need further investigation, given that much of the research is derived from adult alliance theory that potentially may not apply to youth and how they form relationships. Regardless, research in the youth treatment domain suggests that the single item used in the current study captures the critical component of alliance development (i.e., bonding and likeability; Bickman et al., 2012; Karver, et al., in press; Shirk et al., 2011). As this was a group-based prevention program in which participation was part of standard health curricula and not targeted at participants’ personal therapeutic concerns, there may also have been fewer opportunities for shared goals and task collaboration directly between trainer and participant, thereby potentially rendering these components less relevant within the prevention context. Additional research should be conducted in prevention to determine the key aspects of youth alliance predictive of youth outcomes, as well as with the PBIQ to further establish its validity and utility. For instance, while it is speculated that alliance fosters trust among participants, which then may contribute to program outcomes, this was not specifically assessed in this study.

Additionally, only involvement, and not motivation to participate, was measured, leaving unclear whether involvement was driven by a desire to participate and learn skills or if participation was motivated by other factors (e.g., grade seeking). Assessing motivation would give insight into whether participants engaged sufficiently to acquire an effective level of skill development. As an extension, actual skill development was not measured as a function of involvement. Even if participants have high intentions to use their skills, if they do not have the opportunity to demonstrate them in the training context, there is no guarantee they will be able to do so in real world situations. Furthering this limitation is the inclusion of intentions to refer rather than capturing actual referral behaviors. Although the Theory of Planned Behavior (Azjen, 1985) suggests that intentions are a close proxy to engagement in behaviors, they are not a perfect correlate – one may have intentions and not behave consistently with them. Yet, this study did not involve a lengthy evaluation design and likely would not have captured an adequate number of referral behaviors to meet statistical power needs. Therefore, intentions to refer at-risk youth was determined the best outcome immediately following program participation. Future research should focus on incorporating longer evaluation periods in program implementation in order to allow greater opportunity for demonstration of prevention behaviors, which are a more true indication of effectiveness. Finally, alliance and involvement were measured in this study within the context of youth suicide prevention. It would be important for further research to test if these effects could be replicated in other prevention areas and with different populations.

Despite these limitations, the study employed a sizeable diverse sample of participants to be able to test multiple mediation and moderation models. Additionally, the study used strong statistical methods to determine the important mechanisms underlying intentions to use learned prevention behaviors. This exploration is merited, as little focus has previously been paid to the role of alliance and involvement in prevention, particularly in regard to a sensitive topic such as youth suicide prevention, for which implementation may be more challenging in community settings. Without doubt, trainee involvement in prevention activities is necessary to foster the intention to utilize newly trained knowledge and skills. Ideally, acquired knowledge and skills would translate to prevention-related behaviors (Ajzen, 1985; 2017); however, they do not automatically translate into intention to act. Likewise, having intentions to act does not suggest that a sufficient amount of prevention skill development occurred to act effectively. There may be circumstances in which involvement in programming is less than optimal, particularly those that may carry higher potential for stigma such as suicide prevention.

Future research should examine other moderated-mediated relationships to better understand the mechanism behind intentions to engage in prevention behaviors and actual behavior change in prevention programming. For instance, further investigation is warranted to determine whether intentions are internally motivated, or if participants are more open to taking action in a manner that is perceived as favorable by their peers or a trainer they like. Future research should also examine if alliance and involvement contribute similarly to perceived competency to use skills and actual prevention behaviors, especially with regard to other community prevention priorities. Fundamentally, the present study suggests that improving aspects of the prevention implementation climate (i.e., alliance) can mitigate potential problems with involvement and intentions to utilize learned prevention skills.

Acknowledgements:

Special thanks to Sarah Gunderson, Renee Brown Hangartner, Jason I. Chen, Sarah J. Tarquini, and Krista Kutash for their invaluable assistance.

Funding: This research was supported by Substance Abuse and Mental Health Administration grants (#SM57442-01, 02-3) to Drs. Karver, Kutash, and Totura. Dr. Labouliere was supported as a post-doctoral fellow by Award #2T32 MH16434-34 from the National Institute of Mental Health (NIMH) at the National Institutes of Health (NIH). The contents of the manuscript are solely the responsibility of the authors and do not necessarily represent the official views of NIMH or SAMHSA.

Footnotes

Compliance with Ethical Standards:

Conflict of Interest Disclosure: The authors declare that they have no conflicts of interest.

Ethical approval: All procedures performed in this study involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed consent: Informed consent was obtained from all participants included in the study via university Institutional Review Board approved waiver of parental signed informed consent.

References

  1. Ajzen I (1985). From intentions to actions: A theory of planned behavior. In Kuhl J & Beckman J (Eds.), Action-control: From cognition to behavior (pp. 11–39). Heidelberg: Springer. [Google Scholar]
  2. Ajzen I (2017). Constructing a Theory of Planned Behavior Questionnaire Retrieved from: http://people.umass.edu/aizen/pdf/tpb.measurement.pdf
  3. Berkel C, Mauricio AM, Schoenfelder E, & Sandler IN (2011). Putting the pieces together: An integrated model of program implementation. Prevention Science, 12, 23–33. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Berry K, Gregg L, Lobban F, & Barrowclough C (2016). Therapeutic alliance in psychological therapy for people with recent onset psychosis who use cannabis. Comprehensive Psychiatry, 67, 73–80. [DOI] [PubMed] [Google Scholar]
  5. Bickman L, deAndrade ARV, Athay MM, Chen JI, De Nadai AS, Jordan-Arthur BL, & Karver MS (2012). The relationship between change in therapeutic alliance ratings and improvement in youth symptom severity: Whose ratings matter the most? Administration and Policy in Mental Health and Mental Health Services Research, 39, 78–89. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Bordin ES (1979). The generalizability of the psychoanalytic concept of the working alliance. Psychotherapy: Theory, Research, and Practice, 16, 252–260. [Google Scholar]
  7. Brown Hangartner R, Totura CMW, Labouliere CD, Gryglewicz K, & Karver MS (2018). Benchmarking the Question, Persuade, Refer program against established suicide prevention gatekeeper trainings. Suicide and Life-Threatening Behavior, in press. [DOI] [PMC free article] [PubMed]
  8. Brown CH, Wyman PA, Brinales JA, & Gibbons RD (2007). The role of randomized trial in testing interventions for the prevention of youth suicide. International Review of Psychiatry, 19, 617–631. [DOI] [PubMed] [Google Scholar]
  9. Bruffaerts R, Demyttenaere K Hwang I, Chiu WT, Sampson N, Kessler RC, … & Nock MK (2011). Treatment of suicidal people around the world. British Journal of Psychiatry, 199, 64–70. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Castro-Blanco D, Karver MS, & Chiechi J (2010). Conclusions: Looking ahead—Future directions in treatment engagement with high risk adolescents. In Castro-Blanco D & Karver MS (Eds.), Elusive alliance: Treatment engagement strategies with high-risk adolescents Washington, D.C.: American Psychological Association. [Google Scholar]
  11. Centers for Disease Control and Prevention. (2016). Web-based Injury Statistics Query and Reporting System, Fatal Injury Data Retrieved on January 25, 2018 from https://www.cdc.gov/injury/wisqars/fatal.html.
  12. Cummings C, Caporino N, Settipani C, Read K, Compton S, March J… & Kendall P (2013). The therapeutic relationship in cognitive-behavioral therapy and pharmacotherapy for anxious youth. Journal of Consulting and Clinical Psychology, 81, 859–864. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Durlak JA & DuPre EP (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350. [DOI] [PubMed] [Google Scholar]
  14. Fixsen DL, Naoom SF, Blase KA, & Friedman RM (2005). Implementation research: Synthesis of the literature Tampa, FL: University of South Florida. [Google Scholar]
  15. Fluckiger C, Del Re AC, Wampold BE, Symonds D, & Horvath AO (2012). How central is the alliance in psychotherapy? A multilevel longitudinal meta-analysis. Journal of Counseling Psychology, 59, 10–17. [DOI] [PubMed] [Google Scholar]
  16. Fredricks JA, Bohnert AM, & Burdette K (2014). Moving beyond attendance: Lessons learned from assessing engagement in afterschool contexts. New Directions for Student Leadership, 144, 45–58. [DOI] [PubMed] [Google Scholar]
  17. Garvey C, Julion W, Fogg L, Kratovil A, & Gross D (2006). Measuring participation in a prevention trial with parents of young children. Research in Nursing & Health, 29, 212–222. [DOI] [PubMed] [Google Scholar]
  18. Hagger MS, Chan DK, Protogerou C, & Chatzisarantis NL (2016). Using meta-analytic path analysis to test theoretical predictions in health behavior: An illustration based on meta-analyses of the theory of planned behavior. Preventive Medicine, 89, 154–161. [DOI] [PubMed] [Google Scholar]
  19. Hayes AF (2013). Introduction to mediation, moderation, and conditional process analysis: A regression-based approach New York: Guilford Press. [Google Scholar]
  20. Hayes AF (2018). Introduction to mediation, moderation, and conditional process analysis: A regression-based approach (2nd Edition). New York: Guilford Press. [Google Scholar]
  21. Hogue A, Dauber S, Stambaugh LF, Cecero JJ, & Liddle HA (2006). Early therapeutic alliance and treatment outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting and Clinical Psychology, 74, 121–129. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Horvath AO, Del Re AC, Flückiger C, & Symonds D (2011). Alliance in individual psychotherapy. Psychotherapy, 48, 9–16. [DOI] [PubMed] [Google Scholar]
  23. Johansson P & Høglend P (2007). Identifying mechanisms of change in psychotherapy: Mediators of treatment outcomes. Clinical Psychology & Psychotherapy, 14, 1–9. [Google Scholar]
  24. Kalafat J, & Elias M (1994). An evaluation of a school-based suicide awareness intervention. Suicide and Life-Threatening Behavior, 24, 224–233. [PubMed] [Google Scholar]
  25. Karver MS, De Nadai AS, Monahan M, Shirk SR (in press). Meta-analysis of the prospective relation between alliance and outcome in child and adolescent psychotherapy. Psychotherapy [DOI] [PubMed]
  26. Karver M, Shirk S, Handelsman J, Fields S, Crisp H, Gudmundsen G, & McMakin D (2008). Relationship processes in youth psychotherapy: Measuring alliance, alliance-building behaviors, and client involvement. Journal of Emotional & Behavioral Disorders, 16, 15–28. [Google Scholar]
  27. Kim HY & Cappella E (2016). Mapping the social world of classrooms: A multi-level, multi-reporter approach to social processes and behavioral engagement. American Journal of Community Psychology, 57, 20–35. [DOI] [PubMed] [Google Scholar]
  28. Labouliere C, Reyes JP, Shirk S, & Karver M (2017). Therapeutic alliance with depressed adolescents: Predictor or outcome? Disentangling temporal confounds to understand early improvement. Journal of Clinical Child & Adolescent Psychology, 46, 600–610. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Labouliere C, Tarquini S, Totura C, Kutash K, & Karver M (2015). How much is learned by students participating in suicide prevention gatekeeper training? Crisis, 36, 274–280. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Low S, Van Ryzin MJ, Brown EC, Smith BH, & Haggerty KP (2014). Engagement matters: lessons from assessing classroom implementation of steps to respect: A bullying prevention program over a one-year period. Prevention Science, 15, 165–176. [DOI] [PubMed] [Google Scholar]
  31. McCormick MP, O’Connor EE, Cappella E, & McClowry SG (2013). Teacher–child relationships and academic achievement: A multilevel propensity score model approach. Journal of School Psychology, 51, 611–624. [DOI] [PubMed] [Google Scholar]
  32. McLeod BD, Islam NY, Chiu A,W, Smith MM, Chu BC, & Wood JJ (2014). The relationship between alliance and client involvement in CBT for child anxiety disorders. Journal of Clinical Child & Adolescent Psychology, 43, 735–741. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Michie S, van Stralen M, & West R (2011). The behavior change wheel: A new method for characterizing and designing behavior change interventions. Implementation Science, 6, 1–11. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Murphy R, & Hutton P (2018). Practitioner review: Therapist variability, patient-reported therapeutic alliance, and clinical outcomes in adolescents undergoing mental health treatment – a systematic review and meta-analysis. The Journal of Child Psychology and Psychiatry, 59(1), 5–19. [DOI] [PubMed] [Google Scholar]
  35. Nix RL, Bierman KL, McMahon RJ, & The Conduct Problems Prevention Research Group. (2009). How attendance and quality of participation affect treatment response to parent management training. Journal of Consulting and Clinical Psychology, 77, 429–438. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Prochaska JO, DiClemente CC, & Norcross JC (1992). In search of how people change: Applications to the addictive behaviors. American Psychologist, 47, 1102–1114. [DOI] [PubMed] [Google Scholar]
  37. Sandler I, Wolchik SA, Cruden G, Mahrer NE, Ahn S, Brincks A, & Brown CH (2014). Overview of meta-analyses of the prevention of mental health, substance use, and conduct problems. Annual Review of Clinical Psychology, 10, 243–273. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Shirk SR, Karver MS, & Brown R (2011). Alliance in child and adolescent psychotherapy. Psychotherapy, 48, 17–24. [DOI] [PubMed] [Google Scholar]
  39. Thompson SJ, Bender K, Lantry J & Flynn PM (2007). Treatment engagement: Building therapeutic alliance in home-based treatment with adolescents and their families. Contemporary Family Therapy, 29(1–2), 39–55. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Totura CMW, Kutash K, Labouliere CD, & Karver M (2017). Evaluating active parental consent procedures for school programming: Addressing the sensitive topic of suicide prevention. Journal of School Health, 87, 114–120. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Totura CMW, Lewis Figueroa H, Wharton C, & Marsiglia FF (2015). Assessing implementation of evidence-based childhood obesity prevention strategies in schools. Preventive Medicine Reports, 2, 347–354. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Totura CMW, Tarquini S, Naoom S, Karver MS, & Kutash K (2008). Planned Behavior & Implementation Questionnaire Unpublished Tampa, FL: University of South Florida. [Google Scholar]
  43. Wandersman A, Duffy J, Flaspohler P, Noonan R, Lubell K, Stillman L, … & Saul J (2008). Bridging the gap between prevention research and practice: The interactive systems framework for dissemination and implementation. American Journal of Community Psychology, 41(3–4), 171–181. [DOI] [PubMed] [Google Scholar]
  44. Weisz JR, Doss AJ, & Hawley KM (2005). Youth psychotherapy outcome research: A review and critique of the evidence base. Annual Review of Psychology, 56, 337–363. [DOI] [PubMed] [Google Scholar]
  45. Whittaker KA, & Cowley S (2012). An effective programme is not enough: A review of factors associated with poor attendance and engagement with parenting support programmes. Children & Society, 26, 138–149. [Google Scholar]
  46. Xia Y, Taylor S, & de Guzman MRT (2017). Evaluation of the Health Rocks! Program: The association of youth engagement with program outcomes. Journal of Youth Development, 11, 95–104. [Google Scholar]
  47. Zhao X, Lynch JG, & Chen Q (2010). Reconsidering Baron and Kenny: Myths and truths about mediation analysis. Journal of Consumer Research, 37, 197–206. [Google Scholar]

RESOURCES