Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2022 Jul 1.
Published in final edited form as: Health Psychol. 2021 Jul;40(7):450–458. doi: 10.1037/hea0001094

Replicating dissemination and identifying mechanisms of implementation of an empirically supported treatment

Stephen B Lo 1, Claire C Conley 2, Brittany M Brothers 3, Marlena M Ryba 4, Georita F Frierson 5, Rebecca A Shelby 6, Lisa M Thornton 7, Kristen M Carpenter 7, Barbara L Andersen 1
PMCID: PMC8432620  NIHMSID: NIHMS1729843  PMID: 34435796

Abstract

Objective:

Implementation research is needed in cancer control. Replication of the dissemination of empirically supported treatments (ESTs) is important as is the identification of mechanisms by which dissemination leads to implementation. Addressing these gaps, Study I (cohorts 3–6, N=104) tests for replication of a successful dissemination to community providers (Brothers et al., 2015; cohorts 1–2; N=62) and Study II (cohorts 1–6) tests providers’ changes on dissemination outcomes as mechanisms of EST usage.

Methods:

The Biobehavioral Intervention (BBI), a psychological EST in cancer control, was disseminated to oncology mental health providers using manual provision, didactics, roleplays, and other strategies. Study I tested for pre/post changes in dissemination outcomes (BBI knowledge/skills and attitudes toward and self-efficacy to deliver ESTs/BBI) between cohorts (1–2 vs. 3–6) with repeated measures ANOVAs. In Study II, the implementation outcome was providers’ (N=166) BBI usage with patients (percent treated). Structural equation models tested dissemination outcome changes as predictors of usage at 2-and 4-months.

Results:

Study I replicated high dissemination outcomes and significant gains in BBI knowledge (p<.001) in cohorts 3–6. Unlike cohorts 1–2, significant gains were observed in self-efficacy (ps<.001) but not attitudes towards ESTs (p=.523) in cohorts 3–6. In Study II, gains in providers’ self-efficacy (ps<.05) and EST attitudes (p=.008) predicted greater 2-month (58.4%±35.5%) and 4-month (66.2%±35.0%) usage, respectively of the BBI with patients.

Conclusions:

This is the only replication of dissemination of a psychological EST in cancer control. Disseminations enhancing providers’ self-efficacy to deliver and instilling positive attitudes towards ESTs are mechanisms for achieving EST implementation.

Keywords: dissemination, implementation, mechanisms, biobehavioral, empirically supported treatment

Researchers may spend decades to develop, refine, and demonstrate empirical support for treatments prior to dissemination—the education of a target audience such as community providers on how to use the treatment (Rabin et al., 2008). For complex innovations, such as psychological empirically supported treatments (ESTs) that require learning both content and how to deliver it, dissemination must be comprehensive to achieve the distal outcome of providers’ EST implementation—the integration and use of the EST in the providers’ setting. Recognized decades ago in medicine (Haines & Jones, 1994), the science to practice gap for psychological/behavioral ESTs persists. Dissemination and implementation studies are needed, and particularly in cancer control (Mitchell & Chambers, 2017; Rankin et al., 2019). Unsurprisingly, a literature search finds no replications of dissemination studies of ESTs targeting stress or psychological responses in patients with cancer to inform dissemination best practices. Further, the study of mechanisms by which dissemination leads to implementation offers efficiency and clarity to tailor disseminations to maximize the likelihood of provider implementation (Lewis et al., 2020).

In general, more dissemination research is needed in health psychology. There is little consensus on the dissemination strategies to achieve providers’ EST usage in their settings (Powell et al., 2015; Tabak et al., 2012) or the conceptual models and frameworks to guide these efforts (Damschroder et al., 2009; Nilsen, 2015). Implementation can be viewed as a process of behavior change with dissemination needing to be sufficiently positive and rigorous to instigate change in providers. Thus, an important task is identifying the processes (mechanisms) of dissemination that led to implementation.

The only systematic review on this topic identified 17 quantitative studies examining mechanisms of implementation of complex treatments such as ESTs (Lewis et al., 2020). Of them, three were disseminations to teachers or mental health providers to deliver guideline-prescribed (Atkins et al., 2008) or evidence-based treatment (Aarons et al., 2009; Williams et al., 2017). More common were cross-sectional studies correlating providers’ current usage to potential mechanism variables. Relevant theories for such research include Fishbein & Ajzen’s Theory of Planned Behavior (Fishbein & Ajzen, 1975) to test, for example, if providers’ positive EST attitudes covary with usage (e.g., Ryba et al., 2019). Also used is social cognitive theory, with its emphasis on providers’ self-efficacy (Rapport et al., 2018), in which confidence or perceived ease to deliver an EST is correlated with usage (David & Schiff, 2017; Harned et al., 2013). Conversely, low self-efficacy may be a barrier to implementation (Couineau & Forbes, 2011). Knowledge building during dissemination has been well studied, but knowledge gains are unrelated to implementation (Lavoie et al., 2017; Pinto et al., 2018).

Although coming from different conceptualizations, these studies provide early evidence to test if disseminations that yield gains in providers’ knowledge, positive attitudes toward the EST, or high self-efficacy to deliver the EST might portend future EST usage (Powell et al., 2019). Indeed, prospective tests of mechanisms could discover key variables upon which disseminations should focus (Powell et al., 2015; Waltz et al., 2015). Further, repeated assessments of usage could determine if the mechanisms for achieving providers’ early implementation of an EST have similar relevance for its continuation. Two studies are provided here to address the gaps in dissemination and implementation research in cancer control—the need for replication of successful dissemination and identification of mechanisms of implementation.

Data come from a project funded through the National Institutes of Health/National Cancer Institute (NIH/NCI) program (R25E) to train cancer care providers in new evidence-based strategies or to use current strategies in new ways or with underserved groups. This particular effort enrolled full-time oncology mental health providers practicing in diverse settings to learn a psychological EST in cancer control, the Biobehavioral Intervention (BBI). As designed and manualized, the BBI is a multicomponent treatment (i.e., stress conceptualization with progressive muscle relaxation, information about disease/treatment, problem-solving, assertive communication, social support, sexuality, and health behaviors; Andersen et al., 2009). When tested, it yielded robust improvements in psychological, behavioral, immune, health, and disease outcomes (Andersen et al., 2004; Andersen et al., 2010; Andersen et al., 2008).

A conceptual framework (Andersen & Dorfman, 2016) was used to design BBI dissemination to achieve implementation that could be sustained. Three-day BBI training Institutes, the vehicle for dissemination, used evidence-based learning activities and principles of adult learning effective for skill and knowledge acquisition generally (DeNeve & Heppner, 1997) and clinical skills in particular (Sterling-Turner, 2002). The aim was to achieve provider gains in knowledge, clinical skill, attitudes, and self-efficacy, and eventual implementation of BBI with patients. Data from the first cohorts (1–2; N=62) found that providers rated the quality of BBI instruction highly (M=4.66 ± 0.26; possible range=0–5) and had improvements in proximal dissemination outcomes—BBI knowledge and positive attitudes toward ESTs (Brothers et al., 2015).

Building upon this, two studies are provided. Study I is a replication test of Brothers et al. (2015), using a new sample, to study the efficacy of the Institute training at improving dissemination outcomes: providers’ EST knowledge, attitudes, and self-efficacy. Two tests of replication were planned, with hypotheses being that 1) post-Institute outcomes would not be significantly different between the earlier cohorts (1–2) vs. those later (3–6), and 2) pre- to post-Institute outcome gains for cohorts 3–6 would replicate those of cohorts 1–2, with similar if not larger effect sizes. Study II is an extension, testing mechanisms by which dissemination outcomes impact implementation. As the providers alone were responsible for implementation, many faced early barriers to implementation within their facility/practice (Williams et al., 2015) or needed to make changes in their clinical practices to enable BBI delivery. Nevertheless, providers’ (cohorts 1–6) usage of the BBI was assessed at 2- and 4-months post-Institute. Using both dissemination and implementation data, changes in EST knowledge, attitudes, and self-efficacy arising from dissemination were viewed as mechanisms and tested as predictors of providers’ early (2-month) and later (4-month) EST usage. As discussed above (e.g., David & Schiff, 2017; Pinto et al., 2018; Ryba et al., 2019), gains in attitudes and self-efficacy, rather than knowledge, from dissemination were hypothesized to predict EST usage.

Study I: Replication test of Brothers et al. (2015)

Methods

Participants

Oncology mental health providers (cohorts 3–6; N=104) attending BBI Institutes (described below) were studied. The sample was predominantly female (n=95; 91%), mid-age (M=41.1 years ± 9.4; range=25–62), and Caucasian (n=81; 78%). Providers came from 29 US states and Puerto Rico and two foreign countries (Kenya and Mexico). Providers’ disciplines were social work (n=42; 40%), clinical psychology (n=42; 40%), post-doctoral fellows in psychology (n=6; 6%), nursing (n=1; 1%), and others (n=13; 13%). All were full-time clinicians employed at academic medical centers or Veterans Administration Hospitals (n=47; 45%), community hospitals/centers (n=35; 34%), community supportive care facilities (n=12; 12%), private practices (n=8; 8%), or other settings (n=2; 2%). Employed an average of 5.3 years (SD=5.8; range=0–30) in their current position, providers spent most of their time in clinical service provision (70% ± 26) with the remainder being administration or teaching. The majority were licensed (n=91; 88%) and had been so for an average of 11.1 years (SD=8.5; range=1–36).

Procedures

Complete descriptions of enrollment procedures have been provided (Brothers et al., 2015). Briefly, BBI Institute announcements were placed on listservs (e.g., Society for Health Psychology) and the Institute website. Applications were sought from licensed, full-time, oncology mental health providers. A letter of reference from one’s supervisor (i.e., the person who could authorize BBI implementation in the setting) was required. Six 3-day (18 hours) Institutes were conducted from 2012–2016 with results from the last four cohorts (2014–2016) reported here. Following informed consent and before attending, providers completed self-report measures using Qualtrics; posttest measures were completed via paper/pencil on the last day of training. Continuing education credit up to 18 hours was available and research funding provided small travel stipends.

The BBI Institutes were designed using principles of adult learning (andragogy; Knowles, 1980), i.e., 1) cooperative climate; 2) prior assessment of providers’ skills, setting, and patients; 3) learning objectives for each component; 4) sequential activities; and 5) continuous provider evaluations of Institute quality. Providers’ readiness and motivation to learn BBI content and skills directly applicable to his/her patients were assumed. The goals for providers were as follows: 1) understand the conceptualization and empirical support for BBI and its components; 2) become BBI content knowledgeable; 3) develop BBI clinical skill as well as positive attitudes and confidence to use BBI; 4) use BBI manuals and patient-reported outcome (PRO) measures; and 5) formulate a plan for BBI adaptation, if necessary, and implementation. Seven clinical psychologists with expertise in the BBI were trainers for all Institutes. Instruction modes were the following: 1) therapist and patient manuals; 2) didactics; 3) large group discussion and roleplays of BBI component delivery amongst providers with trainer monitoring, feedback, discussion; 4) practice within small groups with individualized feedback; and 5) five 10-minute videotaped “sessions” with a patient confederate (see below).

This study was approved by the IRB at the Ohio State University. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Measures

Knowledge

The BBI Knowledge Test [BBI-Know (Brothers et al., 2015)] assessed the 7 different components of the BBI with 32 multiple choice and true/false items. Items were summed and percent correct is reported. Internal consistency was α=0.542.

Clinical Skill

The Clinical Analogue Assessment (BBI-Clin) consisted of five tasks to evaluate providers’ conceptual and clinical skills using the BBI. Providers received a vignette describing a patient with a specific concern that could be treated with a BBI component (stress conceptualization, problem-solving, social support, sexuality, and health behaviors). Each provider completed five different audio/video taped vignette sessions during which the provider had 6-minutes to interact with a trained patient confederate. Recordings were rated for mastery of content delivery and skills. Ratings were adapted from the cognitive therapy rating scale (CTRS; Blackburn et al., 2001; Young & Beck, 1980). For each of the five vignettes, the provider was rated on 19 items assessing content/skill delivery using a 0 to 6 scale (e.g., for problem-solving: 0=only mentions brainstorming for generating solutions; 6=3 of 3 principles discussed with rationales included). A BBI-Clin score was computed by averaging the 19 items across the 5 vignettes, for an overall BBI-Clin score ranging from 0–6. BBI competence was defined as an individual score of ≥ 3.5 out of 6 (i.e., ≥ 60% of the total possible score), consistent with the CTRS standard. The training of raters and inter-rater reliability >0.80 has been described (Brothers et al., 2015). BBI-Clin data is available for Institutes 1–5 but not 6 (tapes exist but were unscored due to funding limitations).

Attitudes

Two measures were used. 1) The Evidence-Based Practice Attitudes Scale (EBPAS; Aarons, 2004) assessed general attitudes towards evidence-based practices and were operationalized as attitudes towards ESTs (EST-Att). The EBPAS consists of 15 items (e.g., “I am willing to try new types of therapy/interventions even if I have to follow a treatment manual”) rated on a 5-point Likert scale (0 = not at all to 4 = to a very great extent). Items were summed, with scores ranging from 0–60; higher scores indicate more positive attitudes. Internal consistency was α=.736. 2) BBI-specific attitudes (BBI-Att) were assessed using the semantic differential method (Osgood et al., 1957). Thirty pairs of bipolar adjectives descriptive of the BBI assessed the dimensions of evaluation (11 items; e.g., worthless/valuable), potency (13 items; e.g., complex/simple), and activity (6 items; e.g., dull/stimulating). Providers were asked to rate each pair in terms of how well the adjectives described BBI. The two words of each pair were anchors, for which a 7-point Likert scale (−3 = worthless to +3 = valuable) was used. The score for each item pair was summed, with possible scores ranging from −90 to +90; negative scores reflected negative attitudes about the BBI, and positive scores reflected positive attitudes. Internal consistency was α=.895.

Self-efficacy

Two measures were used. 1) The Counselor Activity Self-Efficacy Scale (Lent et al., 2003) consists of 25 items assessing general self-efficacy of basic (e.g., performing helping skills) and advanced (e.g., handling challenging counseling situations) counseling skills (General SE) rated on a 10-point Likert scale (0 = not at all confident to 9 = totally confident). Items were summed, with scores ranging from 0 to 225; higher scores indicate greater general counseling self-efficacy. Internal consistency was α=.952. 2) The BBI-specific self-efficacy scale (BBI-SE) used 8-items to assess providers’ confidence to deliver BBI components (e.g., progressive muscle relaxation, problem-solving), each rated on a 10-point Likert scale (0 = not at all confident to 9 = totally confident). Items were summed, with scores ranging from 0–72; higher scores indicate greater self-efficacy to deliver the BBI. Internal consistency was α=.852.

Analytic Plan

Preliminary analyses identified covariates for cohorts 3–6 repeated measures analysis of variance (rANOVA) by using one-way analysis of variance (ANOVA) to test for any differences in knowledge, attitudes, or self-efficacy scores by cohort (3–6), profession (psychologist vs other), and providers’ employment setting [hospital (university, Veterans Administration, community hospitals) vs. non-hospital].

The first test of replication used ANOVAs to compare cohort groups (1–2 vs. 3–6) on post-Institute dissemination outcomes, and the second used rANOVAs with covariates as needed to examine the change in outcomes pre- to post-Institute for cohort 3–6. All analyses used two-tailed tests with p <.05 considered significant and were completed on SPSS v22.0.

Results

Descriptive and Preliminary

See Table 1 for mean and standard deviation statistics and observed ranges on dissemination measures for both cohort groupings at pre- and post-Institute. Inspection of post-Institute data for cohorts 3–6 shows that providers scored high relative to possible ranges on knowledge of the BBI (BBI-Know mean = 83.2 ± 8.9; potential range = 0–100) and were rated as having a high skill level when delivering BBI (BBI-Clin mean = 3.9 ± 0.9; competence defined as > 3.5), with 74% of providers (52 of 70 rated) achieving competence. Providers reported positive attitudes towards ESTs (mean = 48.2 ± 5.9; potential range = 0–60) and BBI (mean = 60.6 ± 15.7; potential range = −90 - +90), and high self-efficacy (e.g., general SE mean = 199.2 ± 18.2; potential range = 0–225) to deliver the BBI. Inspection of the means for cohorts 1–2 showed similarly high scores.

Table 1.

Descriptive statistics for dissemination outcomes and rANOVAs results of pre- to post-Institute change for providers from cohorts 3–6 and published findings from cohorts 1–2 (Brothers et al., 2015).

Cohorts 3–6 (N=104) Cohorts 1–2 (N=62)
Dissemination Outcome Pre Mean ± SD (Range) Post Mean ± SD (Range) Effect size of time (partial η2) Pre Mean ± SD (Range) Post Mean ± SD (Range) Effect size of time (partial η2)
Knowledge/skills
 Knowledge (BBI-Know) 61.6 ± 12.8 (18.8–87.5) 83.2 ± 8.9* (56.3–100) .789 63.3 ± 9.8 (38–88) 83.8 ± 8.2* (63–97) .73
 Clinical skill (BBI-Clin)b —a 3.9 ± 0.8 (1.7–5.2) —a 3.5 ± 0.7 (1.8–4.7)
Attitudes (Att)
 Toward ESTs (EST-Att) 47.8 ± 5.4 (34–60) 48.2 ± 5.9 (29–60.0) .004 45.9 ± 6.9 (29–58) 47.6 ± 7.0* (34–59) .11
 Towards BBI (BBI-Att) —a 60.6 ± 15.7 (4–86) —a 56.2 ± 14.3 (24–85)
Self-efficacy (SE)
 Counseling (General-SE) 186.0 ± 21.6 (120–225) 199.2± 18.2* (125–225) .146 183.6 ± 27.8 (103–225) 196.1 ± 18.2 (156–225) .02
 BBI (BBI-SE) 56.2 ± 8.8 (30–72) 63.6 ± 6.8* (40–72) .262 54.9 ± 9.8 (34–72) 6.21 ± 6.1 (47–72) .02
Open in a new tab
a

Assessed at post-Institute only.

b

Score calculated for providers with vignette scores available (n=54 for Cohorts 1–2; n=70 for Cohorts 3–6).

*

Significant (p<.01) within group, pre- to post-Institute change. Analyses of self-efficacy included covariates of profession.

Note: Cohen’s rule of thumb for interpreting partial η2 effect sizes is as follows: small ≥0.01, medium ≥0.06, and large ≥0.14 (Cohen, 2013). ESTs=empirically supported treatments; BBI = Biobehavioral Intervention; BBI-Know = BBI Knowledge; BBI-Clin = Clinical Analogue Assessment; EST-Att = attitudes towards ESTs; BBI-Att = BBI-specific attitudes; general SE = general self-efficacy of counseling skills; BBI-SE = general self-efficacy

Regarding covariate selection for rANOVAs for cohorts 3–6, only providers’ profession was identified and only for self-efficacy. Compared to those in other professions, psychologists reported higher self-efficacy for providing ESTs [general SE mean = 203.5 vs 196.3; F(1,101)=3.984, p=.049] and the BBI [BBI-SE mean = 65.4 vs 62.3; F(1,101)=5.265, p=.024].

Dissemination Replication Tests

Results show that the positive post-Institute outcomes for cohorts 3–6 were not significantly different from those found for cohorts 1–2, indicating full replication. Providers had similarly high knowledge (BBI-Know M=83.2% vs. 83.8%, p=.637), clinical skill (BBI-Clin M=4.0 vs. 3.5, p=.051), self-efficacy (general SE M= 199.2 vs. 196.1, p=.291; BBI-SE M=63.6 vs. 62.1, p=.163), and positive attitudes (EST-Att M= 48.2 vs. 47.6, p=.163; BBI-Att M=60.6 vs. 56.2, p=.079).

rANOVAs show pre- to post-Institute gains for cohorts 3–6 to be largely equivalent, if not of a larger magnitude than those for cohorts 1–2. Significant gains in BBI Knowledge observed by Brothers et al. (2015) were replicated [partial η2=.789, F(1,102)=382.431, p<.001]. Unlike Brothers et al. (2015), cohorts 3–6 showed significant gains in self-efficacy, both general [partial η2=.146; F(1,101)=17.278, p<.001] and BBI-specific [partial η2=.262; F(1,101)=35.921, p<.001]. There was a failure to replicate significant gains in positive EST attitudes reported in Brothers et al. (2015), although effect sizes for both cohort groups were low (partial η2=.11 vs. .004).

Study II: Test of dissemination outcomes as mechanisms for EST usage

Methods

Participants

All providers (cohorts 1–6; N=166) were studied. The group was predominantly female (n=150; 90%), middle-aged (M=42.5 years ± 10.4; range=25–67), and Caucasian (n=136; 82%). Providers came from 35 US states and Puerto Rico and three foreign countries (Kenya, Mexico, and Malaysia). Providers were employed at academic medical centers or Veterans Administration Hospitals (n=68; 41%), community hospitals/centers (n=62; 37%), community supportive care facilities (n=15; 9%), private practice (n=8; 5%), and other settings (n=13; 8%). Providers had been employed an average of 5.8 years (SD=6.2; range=0–30) in their current position, primarily providing clinical services (71% ± 25). The majority were licensed (n=146; 88%) and had been so for an average of 12.5 years (SD=9.2; range=0–36). Providers’ disciplines were social work (n=69; 42%), clinical psychology (n=65; 39%), clinical psychology post-doctoral fellows (n=9; 5%), nursing (n=4; 2%), or others (n=19; 11%).

Procedures

See Study I for general procedures and IRB approval. Providers reported BBI usage 2- and 4-months post-Institute using Qualtrics and received a $15 gift card for completing a report.

Measures

BBI Implementation

Each provider reported, for the last two months, a) the total number of adult cancer patients treated, and b) the total number of adult cancer patients treated with the BBI. Usage was defined as the percentage of each provider’s patients treated with the BBI out of the total number of patients treated in the past two months. Since providers worked full-time providing clinical services, we anticipated that they could readily report the number of patients treated, corresponding to billing hours for many. Of note, the BBI was not conceptualized as appropriate for all cancer patients nor would it meet all clinical needs. This was discussed with providers during training.

Dissemination Outcomes

See Study I for descriptions of knowledge (BBI-Knowledge), attitudes (EST-Att), and self-efficacy [general SE, BBI-specific (BBI-SE)] measures.

Analytic Plan

Missing usage data were anticipated and considered. ANOVAs tested for differences between providers with and without usage data in knowledge, attitudes, self-efficacy, and provider/setting characteristics (profession and healthcare setting). Spearman correlations of BBI usage at 2 and 4 months with profession (psychologist vs other) and healthcare setting [hospital (university/Veterans Administration/community hospitals) vs. non-hospital] identified potential covariates for analyses.

For tests of mechanisms, structural equation modeling tested change for each dissemination outcome from pre- to post-Institute as a predictor of usage using a robust maximum likelihood estimation and a probit link function (Hayes, 2013; Muthén, 1984). Characteristics significantly correlated with usage, pre-Institute scores, and Institute cohort were entered as covariates for each model. Post-hoc analyses tested the models again using a conservative approach, i.e., replacing missing usage values with 0%. All analyses used two-tailed tests with a p <.05 considered statistically significant and were completed on R v3.4.0.

Results

Data Availability

Of the 166 providers, 119 (72%) and 114 (69%) provided usage data at 2 and 4 months, respectively. Of the non-reporting providers, 8 (17%) and 13 (25%) at 2 and 4 months, respectively, reported not having an opportunity to use the BBI (i.e., left the institution; needed more time to implement; medical leave). No differences in provider/setting characteristics or post-Institute knowledge, attitudes, and self-efficacy were found between providers with and without 2-month (ps>.056) or 4-month usage data (ps>.156).

Implementation

On average, providers reported using the BBI with 58.4% ± 35.5% of their patients at 2 months (N=119) before increasing to 66.2% ± 35.0% at 4 months (N=114). A minority of providers reported 0% usage at 2 (n=4; 2.4%) and 4 months (n=3; 1.8%). Usage was not correlated with healthcare setting or profession (ps>.05). Conservative usage estimates, replacing missing values with 0%, were 41.4% ± 38.9% at 2 months and 45.0% ± 41.4% at 4 months.

Tests of Dissemination Outcomes as Mechanisms for Implementation

Implementation at 2 Months

See Table 2 for a summary of model estimates for usage. The increase in providers’ self-efficacy, both general (general SE; b=.004; z=2.078; p=.038) and BBI-specific (BBI-SE; b=0.011; z=1.965; p=0.049), from dissemination predicted usage, controlling for pre-Institute scores and cohort. Practically speaking, providers reporting an increase of one standard deviation in general SE (18 points; range 0–225) or BBI-SE (7 points; range 0–72) at the end of dissemination would then go on to use the BBI with an additional 7% of their patients at 2 months. Changes in knowledge (BBI-Know; p=.185) and EST attitudes (EST-Att; p=.879) did not predict usage.

Table 2:

Model estimates of pre- to post-Institute change in dissemination outcomes predicting EST/BBI implementation (usage) at 2 and 4 months.

2 months 4 months
Dissemination Outcome b Z p-value b Z p-value
Δ Attitudes toward ESTs (EST-Att) −0.001 0.152 0.879 0.017 2.641 0.008*
Δ Self-efficacy for counseling skills (General SE) 0.004 2.078 0.038* 0.004 1.975 0.048*
Δ Self-efficacy for BBI (BBI-SE) 0.011 1.965 0.049* 0.011 1.791 0.073
Δ BBI knowledge (BBI-Know) 0.005 1.325 0.185 −.002 0.409 0.683
Open in a new tab
*

p<.05

Note: ESTs=empirically supported treatments; BBI = Biobehavioral Intervention; pre-Institute scores and cohort were controlled.

Implementation at 4 Months

Change in providers’ general self-efficacy (general SE; b=0.004; z=1.975; p=0.048) and attitudes (EST-Att; b=.017; z=2.641; p=.008) from dissemination predicted usage, controlling for pre-Institute scores and cohort. Providers reporting a standard deviation increase in EST-Att (6 points; range 0–60) and general SE (18 points; range 0–225) at the end of dissemination would use the BBI with an additional 10% and 7%, respectively, of their patients at 4 months. Changes in knowledge (BBI-Know; p=.683) and BBI-specific self-efficacy (BBI-SE; p=.073) did not predict usage.

Post-hoc Analyses

Analyses were repeated with missing usage values replaced with 0%. Greater change in general SE (b=.005; z=3.355; p=.001) and BBI-SE (b=.013; z=2.852; p=.004) predicted more usage at 2 months, as found in the primary analyses. BBI-SE was again significant (b=.013; z=2.716; p=.007) in predicting usage at 4 months, although EST-Att was not (p=.125). Unlike the prior 4-month analyses, change in general SE significantly (b=.005; z=2.833; p=.004) predicted more usage.

Discussion

New to the implementation science literature, this is the only replication of the effectiveness of dissemination of a health psychology EST. The discovery of mechanisms of EST implementation is also novel. Replication is fundamental to science (Ioannidis et al., 2014), and particularly so for a field as new as implementation science. With only a few examples in healthcare (Chinman et al., 2018; Katzman et al., 2016), this replication provides evidence for the reliability of a goal-driven, multimodal training using principles of adult education to produce gains in dissemination outcomes, i.e., knowledge and skill gains, positive attitudes toward the EST, and enhanced self-efficacy to deliver it. Providers’ usage of the EST was high, which enabled tests of mechanisms of implementation. Novel data show that enhancing providers’ self-efficacy and instilling positive attitudes during dissemination were instrumental for EST implementation.

Using a larger but similarly diverse sample, the dissemination outcomes reported for the later cohorts largely replicated those of the earlier cohorts (Brothers et al., 2015) with effects at least as large or if not larger. As experts view the science to practice gap to be due, in part, to using ineffective dissemination methods (Brownson et al., 2018; Powell et al., 2019), we consider the design and teaching methods that may have been important for achieving implementation. This might provide directions for specific strategies for future trainings of providers to deliver complex ESTs. For example, the first morning of the Institute provided the conceptualization for BBI and empirical support for the BBI. The intent was to provide a rationale for the importance of using each BBI component and to generate positive attitudes about the treatment. The BBI attitudes measure was only administered at post-Institute, but inspection shows high mean scores across cohorts. Providers becoming knowledgeable and having an early sense that they could deliver BBI clinically were Institute goals. The latter was done with multiple instruction modes (see above). The components selected for the multimodal training Institute [e.g., experientially-based training (Chard et al., 2012; DeNeve & Heppner, 1997; Karlin et al., 2010)], additional planning and goal setting (Schunk, 2003)], and selecting providers with supportive supervisors insofar that they wrote a letter of reference for the provider (Borrelli et al., 2008), may have contributed to the positive dissemination outcomes. Having multiple learning experiences may have increased self-efficacy to deliver the BBI when providers returned to their setting. This is consistent with other data showing that providers seeing the value in ESTs and having confidence in their skills are more likely to implement an EST (Turner et al., 2011). Additionally, providers’ profession was influential in that psychologists reported greater self-efficacy in using the EST compared to non-psychologists, perhaps because their training included other EST instructions. Attending to individual differences at the provider level has the potential to amplify the effectiveness of dissemination and implementation efforts.

The overall absence of cohort effects suggests training quality was maintained with some evidence for improvement. For instance, there were larger effect sizes for self-efficacy in cohorts 3–6, which may be due to the use of providers’ evaluations to improve Institute didactics, experientials, etc. Trainers had the impression that quality was, in fact, improving. The pre- and post-Institute self-efficacy scores of cohort groups were similar, though the differences in effect size may be due to the covariates used in rANOVA. Similarly, the null effects of EST attitude changes with cohorts 3–6 are not surprising, as the change was minimal (mean change=0.4) from the already high pre-Institute score. The change in EST attitudes found by Brothers et al. (2015) =was also small (mean change=1.7 on a possible range of 0–60) but found to be significant.

A prospective design is required to identify predictors of implementation, and usage must be high enough to predict it. One hundred percent usage was never anticipated as told to providers—the BBI is not appropriate for all patients nor all problems. Providers were from diverse settings and treated patients differing in cancer types, including those for which the BBI was not tailored (e.g., recurrent disease) for which modifications were needed (Thornton et al., 2014). But for most providers and patients, the BBI was the treatment of choice as the observed usage rates were 58–62% and conservatively, 41–45%. Data show that achieving “high” usage is difficult. An early Cochrane review of health care professionals randomized to one of 30 trials testing the efficacy of workshops found subsequent compliance to the practices taught was 6% (Forsetlund et al., 2009). Another example comes from the Center for Disease Control and Prevention initiative to train over 12,000 providers in HIV/AIDS risk-reduction interventions. Six months after training, the interventions were offered to an average of 4 patient groups per setting (Collins et al., 2010; Kelly et al., 2000). Other large projects have reflected on the reasons for failure in both the dissemination and implementation phases and made suggestions for improvements (Oxman et al., 1995; Wensing, 2017).

Changes in dissemination outcomes were tested as mechanisms of implementation. Knowledge was not one of them, despite large effect sizes for gains from dissemination. Objectively, treatment knowledge is important to the delivery of an EST with fidelity (Ashmore et al., 2019). But dissemination focused on knowledge gains alone—like most continuing education opportunities—is insufficient for implementation (e.g., Godin & Kok, 1996; Harned et al., 2013). Indeed, basic research also supports the finding that knowledge is unrelated to behavior outcomes (Ajzen et al., 2011).

By contrast, changes in providers’ positive attitudes toward and self-efficacy to use an EST were mechanisms of implementation. Prior data show changes in providers’ attitudes influence an indirect outcome, i.e., intentions to implement (Brothers et al., 2015). Extending this finding, these data show the importance of providers’ attitudes to implementation (Fishbein & Ajzen, 1975; Lo et al., 2019). Another mechanism was change in providers’ self-efficacy which predicted usage in the primary analyses and conservative post-hoc tests. In doing these tests, the research design used two data points (2 and 4 months) which enabled mechanism tests for early and later implementation. The analyses show some differences, but overall, there is support for the importance of instilling positive attitudes and confidence (self-efficacy) in providers to achieve implementation of a new EST to patients.

The context of the study is considered. An overarching goal of the NIH/NCI R25E program announcement was to facilitate the training of the oncology workforce. This particular grant was successful in the dissemination goal for mental health providers in oncology, but the study also determined if training resulted in providers’ using the EST and further and their usage of the EST could reduce patients’ distress (Ashmore et al., 2019). Full-time providers from across the country attended and were diverse in age, clinical experience, discipline, home institution, but the sample had little diversity in ethnicity and sex. Providers interested in learning BBI self-selected into the training and it is not known whether similar results would be found, for example, in a system-wide dissemination such as that done in the Veterans Administration Hospitals (Karlin et al., 2010). The findings provide multiple directions for future research, such as using the same instructional principles in training providers to use other ESTs or, for example, testing if training via online instruction could achieve similar gains.

Conclusion

Implementation science regards the translation of research to be on a continuum (Moullin et al., 2015; Tabak et al., 2012). Without studies across the continuum like that described here, it is unknown if changes during dissemination reap implementation benefits. The expansion of clinical guidelines for psychological care of patients with cancer has not been coupled with the expansion of clinical services or use of evidence-based psychosocial care (Travado & Rowland, 2020). Superficially, the BBI might be seen as an unlikely candidate for dissemination because it is a multicomponent EST. However, data disconfirm such a view and replicated the enduring effects of multimodal training of treatment conceptualization and complex clinical skills. Success in knowledge provision is a necessity, but it is an irrelevant one for implementation. In contrast, building providers’ self-efficacy and positive EST attitudes during dissemination are suggested as the mechanisms by which implementation occurs. Taken together, this research illustrates the importance of replicating dissemination efforts in determining the reliability of their effect and the need for the identification of mechanisms of the dissemination/implementation continuum.

Acknowledgments:

We thank the Stress and Immunity Cancer Projects and Cancer to Health staff and graduate students, Dale Kiss, and the Cancer to Health Institute attendees and their supervisors for their participation and expertise.

Funding Source:

National Institutes of Health/National Cancer Institute R25E CA163917, K05 CA098133.

Footnotes

Conflicts of Interest: Stephen B. Lo, Claire C. Conley, Brittany M. Brothers, Marlena M. Ryba, Georita F. Frierson, Rebecca A. Shelby, Lisa M. Thornton, Kristen M. Carpenter, and Barbara L. Andersen declare that they have no conflicts of interest.

References

  1. Aarons GA (2004). Mental health provider attitudes toward adoption of evidence-based practice: The evidence-based practice attitude scale (EBPAS). Mental Health Services Research, 6(2), 61–74. 10.1023/b:mhsr.0000024351.12294.65 [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Aarons GA, Sommerfeld DH, & Walrath-Greene CM (2009). Evidence-based practice implementation: The impact of public vs. Private sector organization type on organizational support, provider attitudes, and adoption of evidence-based practice. Implementation Science, 4. 10.1186/1748-5908-4-83 [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Ajzen I, Joyce N, Sheikh S, & Cote NG (2011). Knowledge and the prediction of behavior: The role of information accuracy in the theory of planned behavior. Basic and Applied Social Psychology, 33(2), 101–117. 10.1080/01973533.2011.568834 [DOI] [Google Scholar]
  4. Andersen BL, & Dorfman CS (2016). Evidence-based psychosocial treatment in the community: Considerations for dissemination and implementation. Psycho-Oncology, 25(5), 482–490. 10.1002/pon.3864 [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Andersen BL, Farrar WB, Golden-Kreutz DM, Glaser R, Emery CF, Crespin TR, Shapiro CL, & Carson WE 3rd. (2004). Psychological, behavioral, and immune changes after a psychological intervention: A clinical trial. Journal of Clinical Oncology, 22(17), 3570–3580. 10.1200/jco.2004.06.030 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Andersen BL, Golden-Kreutz DM, Emery CF, & Thiel DL (2009). Biobehavioral intervention for cancer stress: Conceptualization, components, and intervention strategies. Cognitive and Behavioral Practice, 16(3), 253–265. 10.1016/j.cbpra.2008.11.002 [DOI] [Google Scholar]
  7. Andersen BL, Thornton LM, Shapiro CL, Farrar WB, Mundy BL, Yang H-C, & Carson WE (2010). Biobehavioral, immune, and health benefits following recurrence for psychological intervention participants. Clinical Cancer Research, 16(12), 3270–3278. 10.1158/1078-0432.ccr-10-0278 [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Andersen BL, Yang H-C, Farrar WB, Golden-Kreutz DM, Emery CF, Thornton LM, Young DC, & Carson WE (2008). Psychologic intervention improves survival for breast cancer patients. Cancer, 113(12), 3450–3458. 10.1002/cncr.23969 [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Ashmore JA, Ditterich KW, Conley CC, Wright MR, Howland PS, Huggins KL, Cooreman J, Andrews PS, Nicholas DR, Roberts L, Hewitt L, Scales JN, Delap JK, Gray CA, Tyler LA, Collins C, Whiting CM, Brothers BM, Ryba MM, & Andersen BL (2019). Evaluating the effectiveness and implementation of evidence-based treatment: A multisite hybrid design. American Psychologist, 74(4), 459–473. 10.1037/amp0000309 [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Atkins MS, Frazier SL, Leathers SJ, Graczyk PA, Talbott E, Jakobsons L, Adil JA, Marinez-Lora A, Demirtas H, Gibbons RB, & Bell CC (2008). Teacher key opinion leaders and mental health consultation in low-income urban schools. Journal of Consulting and Clinical Psychology, 76(5), 905–908. 10.1037/a0013036 [DOI] [PubMed] [Google Scholar]
  11. Borrelli B, Lee C, & Novak S (2008). Is provider training effective? Changes in attitudes towards smoking cessation counseling and counseling behaviors of home health care nurses. Preventive Medicine, 46(4), 358–363. 10.1016/j.ypmed.2007.09.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Brothers BM, Carpenter KM, Shelby RA, Thornton LM, Frierson GM, Patterson KL, & Andersen BL (2015). Dissemination of an evidence-based treatment for cancer patients: Training is the necessary first step. Translational Behavioral Medicine, 5(1), 103–112. 10.1007/s13142-014-0273-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Brownson RC, Eyler AA, Harris JK, Moore JB, & Tabak RG (2018). Getting the word out: New approaches for disseminating public health science. Journal of Public Health Management and Practice, 24(2), 102–111. 10.1097/PHH.0000000000000673 [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Chard KM, Ricksecker EG, Healy ET, Karlin BE, & Resick PA (2012). Dissemination and experience with cognitive processing therapy. Journal of Rehabilitation Research and Development, 49(5), 667–678. 10.1682/jrrd.2011.10.0198 [DOI] [PubMed] [Google Scholar]
  15. Chinman M, Ebener P, Malone PS, Cannon J, D’Amico EJ, & Acosta J (2018). Testing implementation support for evidence-based programs in community settings: A replication cluster-randomized trial of getting to outcomes®. Implementation Science, 13(1), 131. 10.1186/s13012-018-0825-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Cohen J (2013). Statistical power analysis for the behavioral sciences. Academic Press. [Google Scholar]
  17. Collins CB, Hearn KD, Whittier DN, Freeman A, Stallworth JD, & Phields M (2010). Implementing packaged hiv-prevention interventions for hiv-positive individuals: Considerations for clinic-based and community-based interventions. Public Health Reports, 125(Supplement 1), 55–63. 10.1177/00333549101250S108 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Couineau A-L, & Forbes D (2011). Using predictive models of behavior change to promote evidence-based treatment for ptsd. Psychological Trauma: Theory, Research, Practice, and Policy, 3(3), 266–275. 10.1037/a0024980 [DOI] [Google Scholar]
  19. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, & Lowery JC (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4, 50. 10.1186/1748-5908-4-50 [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. David P, & Schiff M (2017). Self-efficacy as a mediator in bottom-up dissemination of a research-supported intervention for young, traumatized children and their families. Journal of Evidence-Informed Social Work, 14(2), 53–69. 10.1080/23761407.2017.1298072 [DOI] [PubMed] [Google Scholar]
  21. DeNeve KM, & Heppner MJ (1997). Role play simulations: The assessment of an active learning technique and comparisons with traditional lectures. Innovative Higher Education, 21(3), 231–246. 10.1007/bf01243718 [DOI] [Google Scholar]
  22. Fishbein M, & Ajzen I (1975). Belief, attitude, intention, and behavior: An introduction to theory and research. Addison-Wesley. [Google Scholar]
  23. Forsetlund L, Bjørndal A, Rashidian A, Jamtvedt G, O’Brien MA, Wolf F, Davis D, Odgaard-Jensen J, & Oxman AD (2009). Continuing education meetings and workshops: Effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews, 2009(2), Cd003030. 10.1002/14651858.CD003030.pub2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Godin G, & Kok G (1996). The theory of planned behavior: A review of its applications to health-related behaviors. American Journal of Health Promotion, 11(2), 87–98. 10.4278/0890-1171-11.2.87 [DOI] [PubMed] [Google Scholar]
  25. Haines A, & Jones R (1994). Implementing findings of research. BMJ, 308(6942), 1488–1492. 10.1136/bmj.308.6942.1488 [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Harned MS, Dimeff LA, Woodcock EA, & Contreras I (2013). Predicting adoption of exposure therapy in a randomized controlled dissemination trial. Journal of Anxiety Disorders, 27(8), 754–762. 10.1016/j.janxdis.2013.02.006 [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Hayes AF (2013). Introduction to mediation, moderation, and conditional process analysis: A regression-based approach. The Guilford Press. 10.1111/jedm.12050 [DOI] [Google Scholar]
  28. Ioannidis JPA, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, Schulz KF, & Tibshirani R (2014). Increasing value and reducing waste in research design, conduct, and analysis. The Lancet, 383(9912), 166–175. 10.1016/S0140-6736(13)62227-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Karlin BE, Ruzek JI, Chard KM, Eftekhari A, Monson CM, Hembree EA, Resick PA, & Foa EB (2010). Dissemination of evidence-based psychological treatments for posttraumatic stress disorder in the veterans health administration. Journal of Traumatic Stress, 23(6), 663–673. 10.1002/jts.20588 [DOI] [PubMed] [Google Scholar]
  30. Katzman JG, Galloway K, Olivas C, McCoy-Stafford K, Duhigg D, Comerci G, Kalishman S, Buckenmaier CC III, McGhee L, Joltes K, Bradford A, Shelley B, Hernandez J, & Arora S (2016). Expanding health care access through education: Dissemination and implementation of the echo model. Military Medicine, 181(3), 227–235. 10.7205/milmed-d-15-00044 [DOI] [PubMed] [Google Scholar]
  31. Kelly JA, Somlai AM, DiFranceisco WJ, Otto-Salaj LL, McAuliffe TL, Hackl KL, Heckman TG, Holtgrave DR, & Rompa D (2000). Bridging the gap between the science and service of hiv prevention: Transferring effective research-based hiv prevention interventions to community aids service providers. American Journal of Public Health, 90(7), 1082–1088. 10.2105/AJPH.90.7.1082 [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Lavoie KL, Rash JA, & Campbell TS (2017). Changing provider behavior in the context of chronic disease management: Focus on clinical inertia. Annual Review of Pharmacology and Toxicology, 57(1), 263–283. 10.1146/annurev-pharmtox-010716-104952 [DOI] [PubMed] [Google Scholar]
  33. Lent RW, Hill CE, & Hoffman MA (2003). Development and validation of the counselor activity self-efficacy scales. Journal of Counseling Psychology, 50(1), 97–108. 10.1037/0022-0167.50.1.97 [DOI] [Google Scholar]
  34. Lewis CC, Boyd MR, Walsh-Bailey C, Lyon AR, Beidas R, Mittman B, Aarons GA, Weiner BJ, & Chambers DA (2020). A systematic review of empirical studies examining mechanisms of implementation in health. Implementation Science, 15(1), 21. 10.1186/s13012-020-00983-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Lo SB, Ryba MM, Brothers BM, & Andersen BL (2019). Predicting implementation of an empirically supported treatment for cancer patients using the theory of planned behavior. Health Psychology, 38(12), 1075–1082. 10.1037/hea0000794 [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Mitchell SA, & Chambers DA (2017). Leveraging implementation science to improve cancer care delivery and patient outcomes. Journal of Oncology Practice, 13(8), 523–529. 10.1200/JOP.2017.024729 [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Moullin JC, Sabater-Hernández D, Fernandez-Llimos F, & Benrimoj SI (2015). A systematic review of implementation frameworks of innovations in healthcare and resulting generic implementation framework. Health Research Policy and Systems, 13, 16. 10.1186/s12961-015-0005-z [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Muthén B (1984). A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators. Psychometrika, 49(1), 115–132. [Google Scholar]
  39. Nilsen P (2015). Making sense of implementation theories, models and frameworks. Implementation Science, 10, 53. 10.1186/s13012-015-0242-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Oxman AD, Thomson MA, Davis DA, & Haynes RB (1995). No magic bullets: A systematic review of 102 trials of interventions to improve professional practice. CMAJ: Canadian Medical Association Journal, 153(10), 1423–1431. [PMC free article] [PubMed] [Google Scholar]
  41. Pinto D, Heleno B, Rodrigues DS, Papoila AL, Santos I, & Caetano PA (2018). Effectiveness of educational outreach visits compared with usual guideline dissemination to improve family physician prescribing—an 18-month open cluster-randomized trial. Implementation Science, 13(1), 120. 10.1186/s13012-018-0810-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Powell BJ, Fernandez ME, Williams NJ, Aarons GA, Beidas RS, Lewis CC, McHugh SM, & Weiner BJ (2019). Enhancing the impact of implementation strategies in healthcare: A research agenda. Front Public Health, 7, 3. 10.3389/fpubh.2019.00003 [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, Proctor EK, & Kirchner JE (2015). A refined compilation of implementation strategies: Results from the expert recommendations for implementing change (ERIC) project. Implementation Science, 10(1), 21. 10.1186/s13012-015-0209-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Rabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, & Weaver NL (2008). A glossary for dissemination and implementation research in health. Journal of Public Health Management and Practice, 14(2), 117–123. 10.1097/01.PHH.0000311888.06252.bb [DOI] [PubMed] [Google Scholar]
  45. Rankin NM, Butow PN, Hack TF, Shaw JM, Shepherd HL, Ugalde A, & Sales AE (2019). An implementation science primer for psycho-oncology: Translating robust evidence into practice. Journal of Psychosocial Oncology Research and Practice, 1(3), e14. 10.1097/or9.0000000000000014 [DOI] [Google Scholar]
  46. Rapport F, Clay-Williams R, Churruca K, Shih P, Hogden A, & Braithwaite J (2018). The struggle of translating science into action: Foundational concepts of implementation science. Journal of Evaluation in Clinical Practice, 24(1), 117–126. 10.1111/jep.12741 [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Ryba MM, Lo SB, & Andersen BL (2019). Sustainability of a biobehavioral intervention implemented by therapists and sustainment in community settings. Translational Behavioral Medicine. 10.1093/tbm/ibz175 [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Schunk DH (2003). Self-efficacy for reading and writing: Influence of modeling, goal setting, and self-evaluation. Reading & Writing Quarterly, 19(2), 159–172. 10.1080/10573560308219 [DOI] [Google Scholar]
  49. Sterling-Turner HE (2002). The effects of direct training and treatment integrity on treatment outcomes in school consultation. School Psychology Quarterly, 17(2), 47–77. 10.1037/h0089179 [DOI] [Google Scholar]
  50. Tabak RG, Khoong EC, Chambers DA, & Brownson RC (2012). Bridging research and practice: Models for dissemination and implementation research. American Journal of Preventive Medicine, 43(3), 337–350. 10.1016/j.amepre.2012.05.024 [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Thornton LM, Cheavens JS, Heitzmann CA, Dorfman CS, Wu SM, & Andersen BL (2014). Test of mindfulness and hope components in a psychological intervention for women with cancer recurrence. Journal of Consulting and Clinical Psychology, 82(6), 1087–1100. 10.1037/a0036959 [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Travado L, & Rowland JH (2020). Supportive care and psycho-oncology issues during and beyond diagnosis and treatment. In Gentilini O, Partridge AH, & Pagani O (Eds.), Breast cancer in young women (pp. 197–214). Springer International Publishing. 10.1007/978-3-030-24762-1_17 [DOI] [Google Scholar]
  53. Turner KMT, Nicholson JM, & Sanders MR (2011). The role of practitioner self-efficacy, training, program and workplace factors on the implementation of an evidence-based parenting intervention in primary care. The Journal of Primary Prevention, 32(2), 95–112. 10.1007/s10935-011-0240-1 [DOI] [PubMed] [Google Scholar]
  54. Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, Chinman MJ, Smith JL, Proctor EK, & Kirchner JE (2015). Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: Results from the expert recommendations for implementing change (eric) study. Implementation Science, 10(1), 109. 10.1186/s13012-015-0295-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  55. Wensing M (2017). The tailored implementation in chronic diseases (ticd) project: Introduction and main findings. Implementation Science, 12(1), 5. 10.1186/s13012-016-0536-x [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Williams KC, Brothers BM, Ryba MM, & Andersen BL (2015). Implementing evidence-based psychological treatments for cancer patients. Psycho-Oncology, 24(12), 1618–1625. 10.1002/pon.3937 [DOI] [PMC free article] [PubMed] [Google Scholar]
  57. Williams NJ, Glisson C, Hemmelgarn A, & Green P (2017). Mechanisms of change in the arc organizational strategy: Increasing mental health clinicians’ ebp adoption through improved organizational culture and capacity. Administration and Policy in Mental Health, 44(2), 269–283. 10.1007/s10488-016-0742-5 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

Of the 166 providers, 119 (72%) and 114 (69%) provided usage data at 2 and 4 months, respectively. Of the non-reporting providers, 8 (17%) and 13 (25%) at 2 and 4 months, respectively, reported not having an opportunity to use the BBI (i.e., left the institution; needed more time to implement; medical leave). No differences in provider/setting characteristics or post-Institute knowledge, attitudes, and self-efficacy were found between providers with and without 2-month (ps>.056) or 4-month usage data (ps>.156).

RESOURCES