Sustainability of a biobehavioral intervention implemented by therapists and sustainment in community settings

Abstract

The ultimate aim of dissemination and implementation of empirically supported treatments (ESTs) in behavioral medicine is (a) sustainability of the therapist/provider’s EST usage and (b) sustainment of EST delivery in the setting. Thus far, sustainability has been understudied, and the therapist and setting variables that may be influential are unclear. The purpose of the study was to test the therapists’ sustainability of a cancer-specific EST using a prospective longitudinal design and examine its predictors. Oncology mental health therapists (N = 134) from diverse settings (N = 110) completed training in the biobehavioral intervention (BBI) and were provided with 6 months of support for implementation, with no support thereafter. BBI usage (percent of patients treated) was reported at 2, 4, 6, and 12 months. Using a generalized estimating equation with a logistic link function, 12-month sustainability (a nonsignificant change in usage from 6 to 12 months) was studied along with therapist, supervisor, and setting variables as predictors. BBI usage increased through 6 months and, importantly, usage was sustained from 6 (68.4% [95% CI = 62.2%–73.9%]) to 12 months (70.9% [95% CI = 63.6%–77.3%]), with sustainment in 66 settings (60.0%). Predictors of implementation-to-sustainability usage were therapists’ early intentions to use the BBI (p < .001) and from the setting, supervisors’ positive attitudes toward ESTs (p = .016). Adding to the DI literature, a health psychology intervention was disseminated, implemented, and found sustainable across diverse therapists and settings. Therapists and setting predictors of usage, if modified, might facilitate future sustainability/sustainment of ESTs.

Implications.

Practice: Improving providers’ intentions to use empirically supported psychological treatments (ESTs) and supervisors’ positive attitudes towards the EST could achieve both sustainability and sustainment.

Policy: Leaders wanting to achieve sustainability of an EST must consider providers’ and supervisors’ perception of the EST in addition to providing education in the EST.

Research: Sustainability and sustainment are key outcomes for dissemination to implementation efforts.

INTRODUCTION

The dissemination of health psychology interventions to therapists in the community remains limited, with successful implementations fewer still. The endpoints of dissemination and implementation (DI) are sustainability—continued delivery of the empirically supported treatment (EST) to the intended patients/individuals by the therapist—and sustainment, the continued provision of the EST in the setting. The setting is most relevant to implementation for managed care organizations, policy analysts, and related stakeholders [1,2]. For others, the therapist is key, with the view that therapists must sustain EST usage and do so independently [3].

The empirical literature supports both setting and therapist variables as influential, although the predominance of papers examines setting/organizational factors (e.g., [4]). For example, a review of 60 studies by Stirman et al. [1] found 82% examining sustainment with only 18% (11 of 60) having sustainability data, with similar findings from others [2,5]. A notable study is that of Swain, Whitley, McHugo, and Drake [6] which began with dissemination. Staff in 49 community mental health practices were trained to deliver five ESTs. Afterward, the investigators provided 2 years of support to therapists to achieve implementation [7]. Six years later, 47% of the sites showed sustainment, 16% restarted EST delivery after a period of discontinuation, and 37% discontinued the ESTs permanently. Surveying the leaders/managers for clues to achieving sustainment [8], managers at successful sites reported having adequate funding for its delivery, providing supervision for therapists, and having a system to monitor EST fidelity, whereas nonsustainment settings reported the converse, i.e., inadequate financial resources, insufficient numbers of trained therapists/supervisors, and a lack of prioritization for EST delivery. These data and others from system-wide implementations (e.g., [9,10]) illustrate the difficulty and the multiple circumstances influencing sustainment.

Sustainability data come from studies focused on the processes of dissemination or implementation (rather than the combination) and are infrequently framed to achieve sustainability. Relevant to dissemination is literature on qualities of EST training or continuing education. They suggest the intensity or duration of dissemination training as implementation predictors [11]. Regarding implementation, providing support to therapists has been long regarded as essential [12]. What constitutes “support” varies from study to study (e.g., [13,14]), but primary elements are assisting or supervising therapists in EST usage and/or helping to problem solve implementation barriers, with support durations ranging from days to a year or more [6,15]. Support is viewed as key to implementation, but its relationship to sustainability is little studied. But relevant is the finding that therapists are at greatest risk of abandoning EST usage when support ends [8,16].

In this context, a multi-level research effort was conducted to study dissemination, implementation, and ultimately, the sustainability of therapists’ use of a behavioral medicine EST. A conceptual framework was used [17] with sustainability as the endpoint and its antecedents being quality dissemination and support for implementation. Within each step were therapist and setting variables hypothesized to covary with its success and the prediction of the next step. Reported here are data from the last step, sustainability. Data come from oncology mental health therapists’ usage of an EST, i.e., the biobehavioral intervention (BBI), designed to reduce stress, enhance health behaviors and treatment compliance, and improve the health of the cancer patients [18,19].

Previously published results from the first two steps are described. First, the multimodal dissemination education yielded significant gains in therapists’ knowledge and clinical facility to use the BBI. At the end of the training, therapists left with positive attitudes, high self-efficacy, and high intentions to use the BBI [20]. Next, implementation support was provided for six months to achieve early BBI usage and its maintenance thereafter. BBI usage was high absolutely and in comparison to rates found in the literature, as therapists’ provided BBI to 58%–68% of their patients across the support months [21].

Thus, data show that a cancer-specific EST could be successfully disseminated and implemented, but could it last? For this final step, sustainability was operationalized as a non-significant change in patients receiving the BBI from 6 months—the end of implementation support—to 12 months. Failure to achieve sustainability was operationalized as a significant decrease in BBI-treated patients from month 6 to month 12. Secondarily, the number of settings showing sustainment at each timepoint is reported. As has been suggested [1], predictors of usage were studied to determine mechanisms/moderators of effects and generate hypotheses for future research. Contextual variables of the therapists, their managers/supervisors, and the treatment settings were selected from the framework guiding the research, [17] and more generally, ones relevant to the prediction of behavior [45], dissemination [22,23], and implementation outcomes [13].

METHODS

Participants

Therapists

Oncology mental health therapists (N = 134) attending five BBI Institutes (description below) were studied. The sample was predominantly female (89%), middle-age (M = 43.2 years; range 25–67), and Caucasian (81%). All were paid, full time, clinical service providers with limited time (27%) in other activities (e.g., administration, teaching). Virtually all (90%) were licensed and had been so for an average of 13 years (range 0–36), as clinical psychologists (42%), social workers (42%), or other mental health providers (16%).

Managers/supervisors and settings

Managers/supervisors of the therapists provided supporting data. Of the 134 therapists, 11 had the same supervisor, resulting in 123 individuals. Of them, 104 consented (85%), but 7 did not complete data entry, for a final N = 97 (79%). Supervisors were in the disciplines of mental health (60%), medicine (25%), or business/administration (15%).

Therapists/supervisors were employed in 110 unique sites, from 33 U.S. states, Washington, DC, Puerto Rico, and two foreign countries (Kenya and Malaysia). The healthcare settings were exceptionally diverse, ranging from community facilities (55%; e.g., single provider private practices, community hospitals, private practices) to university affiliated comprehensive cancer centers and Veterans’ Administration Hospitals (VAHs; 45%). More specifically, therapists came from academic medical centers (n = 49; 37%), community hospitals (n = 34; 25%), cancer support communities (n = 12; 9%), VAHs (n = 11; 8%), private practices (n = 8; 6%), community oncology practices (n = 7; 5%), or and other settings (n = 13; 10%). Therapists’ described the “average” patient served 57% female (range 5%–100%), with an ethnicity/race of 69% Caucasian (range 0%–100%), 18% African-American (range 0%–80%), 14% Latino (range 0%–100%), 5% Asian (range 0%–100%), and 3% other race (range 0%–42%). Many therapists (36%) treated patients of all cancer types, while others specialized (e.g., breast, lymphoma, prostate). Therapists reported 36% of their patients resided in rural areas (range 0%–100%) and 37% with income $35,000 or less for a family of four (range 0%–100%).

Procedures

Approved by the Institutional Review Board (Ohio State University), informed consent was obtained from participants in accordance with the Declaration of Helsinki. Procedures for accrual and dissemination education have been detailed [20]. Briefly, announcements were posted on listservs and the Institute website, with five conducted from 2012 to 2015. A recommendation from one’s direct manager/supervisor was required for application. Therapists attended 3-day BBI Institutes taught by six expert trainers using lectures (40%), role play and group discussions (35%), and practice experiential (25%), providing “hands-on” familiarity with the BBI therapist manual and patient guidebook [see 18 for a description]. Therapists completed self-report measures, the majority has given pre- and immediately post-institute. Permission was sought from therapists to contact his/her supervisor for data collection. Post-institute, the supervisor was informed of the therapist’s completion of training and was invited to participate in the research. Supervisors received $5 gift cards for participation.

As previously described [21], implementation support consistent with recommendations [12] was provided. Components were the following: (a) adaptation planning, i.e., individual guidance to determine the fit of the BBI for the patients/setting and discuss adaptations as needed; (b) quality monitoring, i.e., six monthly group conference calls with trainers to review BBI principles and problem solve implementation challenges; (c) provision of BBI information/marketing materials for administrators, physicians, nurses, patients, and others; and (d) financial, i.e., business plan templates to secure setting resources for implementation.

To obtain BBI usage data, therapists used a web-based Qualtrics log at 2, 4, and 6 months during support and then again at 12 months for sustainability. Therapists reported usage in the last month, along with the BBI components (10) used and homework assigned (data not reported). Therapists also completed measures of self-efficacy, attitudes, and intent to use the BBI. Therapists received a $15 gift card for each assessment.

Measures

Sustainability

Using the Qualtrics log, therapists reported service provision (treatment) to individuals and groups within the previous month. As full-time providers, it was anticipated that therapists could readily report the total number of patients treated/billed. In another portion of the log, therapists reported the total number treated with the BBI. Usage was calculated as a percentage: (1) the total number of adult cancer patients who received the BBI (“How many adult cancer patients have you treated using any of the BBI techniques, sessions, or modules?”), in relation to (2) the total number of all adult cancer patients treated (“How many adult cancer patients have you seen for treatment?”). Spearman correlations across time for therapists’ usage reports are as follows: .557 (2 and 4 months), .619 (4 and 6 months), and .648 (6 and 12 months).

Predictors of sustainability

Therapist

Six areas were studied. (1) Professional characteristics. Variables were age, years licensed, years in current position, and profession (psychology, social work, or other). (2) Clinical service provision. Percent of full-time equivalent effort in clinical service versus other tasks (e.g., teaching). (3) Knowledge. A BBI content knowledge measure [20] consisted of 32 multiple choice and true/false items. Percent correct (0–100%) was calculated. Consistent with its heterogeneous content, internal consistency was α=0.504. (4) Self-efficacy. Three measures were completed. (a) The counselor activity self-efficacy scale [24] uses 25 items to assess self-efficacy to perform basic (e.g., helping skills) and advanced (e.g., handling challenging situations) counseling skills. Items were rated on a 10-point Likert scale (0 = not at all confident to 9 = totally confident) and summed (range 0–225) with higher scores indicating greater general counseling self-efficacy. Internal consistency was α = 0.963. (b) The BBI Self-Efficacy Scale is an eight-item measure of BBI-specific therapeutic skills (e.g., progressive muscle relaxation, problem-solving) including one item assessing confidence in performing these skills in a group treatment format [20]. A 10-point Likert scale (0 = not at all confident to 9 = totally confident) used, items were summed (range 0–72), with higher scores indicating greater self-efficacy with BBI-specific skills. Internal consistency was α = 0.840. (c) A six-item measure assessed confidence in learning, practicing, and applying the BBI. Items were rated on a five-point Likert scale (1 = not confident to 5 = totally confident), items were summed (range 6–30), with higher scores indicating greater confidence in conducting the BBI. Internal consistency was α=0.870. (5) Attitudes. Two measures were used. (a) Attitudes toward evidence-based practices were assessed with the 15-item Evidence-based Practice Attitude Scale[25]. Items (e.g., “I am willing to try new types of therapy/interventions even if I have to follow a treatment manual”) were rated on a five-point Likert scale (0 = not at all to 4 = to a very great extent) and summed (range 0–60), with a higher score indicating more positive attitudes. Internal consistency was α = 0.831. (b) A 30-item semantic differential scale [26] assessed attitudes towards the BBI. Ten items each for the dimensions of evaluation, potency, and activity were rated on a seven-point scale ranging from −3 (negative attitude) to +3 (positive attitude) and summed (range −90 to +90). Internal consistency was α = 0.858. (6) Intent. Intent to use each BBI component (e.g., progressive muscle relaxation, assertive communication) was assessed with 10 items. Items were rated on a five-point Likert scale (0 = never to 4 = always) and summed (ranged 0–40), with higher scores indicating greater intentions to use the BBI as manualized. Internal consistency was α = 0.871.

Manager/supervisor

Two areas were studied. (1) Professional discipline. Coded variables were mental health, medical, or business administration. (2) Attitudes. The Evidence-based Practice Attitude Scale [25], described above, was used. One item not applying to supervisors was omitted (“How likely would you be willing to adopt it at your institution if it was required by your supervisor?”). Using 14 items, internal consistency was α = 0.702.

Setting

Three measures were used. (1) Type. Healthcare settings were coded as community (e.g., nonaffiliated, community hospitals, supportive care facilities, private practice) versus noncommunity (i.e., academic medical centers, Veteran’s Administration Hospitals). (2) Resource availability. Therapists were asked two brief questions, coded as yes (1) or no (0): “Does the setting have rooms available for clinical use?” and “Does the setting have the capability to promote/advertise BBI program availability?” (3) Implementation challenges. Supervisors rated the perceived ratio of challenges/benefits to BBI implementation in the setting on a visual analogue scale with anchors of “challenges will always outweigh the benefits” (scored 0) to “benefits will always outweigh the challenges” (scored 0–100).

Analytic plan

Descriptive statistics for BBI usage and potential predictors were considered. Therapist sample size at each time point is reported (see Fig. 1); analyses included all usage data at all time points (2, 4, 6, and 12 months). Usage reports specifying that no adult cancer patients were treated in the past month (i.e., reporting “0” total cancer patients treated) were considered missing (i.e., no data). In this case, no usage reflects an absence of opportunity to use the BBI or having the opportunity to use BBI but not choosing/able to do so. For the interest of the reader, settings (n = 110) having used more than 0% were tallied for each timepoint.

Fig 1.

Model estimate of BBI usage from end of implementation support (6 months) to sustainability (12 months). Generalized estimating equation (GEE) model estimates of therapists’ provision of the BBI to patients at assessments from 2 to 12 months are displayed with 95% confidence intervals. The white background represents months 1–6 of implementation support provision and gray shading represents sustainability, months 7–12. *p < .05; NS: p > .05

To model BBI usage, Institute attended (five cohorts) was controlled. A generalized estimating equation (GEE) approach [27] was used, with a logistic link function [28] to analyze skewed data bounded by response scales (e.g., percentages that range from 0% to 100%). The GEE approach is a nonparametric extension of a generalized linear model and adjusts the variance using a working matrix of correlation coefficients to account for correlation between observations. A binomial distribution assuming an unstructured correlation matrix was used. Model coefficients were transformed [28] and interpreted as the percentage change in BBI usage. Sustainability was determined by fitting time to the model detailed above to predict usage over time.

Factors, which may have influenced usage–therapist, supervisor, and setting variables, were examined. Controlling for time and cohort, a variable was fit into the model’s level of analysis (therapist, supervisor, setting). Variables within a level found to be significant by univariate analysis were added to a subsequent multivariate model for the level. For each level analysis, nonsignificant variables were removed until the final model retained only significant variables. All analyses were performed using R [29].

RESULTS

Sustainability and sustainment

Observed BBI usage with patients ranged from 0% to 100%, with overall usage increasing across time and then stabilizing. Descriptively, the average percentage of therapists’ patients receiving BBI was 60.2% ± 34.3% at 2 months, 67.2% ± 33.9% at 4 months, 69.1% ± 34.0% at 6 months, and 72.7% ± 30.7% at 12 months.

GEE model estimates are depicted in Fig. 1. At 12 months, therapists reported an average of 70.9% (95% CI = 63.6%–77.3%) of patients treated with the BBI. This level of usage was comparable to that at 6 months (68.4%; 95% CI = 62.2%–73.9%), showing BBI sustainability (z = .792, p = .428). To test the reliability of these results, a conservative method was used, imputing all missing usages as 0% usage (see therapist ns in Fig. 1). These analyses also showed no significant difference between 6- and 12-month usage (z = .421, p = .674).

Regarding sustainment, the number/percentage of sites (n = 110) reporting usage more than 0% was 79 (71.8%) at 2 months, 76 (69.1%) at 4 months, 75 (68.2%) at 6 months, and 66 (60.0%) at 12 months. Across time, only two to four sites (1.8%–3.6%) reported usage = 0. Across time, there were missing data from 28 to 42 sites (25.5%–38.2%).

Analyses with predictors

See Table 1 for post-institute descriptive statistics of contextual therapist, supervisor, and setting measures.

Table 1.

Means and standard deviations (SD) or percentage, at post-institute, of contextual therapist, supervisor/manager, and setting measures used in the generalized estimating equation (GEE) analyses

Level	Mean ± SD	Observed range	Possible range
Variable
Measure
Therapist
Attitudes
EST attitudes	48.7 ± 6.2	29–60	0–60
BBI attitudes	59.8 ± 14.3	15.5–85	−90 - +90
Intent to use BBI	31.4 ± 4.9	19–40	0–40
Self-Efficacy
General self-efficacy	198.4 ± 18.4	125–225	0–225
BBI self-efficacy	63.5 ± 6.1	40–72	0–72
Confidence in conducting BBI	26.2 ± 3.2	18–30	6–30
Knowledge	83.7% ± 8.7	56.3–100	0–100%
Manager/supervisor
Attitudes
EST attitudes	42.5 ± 5.6	30–54	0–56
Setting
Community (vs. non)	55% (n=74)
Resource availability
Rooms available (% yes)	69% (n=92)
Able to promote BBI (% yes)	58% (n=77)
Challenges/benefits ratio	69.5 ± 19.3	2–100	0–100
Sustainment across assessments (# of sites)	74.0 ± 5.6	66–79	0–110

EST empirically supported treatment; BBI biobehavioral intervention.

Therapist

In the final model, therapists’ greater intent to use the BBI (p < .001) was positively associated with usage, controlling for cohort and time. In the first step of model selection (univariate analysis), self-efficacy, both general and BBI-specific, confidence in conducting BBI, and attitudes toward the BBI were significant but not retained in the final model.

Supervisor

In the final model, supervisors’ positive attitudes toward the adoption of evidence-based practice (p = .016) were associated with greater usage, while controlling for cohort and time. Other variables were not significant in univariate analyses.

Setting

No variables were significant in univariate analysis.

DISCUSSION

Beginning with the dissemination and continuing through early implementation, these data demonstrate sustainability of an efficacious health psychology intervention used by oncology mental health therapists treating cancer patients. Sustainability was operationalized as a non-significant change in usage of a biobehavioral intervention (BBI) from 6 months (68% usage)—the end of implementation support—to 12 months (71% usage). Exploration of therapist, manager/supervisor, and setting variables as predictors of usage identified two: therapists’ high intentions to implement the BBI and supervisors’ positive attitudes toward ESTs. A significant contribution to the current to the dearth of sustainability data, findings are considered in the context of the steps leading to sustainability and the DI literature.

From the perspective of this group, implementation hinges on the quality of the first step, dissemination [20]. Data reliably show that brief training or ones primarily didactic are insufficient [30]. Knowledge gains, a bare minimum, are short-lived and insufficient for skill development. Prior data show therapists gained knowledge and clinical facility in using BBI strategies, even though all came to dissemination with no familiarity with the BBI. Even BBI components common to cognitive behavior therapy (e.g. assertive communication, problem-solving) were ones that at least some had neither sufficient clinical skill nor confidence to use. Therapists with low confidence, insufficient skills, and/or negativity or disinterest in ESTs—all are conditions for failure to implement or implementation that deteriorates in quality [31]. Strong gains/high scores on these dimensions and others provided therapists with solid footing to take the step to implementation.

Implementation support was designed to counter the reasons for implementation failure: (1) poor fit of the EST [32]; (2) therapist and treatment delivery factors [31,33]; and (3) barriers in the setting [34,35]. As the therapists, settings, and patients [20] were diverse, the need for BBI adaptation was anticipated. To do so, therapists were guided to retain the core components of the BBI. Other data show the primary adaptations were reduction in numbers of sessions and/or deletion of non-core components [36]. Assisting therapists may have helped maintain confidence and self-efficacy to use the BBI and their positive view of it. This may have been important for therapists’ achievement of high early rates of usage 2 months after dissemination. Dissemination content and support materials appeared to help therapists anticipate and manage setting barriers [37], as even a null environment, one without barriers but also no support, would hinder implementation. Although unknown, it is plausible that helping therapists to address early barriers aided in their securing of support and resources to deliver the BBI through the year.

Although the literature is limited, it shows sustainability to be difficult [16], but not impossible (e.g., [38,39]) to achieve. The analyses estimated therapists using the BBI with 58%–68% of their patients during the 6 months of implementation support provided an adequate baseline for a sustainability test. Analyses showed no slippage of usage at 12 months. The conditions for achieving implementation and sustainability have been unclear, with views that they are similar [40], different but interacting with one another [41], or are from a dynamic process of adaptation [42]. DI theories and models provide a plethora of variables as potential drivers of sustainability/sustainment.

Here, three domains were studied. Studies focused on therapist individual differences predicting implementation find their confidence to use the EST [43], their positive attitudes [13] and high intentions [44] to be important. Of personal and professional characteristics, and therapists’ judgments of competence, self-efficacy, attitudes, and intentions, only intentions to deliver the BBI remained in the final model. The Theory of Planned Behavior posits intentions as proximate to behavior [45], with many demonstrations, including those with intentions predicting clinicians’ provision of healthcare to patients [44]. Data from a subset of the present sample (Institutes 1–2, N = 62) suggest hypotheses as to why intent was so important. At the end of dissemination training, therapists’ attitudes toward the BBI and their self-efficacy to deliver it accounted for 7% and 11% of the variance, respectively, in predicting their intentions to deliver the BBI [20]. In absolute terms, the level of intentions at that time was high (mean of 31 of 40 possible, with 19 being the lowest observed score; see Table 1), suggesting a collective momentum to use the BBI. In the short term, high intentions may have been instrumental in achieving the 2-month usage of 58% and, in the long term, sustainability.

In the DI literature, studies of the organization/setting in relation to implementation, have included ones examining organizational culture [33], climate [46], and personal characteristics such as leadership [34,35], among others. When negative, these elements prevent EST adoption [34]. Therapists anticipated barriers to implementation arising from the setting or persons within it [37]. For the former, there might be insufficient funds or resources to provide a new EST. For the latter, physicians, managers/supervisors, or others might lack knowledge about psychological ESTs, for example. Of the characteristics studied, supervisors’ positive attitudes toward ESTs predicted higher levels of implementation/sustainability. Such individuals may have been mid-level “champions” [47], reducing barriers and providing stable support for implementation.

To consider these findings, evaluation of the research design and methodology is provided. This was a longitudinal design with multiple assessments after the “treatment” of dissemination and implementation support. To achieve an effect, both were conducted with the same expert trainers using standardized procedures and manualized content done reliably across cohorts [20,21]. As these therapists had not used the BBI previously, the jump to BBI implementation in the range of 60%–80% of patients treated with sustainment in 60% in diverse settings are compelling, and unlikely to be due to the passage of time but instead dissemination and support effectiveness. BBI delivery by individual therapists to individual patients is a face valid metric. The observed usage level is high but not unreasonably so, as the multi-component BBI is relevant to the majority patients, though not all. The possibility that usage was over-reported is made less plausible by the observed range (0%–100%) of usage. This suggests no reluctance to report no/low usage and also, therapists readily described their implementation difficulties and low usage on support telephone calls [21]. That there was no bias (overreporting) cannot be ruled out, but it done, it might have been done systematically as the reliability of the usage reports ranged from .56-.65. A potentially greater threat to internal validity is therapist/site attrition. However, confidence in the usage findings comes from the additional analyses with missing data recoded as no (0) usage.

Regarding the participants, eligibility criteria were minimal: full-time employment in licensed, service provision in oncology mental health. However, the self-selection of participants to attend an intensive EST training weakens internal validity. Additionally, the therapists had generally positive EST attitudes from the beginning, although their attitudes improved by training’s end [20]. The novelty of the training opportunity and the BBI may have increased therapists’ enthusiasm/motivation to use it initially, but it is unknown if it contributed to sustainability. Regarding the context of the study, the Institutes and BBI implementations occurred during the years of a U.S. economic recession. Hiring freezes/cuts in psychosocial services, hospital re-organizations, and shrinking resources did account for some failures to implement. To summarize, internal validity of the “treatments” and the timing of their effects is strong. Although attrition occurred, it did not appear to affect the reliability of the results. Also, the findings may have been positively impacted by the prior history of the participants but negatively by the calendar period during which the study was conducted.

The heterogeneity of the therapists (age, discipline, experience, type of patients treated; though not race. and gender), supervisors (professional discipline), settings (e.g., financial base, models of care delivery, number of cancer patients served), and geographical diversity provide significant generalizability for the findings. A single EST was studied, as is the case for the majority (e.g., 73%) of implementation studies [2]. This is a limiting factor, but when considered broadly, the BBI is thus far the only cancer control intervention [48] disseminated, implemented, and now found sustainable.

In conclusion, an empirical look at EST sustainability is provided, following therapists from their dissemination training, through implementation support, to months without support. The data show that the BBI is a cancer control treatment sustainable across therapists, patients, and settings. Therapists high intentions to deliver the BBI and supervisors’ positive attitudes toward evidence-based practice were key predictors. In the DI literature, this is a unique demonstration of EST sustainability, with exemplars and implications for the transfer of other ESTs and public health programs to community settings where they are most needed.

Compliance with Ethical Standards

Conflict of Interest: None declared

Authors’ Contributions: MMR: Conceptualization, investigation, writing - review and editing, supervision, project administration. SBLo: Conceptualization, validation, formal analysis, writing- original draft, visualization. BLA: Conceptualization, methodology, investigation, resources, writing - review and editing, supervision, project administration, and funding acquisition.

Ethical Approval: All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. Informed consent was obtained from all individual participants included in the study. This article does not contain any studies with animals performed by any of the authors.