Multilevel Barriers and Facilitators to Sustainability of a Universal Trauma-Informed School-Based Mental Health Intervention Following an Efficacy Trial: A Qualitative Study

Abstract

Limited research exists about factors that influence the sustainability of preventive school mental health interventions when research support ends. This study assessed barriers and facilitators to sustaining RAP (Relax, be Aware, do a Personal rating) Club, a trauma-informed universal mental health intervention, in urban schools following efficacy trial implementation. Between 2016-2018, 13 Baltimore City schools implemented RAP Club. We evaluated RAP Club sustainability using semi-structured interviews with administrators (n=10) and school staff trained to deliver the intervention (n=11), as well as review of intervention fidelity logs (n=137) and notes from supervision calls (n=10) with school staff and research team members who implemented the program. Although most school staff described RAP Club as acceptable and beneficial, none of the 13 schools sustained the intervention. Barriers to sustainability included low self-efficacy among school staff trained to deliver the intervention, school staff turnover, logistical challenges with space and time in the regular school calendar to deliver the program, insufficient funding to sustain the program outside of the research context, and limited planning and ongoing communication between school personnel and researchers about sustainability. Recommendations for increasing post-trial program sustainability include developing sustainability plans with schools during the pre-implementation phase, enhancing ongoing support for school staff during the implementation phase, and using academic-community partnerships to facilitate sustainability and intervention scale-up during the post-implementation phase. Increasing sustainability of beneficial school-based mental health programs has the potential to reduce mental health disparities and promote health equity.

Childhood trauma is a pervasive public health problem associated with adverse academic, emotional, behavioral, social, and health outcomes across the lifespan (Substance Abuse and Mental Health Services Administration [SAMHSA] 2014). Over two-thirds of children in the United States (U.S.) have experienced at least one situation that is physically or emotionally harmful (e.g., abuse, emotional neglect, racism, community violence) by age 16 (SAMHSA 2015) with higher rates of trauma exposure and less access to mental health services among youth of color and youth living in poverty or economically disadvantaged areas (Larson et al. 2017; Slopen et al. 2016). Schools are a promising setting for delivering interventions to reduce or prevent the detrimental effects of trauma because most children and adolescents spend the majority of their time in schools (Fazel et al. 2014; Murphy et al. 2017). However, few trauma-informed prevention interventions—even those shown in research studies to have benefits for youth (Dorsey et al. 2017; Gillies et al. 2013; Powell et al. 2020)—have been successfully implemented or sustained in schools (Eiraldi et al. 2015; Forman et al. 2009; Owens et al. 2014).

Sustainability is the extent to which a newly implemented program, intervention or treatment is maintained or institutionalized within an organization’s ongoing, routine operations (Proctor et al. 2011). Significant resources have been invested in research to develop and test health interventions, but there has been less investment in research related to sustaining effective interventions. This limits the public health reach and potential impact of effective interventions and contributes to the approximately 17-year gap between research and practice (Morris et al. 2011). Additionally, not sustaining beneficial interventions in underserved educational and community settings beyond the context of research studies could exacerbate health disparities by removing access to high quality health programs.

Sustainability is a persistent challenge across numerous settings and service delivery sectors in which health interventions are delivered (including education, health care, and social services) and for a range of health behaviors and outcomes (Shelton et al. 2018). Common factors theorized to promote program sustainability across settings include favorable intervention characteristics (e.g., low cost, low complexity); positive individual characteristics of implementers (e.g., favorable attitudes, strong self-efficacy); presence of intervention champions and organizational leaders who are committed to sustainability; supportive organizational climate and culture (e.g., collegial relationships, reliable procedures; Hoy et al. 1991; Parcel et al. 2003); use of monitoring and evaluation; low staff turnover; and availability of external policies and financing (Fleiszer et al. 2015; Herlitz et al. 2020; Stirman et al. 2012).

There are few studies that examine health program sustainability in education settings. However, previous research in this area highlights unique factors related to sustaining newly implemented health interventions in schools (Herlitz et al. 2020). First, sustaining health programs may not be a top priority of schools, particularly if they require resources that may divert from the educational mission of the school (e.g., health curriculum, intervention manuals and materials). Second, health interventions delivered in schools may require professionally trained health providers (e.g., nurses, school psychologists) and/or school staff who do not specialize in health, such as teachers and guidance counselors. Acceptability of health interventions may vary among school staff, which typically translates into variation in their commitment to, comfort with, and confidence in delivering health interventions. Third, limited interaction and collaboration between the education and health sectors contribute to challenges with funding, resources, and training—all of which are essential to sustainability (Herlitz et al. 2020).

To date, few studies on the sustainability of school-based health interventions have focused on mental health (Herlitz et al. 2020). A recent review of 18 school-based health interventions reported that only 2 interventions were focused on mental health (Herlitz et al. 2020), specifically suicide prevention (Rauscher et al. 2015) and post-traumatic stress disorder (Nadeem & Ringle 2016). While one or more core components of some interventions were sustained, none of the interventions were entirely sustained. There was also no consistent pattern between evidence of effectiveness during implementation and sustainability of the intervention post-implementation (Herlitz et al. 2020). In addition to explaining the importance of dedicated school leaders, challenges with staff turnover, and continued need for resources and training, the authors also emphasized feasibility—which is the degree to which an intervention can be integrated into existing school routines, policies, and plans—as a key factor in sustaining health interventions in schools (Herlitz et al. 2020).

Little is known about sustaining preventive mental health interventions in schools following initial program implementation during research trials and the extent to which implementation determinants in this context are consistent with the broader sustainability literature. To bridge the research-to-practice gap, increase the public health impact of research studies, and promote health equity, it is important to understand barriers and facilitators to sustaining school-based interventions following their delivery in research trials. Researchers have also advocated gathering data from individuals in different roles when studying program sustainability within school-based research studies (Friend et al. 2014). This qualitative study sought to extend existing literature by describing perceptions of school administrators and staff members about multilevel factors that impacted sustainability of a trauma-informed universal mindfulness intervention for 8^th graders across 13 schools in a large urban school district following implementation in an efficacy trial.

The RAP Club Intervention and Project POWER Trial

RAP (Relax, be Aware, do a Personal rating) Club is a trauma-informed universal mental health intervention implemented as part of a larger efficacy trial called Project POWER (Promoting Options for Wellness and Emotion Regulation; Mendelson et al. 2020). RAP Club was adapted from a clinically-based treatment program, Structured Psychotherapy for Adolescents Responding to Chronic Stress (SPARCS; DeRosa et al. 2006). RAP Club uses evidence-based practices for enhancing mental health including mindfulness strategies to promote emotion regulation skills, cognitive behavioral therapy strategies to teach problem solving and communication skills, and psychoeducation about the effects of stress and trauma. Pilot research demonstrated that RAP Club improved academic and social competence, emotion regulation, and classroom behavior among adolescents when compared with regular school programming (Mendelson et al. 2015). In a previous study, school administrators explained that they adopted RAP Club to provide support for students affected by trauma and to prevent students from engaging in unhealthy coping behaviors associated with trauma (Arnold et al. 2020).

RAP Club was implemented with 8^th graders in 13 schools during the first two years of the Project POWER trial (2016-2018). General education students who provided parent permission and assent were eligible for inclusion in the trial, with an average of 22 students enrolled per school. The principal at each participating school selected 1-2 school mental health professionals (or teachers, if mental health personnel were not available) to be trained in the intervention curriculum and techniques. The study team trained and supervised school personnel during the trial to build schools’ capacity to sustain the program beyond the trial. Participating school personnel attended a two-day interactive RAP Club training led by two SPARCS trainers in which they received training materials, an instructor manual, and student intervention materials. School staff then attended the 12-session program during the fall, which was primarily facilitated by a trained study team member and young adult community member (called a “mentor”). The school personnel-in-training provided assistance with program delivery and also participated in group supervision calls over the six weeks of program delivery; they received financial compensation from the study for their time in training and assisting with program delivery. Schools had the option to sustain RAP Club after the trial ended.

To facilitate sustainability following research participation, schools were encouraged to send school staff to RAP Club training for free in subsequent study years as a refresher or to expand the number of trained school personnel who received training. The study team also offered free consultation regarding program implementation after the trial concluded. The modeling of RAP Club’s implementation for school staff by the research team prior to schools taking on the full responsibility for program delivery, providing ongoing supervision for additional support during the intervention, and offering financial incentives for time spent in training and implementation-related activities were all strategies used to promote sustainability of RAP Club (Cook et al. 2019; Koschmann et al. 2019).

Current Study

This study adapted and applied a multilevel model of factors hypothesized to affect the implementation and sustainability of preventive interventions in schools (Domitrovich et al. 2008) to evaluate sustainability of a preventive school-based mental health intervention. Individual-level factors in the framework include professional characteristics (e.g., education and training), psychological characteristics (e.g., burnout, self-efficacy), and program perceptions (e.g., attitudes, perceived effectiveness). School-level factors include resources, decision structure, leadership, alignment of school policies/mission with intervention goals, school climate and culture, and personnel expertise. Macro-level factors include federal, state, and district policies and financing, community-university partnerships, and leadership and human capital. This study sought to examine how factors across all these levels impacted sustainability of RAP Club beyond research participation.

Methods

Study Context

This study was conducted in the Baltimore City Public Schools (City Schools) district in Maryland. There are over 79,000 students in the City Schools district, most of whom are Black (76.6%; 13.5% are Hispanic/Latinx and 7.6% are White); 50.4% of students are estimated to be part of families with low income (as measured by participation in public assistance programs including Temporary Assistance for Needy Families and Supplemental Nutrition Assistance) (Baltimore City Public Schools [City Schools] 2020). City Schools provides free and reduced meals to all students in the district because low-income status is under-reported (City Schools 2020). City Schools is considered to be “under-resourced” because it is located in a low-income school district. The district has a commitment to promoting the holistic well-being of students through school-based programs and community partnerships that seek to improve health and academic outcomes. Local universities in Baltimore City often partner with schools to implement innovative and evidence-based interventions that are part of research studies, but these interventions are rarely scaled-up or sustained beyond research participation.

Participants

The sample for this study included school administrators (i.e., principals, vice principal) who adopted and implemented RAP Club as part of participation in the Project POWER trial, and school staff (e.g., social workers, school psychologists, guidance counselors, and teachers) trained to deliver RAP Club at the 13 schools that were enrolled in the first two cohorts of the trial. A total of 13 principals and 14 staff members (one school had 2 staff members trained) were contacted via phone, email, and school visits to participate in a semi-structured key informant interview about factors that influenced RAP Club’s sustainability at their school. Participants received $15 for their time.

Procedures

Key informant interviews were conducted by the first author 1-2 years after schools completed implementation of RAP Club during the Project POWER trial. Semi-structured interview guides for administrators and school staff were initially developed by co-investigators of the trial and later expanded by the first author to more explicitly investigate individual, school, and macro level factors drawn from Domitrovich and colleagues’ multilevel framework (Domitrovich et al., 2008). Interview guides included questions about how these multilevel factors influenced sustainability of RAP Club. With interviewees’ permission, the interviews were recorded. All recordings were transcribed by a professional transcription service.

Intervention fidelity logs were completed by each RAP Club group facilitator (independently or with the community mentor who helped deliver the intervention) after each RAP Club session. In addition to ratings of intervention implementation adherence and quality for each session, fidelity logs also included questions about the most successful part of the session, the most challenging part of the session, and any issues that would be helpful to discuss during supervision. All RAP Club implementers (research team members, community mentors, and school staff members in training) were invited to participate in group supervision calls led by the Project POWER principal investigator and/or senior program coordinator. During the calls, intervention implementers from each school reported student attendance and engagement, successful and challenging aspects of intervention sessions, and other information relevant to RAP Club implementation. Supervisors took notes to document the key content of the calls. Intervention fidelity logs and supervision call notes were reviewed to capture potential barriers and facilitators to intervention sustainability.

Data Analysis

Interview transcripts and intervention documents (i.e., fidelity logs and supervision call notes) were analyzed using Yin’s 5-phase approach to qualitative data analysis (i.e., compile, disassemble, reassemble, interpret data and conclude) (Yin 2011). Data from the interview transcripts, fidelity logs, and supervision call notes were coded in Atlas.ti, a qualitative data management software program (Hwang 2008). Preliminary codes for interview transcripts and intervention documents were derived using an inductive approach. The initial list of 16 codes was expanded to 30 and then applied to all transcripts and intervention documents; additional codes and sub-codes were added as needed during the analysis phase based on the data. The constant comparative method was used to identify patterns in the data within and between schools (Bowen 2009; Glaser et al. 1967). Emergent themes were organized into categories based on the conceptual framework by Domitrovich et al. (2008). Data triangulation and debriefing among authors enhanced confidence in the resulting thematic findings (Baxter et al. 2008; Creswell 2007; Denzin 1989; Merriam 1998).

Results

A total of 21 administrators and school staff members from 13 schools that implemented RAP Club during the first two cohorts of the Project POWER trial (2016-2018) participated in this sustainability study (Table 1). Twenty of the 27 principals and staff members contacted were interviewed, and the principal of one of the schools requested that the vice principal be invited to participate in a joint interview; the consent rate was 77.7%. The school representation rate was 100%; at least 1 administrator or staff member from each school provided information about factors that influenced sustainability of RAP Club at their school. Interviews were conducted with both the principal and staff member at 6 schools, either the principal or staff member at 6 schools, and the principal at the last school sent email responses in lieu of a formal interview due to scheduling conflicts; the staff member at the same school did not reply to outreach attempts. Interviews were conducted in person when possible (n=10) and by phone when preferred by the participant (n=10), with an average length of 30-45 minutes per interview. A total of 147 intervention documents were analyzed from all 13 schools (n=137 fidelity logs; n=10 supervision call notes).

Table 1:

Background Characteristics from n=21 School Administrators and Staff

School	Role	Gender	Cohort
1	Principal	Female	1 n=12
	Staff	Male
2	Principal	Female
	Staff	Male
3	Principal	Male
	Staff	Female
4	Principal	Female
	Staff	Female
	Staff	Female
5	Interim Principal	Male
	Staff	Female
6	Staff	Female
7	Principal	Female	2 n=9
8	Principal	Female
	Staff	Female
9	Principal	Female
	Vice Principal	Male
10	Staff	Female
11	Staff	Female
12	Staff	Female
13	Principal	Female*

^*Principal of School 13 from Cohort 2 was not interviewed but provided responses via email that were included in the analysis.

Although most key informants mentioned during their interview that they found RAP Club to be acceptable, appropriate, and beneficial to their students, none of the 13 schools sustained RAP Club after the trial. Individual, school, and macro level factors that influenced sustainability of RAP Club are described below and summarized in Table 2.

Table 2:

Summary of Factors that Influenced Sustainability of the RAP Club Intervention

Domains from Multilevel Model by Domitrovich et al. (2008)	Key Themes from Data
Individual Level Factors
Intervention Perceptions and Attitudes	Facilitators  ● Acceptability of intervention  ● Perceived need to offer the intervention  ● Perceived effectiveness of intervention
Psychological characteristics	Barriers  ● Low self-efficacy  ● Concerns about professional burnout
School Level Factors
Administrative leadership	Facilitator  ● Administrative support of RAP Club Barrier  ● Administrative changes
Decision structure	Facilitators  ● Decision to sustain is made solely by principals and/or other administrators  ● Decision to sustain is made by principal in collaboration with other administrators and staff members
Personnel expertise	Barriers  ● Turnover and/or shortage of staff trained to deliver intervention
Resources	Barriers  ● Lack of extra space in the school to routinely deliver the intervention  ● Difficulty fitting intervention into school schedule
Macro Level Factors
Policies and financing	Facilitator  ● Intervention aligned with district student wholeness policies and priorities for addressing trauma exposure among students and providing programs to promote social and emotional skills Barriers  ● Lack of funding for intervention
University/community partnerships	Barriers  ● Removal of resources provided by university partner during the efficacy trial including outside staff to help deliver the intervention, incentives for students, and stipends for school staff  ● Lack of sufficient communication between schools and researchers regarding how to sustain intervention beyond research trial participation

Individual Level Factors

Individual level factors relevant to RAP Club’s potential sustainability in schools included perceptions about the intervention—including the perceived need to continue offering the program and perceived effectiveness of the program—and psychological characteristics (e.g., low self-efficacy among school staff, concerns about professional burnout among administrators).

Intervention Perceptions and Attitudes.

Nearly all administrators and staff members trained to deliver RAP Club described the program as acceptable and appropriate for their students. They expressed support for sustaining RAP Club because it filled an important gap in their regular school programming by teaching students about mental health topics such as the potential negative impacts of trauma and positive benefits of mindfulness:

“The information that they can gain from the program is much needed…it is quite useful for them to have that knowledge, which is why I would say, yes, I would want to have [RAP Club] at this school. I would want to use it [again].” (Staff member)

Others mentioned acceptability of RAP Club for equipping students with skills in stress management, communication, and problem-solving:

“I loved the group sessions that the kids had. We need it, because there’s just so many things working against them—[like] social media. They’re turning to the wrong sources for answers, you know?… So, I definitely want it, because my kids just need that constant reinforcement on what to do and how to handle issues and problems, and it’s not to revert to violence.” (Principal)

Most administrators and staff members expressed willingness or an explicit desire to offer the program again based on their perception that RAP Club was a useful and healthy outlet for students to learn about trauma and healthy ways to respond to stressful situations.

Several administrators and many staff specifically expressed interest in sustaining RAP Club because they perceived the program benefited their students:

“I would still want [RAP Club] in the schools. It’s just a benefit to the students, and the more they’re exposed to it, the better coping and the better choices they’ll be able to make in the future.” (Principal)

“Yes, I would [like to continue offering RAP Club] Because I feel like when I’m comparing the students that participated in RAP [Club] with the students who did not, I feel like they are able to better adapt to the craziness… I feel like they have learned some sort of way to deal with it.” (Staff member)

Only one study participant (a staff member) expressed unfavorable perceptions of RAP Club’s effectiveness, saying, “I don’t think as a school we found it to be particularly effective, and we got a lot of pushback from kids about missing their classes…I don’t know that we would’ve gotten the support [to continue] even if I would’ve championed it.” Questionable effectiveness and conflicts with students’ resource classes (e.g., gym, art, music) contributed to negative perceptions shared by this staff member and may have been a barrier to sustainability at her school.

Psychological characteristics.

Concerns about low self-efficacy and professional burnout emerged as potential individual-level barriers to program sustainability. For example, some staff members explained that they were not confident about their ability to deliver the program on their own without research team support. Fidelity logs and supervision call notes revealed that staff attendance and participation in RAP Club sessions and supervision calls (which were implementation strategies used by the research team to promote self-efficacy among school staff) varied widely across schools. While some staff members were very engaged in program delivery and participated in most of the program components, others were not, which likely influenced their self-efficacy and perceived readiness to deliver the intervention on their own. Possibly due to competing responsibilities, school staff attendance was low for supervision calls during Cohorts 1 and 2; none of the supervision calls had 100% attendance from school staff members who were trained to deliver RAP Club. During Cohort 1, staff members from only 1-3 schools were present during each call. Of the 5 supervision calls that occurred during Cohort 1, only 1 call had representation from 3 schools; 1 call had representation from 2 schools and the rest only had staff representation from 1 school. During Cohort 2, school staff members only participated in 3 of 5 supervision calls; 1 call had representation from 4 schools and the other 2 calls had representation from 2 schools.

Some administrators and staff members raised the concern that sustaining RAP Club without staffing or financial support from the research team could contribute to professional burnout (including role strain) among school staff. One school mental health professional noted this may be more problematic for teachers than for mental health personnel who may be accustomed to delivering mental health interventions as part of their work:

“The way that the system worked was kind of like a train-the-trainer model where there wouldn’t have been the support from the [research team the next year], and it would’ve been complicated to have enough staff members…we’re stretched very thin, and to be able to say ‘We need people to do this without compensation on top of your job’ is a big ask, specifically if you’re asking for a teacher. I can run a group as a part of my job [as a school social worker] and not feel as put out, but to say to a teacher ‘Give me your planning period’ I think would never work.” (Staff member)

In a resource-constrained school environment, teachers are at risk for being overburdened by new tasks; thus, administrators may be reluctant to require additional roles from them without the ability to compensate them for the new tasks or reduce existing responsibilities to accommodate new ones (i.e., facilitating a mental health intervention).

School Level Factors

School level factors associated with program sustainability included administrative leadership and decision structure (e.g., top-down vs. collaborative decision-making between administrators and school staff), personnel expertise and staffing issues (e.g., staff turnover), and resources (e.g., space, time in the regular school schedule).

Administrative leadership and decision structure.

Administrative buy-in and support for the program was commonly described as critical for program continuation. Many staff members stated how commitment from the school’s administration (particularly the principal) is a key factor in determining and facilitating program sustainability. Some staff members stated the decision to sustain RAP Club would be made unilaterally by the principal, highlighting the importance of having their support for the program:

“That wouldn’t be my decision. That would be the decision of the principal…That’s probably the big thing. If she decides if she wants to do it. She’ll probably ask if I think it’s beneficial. I’ll give her my feedback…but that’s how that will be determined.”

(Staff member)

Four principals across both cohorts left the school after implementation of RAP Club. A staff member at one school described an administration change as a major barrier to sustaining RAP Club at her school. Just as staff mentioned the importance of having the support of the school’s administrative leadership, some principals mentioned the importance of having the support of their staff members who would be involved in implementation of the intervention if it were sustained. They said they would ask for their staff’s input on the decision to sustain RAP Club.

Personnel expertise and staffing issues.

Administrators mentioned that they would like to continue offering RAP Club and offer it to more students after the research study but needed additional trained staff in the school to implement the program. At a few schools, staff who were trained to deliver RAP Club moved into different positions or left the school following implementation of the program during the trial. For example, when asked why RAP Club did not continue at her school, one principal responded, “My Director of Culture and Climate who was the main facilitator of the program moved into another leadership position so she’s no longer at the school and one of the teachers who [worked with RAP Club] is no longer at the school.” As few staff members were trained to deliver RAP Club from each school (only 1 or 2 people), staff turnover and shortages presented major barriers to sustainability.

Resources: Space conflicts.

RAP Club was delivered in different locations within each partner school, including the library, music room, classrooms of teachers who had planning periods during the time of the intervention, or another unoccupied room. Some administrators and staff described a shortage of space as a barrier to sustainability; these descriptions were supported by documentation of space conflicts in fidelity logs and supervision call notes. Common space issues included not having enough school rooms to consistently accommodate program delivery and difficulty securing space to store intervention materials during and beyond the trial.

Resources: Scheduling conflicts.

Almost all administrators and staff members described time constraints in the school schedule as a barrier to program sustainability. Scheduling was the school-level factor most frequently recorded by research staff as an implementation challenge in the intervention documents reviewed for this research. The primary problems reported were difficulty fitting RAP Club into the school schedule and scheduling conflicts with teachers’ planning periods.

Macro Level Factors

The two macro level factors cited by school administrators and staff members as being most relevant to sustainability of RAP Club across schools were policies and financing (e.g., City Schools whole child policies that encouraged principals to offer mental health programs) and community-university partnerships (e.g., funding and implementation support from the research team).

Policies and financing.

Several administrators and staff mentioned funding as a barrier to sustainability. For example, when asked about barriers to sustaining RAP Club, one staff member relayed, “The main thing would most likely be funding.” City Schools district provides funding for each school to have at least one social worker and/or one psychologist based on the proportion of students at the school with identified mental health needs. Although RAP Club is aligned with the district’s “whole child” policies, preventive mental health programs like RAP Club must be funded through individual schools’ budgets or an external partnership. To sustain RAP Club following participation in the Project POWER trial, administrators would have to include RAP Club in their budget in February for the following school year or secure other funding to implement the program.

Community-university partnerships.

While implementing RAP Club during the Project POWER trial, the research team provided staff to deliver the program, intervention materials, incentives for participating students, curriculum training, supervision calls, and stipends for school staff members-in-training. Once the first formal year of the community-university partnership ended, outside staff, incentives, and stipends were no longer provided. However, schools had the option of receiving free consultation and sending staff for free RAP Club training prior to the following school year. Interview data suggest inconsistent understanding of these sustainability options. Some administrators and staff members said they did not know that they could continue delivering RAP Club after their participation in the trial ended. One staff member thought the school was prohibited from using intervention materials. Another staff member said, “I wasn’t very clear on what was happening afterwards. So, I didn’t know that sustaining [RAP Club] was even an option.” A different staff member stated: “I knew there was that option [to sustain RAP Club], but I just wasn’t sure that my principal was clear.” Some participants noted that they were not contacted by the research team after the study ended: “We weren’t contacted to continue it…I do think that if we had been contacted to, you know, do it again, I’m sure we would have” (Staff member). Miscommunication between the university and school partners about the continuation of RAP Club after initial implementation and lack of planning for sustainability contributed to the program not being sustained.

Discussion

Given the significant potential benefits of universal trauma-informed school-based mental health interventions, it is critical that we identify strategies to promote sustainability of these interventions in schools, particularly in under-resourced schools in economically disadvantaged school districts in which student mental health needs are often high and implementation challenges are typically substantial. The current study is one of the few sustainability studies to gather perspectives from diverse school personnel (Friend et al. 2014)—including administrators, school mental health counselors, and teachers—in addition to analyzing documents pertaining to intervention delivery. Although all but one of the participants in this study described RAP Club to be acceptable, appropriate, and beneficial to their students, none of the 13 schools continued the intervention following participation in the trial in which it was originally delivered.

Positive stakeholder perceptions thus did not lead to program sustainability. The most recent systematic review of sustainability of public health interventions in schools reported no noticeable associations between intervention effectiveness and sustainability outcomes (Herlitz et al. 2020). While the perceived effectiveness of programs for the wellbeing of students was mentioned as influential to sustainability, scientific evidence of program effectiveness did not significantly impact sustainability of programs in the review (Herlitz et al. 2020). Scientific evidence of RAP Club’s efficacy is being assessed as part of a multiyear efficacy trial. Thus, only perceived effectiveness of the intervention on students was assessed among study participants, and yet positive perceptions of effectiveness were not enough to influence sustainability in a way that yielded RAP Club continuing at any of the participating schools.

At the individual level, the barrier most frequently mentioned in relation to sustainability was low self-efficacy among staff members who were trained to deliver RAP Club. Self-efficacy is a factor identified in the literature as an important facilitator to sustainability of school-based health interventions (Domitrovich et al. 2008; Eiraldi et al. 2015; Friend et al. 2014; Han & Weiss 2005; Lytle et al. 2003). Although all staff that we interviewed were trained to facilitate RAP Club, some described not feeling confident in their ability to deliver the intervention on their own beyond the trial. This finding is consistent with literature that highlights the importance of boosting self-efficacy of school staff to deliver an intervention following the conclusion of formal research support (Eiraldi et al. 2015; Friend et al. 2014; Han & Weiss 2005; Owens et al. 2014). Examination of RAP Club intervention documents revealed low participation of school staff in supervision calls designed to enhance their self-efficacy. This finding led to improvements in ongoing consultation and supervision provided for school staff members during future cohorts in the trial to increase school staff’s self-efficacy to deliver RAP Club. Instead of group supervision calls, supervision occurred individually between RAP Club group leaders (who were part of the research team) and school staff. These meetings were set up based on the most convenient time (e.g., after each session, after school), place (e.g., classroom, teacher’s lounge, etc.), and mode (e.g., phone, in-person) for the school staff. Coaching, which is an implementation strategy that provides in-person skills modeling and post-training practice with feedback from an expert during treatment/intervention delivery (Koschmann et al. 2019), may have helped improve self-efficacy among staff who were trained to deliver RAP Club. When planning for sustainment, it is important for schools and university partners to determine the level of involvement of the research team in providing consultation, supervision, and/or coaching during future implementation of school-based interventions beyond the research context. Researchers should ensure school leadership has a plan for assuring these post-training support roles are filled and supported upon conclusion of research participation. Future research should identify and test implementation strategies to increase engagement and self-efficacy of school staff members-in-training, including alternative training modalities (e.g., online vs. in person) and post-training support (e.g., consultation, supervision, and coaching; Koschmann et al. 2019). Feedback regarding self-efficacy should be obtained from school staff throughout the stages of implementation (e.g., training during the pre-implementation phase, supervision calls during the implementation phase, and surveys and interviews during the post-implementation phase) to determine if post-training support strategies need to be improved.

At the school level, the primary barriers were staffing (e.g., turnover, shortage) and resources (e.g., time, space). Staffing issues were described by participants as a major barrier to program sustainability. Other studies have also reported staff turnover as a barrier to sustaining school-based health programs (Eiraldi et al. 2015; Friend et al. 2014; Owens et al. 2014). Possible remedies include training multiple school staff members initially so that “backups” are available in case of staff transfers and providing additional training opportunities during and after the study using online modules and videos (Friend et al. 2014). Schools could also consider alternative staffing options, such as training student leaders to be program co-facilitators.

Consistent with studies on sustainability of school health and mental health programs (Eiraldi et al. 2015; Han & Weiss 2005; Nadeem & Ringle 2016; Owens et al. 2014), we also found that resources such as time in the school schedule and space in the school for consistent program implementation were school-level barriers to program sustainability. While research teams generally work closely with schools to address these resource issues during program delivery within research studies, it is important that plans are made to address anticipated challenges with resources during subsequent years. A school liaison or “champion” could be identified to work with the research team to adapt and integrate the intervention into the regular school routine after the study ends to mitigate these challenges.

Administrative leadership and decision structure also emerged as school-level factors that could facilitate or hinder sustainability of RAP Club. Participants confirmed the primary decision-making role of principals for sustaining school-based interventions, with varying levels of staff involvement in the decision. Some administrators mentioned that not having staff buy-in would hinder sustainability, while some staff reported that not having administrator buy-in would be a barrier to sustainability. This suggests support from both administrators and staff is likely necessary for sustainability, which is consistent with previous research that describes buy-in and support from school stakeholders as key to sustaining school-based interventions (Domitrovich et al. 2008; Forman et al. 2009; Friend et al. 2014; Han & Weiss 2005).

Macro-level barriers to sustainability included lack of funding and limited community(school)-university communication following participation in the trial. Overall, findings from this study highlight the importance of developing a structured plan for program sustainability with school partners during the adoption phase, revisiting the plan during the implementation phase, and reviewing the plan with schools during the post-implementation phase to increase likelihood of program sustainability upon conclusion of the research trial (Nadeem & Ringle 2016; Owens et al. 2014). It is also important for university partners to work with schools early on to identify external funding for the program and/or include it as a line item in the school’s budget prior to district deadlines for school budgets for the following year. That said, advocating for investments in sustainability before evaluation findings are known can be problematic.

Varying perceptions among school stakeholders about the sustainability of RAP Club after the research trial ended suggests that clear and ongoing communication between universities and schools is key to sustainability. More specifically, connecting with principals directly emerged as particularly critical at or near the end of program implementation during a trial in order to revisit their intentions to sustain the program and specific plans for incorporating the program into the school’s regular routine. Sustainability plans could include developing protocols for ensuring consistent communication during and after program implementation within the research context, identifying school staff to continue working with university partners, and determining potential funding sources to pay for future program implementation costs. Methods of communication could include emails, brief periodic surveys for relevant school personnel (e.g., 3-5 questions), regular in-person meetings, and/or phone calls.

Unlike most school-based intervention trials, Project POWER explicitly integrated plans for program sustainability into its study protocol, including identification and training of school staff (including guidance counselors, teachers, and social workers) who could continue to deliver the programming following participation in the trial. Study investigators attempted to alleviate the pressures facing school staff by using multiple implementation strategies, including modeling of group facilitation by the research team prior to taking on sole responsibility for program delivery, ongoing supervision for additional support during the implementation phase, and payment for school staff time spent in training-related activities (Cook et al. 2019). The fact that program sustainability did not occur despite the research team’s use of implementation strategies to promote sustainability and high levels of stated program acceptability and perceived effectiveness among school stakeholders suggests that the anticipated resources required for program sustainment may have exceeded what schools deemed to be feasible outside the research context. Additionally, without explicit prompting from the research team or a program champion within the school, principals likely did not prioritize, or perhaps even remember, intervention implementation following the trial, given demands on them to conduct extensive financial and academic planning for the following year. Intervention planning for the following school year should ideally start during the winter (school budgets are usually due in February), just a few short months after initial implementation. As previously mentioned, having a designated school implementation leader to work with university partners at the end of trials could be a key feature of a practical sustainability plan. Testing strategies to boost the sustainability of school-based public health interventions—including those focused on mental health—is an important area for future research.

Increased use of hybrid effectiveness-implementation trials (Curran et al. 2012) with explicit plans and funding for program sustainment may be important for facilitating sustainability beyond the context of research studies, particularly those conducted in schools with limited resources for delivering preventive mental health interventions. For instance, the trial reported herein did not have adequate staffing or funding to devote significant time toward working with each school following its one-year trial participation, nor was sustainability a study aim the investigators were funded to accomplish. When incorporating implementation outcomes into grant proposals, researchers should consider adding an aim focused on determinants, mechanisms, and/or testing implementation strategies for sustainability.

Limitations

Not all administrators and staff members trained to deliver RAP Club were interviewed; however, we interviewed at least one school administrator or staff member from 12 of the 13 schools and obtained written responses to the interview guide from the principal at the 13th school. Perspectives provided by study participants may not be fully representative of other school stakeholders involved in RAP Club implementation or school stakeholders at other urban schools more broadly. Interview responses may also have been influenced by social desirability bias; however, in most cases, the interviewer was an individual with whom the participant had not previously interacted during the trial, potentially reducing social pressure to respond positively about the program.

Conclusions

Sustainability of trauma-informed universal mental health interventions is a challenging but important issue, as school mental health interventions have potential to improve students’ mental health and academic success. This study identified several facilitators and several barriers at the individual, school, and macro levels to sustaining a preventive mental health intervention in urban schools. Most barriers appeared to be at the school level, including staff turnover, scheduling challenges, and limited space. A key finding was the need for researchers to have ongoing communication with school leadership and to iteratively develop sustainability plans with school partners throughout the phases of implementation to increase the likelihood of sustaining an intervention following research participation. Findings also highlight the need for identification and testing of implementation strategies to promote sustainability, including structured sustainability planning, ongoing communication about subsequent program delivery, use of online modalities for training and ongoing supervision, and maintenance of program skills among school staff. Funders of school-based intervention trials should also consider weighting sustainability concerns more heavily when developing requests for proposals and reviewing applications.

To the extent that evidence-based interventions reduce harmful effects of trauma on students, it is critical to determine how best to sustain these programs beyond research trials, particularly in under-resourced schools serving students of color and students from families with low income who often have less access to high quality mental health services. Not sustaining interventions that have the potential to positively impact the health and academic outcomes of children and adolescents represents a missed opportunity to reduce mental health disparities and improve health, academic, and social outcomes for school-aged youth. School-based research trials offer an important opportunity for researchers to partner with schools to integrate these interventions into the school ecology, increase access to evidence-based mental health services, and achieve health equity. Researchers, school administrators, and education policymakers should collaborate to develop models that best support sustainability of school-based health programs to promote long-term student health and wellbeing.