A Mixed-Method Study Examining Solutions to Common Barriers to Teachers’ Adoption of Evidence-Based Classroom Practices

Abstract

Objectives:

We conducted a mixed-method focus group study to (a) assess the appropriateness and likely effectiveness of strategies that target individual behavior change mechanisms associated with perceived barriers of lack of time and unsupportive leadership and (b) identify recommendations regarding strategies for overcoming the barriers.

Method:

Sample included 39 school-based staff (80% female, 77% White) across two districts in the Midwest. Mixed methods included a simultaneous approach.

Results:

Lack of time and supportive leadership continue to pervade school-based implementation efforts. Recommendations centered around the need for school leaders to give teachers the power to re-prioritize how they spend their time as well as providing protected, facilitated time for teachers to collaborate and learn practical skills targeting self-advocacy.

Conclusion:

Our findings provide compelling evidence for the use of implementation methodology to strategically target mechanisms of individual behavior change during the process of incorporating new and innovative practices in schools.

Approximately 20% of children exhibit social, emotional, and behavioral (SEB) needs that warrant prevention and intervention supports (Merikangas et al., 2010). Compared to other child-serving settings, schools are the most common place where children access needed social, emotional, and behavioral (SEB) supports (Duong et al., 2020). There are numerous evidence-based practices (EBPs) available for use as part of routine service delivery in schools. EBPs represent a systematic approach to determining which programs, practices, interventions, or procedures are supported by a sufficient number of studies with high methodological quality and demonstrate meaningful change over a generalizable sample of participants (Cook et al., 2012). The What Works Clearinghouse, an arm of the Institute for Education Sciences within the U.S. Department of Education, have established “gold standard” requirements for evidence-based practices (i.e. study methodological quality) and serves as a valuable resource for practitioners to learn about which educational practices are effective (https://ies.ed.gov/ncee/wwc/) A few examples of EBPs include the good behavior game (Kellam et al., 2011), Check-In Check-Out (CICO; Crone et al., 2004; Maggin et al., 2015), and behavioral contracts (Mruzek et al., 2007). Despite the availability of evidence-based practices for use in schools, there are significant barriers to implementation even when supportive leadership is in place and high-quality training and consultation are provided (Goldenthal et al., 2021). These barriers perpetuate the research to practice gap that ultimately precludes students from receiving the best available prevention and intervention services.

Individual characteristics (e.g., attitudes, stress, self-efficacy, implementation intentions) of the educators who are expected to adopt and deliver EBPs in schools (e.g., teachers) are one category of barriers that interfere with successful implementation in schools; implementation ultimately rests with designated implementers making choices to adopt an EBP and persist to deliver it with high fidelity (Long et al., 2016; Michie et al., 2011; Tabak et al., 2012; Weston et al., 2018). Implementers’ perceptions regarding time and supportive leadership are two of the most common barriers that impede successful implementation in schools (Long et al., 2016; Ransford et al., 2009), and they have been identified across service sectors: among healthcare professionals (Deenik et al., 2019; Geerligs et al., 2018) and among therapists in mental health clinics (Ringle et al., 2015). Most research to date has focused on merely documenting these barriers, with limited research examining potential solutions to overcome them, and studies that do explore solutions have done so at the systems level (Fixsen et al., 2005; McGoey et al., 2014; Pinkelman et al., 2015; Power et al., 2005).

The purpose of this inquiry was to conduct a mixed-method study with three different stakeholder groups (teachers, school administrators, and support staff) to gather data on the appropriateness and likely effectiveness of solutions to address perceptions of lack of time and supportive leadership as barriers to implementation. We also sought to gather unique recommendations to overcome these common barriers to EBP implementation, including those at the individual level of implementation. Mixed methodology was selected in order to provide both breadth and depth to our findings. Quantitative data allows us to test hypotheses about the appropriateness and effectiveness of the strategies proposed and to determine potential statistical differences between stakeholder groups and geographic settings. Qualitative focus group data allows us to probe deeper into the nuance and rationalization for quantitative ratings (Palinkas et al., 2011; Regnault et al., 2018). Mixed methodology is a common approach in implementation research, particularly in the school context, and several studies have used mixed methods (Brann et al., 2021; Hemphill et al., 2015; McLaughlin et al., 2020).

Implementation Gap in Schools

Due to the increased opportunity for widespread access to needed SEB supports, considerable attention has been paid to ensure that school-based prevention and intervention supports are evidence-based (Kutash et al., 2006; Lyon & Bruns, 2019). Unfortunately, EBPs are rarely adopted or delivered with adequate fidelity in real-world school settings (Owens et al, 2014). Research across disciplines reveals a common “implementation gap” between what research has been demonstrated as effective and what is routinely used in everyday service settings (Flay et al., 2005; Hagermoser Sanetti & Collier-Meek, 2019; Weisz, et al., 2006). As such, research is increasingly moving away from developing new prevention and intervention programs and is instead focusing on understanding the implementation processes, outcomes, and strategies that help transfer research findings into real-world settings (e.g., Cook et al. 2015; Wanless & Domitrovich, 2015). This shift in focus among researchers is now on developing generalizable knowledge that can be used in everyday implementation efforts to overcome common barriers to implementation and facilitate children’s access to high quality supports.

Implementation Determinants

Implementation research suggests that there are determinants across multiple levels and phases of the implementation process that impact EBP uptake and use (Aarons et al., 2011; Damschroder et al., 2009; Domitrovich et al., 2015; Han & Weiss, 2005; Ransford, 2007). Determinants are factors that either obstruct or enable successful implementation and are often referred to as barriers and facilitators (Nilsen, 2015). The field of implementation has reached general agreement regarding the categorization of determinants across levels of influence: (1) outer setting determinants (i.e., factors beyond the immediate setting in which implementation takes place, e.g., policy, funding), (2) inner setting determinants (i.e., factors within the specific setting in which implementation takes place, e.g., school leadership, climate, staffing), (3) determinants associated with individuals who are expected to adopt and deliver the practice (e.g., teachers’ beliefs, self-efficacy, levels of burnout), and (4) determinants associated with the EBP itself (e.g., costs, complexity, appropriateness) (Lyon & Bruns, 2019).

In educational settings particularly, inner setting determinants including leadership, organizational climate, quality of professional development, and quality of follow-up support, have been identified as significant determinants of implementation (e.g., Collier-Meek et al., 2018; Fixsen et al., 2005). Despite the importance of these determinants, even when optimal organizational factors are in place, successful implementation is not guaranteed. For example, research on classroom-based EBPs, has demonstrated that teacher-level factors (i.e., their beliefs, attitudes and perceptions, and self-efficacy) are most predictive of adoption and implementation, while organizational factors less often associated with EBP use (Becker et al., 2013; Locke et al., 2019). Thus, the uptake and use of EBPs likely rests with individual, front-line implementers—most often teachers in school contexts (Forman et al., 2009; Weston et al., 2009)—who make decisions about whether to change their own behavior regarding EBP adoption. Therefore, intervening on individual-level factors that facilitate or impede teacher delivery of EBPs is especially important to improve student outcomes (Cook et al., 2015).

Considering this, it is critical to understand and address individual-level determinants as part of the implementation process. Individuals’ beliefs and perceptions about time and support from their administrators are two of the most widely cited individual-level belief determinants that can impact EBP implementation (Lyon et al., 2016; Pagoto et al., 2007; Pinkleman et al., 2015). For instance, teachers who think they do not have the time to deliver a new practice experience less implementation success than teachers with favorable attitudes and beliefs regarding time (Collinson & Cook, 2001; Skaalvik & Skaalvik, 2010; Weston et al., 2018). Moreover, beliefs about whether one perceives that they lack support from their administrator(s) are another key individual-level influence on EBP implementation (McMahon et al., 2017; Sangster-Gormley, et al., 2011). In a 2020 survey, the Institute for Arts Integration and Steam conducted a State of Teaching survey of around 5,000 elementary, middle, and secondary general education, English language, arts, and STEM (Science, Technology, Engineering, Math) teachers across the country, and found that 55% reported that they felt insufficiently supported by their administrators (De La Rosa, 2020). These feelings or thoughts, whether favorable or unfavorable, are directly related to implementation behavior. McIntosh and colleagues (2013) demonstrated that perceived supportive leadership was the most important facilitator for sustainability of a schoolwide approach to behavior management. Thus, efforts to support teachers’ motivation to overcome their unhelpful or maladaptive beliefs about time and leadership support are needed, as implementers who have facilitative beliefs that produce feelings of motivation are more likely to engage in efforts to implement (Lyon et al.,2013).

Leveraging Facilitators and Overcoming Barriers through Implementation Strategies

Considering these aforementioned individual-level perceptual or belief barriers (lack of time and supportive leadership), there is utility in selecting or developing individual-level implementation strategies to overcome them. Implementation strategies are defined as methods or techniques designed to enhance implementation outcomes, such as acceptability, adoption, and fidelity (Proctor et al., 2013). Implementation strategies can be conceptualized as adult-facing interventions that aim to facilitate implementer behavior change. For example, large group didactic training is considered a cornerstone implementation strategy designed to increase implementer knowledge about a given program or practice to increase the likelihood they will perform behaviors consistent with the program or practice. Though many authors have detailed how common barriers such as lack of time and lack of supportive leadership disrupt the implementation of EBP in schools (Distel et al., 2019; Forman et al., 2009; Gregory et al., 2007; Langley et al., 2010), there is a gap in the literature with regard to individual-level strategies that can address these barriers and facilitate change.

Prior studies have uncovered lack of time and lack of leader support as key determinants for implementation, and others have developed system-level strategies to address them. In two previous studies, the solutions proposed by investigators included inner-setting strategies focused on leaders (e.g., cultivating buy-in from principals prior to EBP implementation; Forman et al., 2009) and outer-setting strategies that redefined the role of school-based mental health clinicians to allow time for EBP implementation efforts (Langley et al., 2010). Importantly, even if these systems-level strategies were successful, school-based EBP implementation ultimately relies on teacher behavior change. Thus, tailored implementation strategies, that involve the front-line implementers such as teachers, are needed to address specific barriers and leverage facilitators in order to promote teacher adoption and delivery of EBPs. Precise implementation strategies that target key individual-level determinants are present in some service delivery settings, such as medicine (Steinmo et al., 2015). However, few individual implementation strategies have been specifically developed to address the barriers of perceived lack of time and leader support for the school context, where the ability to overcome implementation barriers has been particularly difficult (Gregory et al., 2007; Domitrovich et al., 2008). Therefore, there is a need to address common barriers that undermine successful implementation and ultimately place limits on whether EBPs can produce positive impacts on student outcomes when translated from research to real world school settings.

Purpose of the Current Study

The purpose of this study was to engage key stakeholder groups (i.e., teachers, administrators, and professional support staff including counselors, instructional coaches and school psychologists) in a mixed-method information gathering process to understand their perspectives regarding specific strategies to address time and supportive leadership as common barriers to EBP implementation. We used mixed methodology (quantitative and qualitative) to answer the following research questions:

1. Do key stakeholders perceive individual-level implementation strategies targeting two common implementation barriers as appropriate and effective?

2. Are there commonalities and differences across groups (teachers, administrators, support staff) or geographical settings (urban, suburban) in quantitative and qualitative data regarding the appropriateness and effectiveness of strategies to address the barriers of time and supportive leadership?

3. What recommendations do key stakeholders have to address time and supportive leadership as common barriers to EBP implementation?

Answering these research questions helps provide useful directions for future research and practice to design and deploy strategies that address common barriers that undermine successful implementation and ultimately place limits on whether EBPs can produce positive impacts on student outcomes when translated from research to real world school settings.

Methodology

Participants

The university institutional review board and each participating school district approved the study. A total of 39 key stakeholders from two school districts (one urban and one suburban) located in the northern Midwest United States participated in this study, including 15 teachers, 10 support staff (counselors, intervention specialists, school psychologists) and 14 school and central administrators. District administrators were asked to nominate participants working in elementary schools who could represent the opinions and attitudes of their colleagues. The urban district included a racially, linguistically, and economically diverse student population, with 28% of students with limited English proficiency and 66% receiving free/reduced price lunch. The suburban district was primarily White (77%) with 3.3% and 11% of students with limited English proficiency and receiving free and reduced priced lunch, respectively. Overall, focus group participants included 20 females (80%) and 5 males (20%). The average number of years in their current role was 19.70 (SD = 8.31).

Procedures

Data were collected during the 2018-2019 school year. Research staff collaborated with district administrators to determine the day, time, and location of the focus group sessions. In total, two focus group sessions were hosted to provide a convenient location for the two partnering school districts. The focus groups were scheduled for 90 minutes after school. Consistent with recommendations for conducting focus groups, stakeholders were grouped by role (i.e., teacher, support staff, and administrator groups; Krueger & Casey, 2014). Separate, rather than mixed focus groups, were performed to compare responses across each of the stakeholder groups. This minimized potential effects with the inclusion of administrators and subordinates in the same group. Thus, across both school districts there were six focus group sessions in total -- two focus groups for each of the three stakeholder groups. The size of each focus group (6-9 participants) was consistent with recommended procedures for thematic saturation (Guest, et al., 2006).

Focus Group Organization and Content

Each focus group was led by a trained moderator (authors 3, 6, and 10) and an assistant moderator (authors 1, 2, and 4), who used quantitative ratings (detailed below) completed in real time by participants in reaction to presentation content delivered during the whole group. The beginning of the focus group involved exposing all stakeholders (i.e., teachers, support staff and administrators) to the same information, such as the need for a focus on preventive, universal, social-emotional and behavioral supports for elementary-aged students, the existence of a science-practice gap, and activities designed to address solutions to two common implementation barriers: time and supportive leadership.

Following the large group presentation, breakout focus groups were conducted separately with each of the three stakeholder roles. Each focus group was provided with proposed strategies to overcome both barriers. The strategies targeted two key components of theorized mechanisms of behavior change. (1) Social proofing messages were used to address social-norms. These strategies incorporated data to describe, normalize, acknowledge, and validate each barrier separately through the use of statistics highlighting the prevalence of each barrier among teachers nationwide as well as through testimonials of how others had overcome each barrier. For overcoming lack of leader support, a testimonial of an administrator who was once a teacher was provided as well as common misconceptions teachers have toward administrators that all aimed to build empathy toward leaders and recognize. (2) Practical, solution-focused skills were used to increase self-efficacy. For the time barrier, this included how to advocate to leadership for re-prioritizing activities and making own decisions about shifting priorities, engaging in collective and collaborative problem-solving, sharing knowledge, and cultivating social support through facilitated discussions). Other practical recommendations included going to the administrator with another colleague to advocate together, and collecting and providing data to corroborate evidence of the need, and clearly communicating what type of support is needed.

Measures

Demographic Information

Participant demographic data was collected, which included their school distinct, position, grade taught (if applicable), gender, and years of experience.

Quantitative Ratings of Appropriateness and Impact/Effectiveness

Participants completed ratings in real time during the presentation segments of the focus group to indicate their perceptions of the content to enhance teachers’ motivation and capacity to overcome lack of time and supportive leadership as barriers to EBP implementation. The quantitative results were based on the ratings provided by the focus group participants regarding the following four prompts: “To what extent do you agree that the strategies employed in this segment would… (1) be appropriate for teachers (2) positively impact attitudes (3) increase sense of social norms/expectations (4) increase self-efficacy (confidence).”

Appropriateness reflects perceptions of the suitability, fit, and relevance of the content/experience for use with teachers as part of real-world implementation efforts in schools (Proctor et al., 2011). Impact reflects the likelihood that the content/experience would influence the three mechanisms of behavior change derived from the Theory of Planned Behavior (attitudes, social norms/expectations, and self-efficacy; Ajzen, 1991) as it relates to teachers’ motivation to overcome lack of time and leader support as barriers to EBP implementation. Attitudes reflect a person’s favorable or unfavorable perceptions toward adopting and delivering universal EBPs targeting students’ social, emotional, and behavioral functioning. Social norms/expectations capture the degree to which a person perceives that others who are similar and respected engage in adopting and delivering universal EBPs targeting students’ social, emotional, and behavioral functioning. Self-efficacy refers to a person’s degree of confidence in their own ability to adopt and deliver universal EBPs targeting students’ social, emotional, and behavioral functioning. Differences in responses were compared between staff role and between geographic location for two segments: (a) Overcoming the Time Barrier and (b) Overcoming Lack of Supportive leadership. Ratings were on a 9-point scale with the following anchors: “strongly disagree” (1), “moderately disagree” (3), “neither agree nor disagree” (5), “moderately agree” (7), “strongly agree” (9).

Qualitative Focus Group Questions

Each of the three groups were asked to provide their reactions and recommendations in response to the presentation content on addressing lack of time and leader support as barriers to teacher EBP implementation. Focus group moderators used the same probing questions based on participants’ quantitative ratings to elicit qualitative responses around appropriateness and impact. A whip-around strategy was used where each participant took a turn to provide their qualitative responses to the following two probes: first, What appropriateness rating did you give for the segment of content focused on addressing the time barrier? If you gave it a high rating of 8 or 9, why? If you gave it a rating lower than 8, why and what recommendation do you have to increase your rating? and second; What impact rating did you give for the segment of content focused on addressing the time barrier? If you gave it a high rating of 8 or 9, why? If you gave it a rating lower than 8, why and what recommendation(s) do you have to increase your rating? This process was conducted for both content segments (time and leader support) delivered during the whole group session to elicit specific responses and recommendations from participants regarding their ideas to address these barriers to teachers’ EBP implementation. Focus group sessions were audio and video recorded for purposes of transcription and data analysis via a coding process. Thus, according to Palinkas and colleagues (2011), this study employed a QUAL + quan simultaneous approach in which the main objective was to gather participants’ ideas and recommendations about time and supportive leadership as barriers to teachers’ implementation of EBPs and quantitative ratings of each segment were analyzed secondarily.

Data Analytic Approach

Quantitative data were analyzed across both stakeholder groups (administrators, teachers, support staff) and school district location (urban, suburban) to capture ratings on perceived appropriateness, effectiveness, and impact on attitudes, self-efficacy, and social norms. Data were then analyzed using descriptive statistics of central tendency (mean) and dispersion (standard deviation). Means between stakeholder groups and locations were then compared using two-tailed between samples t-tests, and Hedge’s g effect sizes were calculated due to small and uneven samples between groups (Durlak, 2009; Ellis, 2010). Based on the context of this study, effect sizes may be interpreted as follows: 0.70 = small difference, 0.80 = moderate difference, and 0.90 and above indicate a large difference (Durlak, 2009).

For the qualitative analysis, focus group audio recordings were transcribed and uploaded to NVivo QSR 10 for data management. The quantitative constructs measuring appropriateness and impact guided the development of the coding scheme. The coding scheme was developed using a systematic, transparent, and iterative approach. First, trained coders independently coded two initial transcripts line-by-line to identify recurring codes. Second, coders met as a group to discuss recurring codes and developed a codebook using an integrated approach to coding as certain codes were conceptualized during the focus group protocol development (i.e., deductive approach) and other codes were developed through a close reading of the two transcripts (i.e., inductive approach; Bradley et al., 2007; Neale, 2016). Next, coders and principal investigators met to discuss and select common codes interpreted from the transcripts. The group collectively determined which codes were incorporated into the final codebook. Then, operational definitions of each code were documented as well as examples of the code from the data, as well as when to use and not use the code. The coding scheme was applied to the data to produce a descriptive analysis of each code; the coding scheme was then refined throughout the data analytic process (Bradley et al., 2007). Coders overlapped 20% of the selected transcripts to determine inter-rater reliability. Percent agreement and Cohen’s Kappa (k) were calculated; average percent agreement = 98.5%, and average k = .48 across all codes. Quantitative and qualitative data were mixed using an embedded approach for the purpose of expansion.

Results

Research Question 1: Do Key Stakeholders Perceive Individual-level Implementation Strategies Targeting Time and Support as Appropriate and Effective?

Segment 1: Overcoming Lack of Time

Measures of central tendency from the segment, “Overcoming the Time Barrier,” are displayed in Table 1. On average, participants rated this segment moderately-highly across probes (range M = 6.5—7.4, range SD = 1.1—1.4). Qualitatively, participants indicated that the segment was perceived as highly appropriate for teachers delivering evidence-based practices in classrooms, “I think everyone feels that time is thin, and we’re all stretched in a lot of different ways, so I felt like people would relate to it.” Most comments about the appropriateness of the segment were focused on time as a salient experience for all teachers in schools, “I think it captured the real world of teachers.” Many participants felt that the segment was effective due to its ability to acknowledge and normalize time as a barrier, engage in collective and collaborative problem-solving, share knowledge, and cultivate social support through facilitated discussions, “I just think it cues that, not only are we acknowledging it, but it also has a problem-solving component to it. I think any time when you’re providing an opportunity to do some problem-solving and generate solution-focused ideas around something that’s a common barrier, it leaves me with something I can contemplate and try to move forward with.”

Table 1.

Ratings of Segment 1: Lack of Time as a Barrier

Respondent	Appropriateness M (SD)	Impact M (SD)	Social Norms M (SD)	Self-Efficacy M (SD)
All (N = 39)	7.38 (1.44)	6.56 (1.12)	6.74 (1.23)	6.56 (1.19)
Teachers (n = 15)	7.13 (1.19)	6.13 (1.30)	6.73 (1.10)	6.80 (0.68)
Urban (n = 6)	6.50 (1.38)	5.83 (1.94)	5.83 (1.17)	6.50 (0.84)
Suburban (n = 9)	7.56 (0.88)	6.33 (0.71)	7.33 (0.50)	7.00 (0.50)
Support Staff (n = 10)	7.20 (1.99)	7.10 (0.88)	6.60 (1.26)	6.50 (1.51)
Urban (n = 6)	7.17 (2.14)	6.83 (0.41)	6.33 (1.21)	6.33 (1.21)
Suburban (n = 4)	7.25 (2.06)	7.50 (1.29)	7.00 (1.41)	6.75 (2.06)
Administrators (n = 14)	7.79 (1.25)	6.64 (0.93)	6.86 (1.41)	6.29 (1.38)
Urban (n = 7)	7.86 (0.38)	6.29 (0.95)	6.43 (1.62)	5.86 (1.35)
Suburban (n = 7)	7.71 (1.80)	7.00 (0.82)	7.29 (1.11)	6.71 (1.38)

Segment 2: Overcoming Perceived Lack of Leader Support

Measures of central tendency and dispersion for the segment, “Lack of Leader Support”, are displayed in Table 2. On average, participants rated this segment moderately-highly across probes (range M = 7.1—7.9, range SD = 1.1—1.2). Participants across stakeholder groups described the segment as favorable with regard to appropriateness, largely because they felt it provided actionable solutions, allowed teachers to consider the perspectives of their administrator, and gave teachers a chance to share each other’s experiences asking for help, “this [segment] showed that administrators are just people who are doing their job, and sometimes I think staff can expect them to be, you know, something other than that. It’s just a person; Normalize the person; go talk to the person; ask for what you need; treat them like a human, and they’re going to treat you like a human. They’re going to support you and they’re going to help you.”

Table 2.

Ratings of Segment 2: Lack of Supportive Leadership as a Barrier

Respondent	Appropriateness M (SD)	Impact M (SD)	Social Norms M (SD)	Self-Efficacy M (SD)
All (N = 39)	7.90 (1.14)	7.13 (1.26)	7.18 (1.14)	7.15 (1.20)
Teachers (n = 15)	7.53 (1.46)	7.27 (0.88)	7.27 (0.80)	7.20 (1.01)
Urban (n = 6)	6.83 (1.72)	6.83 (0.98)	7.67 (0.82)	7.17 (0.75)
Suburban (n = 9)	8.00 (1.12)	7.56 (0.73)	7.00 (0.71)	7.22 (1.20)
Support Staff (n = 10)	8.20 (0.79)	6.90 (1.73)	7.00 (1.49)	7.20 (1.40)
Urban (n = 6)	8.17 (0.75)	6.33 (1.86)	6.83 (1.94)	6.83 (1.72)
Suburban (n = 4)	8.25 (0.96)	7.75 (1.26)	7.25 (0.50)	7.75 (0.50)
Administrators (n = 14)	8.07 (0.92)	7.14 (1.29)	7.21 (1.25)	7.07 (1.33)
Urban (n = 7)	7.43 (0.79)	6.29 (1.11)	6.57 (1.40)	6.29 (1.38)
Suburban (n = 7)	8.71 (0.49)	8.00 (0.82)	7.86 (0.69)	7.86 (0.69)

Participants across stakeholder groups indicated that the segment would likely be effective, because it could build confidence and self-advocacy among teachers, “In a nutshell I think this segment is very relational and gives a lot of hope… This whole segment is about confidence and advocacy to yourself and so forth, so I think that it would probably work.” Others indicated that it would improve communication and create a sense of camaraderie among teachers and the school as a whole, “This would open up that communication channel to instill the idea that we are all in this together.” One final theme was that the action-oriented strategies would instill self-efficacy among teachers, leading to greater ownership, “the [segment] is really action-oriented versus the blame game, which sometimes we kind of just stumble and stop and do the ‘yeah, but…’ I think the choices and the examples with that really did give options for self-efficacy and to go ‘you know what? Yeah, I do have a stake in this. I am part of this’.”

Research Question 2: What Are the Commonalities and Differences Across Groups and Settings?

Segment 1: Overcoming Lack of Time

Group comparisons, t-test results, and effect sizes for both segments are displayed in Table 3. For this segment, there was a significant difference [t(23) = −2.2, p = .03, Hedge’s g = 0.81] between teachers (M = 6.1) and support staff (M = 7.1) at the aggregate district level regarding the impact of this segment on attitudes. Qualitatively, many teachers from the urban district indicated that they felt the segment could be effective if they were guaranteed that other responsibilities were removed by leaders to make room for implementing a new EBP, “Whatever it is, your immediate thought is, ‘how are you going to help me make it happen?’ The best way I know how to make it happen is take something away from me. Or--not minimize it, but make it really manageable.”

Table 3.

Group Comparisons of Ratings for Each Barrier

Barrier and Comparison Group	Appropriateness				Impact				Social Norms				Self-Efficacy
	df	t	p	g	df	t	p	g	df	t	p	g	df	t	p	g
Time Barrier
Teachers x SS	13	−0.1	.93	0.04	23	−2.2	.03*	0.81	24	0.3	.79	−0.11	24	0.6	.57	−0.27
SS x Admin	14	−0.8	.42	0.59	20	1.2	.23	−0.49	20	−0.5	.64	−0.19	28	0.4	.73	−0.14
Teachers x Admin	26	−1.4	.16	−0.21	25	−1.2	.23	0.52	25	−0.3	.79	0.10	19	1.3	.22	−0.47
Urban x Suburban	37	−0.7	.47	0.23	34	−1.4	.18	.43	32	−2.9	.01**	0.90	37	−1.7	.09	.54
Admin. Support Barrier
Teachers x SS	24	−1.2	.24	0.52	12	0.6	.54	−0.28	12	0.5	.61	−0.23	15	0.0	1	0.0
SS x Admin	21	0.4	.72	−0.14	16	−0.4	.71	0.16	17	−0.4	.72	0.15	19	0.2	.82	−0.09
Teachers x Admin	22	−1.5	.15	0.43	23	0.3	.77	−0.11	22	0.1	.89	−0.11	24	0.3	.77	−0.11
Urban x Suburban	34	−2.4	.02*	0.75	31	−3.6	.001***	1.14	27	−0.9	.36	0.30	32	−2.2	.04*	0.70

Note: SS = Support Staff (i.e., school psychologists, counselors, and instructional coaches);

^*= p < .05,

^**= p < .01,

^***= p, .001;

df = degrees of freedom; t = t-value, g = Hedge’s g

A subset of teachers across districts reported that some aspects of the segment could negatively influence effectiveness. Specifically, one teacher indicated that effectiveness could be compromised by teacher attendees who speak out negatively during activities “What’s the group like that you are with? Do they come in all sour? Are they receptive to trying something new? Because that affects not only the attitude of the group, but my attitude. If everyone’s negative, it brings me down.” One final concern regarding impact among teachers was knowing that the quality of the EBP they were about to be trained in was high, “I think it depends on what it is we’re learning. If it’s a behavior thing, I am willing to put the time in up front, because then in the end you’re dealing less with behavior and you’re doing more teaching. So, it really depends on what the product is.”

There was also a significant difference [t(32) = −2.9, p = .008, Hedge’s g = 0.90] between aggregated urban (M = 6.2) and suburban (M = 7.3) participants regarding the overall impact on social norms. Highlighting this effect, one administrator from a suburban school noted, “I think alone, [reflecting on time as a barrier] is not going to give you implementation or any action. I think it’s still [problem] admiration. So, yes it can be good, and it gets you to think. I think you need that coupled with teachers getting together and saying that ‘oh, yes this is something common’ and, ‘hey, I figured this out, and I managed my time better’.” Adding to this, an urban school district participant indicated that the ability of the facilitator to guard against negativity, re-frame negative comments into opportunities for proactive problem solving, and restore a positive training climate was important to overall effectiveness, “It has to be very carefully facilitated in order to increase that positive attitude, which obviously increases self-efficacy. So, that’s why I am in the middle there. It really depends.” Although both administrators and support staff from the urban district indicated that the effectiveness would depend on the qualities of the facilitator, this concern was not directly brought up by participants in the suburban district.

Segment 2: Overcoming Perceived Lack of Supportive leadership

Results of group comparisons from this segment indicated that there were no significant differences in terms of role across the four prompts (Table 3). There were significant differences between urban and suburban district participants regarding appropriateness (suburban M = 8.3, urban M = 7.5; t(34) = −2.4, p = .022, Hedge’s g = 0.75) impact on attitudes, (suburban M = 7.8, urban M = 6.5, t(31) = −3.6, p = .001, Hedge’s g = 1.14), and self-efficacy (suburban M = 7.6, urban M = 6.7, t(32) = −2.2, p = .035, Hedge’s g = 0.70).

Qualitatively, administrators in urban districts questioned whether lack of leader support was a universal problem, which may have impacted quantitative ratings, “I would be surprised if that was the experience of teachers in my building. I just may be naive, but I don’t think [lack of leader support]is a universal experience -- or it’s experienced differently.” A teacher from the urban district also indicated that the problem is not that of the teacher, but is a skill deficit among administrators that should be targeted. Another teacher indicated that, “inevitably, I feel like teacher-to-teacher support is way more valuable than supportive leadership.” These views are juxtaposed with suburban administrators who believed the segment was important and would be helpful for all teachers by validating feelings, sharing perspective, and the providing advocacy skill: “I think it’s wholly appropriate for helping them learn our perspective. They don’t always know what’s on our plates, and, so, how do we broaden that perspective? But I think it’s totally appropriate, and it does go both ways.”

A subset of urban administrators shared that this segment’s effectiveness depended on whether or not the teacher had an existing trusting relationship with their administrator, “I think, that if a teacher went through this segment and got ideas and had a strategy, and then went to their administrator, and it was shut down for whatever reason, that would be this like negative feedback and would lead back into this like, hands up in the air, ‘well, I tried and I did the strategy, and I thought it out, and then it backfired’.”

Support staff, however, indicated that the appropriateness of this segment could depend on the quality of existing relationships between staff and administrators, including existing feelings of safety and trust, “The level of trust that the individual has with the administrator would make a big difference. If there isn’t the trust there, I think it would be really hard to engage in that conversation and believe that things might happen.”

Research Question 3: Recommendations

Segment 1: Overcoming Lack of Time

Participants from both locations and groups provided recommendations to improve the appropriateness and effectiveness of the “Time Barrier” segment. Participants indicated that it would be helpful to break implementation activities down in order to allow participants to “start small” and work up to full fidelity, particularly for multi-component interventions. Participants indicated that they felt this would reduce feelings of being overwhelmed following EBP training, which contribute to non-adoption or suboptimal implementation and sustainment, and promote feelings of self-efficacy associated with implementation behavior, “…just baby steps, where you don’t feel like you have to implement this [large gesture] from the beginning but you can implement this [small gesture], and this will be good for now.”

Participants also provided recommendations regarding placing the onus for change on building- and district-level leadership. For example, teachers working in the suburban districts noted that they felt they needed permission from building-level leadership to redistribute their time or teaching priorities to accommodate for new practices (e.g., “there’s a lot added to the plate that it’s not an option for us to take off,” and “I think we’re all feeling powerless in controlling that, and I don’t know where that power will come from.” This was corroborated by administrators working in the suburban district, “Sometimes they just need that permission to know that they can go outside the box.” To remedy this factor, participants indicated that focusing on what teachers can control and empowering them with strategies to re-prioritize their time effectively, such as action and coping planning, would increase effectiveness, “…we need to empower [teachers] with more tools for how to manage time and how to give yourself permission to prioritize differently.”

Similarly, comments from teachers working in the urban district indicated that their time was largely dictated by upper-level leadership and that they had relatively little ability to re-prioritize required activities. They also indicated that they lacked trust in administration to sustain an effort due to the large number of initiatives started and forgotten, which lowered perceptions of potential effectiveness. These perceptions were corroborated by comments made by administrators working in the urban district (e.g., “There has to be stuff taken off the plate, or concern for that, and reaching out to teachers and saying ‘what can we do to make it better’ instead of saying just ‘ you need to find more time’.” To remedy this factor, participants indicated that outlining an overall vision of initiatives to depict those that are most important for continued improvement, and to show that leaders are committed to a strategic effort.

Segment 2: Overcoming Perceived Lack of Supportive leadership

Participants from both locations and groups provided recommendations to improve the appropriateness and effectiveness of this segment. Some recommended beginning the segment with the testimonial of an administrator who was a former teacher to begin developing empathy and open-mindedness toward later strategies for overcoming this barrier. Another recommended including a testimonial of a person who struggled with a lack of supportive leadership and then steps they took to overcome it. Others recommended broadening the definition of who an administrator or leader is and providing strategies for identifying other means of support – or else providing indirect strategies for influencing a professional relationship. It was argued that, inevitably, someone will come in who has a negative relationship with their principal despite an unsuccessful history of attempts to repair it directly. Both teacher and administrator participants indicated that they felt a shared experience facilitated between teachers and their administrators would help build more effective working relationships and improve the segment’s effectiveness. Participants recommended including administrators in some portion of the training and being informed that teachers were being encouraged to seek support from them would address this recommendation.

Discussion

Staff rated and reported that practical, solution-focused strategies targeting social norms and self-efficacy would be appropriate, acceptable, and feasible for teachers and impactful in reducing barriers of time and unsupportive leadership, which aligns with extant literature and theory (Ajzen, 1991; Knapp et al., 2010). Teachers, administrators, and support staff did not significantly differ regarding their ratings, and agreed that focusing on practical skills that gave teachers increased autonomy over their priorities would be beneficial. The extent literature also indicates that achieving this requires open communication on the part of administrators and advocacy skills on the part of teachers (Ismail & Mydin, 2019; Smylie & Eckert, 2018; Pearson & Moomaw, 2005).

Our results indicated differences in perceptions between suburban and urban school districts. This is supported by previous literature, which shows differences in the experience of teachers as well as major differences in organizational structure across rural, urban, and suburban districts: where urban districts report more oversight and strict hierarchical structures and rural schools employ more distributed leadership and less formal leadership structures (Eckert, 2018; Martin et al., 2019).

Recommendations for improvements included providing protected time and space to allow school staff to collectively engage in facilitated problem-solving for overcoming common implementation barriers may be an effective strategy in itself to reduce barriers and spur change. Protected time for teaming and collaboration has been shown to improve implementation of EBPs in schools (Kittleman et al., 2021). It was cautioned by some participants, however, that the qualities of the facilitator were crucial to its success as well as the nature of the EBP being implemented. These recommendations align with other literature indicating the importance of trainer and facilitator qualities (Larson et al., 2022).

Limitations

Overall, findings from this study should be interpreted in light of several limitations. First, we acknowledge the small sample size, significant homogeneity of the participant sample and lack of key demographic variables, such as race/ethnicity and level of education. In addition, our sample included only schools from suburban and urban districts. Rural districts make up roughly half of school districts in the United States (NCES, 2016) and have unique barriers that likely warrant contextual tailoring (Gottfredson & Gottfredson, 2002; Lavalley, 2018). Our limited sample size also prevented us from analyzing disaggregated, within-district differences in quantitative ratings among staff.

Further, we are aware of the influence we as researchers have on the people and topic being studied, and simultaneously recognizing how the research experience is affected by the backgrounds, beliefs, and biases of our collective team and individual team members (Gilgun, 2008). As our analyses were carried out by exclusively White members of our research team from largely middle-class backgrounds, we recognize potential limitations in the representativeness of our interpretations. Lastly, we studied two barriers to implementation that, in our current research and in previous studies, are influential and prevalent in schools; however, there are most certainly additional barriers to implementation that warrant further investigation and may be present at different levels of urgency in different settings.

Future Research

The findings and limitations of the current study warrant further investigation. First, the “urban/suburban divide” found in this study shows how implementation in urban districts may be uniquely undermined by the influence of outer contextual factors (e.g., unions, district level leadership). This may be even more exacerbated in schools not included in this study that serve traditionally marginalized populations receiving increased levels of federal and state subsidies based on population needs. These types of supports often come with even higher levels of oversight and less autonomy in decision-making. Research indicates that this lack of perceived autonomy in highly hierarchical organizations can actually impede motivation at the staff level and lead to poorer work quality and productivity (Lavalley, 2018). Thus, it may be important for leaders at the outer context level (e.g., district-level leaders) to prioritize relationship building strategies that balance oversight with support for autonomy and self-determination that can positively impact continuous improvement at the school level (Parker et al., 2018; Weston et al., 2018).

Findings indicate a continued need to develop and tailor strategies based on organizational and population differences, which is echoed in implementation research across disciplines (Powell et al., 2017). Additionally, researchers should explore other barriers that may be more population or context specific, such as those that might be unique to rural school settings, which were not explored in this study, or nuances of how the barriers investigated in this study manifest uniquely in different settings.

Implications for Research and Practice

Findings from this study have implications that compel researchers and practitioners to incorporate implementation science methodology into routine use in schools. When implementing a new practice in schools, our results indicate that it is important for school leaders, in partnership with purveyors or intermediaries, to assess teacher buy-in and engage them throughout the implementation process. Teachers indicated that the strategies addressed in this study depended on which EBP was being implemented. Therefore, involving teachers in the decision-making process when selecting an EBP is recommended. This would include assessing for factors, such as perceived need of a new practice, attitudes toward the proposed innovation to be implemented among staff, the extent to which implementers will be supported, and organizational health and readiness prior to implementing a new practice in a school system. Practically, utilizing an implementation science model or framework, such as the Exploration, Preparation, Sustainment model (Aarons et al., 2011), could help facilitate this process as it provides important multilevel factors to consider at various phases of the implementation process.

Our findings regarding differences in perceptions depending on urbanicity may warrant strategic tailoring of implementation strategies to suburban and urban districts, as one-size-fits-all approaches are likely to be ineffective due to the significant differences in organizational structure. For example, differences in hierarchical leadership structures were staunchly apparent and had a marked influence on teachers’ perceptions of leadership support as well as their perceptions of the level of autonomy they felt. School staff across roles acknowledged that lack of time and supportive leadership continue to pervade school contexts and inhibit implementation of evidence-based practices. Evidence from this study suggests that targeting these determinants by developing and testing implementation strategies in a variety of contexts to reduce the impact of these barriers. Teacher responses showed that providing protected space for teachers, support staff, and administrators to acknowledge common barriers, and providing a facilitated, solution generation session, may boost self-efficacy and improve climate, which is a known facilitator of increased implementation outcomes (Authors, in press).

Incorporating implementation science in the rollout of innovations in schools is also an implication for future intervention research and among EBP purveyors and intermediary groups. If intervention researchers hope to replicate outcomes in effectiveness trials, it is important to ensure system readiness to implement and having a strategic implementation plan. Any implementation effort requires behavior change among front-line practitioners, which is a complex process unique to each individual; however, a wide variety of models, theories, and frameworks exist to guide this process and reduce barriers prior to and during implementation of a new practice (Michie et al., 2011; Moullin et al., 2019; Schwarzer, 2008). Additionally, hybrid designs can be utilized to not only test the effects of the innovation on client outcomes, but also the effect of an implementation strategy on implementation outcomes (Curran, 2012).

Conclusion

The phenomena of having “too much on one’s plate” and “not enough support” continue to be salient barriers for teachers who are regularly asked to incorporate practice changes. These, along with other factors can lead to burn-out, diminished implementation outcomes, and fewer students accessing EBPs, which impacts their outcomes. Other unmentioned barriers, such as turnover, cutbacks, and under-compensation make change initiatives challenging and widen the research-practice gap in schools. Findings from this study indicate that (a) addressing these barriers and normalizing them, (b) validating teacher experiences, (c) providing space for collective problem-solving and idea sharing among staff, and (d) providing practical strategies aimed at increasing individual-level determinants of behavior change, such as self-efficacy and perceived norms, were rated as overall moderately-to-highly appropriate and effective across participants. Incorporating similar individual-level strategies to overcome these common barriers are important prior to and during any change initiative.