An Integrated Approach to Universal Prevention: Independent and Combined Effects of PBIS and SEL on Youths’ Mental Health

Abstract

Mental health among children and adolescents is a growing national concern and schools have taken center stage in efforts to prevent problems and promote wellness. Although research and policymakers support the integration of mental health services into the schools, there is limited agreement on the ways to package or combine existing supports to achieve prevention-oriented goals. Positive Behavioral Interventions and Supports (PBIS) and Social Emotional Learning (SEL) are two of the most widely-adopted, evidence-based approaches that have been advocated to address student mental health. These universal prevention approaches, however, stem from different theoretical camps and are often advocated and implemented apart from one another. The purpose of this study was to examine the independent and combined effects of PBIS and SEL on student mental health outcomes. A quasi-randomized control design at the classroom-level was used to make comparisons across four conditions: business-as-usual (BAU), PBIS alone, SEL alone, and COMBO condition with regard to their acceptability to teachers, integrity of program delivery, and student outcomes. As predicted, the COMBO condition produced significantly greater improvements in overall mental health and reductions in externalizing behaviors when compared to all other conditions. The results also indicated that the PBIS and SEL only conditions were both able to produce significant improvements in overall mental health functioning as compared to the BAU control. The implications of an integrated approach for school-based universal prevention and directions for future research are discussed.

Mental health among children and adolescents is a growing national concern given the prevalence rates of mental health disorders and the costs to society when young people transition into adulthood. Research indicates that roughly 1 out of 5 children have diagnosable mental health disorders (Costello, Mustillo, Erkanli, Keeler, & Angold, 2003; Hoagwood & Erwin, 1997), but that roughly 70% do not receive indicated services (Kataoka, Zhang, & Wells, 2002). Furthermore, those who do are often provided with inadequate care (Kazdin & Wassell, 2000; Rones & Hoagwood, 2000). These findings are troubling considering that youth mental health problems portend a variety of negative short- and long-terms outcomes. For example, mental health problems have been linked to lower academic performance (Brown et al., 2005; Roeser, Eccles, & Strobel, 1998), increased interpersonal problems (Cook et al., 2010), higher rates of school dropout and incarceration (Moore, 2009), and adult unemployment (Nielsen, Madsen, Bültmann, Christensen, Diderichsen & Rugulies, 2010).

Schools represent the most common setting in which both mental illness prevention and mental wellness promotion programs are delivered and, more generally, are widely considered to be the de facto mental health service setting for youth (Burns et al., 1995; Costello et al., 2003; Leaf et al., 1996; Zahner, Pawelkiewicz, DeFrancesco, & Adnopoz, 1992). As a result, there is increased pressure for schools to adopt programs and practices that address youths’ mental health, ensuring that all students have the competencies necessary for succeeding socially, emotionally, and academically (Adelman & Taylor, 2006; Wagner et al., 2006, Vander Stoep, et al., 2003). Indeed, federal reports (e.g., U.S. Surgeon General’s Report 1999; President’s New Freedom Commission on Mental Health, 2002) and legislation (Individuals with Disabilities Improvement Education Act of 2004) have identified the promotion of student mental health as one of the top priorities of elementary and secondary schools. In response to these calls for action, research has identified several evidence-based practices that can be applied in schools to prevent or remediate mental health problems and promote wellbeing and academic success (Cornell, Allen, & Fan, 2012; Durlak et al., 2011; Horner et al., 2009). However, the school infrastructure for addressing these needs is often fragmented (Adelman & Taylor, 2007; Domitrovich, 2008) and the quality and effectiveness of practices targeting these areas needs to be strengthened (Evans & Weist, 2004; Rones & Hoagwood, 2000; Wilson et al., 2001). Despite these concerns with the current and long-term viability of school mental health practices and programs, each represents an opportunity for refining such practices to help better address the mental health needs of youth in the schools. In order to capitalize on this opportunity, schools need efficient, effective, and socially-valid organizational frameworks for integrating mental health services with preexisting academic practices.

Multi-tiered Systems of Support

Numerous researchers have embraced and advocated for the use of multi-tiered systems of support (MTSS) as a way to efficiently and effectively organize and deliver a continuum of school mental health services (Cook, Burns, Browning-Wright, & Gresham, 2010; Doll & Cummings, 2008). MTSS represents a service delivery framework grounded in the public health model of prevention and consists of providing a continuum of evidence-based practices and making data-driven decisions. The aims of MTSS are to prevent, reverse, and minimize mental health problems while promoting social, emotional, and academic success among all individuals in a school (Strein, Hoagwood, & Cohn, 2003). The foundation of MTSS is the universal level of support, which entails the delivery of evidence-based programs and practices to all students in order to prevent the emergence of mental health problems and promote social, emotional, and academic success (Rones & Hoagwood, 2000; Walker, 1996).

Given the importance of the universal level of support, researchers and organizations have developed programs and practices for implementation within school settings targeting mental health or behavior (Catalano, Berglund, Ryan, Lonczak, & Hawkins, 2002; Gottfredson &Wilson, 2003; Wilson & Lipsey, 2007). While many support the goal of integrating mental health services into the schools, there is limited agreement on the ways to package or integrate these supports to achieve prevention-oriented goals (Atkins, Hoagwood, Kutash, & Seidman, 2010). As a result, educational leaders are confronted with difficulties in deciding which programs and practices to integrate to create a comprehensive system of social, emotional, behavioral, and academic supports.

One aspect that makes it challenging is the fact that universal programs are predicated upon different theoretical frameworks, that target different social, emotional, or behavioral outcomes, and that emphasize different intervention components (Guerra & Bradshaw, 2008). Indeed, a cursory review of the school-based literature reveals that nearly all of the existing universal programs have been researched and implemented as standalone programs (Rones & Hoagwood, 2000). A singular or standalone approach to universal prevention, however, may be shortsighted if the goal is to address the diverse mental health needs of students. At the same time, adopting a program-for-every-problem approach (e.g., bully prevention program, substance use prevention program, suicide prevention program, child maltreatment, violence prevention) to universal prevention is untenable because it causes system overload and minimizes sustainability (Fixsen, Blasé, Naoom, & Wallace, 2009). Instead, a more promising and potentially effective approach is to develop an integrated universal support system that provides broad-spectrum coverage of a wider range of mental health targets that are relevant to students’ academic and life success (Domitrovich, Bradshaw, Greenberg, Embry, Poduska & Ialongo, 2010; Osher & Fleischman, 2005). However, to date, there are only a few studies that have investigated an integrated approach to universal prevention (e.g., Domitrovich, Ialongo, Embry, & Greenberg, 2008; Reinke, Herman, & Ialongo, 2012).

Integrated Universal Prevention

The theoretical and practical differences that exist among universal programs have the potential to be advantageous when it comes to combining intervention ingredients and theoretical perspectives that offer a more comprehensive, complementary approach to universal prevention. In light of the shortcomings of standalone approaches to universal prevention, Domotrovich et al. (2010) called for the development and evaluation of integrated models of school-based prevention. An integrated model consists of merging different independent universal programs or practices which target various risk and protective factors into a single program that has the potential to produce better outcomes than could either program implemented alone. This approach provides greater theoretical breadth, is characterized by a well-rounded repertoire of skill development, and draws from the strengths of each of the individual models to facilitate potential synergistic effects. Integrated approaches may also serve to address some of the resource limitations encountered when implementing more intensive secondary and tertiary interventions for students with chronic and/or severe emotional and behavioral problems (Domitrovich et al., 2010). Therefore, when integrating universal supports, it is important that they are complementary, rather than redundant, to ensure that program components are not unnecessarily burdensome to the school system (Fixsen et al., 2009). Although the extant literature is spare, there are a few examples of empirical studies examining integrated approaches to universal prevention.

One such study comes from the research conducted on the integration of the Promoting Alternative Thinking Skills (PATHS; Greenberg, Kusche, Cook, & Quamma, 1995) social emotional curriculum with the PAX Good Behavior Game (PAX GBG; Embry, Staatemeier, Richardson, Lauger, & Mitich, 2003) classroom management program as a horizontally (that is, within a particular level of supports) integrated approach to universal prevention (Domitrovich, Bradshaw, Greenberg, Embry, Poduska, & Ialongo, 2010). Results have indicated that the combination significantly improved student outcomes, with the PAX GBG facilitating increased engagement and appropriate student behavior and PATHS targeting the acquisition of social-emotional skills. A similar study was conducted by Reinke et al. (2012) who examined the feasibility and impact of combining the Classroom Check-Up, which is a teacher-directed coaching model based on motivational interviewing that enhances delivery of classroom practices, with the PATHS to PAX integrated model of universal prevention. Findings from this study supported the feasibility and impact of this triply integrated approach, suggesting that teachers significantly improved the quality of implementation of classroom management practices and the delivery of the PATHS curriculum—both of which have been linked to a range of student outcomes. Although there is an emerging body of promising research on integrated models of universal prevention, there is a paucity of research in this area.

Based on the integrated approach advocated by Domitrovich et al. (2010), the current study sought to examine the independent and combined effects of two widely used school-based universal programs when implemented at the classroom-level. Specifically, this study focused on Positive Behavioral Interventions and Supports (PBIS) and Social-Emotional Learning curricula (SEL), which are widely implemented universal programs that possess standalone evidentiary support (Bradshaw, Mitchell, & Leaf, 2010; Durlak et al., 2010). These two approaches have traditionally been discussed, implemented, and evaluated apart from one another. However, researchers have suggested that these two approaches are not antithetical to one another (Bear, 2010; Osher, Bear, Sprague & Doyle, 2010; Bradshaw, Mitchell & Leaf, 2010). Instead, each of the approaches offers unique theoretical underpinnings and practices that complement the other and potentially produce synergistic effects. For example, as Osher et al. (2010) explicated, while PBIS emphasizes more of a teacher-centered approach that focuses on extrinsic rules and use of positive reinforcement to prevent problems and manage behavior, SEL emphasizes more of a student-centered approach in that it teaches students skills to regulate their own actions towards self and others. Moreover, the integration of PBIS and SEL makes logical sense on at-least three levels. First, they both focus on the prevention of problems that interfere with academic success and the promotion of positive skills and environments. Second, they emphasize the value of positive approaches to students rather than punitive ones. Last, they put high value on the importance of teaching practices in order for students to learn the skills that will enable them to be socially and academically successful. In sum, implementing SEL and PBIS in tandem represents a horizontal integration of universal supports that potentially allows for the blending of complementary theories of change, greater exposure to a variety of preventive supports, and establishment of a more supportive and nurturing environment that empowers students to develop competencies that prevent mental health problems and promote wellbeing and academic success (Biglan, Flay, Embry, & Sandler, 2012).

SEL and PBIS Defined

SEL curricula are primarily derived from social-cognitive or cognitive-behavioral theories and focus on teaching skills that are the foundation for social competence and resilience, such as self-regulation, emotion management, empathy, interpersonal problem-solving, and future orientation (Zins, Bloodworth, Weissberg, & Walberg, 2004). A meta-analysis conducted by Durlak et al. (2011) demonstrated that SEL is linked to a range of beneficial outcomes, including improvements in social–emotional skills, attitudes, positive social behavior, conduct problems, emotional distress, and academic performance—noting an average increase of 11 percentile points on standardized academic measures. Although SEL has been shown to be an effective approach to universal prevention, drawbacks include its limited emphasis on teaching practices that promote orderly and productive learning environments in which students can acquire and then generalize skills from the curriculum (Gresham, 1995; Osher et al., 2010). PBIS, on the other hand, is grounded in applied behavior analysis and consists of teaching, modeling, cueing and reinforcing observable behaviors and developing a progressive system of systematically responding to problem behavior (Sugai & Horner, 2009). Like SEL, there are several studies that support the use of PBIS, particularly as it relates to the reduction of externalizing behaviors and promotion of more safe and orderly learning environments (Horner et al., 2009). However, to date, there is limited research examining the effects of PBIS to address internalizing problems (McIntosh, Ty, & Miller, 2014), whereas prior literature has shown that SEL programs are able to significantly reduce those symptoms in students (Merrell, Juskelis, Tran, & Buchanan, 2008).

While SEL consists of adopting a specific curriculum to deliver lessons that teach social, cognitive or emotional skills that help guide students’ decision making and behavior (e.g., Elias, Arnold, & Hussey, 2003), PBIS emphasizes the teaching of observable behavioral expectations to reduce problem behavior and altering aspects of the environment to create more safe, orderly and productive learning environments (McIntosh, Predy, Upreti, Hume, Turri, & Mathews, 2014). If integrated, these universal programs can enable students to learn a broader range of social, emotional, and behavioral skills that prevent mental health problems and promote student wellbeing and academic success (Bear, 2010; Osher et al., 2008).

Approaches to Integrating Multiple EBPs

Presently, there is growing emphasis on implementation across service sectors and disciplines (Eccles, Foy, Sales, Wensing, & Mittman, 2012), with many systems and organizations being faced with the prospect of implementing multiple EBPs simultaneously. The literature offers limited guidance on how to integrate different EBPs; however, there are different conceptual ways to approach the integration of multiple EBPs. One approach is to implement the interventions in a parallel manner by implementing the interventions side-by-side with limited to no attention paid to (a) theoretical linkages, (b) alterations to the core practices to remove redundancies, or (c) how certain practices from one EBP complement or enhance the other EBP. An alternative approach is to systematically blend the interventions by explicitly discussing the differences between the interventions, while also emphasizing the theoretical overlap between them and breaking down specific practices to remove redundancies and identify how specific practices from one EBP complement or enhance practices from the other.

For the purposes of this study, a blended approach was utilized in which specific points of difference between PBIS and SEL were emphasized (e.g., SEL is about delivering a curriculum while PBIS is about teaching and reinforcing observable behavioral expectations), as well as how PBIS practices could facilitate skills learned via the SEL curriculum (e.g., use of specific praise statements to promote desired behaviors) and likewise how the SEL skills could be linked to PBIS expectations (e.g., empathy helps us engage in respectful behaviors towards others). Additionally, explicit description of how the integration of PBIS and SEL provide a more comprehensive theoretical and practical approach to universal prevention was provided. Specifically, the combination of PBIS and SEL provides a more comprehensive cognitive-behavioral theoretical approach that entails creating a positive, orderly, and productive classroom setting in which students learn cognitive, social, emotional and behavioral skills and educators can utilize positive reinforcement techniques to promote the acquisition and maintenance of new skills and behaviors.

With all intervention research, it is important to examine the acceptability and feasibility of the intervention, because adoption and effective implementation of an intervention involves more than whether an intervention will produce results (Noell & Witt, 1999). Moreover, interventions found to be acceptable may not actually be implemented with integrity (Sanetti, & Kratochwill, 2009). Thus, the acceptability, feasiblity and integrity of the interventions should be assessed based on the implementers’ familiarity or direct experience with implementing different aspects of the intervention (Proctor et al., 2011).

Purpose of the Present Study

In light of the absence of research examining integrated approaches to universal prevention, the purpose of this study was to evaluate the impact, acceptability, and integrity of integrating SEL and PBIS on students’ mental health outcomes. Specifically, this study represented a participatory action research (PAR) evaluation, where a collaborative partnership was established between a school system and a research institution, and practitioners were involved in the research process from the initial design of study through data gathering and analyses to inform future actions for the school district (Nastasi, Varjas, Schensul, Silva, Schensul, & Ratnayake, 2000; Whyte, 1991). Utilizing the PAR approach, a quasi-randomized control study was conducted to evaluate the independent and combined effects of PBIS and SEL when implemented at the classroom-level for upper elementary students. The focus of this research was to examine the preventative effects of an integrated approach to universal prevention. The following research questions guided this study:

1. To what extent were the interventions found to be acceptable and implemented with adequate levels of integrity?

2. To what extent does the integration of PBIS and SEL produce significant reductions in negative mental health outcomes relative to the PBIS and SEL only conditions?

3. To what extent do the PBIS and SEL only conditions produce significant reductions in negative mental health outcomes compared to the business-as-usual control condition?

4. To what extent will there be differential effects between the PBIS and SEL only conditions?

We hypothesized that the integrated approach would result in the greatest reduction in negative mental health outcomes, while PBIS and SEL would have differential reductions in externalizing and internalizing problems, respectively.

Methods

Setting and Participants

This study took place in two large elementary schools located in the Southeastern Region of the United States. Given the PAR framework, district administrators were allowed to select the elementary schools, which was based on two criteria: (a) neither school was actively implementing universal practices to prevent mental health problems (e.g., PBIS or an SEL curriculum) and (b) both schools served a high proportion of economically disadvantaged youth (School 1 = 84% and School 2 = 91% free and reduced lunch). Also, the principals from the two participating schools were allowed to select the grades they wanted to participate in the study. To maintain consistency across the school sites, the principals were asked to work collaboratively to identify the same grades. Through this process, principals identified 4^th and 5^th grade classrooms based on two criteria: (a) perception that students in the classes exhibited social, emotional, and behavioral issues and (b) there was a need to improve the orderliness, productivity, and safety of the classroom environments. A total of eight 4th and 5th grade classrooms (four from each school) were identified to participate in this study, representing all the classrooms for each grade in each school. After identifying the eight classrooms, parental permission letters were sent out to parents and permission was obtained for all but two of the students in the 8 classrooms. This resulted in a total of 191 students who participated in this study. The demographic information of these students is depicted in Table 1. The average age of the participants was 9.8 years old (SD = 1.07) and 14.7% of the students were receiving special education services as part of an individualized education program.

Table 1.

Student Demographic Information for Each of the Participating Schools

	Gender		Ethnicity			FRL
	Male	Female	White	AA	Other	Yes	No
School 1 (n = 97)	49% (n = 48)	51% (n = 49)	82% (n = 80)	16% (n = 16)	2% (n = 4)	84% (n = 81)	16% (n = 16)
School 2 (n = 94)	52% (n = 49)	48% (n = 45)	22% (n = 21)	73% (n = 69)	5% (n = 4)	91% (n = 86)	9% (n = 8)

Note: AA = African American; FRL = Free and reduced lunch.

Teachers in the participating classrooms (n = 8) had an average of 8.6 years (SD = 3.12) of teaching experience. All but one of the teachers was female and six out of the eight identified as White, while the other two identified as African American. With regard to previous training, two of the eight teachers reported that they had taken a formalized course in behavior management during their university preparation, while the other six had not. None of the teachers had received specific training on implementing PBIS or delivering an SEL curriculum.

Procedures

As stated above, this work stemmed from a PAR framework. A problem-solving approach was used to communicate with district stakeholders to identify the problem to be addressed, analyze the identified problem, develop and implement a plan, and evaluate the plan. This resulted in the identification of students’ mental health issues interfering with academic success as the problem and the development of an integrated approach to preventing mental health problems that involved the combination of PBIS and SEL. Given the research questions guiding this study, a total of four treatment conditions were used to examine the isolated and combined effects of PBIS and SEL: (a) PBIS only, (b) SEL only, (c) PBIS-SEL combined (i.e., COMBO), and (d) business-as-usual control condition. Two classrooms were assigned to each condition to ensure comparable sample size across conditions, with conditions being equally spread across both of the schools. A matched quasi-randomized design was used to equate groups at baseline. Classrooms were matched into pairs according to pre-test data (overall mental health outcome measure; see below) and then each pair was randomly selected to undergo a quasi-random assignment procedure to one of the four treatment conditions. Each class within a pair was assigned to a different condition. Due to the number of classrooms and conditions, the first pair of classrooms had an equal probability of being assigned to one of the four conditions. However, once a condition was filled with two classrooms, then subsequent classrooms could not be assigned to this condition. As a result, not all classrooms had an equal probability of being assigned to every condition. Thus, this study employed a quasi-experimental procedure and not pure random assignment. The results of the matching process resulted in groups that were not statistically significantly different on baselines measures of mental health: internalizing F(3, 187) =.78, (p =.50) and externalizing F(3, 187) = .85, (p =.47).

The professional development training was delivered after the collection of baseline data. The professional development training was provided over the course of one day for the PBIS only and SEL only conditions and two days for the PBIS and SEL combined condition (see below for a more detailed description of the training). The trainings utilized a tell–show–do approach (Birman, Desimone,, Porter, & Garet, 2000) and “how-to” scripts were provided to teachers as reminders of key implementation components. PowerPoints of the training content with detailed note sections were provided to the teachers to use as a guide and basis for review. The site principals attended all the trainings to provide support to the teachers based on their respective condition. Teachers were provided with time to ask questions and develop necessary materials (e.g., posters cueing the expectations and/or SEL skill, setting by behavioral expectation matrix) to support the implementation of their respective universal program. For teachers in the COMBO condition, they received specific training in how PBIS and SEL practices are integrated as complementary yet distinctive approaches (i.e., discussion of theoretical differences and programmatic overlaps that combine to create a potentially more powerful approach to universal prevention). For example, they received training on how to use cueing and specific contingent praise (PBIS) to promote the acquisition and maintenance of SEL skills. Additionally, teachers were instructed that modeling and providing feedback about specific SEL skills (e.g., emotion management and interpersonal problem solving) can help students be linked to the established behavioral expectations (PBIS). After the in-service training, the teachers were allowed a week to review and develop materials. At the end of the week, a 90-minute follow session via Skype was held with the teachers to review content, answer any questions, and assess understanding using brief competency exam based on their assigned condition. Teachers could not fail the competency exam, because if they answered a particular question incorrectly, the first author continued to review the question using Socratic questioning techniques until they fully comprehended the answer. To avoid contamination effects across conditions, teachers were instructed to not share resources with other teachers and principals were also asked to help enforce this rule via supervision.

Only teacher-report measures of student mental health problems were used given that there were limited resources to conduct direct observations of student behavior and that teacher-report measures have been shown to be reliable and valid indicators of students’ mental health, both externalizing and internalizing (Youngstrom, Loeber, & Stouthamer-Loeber, 2000). Baseline data were collected four weeks into the beginning of the academic year to allow teachers to become familiar with their students and control for potential honeymoon effects. Immediately following baseline data collection, the professional development activities took place. Post-test data were collected five months after the baseline data collection to allow for sufficient time for the PBIS practices and the SEL curriculum to be implemented. During this time, there were two booster sessions conducted after school via Skype with participating teachers. The booster sessions were spaced one month apart and consisted of a review of key concepts, discussion regarding current implementation, and provision of feedback to improve future implementation. Moreover, participating teachers in the PBIS only, SEL only, and COMBO conditions were allowed to contact the first author if they had any questions regarding implementation.

Universal Prevention Programs

Social–Emotional Learning Curriculum

The Strong Kids social–emotional learning curriculum was used in this study. Strong Kids was designed to teach students social-emotional skills that promote emotion regulation, personal resilience, and interpersonal problem-solving (Merrell, Carrizales, Feuerborn, Gueldner, & Tran, 2007). The Strong Kids curriculum is used with students in grades 3 to 5, who are between the ages of 8 and 12. Each lesson takes 40 to 50 minutes to implement and is semi-scripted and outlined in an easy-to-follow manualized format. Lessons have been developed to be implemented once a week with corresponding generalization practices used throughout the remainder of the week (e.g., pre-teaching, praise students when they use skill outside of the lesson, daily brief review of learned skill). The curriculum requires minimal specialized training to administer, and the manual includes specific guidance for becoming proficient in delivering it. The 12 lessons included in the Strong Kids are as follows: (1) Emotional Strength Training, (2) Understanding Your Feelings: Part I (3) Understanding Your Feelings: Part II, (4) Dealing With Anger, (5) Understanding Other People’s Feelings, (6) Clear Thinking: Part I (7) Clear Thinking: Part II, (8) The Power of Positive Thinking, (9) Solving People Problems, (10) Letting Go of Stress and How to identify stressors, (11) Behavior Change: Setting Goals and Staying Active, and (12) Finishing UP. Previous research has demonstrated that The Strong Kids curriculum is effective for addressing externalizing and internalizing problems among elementary students (Marchant, Brown, Caldarella, & Young, 2010a, 2010b).

Positive Behavioral Interventions and Supports

The PBIS model used in this study was adapted from the universal supports included in the BEST Behavior approach to PBIS (Sprague & Golly, 2004). The primary components of this approach consist of establishing 3 to 5 core behavioral expectations, developing a progressive method of responding to problem behavior, ongoing teaching and modeling of the expectations, cueing the use of the expectations via posters and signals, and reinforcement of students when they exhibit the behavioral expectations through the use of contingent praise and issuance of tickets that could be exchanged for items included in a class store.

The first step of the BEST approach involves establishing three common behavioral expectations that were positively stated, teachable, and memorable. The ubiquitous behavioral expectations of be safe (e.g., hands and feet to self, walk while in doors, gently hand things to others), be respectful (e.g., say nice things or nothing at all, follow adult directions the first time, listening while others are talking), and be responsible (e.g., be on time, have materials out and ready to learn, keep sitting area clean) were selected to ensure consistency across classrooms, with the anticipation of the school- and district-wide adoption of PBIS that was scheduled for the following academic year. The teachers were instructed to teach and review the behavioral expectations on a weekly basis using a tell–show–do approach.

The progressive method of responding to problem behavior consists of defining problem behaviors as either minor (e.g., failure to have materials out and ready, talking to peers about non-academic task, not following directions, running in the classroom) or major (e.g., threatening to harm another student, physically hurting another student, destruction of classroom property, throwing dangerous object at another person) and then developing specific methods of responding to each category of behavior. For example, for minor problem behaviors, teachers were instructed to have students remain in the class and to use the following sequence of responses: proximity control, redirection tactic, prompt appropriate behavior, teaching interaction with delivery of a warning first and in-class disciplinary consequence second. Whether or not the student corrected his behavior determined whether the teacher would move onto the next step in the sequence of responses. For major problem behaviors, the teachers were instructed to calmly communicate with the student and complete an office support form that requested administrative involvement. The cueing system consisted of posters that were located in highly visible places in the classroom and developing a hand signal to remind students to exhibit the expectations. The reinforcement system consisted of a ticket system in which students could use tickets to purchase items, activities, and privileges from a class store one time per week. The teachers were instructed to hand out a minimum of 50 tickets per week based on students exhibiting the expectations.

Measures

Student Internalizing Behavior Screener (SIBS)

The SIBS (Cook et al., 2010) is a 7-item, teacher-completed measure of internalizing behavior problems. The items were derived from expert consensus regarding the key indicators of internalizing behavior patterns. The 7-items include: (1) Nervous/worried or fearful, (2) Bullied by peers, (3) Spends time alone, (4) Clings to adults, (5) Withdrawn, (6) Seems sad or unhappy, and (7) Complains about being sick or hurt. Each item is arranged on a four-point response scale to assess the frequency with which a teacher observers the target behaviors (i.e., 1 = Never, 2 = Rarely/Seldom, 3 = Occasionally/moderately, 4 = Frequently/almost always). Previous research has demonstrated that the SIBS possesses strong reliability, validity, and classification accuracy (Cook et al., 2010). The internal consistency reliability estimate for the current sample was α = .78.

Student Externalizing Behavior Screener (SEBS)

The SEBS (Cook, 2012) is a 7-item, brief measure of externalizing behavior problems, modeled after the Student Risk Screening Scale (Drummond, 1994). The 7-tiems include: (1) Defiant or oppositional to adults, (2) Fights or argues with peers, (3) Bullies others, (4) Gets angry easily, (5) Lies to get out of trouble, (6) Disrupts class activities, and (7) Has difficulty sitting still. Each item is arranged on the same four-point response scale as the SIBS (see above). Previous research has demonstrated that the SEBS possesses strong reliability, validity, and classification accuracy (Cook, 2012; Cook, Volpe, & Gresham, 2013). The internal consistency estimate for the current sample was α = .82.

Treatment Acceptability and Feasibility

Acceptability was measured using a modified version of the Intervention Rating Profile (IRP)-15 that included language specific to the interventions of interest in the present study. The IRP-15 was selected because it is widely used to assess teachers’ perceived acceptability and feasibility of interventions (Martens, Witt, Elliott, & Darveaux, 1985). All items are arranged on a six-point Likert-type scale ranging from “strongly disagree” to “strongly agree.” It was administered at the post-test data collection after the interventions had been implemented. Previous research has revealed that the IRP-15 has demonstrated reliability (α >.70) and validity evidence in support of scores (Lane et al. 2009).

Treatment Integrity

Treatment integrity data were collected via self-report checklists assessing the implementation of key components of PBIS and SEL. For the conditions involving implementation of the SEL curriculum, teachers completed a 4-item yes/no checklist after every lesson. The items included: (1) Did you deliver the lesson as it was planned/written this week?, (2) Did you refer to the skill targeted in the lesson throughout the week?, (3) Did you praise and recognize students when you caught them using the skill?, and (4) Did you give the students a heads up when they could use a skill? Teachers implementing PBIS completed a four-item yes/no checklist once a month (5 times total) during the study. The checklist included the following four items: Within the last month did you (1) teach/review the three behavioral expectations?, (2) deliver tickets to reinforce students for exhibiting the behavioral expectations?, (3) strive to maintain a 5 to 1 ratio of positive to negative comments/interactions?, (4) follow the progressive response system when responding to problem behavior?

Data Analytic Strategy

Two separate one-way, between-group analyses of variance (ANOVA) were conducted to explore differences in pretest–posttest change scores for internalizing, externalizing, and overall mental health problems among the four intervention groups (SEL, PBIS, COMBO, and BUA). Post-hoc follow-up comparisons were then completed on each of the measures to maintain the Type I error rate at an acceptable level. A Bonferroni adjustment was used to control the family-wise Type I error (i.e., separately for each ANOVA analysis), which reflected a total of six post-hoc comparisons (p ≤ .008). Given that statistical significance is not a direct indicator of the size of the effect and the preliminary nature of this study, the authors proceeded with conducting data analyses at the individual student level. Interpretations of results were grounded in a combination of significance findings and effect size (Cohen’s d) estimates, as recommended by the American Psychological Association (2010). Specifically, Cohen’s (1988) guidelines were used: small d > .2; medium d > .5, and large d > .8.

Results

Descriptive Statistics

Readers are referred to Table 2 for descriptive statistics concerning the outcomes for the SEBS and SIBS. Examination of the mean change scores for the SEBS indicated that the COMBO condition was associated with the greatest change from pre to post followed by the PBIS condition, SEL condition and finally, the BAU control. Finally, inspection of the mean change scores on the SIBS revealed the greatest change from pre to post on teachers’ ratings of internalizing behaviors was evidenced by the COMBO condition followed by the SEL condition and PBIS condition. The BAU condition demonstrated an increase in reported internalizing behaviors from pre to post.

Table 2.

Descriptive Statistics for SEBS and SIBS

	Pre		Post		Change Score
Condition	Means	Std. Dev.	Means	Std. Dev.	Means	Std. Dev.
SEL Only
SEBS	5.77	4.63	4.70	3.76	1.06	1.71
SIBS	3.06	3.99	2.47	3.29	.62	1.42
PBIS Only
SEBS	5.63	4.60	4.53	3.75	1.11	1.37
SIBS	3.15	3.85	3.06	3.50	.09	1.21
SEL/PBIS COMBO
SEBS	5.88	4.39	3.56	2.93	2.37	2.76
SIBS	3.04	4.13	1.76	2.60	1.25	2.29
Business-as-usual Control
SEBS	5.76	4.96	5.73	5.13	.02	1.12
SIBS	3.04	3.43	3.07	3.37	−.02	.84

SEL = Social emotional learning; PBIS = Positive behavior supports

Acceptability/Feasibility and Integrity

Acceptability and feasibility

Post intervention acceptability data were collected from teachers in the COMBO, SEL only, and PBIS only conditions and results demonstrated that the teachers perceived SEL and PBIS to be acceptable, feasible, and fair. Results of these data were inspected as an average across all items, as well as at the individual item level to examine the extent to which the teachers found the PBIS, SEL, and combined to be acceptable and feasible to implement. Specifically, teachers who implemented the SEL curriculum had a mean of 5.2 (min. 5.0 and max 5.4) on a scale to 6 and teachers who implemented the PBIS program had a mean rating of 5.5 (min. 5.1 and max. 5.9), suggesting that they perceived the SEL and PBIS programs to be acceptable, feasible, and fair to implement in their classroom. Of particular note are the two teachers who implemented both SEL and PBIS. The average mean of 5.3 (min. 5.0 and max. 5.6) indicated that both teachers found the combination of programs to be acceptable, feasible and fair.

Integrity

Results from the self-report evaluation of integrity indicated that PBIS only, SEL only, and COMBO were implemented with sufficient integrity (see Table 3). Whereas the PBIS only condition was associated with the highest level of integrity, the COMBO condition was associated with the lowest, but all interventions were implemented with greater than 80% integrity on average across the observation sessions.

Table 3.

Self-Report Treatment Integrity Data for COMBO, PBS only, and SEL only Conditions

	SEL Integrity Percentages					PBIS Integrity Percentages
Condition	Item 1	Item 2	Item 3	Item 4	Mean	Item 1	Item 2	Item 3	Item 4	Mean
COMBO	85%	85%	80%	75%	81%	100%	90%	80%	80%	85%
SEL Only	95%	95%	95%	85%	93%	--	--	--	--	--
PBIS Only	--	--	--	--	--	100%	100%	100%	90%	97%

Inferential Statistics

SEBS

Findings from the one-way ANOVA using change scores from the SEBS revealed a statistically significant main effect among the four conditions (F (3, 187) = 12.209, p = <.01, η² = 0.16). The effect size associated with this analysis indicated that 16% of the variance in the outcome measure was accounted for by the differences between the group means. Again, six pair-wise comparisons were performed as follow-up post-hoc analyses to the omnibus F test using the Bonferroni adjusted p-value of 0.008.

The results from the pair-wise comparisons are depicted in Table 4. Beginning first with the pair-wise comparisons between the COMBO condition and all other conditions, results revealed that the COMBO condition demonstrated significantly greater change from pre to post on the SEBS than all other conditions: SEL only condition, PBIS only condition, and BAU control. The standardized mean difference effect sizes were in the moderate to large range, indicating differences that would likely be noticeable. Significant differences were also found when comparing the BAU control to both the PBIS intervention and the SEL group. The obtained effect size for the difference between the PBIS and BAU conditions was large, while the difference between the SEL and BAU conditions was moderate. There was no statistically significant difference on externalizing behavior measures when comparing the SEL and PBIS treatment conditions; however, the mean change produced by the PBIS only condition was slightly larger than that of the SEL only condition.

Table 4.

Results from Post-Hoc Pair-wise Comparisons

Externalizing Comparison	t	p-value	df	Cohen’s d [95% CI]
SEL vs. PBIS	−0.13	0.89	92	0.03 [−.35, .39]
SEL vs. COMBO	−2.71	0.008*	97	0.57 [.12, 1.02]
SEL vs. BAU control	3.44	≤ .001*	90	0.72 [.43, 1.02]
PBIS vs. COMBO	−2.75	0.007*	97	0.58 [.14, 1.01]
PBIS vs. BAU control	4.15	≤ .001*	90	0.87 [.63, 1.13]
COMBO vs. BAU	5.26	≤ .001*	95	1.12 [.67, 1.52]

Internalizing Comparison	t	p-value	df	Cohen’s d [95% CI]
SEL vs. PBIS	1.95	0.054	92	0.40 [.14, .67]
SEL vs. COMBO	−1.64	0.100	96	0.33 [.05, .71]
SEL vs. BAU control	2.61	0.011	90	0.54 [.32, .79]
PBIS vs. COMBO	−3.12	0.002*	96	0.64 [.27, .99]
PBIS vs. BAU control	0.49	0.623	90	0.10 [−.11, .32]
COMBO vs. BAU	3.54	≤ .001*	94	0.74 [.38, 1.07]

^*Significant at p ≤ .001

SIBS

Findings from the one-way ANOVA using change scores from the SIBS revealed a statistically significant main effect among the four conditions F (3, 186) = 6.846, p = <.01, η² = 0.10. The effect size associated with this analysis indicated that 10% of the variance in the outcome measure was accounted for by the differences between the group means. Again, the significant main effect was followed up with six pair-wise using the Bonferroni adjusted p-value of 0.008.

The results from the pair-wise comparison are depicted in Table 4. Students who received the combined intervention (COMBO) demonstrated significantly greater change from pre to post than the PBIS only intervention and the BAU control. The standardized mean difference effect sizes were in the moderate range, indicating differences that would likely be noticeable by observers. While there was no statistically significant difference on the SIBS between in the SEL vs. PBIS, SEL vs. COMBO, and SEL vs. BAU control comparisons, examination of the magnitude of the effect sizes indicated that they all fell within the small to moderate range.

Discussion

Although significant strides have been made in the area of school-based mental research, there remains considerable room for researchers to develop and evaluate more integrated approaches to the prevention of students’ mental health problems. In this vein, the present study responded to Domitrovich et al.’s (2010) call for additional research to examine integrated models of school-based prevention by investigating the independent and combined effects of SEL and PBIS on students’ mental health outcomes. The aim of this research was to conduct a preliminary investigation examining whether the combination of these two popular programs provides an effective foundation of universal supports to integrate within a MTSS framework.

Interpretation of Findings

Consistent with predictions, findings provided promising support for the integration of SEL and PBIS. When examining the impact of the COMBO, SEL only, and PBIS only conditions on the measures of teacher ratings on externalizing and internalizing behaviors, the findings provided support for the utility of an integrated approach to address externalizing behaviors, specifically, as students in the COMBO condition evidenced significantly greater change from pre to post than the SEL and PBIS only conditions. The effect sizes for these significant differences were in the moderate range. Of particular note was the effect size for the significant difference between the COMBO and BAU conditions, which exceeded 1.0. An effect size of this magnitude indicates that someone unfamiliar with school practices would likely be able to observe students at pre and post and notice significant differences between the two groups with the naked eye (Cohen, 1988). Although students in the SEL and PBIS only conditions did not evidence as much change as the students in COMBO conditions, results indicated that they were associated with significantly greater change than students in the BAU condition. The effect sizes derived from these comparisons were all in the moderate to large range. The analysis comparing the SEL and PBIS only conditions did not result in a significant difference between the two.

Examination of the effect of the interventions on internalizing behaviors yielded results consistent with the literature on universal prevention programs. Park-Higgerson et al. (2008) argue that detecting an effect for a single-intervention model may be difficult with small sample sizes as these universal interventions have demonstrated modest effect sizes on outcomes to date, especially on internalizing behavioral outcomes (Greenberg, 2004; Weisz, Sandler, Durlak, & Anton, 2005). We too found fewer significant effects on internalizing behaviors, relative to externalizing problems. This could be due to the fact that students are often better reporters of internalizing problems (Cantwell, Lewinsohn, Rohde, & Seeley, 1997; Edelbrock, Costello, Dulcan, Conover, & Kala, 1986) and only teacher-report measures were used. Significant differences from pretest to posttest on the SIBS were only evidenced between COMBO classrooms and PBIS only and BAU conditions respectively (both with moderately large effect sizes), demonstrating that the COMBO intervention significantly improved internalizing behavior to a greater extent than the BAU and PBIS only conditions. While not statistically significant, it is important to consider that the effect size representing the COMBO vs. SEL only comparison was in the moderate range and was associated with an interaction effect that was approaching significance according to the standard .05 p value. Although discretion is warranted when interpreting this outcome, the magnitude of the effect between the COMBO and SEL only conditions was in favor of the integrated model of prevention.

Overall, results were generally straightforward in that one would predict that the combination of the two programs would be better than either one alone. However, this study is unique in that there is the paucity of research investigating integrated approaches to universal prevention (see Domitrovich et al., 2010; Reinke, Herman, & Ialongo, 2012). This research is important because it is quite reasonable to expect that the integration of two programs would not produce any additive benefits beyond either one alone and actually may exceed the capacity of a person or system to implement them both in tandem. Thus, research such as the present study can help guide educational decision-makers in knowing whether the integration of practices produces enhanced outcomes and can be reasonably implemented in the schools.

Implications for Practice

Results of this study contain many implications for school-based universal prevention practices within a school’s MTSS that targets mental health. First, our findings serve to provide additional support for the continued adoption and implementation of SEL and PBIS practices as standalone programs in schools as both demonstrated practical effects in improving students’ overall mental health relative to a BAU condition. Second, our initial findings suggest that MTSS models should include a more comprehensive, integrated framework of universal prevention by combining standalone programs or interventions. Our findings speak to the power of implementing a more comprehensive structure of universal supports by integrating PBIS and SEL interventions together using a blended approach by combining these two interventions both theoretically- and practically-speaking. In the current project, this combined approach appeared to produce additive effects on mental health outcomes including internalizing and externalizing behavior problems beyond changes that occur when implementing only one intervention. Additionally, the combined approach was implemented with acceptable levels of treatment integrity and teachers reported that it was feasible to implement and mutually beneficial for both teacher and students, demonstrating the realistic possibility of using this combined approach to universal prevention in the school setting. Future research should examine more closely the programmatic overlap of these two interventions to devise the most cost and time efficient approach to integrating practices to achieve prevention-oriented goals. In the case of PBIS and SEL, reinforcement practices tailored to promote skill acquisition and maintenance represent clear areas of programmatic overlap and should be considered to ensure continuity and alleviate redundancy.

It is important to recognize that the class-wide PBIS and SEL programs differ from school-wide adoptions in which common language and practices are adopted across all educators or teachers within a building. For example, school-wide PBIS is focused on teaching, modeling, cueing, and reinforcing expectations in all settings throughout the school by all staff, as well as the inclusion of ongoing data-based decision making by tracking disciplinary referrals (Sugai & Horner, 2005). Moreover, most SEL curricula, like Strong Kids, are supposed to be implemented for all students across each grade. As a result, the school-wide versions of PBIS and SEL may produce results that are different than those obtained in this study. Future research should explore the integration of school-wide models of universal prevention and gather data on the efficacy, acceptability, feasibility, and integrity of implementation.

Limitations and Future Directions

As with most preliminary studies, this study is not without its limitations. First, it is important to point out many of the limitations that come along with using PAR. As discussed by Wallerstein and Duran (2006), the idea of PAR is to bridge the gap between scientists and practitioners helping to bring research to life in a practical way. This is an important undertaking in the field of mental health as some studies show that it typically takes well over a decade to translate laboratory research into common practices implemented in the field (Balas & Boren, 2000).

According to the Institute of Educational Sciences (IES), the average cost for supporting a school-based randomized control trial is between $3 and $3.5 million (Slaven, 2008). As with our study, many community-based participatory research studies consist of cost-neutral partnerships between research organizations or universities and those putting research into practice in places like schools. Often times, there is not extensive funding for the type of research conducted in this study (Wallerstein & Duran, 2006) and conducing cost-neutral research does not allow for the extremely large scale studies necessary to operate with sufficient statistical power to detect significant effects. In contrast, routinely situating evaluations such as those described in the current paper within collaborative partnerships with existing agencies has the potential to advance the state of the science in ways likely to demonstrate a high degree of contextual fit and that would not otherwise be possible.

This study focused solely on mental health problems according to universal screening measures. Future research should examine the extent to which integrated approaches promote strengths or positive skills in students that are associated with wellbeing and enhance quality of life. This research would be consistent with the dual continua of mental health that has demonstrated that mental health exists on two dimensions: mental illness and wellbeing (Suldo & Shaffer, 2008). Moreover, this study only utilized teacher-reports of mental health problems and treatment acceptability and integrity. Multi-method, multi-informant approaches to collecting data and evaluating the efficacy, feasibility, and integrity of integrated approaches are needed to cross-validate these findings with other measures (e.g., direct observation) and other sources (e.g., parent-report or self-report).

While we recognize the limitations of our study given the small number of randomized units (i.e., classrooms) and the nesting of students within classrooms, the purpose of this study was to conduct a preliminary evaluation of an integrated approach to school-based prevention. The results should be replicated with a larger sample of classrooms or schools to take into account the nested structure of the data and cross-validate the findings. Future research should also attempt to generalize the results to more diverse settings and samples as well as different types of SEL curricula (e.g., Second Step; Committee for Children, 2011) and approaches to PBIS (e.g., Project Achieve; Knoff & Batsche, 1995). A study such as this could serve to establish the most effective combinations of PBIS and SEL. Although this study did not explicitly evaluate the costs associated with implementing the two approaches, future studies should examine the cost-effectiveness of the blended approach. It may be that there are cost savings and efficiencies associated with a blended, rather than a parallel, approach to integration. To best assess these costs, future research should also include implementation at a school-wide level.

This study supports the use of a PAR framework to conduct research that is practically important and offers value-added contributions to the scientific literature. While this study did not employ the most rigorous design possible, as other large scale grant funded randomized control trials, it serves to bridge the gap between research and practice in a cost-effective manner that has practical implications for educators, students and families and represents an important preliminary investigation to advance research forward. As evidenced by the findings of this study, implementing high quality EBP includes taking into account the diverse and complex mental health needs of students and working to design and implement systems of universal prevention that are comprehensive in nature. This involves interventions that draw from multiple theoretical orientations having the potential to impact both externalizing and internalizing behavioral outcomes. This study used a community-based research design to establish the potential of “marrying” two widely implemented interventions – SEL and PBIS – and demonstrating their synergetic effects on overall mental health functioning as well as internalizing and externalizing problems for students in a school setting. Considering the long-term implications of mental health for many key functional outcomes (e.g., academic success), continued investigation of the most effective universal prevention strategies and advocacy for the integration of high quality EBP are of the utmost importance.