Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2011 Jun 1.
Published in final edited form as: J Sch Violence. 2011 Jan 1;10(2):165–184. doi: 10.1080/15388220.2010.539167

Developing and Validating a New Classroom Climate Observation Assessment Tool

Stephen S Leff 1, Duane E Thomas 2, Edward S Shapiro 3, Brooke Paskewich 4, Kim Wilson 5, Beth Necowitz-Hoffman 6, Abbas F Jawad 7
PMCID: PMC3106103  NIHMSID: NIHMS270466  PMID: 21643447

Abstract

The climate of school classrooms, shaped by a combination of teacher practices and peer processes, is an important determinant for children’s psychosocial functioning and is a primary factor affecting bullying and victimization. Given that there are relatively few theoretically-grounded and validated assessment tools designed to measure the social climate of classrooms, our research team developed an observation tool through participatory action research (PAR). This article details how the assessment tool was designed and preliminarily validated in 18 third-, fourth-, and fifth-grade classrooms in a large urban public school district. The goals of this study are to illustrate the feasibility of a PAR paradigm in measurement development, ascertain the psychometric properties of the assessment tool, and determine associations with different indices of classroom levels of relational and physical aggression.

Keywords: classroom climate, teacher practices, behavioral observations, participatory action research, aggression prevention

Recent research has shown that the social climate of classrooms can have a significant bearing on students’ psychosocial functioning and adaptation during the elementary school years, including their social competence (Wilson, Pianta, & Stuhlman, 2007) or, alternatively, their levels of aggression towards peers (Thomas, Bierman, Thompson, Powers, & the Conduct Problems Prevention Research Group [CPPRG], 2008; Thomas, Bierman, Powers, & the CPPRG, in press). In addition, an increasing number of bullying prevention programs have recognized the importance of the classroom climate and have developed classroom-based components to improve teacher responsiveness to interpersonal aggression and conflict situations among students (Doll, Song, & Siemers, 2004; Olweus & Limber, 1999). Despite the growing emphasis on enhancing classroom climate, there are few validated, theoretically grounded assessment tools that have been designed specifically to assess the social quality of classrooms and to ascertain actual mechanisms within these settings associated with student aggression. This article describes the development of a classroom-based observation tool that was designed to provide a brief index of classroom climate. We discuss the rationale behind its development (particularly as it relates to challenging urban classroom settings), its current psychometric properties, and its utility for aggression and bullying prevention programming and teacher training.

Structured observations of student behaviors in the classroom are one of the most commonly used assessment methods by school-based practitioners (Shapiro & Heick, 2004). However, current observational practice has several limitations. First, professionals tend to rely upon informal and anecdotal observations as opposed to using systematic observational systems with predefined target behaviors (Volpe, DiPerna, Hintze, & Shapiro, 2005). Second, many classroom-based observation systems are often helpful in planning and monitoring the treatment of specific at-risk children (Volpe et al., 2005), but they do not generally provide information about the overall classroom environment, how teachers relate to their students, or the quality of the teaching and learning environment. The classroom observation assessment tool described in this article was developed through an extensive partnership-based process in an effort to address these gaps in the literature base and also to potentially serve as a useful and brief assessment tool for determining the impact of classroom-based prevention programming and/or teacher training.

The Importance of Teacher Practices and Classroom Management

Seminal studies related to effective teacher practices highlight the importance of better understanding the classroom climate to which students are exposed. For example, early research by Kounin (1970) found that effective and ineffective teachers were differentiated primarily by how they used strategies to prevent student misbehavior from occurring. Further, effective classroom management may also be related to teachers’ ability to use proactive and positive encouragement, to actively involve their students in learning, and to be organized in their classroom and teaching practices (Brophy, 1983). Additionally, over four decades of research has shown that teacher praise and reinforcement of appropriate student behaviors, while ignoring less appropriate behaviors, is associated with positive student behavioral outcomes in the classroom (e.g., Madsen, Becker, & Thomas, 1968; Swinson & Harrop, 2001). Finally, classrooms in which teachers provide more clearly stated, positively-oriented strategies combined with appropriate levels of student redirection, are associated with students who attain greater academic progress (Kern & Clemens, 2007).

The Influence of Teacher-Student Relationships

Research has found that classroom environments in which students have strong and supportive relationships with their teachers are associated with children’s positive peer relationships and social adjustment (Howes & Hamilton, 1993; Stuhlman & Pianta, 2001). Strong student-teacher relationships can also serve as a buffer against student behavioral problems (Meehan, Hughes, & Cavell, 2003; Stuhlman & Pianta, 2001). For instance, Meehan et al. (2003) showed that teacher-student relationships had a significant impact on the rates of aggressive behaviors, particularly among African American and Hispanic students. Thus, early positive relationships between students and teachers are associated with higher levels of school adjustment and the attenuation of negative student behavior outcomes.

Classroom Climate and Aggressive Behaviors

Accumulating evidence suggests that the social climate of classrooms, shaped by a combination of teacher practices and peer processes, is a primary factor affecting children’s development of aggression (Thomas, Bierman, & CCPRG, 2006; Thomas et al., 2008). For instance, Thomas et al. (2008) found that a positive classroom climate partially attenuated the impact of child cognitive and behavioral vulnerabilities on students’ behavioral adjustment during the first grade, whereas similarly vulnerable students placed in classrooms with more negative climate experience increased risk for the development of aggressive-disruptive behavior problems. Moreover, classroom climate exerted a unique and stronger influence on individual student aggressive behavior problems than more distal contextual risk factors at school, such as school poverty levels.

Negative classroom contexts, characterized by high concentrations of aggressive-disruptive peers, place children at considerable risk for elevated and persistent aggressive behavior problems in the school setting (Kellam, Ling, Merisca, Brown, & Ialongo, 1998; Thomas et al., 2006). For instance, children exposed in first grade to classmates who start fights, are disruptive, and generally get into trouble with teachers and other authority figures are vulnerable to displaying high levels of aggressive behaviors in classrooms six years later in middle school (Kellam et al., 1998). High levels of aggressive students may affect the degree to which teachers can exert positive control in a classroom, thereby resulting in the heavy use of inflexible classroom rules and a reliance on punishments to control student misbehavior. Such practices, particularly when used without positive support strategies, may fuel student-teacher conflict, creating classroom conditions that elicit and exacerbate aggressive student behaviors (CPPRG; 1999; Thomas et al., 2006).

Strengths of Observational Tools

Naturalistic direct observations are regarded as highly objective ways to evaluate children’s behavior in schools and are one of the most common assessment procedures used by school-based professionals (Shapiro & Heick, 2004). The majority of classroom-based observational assessment tools have been designed to be useful in the diagnosis of emotional and behavioral difficulties, to determine intervention effectiveness for specific target students, and/or to identify particular at-risk children and youth (Shapiro & Heick, 2004). The use of observational methodologies for better understanding the social ecology of the school classroom is another important, yet relatively understudied, area of research. Specifically, the use of observational technologies to simultaneously assess student behavior and naturally-occurring, teacher classroom-management practices shows promise in understanding factors that affect children’s behavior functioning in the learning environment (Greenwood, Carta, Kamps, Terry, & Delquadri, 1994). Educators and applied behavioral analysts refer to this approach as ecobehavioral assessment, and although most research in this area occurs during the preschool years (Greenwood et al., 1994), it provides an important heuristic for methodically identifying classroom situational factors that either promote or impede the occurrence of specific student behaviors during subsequent school years and grade levels.

Classroom Climate Observational Tools

As described, most observational assessment tools used within the elementary school classroom context have focused on individual student academic functioning or behavior. However, other investigations have utilized these methodologies to assess classroom climate in an effort to identify key features of social interactions in elementary-school classrooms that are associated with elevated levels of learning engagement among students, enhanced student social competence, and reduced behavior problems (see Emmer, 1971; Thomas et al., in press). The Multi-Option Observation System for Experimental Studies (MOOSES; Tapp, Wehby, & Ellis, 1995) and the ASKER (Tapp & Fiel, 1991) are two prevalent classroom climate observation systems used in prior studies. In the MOOSES system, classroom behaviors and activities of a target child are observed by trained observers. Specific codes relate to student engagement in classroom activities, teacher involvement in student activities, and teacher-child interactions. The primary limitation of the MOOSES is its exclusive focus on individual students.

The MOOSES has also been used in conjunction with the ASKER system (Tapp & Fiel, 1991), a computer-assisted rating tool that assesses the entire classroom atmosphere across several facets of student and teacher behaviors (e.g., level of disruption during academic time, level of cooperation amongst students, level of interest and enthusiasm). The ASKER system provides ratings on several critical dimensions of classroom climate and has demonstrated considerable utility in longitudinal investigations examining the impact of interventions in the classroom (see CPPRG, 1999). On the other hand, the ASKER system may be somewhat subjective given that it has been used as an impression rating instead of as an observation system in which direct classroom behaviors are observed and coded (CPPRG, 1999). In addition, it is unclear how the overall classroom climate ratings provided through the ASKER system may be impacted by completing them in the context of an individually-focused observation tool such as the MOOSES. Nevertheless, given that the ASKER system captures many of the foremost aspects of classroom climate, it served as a starting point in the development of the classroom observation assessment tool described in this paper.

The Classroom Assessment Scoring System (CLASS; Pianta, La Paro, & Hamre, 2008) is another recently developed and well-respected classroom climate observational system. The CLASS examines three domains of behavior including emotional support (e.g., positive climate, negative climate, teacher sensitivity), classroom organization, and instructional support. It is an impression-based system that has been validated primarily for children in preschool through third grade. Despite the strengths of the CLASS, the current authors adapted the ASKER system in the design of their tool given that it was developed for older elementary school children and used previously as an outcome assessment tool in the context of a large aggression prevention trial.

The Need for Understanding and Improving Classroom Climate in Urban Schools

Classroom climate can affect urban, African American students more than students from other cultural groups because they face considerably more school-based social challenges (Thomas et al., 2006). Many African American students attending urban schools face a disproportionate number of chronic and acute stressors that could negatively affect their behavior at school, including inequitable teacher practices (Thomas & Stevenson, 2009), aggressive peer norms (Henry et al., 2000), and exposure to community violence (Guerra, Huesmann, & Spindler, 2003). Academic and social achievement of ethnic minority students are improved when teacher practices are carried out in a culturally-responsive and supportive manner (Gay, 2000), however, many teachers, irrespective of race, feel that they have not received proper training in how to teach or relate to urban, low-income, racial minority students (Brown & Medway, 2007; Moore, 2002). As such, the current authors sought to develop and validate a classroom climate observation assessment tool specific to the needs of these youth.

Use of Participatory Action Research (PAR) to Develop Assessment Tools

Participatory action research (PAR) is a method that takes empirically-supported assessment tools, interventions, or strategies and adapts them through key stakeholder input (PAR; Leff et al., 2006; Nastasi et al., 2000). Within schools, PAR involves a nonhierarchical alliance between administration and staff, community organizers, service providers and researchers to develop effective methods to influence positive developmental outcomes for students (Leff, Costigan, & Power, 2004). These methods are sensitive to the sociocultural norms of school communities, build upon the capacity of schools to carry out sustained effective approaches (Hughes, 2002), and help to maximize meaningful outcomes and assessment tools (Leff, Power, Costigan, & Manz, 2003; Nastasi et al., 2000). Given that PAR has been used effectively to develop schoolwide, violence-prevention programming in under-resourced, urban, elementary schools (see Leff et al., 2004), the authors utilized that research paradigm for the development of an observational tool to assess classroom climate in this setting.

Goals for the Current Article

Relative to what is known in the empirical literature about the value of observational systems for assessing individual child behaviors, much less is known about the viability and efficacy of these methods for assessing the social climate of classrooms and its affect on children’s behavioral functioning. Moreover, few observation assessment tools have been designed by combining empirically-based strategies with key stakeholder feedback to ensure the developmental and cultural appropriateness of the resultant assessment tool, a practice that is increasingly being recognized as a key ingredient in the provision of best practice culturally-competent research (Leff, Power, Manz, Costigan, & Nabors, 2001). Hence, there were several primary goals of the current paper. First, the authors sought to illustrate how a PAR approach could be used within the context of measurement development for urban elementary schools. Second, analyses were conducted to determine the initial psychometric properties and associations of the new assessment tool, called the Classroom Climate Assessment Tool (C-CAT). Specifically, the authors investigated whether a Teacher Responsiveness ratio (a proportion of Teacher Praise to Teacher Reprimands) would hold promise as an efficient way of understanding how effective teachers are in providing necessary support and structure to students in their classrooms. As such, it was hypothesized that this Teacher Responsiveness ratio would be negatively correlated with classrooms that demonstrated high levels of student aggression and peer rejection. Finally, next steps in research and practice implications will be discussed.

Method

The Context for the Design and Validation of the Classroom Observation Assessment Tool

The classroom climate assessment tool was designed and validated in two elementary schools within a large, Northeastern urban school district. One of the schools was extremely large, comprised of 700 kindergarten to fourth-grade students who were predominately African American (87%) and economically-disadvantaged (85%; as defined by the school district as those students who receive free or reduced lunch). The second school was a large kindergarten to eighth-grade school (approximately 600 students) comprised almost entirely of an African American (95%) and economically-disadvantaged (90%) student body. The two schools were chosen based on convenience and were both representative of their larger urban public school district. All 18 of the third through fifth-grade teachers within the two schools were given the opportunity and chose to participate in the study. Although parental permission was not necessary given that observations were of regular instruction within the classroom, researchers were available if students or teachers had any questions about the procedures. All aspects of the research were approved by the Institutional Review Board (IRB) of the authors’ institution and by the IRB from the school district.

Peer nominations and classroom observations were also collected as part of a preintervention assessment battery the following year at the second school mentioned above across 8 third through fifth-grade classrooms. On average, 15 ten-minute intervals were observed across each of the eight classrooms. All third to fifth graders who had provided child assent and parent permission (n = 179, 84%) participated in the aforementioned preintervention assessments.

Stages of Measurement Development

The authors employed a PAR paradigm to combine strong psychological theory and past empirical research on classroom climate assessment tools with extensive feedback from urban teachers, community members, and a behavioral observation consultant/expert to develop an empirically-supported and culturally-sensitive classroom climate observational assessment tool. The input of key stakeholders allows for the creation and use of assessment tools that are sensitive and meaningful to the urban school context (see Leff et al., 2006).

Key stakeholder and advisory group

The authors enlisted the support of three teachers, two community partners, and a content area expert. The teachers, one of which had previously collaborated with the research team, were drawn from two public elementary schools within the broader urban school district, and represented different grades (third through fifth), years of experience, and teaching backgrounds. The community partners served several roles at the participating schools (classroom assistant, home-liaison coordinator, etc.) and were familiar with classroom management practices, school policies, and issues within the flanking neighborhoods that impact their respective school settings. Researchers also consulted with a psychology expert specializing in the development and validation of classroom-based observational techniques in order to ensure that adaptations to the ASKER system were in line with local school and classroom practices, and adhered to best practice empirically-supported procedures.

The research team held several meetings with the teachers and community partners to discuss the original ASKER system, as well as the adapted assessment tool, initial target behaviors, and the rating/coding options. In addition, the research team observed a number of classroom periods within these teachers’ classrooms at varying times of the day, and then discussed their impressions and ratings with the teachers. This process helped to ensure that the selected codes were behaviorally grounded, reflected actual behaviors occurring within their classroom, and were clear, meaningful, and easy to code. In addition, the psychology consultant provided ongoing advice related to the aforementioned issues for the duration of the project.

Piloting the classroom atmosphere ratings of the ASKER System

The classroom atmosphere ratings of the ASKER system were used as the starting point for the development and adaptation of the current assessment tool. Thirty-minute observations are conducted, resulting in ratings for three domains of classroom functioning: (a) Disruptive Behavior and Compliance (e.g., student level of compliance, students handle transitions well, and students consistently follow rules appropriate to settings); (b) Cooperation, Communication, and Problem-Solving (e.g., students level of cooperation, students attempting problem-solving, and students express feelings appropriately); and (c) Classroom Interest Level, Focus, and Responsiveness (e.g., students level of interest/enthusiasm, classroom is responsive to individual differences in students’ needs). Ratings are made on a 5-point scale, corresponding to the level of the observed behavior (1 = very high to 5 = very low).

Coders practiced the ASKER system for several months, while also obtaining ongoing consultation from the aforementioned advisory group. Their experience suggested that the ASKER’s target behaviors were important and relevant dimensions of class climate for the urban under-resourced elementary school classrooms. However, some of the ASKER’s target behaviors, especially in the Cooperation/Communication/Problem-Solving Domain, were infrequent and/or challenging to observe. For instance, students attempt at problem-solving was rarely observed and difficult to judge without having spent considerable time within a classroom prior to the observation period. Second, students consistently follow rules appropriate to the setting (within the Disruptive Behavior/Compliance Domain) required in-depth knowledge of the classroom and/or teacher practices and varied considerably by class. Third, the rating scale was somewhat confusing because it was anchored based upon a combination of the frequency of occurrence along with a percentage of the observation period in which the behavior occurred. Fourth, the ASKER system did not have any codes directly related to the teachers’ behavior (e.g., use of praise and reprimands). Finally, our observation team and our teacher partners felt classroom climate often changed over short periods, and as such, it seemed too long to wait 30 minutes before coding. Teachers suggested that 10-minute intervals would be most appropriate for recording the majority of classroom climate variables. Given these challenges, the research team further utilized the PAR paradigm to adapt this theoretically-grounded and empirically supported system through extensive feedback from key stakeholders.

Revised Classroom Climate Assessment Tool (C-CAT)

Based upon the previous PAR process, researchers included in the revised assessment tool only two domains of the initial ASKER system: Disruptive Behavior and Compliance Domain and Classroom Responsiveness Domain (see Table 1). The Disruptive Behavior and Compliance Domain is comprised of four primary target behaviors: (a) Noncompliance/Disruptive Behavior, (b) Teacher Reprimands, (c) Transition in Classroom Teaching, and (d) Interruption in Classroom Teaching. The occurrence of each of these behaviors was recorded and tallied for each 10-minute interval. For Noncompliance/Disruptive Behavior, the frequency of five sub-behaviors (arguing, fighting, loud talking/yelling, horseplay, and defiance) was recorded such that one instance was recorded any time a child or group of children exhibited these behaviors. For the Teacher Reprimand code, coders wrote down and tallied any time a teacher responded to a child or group of children with a reprimand or redirection. This code was based solely on the individual teacher behavior and was independent of student behavior, which was an important distinction highlighted by the stakeholder group since teachers have varying thresholds for the use of reprimand and/or redirection. Finally, the number of transitions between different teaching topics or activities and the number of interruptions to the classroom teaching environment were also recorded, however, these are not discussed because they are not the focus of the current article.

Table 1.

Primary Domains and Behaviors Recorded in the Classroom Climate Assessment Tool

Primary
Domain		 Behavioral Codes Sub-behaviors How Recorded Comments
Disruptive Behavior & Compliance Noncompliance/Disruptive Behavior

  • Arguing

  • Fighting

  • Loud talking

  • Horseplay

  • Defiance

 Behaviors are recorded and tallied per 10- minute interval Any time a child or group of children exhibit behavior it is coded
Teacher Reprimand

  • Any reprimand or redirection exhibited by teacher

 Behaviors are recorded and tallied per 10- minute interval This behavior is based solely on the teacher’s behavior
This is a new code in the adapted system
Classroom Responsiveness Teacher Praise

  • Verbal or nonverbal praise

  • Teacher assistance

  • Teacher encouragement

 Behaviors are recorded and tallied per 10- minute interval This is a new code in the adapted system
Classroom Level of Interest/Enthusiasm

  • How interested and enthusiastic the class appears to be

 Code is rated on a 3-point scale every 2 minutes, and summed every 10 minutes Coded based upon whether the majority, about half, or minority of students are enthusiastic and interested in teaching content
Classroom Level of Focus/Being on Task

  • How focused and on-task the students appearto be

 Code is rated on a 3-point scale every 2 minutes, and summed every 10 minutes Code indicates whether majority, about half, or minority of students are focused and on task with teaching content
Open in a new tab

For the Classroom Responsiveness Domain, three target behaviors were coded: (a) Teacher Praise, comprised of verbal or nonverbal praise and teacher assistance or encouragement, (b) Classroom Level of Interest/Enthusiasm, and (c) Classroom Level of Focus and Being On-Task. For Teacher Praise, coders recorded and tallied the number of times teachers exhibited these behaviors within each 10-minute observation period. Teacher feedback on the other two target behaviors suggested that these behaviors change across shorter intervals than 10 minutes. In response, researchers rated each of these codes every two minutes during the 10-minute observation period on a 1 to 3 scale, consisting of 1 = the majority of students are demonstrating this behavior, 2 = about half of the students are exhibiting these behaviors, or 3 = a minority of students are demonstrating these behaviors. A total score for these two variables was derived by tallying the ratings across the five two-minute periods.

The C-CAT was fine-tuned as described based on classroom observations and meetings with key stakeholders over the course of approximately six months. The resultant assessment tool had clearly defined and meaningful target behaviors, an unambiguous manner of recording responses (typically a frequency count), and a feasible and nonintrusive way to observe and record behaviors using a paper-and-pencil format. Please refer to Table 1 for more details.

Coder training

Four students (graduate and undergraduate) were trained in the C-CAT system. They completed didactic readings related to observational systems, issues within the urban schools, and PAR. They then progressed through research lab-based and classroom-based trainings in the coding system. Using synchronized watches, coders recorded behaviors for 10-minute periods. The team supervisor also coded sessions and those ratings were compared to ratings of other team members so that retraining could occur as appropriate.

Base rate and interrater reliability

Randomization was used to assign coders to two to three classrooms each day within a school and to determine the order by which each class was observed. Three consecutive 10-minute observations were conducted in each classroom. All coders conducted observations multiple times in each of the 18 classrooms. Coders were instructed to sit relatively close to one another in an unobtrusive part of the classroom (often the back of the class), and to synchronize their watches during periods used for interrater reliability.

On average, each classroom was observed for five 30-minute observations (e.g., 15 10-minute periods) with a range consisting of 9 to 18 10-minute intervals. Fifty-four percent of the sessions were observed simultaneously by a coder pair. Based on consultation with a behavioral observation expert, interrater reliability was calculated by deriving a simple proportion of the number of instances that Coder 1 recorded each behavior divided by the number of instances that Coder 2 recorded each behavior in each 10-minute interval. For example, if Coder 1 recorded 8 teacher reprimands and Coder 2 recorded 10 teacher reprimands, the agreement on that behavior was calculated at 80%. In the case that no disruptive behaviors were coded, researchers assigned a frequency of 1 for the purposes of determining interrater reliability.

Base rates of each behavior were determined by including any time (a) a single coder observed a classroom period, and (b) ratings from one coder of a coder pair when they observed classroom sessions simultaneously. When two coders coded sessions simultaneously, the research team randomly selected one coder’s data to use.

Peer nomination procedure

Peer nominations have been conducted across many studies and diverse samples given that they have strong internal consistency, stability across short time intervals, and moderate to high associations with other indices of youth behavior and adjustment (DeRosier & Thomas, 2003). Standard unlimited peer nominations were conducted in order to determine each participating students’ social status (e.g., how liked and/or disliked each student was), relational aggression status (gossiping, threatening to withdraw relationships), and physical aggression status (hitting, pushing). Peer nomination items included Crick and Grotpeter’s (1995) standard five relational and three physical aggression items. Raw score nominations on the relational aggression items were standardized within the nominator group (each grade), resulting in a relational aggression z-score for each child that was then averaged to calculate an overall relational aggression average score for the classroom. A similar procedure was used for physical aggression.

Results

Base Rate and Interrater Agreement

Base rates are presented separately by school for each of the classroom observation variables. As shown in Table 2, the occurrence of Noncompliance/Disruptive Behavior was low in both schools (about 1.0 instance in School 1 and about 1.5 instances in School 2, per 10-minute interval). In contrast, there were relatively higher levels of Teacher Reprimands per 10-minute interval (about 4.0 instances in School 1 and about 6.5 instances in School 2). Interestingly, Teacher Praise occurred somewhat less frequently than Teacher Reprimands (about 2.3 and 2.5 times per 10-minutes across schools). Finally, ratings of Classroom Interest/Enthusiasm and Classroom Focus/On-Task were relatively high across both schools.

Table 2.

Base Rates of Primary Behaviors by School

Behaviors N Mean SD Range
School 1
       Noncompliance/Disruptive Behavior 135 0.90 1.90 0–15
       Teacher Reprimands 135 3.93 3.80 0–16
       Teacher Praise 135 2.31 3.33 0–25
       Class Enthusiasm/Interest 134 13.44 2.29 5–15
       Class Focus 134 13.66 2.10 5–15
School 2
       Noncompliance/Disruptive Behavior 126 1.53 2.35 0–10
       Teacher Reprimands 126 6.58 4.85 0–22
       Teacher Praise 126 2.54 3.15 5–15
       Class Enthusiasm/Interest 124 11.75 2.39 5–15
       Class Focus 124 12.08 2.48 5–15
Open in a new tab

Note. Behaviors are the number of occurrences in teacher or student classroom behaviors per 10-minute interval; Noncompliance = Student noncompliance or disruptive behavior; Student Enthusiasm and Student Focus were reverse scored for the purposes of these analyses, meaning that higher scores in this table represent greater enthusiasm and focus.

Interrater agreement was strong for each of the five primary classroom observation variables. For Noncompliance/Disruptive Behavior, the average percent agreement between coders was 93.7% (range 88.7% to 100%). For Teacher Reprimand and Teacher Praise, the average percent agreement among coders was 92.2% (range 86.3% to 97.0%) and 87.6% (range 83.2% to 95.1%), respectively. The average percent agreement for Classroom Level of Interest/Enthusiasm was 96.5% (range 93.0% to 100%), and for Classroom Level of Focus and Being On-Task it was 96.4% (range 96.0% to 99.0%).

Intercorrelation Among Classroom Climate Variables

As seen in Table 3, most intercorrelations between target behaviors were in the expected direction. Noncompliance had a strong positive correlation with Teacher Reprimands, a strong negative correlation with Classroom Enthusiasm/Interest and Classroom Focus/Being on Task, and a moderate negative correlation with Teacher Praise. Teacher Praise and Teacher Reprimands were not correlated, although Teacher Reprimands was strongly negatively correlated with Classroom Enthusiasm/Interest and Classroom Focus/Being on Task. Classroom Enthusiasm/Interest and Classroom Focus/Being on Task were extremely highly correlated (r = .98) and given this, may represent one construct instead of two.

Table 3.

Intercorrelations Among Primary Behaviors

1 2 3 4 5 6
1. Noncompliance — .50* −.38 −.70*** −.75** −.60**
2. Teacher Reprimand — .10 −.78*** −.72*** −.61**
3. Teacher Praise — .12 .11 .54*
4. Student Enthusiasm — .98*** .42
5. Student Focus — .40
6. Teacher Responsiveness —
Open in a new tab

Note. Each teacher’s (n = 18) total score for each coded variable was calculated across all observation periods, then correlations were conducted. Noncompliance = Student Noncompliance or Disruptive Behavior. Student Enthusiasm and Student Focus were reverse scored for the purposes of these analyses, meaning that higher scores on these variables represent greater enthusiasm and focus.

*

p < .05,

**

p < .01,

***

p < .001.

In addition to the primary target behaviors, a Teacher Responsiveness ratio was derived by dividing Teacher Praise by Teacher Reprimand. It was thought that the Teacher Responsiveness ratio could be used as an overall index of teacher-student relationship. As predicted, the Teacher Responsiveness Ratio was negatively correlated with Noncompliant/Disruptive Behavior, and positively correlated with Classroom Enthusiasm/Interest and Classroom Focus/Being on Task. Also expected, given the derivation of the ratio, Teacher Responsiveness was positive correlated with Teacher Praise and negatively correlated with Teacher Reprimands. In general, higher levels of teacher responsiveness (higher ratios of praise/reprimands) were associated with lower levels of student noncompliance and higher levels of student enthusiasm and focus.

Association Between Teacher Responsiveness and Peer Nominations

Researchers also determined the association between the Teacher Responsiveness ratio and overall classroom average z-scores for peer nominations of relational aggression, physical aggression, being liked most, and being liked least. Pearson correlational coefficients indicated that the Teacher Responsiveness ratio was negatively correlated with classroom average z-scores for Relational Aggression (r = -.38) and Physical Aggression (r = -.47). The Teacher Responsiveness ratio was moderately positively correlated with nominations for being liked most (r = .23) and negatively correlated with nominations for being liked least (r = -.18).

Discussion

The current study investigated how to refine and adapt an empirically-supported and theoretically-grounded classroom climate assessment tool through participatory action research to create a meaningful and culturally-responsive observation tool for use within urban, under-resourced elementary school settings. By partnering with a diverse group of key stakeholders and eliciting their feedback on an ongoing basis, this study represents an exemplar of the participatory action research (PAR) paradigm. PAR approaches highlight the merit of designing and/or adaptation assessment tools so that they are both psychometrically and theoretically sound, and meaningful to specific populations and settings (Leff et al., 2003; Nastasi et al., 2000). In the current study, the development of the C-CAT was additionally meaningful and promising given that observations of student behavior are the most commonly used assessment modality by school psychologists (Shapiro & Heick, 2004). Further, since almost all pre-existing observation tools provide information related to specific at-risk children’s behavioral functioning, on-task behavior, and/or seatwork (see Volpe et al., 2005), the C-CAT meets a clear need in representing a brief classroom observation tool that can help to inform aggression prevention programming and teacher training.

A preliminary base rate and interrater reliability study suggests that the primary behaviors comprising the C-CAT can be reliably observed and coded. Many of the target behaviors within the Noncompliance/Disruptive domain occurred somewhat frequently across the three grade levels within the urban school classrooms, as did the use of teacher reprimands (about 4.0 to 6.5 times per 10-minute period). With regard to the Classroom Responsiveness domain, teacher praise occurred less frequently (about 2.5 times per 10-minute period). Given mounting research indicating the importance of teacher praise and reinforcement on student behaviors (Madsen et al., 1968; Swinson & Harrop, 2001), the C-CAT meets a critical need in measuring this variable.

The C-CAT may have future utility for school practitioners. For instance, the use of a Teacher Responsiveness ratio (teacher praise/teacher reprimand) may serve as a proxy for obtaining more detailed and comprehensive observations of classroom climate. Specifically, correlational analyses suggest that lower levels of teacher responsiveness are associated with higher average classroom levels of relational and physical aggression, and noncompliant and disruptive behaviors. Conversely, higher levels of teacher responsiveness are associated with students being “liked most” by their peers and higher average classroom levels of enthusiasm and focus. Given this, the C-CAT may be an efficient and cost-effective tool for professionals to evaluate these two very important domains. Future research could examine the ease and efficiency by which school-based practitioners can be trained in using this tool, and how best to utilize the C-CAT’s findings to provide feedback to teachers on their teaching style, relationship with their students, and overall classroom climate. Future research could also examine the sensitivity of the Teacher Responsiveness ratio to classroom-based aggression prevention programming and teacher training on the use of positive reinforcement in the classroom.

Study Limitations

Results of this study should be considered in light of several limitations. First, the study was conducted in a limited number of elementary schools that were chosen as a convenience sample. Although these schools are similar to most of the elementary schools within the large urban school district from which they were drawn, results should be interpreted cautiously until the study is replicated across more urban elementary schools. Further, study results cannot be generalized to elementary schools outside of the urban school setting or to elementary schools that do not have a predominately African American, low-income student body. Examining the C-CAT’s applicability in other setting would be an important avenue of future research. Another limitation is the cross-sectional nature of the data. It will be important to examine whether this new classroom climate observation tool is sensitive to detecting aggression prevention and intervention changes within schools across different time intervals. Further, it would be important to compare the C-CAT to other existing classroom climate assessment tools, such as the CLASS (Pianta et al., 2008), in order to determine the unique contributions of each tool.

Conclusion

Despite these limitations, findings from the present study have important implications for research and practice in the area of bullying prevention. First, behavioral observation systems are regarded as vital tools to gain insight into the aggressive behavior and co-occurring problems of youth involved in bullying, the antecedents of these behaviors, and their consequences (Orpinas & Horne, 2006, p. 184). Second, given the C-CAT’s straightforward nature, it is suggested that school-based professionals would need minimal training in its use. Third, it appears that the two domains comprising the Teacher Responsiveness ratio would enable schools to glean information and/or provide feedback on the teacher practices linked to student behavior. Further, it may be possible to couple this index with frequency data of noncompliant classroom behaviors in order to determine the effectiveness of classroom-based prevention initiatives and/or teacher in-service programs that are designed to change aspects of the classroom climate that may foster or exacerbate interpersonal aggression and bullying in the school setting. Finally, given that the C-CAT was designed through partnership, it may allow urban school districts to obtain important classroom climate information in a culturally-sensitive, meaningful manner. While the C-CAT should be examined across more classrooms and schools, it offers promise for providing critical information to help guide practice in urban schools with regard to creating positive and safe learning environments for all children.

Acknowledgments

Funding for this work was provided by an NIMH grant to the first author, R34MH072982, and by cooperative agreement number 5 U49 CE001093 from The Centers for Disease Control and Prevention. Its contents are the sole responsibility of the authors and do not represent the official position of the National Institute of Mental Health or the Centers for Disease Control and Prevention. This research was made possible, in part, by the School District of Philadelphia. Opinions contained in this report reflect those of the authors and do not necessarily reflect those of the School District of Philadelphia

Contributor Information

Stephen S. Leff, The Children’s Hospital of Philadelphia and University of Pennsylvania School of Medicine, and The Philadelphia Collaborative Violence Prevention Center, Philadelphia, Pennsylvania USA

Duane E. Thomas, The Philadelphia Collaborative Violence Prevention Center and The University of Pennsylvania, Graduate School of Education, Philadelphia, Pennsylvania USA

Edward S. Shapiro, Center for Promoting Research to Practice, Lehigh University, Lehigh, Pennsylvania USA

Brooke Paskewich, The Children’s Hospital of Philadelphia and University of Pennsylvania School of Medicine, Philadelphia, Pennsylvania USA.

Kim Wilson, The Children’s Hospital of Philadelphia and University of Pennsylvania School of Medicine, and The Philadelphia Collaborative Violence Prevention Center, Philadelphia, Pennsylvania USA.

Beth Necowitz-Hoffman, The Children’s Hospital of Philadelphia and University of Pennsylvania School of Medicine, Philadelphia, Pennsylvania USA.

Abbas F. Jawad, The Children’s Hospital of Philadelphia and University of Pennsylvania School of Medicine, Philadelphia, Pennsylvania USA

References

  1. Brophy J. Classroom organization and management. Elementary School Journal. 1983;83:265–285. [Google Scholar]
  2. Brown KE, Medway FJ. School climate and teacher beliefs in a school effectively serving poor South Caroline (USA) African-American students: A case study. Teaching and Early Education. 2007;23:529–540. [Google Scholar]
  3. Conduct Problems Prevention Research Group. Initial Impact of the Fast Track Prevention Trail for Conduct Problems: II. Classroom Effects. Journal of Consulting and Clinical Psychology. 1999;67:648–657. [PMC free article] [PubMed] [Google Scholar]
  4. Crick NR, Grotpeter JK. Relational aggression, gender, and social psychological adjustment. Child Development. 1995;66:710–722. doi: 10.1111/j.1467-8624.1995.tb00900.x. [DOI] [PubMed] [Google Scholar]
  5. DeRosier ME, Thomas JM. Strengthening sociometric prediction: Scientific advances in the assessment of children peer relations. Child Development. 2003;74:1379–1392. doi: 10.1111/1467-8624.00613. [DOI] [PubMed] [Google Scholar]
  6. Doll B, Song S, Siemers E. Classroom ecologies that support or discourage bullying and intervention. In: Espelage DL, Swearer SM, editors. Bullying in American schools: A social-ecological perspective on prevention. Mahwah, NJ: Erlbaum; 2004. pp. 161–183. [Google Scholar]
  7. Emmer E. Classroom observation scales. Austin, TX: Research and Development Center for Teacher Education, University of Texas; 1971. [Google Scholar]
  8. Gay G. Culturally responsive teaching Theory, research, and practices. New York, NY: Teachers College Press; 2000. [Google Scholar]
  9. Greenwood CR, Carta JJ, Kamps D, Terry B, Delquadri J. Development and validation of standard classroom observation systems for school practitioners: Ecobehavioral Assessment Systems Software (EBASS) Exceptional Children, Special Issue: Technology-Based Assessment within Special Education. 1994;61:197–210. Retrieved from http://journals.cec.sped.org/ec/ [Google Scholar]
  10. Guerra NG, Huesmann LR, Spindler A. Community violence exposure, social cognition and aggression among urban elementary school children. Child Development. 2003;74:1561–1576. doi: 10.1111/1467-8624.00623. [DOI] [PubMed] [Google Scholar]
  11. Henry D, Guerra N, Huesmann R, Tolan P, VanAcker R, Eron L. Normative influences on aggression in urban elementary school classroom. American Journal of Community Psychology. 2000;28:59–81. doi: 10.1023/A:1005142429725. [DOI] [PubMed] [Google Scholar]
  12. Howes C, Hamilton CE. The changing experience of child care: Changes in teachers and in teacher-child relationships and children’s social competence with peers. Early Childhood Research Quarterly. 1993;8:15–32. [Google Scholar]
  13. Hughes JN. Commentary: Participatory action research leads to sustainable school and community improvement. School Psychology Review. 2002;32:38–43. Retrieved from http://www.nasponline.org/publications/spr/sprmain.aspx. [Google Scholar]
  14. Kellam SG, Ling X, Merisca R, Brown CH, Ialongo N. The effect of the level of aggression in the first-grade classroom on the course and malleability of aggressive behavior into middle school. Development and Psychopathology. 1998;10:165–185. doi: 10.1017/s0954579498001564. [DOI] [PubMed] [Google Scholar]
  15. Kern L, Clemens NH. Antecedent strategies to promote appropriate classroom behavior. Psychology in the Schools. 2007;44:65–75. [Google Scholar]
  16. Kounin J. Discipline and group management in classrooms. New York, NY: Holt, Rinehart & Winston; 1970. [Google Scholar]
  17. Leff SS, Costigan TE, Power TJ. Using participatory-action research to develop a playground-based prevention program. Journal of School Psychology. 2004;42:3–21. [Google Scholar]
  18. Leff SS, Crick NR, Power TJ, Angelucci J, Haye K, et al. Understanding social cognitive development in context: Partnering with urban African American girls to create a hostile attributional bias measure. Child Development. 2006;77:1351–1358. doi: 10.1111/j.1467-8624.2006.00939.x. [DOI] [PubMed] [Google Scholar]
  19. Leff SS, Power TJ, Costigan T, Manz PH. Assessing the climate of the playground and lunchroom: Implications for bullying prevention programming. School Psychology Review. 2003;32:418–430. Retrieved from http://www.nasponline.org/publications/spr/sprmain.aspx. [Google Scholar]
  20. Leff SS, Power TJ, Manz PH, Costigan TE, Nabors LA. School-based aggression prevention programs for young children: Current status and implications for violence prevention. School Psychology Review. 2001;30:344–362. Retrieved from http://www.nasponline.org/publications/spr/sprmain.aspx. [Google Scholar]
  21. Madsen CH, Becker WC, Thomas DR. Rules, praise, and ignoring: Elements of elementary classroom control. Journal of Applied Behavior Analysis. 1968;1:139–150. doi: 10.1901/jaba.1968.1-139. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Meehan BT, Hughes JN, Cavell TA. Teacher-student relationships as compensatory resources for aggressive children. Child Development. 2003;74:1145–1157. doi: 10.1111/1467-8624.00598. [DOI] [PubMed] [Google Scholar]
  23. Moore AL. African-American early childhood teachers’ decisions to refer African-American students. International Journal of Qualitative Studies in Education. 2002;15:631–652. [Google Scholar]
  24. Nastasi BK, Varjas K, Schensul SL, Silva KT, Schensul JJ, Ratnayake P. The participatory intervention model: A framework for conceptualizing and promoting intervention acceptability. School Psychology Quarterly. 2000;15:207–232. [Google Scholar]
  25. Olweus D, Limber S. Blueprints for violence prevention: Bullying Prevention Program. Boulder, CO: Institute for Behavioral Science, University of Colorado; 1999. [Google Scholar]
  26. Orpinas PA, Horne AM. Bullying prevention: Creating a positive school climate and developing social competence. Washington, DC: American Psychological Association; 2006. [Google Scholar]
  27. Pianta RC, La Paro KM, Hamre BK. Classroom Assessment Scoring System. Baltimore, MD: Paul H. Brookes; 2008. (CLASS) [Google Scholar]
  28. Shapiro ES, Heick P. School psychologist assessment practices in the evaluation of students referred for social/behavioral/emotional problems. Psychology in the Schools. 2004;41:551–561. [Google Scholar]
  29. Stuhlman M, Pianta R. Teachers’ narratives about their relationships with children: Associations with behavior in classrooms. School Psychology Review. 2001;31:148–163. Retrieved from http://www.nasponline.org/publications/spr/sprmain.aspx. [Google Scholar]
  30. Swinson J, Harrop A. The differential effects of teacher approval and disapproval in junior and infant classrooms. Educational Psychology in Practice. 2001;17:157–167. [Google Scholar]
  31. Tapp JT, Fiel D. ASKER: A computerized program for administering Likert-scale questions. Nashville, TN: Vanderbilt University; 1991. (Unpublished computer program) [Google Scholar]
  32. Tapp J, Wehby J, Ellis D. A multiple option observation system for experimental studies: MOOSES. Behavior Research Methods, Instruments & Computers. 1995;27:25–31. Retrieved from http://brm.psychonomic-journals.org/ [Google Scholar]
  33. Thomas DE, Bierman KL the Conduct Problems Prevention Research Group. The impact of classroom aggression on the development of aggressive behavior problems in children. Development and Psychopathology. 2006;18:471–487. doi: 10.1017/S0954579406060251. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Thomas DE, Bierman KL, Powers CJ the Conduct Problems Prevention Research Group. The impact of peer aggression and classroom climate on the development of aggressive behavior problems. Child Development. (in press) [Google Scholar]
  35. Thomas DE, Bierman KL, Thompson CI, Powers CJ Conduct Problems Prevention Research Group. Double jeopardy: Child and school characteristics that predict aggressive-disruptive behavior in first grade. School Psychology Review. 2008;37:516–532. Retrieved from http://www.nasponline.org/publications/spr/sprmain.aspx. [PMC free article] [PubMed] [Google Scholar]
  36. Thomas DE, Stevenson H. Gender risks and education: The particular classroom challenges for urban low-income African American boys. Review of Research in Education. 2009;33:160–180. [Google Scholar]
  37. Volpe R, DiPerna J, Hintze J, Shapiro E. Observing students in classroom settings: A review of seven coding schemes. School Psychology Review. 2005;34:454–474. Retrieved from http://www.nasponline.org/publications/spr/sprmain.aspx. [Google Scholar]
  38. Wilson HK, Pianta RC, Stuhlman M. Typical classroom experiences in first grade: The role of classroom climate and functional risk in the development of social competencies. The Elementary School Journal. 2007;108:81–96. [Google Scholar]

RESOURCES