Commentary on Two Classroom Observation Systems: Moving Toward a Shared Understanding of Effective Teaching

Abstract

In this commentary, I make five points: that designing observation systems that actually predict students’ outcomes is challenging; second that systems that capture the complex and dynamic nature of the classroom learning environment are more likely to be able to meet this challenge; three, that observation tools are most useful when developed to serve a particular purpose and are put to that purpose; four that technology can help; and five, there are policy implications for valid and reliable classroom observation tools. The two observation systems presented in this special issue represent an important step forward and a move toward policy that promises to make a true difference in what is defined as high quality and effective teaching, what it looks like in the classroom, and how these practices can be more widely disseminated so that all children, including those attending under-resourced schools, can experience effective instruction, academic success and the lifelong accomplishment that follows.

As education researchers, we are dedicated to improving the lives of children. Many of us do this by investigating ways to improve students’ schooling experiences so that they learn to read and do math proficiently, learn about science and the world around them, build their capacity to learn complex concepts and generate new ideas, and nurture their overall cognitive, social, and emotional development. It is not surprising, therefore, that the classroom learning environment is important, and finding ways to elucidate the active ingredients of this environment are imperative but difficult. The three articles in this issue of School Psychology Quarterly (Crawford et al., pp. 277–300; Reddy et al., pp. 301–316 and pp. 317–341) and the two systems they describe, the Classroom Observation Tool (COT) and Classroom Strategies Scale-Observer Form (CSS), make critical contributions to measuring and improving the classroom learning environment in large part because they were specifically designed using available research findings with the goal of providing feedback to teachers and guiding professional development. They are excellent testimonies to the importance of funding this kind of research. In my commentary, I want to make five points: one, that designing observation systems that actually predict students’ outcomes is challenging; two, that systems that capture the complex and dynamic nature of the classroom learning environment are more likely to be able to meet this challenge; three, that observation tools are most useful when developed to serve a particular purpose and are put to that purpose; four that technology can help; and five, there are policy implications for valid and reliable classroom observation tools.

Designing Systems That Actually Predict Students’ Outcomes Is Challenging

Although we generally accept that teachers and teaching are important sources of influence on children’s learning (e.g., Darling-Hammond, 1997; Goldhaber & Anthony, 2003; Kane, Staiger, & McCaffrey, 2012; Taylor, Roehrig, Connor, & Schatschneider, 2010), other sources of influence account for many individual student differences including genetics, family, home, and community (e.g., Byrne et al., 2010; Connor, Son, Hindman, & Morrison, 2005; Morrison, Bachman, & Connor, 2005; Raudenbush, 2009). Thus, it should not be surprising that even the most carefully designed observation systems explain relatively modest amounts of the variability in students’ achievement. Indeed, the large Gates Foundation–funded study (Kane et al., 2012) that investigated highly regarded classroom observation systems found only modest correlations among teachers’ value-added scores (i.e., student achievement) and their scores on the various rating systems.

With the CSS and COT in good company, it is important to keep in mind that these tools are designed to be used by educational leaders including principals and literacy coaches, during the busy school day, with the express aim of improving teaching. Hence, one would expect weaker associations among student outcomes and ratings on the CSS and COT because the raters would be less knowledgeable about how to use the rating systems than researchers with more training (and thus less reliable), they would not have the luxury of viewing video recordings multiple times, and the observations are relatively brief. And yet, specific characteristics of instruction on the COT (e.g., phonological awareness, letter knowledge, print concepts) significantly predicted students’ letter knowledge and phonological awareness (although not vocabulary or mathematics). Similarly, the CSS Instructional Strategies Scale discrepancy scores (the absolute value of the recommended frequency of a behavior minus the observed frequency) were moderately correlated with students’ mathematics and English language arts scores (although ratings on the Behavior Management Scale did not predict student outcomes). This level of predictive validity is highly encouraging, and although predictive validity for some aspects of the observational systems is lacking, as the authors note, they represent an opportunity as well. This brings me to my next point.

Systems That Capture the Complex and Dynamic Nature of the Classroom Learning Environment Are More Likely to be Able to Predict Student Outcomes

Reddy and colleagues (pp. 317–341) state that “CSS is grounded in teaching model research…[and that their results characterize] effective teaching as an interactive and integrative process that draws from multiple models of instruction.” This includes both direct or explicit and systematic instruction as well as teaching strategies based on “constructivism and teacher-student interactions.” Similarly, the COT rates teaching on multiple dimensions. These systems are innovative, rigorous, and actually predict student outcomes because they recognize the complexity of instruction and the classroom environment. They also show that no one theory of instruction is superior to another and that it is not either constructivist instruction or explicit systematic instruction that is effective. Effective teachers differentiate students’ instruction using assessment information to tailor each student’s instructional regime (e.g., Connor, Piasta, et al., 2009; Raudenbush, 2009) and may use a number of different models of teaching depending on the learning needs of their students.

I have spent my entire career trying to figure out how to measure what goes on in classrooms and what aspects of the classroom learning environment contribute to students’ learning. With an important acknowledgment to my great collaborators, we have and are continuing to develop observation systems that rely on observing the amount of the different types and content of literacy instruction that individual students receive, as well as the quality of this instruction (Connor, Morrison, et al., 2009). Neither the CSS nor the COT system captures this critical element of the complexity of the classroom-learning environment—the individual students in the classroom— even though both systems attempt to measure differentiated instruction. In our work, we have discovered that children who share a classroom still have very different learning opportunities and experiences. Some of these differences may be intentional, for example, differentiated instruction, but others may be unintentional (e.g., providing less instruction to children with behavior problems). We have also found that the extent to which the literacy instruction provided matches the individual students’ instructional recommendations, based on assessments, the stronger students’ outcomes (Connor, Morrison, Fishman, et al., 2011; Connor, Morrison, Schatschneider, et al., 2011; Connor, Piasta, et al., 2009).

Returning to the finding that the CSS Instructional Strategies Scale predicted student outcomes but the BMS did not, in our research, we have found that well-organized classrooms and strong classroom management, which, along with the social-emotional climate, teacher knowledge, education, and experience, constitute foundational dimensions of the classroom learning environment (Connor, Morrison, et al., 2009). These foundational dimensions are necessary but are generally not sufficient to ensure student learning (Connor et al., in press). Thus, they are not as predictive of student outcomes as descriptions of effective instructional strategies related to content. Crawford and colleagues make the point that many preschool observation systems are content specific (e.g., literacy) and that observation tools that can be used across content areas are needed. However, given the results for both systems, I would suggest that systems that focus on content specific instruction are more likely to predict student outcomes in that content area than are more general systems. This is because a large part of effective teaching taps teachers’, coaches’, and educational leaders’ specialized knowledge about the content area—reading, math, science, social studies, and so forth—and how to use this knowledge in the classroom (e.g., Piasta, Connor, Fishman, & Morrison, 2009). For example, there are highly organized teachers who run a tight ship but who do not teach reading in ways that the students can comprehend and learn (Connor et al., in press). In contrast, there are teachers whose classes look highly disorganized but who can lead class discussions in ways that support students’ learning and comprehension (Carlisle, Kelcey, Berebitsky, & Phelps, 2011). At the same time, truly chaotic classrooms are not good for anybody (Wachs, Gurkas, & Kontos, 2004). Rigorous observation systems will be able to differentiate among the three.

Increasing complexity in the classroom are peer effects (Justice, Petscher, Schatschneider, & Mashburn, 2011; Skibbe, Phillips, Day, Brophy-Herb, & Connor, 2012). The characteristics that students bring to the classroom, their language, literacy, content knowledge, self-regulation, and social-emotional status, all contribute to the classroom learning environment. For example, it might be easier for teachers to demonstrate the kinds of instructional behaviors and strategies that the CSS Instructional Strategies Scale measures when children are higher achieving; certainly the concurrent results can be interpreted in this way. All of us who teach know that some classes are easier to teach than others and that a large part of our ability to, for example, generate meaningful discussions has much to do with the extent to which the students are willing and able to discuss, their knowledge of the topic, and their willingness to disagree with others. Skibbe and colleagues (2012) discovered that having a higher number of children with poor self-regulation not only lessened literacy gains for the children with weak self-regulation but also led to generally weaker gains for all of the children in the classroom. When developing observation tools that are designed to help teachers improve their teaching, attention to peer effects on the classroom-learning environment may strengthen their utility.

Observation Tools Are Most Useful When Developed to Serve a Particular Purpose and Are Put to That Purpose

One of the outstanding characteristics of the CSS and the COT is that they have been explicitly designed to support teachers’ understanding of effective teaching. They have been designed with the end in mind. Many of the observation system out there, including ours, have been designed for specific research purposes— understanding how teachers individualize instruction and what it means for children with different skills and aptitudes (e.g., the ISI system), understanding the social-emotional climate of the classroom (e.g., the CLASS), and so forth. Instead, the CSS and COT have been designed to be used by educational leaders to inform the professional development of teachers and the improvement of their classroom instruction. Their utility would be undermined if they were suddenly used for teacher evaluation and retention. As they are designed, they can be used as the centerpiece of communities of practice (Bos, Mather, Narr, & Babur, 1999; Stigler & Hiebert, 1999) where teachers and educational leaders work together to improve student outcomes.

Technology Helps

Technology is and will be a boon that allows us to improve the predictive validity and reliability of these and other observation tools, as shown by the COT. The use of tablets (e.g., iPads) will allow us to create and take into the classroom highly complex systems that preserve a user interface that is easy for teachers and educational leaders to utilize. More complex psychometric data can be generated (Carlisle et al., 2011) and used to differentially weigh teacher practices in the rating systems that lead to stronger versus weaker student outcomes. Data mining (Baker & Yacef, 2009) can uncover more complex associations between the instruction individual students receive and whether this instruction is optimized for their individual skills and aptitudes. Such systems can facilitate the observation of individual students in the classroom.

Returning to the observation of individual students, I remind researchers that recording the instruction for individual students will be much easier for teachers and educational leaders because they already know the children, interact with them daily, and can have their assessment results handy. In contrast, as researchers, we watch video and pray that the observer took adequate field notes, that it isn’t an “everybody wear yellow” day, and that the school hasn’t implemented a school uniform policy.

Policy

Supporting the development of observation tools expressly designed to improve effectiveness in teaching is an important policy decision (Brock, 2013) and this is why: Teacher value-added scores, even appropriately used (Raudenbush, 2004), offer only a black box called the classroom. They do show us that there is tremendous variability in the effectiveness of teaching, which has direct implications for students’ success or failure (Konstantopoulos & Chung, 2011). Steve Raudenbush (2009) describes the previous (and some current) theories of teaching as “privatized idiosyncratic practice” (p. 172) whereby teachers close their classroom doors and teach in the ways they believe to be best. The ideal teacher develops her own curriculum. For the most expert teachers, who have a good grasp of the current research, have expert and specialized knowledge of their content area, and who understand how to use research evidence to inform their practice, this instruction is probably highly effective. However, for teachers with less experience or limited grasp of the research base, such privatized idiosyncratic practice may be highly ineffective and disastrous, particularly for the most vulnerable children—those from low SES families whose home learning environment and access to resources is limited. What research-based observation tools do is allow us to open the black box of the classroom and begin to move toward what Raudenbush calls, “shared instructional regimes” where the best and most rigorous research on effective teaching in a specific content area is brought to bear. “Such explicit notions of instruction define the work of teaching, the expertise required for classroom success, and the role of incentives and accountability in motivating expert instruction” (p. 172). And, I would add, help to illustrate what effective expert practice in the classroom actually looks like so that it can be shared among a community of professionals—both educators and researchers.

In all three articles, the authors make it clear that more work needs to be accomplished and that these observation tools need continuous investigation and refinement to support effective teaching—we still do not understand everything that it takes to improve students’ outcomes. However, these tools represent an important step forward and a move toward policy that promises to make a true difference in what is defined as high quality and effective teaching, what it looks like in the classroom, and how these practices can be more widely disseminated so that all children, including those attending under-resourced schools, can experience effective instruction, academic success, and the lifelong accomplishment that follows.