The Individualized Classroom Assessment Scoring System (inCLASS): Preliminary Reliability and Validity of a System for Observing Preschoolers’ Competence in Classroom Interactions

Abstract

This paper introduces the Individualized Classroom Assessment Scoring System (inCLASS), an observation tool that targets children’s interactions in preschool classrooms with teachers, peers, and tasks. In particular, initial evidence is reported of the extent to which the inCLASS meets the following psychometric criteria: inter-rater reliability, normal distributions and adequate range, construct validity, and criterion-related validity. These initial findings suggest that the inCLASS has the potential to provide an authentic, contextualized assessment of young children’s classroom behaviors. Future directions for research with the inCLASS are discussed.

A generation of longitudinal studies point to the importance for early school success of social relationships, self-regulation (e.g., attention) during interactions with materials, and language development, all of which involve behaviors embedded within home or classroom contexts (e.g., Blair, 2002; Burchinal, Peisner-Feinberg, Pianta, & Howes, 2002; Duncan et al., 2007; Hamre & Pianta, 2001; Ladd, 2005; NICHD Early Child Care Research Network [ECCRN], 2003; Wasik, Bond, & Hindman, 2006). These findings underscore the contention of Pianta and Walsh (1996) and others (Ladd, 2005; Howes & James, 2002) that early childhood skills and abilities develop within child-context interactions, including those with adults, peers, and learning activities/materials. However, much of what we know about children’s relationship, self-regulatory, and language competencies come from retroactive, informant-report surveys and one-on-one, standardized assessments, without real-time observational data about how behaviors are expressed in early childhood classroom interactions.

From the perspective of a number of leading investigators, using standardized procedures to gather process-oriented, observationally-based information on children’s competence in classroom interactions adheres to key professional standards for multi-method assessment and provides unique information on school adjustment that complements other existing measures (Meisels, Xue, & Shamblott, in press; Merrell, 1999; Neisworth & Bagnato, 2004). Until recently, however, the development of classroom observational systems for describing early childhood competencies has been slowed by a lack of consensus around developmentally appropriate constructs and limits to observational methodology. Growing consensus in the field indicates that children engage in classroom interactions that reflect patterns of adaptation to three core developmental tasks--competent exchanges with teachers, peers, and tasks--that in turn relate to building effective social relationships and acquiring skills/knowledge through instructional opportunities (see Bruner, 1966; Pianta, 1999; or Sroufe, 1996 for a discussion of developmental tasks and global patterns of function). By pairing this conceptual framework for a child’s classroom interactions with recent advancements in the development of reliable, valid observational methods, this paper introduces the newly developed Individualized Classroom Assessment Scoring System (inCLASS), a system for observation that targets children’s interactions in preschool classrooms with teachers (i.e., Positive Engagement with the Teacher, Teacher Conflict, and Teacher Communication), peers (i.e., Peer Sociability, Peer Conflict, Peer Assertiveness, and Peer Communication), and tasks (i.e., Engagement within Tasks and Self-Reliance; see Table 1). In particular, preliminary evidence will be reported of the extent to which the inCLASS meets the following psychometric criteria in a sample of 3-, 4-, and 5-year old children: inter-rater reliability, normal distributions and adequate range, construct validity, and criterion-related validity (concurrent and discriminant).

Table 1.

The inCLASS Domains, Dimensions, and Specific Behaviors

Domain	Dimension	Definition	Specific Behaviors
Teacher Interactions	Positive Engagement with the Teacher	Reflects the degree to which the child is emotionally connected to teachers, seeking and apparently enjoying interactions with them. The use of the teacher as a “secure base” is important to this rating.	Attunement (Tracking, Attention, Shared experience, Cooperation) Proximity Seeking (Physical, Conversation, Eye Contact, Response to presence) Shared positive affect (Matching, Reciprocity, Response to affection)
	Teacher Communication	Encompasses the child’s communication with all teachers and adults, as well as his use of speech as a functional tool to accomplish a variety of intentional acts (e.g., requesting, commenting, and questioning). The child’s initiations, maintenance of conversation, as well as use of language to make his needs, emotions, and opinions known should be the focal point of this rating.	Initiates Communication (Leads, Joins, Persists) Sustains Conversation (Responses, Contingency, Turn- taking, Topic prolonged) Varied Purposes for Speech (Practical/Requests, Social/Comments)
	Teacher Conflict	Reflects the degree to which the child’s interactions with the teacher are characterized by tension, resistance, and negativity. The child’s use of aggressive acts and attention- seeking behaviors along with having a negative affect towards the teacher and not complying with the teacher are all considered.	Aggression (Verbal, Physical) Noncompliance (Argumentative, Resists connection, Defiant) Negative Affect (Facial expressions, Body language) Attention-seeking (Whining, Complaining, Drains teacher energy)
Peer Interactions	Peer Sociability	Peer sociability refers to the child’s experience of positive emotions and behaviors with peers, as seen in the propensity to seek peer interactions, social awareness and responsiveness within such interactions, and the manner in which peers respond to the child.	Proximity-seeking (Physical, Conversation, Eye contact) Shared Positive Affect (Reciprocity, Matching, Affection) Popularity (Treatment by peers, Friendships) Perspective-taking (Awareness, Responsivity to cues, Consideration, Politeness) Cooperation (Sharing, Fairness)
	Peer Assertiveness	Assertiveness is characterized by successful initiation of peer interactions, leadership in peer interactions, and self- confidence with peers; all expressed through positive strategies.	Initiation (Conversation, Play, Joining groups) Leadership (Organizes play, Imitated by peers, Teaches peers) Self-advocacy (Speaks up in self-defense, Communicates needs to peers)
	Peer Communication	Peer communication encompasses the child’s communication with peers, including his ability to use speech as a functional tool to accomplish a variety of intentional acts (e.g., requesting, commenting, and questioning). The child’s initiations and maintenance of communication, as well as use of speech for social as well as practical needs, should be the focal point of this rating.	Initiates Communication (Leads, Joins, Persists) Sustains Conversation (Responses, Contingency, Turn- taking, Topic prolonged) Varied Purposes for Speech (Practical, Social)
	Peer Conflict	Peer conflict is characterized the child’s typical affect around peers, the levels of aggression and/or confrontation present in his interactions with peers, and the levels of disrespect and/or attention-seeking present in his behavior toward peers.	Aggression (Verbal, Physical, Relational) Confrontation (Argumentative, Uncooperative) Negative Affect (Facial expressions, Body language) Attention-seeking (Whining, Complaining, Intrusive)
Task Orientation	Engagement within Tasks	Engagement measures the degree to which the child is actively involved in classroom tasks and activities, including the amount of time the child remains focused on any given activity and the level of enthusiasm or intensity displayed, as well as the proportion of time the child spends on activities that are assigned.	Sustained Attention (Focus/distractibility, On-task, Follows directions) Active Engagement (Enthusiasm, Volunteering, Intensity)
	Self-Reliance	Self-Reliance measures the degree to which the child takes learning into his own hands. This includes seeking opportunities rather than passively waiting for teacher direction. The child’s ability to make best use of classroom resources (including the teacher) is also considered.	Personal Initiative (Initiates, Novelty seeking, Inquisitive) Independence (Needed guidance, Confidence in abilities, Bids for assistance) Persistence (Frustration tolerance, Calmness) Self-directed Learning (Making connections)

Developmentally Salient Domains of Interaction

There is no shortage of discussion regarding the key domains of functioning that describe competence in interactions with teachers, peers, and activities in the 3–5 age range (see Meisels, 1999; Pianta, 1999; Raver & Zigler, 2004). These interactions in early childhood settings develop in complexity over this period and forecast performance in elementary school peer nominations (e.g., popularity; Denham & McKinley, 1993), teacher ratings (e.g., problem behavior; Hightower et al., 1986), and achievement tests (Duncan et al., 2007; Ladd & Price, 1987; NICHD ECCRN, 2008). Given that child-context interactions are a key facet of the classroom experience that contribute to both social and academic outcomes, observing them can provide important information about developing skills that complements other assessment methods and leads directly to aligned approaches that support these developing competencies.

Teachers

For young children, social/relational and expressive language skills expressed within adult-child interactions are among the best predictors of children’s success in the early school years and even into late elementary school (Barth & Parke, 1993; Denham, 1993; Morrison, Rimm-Kaufman, & Pianta, 2003; NICHD ECCRN, 2002, 2004a, 2005; Pianta & Harbers, 1996; Welsh, Parke, Widaman, & O’Neil, 2001). The inCLASS addresses three specific dimensions of behavior within interactions with teachers, including positive engagement (or approach), communicative efficacy, and expressed conflict.

Children’s positive engagement and connection with adults in early education settings have reliable and detectable relationships with children’s achievement and social competence both concurrently and in elementary school (e.g., Burchinal, Peisner-Feinberg, Pianta, & Howes, 2002; Howes, Phillipsen, & Peisner-Feinberg, 2000; Meyer, Wardrop, Hastings, & Linn, 1993; NICHD ECCRN, 1996; NICHD ECCRN, 2003; NICHD ECCRN, 2004b; Peisner-Feinberg & Burchinal, 1997). Characteristics of child-teacher interactions, such as use of the teacher as a source of support and help, are related to gains in children’s performance in early childhood classrooms (Matsumura, Patthey-Chavez, Valdes, & Garnier, 2002; NICHD ECCRN, 2005; Nelson-LeGall & Resnick, 1998). Other research has shown that children’s relational styles with teachers uniquely and positively predict their social behavior with peers and academic skills in preschool, kindergarten, and other early grades (Howes & James, 2002; Howes et al., 2000; Ladd, 2004; Ritchie & Howes, 2003).

Language-based interactions with adults are also important for children’s development (Hart & Risely, 1995; Snow, 2006), with social-interactionist theory suggesting that language acquisition occurs through positive verbal interactions with adults (Baumwell, Tamis-LeMonda, & Bornstein, 1997; Chapman, 2000). There is a rich history of work that demonstrates the positive effects of teachers and parents engaging in formal and informal interactions that facilitate the use and understanding of language by young children (Huttenlocher, Vasilyeva, Cymerman, & Levine, 2002; McKeown & Beck, 2006; Vasilyeva, Huttenlocher, & Waterfall, 2006; Wasik et al., 2006). For example, vocabulary is an area of language weakness for children reared in poverty (Justice, Meier, & Walpole, 2005; Whitehurst & Lonigan, 1998) that can be accelerated through adult-child shared storybook reading (e.g., Hargrave & Sénéchal, 2000; Lonigan, Anthony, Bloomfield, Dyer, & Samwel, 1999; Penno, Wilkinson, & Moore, 2002).

It is also important to consider expressed conflict within teacher interactions, such as anger, resistance to connection, and discomfort. Research suggests that teacher-child conflict is linked with a range of negative school outcomes, including negative attitudes toward school, academic problems, and behavioral difficulties (Birch & Ladd, 1998; Hamre & Pianta, 2001; Howes et al., 2000; Werner & Smith, 1982).

Peers

In their comprehensive review of the peer relations literature, Gifford-Smith and Brownell (2003) document that a range of developmental outcomes, including academic achievement, have been associated with various aspects of peer functioning that begin to emerge in early childhood (e.g., social acceptance and peer network status). It is also well-established that children’s inability to successfully negotiate play interactions and develop friendships is linked to the development of externalizing and internalizing problems and school difficulties (Denham & Holt, 1993; DeRosier, Kupersmidt, & Patterson, 1994; Hatzichristou & Hopf, 1996; Lindsey, 2002; Ollendick, Weist, Borden, & Greene, 1992). The inCLASS addresses four specific dimensions of behavior within peer interactions, including sociability, assertiveness, communicative efficacy, and expressed conflict.

Initiation and maintainenance of positive social relationships with peers, referred to here as peer sociability, has proven to be a developmentally salient task within early educational settings (Guralnick, 1993; Ladd, 2005). Ladd and Price (1987) report that children’s observable social behavior with peers during preschool is a strong predictor of their adjustment to kindergarten, both in terms of how sociable they are perceived to be by teachers and how well liked they are by peers. More recent research has bolstered those early findings, making it evident that a child’s ability to seek out and maintain positive peer relationships in early childhood is a primary indicator of social competency (Fantuzzo, Bulotsky-Shearer, Fusco, & McWayne, 2005) and a consistent predictor of overall school readiness (Fabes, Martin, Hanish, Anders, & Madden-Derdich, 2003; Fantuzzo et al., 2005; Fantuzzo & McWayne, 2002).

Another important behavior within peer interactions is assertiveness, encompassing both initiation and leadership skills. Assertive behaviors in young children, such as proactively joining a play group or leading peers in a game, have been positively associated with peer sociability, self-esteem, and frustration tolerance (Adams, Ryan, Ketsetzis, & Keating, 2000; Fantuzzo & McWayne, 2002). In reviews of the literature on peer relations, Ladd (2005) and Kim (2003) identified assertiveness as a key dimension of healthy functioning with peers during early childhood.

Though most research has focused on children’s exposure to and use of language with parents and teachers, there is evidence that preschoolers are fully capable of initiating and sustaining conversations with peers (Schuele, Rice, & Wilcox, 1995). A few studies also suggest that peer interactions can contribute to young children’s language acquisition (McGregor, 2000; Schecter & Bye, 2007). And, peers’ expressive language within classroom peer interactions is increasingly advocated as one of the most valid and culturally sensitive means for examining children’s language use (e.g., Craig, Washington, & Thompson-Porter, 1998; Stockman, 1996), particularly for children whose culture or dialect may differ from that of the mainstream. However, few if any standard approaches exist for this type of assessment.

Finally, it is also important to consider any expressed conflict within peer interactions, including aggression, confrontation, negative affect, and attention-seeking behaviors. Research suggests that conflict and aggression can be problematic in the development of young children (Dodge, 1983), especially when considering implications for later peer relationships (Ostrov & Keating, 2004) and future social adjustment (Crick, 1996).

Tasks

Task orientation, referring to a child’s use of on-task, self-directed, and self-reliant behavior in managing the social and academic/learning demands of the classroom, has been established as an observable feature of young children’s development, which appears to play a role in their school success (Fantuzzo, Perry, & McDermott, 2004). For example, the National Center for Educational Statistics reports that children with a more positive approach to learning in kindergarten (e.g., child’s attentiveness, task persistence, eagerness to learn) show better pre-reading and math skills at the end of kindergarten and are more likely to be proficient in math skills by the spring of first grade (Denton & West, 2002). The inCLASS addresses two specific dimensions of task-oriented behavior, engagement and self-reliance, which refer to how children orient themselves in the classroom toward tasks and activities.

Research supports the examination of a child’s engagement within tasks and activities around the classroom (Matsumura et al., 2002). Perhaps most compelling is that academic growth is more successfully modeled when information is available about a child’s attention to and engagement with specific forms of instruction, rather than simply their exposure to instruction (Christian, Bachnan, & Morrison, 2001; Morrison & Connor, 2002). Attention to classroom tasks and activities has also been associated with prosocial behaviors such as comforting and helping other children, showing creativity in play, encouraging others to join play, and helping settle peer conflicts in a Head Start sample (Coolahan, Fantuzzo, Mendez, & McDermott, 2000).

Self-reliance, conceptualized here as the ability of a child to take learning into his or her own hands during activities and tasks in the classroom by taking initiative, showing independence and persistence, and making connections to previous events or ideas, is another key component of task orientation. The importance of these capacities is evident in the finding that self-reliance is a key mediator of children’s achievement during early schooling (NICHD ECCRN, 2008). Also, kindergarten teachers identify elements of self-reliance, such as independence and task persistence, as key factors during a child’s transition into school (Rimm-Kaufman, Pianta, & Cox, 2000).

How the inCLASS Complements Typical Early Childhood Assessments

There are many measures and tools available for educators and researchers to assess children’s competencies as related to school readiness. However, contemporary and historical approaches to measuring young children’s competencies in interactions with teachers, peers, and tasks have several limitations, highlighting a need in the field for complementary observational tools like the inCLASS. First, there is a notable mismatch between informant-based, retroactive methods of measuring preschool children’s social and emotional growth and the well-established finding that young children’s competencies are embedded within relationships and contexts. Observational approaches, which focus on children’s responses to situational demands, allow for examining how children calibrate their behavior in classroom interactions (Volpe, DiPerna, Hintze, & Shapiro, 2005) rather than defining competence in terms of the presence or absence of a specific, isolated behavior. As such, observations are somewhat more flexible and sensitive to individual ways in which children display their competence in a given context (Meisels & Atkins-Burnet, 2006). And, research suggests that observations may operate as predictors of later classroom adjustment in ways that decontextualized assessments of traits or behaviors do not (Gifford-Smith & Brownell, 2003; Sroufe, 1996). The inCLASS draws from the recent success of its parent measure, the Classroom Assessment Scoring System (CLASS; Pianta, La Paro, & Hamre, 2008), and developmental science research (e.g., NICHD ECCRN, 2008) which use global ratings to code complex classroom and interpersonal interactions. This approach to observing and documenting patterns of behavior provides a rich, contextualized assessment of classroom behaviors.

Second, although informant-based assessments of social, emotional, and behavioral outcomes are among the most widely used indicators of children’s readiness for school, meta-analyses and individual studies regularly report the association between teacher and parent report (the most commonly-used informants) to be low, in the range of 0.25 (see Achenbach, McConaughy, & Howell, 1987). Careful analysis of ratings made across situations and informants indicate the lack of correspondence between informants ratings can be interpreted to reflect properties of raters or of situations (Konold & Pianta, 2007; Kraemer et al., 2003), resulting in an imperfect estimate of a child behavior. A reliable observational assessment, like the inCLASS, therefore offers the opportunity to collect information that is complementary to these informant ratings.

Third, an observational measurement system focusing on classroom-based competencies is likely to provide information about children’s behaviors that has direct relevance to instruction and intervention in an early childhood education classroom setting (Meisels, 1997). The inCLASS observational system provides a comprehensive and detailed set of behavioral descriptors that can function as indicators of desired outcomes in early childhood. Such indicators, when mapped onto the classroom conditions that co-vary with, or give rise to their display, could provide interventionists and early childhood educators with a powerful tool for professional development and interventions, by linking observable indicators of classroom competence with the properties of classroom environments that support their display and development. This is an entirely different approach to classroom intervention (see Pianta, 1999) than is one that identifies behavioral problems (e.g., aggression) through teacher report and focuses on classroom interventions to reduce problems rather than enhance competence (e.g., prosocial behavior).

Study Aims and Hypotheses

For children between the ages of three and five, their “readiness” to function competently in school is perhaps best understood in terms of the nature and quality of their behavioral, social, and language-based interactions in early childhood classrooms. Observational methodology is considered “best practice” for authentically assessing these interactions as part of multi-method assessment protocols, yet historically researchers and schools rely on parent and teacher reports of these child competencies. The central purpose of this study was to develop and evaluate the inCLASS, an observational assessment of three-, four-, and five-year-olds competence during everyday interactions with teachers, peers, and tasks in a preschool classroom environment. The specific study aims were as follows:

1. To establish that observers could be trained in a short period of time to conduct inCLASS observations reliably, both live and on videotape;

2. To determine the extent to which inCLASS observations capture individual differences in children ages 3 to 5;

3. To evaluate the proposed three-factor structure of young children’s interactions using exploratory factor analysis;

4. To test whether inCLASS scores covary in expected directions with key sociodemographic variables, namely children’s age and gender; and,

5. To assess criterion-related validity of inCLASS observations with teacher ratings of children’s behavior.

It was hypothesized that moderate to excellent inter-rater reliability would be observed for the training videotapes and live observations across coders, and that inCLASS observational data would adequately reflect variability in children’s behavior. Furthermore, there was an expectation that inCLASS data would fit the hypothesized 3-factor structure, with high internal consistency for each of the resulting factors (Teacher Interactions, Peer Interactions, and Task Orientation; also referred to throughout as ‘domains’ of interaction). Given that the inCLASS was scaled to reflect typical developmental progress, it was expected that younger children would have lower scores than older children across all domains. And, boys were hypothesized to have lower scores than girls (Maccoby, 1998). Finally, it was expected that inCLASS observations would be associated with teacher ratings on similar constructs of behavior in appropriate directions (concurrent validity) and less related to dissimilar constructs of behavior (discriminant validity). For example, it was expected that observations of teacher-child interactions would be positively associated with teacher ratings of closeness, but not ratings of peer social skills.

Method

Participants

The sample consisted of 164 children, aged 3 to 5, enrolled in 20 different preschool programs in the central region of a mid-Atlantic state (see Table 2 for child, teacher, and classroom characteristics). Complete data were available for only 145 children (19 were absent for one of the classroom visits), and it is these 82 girls and 63 boys who were included in the current analyses. Although classrooms were 50% male, more consent forms were received for girls, and girls were more likely to be present across both visits, which led to this slight gender imbalance. Fifty-five of the participating children were three years old at the time of data collection, 73 were four, and 16 were five (one missing date of birth). The majority of children were of Caucasian race/ethnicity (91%) with the second largest group being African American (5%). The sample of participating children was similar in family (maternal education, family income, race/ethnicity) and classroom (teacher age, teacher education) demographic characteristics to the children who were excluded from analyses due to incomplete observation data, except that participating children were in somewhat smaller classrooms than excluded children (t[163] = 3.10, p ≤ 0.01).

Table 2.

Teacher, Classroom, and Child Demographics

Teacher demographicsa
	M	SD	Min	Max
Age (years)	42.7	9.4	24	61
Years experience pre-K	9.3	5.9	1	22
Classroom demographicsb
	M	SD	Min	Max
Class size	15.36	6.8	8	39
% male	50.4	11.7	20	75
% Caucasian	76.7	27.7	0	100
% African-American	5.2	12.0	0	56
% Asian	4.2	7.2	0	33
% Hispanic	1.9	4.0	0	17
Child demographics	Participatingc		All consentedd
	M	SD	M	SD
Age (yrs; mos)	4;1	0;8	4;0	0;8
Family income	$71,422	$20,829	$70,507	$21,671
Maternal education (years)	16.3	2.32	16.4	2.33

^aN = 40,

^bN = 44,

^cn = 145,

^dN = 295

The 44 classrooms were slightly more diverse, as reported by the teachers, than the sample of participating children, as noted in Table 2. Most of the classrooms (64%) reported no students with limited English proficiency (LEP); the remaining classrooms had between one and four LEP students. Similarly, most classrooms (84%) reported no students with known individualized education plans (IEP); the remaining classrooms had between one and three students with an IEP.

Forty lead teachers participated (some teachers led more than one participating classroom, since some classes only met on alternate days). All teachers were female, with 95% reporting Caucasian race/ethnicity. Twenty-three percent of teachers reported a bachelor’s degree as their highest level of education, and 28 percent had majored in early childhood education. Additional teacher demographics are also reported in Table 2.

Procedures

Recruitment

All 147 preschools, public and private, within a 40-mile radius of the research office were contacted. Principals/directors at 59 schools declined to participate, and those at 51 schools did not respond, while 37 preschools authorized participation. After permission was granted by the principal/director, lead teachers at each preschool were invited to participate in the project. A total of 47 teachers from 23 schools returned consents for their 51 classrooms (some teachers led two classrooms). Teachers were offered a choice of either a monetary stipend ($100) or a new video camera in compensation for their participation: allowing access to their classroom for observations, completing multiple teacher rating forms, and assisting with the parental consent process.

All parents or guardians in each participating classroom were given an informational consent letter and short family demographic survey through the preschool. Parents or guardians returned the consent form and survey to their child’s preschool teacher. Seven teachers in three schools chose not to participate in the study at this point because they felt they did not receive sufficient parental consents. In the final remaining pool of 20 schools, 40 teachers, and 44 classrooms, the response rate from parents was 44%, with a total of 291 children consented.

Of the parental consents received, four children were randomly selected from each classroom for participation: two girls and two boys whenever possible. Seven classrooms received fewer than four consents, in which case gender could not be balanced, and all consented children participated. Two classrooms had only three participants, and five classrooms had only two. Across all classrooms, a total of 164 children were selected for participation; of these, 145 children were present for both observation visits and therefore included in current analyses.

Training

All inCLASS observers were required to attend an intensive training session and reliably code video training clips before observing live in the field. Seven coders were trained on the inCLASS scale and protocol: two research scientists, four graduate students, and one undergraduate student. Training occupied two full days, and began with the trainer describing each domain and dimension in detail. All trainees watched five training clips (10 min. each), which they coded using the inCLASS manual and discussed extensively. At the end of training, all observers were required to code five reliability clips independently (without discussion), and had to score within one point of the mastercode on 80% of their scores to be deemed reliable and ready for live data collection. All training and reliability video clips were mastercoded by a group of researchers, educators, and designers of the inCLASS observation system in its early development. Inter-rater reliability was investigated in two ways: using results from these initial inCLASS training clips, as well as double-coded, live observations during data collection (also independent and without discussion; these will be discussed further in the results section). As a team, the coders were within one point of the mastercode 85% of the time across all five training videos (a range of 74 to 92% across the 9 dimensions). In addition, the intraclass correlation was 0.65, considered a good level for observational assessments (Cichetti & Sparrow, 1981).

Observation protocol

Two observational visits were made to each classroom in a 3–4 month period during the fall (typically one week apart, and not more than two weeks). Observations were scheduled at the teachers’ discretion and lasted for an entire morning, typically between 8 A.M. and 12 P.M. During this time, observers watched each of the participating children in turn, in a series of alternating 15-minute cycles, for an average of 16 observations (four per child). Observations continued throughout all activity settings: at each cycle, observers recorded relevant setting information, such as the type of activity (e.g., whole group, free choice/centers) and number of adults and children present in the room. The same children were observed during the first and second visits. After the first visit, teachers were provided with rating forms that assessed various aspects of functioning for each participating child, which they completed within two weeks of the initial observation.

Measures

Family demographics

A family demographic survey was adapted from one used in multiple studies as part of the National Center for Early Development and Learning’s (NCEDL) work in state-funded pre-kindergarten classrooms in 11 different states. The survey requested information about language spoken within the home, family income, highest level of education of the mother/female guardian living in this household, and other demographics.

Teacher and classroom demographics

The teacher and classroom demographic form was also patterned after NCEDL surveys and designed to provide information about teachers’ professional experience (e.g., education level, years teaching) and their classroom composition (e.g., gender, ethnicity, and language).

Teacher ratings of child competencies

The lead teacher in each classroom completed a survey packet on each participating child. The Academic Rating Scale (ARS; Rock & Pollack, 2002) was developed for the Early Childhood Longitudinal Study-Kindergarten Cohort (see National Center for Education Statistics, 2000). This study used the Literacy and Language subscale, which asks teachers to use a 5-point scale in rating children on items such as “Reads simple books independently.” The ARS shows excellent internal consistency (reported alpha above 0.90 in a range of analyses) and correlates above 0.70 with individual assessments of early literacy skills (see Perry & Meisels, 1996); the alpha in the current study was 0.94.

The Student-Teacher Relationship Scale (STRS; Pianta, 1992) examines teachers’ perspectives of their relationships with an individual child in their classroom. The 15-item, 5-point scale shortened form yields scores on conflict (assessing the degree of negative interactions and emotions involving the teacher and child) and closeness (assessing the degree of warmth, positive emotions, and open communication between the two) and has shown excellent psychometric properties across multiple studies and samples (Pianta, 1992; see also Pianta, 2001). Scale scores on the STRS predict teachers’ ratings of children’s classroom behavior, school retention for school-age children, and academic outcomes (Hamre & Pianta, 2001; Pianta, Steinberg, & Rollins, 1995). Internal consistency was 0.87 for both conflict and closeness scales in this study.

In the domain of social/relational and self-regulatory functioning, the Teacher-Child Rating Scale (TCRS; Hightower et al., 1986) is a 38-item measure that requires 5-point scale responses to items such as “Well liked by classmates” and “Copes well with failure.” The TCRS results in scores on problem behavior and competence scales, and has been widely used as part of multi-site batteries with children in the pre-kindergarten and kindergarten years. This measure shows excellent psychometric properties at these ages (see Bryant et al., 2002). In addition to the problem behavior scale (α = 0.89), there were four subscales from the competence scale utilized in the current study, all with excellent internal consistencies: frustration tolerance (e.g., statements about ignoring teasing or accepting limitations; α = 0.88), assertiveness (e.g., statements about being a leader or defending personal ideas; α = 0.86), task orientation (e.g., statements about completing work and being organized; α = 0.88), and social skills (e.g., statements about having and making friends; α = 0.93).

The California Preschool Social Competency Scales (CPSCS; Levine, Elzey, & Lewis, 1970) is a 30-item questionnaire asking the caregiver to rate the target child using 4-point scales. The items include measures of the child’s abilities to interact with peers, persist on tasks, follow instructions, communicate effectively, and respond confidently in unfamiliar situations. For the current study, a subset of 16 items was used that reflects social communication; these items yielded an alpha of 0.78 in the NICHD Study of Early Child Care and Youth Development sample at 54 months of age, and an alpha of 0.82 in the current sample.

Child observations

The Individualized Classroom Assessment Scoring System (inCLASS) is comprised of nine dimensions: Positive Engagement with the Teacher, Teacher Communication, Teacher Conflict, Peer Sociability, Peer Assertiveness, Peer Communication, Peer Conflict, Engagement within Tasks, and Self-Reliance. These dimensions and their behavioral markers were identified based on developmental and education literatures reviewed in the introduction (refer to Table 1 for definitions of each dimension, and relevant codeable behaviors). Following the effective approach to scaling of its parent measure, the CLASS (Pianta et al., 2008), the child receives a global score on a 7-pt. scale for each dimension based on the observation of specific behavioral markers, which are developmentally graded. The 7-pt. global ratings are designed to counter the isolated nature of frequency counts or time sampling approaches by allowing observers to make judgments about patterns of behavior across different situations. There is evidence that this type of integrative rating can be a particularly successful approach to capturing key aspects of adaptation with respect to the multiple dimensions of the classroom context (NICHD ECCRN, 2008).

Each inCLASS observation cycle lasts 15 minutes – ten minutes of watching and note taking, followed by five minutes of scoring. During the note-taking phase, observers write down instances of relevant behavior in each dimension. Then, during the scoring phase, observers refer to the inCLASS manual, and compare the behavioral descriptors it provides with their own observation notes to determine a score in each dimension. This is a multi-step process. First, using the face page for a dimension (see Appendix for a sample from Positive Engagement with Teacher), observers determine whether the behaviors observed fall within the High, Mid, or Low range for the dimension. Next, observers read through a detailed description of the selected range (e.g., High), which provides specific examples for each behavioral marker of the dimension (examples are drawn from multiple classroom settings). In the case of Positive Engagement, the markers are Attunement, Proximity Seeking, and Shared Positive Affect; the examples include such behaviors as joint smiling and laughing, maintaining eye contact, and responding emotionally to praise. The observer uses this full description to arrive at a final numerical score, by determining how well the description fits the behaviors observed. For example, a child whose behavior perfectly fit the High-range description for Positive Engagement with Teacher would receive a score of 7, while a child who mostly fit the High-range description, but had some behaviors in the Mid-range would receive a score of 6. A child who perfectly fit the Mid-range description would have a score of 4, and one who mostly fit the mid-range but had a few Low-range behaviors would receive a score of 3.

Data from each of the 15-minute observation cycles were averaged to obtain final dimensions scores for each child. Since two full-morning visits were completed for this study, observational ratings from a single coder (the lead coder in cases of double-coding) for each of the nine dimensions were averaged across all observed cycles from both visits. In addition, observers used checklists to record ecological factors that co-occurred with inCLASS ratings, such as the activity setting observed (e.g., whole group, free play), the physical setting (e.g., desk, floor, playground), the number of adults and children present, and whether the activity was teacher-directed.

Results

Exploratory Factor Analysis

An exploratory factor analysis was first conducted to identify the underlying factor structure of the inCLASS observational data. The factor analysis (and all subsequent correlational analyses) was conducted using both Pearson product-moment correlations and Spearman’s rho; results were parallel, so only Pearson correlations are reported throughout the Results section. In order to minimize cross loadings, allow factors to correlate, and simplify interpretation of factors, an oblique rotation was used (Tabachnick & Fidell, 2007). The rotated structure matrix (variance explained by the factor on both a unique and common contributions basis), eigenvalues, and percent of shared variance accounted for are presented in Table 3. Findings solidly supported a 3-factor model, with some evidence of a fourth factor. All items had loadings of at least 0.68, and no dimensions had cross-loadings above 0.53. The three main factors were as follows: Teacher Interactions (Positive Engagement with the Teacher and Teacher Communication), Peer Interactions (Peer Sociability, Peer Assertiveness, and Peer Communication), and Conflictual Interactions (Teacher Conflict and Peer Conflict). Though the eigenvalue for the fourth factor was lower than one, this factor fits with the hypothesized conceptual model, had adequate internal consistency (α = 0.72), and included dimensions that did not strongly load onto any of the other three factors. Therefore, Task Orientation (Engagement within Tasks and Self-Reliance) was retained as a fourth factor in subsequent reliability and validity analyses. Overall, this 4-factor model accounts for 85.71% of the variance in inCLASS observations.

Table 3.

Exploratory Factor Analysis of the inCLASS Dimensions (n=145)

Items	Teacher Interactions(α = 0.80)	Peer Interactions (α = 0.92)	Task Orientation (α = 0.72)	Conflict Interactions (α = 0.71)
Positive Engagement with Teacher	0.78	−0.15	0.15	−0.05
Teacher Communication	0.90	0.16	0.25	0.19
Peer Sociability	−0.08	0.90	0.44	0.12
Peer Assertiveness	0.05	0.88	0.50	0.33
Peer Communication	0.00	0.94	0.40	0.29
Engagement within Tasks	0.13	0.24	0.68	−0.25
Self-Reliance	0.23	0.53	0.88	0.05
Teacher Conflict	0.17	0.05	−0.29	0.82
Peer Conflict	−0.04	0.31	−0.03	0.75
Eigenvalue	1.85	3.35	0.78	1.74
% of variance	20.53	37.18	8.69	19.31
Cumulative %	20.53	57.71	66.40	85.71

Note: Extraction Method: Principal Axis Factoring, Rotation Method: Oblimin with Kaiser Normalization. indicates factor loadings above 0.68.

Inter-rater Reliability during Live Observations

Inter-rater reliability was calculated across 20% of all live classroom observations, as two coders observed and independently rated the same children. Coders were within one point of each others’ scores 87% of the time (with a range of 71% to 99% across the nine inCLASS dimensions). An intraclass correlation was also calculated across all dimensions and reached 0.84, within the excellent range according to standards in the field (Cicchetti & Sparrow, 1981). Intraclass correlations at the domain and dimension levels ranged from moderate to excellent (0.42 – 0.83; see Table 4 for details).

Table 4.

Descriptive Statistics and Inter-rater Reliability for the inCLASS (n=145)

Domain	Dimension	M	SD	Range		Inter-rater Reliability
				Min.	Max.	Within 1	Intraclass Correlation
Teacher Interactions		3.20	0.85	1.38	5.92	0.90	0.83
	Positive Engagement	3.61	0.86	1.62	6.33	0.93	0.81
	Teacher Communication	2.78	1.00	1.12	5.50	0.87	0.82
Peer Interactions		3.42	0.93	1.33	5.71	0.86	0.79
	Peer Sociability	4.26	0.86	2.00	6.17	0.87	0.72
	Peer Assertiveness	2.82	1.00	1.00	5.88	0.83	0.70
	Peer Communication	3.18	1.12	1.00	6.00	0.87	0.82
Task Orientation		4.64	0.64	2.08	6.08	0.77	0.57
	Engagement within Tasks	5.12	0.68	2.50	6.54	0.83	0.54
	Self-Reliance	4.15	0.76	1.67	5.83	0.71	0.50
Conflict Interactions		1.27	0.29	1.00	2.50	0.97	0.53
	Teacher Conflict	1.19	0.27	1.00	2.33	0.99	0.42
	Peer Conflict	1.36	0.38	1.00	3.00	0.95	0.57

Descriptives

Descriptive statistics for the inCLASS are presented in Table 4. Ranges reported here are based on average codes across two classroom visits and an average of eight 15-minute cycles. It is important to note that before aggregating across visits and cycles, codes generally covered the entire 1 to 7 rating scale (except for the Conflict dimensions, which ranged from 1 to 6). Even when scores were averaged per child across cycles, all dimensions ranged from a minimum of at least 2.5 to a maximum of at least 5.5; some, in fact, maintained a range as large as 1 to 6 (e.g., Peer Communication). Most inCLASS dimensions were observed with roughly normal distributions, though the Conflict dimensions were heavily positively skewed (with few instances of conflict observed).

Additionally, Pearson product-moment correlations by factor score (generated from the factor analysis to account for weighting of dimensions) and dimensions are presented in Table 5. The inCLASS items were generally correlated in expected directions and magnitudes, suggesting that the separate domains and dimensions of the inCLASS capture related yet distinct aspects of children’s competencies in the classroom.

Table 5.

Bivariate Pearson Correlations for the inCLASS by Factor Score and Dimension (n=145)

	1	2	3	4	5	6	7	8	9	10	11	12	13
1. Teacher Interactions		−0.03	0.23**	0.11	0.83***	0.96***	−0.09	0.05	0.00	0.14	0.25**	0.18*	−0.04
2. Peer Interactions			0.52***	0.30***	−0.16	0.17*	0.93***	0.91***	0.96***	0.24**	0.55***	0.05	0.32***
3. Task Orientation				−0.16*	0.16	0.28***	0.48***	0.55***	0.43***	0.74***	0.96***	−0.31***	−0.03
4. Conflict Interactions					−0.06	0.22**	0.13	0.37***	0.32***	−0.28***	0.06	0.91***	0.83***
5. Positive Engagement with Teacher						0.67***	−0.15	−0.10	−0.12	0.15	0.13	0.06	−0.12
6. Teacher Communication							0.07	0.24**	0.19*	0.13	0.32***	0.21**	0.07
7. Peer Sociability								0.78***	0.85***	0.25**	0.47***	−0.05	0.19*
8. Peer Assertiveness									0.82***	0.23**	0.59***	0.11	0.34***
9. Peer Communication										0.21**	0.46***	0.08	0.34***
10. Engagement within Tasks											0.57***	−0.31***	*0.12
11. Self-Reliance												−0.10	0.10
12. Teacher Conflict													0.58***
13. Peer Conflict

^*p ≤ 0.05,

^**p ≤ 0.01,

^***p ≤ 0.001

Construct Validity

Because one goal of the inCLASS is to identify individual differences in school readiness, it was also important to assess its sensitivity to gender and developmental differences across the sampled age range (3–5 year olds). There were no differences in inCLASS scores across boys and girls. However, age in months was positively correlated with both the Peer Interactions (r = 0.48, p ≤ 0.001) and Task Orientation (r = 0.22, p ≤ 0.01) domains. Age was not significantly correlated with the Teacher or Conflict Interactions domains.

Criterion-related Validity

To establish criterion-related validity, bivariate correlations were conducted to compare inCLASS Teacher Interactions, Peer Interactions, Task Orientation, and Conflict Interactions factor scores with teacher ratings from several established measures. Results from the correlational analyses generally supported the concurrent and discriminant validity of the inCLASS domains, some of which are reported below (see Table 6 for all correlations).

Table 6.

Bivariate Pearson Correlations for the inCLASS Factor Scores and Teacher Ratings (n=145)

	Teacher Interactions	Peer Interactions	Task Orientation	Conflict Interactions
Closeness (STRS)	0.25**	0.11	0.31***	−0.08
Conflict (STRS)	0.05	0.19*	−0.09	0.53***
Language and Literacy (ARS)	−0.08	0.31***	0.30***	−0.10
Social Communication (CPSCS)	−0.03	0.23**	0.37***	−0.27**
Frustration Tolerance (TCRS)	−0.10	−0.24**	0.02	−0.50***
Assertiveness (TCRS)	0.23**	0.41***	0.37***	0.17*
Task Orientation (TCRS)	−0.03	0.05	0.26**	−0.28**
Social Skills (TCRS)	0.01	0.16*	0.26**	−0.21**
Problem Behaviors (TCRS)	0.08	0.03	−0.28**	0.41***

Note: STRS = Student-Teacher Relationship Scale; ARS = Academic Rating Scale; CPSCS = California Preschool Social Competency Scale; TCRS = Teacher Child Rating Scale.

^*p ≤ 0.05,

^**p ≤ 0.01,

^***p ≤ 0.001

Within the Teacher Interactions domain, observations of the target child’s interactions with the teacher were positively correlated with teacher ratings of closeness with that child (r = 0.25) and teacher reports of assertiveness (r = 0.23, p’s ≤ .01). Within the Peer Interactions domain, significant correlations were found between inCLASS observations and teacher ratings of assertiveness (r = 0.41, p ≤ .01), social communication (r = 0.23, p ≤ .01), language and literacy skills (r = 0.31, p ≤ .01), and social skills (r = 0.16, p ≤ .05). Interestingly, observations of Peer Interactions were also positively related to teacher ratings of conflict (r = 0.19, p ≤ .05), and negatively correlated with teacher ratings of frustration tolerance (r = −0.24, p ≤ .01).

Within the Task Orientation domain, children who were observed as having higher quality interactions with classroom tasks and activities were also rated more highly (p ≤ .01) by their teacher on a host of skills, including task orientation (r = 0.26) and language and literacy skills (r = 0.30). In addition, ratings on the Task Orientation domain were negatively related to teacher ratings of problem behavior (r = −0.28). Observations within the Conflict Interactions domain were also significantly associated with similar teacher ratings. Moderately positive correlations were observed for reports of conflict and problem behaviors (r’s = 0.53, 0.41, p’s ≤ .001), whereas other significant associations in this domain were negative, such as frustration tolerance (r = −0.50). Unexpectedly, Conflict Interactions were positively associated with teacher ratings of assertiveness (r = .17, p ≤ .05). In summary, the pattern of these small to moderate correlations suggest preliminary criterion-related validity for the inCLASS.

Discussion

The purpose of this study was to develop and establish initial reliability and preliminary criterion-related validity of a new observation tool, the Individualized Classroom Assessment Scoring System (inCLASS). The inCLASS is a standardized observation protocol for use in preschool classrooms, with a focus on 3- to 5-year-olds’ developmentally salient interactions with teachers, peers, and tasks. Findings from the study suggest that the inCLASS has the potential to provide an authentic, contextualized assessment of young children’s classroom behaviors, with implications for complementing other early childhood education assessments, understanding children’s adjustment to classroom situations, and linking assessment with relevant intervention.

Inter-rater Reliability

One of the most challenging aspects of developing and applying classroom observation protocols is the process of training observers to reach a high standard of reliability (Volpe et al., 2005). Data from observations are only usable, and potentially valid, when assurances can be made that different observers are rating child behaviors in similar ways. The inCLASS manual and training protocol were based heavily on the past successes of similar observation systems (La Paro, Pianta, & Stuhlman, 2004; Pianta et al., 2008), with an emphasis on defining constructs in concrete, behavioral terms and using repeated exposure and practice with diverse samples of children’s videotaped behavior in naturalistic preschool environments. Our inter-rater reliability findings suggest that the manual and training protocol, as well as on-going drift testing with observers, were successful. Even the lowest intraclass correlations, for Teacher Conflict and Self-Reliance, were still soundly in the acceptable range, perhaps being more difficult to observe due to sparse incident rates for the former and the co-occurrence with peer interactions for the latter. Despite room for improvement that will come from minor scale revisions and even tighter training protocols, whether observing and coding child behaviors on training videotapes or during live classroom visits, observers were regularly seeing and rating children’s classroom interactions in parallel fashion. These solid inter-rater reliability findings permitted further examination of the measure and its validity.

Individual Differences

A central goal of the current study was to examine the extent to which the inCLASS dimensions adequately captured individual differences. Findings indicated sufficient variability in most of the inCLASS dimensions, with relatively normal distributions for the competence-oriented constructs. This distribution of competencies within classroom interactions, such as Peer Assertiveness and Self-Reliance with tasks, suggests that these observational data could provide meaningful information about individual differences in children’s behavioral responses to different classroom interactions. However, the two deficit-focused constructs, Teacher Conflict and Peer Conflict, had insufficient variability and significantly skewed distributions. This was not completely unexpected in that these dimensions include many low-incidence behaviors, such as physical aggression and active non-compliance, which have long been challenging to capture in observational assessments (e.g., Leff & Lakin, 2005; McEvoy, Estrem, Rodriguez, & Olson, 2003; Ostrov & Crick, 2007) and are often skewed even in teacher ratings (e.g., McEvoy et al., 2003). Revision of these dimensions to capture more subtle signs of conflict may improve variation in scores, as indeed may conducting observations in a larger and more diverse sample. Or, it may be that observation of these types of Teacher and Peer Interactions should be conducted during particular classroom activities, when increased demands require children to actively regulate their emotions and behavior.

Factor Structure

The aforementioned inCLASS dimensions were established after an extensive review of education and developmental science literatures, and reflected an a priori hypothesis that preschool children’s classroom behaviors relevant to school readiness tend to fall into three contextualized domains: Teacher Interactions, Peer Interactions, and Task Orientation. However, an initial exploratory factor analysis (EFA) suggested that a slightly different factor structure fit the data best. Indeed, there were three strong factors, encompassing Teacher Interaction, Peer Interaction, and an unexpected Conflict Interaction domains, with some evidence for a fourth factor - Task Orientation. Though contrary to hypotheses, it is not particularly surprising that the two dimensions focused on problem behaviors (Conflict with Teachers, Conflict with Peers) loaded onto an independent factor. These behaviors were clearly distinct from the other competency-focused interactions, and perhaps in these early childhood years dysregulated children are not yet differentiating their reactivity and behavior across teacher and peer contexts (NICHD ECCRN, 2004c).

Evidence for the Task Orientation domain is mixed. A lower eigenvalue in the EFA suggests that the Engagement within Tasks and Self-Reliance dimensions are not holding together as well as expected. And yet, the internal consistency of the Task Orientation domain is reasonable, suggesting that these two dimensions are detecting some signal among the noise. It could very well be that task-related interactions are naturally more difficult to disentangle from children’s other interactions in the classroom. For example, it was apparent that there is some overlap between the Peer Interactions and Task Orientation domains, as evidenced by cross-loadings for Self-Reliance and Peer Assertiveness. These cross-loadings and anecdotal feedback from observers suggest that it can be difficult to distinguish between these domains of interaction in the classroom, especially when a child is actively involved in a learning task with a heavy peer component. Given that there is a strong conceptual argument for Task Orientation being a distinct domain of classroom interactions with important implications for early school success (Fantuzzo et al., 2004), mixed findings point toward retaining it as an independent domain with the need for scale revisions to ensure that behaviors within Engagement toward Tasks and Self-Reliance are more purely task-oriented and distinguishable from Peer Interactions.

Developmental Trends

The inCLASS also demonstrated construct validity, with expected developmental trends observed across the age range sampled. Main effects of age on the measure were present for both the Peer Interactions and Task Orientation domains, indicating that older children were observed interacting more competently with peers and tasks than younger children. The Teacher and Conflict Interactions domains, however, did not capture age-related differences. For Teacher Interactions, it is possible that multiple and conflicting trajectories are in play, as children develop skills with age, but simultaneously become less dependent on the teacher, particularly as they increase and improve their peer interactions. And, there is a good chance that the restricted range of Conflict Interactions stunted the opportunity for the inCLASS to demonstrate developmental differences. Because this is a cross-sectional design, definitive conclusions about individual growth trajectories can not be made without intra-child modeling of inCLASS scores across three or more occasions. Additionally, the observed score differences may not be due to individual child maturation, but rather the different demands placed on children in older vs. younger classrooms or the possibility that older children had spent more time in a preschool classroom. Still, all considered, the detection of positive age effects on several inCLASS domains tentatively suggests that the measure captures important developmental differences in the competencies demonstrated by children within their preschool environment.

Criterion-related Validity

The current study lends support for the criterion-related validity of the inCLASS with teacher reports of early school readiness. In terms of concurrent validity, significant correlations were found between the inCLASS observations and teacher ratings on other well-established informant measures, and for the most part in hypothesized directions for all four domains of behaviors. Likewise, in terms of discriminant validity, the Teacher Interactions domain was largely unrelated to any task-, peer-, and conflict-focused rating scales, and a similarly targeted pattern was seen in the Peer Interactions observations. However, although related to conceptually aligned teacher ratings in task orientation and language/literacy skills, the observed Task Orientation domain was also more broadly associated with teacher ratings of children’s behavior oriented toward peers or teachers. This is perhaps suggestive of the fact that children who are outgoing and sociable may also carry this confidence and self-assuredness into focusing their attention and efforts on tasks and activities. Alternatively, the link between observed Task Orientation and a broad range of teacher ratings may reflect how interactions with teachers and peers are often embedded in or entangled with classroom tasks and activities. Similarly, Conflict Interactions were more globally linked to teacher ratings of children’s behavior, perhaps indicative that dysregulation cuts across types of interactions and settings.

There were also a few unexpected associations between inCLASS domains and teacher ratings of children’s behavior. Specifically, we found that children who demonstrated higher quality interactions with peers were also more likely to have conflict with teachers and less likely to tolerate frustration, whereas children observed to have more conflictual interactions were rated as more assertive by teachers. These results could be interpreted that children who are highly sociable with peers tend to continue engaging in peer conversations and interactions even when they are not supposed to do so, therefore leading to tension with the teacher around noncompliance. And, perhaps children who take the lead and speak up for their beliefs within peer interactions might also create tension and conflict by standing their ground and imposing their ideas on others. Although at first counterintuitive, these findings and interpretations are supported by past literature indicating that externalizing and prosocial behaviors sometimes travel together in young children (Ostrov, Pilat & Crick, 2006; Rudasill, Rimm-Kaufman, Justice & Pence, 2006; Zahn-Waxler, Cole, Richardson, Friedman, Michel, & Belouad, 1994). However, more research examining these specific relations between children’s classroom interactions and teachers’ ratings of their behavior is clearly warranted.

Limitations

This study is the first step in the development and validation of a new observational system focused on understanding how best to capture the developmentally salient interactions that children have within early childhood education classrooms. Findings provide initial support for the inCLASS as a reliable and valid observational assessment of children’s adjustment in preschool environments. Given that the development of the inCLASS is in its infancy, however, it is important to acknowledge several limitations that were not previously addressed.

The study sample was primarily homogenous, consisting mostly of three- and four-year old Caucasian children from families with moderate income levels, attending mostly private preschools with Caucasian teachers who agreed to participate from a larger pool of classrooms. Though the inCLASS was developed to be used across any early education or care setting for three-, four-, and five-year olds, it is difficult to discern the extent to which the tool may be applicable in at-risk samples and settings unrepresented in the current sample such as licensed family daycare, public pre-kindergartens, and kindergarten classrooms. For example, the distribution of domain scores might look different in a set of classrooms designed to serve economically disadvantaged preschoolers, such as having greater range in the Conflict domain, or in kindergarten classrooms with far more structure to the day. Future research should examine the inCLASS observation system in a wider array of early care settings with greater racial, ethnic and socioeconomic diversity at the child and teacher level, which we have already initiated in a large field trial across two urban cities.

Given skewed distributions of some inCLASS dimensions, it is clear that future revisions need to address the scaling of these dimensions. Specifically, the Teacher and Peer Conflict dimensions were observed at very low frequencies and had skewed distributions, as previously noted. For future studies, the scoring manual may need to be modified so that minor conflict behaviors are weighted more heavily, and subtle signs of conflictual relationships are included in scoring (such as low-level discomfort and avoidance). Any major conflict incidents, meanwhile, may require an automatic high level score. On the other hand, it is not uncommon for conflictual interactions to be difficult to capture during the course of naturalistic observation, and despite the restricted range, Conflict Interactions paralleled the perspectives of teachers who have a larger sample of behavior from which to draw.

Future Directions

The development of any new measure is an iterative process that involves establishing increasingly stringent psychometric properties and examining feasibility of practical applications. Given promising results from this initial foray into analyzing inCLASS data, a logical next step needs to involve exploration of stability and predictive validity. The current study involved multiple observations of children within and across days. More complex statistical modeling is needed to determine how many cycles of observation are necessary to provide a reliable estimate of a child’s behavior in the classroom at one point in the year (see Raudenbush & Sadoff, 2008 and Raudenbush & Sampson, 1999 for examples), as well as to determine stability across points in the year. This might point toward needing more cycles per child (Doll & Elliot, 1994) or additional days of observations during one week (Merrell, 2001), which could reduce the practical applicability of the observational tool. This is a critical next step toward establishing whether the inCLASS captures a stable estimate of children’s behavior, as well as determining the feasibility of its utility outside of research projects. In addition, it will be important to move beyond concurrent associations between observed and teacher-rated behaviors and study the extent to which inCLASS scores in the fall of preschool can predict changes in social and academic outcomes during the year and account for variance in kindergarten success.

Beyond serving as a valuable research tool to better understand children’s classroom behavior, there may be practical applications of the inCLASS. The inCLASS observations could produce behavioral profiles of children that guide teachers’ decisions about when and how to adjust their daily interactions to meet children’s individualized needs. Since inCLASS observers simultaneously code children’s behaviors and predominant activity settings, these profiles could also be refined to account for specific classroom contexts. For example, a boy may repeatedly be observed as low on Peer Interactions during free choice/centers, but higher during small groups in which a teacher is involved. This nuanced description of a child’s behavior across the common activity settings in a preschool classroom might uncover strengths and capabilities that are typically masked in global accounts of behavior and provide context-specific targets for classroom intervention. As potential practical applications of the inCLASS are explored, it will be important to determine exactly who collects the observational data. In particular, while attempting to balance efficiency, feasibility, and psychometric rigor, future efforts need to examine the extent to which observations must be conducted by independent, trained professionals (as was the case in the current study), or could be completed by teachers themselves.

It is difficult to extract school readiness from the context within which it unfolds and develops, which calls for approaches to assessment that take into consideration the fact that children’s competencies are reflected within interactions with the interpersonal and material resources available to them in a classroom. The inCLASS is an observational approach that is context-specific and focuses on dimensions of children’s adjustment to common situations in a preschool classroom with known links to academic and social outcomes. It therefore offers exciting opportunities to learn more about patterns of interaction that children experience during early classroom experiences and to capture information about adjustment that is tightly linked to the classroom setting in a way that could inform intervention.

Appendix. Positive Engagement

Reflects the degree to which the child is emotionally connected to teachers, seeking and apparently enjoying interactions with them. The use of the teacher as a “secure base” is important to this rating.

	Low (1, 2)	Mid (3, 4, 5)	High (6, 7)
Attunement Tracking Attention Shared experience Cooperation	Few indications of sharing attention or emotions with the teacher	Sometimes organizes behavior and emotions around the teacher (some tracking, unemotional attention).	Consistently organizes behavior and emotions around the teacher (attention on teacher, matches teacher’s emotional shifts, aware of teacher’s movement).
Proximity Seeking Physical Conversation Eye contact Response to presence	Few indications of seeking to be near the teacher (physical approach, conversation, or eye contact); no reaction when she approaches.	Occasionally seeks teacher (through physical approach, conversation, or eye contact), and reacts positively to teacher- initiated contact.	Maintains consistent contact through physical approach, conversation, or eye contact (depending on activity), and is reassured by such contact.
Shared Positive Affect Matching Reciprocity Response to affection	Very few indications of joint smiling and laughter, physical affection; flat or negative response to teacher affection.	Occasional indications of joint smiling and laughter, and generally positive response to affection from teacher.	Many indications of joint smiling and laughter, physical affection, and responding in relaxed/comforted way to teacher.