The Predictive Utility of a Brief Kindergarten Screening Measure of Child Behavior Problems

Abstract

Objective

Kindergarten teacher ratings, such as those from the Teacher Observation of Classroom Adaptation–Revised (TOCA-R), are a promising cost- and time-effective screening method to identify children at risk for later problems. Previous research with the TOCA-R has been mainly limited to outcomes in a single domain measured during elementary school. The goal of the current study was to examine the ability of TOCA-R sum scores to predict outcomes in multiple domains across distinct developmental periods (i.e., late childhood, middle adolescence, late adolescence).

Method

We used data from the Fast Track Project, a large multisite study with children at risk for conduct problems (n = 752; M age at start of study = 6.55 years; 57.7% male; 49.9% Caucasian, 46.3% African American). Kindergarten TOCA-R sum scores were used as the predictor in regression analyses; outcomes included school difficulties, externalizing diagnoses and symptom counts, and substance use.

Results

TOCA-R sum scores predicted school outcomes at all time points, diagnosis of ADHD in 9th grade, several externalizing disorder symptom counts, and cigarette use in 12th grade.

Conclusions

The findings demonstrate the predictive utility of the TOCA-R when examining outcomes within the school setting. Therefore, these results suggest the 10-item TOCA-R may provide a quick and accurate screening of children at risk for later problems. Implications for prevention and intervention programs are discussed.

Early behavior problems (e.g., aggressiveness, disruptiveness, oppositionality) are widely recognized as a risk factor for later violence and antisocial behavior (Bennett & Offord, 2001; Campbell, Shaw, & Gilliom, 2000). These early problematic behaviors are associated with several severe negative outcomes in later adolescence and adulthood, including school dropout and unemployment (Fergusson & Horwood, 1998; Jessor, 1998; Loeber & Dishion, 1983). Additionally, aggressive and disruptive behavior identified as early as kindergarten has been shown to predict later delinquent behavior and substance use (Hill, Lochman, Coie, Greenberg, & the Conduct Problems Prevention Research Group [CPPRG], 2004; Petras, Chilcoat, Leaf, Ialongo, & Kellam, 2004; Petras et al., 2005), suggesting that problem behaviors observed early in childhood can persist across development (Loeber, 1982; Loeber & Hay, 1997; Moffitt, 1993). Thus, early intervention aimed at high-risk children is important (Lochman & CPPRG, 1995), and the use of accurate and reliable screening tools to identify children for these targeted prevention programs is essential (Jones, Dodge, Foster, Nix, & CPPRG, 2002).

Brief screening measures distributed to parents, teachers, and other caregivers are often used as an initial step in identifying children who will benefit from prevention programs targeting early behavior problems (CPPRG, 1999; Feil, Walker, & Severson, 1995). Studies examining the predictive utility of these screening tools showed that combining parent and teacher ratings of behavior problems during kindergarten predicted more difficulties in interactions with peers and teachers (Wehby, Dodge, Valente, & CPPRG, 1993), higher levels of delinquency (Hill et al., 2004), and lower levels of social competence (Lochman & CPPRG, 1995) in first grade. Additionally, parent and teacher ratings of child aggressiveness and hyperactivity in preschool predicted child behavior problems at the end of preschool (Doctoroff & Arnold, 2004) and 5 years later (Stormont, 2000). However, these multireporter screening measures can be costly and time-consuming. It is therefore important to examine the predictive utility of brief single-rater screening measures, as these may represent a cost- and time-efficient method to identify children at risk for later behavior problems and associated negative outcomes.

Teacher Ratings of Early Behavior Problems

Teacher ratings are a promising resource for the implementation of brief, single-informant screening procedures. Focusing on teacher ratings in the school context provides a highly efficient and cost-effective method of identifying early behavior problems, as compared with gathering information from parents or peers, as a teacher is able to rate an entire classroom in one sitting (Petras et al., 2005). Furthermore, teacher ratings of aggressive and disruptive child behavior in first grade predicted difficulties in classroom behavior, academic achievement, and social adjustment in third grade (Flanagan, Bierman, Kam, & CPPRG, 2003). Additionally, teacher nominations were more accurate than parent nominations in predicting which children would develop behavior problems 1 year later (Dwyer, Nicholson, & Battistutta, 2006). This research suggests that teachers may be an ideal source for identifying children who are likely to show later behavior problems and may therefore benefit from participation in preventive interventions. Additionally, several large-scale intervention programs use teacher ratings as the first step in a multiple-gating assessment procedure (e.g., the Fast Track project; Lochman & CPPRG, 1995). Parents of children identified as higher risk based on these teacher ratings can then be contacted to provide their own ratings of their children’s behavior. Given the wide use of teacher ratings, it is important to determine the accuracy and predictive utility of these screening measures.

The Teacher Observation of Classroom Adaptation–Revised (TOCA-R; Werthamer-Larsson, Kellam, & Wheeler, 1991) is a commonly used teacher rating screening tool. The Authority Acceptance (AA) scale of this measure includes 10 items asking teachers to rate the frequency of their students’ behavior problems in the classroom. Scores on the TOCA-R collected during first through fifth grade have been shown to predict later (through age 18) violent and antisocial behavior in both boys and girls as identified in court records (Petras et al., 2004, 2005). In a recent study (Bradshaw, Schaeffer, Petras, & Ialongo, 2010), children were classified into one of three early starter aggressive-disruptive behavior trajectory groups on the basis of their TOCA-R scores during first through fifth grades: chronic high, low-moderate (for girls), or increasing (for boys). As compared with children who did not display behavior problems in elementary school, children in these early starter trajectory groups were found to be at risk for a greater number of negative nonaggressive life outcomes at age 19–20, including early pregnancy and unemployment (for girls) and high school dropout (for boys). Thus, the TOCA-R has demonstrated predictive validity to later problematic outcomes when examined across multiple assessment periods, but its utility as a screener at a single time point early in children’s development (e.g., prior to first grade) is less well established.

Only a few studies have examined the ability of screening measures like the TOCA-R to predict behavior beyond elementary school (cf. Bradshaw et al., 2010; Petras et al., 2004, 2005). Additionally, many of these studies have only examined outcomes in one particular domain or context (e.g., violent behavior, behavior in the classroom, nonaggressive life outcomes) and at one particular point in time. Thus, it is unknown how well early teacher ratings of aggressive and disruptive behaviors, assessed at a single time point, predict behaviors both within and external to the school context, and how these predictions may extend across different developmental periods. Furthermore, it remains unclear how broadly or narrowly this screening measure could be used. For instance, it could be that this measure is a broad predictor, capturing multiple outcomes (e.g., clinical diagnoses of externalizing disorders, substance use, school problems). Conversely, it could be that this measure is a narrow predictor, only addressing school outcomes. As a result of this uncertainty, the utility of this measure to identify children who would benefit from particular prevention programs is still unknown.

Goal of the Current Study

Previous studies examining the predictive validity of early screening measures of behavior problems such as the TOCA-R have been limited in that they have mostly examined outcomes in one domain or one context (e.g., behavior problems at school), and at one point in time (e.g., third grade). Few studies have examined how well ratings of early behavior problems collected at a single time point predict later negative outcomes and adjustment problems across multiple domains and across distinct developmental periods (i.e., late childhood, middle adolescence, and late adolescence). By examining a variety of outcomes across several developmental periods, it is possible to suggest more targeted interventions for at-risk children. For instance, it may be that high scores on the TOCA-R in kindergarten would predict more school-based behavior problems during late childhood and more delinquency and substance use problems during adolescence. These findings might suggest that for children identified as at risk based on TOCA-R ratings in kindergarten, school-based interventions addressing behavioral and emotional regulation in the classroom would be needed during childhood, whereas more community-based interventions addressing delinquency and substance use would be appropriate during adolescence. To date, no studies using early screening measures have addressed this issue regarding the timing and type of interventions indicated for at-risk children.

Therefore, the goal of the current study was to examine the ability of the TOCA-R measured during kindergarten to predict outcomes at the end of elementary (sixth grade), middle (eighth or ninth grade), and high (11th or 12th grade) school in the following domains: (a) school (behavior problems, social competence, and academic and disciplinary difficulties); (b) clinical diagnoses of externalizing disorders (i.e., attention-deficit/ hyperactivity disorder [ADHD], conduct disorder [CD], and oppositional defiant disorder [ODD]); and (c) substance use (at the end of middle and high school only). In sum, the goal of the current study was to comprehensively examine the prediction of multiple outcomes across multiple domains and developmental periods from the TOCA-R, administered at one time point in kindergarten. To date, no studies have provided this comprehensive evaluation of the predictive utility of the TOCA-R.

Method

Participants

Fast Track project

Participants came from a community-based sample of children drawn from the Fast Track project, a longitudinal multisite investigation of the development and prevention of childhood conduct problems (CPPRG, 1992, 2000). Schools within four sites (Durham, NC; Nashville, TN; Seattle, WA; and rural Pennsylvania) were identified as high risk based on crime and poverty statistics of the neighborhoods that they served. Within each site, schools were divided into sets matched for demographics (size, percentage free or reduced lunch, ethnic composition), and the sets were randomly assigned to control and intervention groups. Using a multiple-gating screening procedure that combined teacher and parent ratings of disruptive behavior, 9,594 kindergarteners across three cohorts (1991–1993) from 55 schools were screened initially for classroom conduct problems by teachers using the AA score of the TOCA-R (Werthamer-Larsson et al., 1991; see also Lochman & CPPRG, 1995, for more details regarding screening procedures). The AA scale of the TOCA-R includes 10 items asking teachers to rate the frequency of their students’ behavior problems in the classroom. Those children scoring in the top 40% on the TOCA-R within cohort and site were then solicited for the next stage of screening for home behavior problems by their parents, using items from the Child Behavior Checklist (Achenbach, 199la) and similar scales, and 91% agreed to participate (n = 3,274). The teacher and parent screening scores were then standardized and summed to yield a total severity-of-risk screen score. Children were selected for inclusion into the high-risk sample on the basis of this screen score, moving from the highest score downward until desired sample sizes were reached within sites, cohorts, and groups. Deviations were made when a child failed to matriculate in the first grade at a core school (n = 59) or refused to participate (n = 75) or to accommodate a rule that no child would be the only girl in an intervention group. The outcome was that 891 children (control = 446, intervention = 445) participated.

In addition to the high-risk sample of 891 children, a stratified normative sample of 387 children was identified to represent the population-normative range of risk scores and was followed over time. This normative sample was selected from the control schools, such that 100 kindergarten children were selected at each site (except for Seattle, WA, where only 87 children were selected). Participants in the normative sample were stratified to represent the population according to race, gender, and level of teacher-reported behavior problems (10 children at each decile of the distribution of scores from the TOCA-R). The normative sample included a portion of high-risk control group children to the proportional degree that they were represented in the school population. Written consent from parents and verbal assent from children were obtained. Parents were paid $75 for completing the summer interviews, and teachers were compensated $10 per child for completing the measures. The Institutional Review Boards of the participating universities approved all study procedures.

Sample description

The current study used data from the high-risk control and normative groups. Participants from the high-risk intervention sample were not included in this study. Because 79 of those recruited for the high-risk control group were also included as part of the normative sample, the total sample included 754 participants. However, two children in this sample were missing TOCA-R scores and were therefore excluded from the current analyses, yielding a final sample of 752 children. Children were, on average, 6.55 years old (SD = .43) at the start of the Fast Track project. As would be expected given the higher prevalence of conduct problems documented among boys as compared with girls (Hinshaw & Lee, 2003), 57.7% of the sample was male. Reflecting the ethnic diversity in the populations at the four sites, the majority of the sample was either Caucasian (49.9%) or African American (46.3%), with 3.8% of the sample representing other ethnic groups (e.g., Hispanic, Asian). Due to the multisite sampling design of the Fast Track project, race and urban/rural status were confounded, as nearly all of the African American participants lived in urban areas. In fact, less than 1% of the entire sample consisted of African Americans living in rural communities. Thus, for the current study, a race/urban status variable was examined representing three groups: urban African Americans (46.0%) urban Caucasians (24.2%), and rural Caucasians (25.7%). For analyses examining ethnicity and race/urban status, other ethnic minorities were not included due to the small sample sizes in these groups. For this final sample of 752 children, 147 teachers provided TOCA-R ratings in kindergarten.

Procedure

Annual home interviews were conducted with primary caregivers (typically mothers) and children. Interviews began during the summer before children’s entry to first grade and concluded 2 years after the child completed (or would have completed) 12^th grade. Caregivers and children completed the interviews separately with two different interviewers over the course of approximately 2 hr. Measures given during these interviews assessed several domains, including parenting behaviors, child behavior problems, family functioning, parent–child relationship quality, peer relationships, academic achievement, and characteristics of the broader neighborhood. Measures included in the current study are described below.

We chose the specific grades included in this study because they aligned with assessments from the Fast Track project that fell within the developmental periods we aimed to address. The timing of assessments during the Fast Track project also took into account the length of the assessment battery at each year. For instance, in years when the lengthy Computerized Diagnostic Interview Schedule for Children (CDISC; Shaffer & Fisher, 1997) was administered, the remainder of the assessment battery was trimmed significantly to reduce participant burden. For this reason, not all measures were administered at all years, leading to the uneven timing of assessments.

Measures

The TOCA-R

The AA scale of the TOCA-R (Werthamer-Larsson et al., 1991) assessed during kindergarten was used as the predictor in all analyses. The entire TOCA-R is a 43-item questionnaire designed for teachers to assess authority and acceptance behavior, concentration problems, and shy behavior relevant to the child’s behavior in a classroom situation on a 6-point Likert-type scale (0 = never, 1 = rarely, 2 = sometimes, 3 = often, 4 = very often, and 5 = always). The AA scale includes 10 items representing aggressive and disruptive behavior problems (e.g., “breaks rules,” “harms others,” and “takes others’ property”). Scores on the TOCA-R are commonly summed to create a composite sum score (e.g., Petras et al., 2004, 2005). Therefore, for the current study, sum scores were created on the basis of the 10-item AA scale of the TOCA-R (hereafter referred to as TOCA-R sum scores).

School outcomes

Several school-related outcomes were measured at sixth, eighth, and 11th grades, including behavior problems in school, social competence, and academic and disciplinary difficulties. Teacher-rated child behavior problems in school were measured in sixth and eighth grade with the T-score of the Externalizing subscale of the Teacher’s Report Form (TRF; Achenbach, 1991b). The Externalizing subscale of the TRF is a 34-item measure that asked teachers to report on the child’s level of multiple problematic behaviors (e.g., disobedient, disruptive, physically aggressive, explosive, stubborn, truant). Items were scored on a 3-point Likert scale, ranging from 0 (not true [as far as you know]) to 2 (very true or often true). Cronbach’s alpha coefficients for this subscale were .96 in sixth grade and .97 in eighth grade, indicating strong internal consistency.

Teacher-rated social competence in sixth and eighth grade was measured with the Teacher Social Competence (TSC) scale, which was developed by the Fast Track project (CPPRG, 1995). The TSC is a 17-item measure that assessed child competence in academic behavior, prosocial skills, and emotional regulation (e.g., performs academically at grade level, handles disagreements in a positive way, cooperates with others, initiates interactions in a positive manner, recognizes and labels feelings, stops and calms down when excited). Items on this measure asked teachers to rate the frequency of these social behaviors on a 6-point scale, ranging from 0 (almost never) to 5 (almost always). The total score on the TSC was calculated as the mean of all 17 items. Internal consistency of the TSC was strong (Cronbach’s α = .94 in sixth grade and .95 in eighth grade).

Parent- and child-reported academic and disciplinary difficulties in sixth, eighth, and 11th grade were measured with the School Adjustment scale, which was developed by the Fast Track project (CPPRG, 1997a, 1997b). The parent report version of this measure included 18 items that evaluated the child’s past school year in terms of academic performance, disciplinary problems, and general worries about school (e.g., school year difficult for child, school work was really hard for child, other kids tried to make child do bad things, child got into trouble by breaking rules). Two items related to the parent’s contact with the school and teachers were not included in this study. This measure used a 5-point response scale, ranging from 1 (strongly disagree) to 5 (strongly agree). The total academic and disciplinary difficulties-parent report score was calculated as the mean of all 16 items. Cronbach’s alpha coefficients were .90 in sixth and eighth grades and .84 in 11th grade, indicating adequate internal consistency.

The child report version of the School Adjustment scale included 20 items related to the child’s academic and disciplinary difficulties, relationships with other students, and general aspects about the school and teachers. The subscale pertaining to academic and disciplinary difficulties was used for the current study. Example items on this subscale included “the school year was difficult,” “school work was really hard,” “I got into trouble this year,” and “teachers were on me because I broke rules.” The eight items on this subscale were on a 5-point scale ranging from 1 (never true) to 5 (always true). The total academic and disciplinary difficulties-child report score was calculated as the mean of all eight items. Internal consistency of this subscale was adequate (Cronbach’s α = .75 in sixth grade, .76 in eighth grade, and .74 in 11th grade).

Externalizing disorders

The externalizing diagnoses considered for this study included ADHD (combined type), ODD, and CD. Diagnoses of externalizing disorders in sixth, ninth, and 12th grade were assessed with the CDISC (Shaffer & Fisher, 1997). The CDISC is widely used to assess Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, psychiatric symptoms and diagnoses in children and adolescents aged 6–17 years. We analyzed both binary diagnosis variables (i.e., child met diagnostic criteria for a particular disorder or not) and symptom count variables (i.e., a count of how many symptoms of a particular disorder were endorsed). Both parent- and child report versions of this measure were used in the current study, such that a child was considered to have displayed a particular symptom or received a diagnosis in the past year if either the parent or the child or both endorsed that symptom or if the report of the parent or the child or both met criteria for a diagnosis. This approach was used because previous research has shown that parents and children provide unique information regarding ADHD, ODD, and CD diagnostic criteria (Colins, Vermeiren, Schutyen, Broekaert, & Soyez, 2008). For analyses, we examined both diagnoses and symptom counts for each disorder for each grade separately and combined (i.e., a diagnosis of ADHD, ODD, CD, or any externalizing disorder at sixth, ninth, and 12th grade; total symptom counts of ADHD, ODD, CD, or any externalizing disorder at sixth, ninth, and 12th grade).

Substance use

Information regarding children’s substance use was collected with the Tobacco, Alcohol and Drugs measure. This measure was adapted by the Fast Track project from the substance use section of the Self-Administered Youth Questionnaire from the National Longitudinal Survey of Youth 1997 (Bureau of Labor Statistics, 2002; Elliot, Huizinga, & Ageton, 1985). We examined child report of tobacco, alcohol, and marijuana use at eighth and 12th grade. Use of other illicit drugs was not included due to extremely low endorsement of these substances in the sample. For analyses, we examined the number of days children reported smoking cigarettes in the past month (range = 0–30), the number of days they reported drinking alcohol in the past year (range = 0–365), and the number of times children used marijuana in the past month (range = 0–100). Additionally, we examined the number of drinks children reported drinking each time they drank alcohol as well as the number of days children engaged in binge drinking (had 5+ drinks in a row) in the past year (range = 0–365). We also multiplied the number of days children drank alcohol with the number of drinks children had each time they drank alcohol to obtain a measure of intensity of alcohol use. All substance use variables were square-root transformed for analyses due to nonnormality and skewness in the data (Tabachnick & Fidell, 2001).

Missing data

The current analyses examined school outcomes measured at the end of sixth, eighth, and 11th grades; externalizing diagnoses measured at the end of sixth, ninth, and 12th grades; and substance use measured at the end of eighth and 12th grades. For school outcomes, 228 (30.3%) children were missing data in sixth grade, 259 (34.4%) in eighth grade, and 282 (37.5%) in 11th grade. For externalizing diagnoses, 119 (15.8%) were missing data in sixth grade, 185 (24.6%) in ninth grade, and 245 (32.6%) in 12th grade. For substance use, 179 (23.8%) were missing data in eighth grade and 205 (27.3%) in 12th grade. Children with missing data on school outcomes were more likely to be in the high-risk control group than in the normative group: sixth grade, χ²(1, N = 752) = 6.16, p < .05; eighth grade, χ²(1, N = 752) = 6.30,p < .05; 11th grade, χ²(1, N = 752) = 6.39,p < .05. For externalizing diagnoses, children with missing data in 12th grade were more likely to be urban African American, χ²(2, N = 752) = 9.39, p < .01. Additionally, for substance use, children with missing data in 12th grade were more likely to be in the high-risk control group, χ²(1, N = 752) = 16.08, p < .001; urban African American, χ²(2, N = 752) = 10.43, p < .01; and male, χ²(1, N = 752) = 3.94, p < .05. There were no other significant differences in attrition by group (high-risk control vs. normative), race/urban status, gender, or TOCA-R sum score at any of the other time points.

Analysis Plan

Analyses were conducted in SPSS version 14.0 (for descriptive statistics) and Mplus version 6.0 (Muthén & Muthén, 2010). To account for the oversampling of high-risk children in the Fast Track project and to increase generalizability to the population, we used a probability weight based on group (normative vs. high-risk control) that had been previously calculated for all normative and high-risk control group participants (see Jones et al., 2002, for a description of the creation and calculation of this weighting variable). Gender, race/urban status, and risk group (normative vs. high-risk control) were included as covariates in all regression analyses. The research design of the Fast Track project involved children who were nested within classrooms. For example, the 891 high-risk children recruited for this project were nested within 401 first-grade classrooms. However, by Grade 3, these same children were nested in 527 classrooms due to transfers and relocations, and some intervention and control children were in the same classrooms at that time. Therefore, we determined that it was not appropriate or possible to account for nesting within classrooms or schools in the outcome data examined for this study (see CPPRG, 2002, for further details on the nested structure of the data in the Fast Track project).

A series of linear regression analyses were conducted to test the prediction of school outcomes (behavior problems, social competence, and adjustment) from the TOCA-R sum scores. Covariates and TOCA-R sum scores were entered simultaneously as predictors of the various outcomes. Full information maximum likelihood was used to handle missing data (amount of missing data ranged from 15.8% to 37.5% across the variables). We used a maximum likelihood estimator that calculated robust standard errors, which provides valid standard error estimates when variables are nonnormal (Asparouhov & Muthén, 2006). For the prediction of externalizing diagnoses (i.e., ADHD, ODD, and CD), we conducted a series of logistic regressions for the binary diagnostic variables as well as linear regression analyses for the continuous symptom count variables. To predict substance use outcomes, we used a series of censored linear regressions. We used censored regression because the distributions of the substance use variables were continuous, positively skewed with an abundance of zeroes, and theoretically left-censored at zero. Censored regressions are used commonly with variables representing behaviors that do not occur frequently in the general population (e.g., substance use among children, serious delinquency), as data transformations that attempt to normalize the distribution of variables are ineffective in managing an abundance of zeros (Long, 1997). The proportion of variance explained (R²) was used as a measure of the effect sizes for the TOCA-R sum scores in the prediction of outcomes (Cohen, Cohen, West, & Aiken, 2003).

Results

Descriptive Statistics

Table 1 provides sample sizes, means, and standard deviations for all continuous outcomes for each year measured. For the binary diagnosis outcomes, the percentage and number of children receiving that diagnosis are presented for each year measured. The zero-order correlations between the TOCA-R sum scores and the school outcomes and between the TOCA-R sum scores and externalizing disorder outcomes are presented in Table 2 and Table 3, respectively. As seen in these tables, the TOCA-R sum scores were significantly correlated with almost all outcomes at all time points. Specifically, higher TOCA-R sum scores were associated with more teacher-reported behavior problems, lower teacher-reported social competence, and lower parent- and child-reported school adjustment. In terms of externalizing diagnoses, higher TOCA-R sum scores were related to higher diagnosis rates (with the exception of diagnoses of CD in 12th grade) and symptom counts across all disorders.

Table 1.

Descriptive Statistics of Outcome Variables Measured at the End of Elementary, Middle, and High School

Variable	6th grade M (SD)/%	8th grade M (SD)/%	9th grade M (SD)/%	11th grade M (SD)/%	12th grade M (SD)/%
Teacher-rated behavior problems	60.38 (10.69)a	59.22 (11.14)b
Teacher-rated social competence	2.80 (1.05)a	2.80 (1.05)a
Parent-rated school adjustment	3.39 (.74)a	3.39 (.73)a		3.41 (.60)a
Child-rated school adjustment	3.54 (.70)a	3.62 (.67)b		3.74 (.62)c
ADHD diagnosis	8.8% (n = 66)a		6.5% (n = 48)a		3.6% (n = 27)a
CD diagnosis	7.8% (n = 59)a		7.7% (n = 58)a		4.9% (n = 37)a
ODD diagnosis	12.5% (n = 94)a		10.0% (n = 75)a		6.9% (n = 52)a
ADHD symptom count	5.53 (4.13)a		3.97 (4.96)b		2.53 (3.90)c
CD symptom count	2.83 (2.76)a		3.75 (3.30)b		4.14 (3.85)c
ODD symptom count	4.13 (2.58)a		4.15 (2.63)a		3.36 (2.46)b
Days smoked cigarettes past montha		.50 (.05)			1.64 (.10)
Days drank alcohol past yeara		.52 (.06)a			1.63 (.11)b
Number of drinks each time past yeara		.40 (.04)a			.99 (.05)b
Days had 5+ drinks in a row past yeara		.25 (.04)a			.84 (.08)b
Alcohol intensity (Days × Drinks)a		1.08 (.15)a			3.71 (.31)b
Used marijuana past montha		.18 (.03)a			.69 (.07)b

Note. Means in the same row with different subscripts differ at p < .05 in t -test comparisons (z-test comparisons for percentages). ADHD = attention-deficit/hyperactivity disorder; CD = conduct disorder; ODD = oppositional defiant disorder.

^aVariables are square-root transformed for all analyses.

Table 2.

Intercorrelations Between TOCA-R and School Outcomes Measured at the End of Elementary, Middle, and High School

Variable	1	2	3	4	5	6	7	8	9	10	11
1. TOCA-R	—
2. Bx problems (TR): Grade 6	.41***	—
3. Bx problems (TR): Grade 8	.35***	.57***	—
4. Social comp (TR): Grade 6	−.38***	−.78***	−.52***	—
5. Social comp (TR): Grade 8	−.33***	−.51***	−.76***	.56***	—
6. School adjust (PR): Grade 6	− 29***	−.46***	−.38***	.57***	.42***	—
7. School adjust (PR): Grade 8	−.27***	−.40***	−.54***	.44***	.58***	.50***	—
8. School adjust (PR): Grade 11	−.41***	−.34***	−.36***	.42***	.36***	.36***	.48***	—
9. School adjust (CR): Grade 6	−.31***	−.44***	−.37***	.49***	.41***	.52***	.40***	.28***	—
10. School adjust (CR): Grade 8	−.21***	−.38***	−.46***	.39***	.49***	.34***	.52***	.29***	.51***	—
11. School adjust (CR): Grade 11	−.16***	−.28***	−.25***	.35***	.37***	.25***	.35***	.32***	.42***	.48***	—

Note. TOCA-R = Teacher Observation of Classroom Adaptation-Revised; Bx = Behavior; TR = teacher rated; comp = competence; adjust = adjustment; PR = parent rated; CR = child rated.

^***p < .001.

Table 3.

Intercorrelations Between TOCA-R and Externalizing Disorders Measured at the End of Elementary, Middle, and High School

Variable	1	2	3	4	5	6	7	8	9	10
1. TOCA-R	—	.19***	.23***	.13**	.08*	.13**	.06	.15***	.13**	.09*
2. ADHD: Grade 6	28	—	.33***	.34***	.18***	.10*	.04	.35***	.21***	.25***
3. ADHD: Grade 9	.26***	.51***	—	.40***	.13**	.24***	.16***	.21***	.35***	.25***
4. ADHD: Grade 12	.23***	.46***	.54***	—	.14**	.23***	.13**	.21***	.18***	.35***
5. CD: Grade 6	.30***	.54***	.39***	.30***	—	.37***	.30***	.26***	.19***	.10*
6. CD: Grade 9	.29***	.44***	.49***	.33***	.64***	—	.23***	.19***	.37***	.17***
7. CD: Grade 12	.24***	.40***	.44***	.47***	.61***	.65***	—	.20***	.16***	.21***
8. ODD: Grade 6	.15***	.63***	.35***	.29***	.56***	.38***	.39***	—	.33***	.23***
9. ODD: Grade 9	.10*	.34***	.52***	.32***	.37***	.61***	.44***	.50***	—	.25***
10. ODD: Grade 12	.09*	.30***	.38***	.53***	.25***	.34***	.54***	.41***	.51***	—

Note. Numbers above the diagonal represent nonparametric correlations (Spearman’s) of externalizing diagnoses (yes/no); numbers below the diagonal represent parametric correlations of externalizing disorder symptom counts. TOCA-R = Teacher Observation of Classroom Adaptation–Revised; ADHD = attention-deficit/hyperactivity disorder; CD = conduct disorder; ODD = oppositional defiant disorder.

^*p < .05.

^**p < .01.

^***p < .001.

Covariates

Many of the covariates included in the regression models were significant predictors of several of the outcomes included in the current study. The following is a summary of the most consistent covariate effects observed in the analyses (the full results are available from the first author). For teacher-rated school outcomes, children living in urban areas (urban Caucasians and urban African Americans) were rated as higher on behavior problems as compared with rural Caucasians (β = .19,p < .001; β = .30,p < .001, for sixth and eighth grade, respectively) as were urban African Americans when compared with urban Caucasians (βs = −.22, ps < .001 for both sixth and eighth grade). Additionally, children living in urban areas were rated as lower on social competence as compared with rural Caucasians (β = −.18, p < .001; β = −.25, p < .001, for sixth and eighth grade, respectively), as were urban African Americans when compared with urban Caucasians (β = .28, p < .001; β = .26, p < .001, for sixth and eighth grade, respectively). For child-rated school outcomes, children living in urban areas were lower on school adjustment than rural Caucasians (β = −.15, p<.01;β = − .21, p < .001; β = −.15, p < .05, for sixth, eighth, and 11th grade, respectively), and females were higher on school adjustment than males (β = .20, p < .001; β = .18, p < .01; β = .21, p < .01, for sixth, eighth, and 11th grade, respectively).

Males were also consistently higher on symptom counts of ADHD and CD as compared with females (βs range from −.12 to −.30, all ps < .05). Additionally, children living in urban areas had higher symptom counts of CD than rural Caucasians (β = .18, p < .001; β = .21, p < .001; β = .11, p < .05, for sixth, ninth, and 12th grade, respectively), as did urban African Americans when compared with urban Caucasians (β = −.11, p < .05; β = −.12, p < .05; β = −.10, p < .05, for sixth, ninth, and 12th grade, respectively). For substance use in 12th grade, males reported higher levels of use (except for number of drinks consumed each time they drank alcohol, where there was no gender difference) than females (β s range from −.14 to −.26, allps < .05).

School Outcomes

Table 4 presents the results from the linear regression analyses predicting school outcomes at the end of elementary, middle, and high school from the TOCA-R sum scores. Above and beyond the covariates, teacher-reported behavior problems during kindergarten were prospectively associated with almost all teacher-, parent- and child-rated school outcomes at sixth, eighth, and 11th grades; however, child-rated school adjustment in eighth and 11th grades were not significantly associated. For example, a one standard deviation increase in the TOCA-R sum score predicted a 0.32-standard deviaion increase in teacher-rated behavior problems in sixth grade and a 0.29-standard deviation unit increase in teacher-rated behavior problems in eighth grade. As seen in Table 4, higher kindergarten TOCA-R sum scores predicted higher levels of teacher-rated child behavior problems and lower teacher-rated child social competence at sixth and eighth grades. Higher TOCA-R sum scores also predicted lower parent- and child-rated school adjustment at sixth, eighth, and 11th grades (however, the school adjustment coefficients at eighth and 11th grades were significant according to parent report only). Across reporters and time points, early teacher ratings of behavior problems using the TOCA-R explained approximately 2%–5% of the variance in later behavior problems at school, social competence, and school adjustment, after controlling for the covariates.

Table 4.

Linear Regression Analyses of TOCA-R Sum Scores Predicting School Outcomes

	TOCA-R sum scores
Variable	ΔR2	β	95% CI for β
Teacher-rated outcomes
Behavior problems: Grade 6	.05***	.32***	[.19, .46]
Behavior problems: Grade 8	.04***	.29***	[.18, .39]
Social competence: Grade 6	.02**	−.22**	[−.35, −.09]
Social competence: Grade 8	.04***	−.28***	[−.39, −.17]
Parent-rated outcomes
School adjustment: Grade 6	.01*	−.13*	[−.24, −.02]
School adjustment: Grade 8	.01*	−.16*	[−.29, −.02]
School adjustment: Grade 11	.01*	−.15*	[−.29, −.01]
Child-rated outcomes
School adjustment: Grade 6	.03***	−.22**	[−.34, −.09]
School adjustment: Grade 8	.01*	−.12	[−.26, .01]
School adjustment: Grade 11	.00	−.06	[−.20, .08]

Note. Covariates entered at Step 1: gender, race/urban status, and group. TOCA-R = Teacher Observation of Classroom Adaptation-Revised; CI = confidence interval.

^*p < .05.

^**p < .01.

^***p < .001.

Diagnosis of Externalizing Disorders (ADHD, ODD, and CD)

Of the externalizing disorders, the TOCA-R sum scores predicted only diagnosis of ADHD in ninth grade (β = .30, SE = .12, p < .01, OR = 1.09, 95% CI for OR [1.02, 1.16]). In other words, the odds of receiving an ADHD diagnosis in ninth grade increased between 1.02 and 1.16 times for every one-unit increase in the TOCA-R sum score. The TOCA-R sum scores did not predict any other diagnosis at any grade, nor did they predict any cumulative diagnosis of ADHD, CD, ODD, or any externalizing disorder (ADHD, CD, and ODD combined).

As expected, utilizing symptom counts, compared with predicting dichotomous diagnoses, provided better predictive utility (MacCallum, Zhang, Preacher, & Rucker, 2002). As seen in Table 5, after controlling for the covariates, TOCA-R sum scores in kindergarten prospectively predicted the number of ADHD symptoms endorsed by the parent or the child or both in sixth grade, such that a one-standard deviation increase in TOCA-R sum scores predicted a 0.14-standard deviation unit increase in ADHD symptoms in sixth grade. Additionally, a one-standard deviation increase in TOCA-R sum scores predicted a 0.14-standard deviation unit increase in ninth grade CD symptoms, a 0.12-standard deviation unit increase in total ADHD symptoms (sixth, ninth, and 12th grades combined), and a 0.12-standard deviation unit increase in total externalizing symptoms (ADHD, CD, and ODD combined) across all three grades combined.

Table 5.

Linear Regression Analyses of TOCA-R Sum Scores Predicting Externalizing Disorder Symptom Counts

	TOCA-R sum scores
Variable	ΔR2	β	95% CI for β
ADHD: Grade 6	.01*	.14*	[.03, .25]
ADHD: Grade 9	.01*	.11	[−.02, .23]
ADHD: Grade 12	.00	.08	[−.08, .23]
CD: Grade 6	.01	.11	[−.03, .24]
CD: Grade 9	.01*	.14*	[.01, .26]
CD: Grade 12	.01	.13	[−.02, .28]
ODD: Grade 6	.00	.03	[−.09, .16]
ODD: Grade 9	.00	.07	[−.06, .19]
ODD: Grade 12	.01	.11	[−.03, .25]
ADHD: all grades	.01*	.12*	[.00, .23]
CD: all grades	.01	.11	[−.02, .24]
ODD: all grades	.00	.07	[−.05, .19]
Total symptom count: all grades	.01	.12*	[.00, .23]

Note. Covariates entered at Step 1: gender, race/urban status, and group. TOCA-R = Teacher Observation of Classroom Adaptation-Revised; ADHD = attention-deficit/hyperactivity disorder; CD = conduct disorder; ODD = oppositional defiant disorder.

^*p < .05.

Substance Use

Results from the censored linear regressions of the substance use variables indicated that, after controlling for the covariates, TOCA-R sum scores in kindergarten were unrelated to any of the substance use outcomes at either eighth or 12th grade (all regression coefficients were not significant at the .05 level), with the exception of cigarette use in 12th grade. This finding indicated that higher TOCA-R sum scores predicted more days smoked cigarettes in the past month (β = .20, p < .05, 95% CI [.03, .36]).

Replication of Findings

It is important to note that sum scores, as reflective of classical test theory, provide equal weight to each behavior on a particular screening measure. Therefore, sum scores may not accurately capture the true severity of a child’s aggressive and disruptive behaviors, as some behaviors (e.g., being stubborn and disobedient) are considered less severe than others (e.g., fighting and harming others). Several researchers have argued that item response theory (IRT), which explicitly assesses differential severity across items (and thus behaviors), may be a more appropriate method for modeling the items used in brief screening scales for behavior problems (Embretson & Reise, 2000). In the current study, the findings from the TOCA-R sum scores were replicated with IRT-based TOCA-R scores (see Wu et al., 2012, for a description of the creation of the IRT TOCA-R scores). The IRT scores demonstrated a similar pattern of prediction to later school outcomes, externalizing diagnoses and symptom counts, and substance use outcomes as seen with the TOCA-R sum scores. Furthermore, the standard errors and confidence intervals were relatively similar between these two sets of analyses. However, the IRT TOCA-R scores tended to account for slightly more variance in the majority of the outcome variables as compared with the TOCA-R sum scores.

We also ran post hoc regression analyses without controlling for the effects of the covariates. A similar pattern of findings was observed in these analyses, as higher TOCA-R sum scores predicted more behavior problems at school, lower social skills, and poorer school adjustment. Additionally, in these analyses higher TOCA-R sum scores predicted more cigarette use in eighth and 12th grades. Higher TOCA-R sum scores also predicted higher externalizing disorder symptom counts (for ADHD, CD, and ODD) as well as higher odds of receiving an externalizing diagnosis. Lastly, although it was not possible for us to account for nesting within classrooms or schools in the outcome data examined for this study, we conducted post hoc regression analyses accounting for clustering in the TOCA-R sum scores collected during kindergarten (clusters were based on classrooms). A similar pattern of findings was observed in these analyses as was seen in the initial regression analyses, with similar beta coefficients obtained for all outcomes.

Discussion

The goal of the current study was to examine the predictive utility of the TOCA-R (specifically, the AA scale), a commonly used teacher screening measure, administered during kindergarten. We extended prior research by examining the TOCA-R at a single time point at an earlier age than examined in previous studies. Additionally, we examined a broader range of outcomes across a broader range of developmental periods than has been documented in the extant literature. Our results suggested that higher TOCA-R kindergarten scores were associated with more behavior problems at school, lower social skills, and poorer school adjustment reported by multiple informants (teacher, parent, and child) at the end of elementary, middle, and high school. The TOCA-R sum scores were also related, although somewhat more inconsistently, to the odds of an ADHD diagnosis, as well as ADHD, CD, and externalizing disorder symptom counts, but not to an ODD diagnosis or symptoms or any substance use outcomes.

The findings from the current study indicated that early teacher ratings consistently predicted later school outcomes, including school adjustment and social competence, as late as the end of high school. The prediction of ADHD diagnosis in ninth grade and symptoms in the sixth and ninth grades further supports the ability of the TOCA-R to predict outcomes mainly within the school setting, as teachers frequently play an important role in initial screenings for ADHD symptoms (Snider, Busch, & Arrowood, 2003; Snider, Frankenberger, & Aspen-son, 2000). Additionally, ADHD symptoms are frequently first observed and most troublesome in the classroom where children are required to sustain their attention and refrain from hyperactive or impulsive behaviors (Barkley, 2003). Overall, these findings suggest that the TOCA-R may be most useful when predicting problematic behaviors that occur within the classroom.

A strength of the current study is the inclusion of a racially and regionally diverse sample. Prior research has demonstrated that the TOCA-R administered in kindergarten exhibits group differences, as boys and African Americans have a higher overall mean, and therefore more frequent behavior problems, than girls or Caucasians, respectively (Koth, Bradshaw, & Leaf, 2009). In previous work with the same larger sample as the current study (Wu et al., 2012), item bias by gender was revealed in the TOCA-R, such that at equivalent levels of latent behavior problems, males received more endorsements of overt behaviors (e.g., harming others, fights, breaks things) from teachers, whereas females received more endorsements of nonphysical behaviors (e.g., stubborn, takes property, lies). Moreover, overt behaviors tended to be better at distinguishing among levels of latent behavior problems for males, and covert behaviors tended to be better at distinguishing among levels of latent behavior problems for females. However, given the similarity in the results obtained from the TOCA-R sum scores and the IRT scores, our findings suggest that the gender bias identified by Wu and colleagues (2012) may not affect the predictive utility of this screening measure.

The importance of including diverse samples in studies examining problem behaviors is underscored by the significant covariate effects in the current study. For instance, children living in urban areas were found to be rated as higher in behavior problems, lower in social competence and school adjustment, and higher in endorsed symptoms of CD than children in rural areas. Compared with urban Caucasians, urban African Americans also received the lowest/highest scores, indicating that this population is particularly at risk for the development of these problematic behaviors. Additionally, when compared with females, males were found to be rated as lower in school adjustment and higher in endorsed symptoms of ADHD and CD. These findings are consistent with other studies indicating that males, ethnic minorities, and children living in urban areas are at most risk for behavior problems (Hinshaw & Lee, 2003). It is therefore important that future studies continue to incorporate diverse samples (i.e., in terms of race, gender, and geographical location). Overall, the findings from the current study suggest that in applied settings, such as schools or community mental health centers, or when a quick screening of behavioral problems is needed, the calculation of a sum score on the TOCA-R may be sufficient to determine which children are at risk for later problems.

The post hoc regression analyses without controlling for the effects of the covariates allowed for exploration of the utility of the TOCA-R in applied settings, as it is unlikely that professionals in these settings would have the resources or information needed to consider these demographic factors in their screening procedures. A similar pattern of findings was observed in these analyses, providing support for the utility of the TOCA-R as a kindergarten screener in practical, applied settings (e.g., schools, community mental health centers, private practice offices). However, it is important to note that the findings regarding diagnostic outcomes were not significant in the regression analyses with covariates, suggesting caution in using the TOCA-R and similar brief screening measures to predict whether a child will receive a diagnosis. Such diagnostic decisions require careful, evidence-based assessments (Mash & Hunsley, 2005) along with the inclusion of demographic variables known to account for a significant amount of variance in these disorders (e.g., race, gender, etc.).

Clinical Implications and Future Directions

Generally, studies examining the development of maladaptive and problematic behaviors have focused mainly on elementary school, with little attention to earlier behaviors during preschool and kindergarten (Campbell et al., 2000). The current study shows that early behavior problems identified in kindergarten with the TOCA-R can reliably predict a range of adverse school-related outcomes in late childhood, middle adolescence, and late adolescence. Therefore, the TOCA-R may be most effective in identifying which children will benefit from prevention programs targeting problems within the school environment and across the school years (e.g., elementary, middle, and high school). It may be that classroom-based intervention programs would be best suited to implement with children at high levels of TOCA-R sum scores. Several existing interventions target improvements in school behavior. Examples include Promoting Alternative Thinking Strategies (Kusche & Greenberg, 1994), Coping Power (Lochman & Wells, 2003), and the Good Behavior Game (Barrish, Saunders, & Montrose, 1969).

The identification of children at risk for a range of negative outcomes beyond the school context may require the application of additional, broader screening measures. Specifically, other screening measures may be needed to identify children at risk for later outcomes that occur in contexts outside of the school setting (i.e., diagnoses of externalizing disorders and substance use), which may require information from additional reporters (Hill et al., 2004; Kerr, Lunkenheimer, & Olson, 2007; Lochman & CPPRG, 1995). Cost–benefit analyses are needed to determine whether adding additional raters is beneficial and outweighs the added cost of both time and resources needed to collect these ratings. Moreover, given previous research demonstrating that the TOCA-R did not adequately cover the lower range of problem behaviors (Wu et al., 2012), it may be that adding additional items to the TOCA-R may improve specificity of prediction to later ages. For example, recent research has suggested that poor self-control observed as early as age 3 predicts substance use disorders at age 32 (Moffitt et al., 2011). Improving prediction to later ages may involve creating measures that blend traditional symptom checklists with measures that tap the underlying processes (e.g., poor self-control) that increase risk for those symptoms. Future research should continue to explore the optimal combination of screening measures to capture the full range of negative outcomes that could be experienced by children and adolescents.

The findings from the current study indicate that early behavior problems that place children at risk for later adverse school-related outcomes can be identified as early as kindergarten. The results from this study therefore suggest that kindergarten may be an appropriate time to begin delivering prevention programs that address aggressive and disruptive behaviors in children. The prompt application of effective prevention programs with these children may interrupt the progression of behavior problems in school before they become entrenched and difficult to change (Lochman & CPPRG, 1995). However, the ability of these programs to effectively address these negative behaviors relies on the accurate identification of children who need these preventive interventions (Hill et al., 2004; O’Connell, Boat, & Warner, 2009). Future studies should therefore continue to explore the predictive validity of methods used in the early identification of at-risk children (Keenan & Wakschlag, 2002; Wakschlag & Keenan, 2001).

The timing of screening measures is an important consideration, and the administration of these measures may need to be altered depending on the purpose of the screening. For instance, previous studies have suggested that prediction to later outcomes is enhanced when TOCA-R ratings collected during first grade are used (Flanagan et al., 2003; Hill et al., 2004). Petras and colleagues (2004, 2005) have reported that the spring of third grade for boys and fifth grade for girls are the optimal times for minimizing both false-negative and false-positive identifications of children at risk for later problems based on TOCA-R sum scores. The findings from the current study suggest that for the general identification of children at risk for a variety of school-based negative outcomes at various developmental stages, administration of the TOCA-R during kindergarten is warranted. Future research should seek to refine conclusions regarding the ideal timing of TOCA-R administration.

Limitations

Limitations of the current study should be noted. First, the early behavior ratings examined in the current study only described a minimal amount of the variance (2%–5%) in later teacher-rated behavior problems, social competence, and school adjustment. More research is needed to identify additional factors that may explain more of the variance in these behavior and adjustment difficulties (e.g., parenting, peer engagement in antisocial behaviors, environmental context). Additionally, although the current study is longitudinal, care should be taken to not assume causal relationships between TOCA-R sum scores and outcomes, as several third variables may influence this relationship (e.g., low or unchanging teacher expectations, negative perceptions of children’s classroom behavior, and academic achievement).

Conclusion

In summary, this study illustrates the predictive validity of TOCA-R sum scores when predicting various outcomes, particularly those observed within the school setting, across a range of developmental periods. The ability of the TOCA-R to predict outcomes into late adolescence speaks to the benefit of using a widely distributed brief teacher report screening instrument. The findings also indicate that children identified as at risk for later behavior problems experience difficulty with school, teachers, and their fellow classmates. Prevention programs working with at-risk children should therefore continue using strategies that promote the development of academic skills, prosocial interactions with peers and teachers, and overall positive attitudes toward school. Taken together, this study, as well as other studies examining teacher-reported screening tools, supports the adage to “catch behavior problems early,” before children are directed into a persistent negative life course trajectory.