Instability in teacher ratings of children’s inattentive symptoms: Implications for the assessment of ADHD

Abstract

Objective

To examine the cross-grade stability of clinically elevated teacher ratings of inattentive symptoms in 3 samples of elementary school children.

Participants and Methods

Samples 1 and 2 included 27 first graders and 24 fourth graders, respectively, identified based on clinically elevated teacher ratings of inattentive symptoms. The third sample included 28 children in grades 1–4 from the Multimodal Treatment Study of ADHD (MTA Study) with a confirmed ADHD diagnosis. Teacher ratings of inattentive symptoms were completed an average of 12–14 months apart so that cross-grade stability of elevated ratings could be computed for each sample.

Results

In all 3 samples, clinically elevated ratings persisted for less than 50% of children and between 25 and 50% had ratings that declined to within the normative range. The decline in attention difficulties was not related to hyperactivity, oppositional behavior, or anxiety at baseline, nor was it explained by children beginning medication treatment.

Conclusions

Many elementary-aged children rated by their teachers as highly inattentive are not considered to demonstrate these problems the following year, even children with a confirmed ADHD diagnosis. The instability in clinically elevated teacher ratings found across 3 independent samples highlights the importance of annual reevaluations to avoid treating children for problems that may no longer be present.

DSM-IV diagnostic criteria for Attention-Deficit Hyperactivity Disorder (ADHD) require symptoms of inattention and/or hyperactivity-impulsivity to have been present for at least 6 months¹. This helps to prevent diagnosing with ADHD children whose symptoms reflect time-limited situational factors, e.g., psychosocial stress or settling in to a new classroom, rather than a chronic neurobiological disorder.

Because the 6-month time frame may not span successive school years, however, children can meet full ADHD diagnostic criteria based on symptoms reported by a single teacher. This may be problematic because classroom context is an important influence on children’s ADHD symptoms and ratings made by different teachers frequently show poor agreement – at least in middle school². Although clinicians evaluating younger children confront this issue infrequently because elementary school children typically spend their time in a single classroom, context effects may also influence symptom ratings that younger children receive. For example, a child in a poorly organized classroom with disruptive classmates may demonstrate more inattentive behavior than if placed in a structured classroom where consistent on-task behavior was encouraged. Thus, when inattentive symptoms have been evident within only a single school year, even if they have persisted for 6 months, they may reflect contextual factors rather than an enduring child characteristic and diagnosing ADHD may be premature.

Examining the cross-grade stability of clinically elevated ADHD symptoms is important because awareness that ADHD is often a chronic condition, coupled with findings that teacher ratings of ADHD symptoms are stable in community samples^3,4, may promote the unwarranted assumption that clinically elevated ratings are also stable. However, this was not the case in the only prior study that directly examined this issue.⁵ Below, we report on the cross-grade stability of elevated teacher ratings of inattentive symptoms in 3 samples of elementary school children. We focused on inattentive symptoms because attention difficulties typically become evident during elementary school and are more closely linked to academic difficulties than hyperactive-impulsive symptoms.^4,6,7 Two samples were comprised of elementary school children who were rated as highly inattentive by their teacher but who did not have a formal ADHD diagnosis. The third sample includes children from the Multimodal Treatment Study of Children with Attention Deficit Hyperactivity Disorder (MTA Study), all of whom had been diagnosed with ADHD, Combined Type.⁸ Including 3 diverse samples enabled us to explore whether cross-grade stability of clinically elevated inattentive ratings is greater in older vs. younger children, or in children with a confirmed ADHD diagnosis vs. those who merely screen positive for attention difficulties.

In each sample, teacher ratings of inattentive symptoms were completed in successive years – roughly 12 to 14 months apart - so that the cross-grade stability of clinically elevated ratings could be examined. We anticipated that cross-grade stability would be greater among diagnosed children and, within the non-diagnosed samples, would be greater for older children. Were cross-grade stability found to be relatively low in each sample, it would highlight the importance of annual reevaluations to determine whether treatments initiated in response to problems during one grade remain evident in the child’s new classroom.

Participants and Methods

Participants

Sample 1

Participants were 27 first grade students (17 males) attending five public elementary schools in a mid-sized city located in the southeastern United States; these children were recruited for a study evaluating different computer interventions for enhancing attention and academic performance in children with attention difficulties.⁹ Those in the current study had been randomly assigned to the wait-list control group so their results are not influenced by participation in any intervention. Fifteen were African American, 8 were Hispanic, 3 were Caucasian, and 1 was identified as multiracial (89% minority overall); this breakdown is consistent with the racial/ethnic composition of participants’ schools. Eighteen (72%) were on free or reduced lunch. Five students (22%) had been diagnosed with ADHD per parent report; only 2 were receiving medication treatment when recruited. This sample is referred to as the first grade sample.

Sample 2

Participants were 24 children (13 males) selected from a larger sample of 177 children recruited from a diverse, suburban school district in the southern US to study the origins and outcomes of social aggression.¹⁰ Children selected for this study were those within the larger sample who had elevated teacher ratings of attention problems in fourth grade (see below). Four were African American, 14 were Caucasian, and 6 were Hispanic. Four parents reported an annual income of less than $25,000, 6 reported an annual income of $26,000–$50,000, 6 reported an annual income of $51,000–$75,000, 3 reported an annual income of $76,000–$100,000, and 1 reported an income of greater than $100,000 per year; 4 parents did not disclose annual incomes. These characteristics were similar to the composition of the larger sample from which they were drawn. Information on ADHD diagnostic status and ADHD medication treatment was not available. This sample is referred to as the fourth grade sample.

Sample 3

Participants were 28 children from the Community Comparison group of the MTA Study⁸, a randomized clinical trial in which 579 7 to 9-year old children meeting DSM-IV criteria for ADHD, Combined Type were randomized to either receive medication management, multi-component behavior therapy, the combination, or self-selected community care (Community Comparison, CC). Our sample was a subset of children within the CC group who had elevated teacher ratings of inattentive symptoms at baseline and who were known not to have initiated medication treatment prior to or during the initial 14-month follow-up. The mean age of participants was 7.8 years; 75% were male, 56% were Caucasian, and 25% were African American. Thirty-seven percent had household incomes of $20,000 or less, 43% earned between $20,000 and $50,000, and 14% earned more than $50,000. Compared to the full MTA sample (n= 579), children in our never-medicated subsample had a greater proportion of families in the lowest income category (37% vs. 21%) and fewer families in the highest income category (14% vs. 36%). This sample is referred to as the MTA sample.

Measures

The primary measure in each sample was teacher ratings of students’ inattentive symptoms. Baseline teacher ratings of hyperactive-impulsive symptoms, oppositional behavior, and anxiety were also obtained to better understand participants’ characteristics and to examine whether difficulties in these areas were related to persistent attention difficulties. Measures used to assess these areas are described below.

Inattentive symptoms

Inattentive symptoms were assessed in the first grade sample using the DSM-IV Inattentive Scale from the Conners’ Teacher Ratings Scale-Revised (CTRS-R:L).¹¹ The DSM-IV Inattentive Scale corresponds to the nine DSM-IV inattentive symptoms of ADHD; teachers rate each symptom using a 0 to 3 scale to indicate how problematic each behavior has been in the past month. Both T-scores (mean of 50 and standard deviation of 10) and total symptom counts were computed for each child; the symptom count followed manual guidelines that only symptoms rated ‘3’ be counted). The DSM Inattentive Scale is internally consistent (alpha = 0.95) and stable (r = 0.70) over a 6–8 week period.¹¹

In the fourth grade and MTA samples, children’s inattention was assessed using the Inattention subscale from the Teacher Report Form (TRF).^12,13 This scale includes 5 items from the TRF Attention Deficit/Hyperactivity Problems Scale that correspond most closely to DSM-IV inattentive symptoms of ADHD, e.g., ‘fails to finish things he/she starts’, ‘can’t concentrate, can’t pay attention for long’ and each item is rated on a 3-point scale. T-scores of 64 or above (there is no raw score corresponding to a T-score of 65) were used to identify inattentive subjects; this corresponds to the top 9% of the normative sample. Coefficient alpha was 0.93 in sample 2 and 0.81 in sample 3. In the MTA sample, the total number of inattentive symptoms was also computed for each child using teacher ratings from the SNAP¹³, in which teachers rate how well each of the 9 DSM-IV inattentive symptoms describe the child using a 0 “not at all” to 3 “very much” scale. Consistent with how symptom counts were computed for sample 1, only symptoms rated ‘3’ were counted as present.

Hyperactive-impulsive symptoms

The DSM-IV Hyperactive-Impulsive Scale from the CTRS-R:L was used to assess participants’ hyperactive-impulsive symptoms for the first grade sample. This scale corresponds to the nine DSM-IV hyperactive-impulsive symptoms (HI) of ADHD and items are rated on a 0–3 scale as described above. Internal consistency is high (alpha = 0.93) and 6–8 week test-retest reliability is moderate (r= 0.47).¹¹ For the fourth grade and MTA samples, hyperactive-impulsive symptoms were assessed using the Hyperactive-impulsive scale from the TRF¹². The 8 items on this scale have substantial overlap with the DSM-IV hyperactive-impulsive symptoms, e.g., ‘can’t sit still’, ‘impulsive’, ‘talks out of turn’. Test–retest reliability over a 15-day period exceeds 0.90¹² and the internal consistency is excellent, alpha = 0.94.

Emotional and behavioral functioning

Emotional and behavioral functioning was assessed in the first grade sample using the Anxious-Shy Problems scale and the Oppositional scale (ODD) of the CTRS-R:L; these scales are internally consistent (alphas of 0.92 and 0.82 respectively) and stable over 6–8 weeks (test-retest reliability exceeds 0.85 for both scales¹¹. In the fourth grade and MTA samples, these areas were assessed using the Anxiety Problems Scale and the Oppositional Defiant Problems scale from the TRF. These scales include items constructed through expert clinical judgment to match selected categories for behavioral/emotional problems as described in the DSM-IV¹². Internal consistency is reported to be 0.91 for the Oppositional Defiant scale and 0.73 for the Anxiety Problems scale. Stability over approximately 2 weeks is 0.90 and 0.73 respectively.

Procedure

First grade sample

Teachers for successive cohorts of first graders rated their students (n = 1109; this represented all students in these classrooms as parental consent was not required to obtain this screening data) on the DSM-IV Inattentive Scale of the CTRS-R:L approximately 5 weeks into the school year to identify children for an attention training study. Two hundred and fourteen children who scored at least 1 SD above the sample mean were potentially eligible for the study; parental consent and student assent to participate was obtained from 167 (78%). To restrict our sample to children with evidence of stable attention difficulties, a second rating of inattentive symptoms was obtained 5 weeks later (baseline) when teachers completed the entire CTRS-R:L. Thirty-two students with a T-score on the DSM-IV Inattentive Scale of 64 or above – this corresponds to the top 9% of the normative sample - were retained for this study. This score was used to identify inattentive students rather than more conventional cut-score of 65 to be consistent with the other 2 samples. Approximately 12 months later, teacher ratings using the CTRS-R:L were collected again on 27 students who had remained at their school.

Fourth grade sample

As part of a larger study on the development and consequences of social aggression, teacher ratings using the TRF¹² were obtained in the spring of fourth and fifth grade on 177 children. These children – who reflected the general population of children at these schools rather than being any type of ‘at-risk’ sample - had been recruited from a diverse, suburban school district in the Southern United States. The data presented below is based on 24 children whose T-score on the Inattention subscale of the TRF was 64 or greater at the initial assessment in fourth grade.

MTA sample

As part of the standard assessment protocol used in the MTA study, participants’ teachers rated their behavior using the Teacher Report Form (TRF)¹² and the SNAP.¹³ These measures were obtained at baseline and at the 14 month outcome assessment.

Results

Baseline characteristics

Table 1 summarizes baseline characteristics for the 3 samples. Because these are 3 distinct samples, statistical comparisons were not appropriate and the data are presented for descriptive purposes only. Baseline inattentive scores are clinically elevated in all 3 samples. Average scores for HI and oppositional (ODD) symptoms were also at least 1 standard deviation above the normative mean in each sample; this is expected given that these problems covary with attention difficulties. Teacher ratings of anxiety symptoms were above average in all 3 samples but within a standard deviation of the normative mean. There was a greater representation of minority students in the first grade sample than in the other samples.

Table 1.

Baseline characteristics of each sample

	First grade sample (n=27)	Fourth grade sample (n=24)	MTA sample 3 (n=28)
Percent male	63%	50%	75%
Percent Caucasian	11%	58%	56%
Inattentive T-score	73.0	68.5	67.4
Hyperactive-impulsive T-score	65.5	62.3	68.3
Oppositional T-score	60.7	60.3	66.0
Anxiety T-score	60.1	56.8	58.9

Stability of inattentive symptoms

Table 2 shows the percentage of children whose symptoms remained clinically elevated (defined as a T score of ≥ 64) or had declined to the normative range, i.e., T < 60, based on ratings made by their teacher the following year. The percent of children with cross-grade elevations in teacher ratings of inattentive symptoms ranged from 33% to 46%, with the greatest stability found within the MTA sample. Inattentive symptoms declined into the normal range, i.e., within 1 SD of the normative mean, for between 25 and 50% of participants, with the smallest decline found for the MTA sample.

Table 2.

Percent of children with clinically elevated and normative scores at follow-up.

	First grade sample (n=27)	Fourth grade sample (n=24)	MTA sample 3 (n=28)
% T >= 64	37%	33%	46%
% T < 60	44%	50%	25%

In the first grade and MTA samples, symptom counts of the 9 DSM-IV inattentive symptoms were available at baseline and follow-up; this allowed us to examine the change in symptoms among children with at least 6 teacher-reported inattentive symptoms at baseline, which is the number required to consider an ADHD diagnosis. Fourteen children in the first grade sample had at least 6 inattentive symptoms at baseline. At follow-up, 10 had 2 or fewer symptoms (70%) including 5 with 0 symptoms (36%); only 2 children (14%) continued to be rated with 6 or more symptoms. Within the MTA sample, there were 15 participants with at least 6 inattentive symptoms at baseline. Six were rated with 2 or fewer symptoms at follow-up (40%) including four (27%) with 0 symptoms; 6 children (40%) continued to be rated with 6 or more inattentive symptoms.

Correlates of stable attention difficulties

A series of exploratory ANOVAs were conducted to test whether baseline scores for HI symptoms, anxiety, or oppositional behavior were higher among children whose attention difficulties persisted across grade. Mean scores for children in stable and unstable groups are presented in Table 3. Across the 3 samples, none of these differences approached significance, i.e., all ps > 0.30, indicating that these symptoms assessed at baseline were not reliably associated with stable attention difficulties.

Table 3.

Baseline T-scores on hyperactivity, anxiety, and oppositional behavior for children with stable (S) vs. unstable (US) attention problems.

	First grade sample (n=27)		Fourth grade sample (n=24)		MTA Sample (n=28)
	S	US	S	US	S	US
Hyperactivity	68.9	63.6	59.4	63.8	67.2	69.3
Anxiety	58.4	61.1	57.6	56.4	57.8	59.9
Oppositional	58.2	62.2	62.0	59.5	65.2	66.7

Discussion

Data from 3 diverse samples indicates that a substantial percentage of elementary school children rated by their teacher as having clinically significant inattentive symptoms are not rated as having significant attention difficulties the following year. In 2 samples of children without a confirmed ADHD diagnosis, cross-grade stability of clinically elevated ratings was found for fewer than 40% of participants and between 40 and 50% had ratings of attention difficulties that had declined to within the normal range, i.e., within 1 SD of the normative mean. In addition, and contrary to expectations, descriptive data did not suggest that elevated attention difficulties were any more likely to persist in older children. As expected, there was some indication that cross-grade stability was higher among children from the MTA sample although direct comparisons could not be made. Even among this carefully diagnosed group, however, fewer than half retained clinically elevated ratings the following year and 25% had ratings that had declined into the normative range. Cross-grade stability in elevated ratings was not related to baseline ratings of hyperactivity, anxiety, or oppositional behavior in any sample. Given the relatively small number of participants in each sample, however, our power to detect such differences was limited.

The change in symptom counts among children from the first grade and MTA samples who were rated with at least 6 inattentive symptoms at baseline was especially striking. Among the 14 such children from the first grade sample, only 2 continued to be rated with at least 6 symptoms by their new teacher and 5 were reported to show 0 symptoms. Even within the MTA sample, only 6 of 15 children were rated with at least 6 symptoms in both years and 4 children were reported to show 0 symptoms by their new teacher. This could not be attributed to the initiation of medication treatment, as the MTA sample analyzed for this study was restricted to children who did not receive medication between baseline and follow-up. And, although we did not have information on medication treatment for children in the fourth grade sample, within the first grade sample there was no indication that children known to have started medication in between the two rating points (n=5) were faring any better than their peers.

Why might the cross-grade stability of clinically elevated teacher ratings of inattentive symptoms be so modest? A variety of explanations are possible including positive change associated with maturation, the resolution of a significant life stressor, or perhaps improved nutrition and/or sleep. Regression to the mean and measurement error is also likely to play a role. In addition, for many children a change in context associated with the transition to a new classroom is likely to be a critical factor. This hypothesis is consistent with Achenbach and McConaughy’s¹⁴ finding of only moderate agreement in the behavior ratings made by different teachers. It also resonates with the limited agreement in middle school teachers’ ratings of ADHD symptoms reported by Evans, Moore, and Strauss², who attributed this to the unique characteristics associated with different classrooms.

Although our study did not directly examine the role of context, it is interesting to note that screening data collected in Study 1 on first graders from 25 classrooms indicated the number of students with clinically elevated ratings of attention difficulties ranged from 0 to 9 across classrooms. And, the percentage of students within each class who received a ‘0’ rating on every inattentive symptom ranged from 5% to 62%. This substantial between-classroom discrepancy seems unlikely to be fully explained by principals disproportionately placing inattentive students with teachers especially skilled at working with them. Rather, it suggests that the class a child is assigned to can significantly increase the risk of being rated as highly inattentive. Even though clinicians may be aware of this, because younger students typically have a single teacher, it is easier than with middle school students to overlook classroom context as a possible factor in a child’s attention difficulties. To the extent that this is the case, it may inadvertently lead some children to be inappropriately diagnosed with ADHD.

There are several limitations to the present work that should be acknowledged. First, the number of students in each sample is relatively small, although the consistency of findings across the diverse samples strengthens our conclusions. Second, because we do not have good information on the rate of medication treatment in 2 of the samples, we do not know how many children may have shown normalized symptoms in year 2 in response to medication treatment. As noted above, however, the initiation of medication treatment could not explain results for the MTA sample. On a related note, we do not know the extent to which non-medical interventions initiated between the 2 assessments may have contributed to a reduction in inattentive symptoms for some children. However, given that the intensive psychosocial intervention implemented with children in the behavioral treatment arm of the MTA Study normalized core ADHD symptoms for only one-third of participants¹⁶, it seems unlikely that the far less intensive intervention that some participants in our samples may have received would have had a significant effect on the findings. Third, we have no information for why symptom reports often declined so substantially and although an important role of classroom context is suggested, we did not directly examine this. However, while understanding the reasons for such declines should certainly be examined in future research, the contribution of this study is in highlighting that even clinically significant attention difficulties at school are likely to be transient for many children, including those with a confirmed diagnosis of ADHD. Finally, although it appeared that children meeting full diagnostic criteria for ADHD were somewhat more likely to have elevated attention difficulties that persisted across years, we could not test this directly. This would be an interesting issue to examine more carefully in subsequent work.

Conclusions

Results from this study have several clinical implications for the use of teacher ratings in the assessment and management of ADHD. First, as stipulated in the ADHD Toolkit¹⁵, findings underscore the importance of not over-relying on symptom counts in making an ADHD diagnosis as doing so may identify many children whose difficulties at school will be transient. Careful diagnostic interviews that elicit information on whether a child’s symptoms have been evident across grades may mitigate this problem. The importance of reevaluating children annually to learn whether inattentive symptoms reported by one teacher are evident in the child’s new classroom is also highlighted. For children who have been taking medication to treat attention problems, this should be done when the child has been off medication for a brief period. In the absence of such a procedure, some children are likely to be maintained on medication to address difficulties that may no longer be present. It is important to note, however, that a substantial decline in inattentive symptom ratings may occur for reasons other than an enduring resolution of the child’s difficulties, e.g., variability in how teachers use behavior rating scales and/or how they organize and manage their classroom. Thus, further clinical assessment will generally be needed to identify the most likely explanation for the decline in attention problems reported by a new teacher.

Finally, two interesting issues are raised by these results. First, our findings suggest that expanding the 6-month symptom duration requirement to require that a child’s symptoms have been evident in more than one grade could help prevent diagnosing with ADHD children whose school attention problems have a good probability of being transient. Second, efforts should be made to identify characteristics that accurately identify children whose attention difficulties are likely to persist across school grades, as such children are at particularly elevated risk for a range of adverse academic outcomes⁹.