Parent and teacher mental health ratings of children using primary care services: Inter-rater agreement and implications for mental health screening

Abstract

Objective

To examine inter-rater agreement when screening for child mental health problems during primary care visits.

Design and Methods

Children age 5-10 (n = 227) and one of their parents were systematically recruited from the waiting rooms of 15 primary care sites located in Baltimore, MD, Washington, DC, and rural New York from 2002-2005. The parent and teacher of the child completed the Strengths and Difficulties Questionnaire to measure the child's emotions, behaviors, and functional impairment.

Results

Parents and teachers identified a similar proportion of children as having high symptoms (25% vs. 23%) and high impairment (27% vs. 32%) but rarely agreed in their assessments of specific children. Parent ratings alone missed 52% of children rated by teachers as having both high symptoms and high impairment (κ = 0.15). Only 6% of these discrepant visits were for mental health problems, making it unlikely that teacher reports would have been solicited.

Conclusion

Parent reports failed to detect half of school-aged children considered to be seriously disturbed by their teachers. Efforts to improve detection of mental health problems using screening tools in primary care may require algorithms that help providers judge when to solicit teacher reports and how to interpret conflicting information from parents and teachers.

Introduction

Researchers and clinicians have proposed that routine screening during primary care visits may improve the identification of children's mental health problems.^1-4 One potential problem with screening school-aged children in primary care is that ratings are often only gathered from the parent present at the visit.⁴ Teachers are usually consulted only if the parent expresses a concern about behavior or achievement in school.⁵

Obtaining mental health ratings from a single informant is inconsistent with findings from community-based studies, which have demonstrated that raters often do not agree in their assessments of child mental health.^6-14 These discrepancies are not necessarily errors in observation. Rather, when parent and teacher reports have been compared to independent psychiatric assessments, the combination of reports is more sensitive than either alone.¹⁵ Thus, discrepancies may reflect that children's symptoms, or the opportunities to observe them, vary across contexts.¹⁶ Obtaining teacher reports may be particularly important for children, given the high correlation of school difficulties with future mental health problems and social dysfunction.¹⁷

The implications of inter-rater agreement for screening in primary care have not been extensively explored. Inter-rater agreement in primary care might be better than in community samples because the prevalence of mental health problems among children in primary care is higher than in the general population, and detection generally improves with severity.^18-20 Further, visits may be initiated because an adult suspects that the child has a mental health problem.²¹

This investigation takes advantage of data collected for another study to examine parent-teacher agreement on mental health screening reports about children seen in primary care. The data represent what would result if PCPs systematically obtained mental health screening reports from both the teacher and parent. We hypothesized that parent-teacher agreement would be better than in community samples, but that, consistent with community studies, teachers would report more externalizing symptoms and impairment compared to parents.¹⁵

Methods

Participants and Procedures

The data were originally collected to assess whether the youth was eligible to participate in an intervention to improve the psychosocial communications skills of PCPs. Data were collected from 15 primary care offices in Baltimore, MD, Washington, DC, and rural New York which were chosen to represent the range of specialties, payers, locations, and practice structures that provide pediatric primary care in the United States.^22-25 All sites served patient populations with a mix of insurance types and who were within the age of infant to 18 years.

Fifty-four PCPs participated; 81.4% (n=44) medical doctors, 16.6% (n=9) nurse practitioners, and 1.8% (n=1) physician assistant. Forty percent were male, 64.8% were pediatricians, and 35.2% were family practitioners.

Recruitment of families took place from December 2002 to August 2005. Research interviewers systematically approached all parents in the waiting area with the goal of recruiting 10 children for each PCP. Children were eligible if they were between 5 -16 years old, scheduled to see a participating PCP, and reported pain as 4 or less on an analog scale of 1-10, in which 1 was no pain. The researcher randomly selected one child per family to participate if more than one child was eligible.

The parent provided written consent and was asked if a researcher could contact their child's primary school teacher. The parent provided the name of the school and teacher. While in the waiting area, the parent completed a questionnaire about the child's mental health (described below). The parent also reported whether the visit was for a mental health problem, well-child visit, acute illness, or follow-up of a medical problem. The parent was compensated $15 for participation.

The teacher identified by the parent was informed by mail that the child was participating in a research project to teach doctors skills and was asked to complete a questionnaire that would not be included in the child's medical record (described below). No information about the teacher was collected. All procedures were approved by the Johns Hopkins Bloomberg School of Public Health Committee on Human Research and the research review committees of participating sites.

Measure

The parent and teacher completed the Strengths and Difficulties Questionnaire (SDQ), a measure of children's emotions, behaviors, and functional impairment that has been widely used to screen for mental health problems in research and clinical settings.^26-27 The SDQ is available in over 40 languages and has been validated as a screening tool for children age 4-17 among community-based and clinic populations in the United States, United Kingdom, Australia, South America, Asia and many European countries.^28-33 Versions are available for parent, teacher, and teen self-report (www.sdqinfo.com).

The SDQ consist of 5 scales (emotion, conduct, inattention-hyperactivity, peer problems, and prosocial behavior), each composed of 5 items which measure the extent to which the item describes the child during the past 6 months using a 3-point Likert scale. Items are scored 0 = “not true,” 1 = “somewhat true,” and 2 = “certainly true” so that each scale ranges from 0-10. The emotion, conduct, inattention-hyperactivity, and peer problems scales are summed to create the Total Difficulties score, which ranges from 0-40.

Five items inquire whether the child has difficulties in four areas (emotions, concentration, behavior or getting along with others) and whether such difficulties interfere with home life, friendships, classroom learning, and leisure activities. These items are scored 0 = “not at all,” 1= “little,” 2 = “medium amount,” and 3 = “great deal,” to yield an Impairment scale score of 0-10.³⁴

U.S. normative data are based on parent reports of 9,878 children age 4-17 years, who completed the SDQ during the 2001 National Health Interview Survey (NHIS).³⁴ In the NHIS, scores in what was designated as the high range of each scale identified children as belonging to the worse 7-10% of the normative population depending on the domain.³⁴ In the NHIS, children who scored within the high range of the Total Difficulties scale were 12.2 times (95% CI, 9.6 -15.4) more likely to have received mental health services from the general medical sector during the past year.³⁴

The Total Difficulties and Impairment scales demonstrated excellent internal consistency in the NHIS investigation (Cronbach's α = 0.83 and 0.80 respectively).³⁴ Internal consistency was also good for the conduct, emotions, inattention-hyperactivity, and prosocial behavior scales (Cronbach's α = 0.63 to 0.77) but not peer problems (Cronbach's α = 0.46).³⁴ Similar internal consistency has been found among European samples.^35-36 In our sample, all parent-reported symptom scales demonstrated good to excellent internal consistency (Cronbach's α = 0.63 to 0.82), as did the Impairment scale (Cronbach's α = 0.85). Teacher-reported symptom scales also demonstrated good internal consistency (Cronbach's α = 0.72 to 0.85), as did the Impairment scale (Cronbach's α = 0.79).

Low levels of agreement between parent and teacher ratings (Pearson's r = 0.39) have been found among samples in the United Kingdom, which was similar to findings among populations in Germany and the Netherlands.^36-38

Statistical Analysis

SDQ scales were summed using the published algorithm (www.sdqinfo.com). SDQ scores for each domain were categorized as high according to the NHIS normative cut-points.³⁴ As in the NHIS investigation, we created a dichotomous variable to categorize children who were rated as having both a high Total Difficulties score and a high Impairment score. These would be children likely to meet diagnostic criteria for a psychiatric disorder in our sample.

We examined agreement between parent and teacher ratings using proportions and kappa. All analyses were conducted using Stata 9.³⁹

Results

Children and Teachers

This report focuses on children age 5-10 because the teacher completed the SDQ only for this age group. Of the 871 children initially screened for the original study, 291 were between the ages of 5-10, met eligibility requirements, and were recruited during the school year. Eighty-six percent (n = 253) of those parents consented to have a researcher contact their child's teacher. There were no differences in race, gender, or insurance status between the children of parents who consented to have a researcher contact the teacher and those who did not (results not shown). Teachers returned the SDQ for 227 children (89.7% of teachers contacted) for an overall response rate of 78.0%. Of the 227 children, 44.9% were female, 71.8% Caucasian, 19.3% African-American, 7.0% Other, and 1.7% American Indian. Ten parents (4.4%) identified as Hispanic and completed the Spanish language SDQ. There were no differences in race, gender, or insurance status between the children with returned teacher SDQ ratings compared with those without a teacher rating (result not shown). Half of visits (51.1%) were well-child appointments, 29.1% acute illness, 11.8% follow-up of a medical problem, and 7.9% were for a mental health problem.

Agreement between parent and teacher ratings

The group of parents and teachers both identified 17.6% of children as having a high Total Difficulties score and a high Impairment score, that is, children who would be most likely to meet diagnostic criteria in our sample (Table 1). However, parents and teachers did not identify the same children. Twenty-six percent of children were rated as having high scores on both the Total Difficulties and Impairment scales by either the parent or teacher, but only 9.3% were rated as such by both the parent and teacher. Among children rated as having both a high Total Difficulties score and high Impairment score by at least one observer, parents did not identify 52.5% of children who were identified by the teacher (k = 0.15). Of the children who the teacher, but not the parent, rated as having high Total Difficulties and high Impairment, only 2 parents reported that the visit was for their child's mental health problem, making it unlikely that teacher reports would have been solicited.

Table 1.

Parent and teacher ratings of children age 5-10 years (n = 227)

Score	Parent ratings	Teacher ratings	Either parent or teacher	Parent and teacher agree	Teacher concerns missed if only parent screened	kappa (κ)
High Total difficulties	25.4%	22.9%	34.3%	14.0%	38.4%	0.44
	58/227	52/227	78/227	32/227	20/52
High Impairment	27.3%	30.4%	42.2%	14.9%	50.7%	0.31
	62/227	69/227	96/227	34/227	35/69
High Total difficulties and high Impairment	17.6%	17.6%	26.4%	9.3%	52.5%	0.15
	40/227	40/227	60/227	21/227	21/40
High emotional	19.3%	11.8%	25.5%	5.7%	51.8%	0.25
	44/227	27/227	58/227	13/227	14/27
High conduct	22.4%	15.8%	29.0%	9.2%	41.6%	0.35
	51/227	36/227	66/227	21/227	15/36
High inattention-Hyperactivity	22.0%	29.9%	37.0%	14.9%	50.0%	0.42
	50/227	68/227	84/227	34/227	34/68
High peer problems	17.6%	21.1%	29.9%	8.8%	58.3%	0.31
	40/227	48/227	68/227	20/227	28/48
Low prosocial behavior	11.8%	22.0%	28.6%	5.2%	76.0%	0.18
	27/227	50/227	65/227	12/227	38/50

Agreement was best for children with attention or conduct problems (k = 0.42 and 0.35, respectively). Teachers reported a higher proportion of children with attention problems while parents reported a higher proportion with conduct problems. The worst agreement was for emotional problems and low prosocial behavior (k = 0.25 and 0.18, respectively). Teachers reported a higher proportion of children as having low prosocial behavior and parents reported a higher proportion as having high emotional symptoms.

Discussion

In our primary care sample, the prevalence of mental health symptoms and impairment was consistent with previous investigations in primary care settings, but parent-teacher agreement was no better than has been observed in community samples that used the SDQ.^{28,36,38,40-42} Our results parallel those from a British population-based study, which found that combining parent and teacher reports on the SDQ detected 62% of psychiatric disorders among children age 5-10, compared with 30% for parent report alone.²⁸

If PCPs were to routinely gather information from both the parent and teacher, the prevalence of children requiring further evaluation would be much higher than if only the parent completed the screening instrument. One alternative would be to ask routinely for teacher reports when the parent brings the child for evaluation of a mental health concern. However, in our sample, this would have led to getting additional information in only 2 of the cases where teachers suspected disorder-level problems but the parent did not.

Limitations

Our study is limited in that we do not have a way of assessing the child's “true” mental health status. However, prior studies with the SDQ and the Ontario Child Health Study Scales have established that parent reports alone underestimate the prevalence of disorder as measured independently with diagnostic instruments.^6,14 Although the SDQ has not been used extensively in clinical settings in the United States, it has been adopted as a component of the National Health Interview Survey, where it has shown strong associations with the use of mental health services.³⁴

Conclusion

Routine screening using only parent reports is likely to under-identify symptoms and functional problems that would be uncovered if reports were also solicited from teachers. Gathering a mental health rating from the teacher, however, greatly increases the prevalence of children needing further evaluation. Efforts to improve detection of mental health problems in primary care may require development of algorithms that help providers judge when to solicit teacher reports and how to interpret contrasting information from parents and teachers.