Surveillance of Middle and High School Mental Health Risk by Student Self-Report Screener

Abstract

Introduction:

A 2009 National Academies of Sciences report on child mental health prevention and treatment concluded that screening for mental health risk is an essential component of service delivery. To date, however, there are few practical assessments available or practices in place that measure individual child risk, or risk aggregated at the school or community level. This study examined the utility of a 30-item paper and pencil student self-report screener of behavioral and emotional risk (BER) for surveying community risk among 7 schools.

Methods:

In 2010, 2,222 students in 3 middle and 4 high schools in a medium-sized school district in Georgia were administered the Behavioral and Emotional Screening System Self-Report Child/Adolescent form (BESS Student). The BESS is designed to measure 4 sub-syndromal BER factors for developing mental health disorders: inattention/hyperactivity, internalizing, school problems, and personal adjustment. Analysis of Variance and Chi Square analyses were used to assess the association between adolescent self-reported BER as an indicator of school BER, grade level, child ethnic identification and gender, socioeconomic status, and special education placement status.

Results:

BESS scores differentiated well between schools for overall BER and special education status, as well as between grade levels, ethnicity, and gender groups. One high school, known by the school administration to have numerous incidents of student behavior problems, had the most deviant 4 BER domain scores of all 7 schools. Girls rated themselves as having a higher prevalence of BER (14%) than boys (12%); middle school students reported fewer difficulties than high school students.

Conclusion:

Middle and high school students were capable of identifying significant differences in their own BER across schools, suggesting that universal mental health risk screening via student self-report is potentially useful for identifying aggregated community risk in a given school that may warrant differential deployment of mental health prevention and intervention strategies. BESS results reliably identified individual mental health risk associated with special education placement, which is documented to lead to poor school outcomes such as school dropout and lack of enrollment in post-secondary education.

INTRODUCTION

Child and adolescent mental health disorders are known to be associated with, or increase the risk for, numerous poor school and life outcomes for children and adolescents, including suicidal ideation and attempts, academic underachievement and school dropout, substance use and disorders, and physical fighting or victimization by a weapon.^1–4 A preventive approach to mitigating associated impairment, morbidity, and poor outcomes in school settings has been advised for at least 4 decades.⁵ The widespread adoption of preventive models, methods, and procedures for achieving this goal, however, has remained nascent in U.S. schools.⁶

Schools have long been identified as the community context of choice for delivering preventive mental health services. As major societal institutions, schools provide an organizational structure that reaches more children with more continuity than primary care, or any other child and family service setting.⁷ Schools, however, are for the most part unprepared to provide preventive mental health services due to limited staff training, a competing focus on and time commitment to educational service delivery, and a lack of assessment methods for delivering services such as universal screening, which is a necessary component of any preventive mental health service delivery program.^{8, 5, 9, 3}

Universal screening is the first step in any preventive, secondary prevention, or early intervention program for mental health problems.⁹ A National Academies of Sciences report identified 4 levels of prevention including, 1) universal prevention where community risk factors, such as school safety, are of interest, 2) selective prevention where high risk groups, such as children exposed to maternal depression, are identified for services, 3) indicated prevention where screening for behavioral and sub-syndromal symptoms is used to identify children for early intervention services (defined as behavioral or emotional risk (BER), for the purposes of this study), and 4) assessment for detection, diagnosis, and treatment of a mental health disorders.³ In this report, the goals of universal screening were defined as:

The goals and design of these initiatives should be targeted to relatively narrow and specific purposes, for example, (1) improving school success for struggling students, (2) preventing bullying and student harassment, (3) improving teacher and peer relationships, (4) increasing school safety and security, or (5) learning to regulate and control behavior.

(p. 230)

Although the results of the study could inform all of these goals, the school administrators in the present study requested surveillance screening for risk factors associated with goals 3 and 4, due to their concerns about an overall level of misbehavior in some schools that was adversely affecting teacher and student morale and culminating in incidents of violence in the schools. In one of the high schools in particular, several incidents of student violence on campus caused alarm and growing concerns about the safety of the students and teachers.

A central impediment to the adoption of universal screening measures for school-based screening of large groups of children has been the practicality of such measures, especially the associated personnel costs and test administration time that competes directly with the demand for academic instructional time.¹⁰ Although newer screening measures such as the one used in this study require only a few minutes per child, the practicality of screening thousands of adolescents in numerous schools is yet to be determined.^11,12 However, emerging evidence suggests that some of the barriers to feasibility can be overcome with administrative support and planning prior to implementing a universal screening program.¹²

Behavioral problem surveillance measures, such as the Youth Risk Behavior Survey (YRBS), have been in use for some time.^13,14 This measure assesses student perceptions of the frank expression of behavior problems (e.g. smoking, sexual activity, etc.), rather than risk per se.^4,14 Mental health risk or BER tests represent a relatively newer class of measures that assess a different construct, the predictors of mental health disorders, special education placement, or behavioral problems such as those assessed on the YRBS or its variants, including the California Healthy Kids Survey (CHKS).^15,10 A recent study by Dowdy and colleagues provides empirical evidence of the difference between the YRBS and BER constructs.⁴ They found that supplementing the CHKS with a self-report measure of sub-syndromal BER, the same measure used in the current study, significantly increased the prediction of cigarette, marijuana, and alcohol use, binge drinking, physical fighting, threatening or injury by a weapon, skipping school, and serious consideration of suicide as measured by the CHKS.⁴

Based on this evidence, we expected a self-reporter screener to be a useful tool for assessing school-level BER. Thus, the current investigation sought to determine:

1. Whether or not a brief adolescent self-report screener of BER could be used universally in middle and high school with little concern about interference with instructional time or other practical concerns.

2. If the screener would produce score differences between schools that were consistent with school administrator concerns, which predicted that some schools were characterized by more adolescent BER than others.

3. Whether or not demographic variables such as child race/ethnicity, gender, SES, or grade level were strongly associated with screener scores.

4. If individual screener results demonstrated discriminant validity by assessing their association with classification as eligible for special education programs due to the presence of severe behavioral and emotional problems or diagnosed mental health disorders.

METHODS

Sample

Data were collected from 3 middle and 4 high schools in a mid-sized city in the Southeastern United States. The school district requested universal screening for each of the participating schools because of concerns that student misbehavior was beyond the typical rate. At the time of screening, the school district had been recently cited for substandard student academic achievement. In addition, several incidents of school violence had occurred on the campus of one of the high schools. The sample collected included 2,222 adolescents in the eighth through 12^th grades, ranging from 198 in grade 12 to 725 in grade 9. Females constituted 53% of the sample. The ethnic/racial group sample sizes were 1,701 African-American, 456 White, 32 Latino, and 26 Other. Sample sizes by school ranged from 95 to 849 (median = 205). The school district has a high poverty rate, with 76% of students in the present study being eligible for a free or reduced- price lunch. Approximately 6% of the sample, or 139 adolescents, were classified as special education eligible.

Procedure and Instrument

The Behavioral and Emotional Screening System (BESS) Student Form was administered to all students in groups, usually in homerooms, by school district employed school psychologists and school psychology doctoral students. Administration amounted to approximately 15 minutes per classroom, including the reading of instructions from a script, completion of forms, and collecting forms from all students. Total administration was less than 1 hour total per middle or high school since all data were collected at the same time interval, in most cases during the homeroom period. The data collection, entry, cleaning, file preparation, and analyses were either conducted or supervised by a post-doctoral fellow supported by the Georgia Measurement and Assessment Training (GMAT) program (funded by grant number R324B080D06 from the Institute of Education Sciences, US Department of Education). This study was approved by both the local school district institutional review board (IRB) and the IRB of the university of the lead researcher.

The BESS Student is a brief screening measure completed by students to identify BER among youth aged 8 through 18 years.¹⁶ It assesses a wide range of behavioral problems and strengths, including internalizing problems, externalizing problems, school problems, and adaptive skills. The BESS requires no explicit training, has 30 items, and can be completed in 5 minutes or less per student. Use of theory and factor analysis to develop the measure resulted in the inclusion of more internalizing items than other screening measures of this length, and a 4-factor solution, including an assessment of inattention/hyperactivity, internalizing problems, school problems, and personal adjustment.^16,17

Students are given 4 rating options—never, sometimes, often, or almost always—for each item and the sum of the items generates a total T-score with higher scores reflecting a higher level of BER.¹⁶ The scoring rubric or risk level for BER is as follows: (a) a T-score of 20-60 suggests a “normal” level of risk; (b) 61–70 suggests an “elevated” level of risk; and (c) 71 or higher suggests an “extremely elevated” level of risk. The risk level classification cut-scores were developed to maximize sensitivity and specificity, and results suggest that sensitivity, specificity, positive predictive value, and negative predictive value were generally high.¹⁶

Student responses on the BESS may be entered by hand or via scanner with computer software, as was done for this investigation. The software report includes raw scores, T-scores, and percentiles based on a normative sample that closely matched U.S. Census population characteristics. The existence of validity scales has been cited as a particular advantage of the BESS by independent reviewers.⁹

BESS split-half reliability estimates range from 0.90 to 0.97. Test-retest reliability estimates are high, ranging from 0.80 to .91. Inter-rater reliability estimates range from 0.71 to 0.83. The concurrent validity of the BESS was examined by administering the items with other social-emotional measures: Achenbach System of Empirically Based Assessment (ASEBA, 0.71–0.77), Conners’ Rating Scales (CRS, 0.51–0.78), and Children’s Depression Inventory (CDI, 0.51).

RESULTS

As stated in the procedure, the BESS self-report screener of BER was administered universally in 3 middle and 4 high schools with little interference with instructional time. Therefore, there was some evidence to support our first hypothesis that screening could be carried out practically and efficiently in schools.

Descriptive statistics for the sample by school are shown in the Table. In order to test whether the screener would produce score differences between schools that were consistent with school administrator concerns, an Analysis of Variance (ANOVA) comparing schools was conducted. This ANOVA revealed statistically significant differences in the 4 factor scores among the 7 schools with F values ranging from 4.73 for the Internalizing factor to 8.20 for the Hyperactivity factor, and all tests being significant beyond p < 0.001. These results supported the a priori predictions of the school administrators in that the high school that had experienced increases in school violence (High School #3) and its feeder middle school (Middle School #3) had the highest means on the deviant factors and the lowest means for the Adjustment factor. However, as demonstrated in Figure 1, these statistically significant differences across school by factor were small.

Table.

Descriptive statistics: factor score means and standard deviations for 4 Behavioral and Emotional Screening System factors by school.

School			Hyperactivity	Internalizing	School problems
		Adjustment factor	Factor	Factor	Factor
Middle school 1	Mean	3.3684	1.7209	1.6852	1.7966
	N	147	147	146	147
	Standard deviation	0.50167	0.51180	0.49201	0.55487
Middle school 2	Mean	3.2468	1.6737	1.7022	1.6825
	N	95	95	95	95
	Standard deviation	0.53925	0.47835	0.51825	0.47582
Middle school 3	Mean	3.1277	1.8622	1.8493	1.9603
	N	105	105	105	105
	Standard deviation	0.58581	0.54774	0.53887	0.60221
High school 1	Mean	3.2878	1.8114	1.7980	1.9635
	N	205	205	205	205
	Standard deviation	0.53559	0.52631	0.54994	0.57015
High school 2	Mean	3.3764	1.6854	1.6355	1.9460
	N	368	368	368	368
	Standard deviation	0.47221	0.46557	0.45389	0.51496
High school 3	Mean	3.2300	1.8855	1.7607	1.9912
	N	849	849	848	849
	Standard deviation	0.54561	0.55194	0.49715	0.55366
High school 4	Mean	3.3151	1.8172	1.7385	1.9836
	N	453	453	453	453
	Standard deviation	0.50231	0.54746	0.46482	0.51579
Total	Mean	3.2820	1.8105	1.7356	1.9521
	N	2222	2222	2220	2222
	Standard deviation	0.52681	0.53454	0.49409	0.54514

Figure 1.

Factor score means for 4 Behavior and Emotional Screening System by school.

The third question of interest was whether or not demographic variables, such as child race/ethnicity, gender, SES, or grade level, were strongly associated with screener scores The ANOVA by child gender was statistically significant for only the Adjustment (F = 6.79, p < 0.009) and Internalizing factors (F = 25.35, p < 0.000), where girls obtained higher scores on each. BESS scores also differed significantly by grade level for all four factors; Adjustment (F = 3.87, p < 0.004), Inattention/Hyperactivity (F = 6.23, p < 0.001), Internalizing (F = 2.82, p < 0.24), and School Problems (F = 6.99, p < 0.000). Figure 2 plots means for the 4 factors by grade level. These data reveal a trend for self-reported problems to increase in high school over levels reported by eighth graders.

Figure 2.

Factor score means and standard deviations for 4 Behavior and Emotional Screening System factors by student grade.

Socioeconomic status produced the most non-significant findings in that free or reduced lunch eligibility status, unlike the other demographic variables, did not produce any statistically significant differences between the BESS factors. BESS results for student ethnicity, by contrast, did differ for the African American and White groups. All four BESS factors differed for these 2 groups (p <0.000) with White students reporting more deviance and fewer positive adjustment skills. The results for the other 2 groups were too small to interpret with confidence.

In relationship to our fourth research question, special education status was linked statistically to only 2 of the BESS factors; Adjustment (F = 60.10, p < 0.001) and Internalizing (F = 47.30, p < 0.001). Special education status was not a significant predictor of Inattention/Hyperactivity or School Problems in the present study.

All of these analyses were repeated using BESS classification scheme as the outcome variable. The BESS BER classification system as noted earlier has 3 levels: normal, elevated, and extremely elevated risk. These results were similar to those found using factor scores. For example, no differences were found between BESS factors by SES, more White adolescents were classified as elevated and extremely elevated in risk than African-American adolescents, and girls (13.5%) acknowledged a higher prevalence rate of risk than boys (11.5%). In comparisons where ANOVA results were significant, all Chi Square tests of differences in proportions were also statistically significant.

DISCUSSION

This study suggests that student self-report screening results may provide schools and community stakeholders with systematic data about mental health risk that may be used to both address and monitor the mental health needs of adolescents in school. These data may also be used, as is the case with the BESS analytic software, to disaggregate BER status for different levels of analysis, and prevention and intervention planning, including the classroom, school, sub-district, and school district levels, and even region and state levels, if desired. Comparisons can then be made across levels of analysis and service delivery to determine where significant BER exists, whether there are Inattention/Hyperactivity, Internalizing, or School Problems, or an absence of strengths as assessed by the Adjustment factor. This “targeting” of systems in need of change or improvement based on data holds the potential to, for the first time, focus prevention and intervention efforts based on evidence.

While a full cost and practicality analysis of this measure and methodology was not undertaken for the purposes of this study, the ability to gather these individual student data from an entire school taking less than 1 hour of instructional time per academic year using a relatively low-cost screening instrument portends greater practicality than has been the case in the past. Brown and Grumet,¹ for example, used clinical interviews to survey suicide risk in 13 middle and high schools in the Washington, D.C. area. They found that 45% of adolescents screened positive for “previous suicide attempt or ideation, symptoms of depression or anxiety, and/or other emotional problems.” Their screening program, however, was cumbersome due to the use of clinical interviews, and it produced cost estimates for the program at about $242 per child, an untenable figure for large and small school districts alike. In contrast, universal screening using a quick and affordable instrument such as the BESS does appear to provide a practical solution for gathering self-report BER data universally.^{12, 18}

Furthermore, information gathered by the universal screening program implemented in the present study was able to provide score differences between schools that were consistent with school administrator concerns, which predicted that some schools were characterized by more adolescent BER than others. Using data to guide their decisions, school administrators and stakeholders can use the data gathered from a universal screening program to drive school and community interventions based on the areas of need. Although statistically significant, the practical significance of these small differences should be assessed. However, as all of the 7 schools in the present study were selected due to anticipated elevations in BER, generally similar levels of BER across schools was not entirely surprising. The small nuances across schools could help administrators prioritize the types of interventions necessary, and triage schools and individuals in order to best focus the limited resources that are available for such intervention.

Differences in BER were found across demographic variables of interest, including gender, grade, and race/ethnicity. Overall, females reported greater internalizing problems and lower levels of adjustment. High school students, particularly those in the ninth grade, reported lower levels of adjustment and higher levels of BER as compared to eighth grade students. Therefore, transitional difficulties from middle school to high school might help to explain this increase in BER. White students reported more BER than their African American peers. Future studies should continue to examine this finding; White students were the minority group in the schools of interest in the present study, so it could be that out-group status is more predictive of BER than membership in any one particular racial/ethnic group. Finally, those in special education reporter higher levels of internalizing problems and lower levels of adjustment than their peers who were not in special education.

Good surveillance data are prerequisite for initiating public health models of service delivery for children with BER inside or outside of school.¹⁹ The BESS and other newer instruments may be more suitable for providing data due to a variety of factors, including their ability to gather BER data for a wide age range (preschool through high school) and the availability of student, teacher, and parent forms. By contrast, the Youth Behavior Risk Survey and other methodologies fail to include a comprehensive assessment of sub-syndromal psychiatric symptoms or risk factors. The National Survey on Drug Use and Health and National Health and Nutrition Examination Survey are both collected at age 12 and above, and both include only a small sampling of psychiatric questions.

LIMITATIONS

The sample sizes for this study among particular demographic subgroups were sometimes minimal. Although screening was conducted universally, the sampling did not include all children in a school due to student absences and the exclusion of some classrooms due to disability classifications that prevented participation. Sample sizes were not adequate to conduct some important analyses, such as including a Latino sample in the ethnic group comparisons.

Due to time limitations only one screener was used. Although the BESS has supportive reliability and validity evidence, it is relatively new. A clear “gold standard” among such school-based screening measures has yet to emerge based on consensus use; therefore, more research is needed regarding the choice of screening assessment. It is too early for a test or few tests to become the standards of practice since the use of BER screeners, particularly student self-report screeners, is still uncommon in U.S. schools.

CONCLUSION

Use of a student self-report screener to identify mental health or behavioral and emotional risk among adolescents in schools produced results confirming the suspicions of school district administrators; one middle and one high school displayed more risk than others in the district. Although this is not a typical study of discriminant validity for a new measure it is a proxy that served the needs of school district administrators. This study offers some evidence that newer screeners may allow for greater implementation of mental health risk surveillance consistent with well-established public health practices and needs.