Cost-Effectiveness of a School-based Emotional Health Screening Program

Abstract

Background

School-based screening for health conditions can help extend the reach of health services to underserved populations. Screening for mental health conditions is growing in acceptability; however, evidence of cost-effectiveness is lacking. This study assesses costs and effectiveness associated with the Developmental Pathways Screening Program, in which students undergo universal classroom emotional health screening, and positive screens are provided on-site clinical evaluation and referral.

Methods

Costs are enumerated for screening and clinical evaluation in terms of labor and overhead and summarized as cost per enrolled student, per positive screen, and per referral. Cost-effectiveness is summarized as cost per student successfully linked to services. School demographics are used to generate a predictive formula for estimating the proportion of students likely to screen positive in a particular school, which can be used to estimate program cost.

Results

Screening costs ranged from $8.88 to 13.64 per enrolled student, depending on the prevalence of positive screens in a school. Of students who were referred for services, 72% were linked to supportive services within 6 weeks. Cost-effectiveness was estimated to be $416.90 per successful linkage when 5% of students screened positive, and $106.09 when 20% screened positive. A formula for estimating the proportion of students screening positive proved accurate to within 5%.

Conclusion

Information concerning costs and effectiveness of school-based emotional health screening programs can guide school districts in making decisions concerning resource allocation.

Introduction

Adolescent mental health conditions are both prevalent and debilitating.¹ In the United States, as many as 21% of adolescents may be affected at any given time, with 65% experiencing sub-clinical mental health conditions by adulthood.^2,3 Serious adverse outcomes including poor school performance, unemployment, criminal activity, and suicide^4-8 are associated with poor mental health, and significant social and economic costs have been documented.⁹ The World Health Organization currently attributes 11% of the global disease burden to mental illness, with depression ranking first among all causes of disability worldwide. Therefore, early identification of mental health conditions with intervention to reduce or eliminate negative sequelae is of great public health importance.^4,6

Screening for mental health conditions has gained in popularity and acceptability over the past 15 years.^4,10-13 When implemented in schools, screening programs can extend the reach of mental health services to underserved populations.^4,13 To date, school-based mental health screening programs have been used primarily to identify children for specific intervention programs.^14-17 However, recent research indicates that screening with clinical follow-up may itself constitute a beneficial intervention. For example, secondary school students participating in the TeenScreen program experienced increased service utilization and decreased self-reported suicide attempts.¹² Other mental health screening programs have fostered improved linkages to support services.¹⁸ Given their potential to increase early detection of mental health problems, referral to services, and positive emotional adjustment, such programs are generally perceived as beneficial,^19,20 despite a lack of evidence-based cost-effectiveness data. Cost-effectiveness of school-based programs such as DARE drug prevention²¹ and universal scoliosis screening^22,23 was called into question long after integration into standard practice. Timely consideration of cost-effectiveness is indicated to ensure appropriate allocation of school resources.^24,25

Evaluation of other screening programs such as those for breast cancer provides strong evidence of the utility of cost-effectiveness studies as well as a solid methodology for evaluation of school-based mental health screening programs.^26,27 However, to the authors' knowledge, only one study¹⁹ has evaluated the cost of a school-based mental health screening program. This program screened for clinical anxiety, depression, and substance abuse problems and provided counseling for students who screened positive. The study followed similar methodology to that found in the breast cancer screening literature. The cost analysis pertained to one specific program model, and effectiveness data were limited to cost per referral to services and per treatment. Therefore, further research is warranted in this field to establish costs of different types of programs, and to further examine costs in light of program effectiveness.²⁸

This study determines costs associated with a two-stage, school-based emotional health screening program utilizing a cost-assessment framework borrowed from the breast cancer screening literature. Costs of the screening program are first enumerated and then summarized in terms of cost per enrolled student, per positive screen, per referral, and per successful linkage to services. Additionally, a predictive formula was generated that utilizes school demographic characteristics to estimate what proportion of students will screen positive.

Methods

Subjects and Procedure

The Developmental Pathways Screening Program (DPSP) was conducted each fall between 2001 and 2005 with 6^th grade students attending selected public middle schools in a large urban school district. The purpose of DPSP was to identify students experiencing emotional distress during the transition to middle school. Sub-clinical symptoms can be detected through screening long before clinical impairment is apparent, and early identification could lead to more effective early intervention. DPSP was implemented under a research study framework with Human Subjects approval from the University of Washington, and further details of the screening protocol are described by Vander Stoep et al.¹⁸

Stage 1 of DPSP involved recruitment and screening. During Stage 2, students who screened positive for elevated depression or disruptive behavior symptoms were clinically evaluated and given referrals to supportive services as indicated. All elements of the program were carried out by DPSP staff who were trained in maintaining confidentiality and either questionnaire administration or clinical evaluation. The screening staff had a minimum of 2 years of college education in psychology or public health, and all had experience working with school-aged children. The clinical evaluators were masters-level child mental health professionals with educational backgrounds in social work or psychology.

Stage 1 Recruitment

Stage 1 emotional health screening was offered to all enrolled sixth grade students with 3^rd grade spoken English comprehension. Although the program was open to all students with adequate English proficiency, participation was voluntary, and required written parental permission. Program information and permission forms were mailed to parents/guardians. DPSP was subsequently introduced to students in the classroom setting, and additional permission forms were distributed to students as needed. On screening day, the assent form was summarized for students, who were also given the opportunity to read it and ask questions. Students with written parental permission who also gave written assent participated.

Stage 1 Screening

Two staff administered a written screening questionnaire in classrooms during one 50-minute period. Additional staff members assisted in classrooms including students with learning disabilities or other special needs and offered individual administration when warranted. As per school policy for programs involving written parental permission, a small incentive (such as a pen, or pencil sharpener) was offered for returning the permission form, and an additional similar incentive was given to each student who participated.

Stage 1 Screening Measures

The Mood and Feelings Questionnaire (MFQ)²⁹ is a 33-item self-report scale, designed for youth aged 8-18 years old. Questions are derived from the DSM-III diagnostic criteria for major depressive disorder and dysthymia and assess affective, melancholic, vegetative, cognitive and suicidal symptoms.³⁰ High test-retest reliability and good convergent validity with a depression diagnosis have been demonstrated.^30-32

The Youth Self-Report (YSR) aggression scale³³ is an 18-item self-report scale, designed for youth aged 11-18 years old. Content, criterion, and construct validity of the YSR have been demonstrated in diverse populations of children and adolescents through four decades of research (Achenbach, 2001).

For both the MFQ and YSR scales, students were asked to report on a 3-point Likert scale; “not true,” “somewhat or sometimes true,” or “very true or often true,” how true each item was for them in the past two weeks. Students who scored above a designated threshold were considered to have “screened positive” for emotional distress. As described by Vander Stoep et al,¹⁸ the threshold was set at ≥ 20 on the MFQ or ≥25 on the YSR.

Stage 2 Clinical Evaluation

Stage 2 involved a 30-minute 1:1 clinical evaluation of students who screened positive. Masters-level, child mental health professionals (CMHPs) carried out structured, confidential evaluations at the school to assess level of distress, functional impairment in academic, emotional and social domains, strength of support network, and need for referrals. Commonly identified problems included poor academic performance, poor attendance, social isolation, interpersonal conflicts, divorce or recent death in the family, and discipline issues at home or school. When deemed appropriate, the CMHP, with the student's input, created a plan for referral to school-based tutoring, after school social activities, school counselors, on-site mental health counselors, and/or community-based mental health agencies

Stage 2 Referral

The CMHP called parents/guardians of students who received an evaluation to report the student's emotional health status. Motivational Interviewing (MI) techniques were used to make referral recommendations, because of the non-judgmental, non-confrontational approach it uses to increase awareness of potential problems and consequences. Envisioning and articulating the steps needed to meet goals has been shown to increase the likelihood of achieving them.³⁴ The CMHP made a follow-up phone call two weeks after the initial call, to assess whether the linkage to the referral had been completed. If needed, further Motivational Interviewing techniques were used to discuss barriers to implementation and to encourage parents/guardians to make the linkage.

Stage 1 and 2 Participation Rates

Over the study period, 2928 6^th grade students from four Seattle middle schools were eligible to participate, and 2190 (74.8%) were screened. Of the study participants, 1025 (46.8%) were female, and 1165 (53.2%) were male, 257 (9.2%) were Special Education students, 30 (1.4%) were seriously behaviorally disordered students, and 204 (7.3%) were English Language Learners. 351 (16.0%) of screened students screened positive, requiring a CE. Of these students, 52.7% were female, 47.3% male, 43 (11.5%) were Special Education students, 14 (1.4%) were seriously behaviorally disordered students, and 41 (11.0%) were English Language Learners.

Data Analysis

Measuring cost

The costs of the universal screening program were calculated. Only direct screening costs were included in the analysis. As outsiders implementing the program within the school setting, staff invested significant time and materials in producing educational brochures and fliers to introduce DPSP to principals and teachers before they agreed to host DPSP. Time was also required to create databases and assign non-identifiable study codes for research purposes. These steps would likely be unnecessary for a standard non-research school program. Indirect costs, such as utilities and office space, were assumed to be marginal and already present within the confines of a school and therefore not evaluated. Finally, DPSP Staff invested significant resources to develop the screening protocols and materials. These methods have already been developed with the intention of freely disseminating them; thus future development costs to modify these materials to a particular school would be marginal. Any research costs, indirect expenses, and development costs which were estimated to total ≤ $0.01 per enrolled student or less are not included in this analysis.

The authors made several assumptions based on consistent findings from 5 years of DPSP screening data. Calculations assumed an 80% participation rate, a 10% refusal rate, and a 10% no response rate. Classroom size was set at 25 students, with 10% Special Education students and 5% English Language Learners requiring additional classroom support, and 1% requiring 1:1 administration. It was estimated that 5% of parents would require translation services for the CMHP phone call. Copying costs were assumed to be $0.05 per sheet. Finally, 60% of students who screened positive were assumed to warrant referral for supportive services (Figure 1).

Figure 1. Screening Flow Chart, Hypothetical School, Population 300.

Stage 1 and Stage 2 costs of screening were calculated by summing personnel and overhead expenses. Costs were first calculated per student participating in each stage; then from this figure, cost per enrolled student was calculated. Personnel costs were calculated by weighting time spent accomplishing each task by the average 2006 institutional salary schedule for a given occupational category. For example, two screening staff members were paid $13.00/hr each for 1.5 hours of time to screen in a regular classroom of 25 students; therefore, the personnel cost of screening = ($13/hr)(2 staff)(1.5hr)/25 students =$1.56 per enrolled student. Overhead costs included materials such as the permission forms and assent forms, the questionnaire, and the clinical evaluation forms. For example, a 5-page questionnaire costs $0.25 per participating student, and therefore costs $0.20 per enrolled student, assuming an 80% participation rate.

The prevalence of emotional distress greatly affected the overall costs of the program, given that Stage 2 clinical evaluation of positive screens was significantly more costly than Stage 1 screening. The prevalence of distress varied widely by school, ranging from 7.8% to 29.9%. Therefore, overall costs per enrolled student, per positive screen, and per referral were calculated for four hypothetical estimates of the prevalence of emotional distress: 5%, 10%, 15% and 20%.

Measuring cost -effectiveness

Additional data were collected in 2006 concerning the outcomes of referral recommendations. Six weeks after the clinical evaluation, parents of referred students were called to determine whether the student had been successfully linked to recommended supports. Cost-effectiveness was then calculated by assessing overall costs of screening per successful linkage.

Estimating prevalence of positive screens

A predictive model based on school demographic and academic characteristics estimated the proportion of a given school population expected to screen positive. The prevalence of emotional distress, estimated as the proportion of participants who screened positive, was available from 6 schools over 5 years, with the number of years a school implemented the intervention program varying from 1 to 4. Forward stepwise linear regression was used to predict prevalence of positive screens, assessed as a continuous variable. Several school-level predictors were evaluated, including % male gender, % receiving reduced-fee lunch, % English Language Learner, % Special Education, % single parent households, and % suspended in the past year, using p<.05 for inclusion and p>.10 for exclusion. A repeated measures analysis using a generalized estimating equation was conducted to determine whether within school clustering across multiple years affected results. For this analysis, school was the “within subjects” variable, normal distribution of prevalence was assumed, and a corresponding linear link function was used. Finally, the prediction model was validated by estimating the proportion of students who would screen positive at two schools new to DPSP in the final year that were not used to develop the prediction model. Analyses were conducted using SPSS, version 13.0³⁵ and STATA version 7.0³⁶ statistical software.

Results

Cost Outcomes

The cost associated with Stage 1 was estimated to be $6.96 per student enrolled (Table 1). The cost associated with Stage 2 was estimated to be $38.40 per student screening positive (Table 2). Using these figures, the hypothetical cost and cost-effectiveness of screening by prevalence of positive screens in a school population was calculated (Table 3). As the prevalence of emotional distress increases, cost per enrolled student is expected to increase, from $ 8.88 per enrolled student when 5% of students screen positive to $13.64 when 20% screen positive. The cost per positive screen decreases from $177.60 to $68.20, and the cost per referral decreases from $296.00 to $113.66 as the prevalence of emotional distress increases from 5% to 20%.

Table 1. Stage 1 Costs of Screening.

	Task	Cost per student receiving service	% of enrolled receiving service	Cost per Enrolled student
Labor	Preparing/analyzing data 1 staff $20/hr 10 hr/100 students	$2.00	100%	$2.00
	Preparing materials 1 staff $13/hr 5 hr/100 students	$0.65	100%	$0.65
	Recruiting students 2 staff 13/hr 1 hr 100% of school 25 students/class	$1.04	100%	$1.04
	Administering screen for regular classes 2 staff $13/hr 1.5 hr 85% of school students/class	$1.56	85%	$1.32
	Administering screen for ELL, SPED classes 3 staff $13/hr 1.5 hr 14% of school 25 students/class	$2.34	14%	$0.33
	Questionnaire admin for students needing 1:1 1 staff $13/hr 1.5 hr/class 1% of school 1 student/class	$19.5	1%	$0.20
Overhead	Information packet for parents with 20% duplicates for students who lost forms	$0.50	120%	$0.60
	Incentives to participate in screening	$0.50	80%	$0.40
	Forms for nonparticipating students	$0.10	20%	$0.02
	Forms for participating students	$0.50	80%	$0.40
Total				$6.96

Table 2. Stage 2 Costs of Clinical Evaluation (CE).

	Task	Cost per student receiving service	Cost per Student having CE
Labor	Administration of the CE 1 staff $25/hr 1.25 hrs	$31.25	$31.25
	Telephone call to parent 1 staff $25/hr .25 hrs	$6.25	$6.25
	Translator used for 5% of phone calls 1 translator $32/hr .25 hrs	$16.00	0.40
Overhead	CE packet	$0.50	$0.50
Total			$38.40

Table 3. Overall costs to a school for screening, by % screening positive.

	5% screen +	10% screen +	15% screen +	20% screen +
$ per enrolled	$8.88	$9.80	$11.72	$13.64
$ per positive screen	$177.60	$98.00	$78.13	$68.20
$ per referral	$296.00	$163.33	$130.22	$113.66
$ per successful linkage	$416.90	$230.05	$183.41	$160.09

Cost effectiveness outcomes

During the final year of DPSP, 80 students were referred to 125 resources, with 74% of all referrals to existing school-based services, most frequently for academic support. At the 6-week follow-up, parents reported that 72% of recommended linkages had been made. The success rate varied from 68.2% for academic referrals to 90% for medical and other referrals (Table 4). The cost per successful linkage ranged from $416.90 when 5% of students screened positive to $160.09 when 20% of students screened positive (Table 3).

Table 4. Clinical Evaluation Referrals and Successful Linkages.

Referral Type	Referrals Made		Successful Linkages
	N	%	N	%
After School Activity	22	17.6	18	81.8
Academic Support	44	35.2	30	68.2
School Counselor	21	16.8	15	71.4
Outside Mental Health Agency	28	22.4	18	69.2
Medical and Other Services	10	8.0	9	90.0
Total	125		90	72.0

Estimating prevalence of positive screens in a school

The demographic and academic indicators (% male gender, % receiving reduced-fee lunch, % English Language Learner, % Special Education, % single parent households, and % suspended in the past year) were moderately to highly correlated, with Pearson's correlation coefficients ranging from r=0.43-0.86, and there was considerable overlap in their associations with emotional health status, with several variables explaining similar portions of the variance (data not presented). Multiple regression analysis was used to develop a predictive model for estimating the percent of students who would screen positive. Forward stepwise linear regression, showed first % ELL (adjusted R² =.49) and then % receiving reduced fee lunch (adjusted R²= .61) significantly improved model fit (Table 5) The final predictive model was as follows:

Table 5. Multiple linear regression model with step-wise forward regression selection for predicting prevalence of positive screens based on school demographics.

Variable	Adjusted R2	Standard Error	P-value
% reduced fee lunch	.53	0.15	0.036
% English Language Learner	.62	0.58	0.032

PREVALENCE OF POSITIVE SCREENS = 3.57 + 0.16(% reduced fee lunch) + 0.59(% ELL) A generalized estimating equations analysis, conducted to account for having data from the same schools over several years, did not alter results appreciably (data not presented). Therefore the simpler model was retained in the final analysis. To gauge the accuracy of the model, the standard residuals were examined and found to be normally distributed, ranging from -1.22 to 1.44, SD .90.

Testing the predictive model on the two new schools yielded percent positive screen estimates of 16.0% and 27.8%, for schools 1 and 2, respectively, compared to observed prevalences of 21.0% (95%CI 12.8-29.1%) and 26.8% (95%CI 18.2-34.8%), respectively.

Discussion

This study describes the cost and cost-effectiveness associated with implementing a school-based universal emotional health screening program. Screening was estimated to cost between $8.88 and $13.64 per enrolled student based on hypothetical screen positive prevalences between 5% and 20%. As the proportion of students who screened positive increased, the cost per enrolled student increased somewhat, whereas the cost per positive screen, per referral, and per successful linkage decreased sharply. Cost effectiveness was estimated to be $416.90 per successful linkage with a 5% screen positive prevalence, and $106.09 for schools with a 20% screen positive prevalence. A formula for estimating the proportion of students screening positive for emotional distress in a given school proved accurate to within 5% when applied to two schools not used to develop the predictive model.

Given the lack of cost-effectiveness studies of other mental health screening programs, comparing costs of DPSP to the costs of other programs is difficult. The only cost-effectiveness study of a school-based mental health program estimated costs to be $1439 per student who received a referral for clinical depression, anxiety or substance use disorders;²⁴ however these costs are not directly comparable to figures from DPSP, who included students with sub-clinical levels of a broader range of emotional health conditions. School-based screening for scoliosis was estimated to cost between $133.63 and $198.27 per student with confirmed disease referred for treatment,²³ which is roughly comparable to DPSP's estimated $113.66-$296.00 per referral.

Implementation costs involved in carrying out a school-based emotional health screening program were based on the DPSP model. In the DPSP, resources were invested to ensure confidentiality and to increase the students' and parents' comfort level with privacy accommodations. For example, DPSP utilized non-school personnel as screening staff to help maintain confidentiality and to elicit unbiased responses, and DPSP attempted to implement universal screening, rather than focusing on selected or indicated groups of enrolled students. Several cost-saving modifications and their potential ramifications are discussed below.

One cost-saving modification would be to substitute teaching staff for university staff when carrying out screening. For example, having teachers distribute and collect permission forms would save up to $1.04 per enrolled student in labor costs; if teachers also administered the screening questionnaire, schools could save up to $1.85 per enrolled student. While enlisting teachers in recruitment and questionnaire administration may be more efficient and less expensive, participation rates or response validity could decline, if parents and students were concerned about school staff knowing a student's emotional health status.³⁷ Additionally, school staff may not wish to conduct screening for a variety of reasons, such as discomfort with the time commitment or concern that screening is beyond the scope of their professional role.

Another alteration would be to reduce accommodations for students requiring 1:1 administration, as has been practiced in other programs.^12,16 In order to be equitable and have the greatest public health impact, DPSP staff included as many students as possible, despite the fact that labor costs of individual administrations to ELL students, students with behavioral disorders, deaf students, and others with special needs were high. For schools with large numbers of ELL students or other special needs students, the costs of screening could become prohibitive. Despite the cost, one potential concern with selective exclusion of these students is that a greater proportion of these groups of students may have unmet mental health needs.³⁵ However, many special needs subgroups are by definition already identified and are receiving additional support. For example, DPSP found that only 2 of the 33 screened students with a “severe behavioral disability” designation had unmet needs requiring referral. On the other hand, an inclusive screening program may be perceived as more equitable and therefore appropriate to the community than one that excludes “difficult to screen” subgroups, even if these excluded subgroups sustain fewer benefits from screening.

Several other alterations to this screening model are worth considering. Screening could be completed with fewer staff per classroom to reduce labor costs, incentives for returning permission forms and completing screens could be reduced or eliminated, and the screening positive threshold could be raised. School nurses or counselors could complete clinical evaluations. Any alteration may result in tradeoffs in areas such as confidentiality, participation rates, acceptability to the community, and early identification of distressed students.

Another option for allocating scarce resources would be for school districts to only provide emotional health screening in schools with a high prevalence of emotional distress. Reducing the number of schools participating in a screening program could result in significant overall cost savings, and targeting high yield schools could result in significant savings per referral and linkage as well. While this approach would avoid the aforementioned tradeoffs in confidentiality and participation rates, it may not be acceptable to the community as a whole.

Several limitations with this study should be noted. First, estimating the sensitivity, specificity, and negative predictive value of the screening program was not possible, because only students who screened positive received clinical evaluations, and the true status of negative screening students was unknown. False negative screens can engender a false sense of well-being, that may lead to undesirable consequences for students who have undetected mental health needs. An MFQ cut-off of 12 has been demonstrated to have optimal sensitivity (88%-90%) and specificity (70%-77%) for predicting clinical depression;^39,40 the cut-off of 20 used in DPSP would be expected to result in fewer false positives and more false negatives. However, the psychometric properties at a 20 cut-off have not been published, and validity of the MFQ for predicting non-clinical depression has not been established; therefore future research is needed.

Another limitation is that student participation in screening required written parental permission, which has been shown to disproportionately reduce participation within subpopulations at higher risk of emotional distress.⁴¹ Given the current sociopolitical climate surrounding mental health screening,^42,43 obtaining passive permission (students may participate unless parents opt out) rather than active permission (written parental permission is required for participation) may prove unacceptable to schools. A further limitation was a lack of comparison between costs of DPSP and costs of less expensive methods for identifying distressed students. For example, teacher nominations would be less resource intensive; however research suggests that this method has less than 50% sensitivity to identify children experiencing internalizing disorders.^43,44 Therefore, less costly strategies such as teacher nomination may be significantly less cost-effective when compared to administering self-report questionnaires. Finally, a small number of schools were used to establish the predictive model for prevalence of emotional distress, and analysis of the standard residuals indicated a large margin of error. Estimates from this model can provide only rough guidelines for expected results.

Another limitation was that the costs associated with expected increased service utilization were not included in this analysis. Referrals were to diverse resources based on individual needs of the students and were highly reliant on the unique resources of the school and community; therefore these costs would be difficult to estimate and generalizability to other school districts may be limited. However, these costs could be significant, depending on the number and type of referrals made, and they are worth careful consideration by any school implementing such a screening program.

Without question, the DPSP program requires a significant investment of resources; however, the savings could far exceed these investments if costly adverse outcomes were avoided. This study unfortunately lacked long-term effectiveness data; however if screening were to influence outcomes such as decreasing emotional distress, reducing intensive mental health care needs, improving school performance, and preventing drop-out, significant societal cost savings could result. When screening for diseases such as breast cancer, a large body of literature supports treatment effectiveness such that often only costs of screening are calculated, and then established costs and effectiveness of various treatment arms are used to calculate overall cost-effectiveness. With emotional health screening, effectiveness of interventions for identified students is not as well established. In the case of DPSP, effectiveness largely depends on the existing resources which can vary greatly among schools and communities. Therefore, questions concerning long-term effectiveness should be considered in future research.

This evaluation provides valuable information for schools considering implementing mental health screening programs. Detailed analyses of the costs and effectiveness of programs will enable districts to make evidence-based decisions about how to support the emotional health of students.