Assessment of Exposure to High-Performing Schools and Risk of Adolescent Substance Use: A Natural Experiment

Abstract

Importance

Although school environments are thought to influence health behaviors, experimental data assessing causality are lacking, and which aspects of school environments may be most important for adolescent health are unknown.

Objective

To test whether exposure to high-performing schools reduces risky adolescent health behaviors.

Design

We used admission lotteries—which mimic random assignment—to estimate the causal effect of school environments on adolescent health. We surveyed 1270 students who applied to high-performing public charter schools in low-income minority communities in Los Angeles. We followed lottery “winners” (Intervention) and “losers” (Control) from the end of 8^th grade/beginning of 9^th grade through the end of 11^th grade. Intent-to-Treat (ITT) and Instrumental Variables (IV) techniques estimate the effects of “winning” the lottery and attending high-performing schools on health behaviors and whether effects varied by gender.

Setting:

Charter and non-charter public high schools in Los Angeles.

Participants:

Students applying to one of the 5 public charter schools in Los Angeles where the majority enrolled were economically disadvantaged, the school’s academic performance ranked in the top tertile of LA County public high schools, there were at least 50 more applicants than seats available, and they used an admissions lottery.

Main Outcomes and Measures:

Primary outcomes were 30-day marijuana use and high-risk marijuana use. Additional health outcomes included 30-day alcohol use, alcohol misuse, ever being in a fight, ever having sex and past year delinquency. We also examined potential intermediate factors (time studying, truancy, school mobility, school culture, school order, teacher support, and proportion of substance using peers in students’ social networks).

Results

Intent-to-treat analysis showed that lottery “winners” (n=694) reported less marijuana misuse than loterry “losers” (n=576), as well as fewer substance using peers, more time studying, less truancy, greater teacher support, more orderly schools, and less school mobility(all p<0.05). Stratified analyses suggest more consistent effects for boys with treatment effects noted as early as 9^th grade.

Conclusions and Relevance

This natural experiment provides evidence that school environments can improve risky behaviors for low-income minority adolescents.

Education is a key social determinant of health.^1–3 Most studies measure education by academic achievement or attainment.⁴ However, there is growing recognition that school environments likely influence health through pathways that do not necessarily depend on academic gains.⁵ In addition to developing students’ knowledge and skills, schools function as social institutions, connecting adolescents with peers and adults, transmitting social norms, and encouraging or discouraging specific behaviors. Understanding whether and how the school environment itself might be intentionally engineered as a platform for health promotion is critical to designing effective health and education policies.⁵

A number of observational studies find associations between measures of a supportive school environment and health behaviors such as substance use.^6–8 A few quasi-experimental studies suggest exposure to better educational environments improves health behaviors.^9–14 However, none of these examine the transition to high-performing schools or provide detailed data about school environments, social networks or other factors that might explain how schools impact adolescent health. Hence it remains unknown whether changes in the high school environment reduce adolescent substance use and, if so, which aspects of the school environment matter most.

To test whether and how school environments impact substance use, we exploit a natural experiment to study whether attending high-performing charter high schools led to improved health behaviors for low-income minority adolescents. Although both charter and traditional public schools vary in terms of composition and outcomes, compared to traditional public schools, charter schools tend to have lower enrollment, serve fewer special education students, and have a higher proportion of low-income and minority students.¹⁵ In the Reducing Inequities through Social and Educational change follow-Up study (RISE Up), we capitalized on the lottery admission for several high-performing charter schools in low-income neighborhoods of Los Angeles (LA) and prospectively followed a cohort of adolescents from high school admission through 11^th grade. In addition to examining our primary substance use outcomes, we examined other risky health behaviors, such as sexual activity, violence, and delinquency. We also test several hypothesized pathways linking school environments and health,¹⁶ including school climate measures and peer network characteristics, to understand the mechanism of how better school environments might lead to better health.

METHODS

This is a longitudinal natural experiment to study the effects of high-performing schools on the health of low-income, minority adolescents.

School Recruitment

RISE UP Study charter schools were selected on the following criteria: 1) location in LA county; 2) majority of students enrolled are economically disadvantaged (qualify for free/reduced price lunch); 3) academic performance in the top tertile of public high schools in LA County based on 2012 state standardized test scores (API); 4) oversubscription of applicants (more than 50 more applicants than seats available); and 5) use of an admissions lottery. We identified 91 public charter high schools in LA County, of which 32 satisfied the first three criteria. All 5 schools that met the remaining two criteria agreed to participate in the study.

Participant Recruitment

From each school, we randomly sampled, stratified by lottery result, from the list of lottery winners and losers over two consecutive years to identify potential study participants. Students were eligible for the study if they applied for 9th grade admission to one of the study schools for the fall of 2013 or 2014, spoke English or Spanish fluently, and resided within LA County. Siblings of current students were excluded since they were admitted outside of the lottery. Potential participants were contacted and parental consent and student assent were obtained. All procedures were reviewed and approved by the RAND Institutional Review Board. Of the 1996 potential participants identified (Figure 1), 487 were ineligible, 239 refused participation and 1270 consented to participate (participation rate=84%). 576 were “lottery losers” (Control Group) and 694 were “lottery winners” (Intervention group). We did not have control over the random assignment via admissions lottery, but we reviewed each school’s lottery procedures to confirm only basic contact information, and not demographic or academic achievement information, were contained in the lottery application.

Figure 1.

Participant Recruitment

Data Collection

Bilingual research assistants completed a face-to-face, baseline computer-assisted survey with students at the end of 8th grade or fall of 9th grade and follow up interviews during 10th and 11th grade. Interviews were conducted at a location of the participant’s choice that afforded sufficient privacy, typically their home or school. Sections asking about risky health behaviors were collected using a computer assisted self-interview (CASI) to encourage honest responses.¹⁷ There was no significant difference in survey completion or retention by study arm and 87.7% of participants were followed through 11^th grade. Data on participants’ schools was collected from the California Department of Education.

Measures

Risky Health Behaviors:

Because early marijuana use is associated with more risky substance use and other negative health behaviors over time, ^18–20 our primary outcomes of interest were 30-day marijuana use and high-risk marijuana use. At each wave, students self-reported whether they used marijuana in the previous 30 days and completed a marijuana misuse scale (alpha=0.85), an index we adapted from the alcohol misuse scale²¹ of 8 high-risk behaviors that are associated with developing a substance use disorder (e.g., getting in trouble because of marijuana use, missing school because of marijuana use, using marijuana at school). The index assessed behaviors in the prior 12 months and ranged from 0–8, with higher scores representing more risky substance use.

Additional health outcomes included 30-day alcohol use, the alcohol misuse scale (alpha=0.88, range 0–9), ever being in a fight, and ever having sex.²² Students also completed a delinquency scale, in which they reported the frequency of engaging in 8 delinquent behaviors that have been associated with negative life outcomes in the previous 12 months.²³ The resulting delinquency index (alpha= 0.60) ranged from 0–8 with higher scores representing a greater number of delinquent behaviors.

Secondary Outcomes:

We measured factors related to diminished opportunities and motivation to engage in risky health behaviors,¹⁶ including social ties to more risky peers.^24,25 We used a standard procedure for collecting personal social network data.^26,27 At each wave, students named 20 people in their social network, identified which of those individuals were peers, and reported whether those peers engaged in alcohol and marijuana use. Based on these responses, we derived separate measures of the proportion of peers in the network using marijuana and alcohol. In addition, increased academic engagement and accountability can reduce unsupervised time outside of school, limiting opportunities for risky behaviors and increasing motivation to engage in pro-social behaviors.^28,29 Hence, students reported the amount of time per day spent studying, whether they cut school in the last 12 months, and completed a 14-item school engagement scale (alpha=0.80 range 1–4).³⁰ To assess school culture, which may impact behavioral norms and social incentives, students completed a 9-item measure of school order (alpha=0.68, range 1–4, higher scores indicate more orderly environment) adapted from a measure of home chaos,³¹ a 4-item measure of school safety (alpha=0.81, range 0–3),³² and rated the level of teacher support for college attendance (range 1–4).³³ We also included a measure of school social culture; participants indicated whether 12 traits or behaviors (“being a good student,” “disrupting class”, “bringing drugs to school”) would increase or decrease popularity (alpha=0.84, range 1–5) in their school. Finally, participants reported whether they changed schools by the final survey wave. School mobility is thought to increase motivation to engage in risky behaviors as adolescents seek to establish new friends and their place in the social order.³⁴

Covariates:

Due to the lottery design, assignment to Intervention or Control is random. However, students who applied to multiple charter schools had a greater chance of “winning” at least one lottery. We followed the charter lottery literature and controlled for the set of schools to which students and their parents reported applying (the “risk set”).³⁵ Since students are randomly assigned to Intervention, we do not need to control for additional covariates to get unbiased estimates, however including covariates improves power and may reduce bias in small samples that are unbalanced due to random chance. We therefore control for a number of covariates collected at baseline: gender, race/ethnicity (Latino vs. not), native language, being born in the United States, grade point average in 8^th grade, parental education, parental birthplace, parental employment, family structure, and parenting style (as measured by the index of parenting style)³⁶. Since substance use might vary by time of the school year, we also controlled for the month that the survey was conducted.

Analytic Strategy

We used t-tests and chi-square tests to compare Intervention and Control participants on baseline covariates as well as parental substance use. Using the xtmixed command in STATA 14.0, we performed hierarchical longitudinal analyses to estimate the causal effect of “winning” the lottery on each outcome. We included random effects for student and school to account for multiple observations over time for each individual and nesting of students within schools. We include the covariates described above, fully interacted with grade. We excluded observations from grade 8 since this is before the “intervention” began. We refer to these as the Intent-to-Treat (ITT) estimates. Some students who won the lottery did not enroll in a top-tertile public school and vice-versa. The ITT estimates will therefore underestimate the impact of actually attending a high-performing school. We used instrumental variables (IV) analyses to estimate the effects of enrolling in a top tertile public school using two-stage least squares (2SLS).^1,13 Under the assumption that winning the lottery affects outcomes only through enrollment in a high-performing school, the IV estimate is the “local average treatment effect” (LATE) for the students whose decision to enroll in a charter school is determined by the lottery results. Finally, because both the factors associated with marijuana use^37,38 (our primary outcome) and the relationship between school environments and health likely differ for boys and girls,³⁹ we conducted gender-stratified analyses. Missing data for all variables (including outcomes) and loss to follow up/refusals were multiply imputed. Less than 1% of data were missing for each variable, except parental education, which had 4.8% missing data. Imputed results were similar to results using un-imputed data.

RESULTS

Study participants attended 147 different high schools at 9th grade. Of these, 79 (54%) were traditional public schools, 41 (28%) public charter schools, 17 (11%) parochial, 4 (3%) private, and 6 (4%) were other school types (continuation, alternative, home school, or out of area). Compared to the other 142 schools in the sample, the 5 charter schools that made up the study’s sampling frame were smaller (mean enrollment 463 vs. 793) had higher API scores (mean API 787 vs. 730), and served a higher proportion of minority students and those with low parental education (all t-tests p<0.001) (eTable 1). As expected, the intervention group was much more likely to matriculate into a top-tertile school than the control group (87.9% vs. 29.6% in 9^th grade, p<0.001), but there was cross-over in both arms, as there are other schools in the region that are also high-performing.

The sample is representative of low-income minority communities in Los Angeles, with 89.5% identifying as Latino and only 52% with at least one parent who graduated from high school. The characteristics of lottery winners and losers were not significantly different, with the exception of baseline grade point average (see Table 1), which was controlled for in all analyses.

Table 1.

Baseline Characteristics of Intervention and Control Groups

	Intervention N=694	Control N=576	P-value
Male	325 (46.8%)	277 (48.1%)	0.65
Race/Ethnicity
Latino	636(91.6%)	501 (87.0%)	0.06
Black	29(4.2%)	38 (6.6%)
White	12(1.7%)	14 (2.4%)
Other	17 (2.5%)	23 (4.0%)
US born	608 (87.6%)	505 (87.7%)	0.97
Native English speaker	286 (41.2%)	233 (40.5%)	0.78
Grade Point Average(8thgrade)			0.001
No transcripts	149 (21.5%)	77 (13.4%)
A (3.5–4.0)	163 (23.5%)	120 (20.8%)
B (2.5–3.4)	246 (35.4%)	236 (41.0%)
C (1.5–2.4)	114 (16.4%)	120 (20.8%)
D or lower (<1.5)	22 (3.2%)	23 (4.0%)
At least 1 parent:
Born in US	182 (26.2%)	154 (26.7%)	0.84
Graduated high school	375 (54.0%)	291 (50.5%)	0.21
Full-time employed	612 (88.2%)	493 (85.6%)	0.17
Family structure			0.14
2-parent family	579 (83.4%)	457 (79.3%)
1-parent family	100 (14.4%)	107 (18.6%)
Non-traditionalguardian**	15 (2.2%)	12 (2.1%)
Parenting Style			0.23
Neglectful	231 (33.3%)	190 (33.0%)
Indulgent	116 (16.7%)	120 (20.8%)
Authoritarian	155 (22.3%)	126 (21.9%)
Authoritative	192 (27.7%)	140 (24.3%)
Student has seen parent drunk	185 (26.7%)	146 (25.5%)	0.62
Student reports parent used marijuana	13 (1.9%)	17 (3.0%)	0.20

Std. dev.= standard deviation;

^*Non-traditional guardian includes a relative (grandparent, aunt, uncle, sibling) or foster parent. Parental substance use was measured by asking participants if they have ever seen their parent drunk and if their parent had ever used marijuana.

The ITT analyses (Table 2) demonstrated that lottery winners had less, marijuana misuse than lottery losers, as well as fewer substance using peers, more time studying, less truancy, greater teacher support, more orderly schools, and less school mobility. Results were similar before (data not shown) and after adjusting for covariates (Table 2). ITT analyses stratified by gender (eTable 2) showed that, that among boys, lottery winners had significantly lower marijuana use and alcohol misuse scores compared to lottery losers, whereas there were no significant health effects noted for girls.

Table 2.

Intent-To-Treat Analyses Testing the Effect of being Offered Admission to a High-Performing School on Health Behaviors and School-Related Outcomes

Health Behaviors	Intervention	Control	Difference (Intervention- Control)	95% CI
Percent using marijuana	8.41	10.85	−2.44	(−5.14, 0.25)
Marijuana misuse score	0.46	0.71	−0.25	(−0.46, −0.05)
Percent using alcohol	11.8	12.6	−0.8	(−3.76, 2.07)
Alcohol misuse score	0.72	0.91	−0.19	(−0.44, 0.06)
Percent fighting	12.92	13.22	−0.30	(−3.36, 2.76)
Delinquency score	0.54	0.64	−0.10	(−0.26, 0.06)
Percent sexually active	11.4	12.0	−0.6	(−3.7, 2.43)
School-Related Outcomes
Percent of peers using marijuana	9.6	12.7	−3.2	(−5, −1.3)
Percent of peers using alcohol	5.94	6.33	−0.39	(−1.77, 1)
Time studying (hours)	2.63	2.49	0.14	(0.01, 0.27)
Percent with no truancy	84.29	77.31	6.97	(3.4, 10.55)
School engagement score	9.02	9.05	−0.04	(−0.13, 0.05)
Teacher support for college score	7.20	7.02	0.18	(0.1, 0.27)
School safety score	4.55	4.52	0.03	(−0.06, 0.12)
School order score	7.06	6.83	0.23	(0.13, 0.33)
School culture score	3.97	3.92	0.05	(−0.05, 0.15)
Percent who changed schools	21.38	28.41	−7.03	(−0.12, −0.02)

95%CI= 95% Confidence Interval of the difference. Bold indicates p<0.05. Table reports the predicted outcome in the control and intervention groups at the mean of the controls; the third column in each set reports the coefficient on winning the lottery with 95% confidence intervals. Models were estimated using mixed-effects multi-level regression models controlling for the following covariates: number of schools to which students applied, gender, race/ethnicity (Latino vs. not), native language, being born in the United States, grade point average in 8^th grade, parental education, parental birthplace, parental employment, family structure, parenting style, and interview month.

Instrumental variables (2SLS) analyses (Figures 2 and 3, eTable 3) estimate the effects of attending a high-performing school on our outcomes, separately by grade and gender. The estimates for boys more consistently suggest benefits of attending a high-performing school than they do for girls, although the differences in treatment effects by gender are not statistically significant.. Among boys, treatment effects on 30-day marijuana use, and marijuana and alcohol misuse were negative (indicating improvement) and significant; there were no significant treatment effects on health for girls. For both genders, attending a high-performing school reduced exposure to marijuana-using peers and truancy and increased report of teacher support for college and an orderly school environment. .

Figure 2.

Effects of Attending a High-Performing High School on Substance Use, by Grade and Gender Notes: Figure plots the estimated treatment effects of attending a high-performing high school with 95% confidence intervals, separately for males and females. We used instrumental variables analysis, and attendance was instrumented with an indicator for winning the charter school lottery. Models control for the number of schools to which students applied, gender, race/ethnicity (Latino vs. not), native language, being born in the United states, grade point average in 8th grade, parental birthplace, parental employment, family structure, parenting style, and interview month.

Figure 3.

Effects of Attending a High-Performing High School on School Outcomes, by Grade and Gender Notes: Figure plots the estimated treatment effects of attending a high-performing high school with 95% confidence intervals, separately for males and females. We used instrumental variables analysis, and attendance was instrumented with an indicator for winning the charter school lottery. Models control for the number of schools to which students applied, gender, race/ethnicity (Latino vs. not), native language, being born in the United states, grade point average in 8th grade, parental birthplace, parental employment, family structure, parenting style, and interview month.

. A sensitivity analysis examining the proportion of substance-using peers in participants’ school-related versus non-school-related social networks revealed similar findings.

DISCUSSION

In this quasi-experimental study, we found that students attending high-performing schools because of “winning” the charter school admissions lottery had lower rates of risky health behaviors, particularly with respect to substance use. The estimated effects were more consistently beneficial for boys, though not statistically distinguishable from the effects for girls. The fact that these findings are from natural experiment, rather than observational data, provides compelling evidence that school environments can impact health for low-income minority students and suggests that investing in healthy schools might yield valuable population health returns.

The potential differences in effects for boys versus girls are consistent with studies suggesting gender differences in both the factors associated with risky health behaviors and the potential for school environments to intervene on these factors.⁴⁰ For example, prior evidence suggests that boys’ social networks are more strongly determined by their school environment.⁴¹ Further, typical school environments may be more negative for boys. This is consistent with findings from a randomized trial of early high-quality education,⁹ which suggested low quality educational environments are more harmful for boys than girls.⁴²

We also noted more consistent effects on marijuana use as compared to alcohol use. This finding may be because marijuana use represents a more deviant behavior than alcohol use. For example, in 2015, 55% of Hispanic high school students in Los Angeles report ever using alcohol, while 36% reported marijuana use.²² It remains to be seen whether this pattern changes as marijuana policies change.

Prior research has posited that the link between education and health outcomes might be driven through acquiring greater knowledge, accessing better resources, or improving non-cognitive skills.⁴ We found evidence of immediate reductions in risky health behaviors, as early as 9th grade, which is likely inconsistent with mechanisms that depend wholly on improved skills or knowledge. The positive impacts noted on intervention students’ social networks and school culture provide an alternative and suggestive pathway. These factors are likely to change upon entering a new school environment and alter the opportunities and motivations for adolescents to engage in risky behaviors.

If, indeed, the impact of school environments on risky health behaviors operates largely through the social interactions that schools facilitate, then greater attention to constructing healthy social environments is warranted. The high-performing schools in this study may have achieved their impact on health behaviors purely by isolating students from more deviant peers in other schools. If so, widespread application of this strategy is problematic, to say the least, and would likely magnify health disparities. Alternatively, the effects noted here may be driven by practices that enhance a positive school culture and supportive adult and student interactions to reinforce those healthy norms. While the current study cannot distinguish between these two potential pathways, the positive findings related to increased perceived teacher support, school order, and a more pro-social school culture (for boys) suggest important mechanisms for future study.

This study presents a rare opportunity to quantify impacts of high-performing schools on health using quasi-experimental data. We did observe cross-over from both study arms, so the simple intervention-control differences (ITT) will under-estimate the impact of school environments on health; the 2SLS instrumental variables estimates adjust for crossovers. The charter schools in this study may not be representative of charter schools on the whole and, while we attempted to characterize how school environments differed for intervention vs. control students, there may be additional important differences that were not measured here. Also, all data are self-reported and, although participants were assured of the confidentiality of their responses, social desirability might have influenced participants’ answers and this impact might vary by group assignment. While intervention and control students might have had different exposure to the neighborhood around their schools, which may impact their behaviors, we believe this effect is likely small, as the study charter schools are located in the same general area as the majority of the comparison schools. The multiple outcomes increase the chances of wrongly rejecting some null hypotheses. However, the outcomes are highly correlated and the consistent pattern in our findings makes this unlikely. Importantly, this sample is predominantly Latino, comes from a single (albeit large) urban area, and is limited to students who applied to an oversubscribed high-performing charter school, so results might not be generalizable to other groups. Also, we present relatively short-term impacts here and a longer follow up time might yield different results.

Despite these limitations, this study provides compelling evidence that school environments can impact health, and that impacts can be immediate and may be particularly beneficial to low-income, minority boys. Social experiments, like this one, are critical for informing effective policies and practices, especially when impacts and implications cross multiple sectors. Our findings suggest that investing in high-quality school environments can yield important health returns in addition to academic returns and hence may be of even greater societal value than previously believed. Further, the intermediate outcomes here suggest specific aspects of a school environment that might be leveraged to support health not just in charter schools. Investing in high-quality public schools may be an important strategy for achieving health equity for all students, and particularly the most disadvantaged. Future studies targeting school-based social networks and school culture, for example, can begin to identify the pathways through which to build healthier schools. Opportunities to rigorously study such interventions and to determine the long-term impacts on health and wellbeing are critically important if we are to move from merely documenting to harnessing the social determinants of health.

KEY POINTS.

Question:

Does exposure to high-performing school environments reduce risky health behaviors for low-income minority high school students?

Findings:

In this natural experiment of 1270 students who applied via admissions lotteries to high-performing public charter schools in low-income minority communities in Los Angeles, lottery “winners” had lower marijuana misuse scores, fewer marijuana using peers, less truancy, greater teacher support for college, more orderly school environments, less school mobility, and spent more time studying than lottery “losers”.

Meaning:

School environments may impact risky health behaviors and constitute an important prevention tool and target for addressing the social determinants of health.