Helped into Harm: Mediation of a housing voucher intervention on mental health and substance use in boys

Abstract

Background:

Interventions can have harmful effects among subgroups they intend to help. The Moving To Opportunity (MTO) experiment, in which families were randomized to receive a Section 8 housing voucher, was one example. Voucher receipt generally resulted in better long-term mental health and lower substance use and risk behavior outcomes among adolescent girls, but resulted in worse outcomes among adolescent boys. Reasons for this discrepancy and the unintended harmful health effects for boys are unclear. We used mediation analysis to estimate processes through which voucher receipt was hypothesized to affect adolescent mental health and substance use.

Methods:

We used longitudinal data (10–15 years) on boys enrolled in MTO. We estimated interventional (also known as stochastic) indirect effects of voucher receipt on mental health and substance use outcomes through mediators capturing aspects of the school environment, neighborhood poverty, and instability of the social environment. We also estimated interventional direct effects not operating through these mediators. We used a robust, efficient, nonparametric substitution estimator in the targeted minimum loss-based framework.

Results:

Housing voucher receipt increased long-term risk of any DSM disorder, any mood disorder, any externalizing disorder, and cigarette smoking among boys. The majority (between 69–90%) of the total negative long-term effects could be explained by indirect effects through the mediators considered.

Conclusion:

This evidence suggests that, even though the intervention had the desired effects on neighborhood poverty and the school environment, these “positives” ultimately negatively impacted the long-term mental health and behaviors of boys.

Introduction

Interventions sometimes have harmful effects among subgroups they intend to help (e.g., ^1–5). The Moving To Opportunity (MTO) experiment was one example of this. MTO was a large-scale longitudinal randomized control trial conducted by the Department of Housing and Urban Development from 1994 to 2010 in which families living in low-income, high-rise public housing in five cities: Baltimore, Boston, Chicago, Los Angeles (LA), and New York (NYC), could sign up to be randomized to receive a Section 8 housing voucher. Such vouchers, which exist today, subsidize the recipient’s rent on the private market, thereby allowing families to move out of public housing. MTO’s effects on economic, education, and health outcomes have been studied extensively.(e.g., ^6–14) In terms of health outcomes among children who were randomized, housing voucher receipt (vs. not) generally resulted in better mental health and lower substance use and risk behavior long-term outcomes among adolescent girls, but resulted in worse outcomes among adolescent boys (e.g., higher rates of post-traumatic stress disorder, smoking, problematic drug use, etc.).(8, 15–19)

Reasons for this discrepancy and the unintended harmful health effects for boys are unclear, though research has suggested possibilities. For example, voucher receipt may have resulted in more moves and school changes,(15, 20) the instability of which has been shown to have detrimental effects on youth mental health, substance use, and risk behavior,(21–25) especially for adolescent boys.(26) Additionally, voucher receipt was hypothesized to benefit families by reducing exposure to neighborhood poverty,(27, 28) yet this may have inadvertently caused harm through incidental exposure to social instability, alienation, and discrimination,(29–31) to which boys may be more sensitive.(15, 32) In addition to moving out of poverty, one primary reason parents reported for participating in MTO was for their children to go to better schools(19, 33), which may improve both child health and lifetime achievement.(34–36) Indicators of the school environment, including academic ranking, students living under the poverty threshold, and student-teacher ratio, improved for some groups moving with the voucher(20); however, most recipients ultimately did not attend schools that were markedly better-performing than those attended by non-voucher recipients,(20) likely mitigating the positive affect of school environment change on long-term health outcomes.

Mediation analysis can examine the hypothesized processes by which the intervention acted to affect long-term outcomes, and consequently the processes by which these unintended harmful effects may have arisen. Recent work has suggested that unintended harmful effects for boys could be in small part due to less participation in extracurricular activities and increased peer drug use.(18, 37) However, these potential mediating effects, examined one-by-one, were responsible for only a tiny fraction of risk—not enough to be meaningful.(18) In reality, the processes mediating voucher receipt on long-term mental health and substance use outcomes among boys in MTO likely exist as an interrelated and complex bundle. It is possible that by estimating the mediated (indirect) effect through many more mediators, considered jointly, one would be able to explain a larger portion of the total effect. The other prior work estimated different estimands despite violation of the identifying assumptions,(37), so is not directly comparable.

We estimate indirect effects of voucher receipt on long-term mental health and substance use outcomes through mediators related to aspects of the school environment, neighborhood, and instability of the social environment. In doing so we focus on outcomes that were negatively affected by the intervention among boys. We also estimate the corresponding direct effects from voucher receipt to each outcome, not operating through the collection of mediators. In MTO, using the voucher to move out of public housing (i.e., intervention take-up or adherence) is a post-treatment variable that could confound the effect of the mediator on the outcome. Natural direct and indirect effects are common mediation estimands, together adding to the total effect, but are not identified in the presence of such a variable.(38) Consequently, we estimate interventional (also known as stochastic) direct and indirect effects, which do not require the absence of post-treatment confounders of the mediator-outcome relationship for identification but are analogous to natural direct and indirect effects in the absence of such variables.(39, 40) In estimating these effects, we use a recently developed, nonparametric, robust, and efficient estimator.(41)

Methods

We used the data from the entire length of follow-up for the MTO experiment, described in detail elsewhere.(6) The baseline survey occurred at the point of randomization, 1994–1998, and the final follow-up visit and survey occurred 2008–2010.(16)

Sample

We used the final youth analytic dataset (N=4,945 youth aged 10–20 years at the final time point). We excluded the Baltimore site, as has been done previously,(18, 42) because of evidence that the intervention differed meaningfully in this site versus the others,(18) and restricted our analysis to boys, resulting in a total rounded sample size of N=2,500 (any reported sample size must be rounded according to a Census internal algorithm.). We incorporated MTO youth weights for all analyses that account for randomization ratios, sampling of the youth within families, and drop-out over follow-up.(16) This secondary analysis of deidentified data was deemed to be nonhuman subjects research.

Measures

Measures consisted of baseline covariates (W), randomized Section 8 voucher receipt (A), using the voucher to move (i.e., adherence to the intervention, Z), mediators (M), and outcome (Y). These were temporally ordered such that W and A preceded Z, which preceded M, which preceded Y.

We considered receipt of a Section 8 housing voucher (A = 1) versus not (A = 0) as the intervention. Families enrolled in MTO were randomized to one of three arms; the first two received a Section 8 voucher, with the first group being directed to use the voucher to move to a low-poverty neighborhood (< 10% persons under the poverty threshold) and receiving assistance in finding housing and moving, and the second group not being given any directive about where to move and not receiving assistance in finding housing. We combined these two groups as has been done previously.(18, 42, 43)

Use of the voucher (Z, adherence/intervention take-up) was operationalized as a binary indicator of whether or not the family used the Section 8 voucher to move (1) or not (0). This was measured in the 90 days that were allotted to use the voucher, following voucher receipt.

We considered a multivariate list of mediators that have been hypothesized to be on the causal pathway between an exposure of neighborhood poverty and mental health and substance use outcomes, as described in the Introduction. We considered mediators together by group (i.e., school environment, instability of the social environment, and neighborhood poverty) as well as all mediators combined. In terms of the school environment, we included: school rank, student to teacher ratio, percent of students receiving free or reduced lunch, and proportion of Title-I schools attended. In terms of instability of the social environment, we included: number of moves, number of schools attended, number of school changes within the school year, and an indicator of whether or not the most recent school at the end of follow-up was in same school district as baseline. In terms of neighborhood, we included neighborhood poverty. Mediators were weighted over the duration of the 10–15 year follow-up (i.e., calculated as a weighted mean where weights were proportionate to the length of follow-up time (e.g., length of time that the youth attended each school, lived in each neighborhood, etc.)).

We considered several outcomes, assessed at the final time point: presence of any Diagnostic Statistical Manual, Fourth edition (DSM-IV) disorder in the past year, any past-year DSM-IV mood disorder (depression, bipolar, anxiety), any past-year DSM-IV externalizing disorder (conduct disorder, oppositional defiant disorder, or intermittent explosive disorder), and whether or not smoked cigarettes in the past 30 days. All DSM-IV diagnoses were assessed using the Composite International Diagnostic Interview (CIDI).(44)

Baseline covariates included sociodemographic and learning characteristics of the child, sociodemographic characteristics of the enrolled parent, the parent’s reasons for wanting to move, prior experience with Section 8 or other public assistance programs, and baseline neighborhood poverty. A detailed description is available in Section 1 of the eAppendix.

A directed acyclic graph relating these measures to each other is given in Figure 1. This depicts an instrumental variable set-up where any effect of A on M or Y necessarily operates through Z.

Figure 1:

Directed acyclic graph relating measures in the Moving to Opportunity analysis.

Statistical Analysis.

We used data from the MTO youth final analytic sample, which includes all youth who were participants at the final time point. Percent missing for each variable is given in Table 1. We used multiple imputation by chained equations to make 10 imputed datasets, including all variables used in the analysis as well as others believed to be helpful in prediction (listed in Section 2 of the eAppendix).(45, 46) The imputation proceeded in order of the data-generating mechanism (baseline variables, interim variables, final variables). For the imputed datasets to be congenial with the analyses that follow, all variables and their higher-order terms used in the analyses must be included in the imputation. Unfortunately, this was not possible due to computational limitations, and we used only main effects. We conducted the analyses on each imputed dataset and combined results using Rubin’s rules.(47)

Table 1:

Characteristics of boys enrolled in Moving to Opportunity, 1994–2010, by randomized voucher assignment. Numbers are percentages unless otherwise specified. Survey weighted and combined across 10 imputed datasets. All results were approved for release by the U.S. Census Bureau, authorization number CBDRB-FY20-ERD002–031

Characteristic	Voucher N=1400	Control N=700	Missing
Baseline
Age at baseline, mean (sd)	4.7 (0.1)	4.8 (0.1)	0
Site			0
Boston, %	24	23
Chicago, %	30	28
LA, %	20	18
NYC, %	26	30
Race/ethnicity			2
White race, %	2	1
Black race, %	60	58
Hispanic/Latino, %	34	38
Other race, %	5	2
Behavioral problems, %	4	4	0
Gifted, %	5	7	0
Adult graduated high school, %	37	35	0
Adult never married, %	64	65	0
Adult under 18 at birth of child, %	28	26	0
Adult working, %	23	22	0
Adult felt unsafe in neighborhood at night, %	49	50	0
Adult very dissatisfied with neighborhood, %	46	46	0
Adult primary reason for moving is schools, %	55	48	0
Household receives AFDC, %	79	78	0
Household member has a disability, %	16	19	0
Household size=2, %	8	7	0
Household size=3, %	22	25	0
Household size>4, %	28	24	0
Household had section 8 before, %	35	44	0
Moved 3+ times previously, %	9	12	0
Baseline neighborhood poverty, %	54.1 (0.6)	54.2 (0.8)	2
Mediators (measured between baseline and final timepoints)
Neighborhood poverty
Follow-up neighborhood poverty, mean (sd)	32.3 (0.4)	40.5 (0.6)	0
School environment
% students receiving free/reduced lunch, mean (sd)	70.0 (5.3)	73.0 (4.2)	9
% Title I schools, mean (sd)	61.2 (6.8)	64.5 (5.3)	8
School ranks, mean (sd)	24.5 (6.8)	21.9 (5.3)	12
Ratio of students to teacher, mean (sd)	17.5 (1.0)	17.5 (0.7)	8
Instability of the social environment
Number of schools, mean (sd)	4.4 (0.5)	4.3 (0.4)	8
Number of moves, mean (sd)	2.9 (0.1)	2.2 (0.1)	0
Number of school changes during year, mean (sd)	0.8 (0.2)	0.7 (0.1)	8
School in baseline district, %	53	56	8
Outcomes (measured at final timepoint)
Any past-year DSM disorder, %	35	30	8
Any past-year DSM mood disorder, %	24	18	8
Any past-year DSM externalizing disorder, %	32	27	8
Smoked cigarettes in past 30 days, %	43	36	16

We first estimated total, adjusted effects of Section 8 voucher receipt on each mental health and substance use outcome for boys (intent-to-treat average treatment effects) and identified the outcomes exhibiting unintended harmful effects (Table 2). One examined outcome, an index of behavioral problems(48), did not exhibit unintended harmful total effects. We estimated these total effects using a targeted minimum loss-based estimator (TMLE),(49) incorporating all baseline variables listed in Table 1, the weights for the final youth sample, and combining results across the 10 imputed datasets. We also estimated: 1) the effect of voucher receipt on each mediator and 2) the effect of each binary mediator on each outcome using the same approach. We estimated the effect of each non-binary mediator on each outcome using a TMLE for the average effect of an intervention that shifts the exposure distribution(50) by the same amount that voucher receipt was estimated to have shifted the mean of each mediator, incorporating all baseline variables, voucher receipt, and moving with the voucher; results were combined across the 10 imputed datasets for all estimates.

Table 2:

Adjusted estimates of effect of voucher on mediators/outcomes among boys in the Moving to Opportunity, 1994–2010. Survey weighted and combined across 10 imputed datasets. All results were approved for release by the U.S. Census Bureau, authorization number CBDRB-FY21-ERD002–001.

Mediator/Outcome	Estimate	95% CI
Follow-up neighborhood poverty1	−0.08	(−0.09, −0.07)
School environment
% students receiving free/reduced price lunch1	−0.03	(−0.05, −0.01)
% Title I schools1	−0.03	(−0.06, 0.00)
School rank2	2.27	(−0.08, 4.63)
Ratio of students to teacher2	0.03	(−0.46, 0.52)
Instability of the social environment
Number of schools2	0.01	(−0.19, 0.53)
Number of moves2	0.62	(0.44, 0.80)
School in baseline district3	−0.01	(−0.05, 0.04)
Number of school changes during year2	0.06	(−0.07, 0.20)
Past-year DSM disorder3	0.04	(−0.03, 0.14)
Past-year DSM mood disorder3	0.06	(0.01, 0.15)
Past-year DSM externalizing disorder3	0.06	(−0.03, 0.14)
Smoked cigarettes in past 30 days3	0.04	(−0.00, 0.13)

¹difference in proportions,

²difference,

³risk difference

The total effect can be decomposed into the natural direct effect (not operating through the mediators) and the natural indirect effect (operating through the mediators).(51) However, as stated in the Introduction, natural direct and indirect effects are not identifiable in the presence of post-treatment confounding variables.(38) This limits their utility, as any intervention that acts through adherence (i.e., actually using the voucher to move) would violate the identifying assumption.

Interventional, also called stochastic,(52) direct and indirect effects(39, 53, 54) relax that assumption. These effects are defined by contrasts between population-average counterfactuals in which a population-level intervention is set deterministically and the mediator is stochastically drawn from counterfactual distributions defined by contrasting values of the population-level intervention and conditional on baseline variables. Interventional direct and indirect effects are analogous to their natural counterparts in the absence of post-treatment confounding,(53) but unlike natural direct and indirect effects, there is no individual-level decomposition of the total effect. Instead, the interventional direct and indirect effects can be written as components of a population-level decomposition of a slightly different total effect (Equation 1), where: Y_a,m represents the counterfactual outcome setting A = a and M = m, possibly contrary to fact, and G_a represents a random draw from the distribution of M_a conditional on W.

$$E\left(Y_{1,G_1}-Y_{0,G_0}\right)=\underset{\text{Indirect effect }\left(\text{through }M\right)}{\underbrace{E\left(Y_{1,G_1}-Y_{1,G_0}\right)}}+\underset{\text{Direct effect }\left(\text{not through }M\right)}{\underbrace{E\left(Y_{1,G_0}-Y_{0,G_0}\right)}}.$$ (1)

In words, the total interventional effect that can be decomposed into the interventional direct and indirect effects, E(Y_1,G1 − Y_0,G0), is the difference in the average expected outcomes among boys in MTO in hypothetical worlds in which voucher receipt is set to received (1) versus not (0) and M is drawn from the joint population distribution of school environment, neighborhood poverty, and social instability mediating variables had the voucher been received (A = 1) versus from the joint distribution of mediator values had the voucher not been received (A = 0). The interventional indirect effect is the population-average difference in expected outcomes setting voucher to be received (A = 1) and stochastically drawing M from the counterfactual joint distribution of mediator values in a hypothetical world in which the voucher had been received versus drawing from the counterfactual joint distribution of mediator values in another hypothetical world in which the voucher had not been received. Similarly, the interventional direct effect is the population-average difference in expected outcomes setting voucher received versus not and stochastically drawing from the counterfactual joint distribution of mediator values in a hypothetical world in which the voucher had not been received.

Interventional direct and indirect effects are identified from observed data assuming: that there is no unmeasured (i) exposure-outcome, (ii) exposure-mediator, or (iii) mediator-outcome confounding, (iv) that the probability of observing all (A, M) combinations is positive, conditional on covariates W. Because Section 8 voucher receipt is randomized, assumptions (i) and (ii) are likely met. We discuss bias due to violation of assumption (iii) in the Discussion. We examined assumption (iv) by assessing the distribution of the estimated influence curve (defined in the eAppendix),(41) which did not indicate positivity problems. We also assume no interference, which means that the treatment of one person does not influence the potential outcome of another person. This is an issue that has been debated in the MTO literature(55), with the primary investigators concluding that the effect of interference is likely minimal.(56)

To estimate these interventional direct and indirect effects, we used a TMLE.(41) This estimation approach offers several advantages over alternative methods: (1) it is multiply robust, meaning that estimates will remain unbiased even if some models are misspecified; (2) it is efficient; (3) it incorporates the post-treatment variable of moving with the voucher (intervention take-up/adherence); (4) it accommodates numerous, mediating variables simultaneously, allowing for them to be interrelated; and (5) it can incorporate machine learning in model fitting while retaining theoretically valid standard errors and confidence intervals. We did not adjust our standard errors to account for our and others’ history of hypothesis testing in the MTO data. We used an ensemble approach(57) that optimally weights predicted values from the following algorithms in model fitting: generalized linear models (including with interaction terms), Bayesian generalized linear models, generalized additive models, and regression splines (MARS). Additional details of this estimation approach are provided in Section 3 of the eAppendix.

In summary, our analysis was as follows. We created imputed datasets, where intermittent missing data (not due to loss-to-follow-up) were imputed by multiple imputation by chained equations among youth who were interviewed at the final timepoint. We applied the interventional (in)direct effect TML estimators to each imputed dataset, incorporating the MTO youth weights into the estimator. We then combined the direct and indirect effect estimates across the 10 imputed datasets using Rubin’s rules.(47)

We used R version 3.5.2 for all analyses. The estimator can be implemented using the medoutcon R package.(58) Code to replicate the analyses is available: https://github.com/kararudolph/code-for-papers/MTOhighdimmediation.

Results

Survey-weighted baseline characteristics of boys in the multiply imputed, final analytic dataset are given in Table 1, below. Overall, 35% of families included in this analytic sample moved with the voucher.

Table 2 shows the estimated adjusted effect of randomized voucher receipt on each mediator and outcome. Table 3 shows the estimated adjusted effect of each mediator on each outcome. Voucher receipt results in unanticipated harmful effects on the long-term outcomes shown here: any DSM-IV disorder, any mood disorder, any externalizing disorder, and current smoking. Voucher receipt also appears to affect many of the mediators we consider, including what would likely be considered improvements in terms of neighborhood poverty, measures of school poverty (i.e., share of students receiving free and reduced price lunch, Title I status), and school ranking; but what would likely be considered negative effects on elements of instability, including more residential moves and more schools attended (Table 2). Each mediator also appears to negatively affect most of the outcomes we consider, albeit slightly (Table 3); note that in the case of continuous mediators, we estimate effects of shifting mediator values.

Table 3:

Adjusted effect estimates of each mediator on each outcome among boys in the Moving to Opportunity, 1994–2010. Survey weighted and combined across 10 imputed datasets. All results were approved for release by the U.S. Census Bureau, authorization number CBDRB-FY21-ERD002–001.

Mediator	Outcome	Estimate	95% CI
School in baseline district	Past-year DSM disorder	−0.13	(−0.28, 0.02)
	Past-year DSM mood disorder	−0.14	(−0.27, −0.01)
	Past-year DSM externalizing disorder	−0.14	(−0.27, −0.02)
	Smoked cigarettes in past 30 days	−0.15	(−0.25, −0.05)
Neighborhood poverty (shift of −0.08)	Past-year DSM disorder	0.008	(−0.011, 0.027)
	Past-year DSM mood disorder	0.008	(−0.005, 0.021)
	Past-year DSM externalizing disorder	0.002	(−0.017, 0.020)
	Smoked cigarettes in past 30 days	0.013	(−0.007, 0.032)
Proportion students receiving free/reduced price lunch (shift of −0.10)	Past-year DSM disorder	0.003	(−0.022, 0.027)
	Past-year DSM mood disorder	0.003	(−0.025, 0.030)
	Past-year DSM externalizing disorder	0.005	(−0.023, 0.033)
	Smoked cigarettes in past 30 days	0.006	(−0.017, 0.029)
Proportion Title 1 schools (shift of −0.03)	Past-year DSM disorder	0.006	(−0.015, 0.016)
	Past-year DSM mood disorder	0.002	(−0.017, 0.021)
	Past-year DSM externalizing disorder	0.001	(−0.015, 0.016)
	Smoked cigarettes in past 30 days	−0.002	(−0.018, 0.013)
School ranks (shift of 2.61)	Past-year DSM disorder	0.002	(−0.020, 0.023)
	Past-year DSM mood disorder	0.002	(−0.026, 0.030)
	Past-year DSM externalizing disorder	0.002	(−0.022, 0.027)
	Smoked cigarettes in past 30 days	0.004	(−0.017, 0.025)
Ratio of students to teacher (shift of 0.07)	Past-year DSM disorder	0.001	(−0.006, 0.007)
	Past-year DSM mood disorder	0.001	(−0.005, 0.006)
	Past-year DSM externalizing disorder	0.000	(−0.007, 0.007)
	Smoked cigarettes in past 30 days	−0.001	(−0.006, 0.005)
Number of schools (shift of 0.19)	Past-year DSM disorder	0.007	(−0.006, 0.021)
	Past-year DSM mood disorder	0.005	(−0.014, 0.024)
	Past-year DSM externalizing disorder	0.006	(−0.012, 0.024)
	Smoked cigarettes in past 30 days	0.005	(−0.011, 0.021)
Number of moves (shift of 0.73)	Past-year DSM disorder	0.004	(−0.008, 0.017)
	Past-year DSM mood disorder	0.002	(−0.008, 0.012)
	Past-year DSM externalizing disorder	0.003	(−0.009, 0.015)
	Smoked cigarettes in past 30 days	0.000	(−0.013, 0.013)
Number of school changes during year (shift of 0.07)	Past-year DSM disorder	0.003	(−0.006, 0.011)
	Past-year DSM mood disorder	0.002	(−0.008, 0.011)
	Past-year DSM externalizing disorder	0.003	(−0.006, 0.011)
	Smoked cigarettes in past 30 days	0.005	(−0.002, 0.011)

Figures 2–5 plot the interventional total, direct, and indirect effect estimates and their 95% confidence intervals (CIs) for each outcome. Note that within mediator-outcome pair, the indirect and direct interventional effects add to the total interventional effect; however, we do not expect the mediation subgroup indirect effects to add to the indirect effect when all mediators are considered together, given that the mediator subgroups are not conditionally independent. We see a similar pattern across outcomes, namely that much of the harmful total effect operates through the combination of mediators we consider.

Figure 2:

Direct and indirect effect estimates of risk of any Diagnostic Statistical Manual (DSM) disorder and 95% confidence intervals by mediator group among boys in the Moving to Opportunity study, 1994–2010.

Figure 5:

Direct and indirect effect estimates of risk of past 30-day cigarette use and 95% confidence intervals by mediator group among boys in the Moving to Opportunity study, 1994–2010.

For example, nearly all of the effect of voucher receipt on increased risk of any DSM-IV disorder in adolescence operates through the mediators considered (Figure 2), explaining 90% of the total interventional effect (Equation 1). All three types of mediators—neighborhood poverty, school environment, and instability of the social environment—appear to play a role in mediating this overall harmful effect with indirect effects ranging from 0.014 (95% CI: −0.028, 0.056) for neighborhood poverty and 0.059 (95% CI: 0.010, 0.108) for all mediators considered together. For the outcome of any DSM-IV mood disorder (Figure 3), the mediators explain 72% of the total effect. In this case, mediators related to aspects of the school environment and instability appear to play a role, but neighborhood poverty does not. For the outcome of any externalizing DSM-IV disorder (Figure 4), again, all three types of mediators appear to play a role in mediating the overall harmful effect. Considering all mediators together explains 80% of the total effect in this case. Finally, considering all the mediators together explains 69% of the overall effect between voucher receipt and increased risk of smoking cigarettes (Figure 5). Again, all mediator types seem to play a role in this indirect effect pathway.

Figure 3:

Direct and indirect effect estimates of risk of any Diagnostic Statistical Manual (DSM) mood disorder and 95% confidence intervals by mediator group among boys in the Moving to Opportunity study, 1994–2010.

Figure 4:

Direct and indirect effect estimates of risk of any Diagnostic Statistical Manual (DSM) externalizing disorder and 95% confidence intervals by mediator group among boys in the Moving to Opportunity study, 1994–2010.

Discussion

Others have reported unintended harmful effects of the Moving to Opportunity study on the mental health, substance use, and risk behavior outcomes of boys whose families were randomized to the voucher groups.(8, 15–17, 19, 59) However, to our knowledge, there has been little quantitative work explaining these unintended harmful effects.(18, 37) We provided some quantitative evidence here, finding that the majority (69–90%) of the total negative long-term effects on mental health and substance outcomes could be explained by indirect effects through mediators related to the school environment, neighborhood poverty, and instability of the social environment, considered as a bundle.

Instability, as measured by the number of home and school moves, worsened due to voucher receipt (Table 2). Thus, it makes intuitive sense that this subset of pathways would contribute to unintended long-term harmful effects. Previous research has found that boys may be more sensitive than girls to moves, school changes, and other aspects of environmental instability.(6, 15, 26, 60)

The other mediator groups of neighborhood poverty and measures of the school environment “improved” due to voucher receipt (Table 2), as the core MTO investigators had hypothesized.(20) However, these “improvements” did not lead to benefits in terms of the long-term mental health and substance use outcomes we examined, as some posited they might(27)—in fact, we estimated that they had the opposite effect. One reason could be that the boys enrolled in MTO, many of whom are black or Hispanic/Latino, may have experienced more racism or discrimination in their new neighborhoods or new schools.(15) Similarly, boys may have been more affected by larger socioeconomic disparities between themselves and their neighbors.(32) Another possible contributor is peer dynamics. Qualitative research suggested that boys in the experimental group were less careful in selecting their peers than those in the control group and were more likely to form friendships with peers involved in risky behavior than other groups.(15) Boys may have therefore suffered multiple unintended consequences from seemingly beneficial improvements to their living and educational environments.

To provide exploratory evidence for or against some of the above hypotheses, we estimated the effect of voucher receipt on self-reported aspects of the school and social environments (results shown in eTable 1). We found that voucher receipt had consistently detrimental long-term effects on the social networks of boys; boys whose families were randomized to receive a voucher were less likely to report any close friends, and less likely to report visiting friends in either their current or baseline neighborhoods. In addition, they were more likely to report being suspended or expelled from school years in the prior 2 years. This lends support to the idea that even though objective indicators of the school and neighborhood environments improved following voucher receipt, that these “improvements” were not experienced as universally positive. (We would have liked to include more explicit experiences of racism and discrimination in the above post-hoc analyses, but these measures were not collected for these participants.)

Ideally, we would have examined the above hypotheses in our mediation framework, as it might be that the perceived environments are actually the most relevant mediators in explaining these unintended harmful effects. However, incorporating these perceptions of the school and social environments as mediators would suffer from two criticisms. First, these measures were collected at the same time as the outcomes, so temporality would be lost. Second, as these measures would be self-reported by the same individual and at the same time as the diagnostic interview for the DSM disorders, it is possible that having such a disorder would influence responses for the mediators (reverse causation) or that another factor (e.g., the individual’s mood that particular day or how they slept the night before) could influence both the mediator and outcome responses, resulting in same-source bias.

Our efforts to avoid sources of bias that we anticipated to be potentially strong led us to exclude such self-reported mediators from our indirect effect estimation. However, our results are still subject to bias from other sources. For example, although the exposure was randomized, unobserved confounding may nonetheless exist between the mediator and outcome, resulting in bias (identification assumption iii). In an effort to mitigate this, we included a large set of 18 covariates that we flexibly controlled for by using an ensemble of machine learning algorithms to fit each model (which can help control for unobserved confounding variables to the extent that they are correlated with observed covariates) coupled with a doubly robust estimation approach such that some combinations of models may be misspecified while maintaining consistency.(41) However, the exclusion of potentially important baseline variables, like school characteristics (for the children who were old enough to attend school at baseline) could result in bias. Another important source of bias of the mediator-outcome relationship can come from other, not-included mediators that are: 1) correlated with the mediators we include, conditional on all measured and unmeasured variables, or 2) precede the mediators we include. As we use mediators measured across the duration of follow-up, we are most concerned with the former’s impact on bias. For this reason, our results incorporating all mediators as a bundle may be least subject to unobserved confounding whereas our mediator-specific subgroup analyses may suffer more from this bias. Measurement error is another omnipresent source of potential bias. However, given that we use administrative measures of the school environment—which are reported to receive public funding—it may be limited in the mediator measures. Measurement error in DSM outcomes among adolescents as assessed by the CIDI has been studied previously.(44, 61)

Still another limitation is that the novel estimator we use does not identify which mediators from the bundle we considered are most important and which may be unimportant. Gaining such an understanding is a natural next step and adding such a “mediator importance” measure, akin to variable importance measures in algorithms like random forests,(62) is an area for future work. In addition, it may be of interest to estimate “per-protocol” or complier effects, which would be the effect of using the voucher to move (Z), using randomized receipt of the voucher as an instrument. To our knowledge there is very little work in decomposing such effects,(63, 64) though we are we plan to continue to contribute to this area in the future.

Despite these limitations, we believe the novel estimation approach we used to be a strength. Prior to this estimator, two estimation approaches existed to estimate interventional (in)direct effects considering multiple mediators.(54) The approach of Vansteelandt and Daniel (2017)(54) also decomposes the total interventional effect; the difference being that the interventional direct effect is separated into the effect of path A → Y and path A → Z → Y. Because A is an instrumental variable in our case, path A → Y is equal to zero, so the effects we estimate align with those of Vansteelandt and Daniel (2017). However, their estimation approach relies on correct specification of the outcome and mediator parametric models.(54) The relationships underlying the research questions we examined herein are extremely complex, operating at the individual, family, school, and societal levels, and likely changing over time. Consequently, any parametric models we attempted would assuredly be misspecified, and such misspecification has the potential to result in substantial bias.(65) Thus, the nonparametric, robust estimator we use may be well suited for complex research questions like this one.

MTO’s unintended harmful effects on boys’ mental health and substance use outcomes have been reported for more than a decade.(15, 16, 19) The reasons underlying this puzzling outcome have remained an open question. Careful estimation of mediating pathways contributing to these harmful overall effects is one strategy to gain some understanding. Using a novel, robust, and efficient nonparametric estimator of interventional direct and indirect effects,(41) we concluded that most of the overall harmful effects operated through objectively measured aspects of the neighborhood and school environments as well as the instability of these environments. Evidence from this paper suggests that, although voucher receipt generally led to what many would think of as better neighborhoods and better schools, that boys who experienced these changes actually had worse outcomes. Thus, it is possible that the instability resulting from changes in neighborhood and schools is detrimental to boys’ mental health, but also that boys may be more likely to be treated poorly in these supposedly better environments, (e.g., more likely to be suspended or expelled from school and socially marginalized (Table 2)). A broader conclusion may be one of humility: that hypothesized improvements may not be perceived as improvements by everyone.