Heterogeneous Effects of Housing Vouchers on the Mental Health of US Adolescents

Abstract

Objectives. To assess the mental health effects on adolescents of low-income families residing in high-poverty public housing who received housing vouchers to assist relocation.

Methods. We defined treatment effects to compare 2829 adolescents aged 12 to 19 years in families offered housing vouchers versus those living in public housing in the Moving to Opportunity experiment (1994–1997; Boston, MA; Baltimore, MD; Chicago, IL; Los Angeles, CA; New York, NY). We employed model-based recursive partitioning to identify subgroups with heterogeneous treatment effects on psychological distress and behavior problems measured in 2002. We tested 35 potential baseline treatment modifiers.

Results. For psychological distress, Chicago participants experienced null treatment effects. Outside Chicago, boys experienced detrimental effects, whereas girls experienced beneficial effects. Behavior problems effects were null for adolescents who were aged 10 years or younger at baseline. For adolescents who were older than 10 years at baseline, violent crime victimization, unmarried parents, and unsafe neighborhoods increased adverse treatment effects. Adolescents who were older than 10 years at baseline without learning problems or violent crime victimization, and whose parents moved for better schools, experienced beneficial effects.

Conclusions. Health effects of housing vouchers varied across subgroups. Supplemental services may be necessary for vulnerable subgroups for whom housing vouchers alone may not be beneficial.

Moving to Opportunity (MTO) was a landmark housing demonstration sponsored by the US Department of Housing and Urban Development that randomly assigned more than 4000 low-income families to receive housing subsidies to move out of distressed public housing into better housing units and safer neighborhoods. Although the MTO program was not designed with health in mind, it substantially improved the mental health of household heads (mothers) and their adolescent daughters.^1,2 However, MTO had adverse effects on boys’ mental health.^3–5

Assessing such treatment heterogeneity is important for determining subgroups for whom the treatment had unintended negative effects, was ineffective, or was particularly beneficial. This information may in turn guide program eligibility and help identify support services to improve program effectiveness, akin to a policy version of precision medicine’s movement to tailor treatment and services to individual variability.⁶ Identifying treatment heterogeneity could also guide changes to future housing subsidy programs and methods of implementation.

Treatment heterogeneity of a randomized exposure has traditionally been assessed by testing treatment interaction terms in regression models, typically 1 by 1. The development of methodological approaches that assess multidimensional data patterns has now made it possible to investigate higher-order patterns of treatment modification in which a series of participant characteristics (e.g., age, gender, race) are considered. Machine-learning approaches, such as model-based recursive partitioning, are particularly well suited for detecting complex interactions that may be difficult to isolate with a priori hypotheses specification and traditional regression techniques.⁷

With traditional regression methods, investigating higher-order interactions with 35 potential effect modifiers (as we do in this study) would entail specifying a 36-way interaction with treatment as well as different combinations of 2-, 3-, 4-, up to 35-way interactions—totaling more than 68 billion unique interaction terms.⁸ Alternatively, by incrementally identifying groups with similar treatment effects, recursive partitioning arguably optimizes the identification of treatment heterogeneity in the data, with a much more parsimonious approach.⁹ Model-based recursive partitioning is suitable for large data sets with many variables. Unlike other methods in which the original variables are condensed into a reduced set, thereby no longer permitting examination of individual variables (e.g., principal components analysis or factor analysis), model-based recursive partitioning can process patterns from many variables and still allow the examination of individual variables.¹⁰

In selecting potential treatment modifiers of the MTO housing experiment on adolescent mental health, we used several theoretical models. Residential mobility models posit that the effect of residential mobility depends on a series of characteristics related to one’s history of and preferences for residential moves, including history of migration, reasons for the move, and features of the new neighborhood.¹¹ We supplemented this theory by hypothesizing that families struggling with chronic stressors, such as health problems, may find the added burden of moving more difficult. Furthermore, social capital theory suggests that social connectedness in the baseline neighborhood may inhibit residential mobility and modify the effects of mobility because of the potential disruption of moving on familial and social network ties.¹¹

We applied these theories to test for treatment heterogeneity in MTO. We implemented model-based recursive partitioning, which is particularly suited for identifying higher-order interactions in large data sets, although it is rarely applied in public health. We adapted the method to preserve MTO’s experimental design, and therefore the strong internal validity, for inferring how the housing policy affected mental health. We believe this is the first such application of the recursive partitioning method to an experimental design. We replicated our results in subsets of the data to demonstrate robustness.

METHODS

MTO was a $70 million housing mobility experiment administered by the US Department of Housing and Urban Development from 1994 to 1997 in Boston, Massachusetts; Baltimore, Maryland; Chicago, Illinois; Los Angeles, California; and New York, New York. The population eligible for enrollment included low-income families with children who lived in public housing or project-based housing in high-poverty neighborhoods (1990 census tract poverty rate ≥ 40%). More than 5300 families volunteered, 4610 of whom were eligible and randomized. Those in the treatment groups were offered Section 8 tenant-based rental assistance (a.k.a., housing vouchers), which could be used to subsidize rent in a private market apartment.

The voucher holder is typically responsible for paying 30% of his or her adjusted household income as rent, and the federal government meets the difference, up to an area-based ceiling set by fair market rents.¹² The treatment group had 90 days from receiving the voucher to identify an apartment and ensure that it met minimal housing quality standards.¹³ One arm of the treatment group could use the voucher in any neighborhood. Another treatment group had to use the voucher in a low-poverty neighborhood (defined as 1990 census tract poverty < 10%) and received housing counseling to help them find an apartment. The control group was given no further assistance but could remain in public housing.¹⁴

Moving to Opportunity Participant Assessments

Adult heads of household and up to 2 children completed a baseline survey (1994–1998) and an interim survey 4 to 7 years after randomization (2001–2002) in the MTO tier 1 Restricted Access Data set. We conducted in-person interviews with computer-assisted interviewing technology, with effective response rates of 90% for adults and 89% for adolescents.^13,14

Mental health outcomes.

We modeled 2 mental health measures to capture both internalizing (distress) and externalizing (behavioral problems) behaviors among adolescents. We assessed past month psychological distress via Kessler’s K6 scale of the following 6 Likert items: so depressed nothing could cheer you up, nervous, restless or fidgety, hopeless, everything was an effort, worthless. We scored the K6 with item response theory latent variable methods¹⁵ to obtain an approximately standardized distress score with a mean of 0 and an SD of 1 (Cronbach α = 0.80). We assessed past 6-month behavior problems via the Behavior Problems Index, an 11-item scale that included items such as “I lie or cheat” and “I have a hot temper.”¹⁶ Similarly, we used item response theory methods to produce an approximately standardized score (Cronbach α = 0.80).

Treatment.

Although MTO was designed for 2 treatment groups (compared with a control group), treatment effects were similar for these 2 treatment groups, and statistical tests of treatment effect heterogeneity were nonsignificant for both psychological distress (P = .94) and Behavior Problems Index (P = .40). Thus, we combined and modeled the voucher treatment groups as randomized versus the control group to simplify analyses and presentation of results.

Effect modifiers.

We evaluated modification of the MTO intervention effect on adolescent mental health using available prerandomization (baseline) variables, which included youths’ demographics (continuous age at baseline of 5–16 years, gender, race/ethnicity), youths’ developmental health (giftedness, behavioral or emotional problems, learning problems, expulsion or suspension from school, school called household head to discuss problems with schoolwork or behavior, developmental problems that interfered with school or playing active games or sports, problems requiring special medicine or equipment), household composition (household size, adolescents in the home); site (New York, Boston, Baltimore, Chicago, Los Angeles), socioeconomic status (household head’s marital status, adolescent parent at birth of first child, welfare receipt, employment status, educational attainment, current enrollment in school, car ownership), family health (a household member had a disability, a household member was victimized by violent crime), residential history and preferences (tenure in neighborhood, household head moved more than 3 times in 5 years, was very dissatisfied with neighborhood, believed the streets near home were very unsafe at night, was very sure he or she could find an apartment in a different area of the city, primary or secondary reason for move was to get away from gangs or drugs, primary or secondary reason for move was to have access to better schools for children, applied for Section 8 before), and social networks (household head chatted with baseline neighbors, would tell neighbor if child was getting in trouble, had family or friends in baseline neighborhood). In total, we tested 35 potential treatment modifiers.

Statistical Analysis

We applied model-based recursive partitioning to examine modifiers of MTO treatment on mental health. We implemented models using R version 2.12.1¹⁷ and the function mob() that is available with the party package.¹⁸ This machine-learning approach creates classification trees that identify groups that best differentiate treatment effects (i.e., treatment minus control differences), on the basis of comparing splits in the data, 1 variable at a time. At each node, the algorithm selects the variable and cutpoint that results in the greatest impurity reduction (i.e., is most able to differentiate between groups of people on the basis of their experienced treatment effects).¹⁰ The selection of the splitting variable and cutpoint within this recursive partitioning method incrementally isolates groups of participants who experienced similar treatment effects on a particular outcome. As designed, this algorithm is uniquely suited to identify higher-order interactions if they exist in the data.

We implemented this approach separately for the 2 adolescent mental health outcomes—psychological distress and behavior problems. We specified MTO treatment as the predictor variable in the model. This implements an intention-to-treat analysis to preserve the strong experimental design and make causal inferences about how this housing subsidy affected mental health in low-income adolescents after 4 to 7 years. We required that splits differentiate treatment effects at an α level of 0.05 and that the smallest subgroup contain at least 20 participants, the minimum node sample size we specified to produce more reliable subgroup estimates.

The resulting classification trees display each significant break in the data (Figures 1 and 2). We have presented the subgroups (defined by cross-classification of effect modifiers) along with observed treatment effects (parameter estimates and 95% confidence intervals). Estimated treatment effects accounted for household clustering, because we sampled up to 2 children per household and weighted them to account for attrition and varying random assignment ratios across time. Negative values indicate a beneficial effect of treatment (i.e., lower distress and fewer behavior problems in the treatment vs control groups), whereas positive values indicate an adverse effect of treatment on mental health.

FIGURE 1—

Recursive Partitioning Results for Psychological Distress Summarizing Higher-Order Treatment Heterogeneity on Adolescent Mental Health, Entire Sample: Moving to Opportunity; Boston, MA; Baltimore, MD; Chicago, IL; Los Angeles, CA; New York, NY; 1994–1997, 2002

Note. CI = confidence interval. Intent-to-treat treatment effects are indicated by b (95% CI) for each node and split. All potential effect modifiers were baseline characteristics. Bold indicates significant treatment effects.

FIGURE 2—

Recursive Partitioning Results for Behavior Problems Index Summarizing Higher-Order Treatment Heterogeneity on Adolescent Mental Health, Entire Sample: Moving to Opportunity; Boston, MA; Baltimore, MD; Chicago, IL; Los Angeles, CA; New York, NY; 1994–1997, 2002

Note. CI = confidence interval; HH = household. Intent-to-treat treatment effects are indicated by the b (95% CI) for each node and split. All potential effect modifiers were baseline characteristics. Bold indicates significant treatment effects.

As a robustness check, for each outcome, we implemented a replication procedure to assess the consistency of identified nodes across 10 randomly generated subsets of the data, each comprising 67% of the original sample. The introduction of randomness produces an advantage in prediction accuracy over any individual classification tree by reducing dependence on prediction error stemming from unstable predictors.¹⁹

RESULTS

Table 1 presents descriptive statistics for the MTO sample. For 2829 adolescents aged 12 to 19 years at MTO’s interim survey, approximately half were girls with a mean age of 15 years in 2002. Most were African American (62.8%) or Hispanic (30.0%) and represented a disadvantaged population at baseline. For example, 74.0% of household heads were unemployed, 64.0% had less than a high school education, and 56.0% were never married.

TABLE 1—

Youth and Family Descriptive Characteristics at Baseline (1994–1997) and at the Interim (2002) Survey: Moving to Opportunity; Boston, MA; Baltimore, MD; Chicago, IL; Los Angeles, CA; New York, NY

Variable	Overall
Baseline survey, 1994–1997
Household member victimized by crime during past 6 mo, %	43
Household member had a disability, %	17
Site, %
Baltimore	16
Boston	19
Chicago	22
Los Angeles	19
New York	25
Household size, %
2	7
3	22
4	25
≥ 5	45
Mean baseline age, y	10
Gender, %
Male	50
Female	50
Race/ethnicity, %
African American	63
Hispanic ethnicity, any race	30
White	1
Other	2
Special class for gifted students or did advanced work, %	15
Developmental problems, %
Special school, class, or help for learning problem in past 2 y	17
Special school, class, or help for behavioral or emotional problems in past 2 y	8
Problems that made it difficult to get to school or to play active games	7
Problems that required special medicine or equipment	9
School asked to talk about problems child having with schoolwork or behavior in past 2 y	26
Expelled from school in past 2 y	10
Family structure, %
Never married	56
Adolescent parent	26
Socioeconomic status, %
Employed	26
On Aid to Families With Dependent Children (welfare)	76
Education, %
< high school	47
General equivalency diploma	17
High school diploma	36
In school	14
Neighborhood and mobility, %
Lived in neighborhood ≥ 5 y	66
No family members living in neighborhood	64
No friends living in neighborhood	37
Had applied for Section 8 voucher before	44
At baseline, respondent was very dissatisfied with neighborhood	45
At baseline, streets near home were very unsafe at night	49
Baseline respondent reported being very sure would find an apartment in a different area of the city	45
Baseline respondent’s primary or secondary reason for moving was to have access to better schools for children	53
Neighbor relationships, %
Chats with neighbors at least once a week	52
Respondent very likely to tell neighbor if saw neighbor’s child getting into trouble	57
Interim survey variables, 2002
Mean interim survey age, y, %	15

Note. The population was N = 2829 adolescents. Data are from tier 1 of the restricted access data set. Our analyses accounted for clustering at the household level, and we weighted them for varying random assignment ratios across time and attrition.

Recursive Partitioning Effect Modification of Entire Sample

Psychological distress.

MTO treatment effects on distress were significantly modified by the following 4 characteristics: Chicago site (vs the 4 other sites), gender, school-identified behavior or schoolwork problems, and adolescent parent status (Figure 1). Recursive partitioning identified 5 different subgroups, defined by the 4 variables, as displaying the most treatment heterogeneity.

MTO treatment was associated with lower psychological distress (beneficial effect) in the subgroup defined by female adolescents living outside the Chicago site who had problems at school and were not children of an adolescent parent (node 6; treatment–control difference = −0.37). Conversely, the treatment resulted in higher psychological distress (adverse effect) for a subgroup of children defined by the same characteristics, except that girls were daughters of an adolescent parent (node 7; treatment–control difference = 0.64). However, the results for adolescent parent should be interpreted with caution because adolescent parent did not appear as a treatment–modifier in replicated samples.

Behavior problems.

As displayed in Figure 2, MTO treatment effects were modified by 8 baseline characteristics. We entered baseline age as an integer, and our statistical model selected a cutpoint of 10 years as relevant for distinguishing treatment effects. Treatment effects appeared most beneficial for the subgroup defined by the following characteristics: children who were older than 10 years at baseline, without learning problems, without household history of victimization, whose family moved for better schools, and whose household head had never married (node 11; treatment–control difference = −0.44).

Conversely, treatment effects were detrimental for a subgroup of the sample: older than 10 years at baseline, without learning problems, with household history of victimization, a household head who was never married, and unsafe baseline neighborhoods (node 18; treatment–control difference = 0.74). We identified detrimental effects of lower magnitude in a group of adolescents with the same characteristics, except that they did not report unsafe baseline neighborhoods (node 17; treatment–control difference = 0.41).

Recursive Partitioning Replication Analysis in 10 Random Subsets

Table 2 displays the results of our replication analysis. For psychological distress, gender appeared in all 10 replications and Chicago site appeared in 9 replications, indicating that these 2 characteristics were consistently related to heterogeneous effects. Problems at school appeared 5 times as a moderately confident modifier. There were 7 other variables that appeared in the classification trees only 1 or 2 times, suggesting less robust patterns of effect modification.

TABLE 2—

Characteristics Identified by Model-Based Recursive Partitioning as Treatment Effect Modifiers on Adolescent Mental Health: Moving to Opportunity; Boston, MA; Baltimore, MD; Chicago, IL; Los Angeles, CA; New York, NY; 1994–1997, 2002

	Psychological Distress Trees										Behavior Problems Trees
Effect Modifier Tested	1	2	3	4	5	6	7	8	9	10	1	2	3	4	5	6	7	8	9	10
Gender is male	∙	∙	∙	∙	∙	∙	∙	∙	∙	∙
Site is Chicago	∙	∙	∙	∙	∙	∙	∙	∙	∙
Age, y											∙	∙	∙		∙	∙	∙	∙	∙	∙
School called about child problems in school			∙		∙	∙	∙			∙				∙
Learning problem		∙															∙	∙	∙	∙
Perceived neighborhood as very unsafe at night		∙	∙
No family members living in neighborhood			∙					∙
Site is Los Angeles										∙						∙
Very dissatisfied with neighborhood																	∙	∙
History of crime victimization												∙
Household head is employed				∙
Child has physical developmental problems						∙
Socially connected to neighbors										∙
Household head was in school												∙
Housing search confidence																		∙
Better schools was primary or secondary reason for moving
Household head never married
Adolescent parent

Note. Each dot indicates that we identified the characteristic as a treatment effect modifier by model-based recursive partitioning in that data subset replication. We conducted 10 replications for each outcome, numbered 1–10. We did not identify variables that are not listed as a classification variable in any of the trees.

The replication analysis for behavioral problems indicated that age was consistently the most important effect modifier, because it appeared in 9 of 10 replication models. An adolescent’s baseline learning problems appeared in 4 of 10 models, indicating moderate confidence as an effect modifier.

DISCUSSION

MTO is one of the few large, randomized social experiments available to investigate causal health effects of neighborhood relocation. We used model-based recursive partitioning to create classification trees to test effect modification of MTO treatment on adolescent mental health on the basis of theoretically postulated characteristics. Investigating whether particular subgroups benefited more or whether some were harmed by treatment may be even more critical among populations facing multiple levels of disadvantage because of their vulnerability.²⁰ We found that gender and site (i.e., being in Chicago vs elsewhere) most differentiated MTO treatment on psychological distress and that age was the most robust effect modifier for behavioral problems. To a lesser degree, developmental health issues at baseline, such as learning problems or problems at school, also emerged as modifiers of MTO on adolescent mental health.

Effect modification is rarely specified a priori within any particular program, and therefore studies are often underpowered to detect subgroup effects.²¹ Using the alternative approach of model-based recursive partitioning allows a transparent approach to explore potential effect modifiers, including the role that simultaneous dimensions of adversity and resources may play in influencing outcomes. Comprehensively investigating higher-order interaction terms is not feasible in a traditional regression framework because of the exponentially increasing number of interactions to be tested and the need to account for these multiple comparisons. In this analysis, we investigated 35 potential effect modifiers. Using traditional models, it is impractical to investigate all possible combinations of this many potential effect modifiers.

Study Findings in Context

Studies that have tested for treatment heterogeneity of the MTO program used treatment–modifier interactions within regression or regression after stratification.^{1–4,13,22,23} Previous studies documented strong effect modification of MTO on adolescent mental health by gender,^1,3,4 which was reinforced in this analysis for distress when we found more beneficial treatment effects among girls. It is surprising that gender did not emerge as a modifier of behavior problems considering previously documented gender treatment effect modification. However, other characteristics more prevalent among MTO boys, such as learning problems, may be acting as a proxy for gender.

Gender is emerging as an important, yet understudied, modifier of neighborhood effects on health, suggesting gendered processes.^24–26 Disadvantaged neighborhoods may carry unique risks to females, including sexual assault and domestic violence.^24–26 Qualitative interviews with MTO adolescent girls suggested that girls in the MTO treatment group experienced less harassment from males, less pressure to engage in sexual behavior, and less fear overall than did controls^24,25—which may explain a critical component of MTO’s beneficial treatment effects on girls’ psychological distress.^2,13 Conversely, MTO boys may have had less adult supervision and participated in fewer structured after-school activities than did girls, possibly increasing their risk exposure in the treatment group.²⁷

For psychological distress, the Chicago site experienced null treatment effects, which could stem from many factors, including site differences in implementation of housing assistance or program delays in Chicago.²⁸ We cannot quantitatively assess whether program delivery explains the null findings, but we estimated sensitivity models to determine whether Chicago was a proxy for program implementation delays at the site, operationalized by randomization date. But randomization date did not emerge as a modifier in the overall tree or any subsample analyses, so later implementation does not explain the Chicago site effect. Site effects may be standing in for housing market factors, including available housing stock in low-poverty areas, fair market rents, exclusionary phenomena such as housing discrimination, and different segregation levels that pattern minority movement throughout the housing market.^12,29

Previous studies have identified baseline health or developmental conditions and history of violent crime victimization as strong effect modifiers of MTO treatment on adolescent mental health.^3,4 Our analyses reinforce these findings by identifying adolescent developmental problems as modifiers of MTO on mental health. Such patterns are consistent with the literature suggesting that frailty may be associated with lower intervention effectiveness for improving health.^30,31 Household vulnerabilities in the form of baseline health issues or social and economic resources may modify the impact of the relocation by, for instance, increasing the difficulty of postmove adjustment among youths.^32,33

Our results align with evidence that documents age as an MTO effect modifier on children’s physical health²³ and labor market outcomes in young adulthood.³⁴ We found that adverse treatment effects on behavior problems were concentrated among youths older than 10 years at baseline. Mobility occurring during key developmental transitions may heighten behavior problems. For example, a large longitudinal study of at-risk children found that housing mobility occurring during key periods of infancy or adolescence was related to increases in behavior problems, whereas no effects were observed among other school-aged children.³⁵

Study Implications

Future voucher programs may investigate whether housing counseling that provides greater guidance for choosing neighborhoods and schools may result in greater improvements in child outcomes.^36,37 Additionally, voucher programs may wish to consider minority male adolescents as a vulnerable subgroup in need of supports for mental health, academic counseling, extracurricular activities, and adult mentoring (particularly from adult male role models).^27,38

Gains in adolescent developmental outcomes may require more directly addressing the traumas and negative events, such as violent victimization, commonly experienced by low-income minority families.^39–41 Tracking cumulative disadvantage among housing voucher households may encourage greater recognition of the need for multidimensional support services in which housing programs coordinate with, for instance, health clinics, youth programs, child care providers, transit authorities, and career services.

Study Limitations

The study is subject to several limitations. Although preserving the experimental design in our analysis supports causal inference, we did not account for the nonadherence among those in the treatment group. About 51% of treatment group families used the MTO voucher to relocate. Lower adherence attenuates the intention-to-treat effect, because it combines voucher users with nonusers. Although analytic techniques have been developed to make valid inferences of treatment effects among voucher users,⁴² such techniques have not, to our knowledge, heretofore been applied to machine-learning approaches, and they require further development. Therefore, our findings should be interpreted as an underestimate of the true effect of the MTO policy.

Another limitation of the relatively novel nature of the application of machine-learning techniques to health-orientated research is that current software (e.g., R software) is not able to accommodate complex survey designs, including survey weights and clustering, in implementing model-based recursive partitioning. However, in estimating treatment effect estimates for the subgroups identified by model-based recursive partitioning, we used Stata version 11 (StataCorp LP, College Station, TX) and accounted for survey weights and clustering at the household level.

Additionally, although we selected a relatively comprehensive list of potential effect modifiers, there are arguably other important characteristics that we omitted, such as parental warmth, parental monitoring, or the involvement of a father figure, which may be important in predicting variation in treatment effects on adolescent mental health.^43–45

Conclusions

Model-based recursive partitioning is underused in health programs, studies, and randomized controlled trials. However, it can help detect higher-order treatment heterogeneity that is typically not investigated because of limited power with regression-based techniques and potential complexity with regard to a priori model specification. In addition to being used to evaluate completed programs, innovative methods like recursive partitioning may assist in the initial stages of program development to identify subgroups of the intervention population who are not benefiting from the treatment, so that treatment may be tailored to those subgroups.

We identified gender, site, age, and adolescent developmental issues (e.g., learning problems, problems at school) in various combinations as important characteristics in differentiating variation in MTO treatment effects. Because MTO was modeled on the largest federal affordable housing program in the United States,⁴⁶ housing voucher programs should consider supplementing their housing services to benefit vulnerable subgroups for whom housing vouchers alone may not be beneficial.