Effects of Housing Vouchers on the Long-Term Exposure to Neighborhood Opportunity among Low-Income Families: The Moving to Opportunity Experiment

Abstract

Tenant-based rental assistance has received much attention as a tool to ameliorate American poverty and income segregation. We examined whether a tenant-based voucher program improves long-term exposure to neighborhood opportunity overall and across multiple domains—social/economic, educational, and health/environmental—among low-income families with children. We used data from the Moving to Opportunity (MTO) experiment (1994–2010) with a 10- to 15-year follow-up period and used an innovative and multidimensional measure of neighborhood opportunities for children. Compared with controls in public housing, MTO voucher recipients experienced improvement in neighborhood opportunity overall and across domains during the entire study period, with a larger treatment effect for families in the MTO voucher group who received supplementary housing counseling, than the Section 8 voucher group. Our results also suggests that effects of housing vouchers on neighborhood opportunity may not be uniform across subgroups. Results from model-based recursive partitioning for neighborhood opportunity identified several potential effect modifiers for housing vouchers, including study sites, health and developmental problems of household members, and having vehicle access.

INTRODUCTION

Neighborhoods are posited to be upstream causes of a variety of outcomes. For example, high-quality neighborhoods offer better schools and safety from crime (Acevedo-Garcia et al. 2008, Galster and Killen 1995, Osypuk and Acevedo-Garcia 2010). Higher quality neighborhoods are also associated with improved physical (Diez Roux et al. 2003, Diez-Roux et al. 1997) and mental health (Kim 2008, Mair, Roux and Galea 2008, Truong and Ma 2006), and decreased substance use (Leventhal and Brooks-Gunn 2000, Leventhal, Dupéré and Brooks-Gunn 2009). However, large disparities in access to higher quality neighborhoods by race and income are prevalent throughout the United States (U.S.). Processes such as racialized housing markets limit neighborhood choice and result in racial segregation (Osypuk and Acevedo-Garcia 2010), perpetuating disparities in well-being as well as opportunity.

The structural factors contributing to residential sorting across race and income are well documented. Massey and Denton (1993) suggested that racial discrimination shapes the residential patterns of minorities by constraining their options. They argued that levels of black-white segregation do not vary significantly by social or economic class, and that racial discrimination by realtors, property owners, and mortgage lenders explains near-apartheid black segregation conditions in the United States. Based on an analysis of the national trend of black segregation in metropolitan regions from 1980 through 2010, Logan (2013) revealed that average black exposure to non-Hispanic whites did not change over three decades. Similarly, studies have suggested that income segregation between neighborhoods increased over the past several decades as rising income inequality widened the gap between high- and low-income families in terms of housing and neighborhoods they can afford (Owens 2016, Reardon et al. 2018).

Some low-income housing policies are designed to mitigate the translation of racial and income inequality into spatial inequality. For example, the Fair Housing Act of 1968 prohibits discrimination based on race, religion, national origin, sex, handicap and family status when people engage in housing-related activities, including the sale, rental, and financing of housing (Massey 2015). In addition, some low-income housing programs, particularly tenant-based housing vouchers, aim to promote upward residential mobility to low-poverty neighborhoods by expanding the range of housing and neighborhoods lower-income people can afford. However, it is unclear whether housing voucher programs are an effective tool to improve neighborhood opportunity for low-income children in the long term, and the persistence of racial and income segregation suggests a need for more effective policy measures.

Housing voucher recipients typically contribute 30 percent of their income toward rent, if they have income, with local housing agencies paying the rest. This permits voucher-holders to lease homes in higher opportunity neighborhoods than they could access without a voucher. The U.S. Department of Housing and Urban Development (HUD) annually sets Fair Market Rents (FMR)—the maximum allowable rent—separately for each metropolitan area and non-metropolitan county. For example, FMR in Cook County, Illinois, is $1,299 for a two-bedroom unit in 2021. A household with no income would receive a subsidy of $1,299; for a household of four with an annual income at the upper bound of the extremely low-income category, which is $27,950, their monthly subsidy would be about half that ($1,299 − [$27,950/12] × 0.3). HUD allows local housing agencies to adjust FMR up to 10 percent, and with HUD approval up to 150 percent to address the scarcity of rental units affordable to low-income households in high opportunity neighborhoods (HUD 2020).

Regardless of intent, housing vouchers may not result in improvement in neighborhood opportunity, or may have a favorable impact only on subgroups of voucher recipients. Structural barriers and personal preferences might constrain upward residential mobility into higher opportunity neighborhoods. Housing voucher recipients are typically resource- and time-constrained for housing searches, and landlords may be reluctant to participate in the housing voucher program, particularly in tight rental markets where unsubsidized renters are plentiful (Pendall 2000). Housing voucher recipients may prefer to stay in low opportunity neighborhoods because such locations offer instrumental supports from families, communities, or institutions (Briggs, Popkin and Goering 2010). Additionally, some voucher recipients use a housing voucher to access a higher quality housing unit in a low opportunity neighborhood in lieu of a higher quality neighborhood (the unit vs. location tradeoff) (Finkel and Buron 2001, Rosenblatt and DeLuca 2012, Wood 2014).

This paper uses data from the Moving to Opportunity (MTO) experiment to examine whether housing voucher recipients with children experience improvement in long-term neighborhood opportunity overall and across domains (e.g., social/economic, educational, and health/environmental). Previous studies on housing voucher programs have started asking the question of “neighborhood opportunity for whom?” (Lung–Amam et al. 2018, Reina and Aiken 2022), suggesting the necessity of defining neighborhood opportunity specifically to a subgroup of housing voucher recipients. Although some prior work on housing voucher programs has used a neighborhood opportunity measure specifically designed for children (Schmidt, Glymour and Osypuk 2021), their shorter timeframe makes it unclear whether the initial improvement in neighborhood opportunity associated with moving out of distressed public housing projects is sustained over a longer timeframe. This paper improves on previous studies by using a multidimensional measure of neighborhood-based opportunity specifically designed for families with children and a longer follow-up period of 10 to 15 years. Lastly, using an underutilized machine learning tool for identifying differential responses to policy interventions in the field of housing policy evaluation, we conduct an explorative study on the potentially heterogeneous effects of housing vouchers on long-term neighborhood opportunity across subgroups.

LITERATURE REVIEW

Residential Mobility Programs and Neighborhood Opportunity

Experimental and quasi-experimental studies of residential mobility programs in the United States, including the Gautreaux program, MTO program, and Creating Moves to Opportunity (CMTO), allow researchers to examine the effects of a housing voucher program on neighborhood opportunity while addressing the methodological limitations of observational studies (see Bergman, Chetty and DeLuca 2020 for CMTO overview, see Rosenbaum 1995 for Gautreaux program overview). By comparing different types of housing assistance programs, these studies tackle a policy question of how housing assistance programs can better serve low-income families. In the Gautreaux program, residents of Chicago public housing received housing vouchers as part of a court-ordered remedy for past racial discrimination by the Chicago Housing Authority (Keels et al. 2005). Participating families in highly segregated neighborhoods of concentrated poverty in Chicago had the opportunity to move to more racially integrated neighborhoods, and families were offered places to move to suburbs or the city, based on availabilities that arose in the program. In contrast, MTO targeted only vouchers, paired with neighborhood poverty reductions, by promoting residential mobility of low-income families who lived in distressed public housing projects into lower poverty neighborhoods. Importantly, the MTO experiment explicitly incorporated random assignment of who received which of the 3 treatments (1 of 2 vouchers, or no voucher) into the study, while Gautreaux did not. Although Gautreaux researchers stated that participants were assigned to a rental unit in a city versus suburban neighborhood regardless of the participant’s locational preference (Rosenbaum 1995), some evidence suggests that program managers screened families they believed were better suited for moving to the suburbs (Goering 2003), thereby systematically patterning the treatment assignment. This non-random assignment of the treatment subjects the findings from the Gautreaux program to selection bias (Acevedo-Garcia et al. 2004, Massey 2013). Moreover, Gautreaux did not include a control group (since both groups moved), and there was considerable loss to follow up which may have biased results (Acevedo-Garcia et al. 2004, de Souza Briggs 1997). As a result, Gautreaux encountered considerable methodological issues that limit causal inference (Acevedo-Garcia et al. 2004).

In contrast, the MTO study employed an experimental design with a large sample of families with strong longitudinal components, in large part to answer the methodological limitations of Gautreaux (Orr et al. 2003). MTO families were randomly assigned by the program either to receive a traditional Section 8 voucher, receive housing counseling plus a low-poverty voucher that could only be used to rent an apartment in a census tract with less than 10% of residents in poverty, or to receive no voucher, and remain in public housing (the control group) (Orr et al. 2003). MTO is also stronger methodologically by enrolling a much larger sample size than Gautreaux (which was around n=114 in Gautreaux, compared to MTO’s 4,604 families). MTO experienced relatively small attrition, because its investigators designed a strong follow up protocol, including employing a 2-stage follow up at each survey to purposefully follow up and control for differential attrition by hard-to-reach respondents (Orr et al. 2003, Sanbonmatsu et al. 2011).

Previously published results of the MTO studies have examined whether initial improvement in neighborhood conditions upon the first lease of a unit using a voucher was sustained over time (Nguyen et al. 2017, Orr et al. 2003, Schmidt, Glymour and Osypuk 2021). The MTO interim evaluation suggested that low-poverty and traditional voucher groups experienced sustained improvement in neighborhood economic conditions and social environments compared with the in-place controls during the 4- to 7-year study period, but somewhat narrowed over time (Orr et al. 2003). For example, the length of time spent in high-poverty areas during the study period substantially declined for families in both voucher groups, with a greater reduction for the low-poverty group. At the time of interim data collection, families in voucher groups lived in neighborhoods with higher proportions of employed adults, two-parent families, high school graduates, and homeowners than their in-place control counterparts. Later studies of the MTO experiment have captured multiple dimensions of neighborhood environments, including economic, social, physical, and environmental characteristics, to which MTO participants were exposed throughout the 4- to 7-year study period, and reported substantial favorable impacts (Antonakos et al. 2020, Nguyen et al. 2017, Schmidt, Glymour and Osypuk 2021).

The MTO final evaluation report suggested that gains in neighborhood quality may further extend over the entire 10- to 15-year study period (Sanbonmatsu et al. 2011). Voucher-induced residential mobility had substantial, favorable impacts on average neighborhood sociodemographic characteristics, including tract poverty rate and median household income, across the entire study period.

Geography of Opportunity and Child Opportunity Index

Tenant-based housing voucher programs are designed to promote upward residential mobility among housing voucher recipients, and thus, participant locational outcomes are one of the important program evaluation criteria. For example, HUD provides an incentive to local housing agencies administering a tenant-based voucher program when they have success in encouraging housing voucher recipients to move to low-poverty neighborhoods (HUD 2022). But when multiple dimensions of neighborhood opportunity are not considered simultaneously, it is possible that increases in economic opportunity by moving to a lower poverty neighborhood could be counteracted by declines in neighborhood opportunity in other domains (Talen and Koschinsky 2014).

Unlike the single program evaluation criterion used by the HUD for the locational outcome of housing voucher recipients, previous studies on housing voucher programs have embraced the multidimensionality of neighborhood opportunity. Those studies have examined how housing voucher recipients experienced change in built environments (e.g., healthy food and green space, and walkability) (Antonakos et al. 2020, Talen and Koschinsky 2014) and accessibility to jobs, public transportation, and service facilities (Park and Choi 2021, Turner et al. 2011, Wang, Larsen and Ray 2015). But previous studies on neighborhood opportunity have also started to ask the question of “neighborhood opportunity for whom?” (Lung–Amam et al. 2018, Reina and Aiken 2022), suggesting the necessity of defining neighborhood opportunity specifically to a subgroup of housing voucher recipients. However, previous studies have rarely examined the locational outcome of families with children receiving a housing voucher using an opportunity measure specifically designed for child opportunity, although some domains in child opportunity could possibly overlap with opportunities pertinent to other subgroups of housing voucher recipients.

This study will revisit the MTO demonstration to examine its long-term effect on neighborhood opportunities for low-income children using an innovative and multidimensional neighborhood measure for households with children: the Child Opportunity Index (COI). This index is grounded in the geography of opportunity scholarship that illuminates the consequences of unequal access to neighborhood-based opportunity by race and ethnicity for children (Acevedo-Garcia et al. 2020). Based on a conceptual model of child development, the COI includes 29 child-relevant indicators of neighborhood resources that facilitate healthy child development and encompass three domains: education, health and environment, and social and economic opportunity. The index captures the multidimensionality of neighborhood environments, and is more suitable for studying families than an index of concentrated disadvantage or vulnerability because it is child specific.

Heterogeneous Treatment Effects and Tailored Housing Policy

Although the intention of housing voucher policy is to improve housing and neighborhoods for low-income families overall, effects may not be uniform. Vulnerable families, such as those with health problems or other stressors, may not benefit from housing vouchers as much as more advantaged families (Osypuk et al. 2012). It is possible that some subgroups may not benefit or may even be harmed. A few studies of the MTO experiment have examined the heterogeneous treatment effects of housing vouchers on neighborhood environments (Arcaya et al. 2016, Jeon, Dawkins and Pendall 2018, Nguyen et al. 2017), but they measure neighborhood opportunity narrowly and reflect shorter follow-up periods (4–7 years after randomization). Methodologically, prior studies have primarily relied on the traditional regression-based subgroup analysis, which tested treatment interaction terms in regression models with a priori hypotheses specification, typically one at a time (see Nguyen, Rehkopf, Schmidt, & Osypuk, 2016, for an exception). But traditional subgroup analysis could hinder exploratory research on critical subgroups that are theoretically less developed. Traditional regression techniques are also limited in their application to detect higher-order patterns of treatment modification because they exponentially increase the number of parameters to be estimated by specifying different combinations of higher-order interactions (Su et al. 2009). Therefore, power is limited to test such 3, 4, or 5-way interactions (for example) unless sample size is very large.

Machine learning methods overcome these limitations, to examine heterogeneous treatment effects of housing vouchers on long-term neighborhood opportunities for children. Machine learning approaches, including model-based recursive partitioning, can identify subgroups with a similar covariate-outcome association defined by a series of participant characteristics (Lipkovich, Dmitrienko and D’Agostino 2017, Zeileis, Hothorn and Hornik 2008). It is suitable for hypothesis generation using large datasets like the MTO data that have extensive information on individual and household characteristics to explore potential factors that modify housing voucher effects on long-term neighborhood opportunity. It also addresses complex interactions in a more parsimonious way than traditional regression methods by incrementally identifying groups with similar treatment effects (Su et al. 2009). But these methodological advantages of the recursive-partitioning method come with some limitations, particularly because the method fails to report the directionality of treatment effects among subgroups. Thus, it serves as a tool for an explorative study to identify potential treatment modifiers whose directionality needs to be determined in future confirmatory studies.

RESEARCH QUESTIONS

This study leverages the MTO experiment to address limitations of prior work on the effects of housing vouchers on the long-term exposure to neighborhood opportunity and their heterogeneous effects across subgroups. This study first asks: Are housing voucher recipients exposed to higher neighborhood opportunity overall and across domains over the entire 10–15-year study period compared to their in-place control counterparts? We expect that voucher recipients might experience improvement in neighborhood opportunity, particularly in the social and economic domain, compared with families in the control group as the MTO demonstration was designed to promote residential mobility into low-poverty neighborhoods using a housing voucher. Families in voucher groups might experience higher neighborhood opportunity in other domains with varying degrees of improvement, depending on how well social and economic opportunity predicts opportunity in other domains. We also expect that improvement in neighborhood opportunity may decline over time as families in the control group leave highly distressed public housing projects during the study period, contributing to narrowing opportunity gap compared with voucher groups.

Secondly, we also examine heterogeneous treatment effects of housing vouchers on neighborhood opportunities to ask: Are there certain subgroups of people who experience more or less beneficial locational outcomes from housing vouchers? We apply model-based recursive partitioning to answer this question among subgroups defined by multiple simultaneously- examined baseline covariates. Only a few studies have selectively examined potential modifiers for housing voucher effects on neighborhood opportunity in the short-term using a traditional regression-based subgroup analysis. It is unclear whether those potential effect modifiers have similar impacts on neighborhood opportunity in the long term. Given the lack of prior research on effect modifiers, it is critical to conduct an exploratory study to identify potential effect modifiers from a variety of baseline covariates, including sociodemographic, economic, and health characteristics of MTO participants.

DATA

The MTO study was a randomized controlled trial conducted with 4,600 volunteer families from highly distressed public housing who were eligible for the federal Section 8 housing voucher program, and had children under 18 years old at baseline (Goering and Feins 2003). This experiment was conducted in five U.S. cities (sites), Baltimore, Boston, Chicago, Los Angeles, and New York, with a surveys at each of three time points: baseline (1994–98), interim (2001–02), and final (2008–10).

Random Assignment

The MTO experiment randomly assigned eligible, volunteer families into one of three treatment groups: the low-poverty neighborhood voucher group (“low poverty”), the traditional Section 8 voucher group (“traditional”), and the in-place control group. The two voucher groups had 90 days to lease a qualified apartment or the voucher was withdrawn. Low-poverty voucher group participants were required to redeem their voucher in a census tract where fewer than 10 percent of residents live in poverty; to help offset this additional locational constraint, local housing agencies collaborated with nonprofits to provide intensive housing counseling to this low-poverty group. The traditional voucher group received a Section 8 voucher without housing counseling, and without any locational constraints on voucher use. The in-place control group serves as a counterfactual of what would have happened for the treatment group if they had not been offered the voucher treatment, since they received no voucher offer from the MTO study, although they remained eligible to continue living in public housing. After random assignment, MTO put no constraints on the in-place control group. They could remain in their baseline public housing, but they were also eligible to apply for other low-income housing assistance programs or leave public housing developments without having any housing assistance.

Residential Histories

Using HUD program administrative data and MTO surveys, HUD constructed the MTO residential history dataset with a tract-level residential address of MTO adult participants in each calendar quarter starting at baseline (1994–98) through to the final survey (2008–10). This study maps the Child Opportunity Index (COI) 2.0¹, a multidimensional assessment of various opportunity for families, onto all census tracts over this 10–15 year follow up period. The COI consists of three subdomains—education, health and environment, and social and economic opportunity—with 29 indicators to capture multiple dimensions of neighborhoods at the census tract level. Nationally normed, overall, and domain-specific COIs rate all neighborhoods in the 50 U.S. states and Washington, D.C. (Acevedo-Garcia et al. 2020). COI 2.0 data is available for 2010 and 2015; we used the former (2010) because the MTO data collection period was 1994–2010.

We limited our analysis to MTO adults who participated in the final survey (N = 3,300). They each contributed 58 person-quarter observations on average, which resulted in 185,000 person-quarter observations in total. For respondents with a longer follow-up period (elapsed time from randomization to the final survey), more person-quarter observations are available. The shortest follow-up period for those who participated in the final survey was 10 years (40 calendar quarters) after randomization for respondents who were interviewed for the baseline survey (1994–98) in 1998 and for the final survey (2008–10) in 2008; the longest follow-up period was 15.5 years (62 calendar quarters) for those who entered the study in 1994 and exited in 2010. We restructured the data to line it up by baseline entry into the study (1994–1998), so that the missing data due to staggered study entry was at the end of the residential trajectory. MTO participants started to be dropped from our analysis 10 years after randomization; elapsed times to 99, 75, and 50 percent of the sample remaining are 12, 13, and 14 years, respectively. After excluding one variable with more than 10% missing values (youth behavioral problem), we mode/mean-imputed baseline missing variables by site. We mean-imputed missing COI values before attrition (< 1%).

MEASURES

Average Post-randomization Child Opportunity Index (COI) Score.

We used overall and domain-specific child opportunity score as an outcome. The COI was developed and validated by a long, deliberate process by researchers at the Heller School for Social Policy and Management, Brandeis University, in collaboration with the Kirwan Institute for the Study of Race and Ethnicity at Ohio State University (Acevedo-Garcia et al. 2014, Acevedo-Garcia et al. 2020, Noelke et al. 2020). The COI 2.0 includes 29 child-relevant indicators of neighborhood resources that facilitate healthy child development and encompass three domains: education, health and environment, and social and economic opportunity. Indicators in the education domain reflect quality and access to early childhood education, quality of elementary and secondary schools and social resources related to educational achievement. The health and environment domain reflects features of healthy environments, such as access to healthy food and green space, and features that are toxic, such as pollution from industry and exposure to extreme heat. The social and economic domain contains nine indicators measuring access to employment and neighborhood social and economic resources (Acevedo-Garcia et al. 2020). To construct Child Opportunity Scores, we ranked all neighborhoods in the U.S. according to their COI z-scores from lowest to highest and then divided them into 100 rank-ordered groups. We used percentiles, weighted using the total number of children in a given tract, to define the cut points dividing neighborhoods into groups that contain 1% of the child population each (Acevedo-Garcia et al. 2020). Each of the groups was assigned a Child Opportunity Score from 1 (lowest opportunity) to 100 (highest opportunity). Next, we averaged quarterly opportunity score from baseline through the final survey for each MTO participant. If MTO participants were assigned into one of the two voucher groups, they were offered a voucher to lease a qualified apartment within 90 days. Thus, we captured post-treatment effects of housing vouchers by excluding the first quarter opportunity score since randomization for all respondents. Only families with at least one child under aged 18 at baseline were included in the MTO experiment. Assuming that the youngest child in each family at baseline lived at home until they were 18 years old, nearly 90 percent of the follow-up time on average occurred with a child in the home. This is 11.2 years of quarterly information on neighborhood child opportunity, which could be an underestimate if families welcomed a new minor at any time during the follow-up periods.

Treatment.

We conducted statistical tests to determine if the treatment effects on the low poverty and Section 8 voucher groups were statistically homogeneous. The results indicated that treatment effects of the low-poverty group and the Section 8 group were significantly different for neighborhood opportunity. Thus we modeled the two voucher treatment groups separately (compared to the control group).

Baseline Characteristics.

We used a comprehensive set of baseline household and household head characteristics as potential moderators of the treatment effects of housing vouchers on neighborhood opportunity. Household characteristics include which city they lived in (1=Baltimore, 2=Boston, 3=Chicago, 4=Los Angeles, 5=New York) when they were selected for MTO demonstration. Household size was categorized into a number of household members (1=two or smaller, 2=three, 3=four, 4=five or more) and included as a categorical variable in the model. We included a variable of whether a household had adolescents in the home and a car at baseline (coded as 1 for a positive value).

Household head characteristics include age (coded as continuous variable measured in years on May 31, 1996), race (1=Black, 2=White, 3=other race), and binary variables that ask whether the household head was of Hispanic ethnicity, had never married, was 17 years old or younger at birth of first child, was working for pay, or was receiving welfare (Aid for Families with Dependent Children or Temporary Assistance for Needy Families) at baseline. We categorized completed education of household head into one of the three categories (1=less than high school, 2=GED, 3=high school diploma or above). We also included a variable of whether the household head was in school (coded as 1 for a positive value).

We also included variables indicating health or developmental problems in the model as binary variables with a positive value coded as 1: household member had a disability, household youth was expelled/suspended from school, household youth got help for a learning problem, school called about household youth problem with schoolwork/behavior, household member was a crime victim, and household youth was hospitalized in the year following birth.

We included baseline mobility history in the model as binary variables with a positive value coded as 1: household head had been living in his/her neighborhood for 5 or more years and had moved more than 3 times in the 5 years prior to baseline.

ANALYTIC METHODS

We first examined imbalance in baseline covariates between randomized voucher treatment and in-place control groups using F-statistics. We expect balance in baseline covariates given the randomized design. Next, to estimate mean trajectories of neighborhood opportunity, we used a linear regression model that includes treatment groups interacted with yearly elapsed time dummies since randomization. We used yearly COI for simplicity since including quarterly time dummies in the model dramatically increases a number of parameters to be estimated. We used cluster-robust standard errors to account for repeated outcome measures. Next, we examined whether families in the low-poverty and traditional voucher groups are exposed to higher neighborhood opportunity overall and across domains compared with the in-place control during the entire 10- to 15-year study period. We examined everyone in the exposure group to which they were randomly assigned, regardless of adherence, or use of the voucher (Gupta 2011), also known as intent-to-treat.

To identify potential heterogeneous treatment effects of housing vouchers on neighborhood opportunity, we used model-based recursive partitioning while accounting for sample attrition and varying random assignment ratios across time by using final survey weights. We implemented models using the function mob() within the partykit R package (Hothorn and Zeileis 2015). Model-based recursive partitioning successively splits the data into subgroups of observations by selecting the best partitioning variable and the cutoff value that most differentiates treatment effects for subgroups (Krzykalla, Benner and Kopp-Schneider 2019, Lipkovich, Dmitrienko and D’Agostino 2017, Tian et al. 2014). We required that splits differentiate treatment effects at the Bonferroni-corrected significance level of .01 and each subgroup have at least 50 cases. Fitting one model using recursive partitioning can result in overfitting the data, which would mean that the results capture not only true treatment-outcome relationships, but also noise that is specific to the data. We implemented random forest recursive partitioning (Garge et al. 2019). Recursive partitioning using the random forest method addresses the overfitting issue by randomly resampling subsets of the data, building a separate model for each subset, and averaging the resulting predictions (James et al. 2017).

Unlike the traditional regression model that provides a p-value, probability of committing a false positive, for each parameter, this conventional way of selecting a significant variable is not available in the recursive partitioning model with random forest method. Model-based recursive partitioning does conduct significance tests for all the potential modifiers every time it searches for a partitioning variable that splits a sample into subgroups with similar treatment effects. A variable is considered to be potentially important in modifying treatment effects when it is selected as a splitting variable that best differentiates treatment effects for subgroups at the predetermined significance level. But for model-based partitioning models with the random forest method, we need further consideration because the same variables identified in one subsample might not necessarily be selected in another subsample. As one way to measure how much each variable is important as a potential modifier, we use a permutation-based variable importance statistic. A key concept with this statistic is to measure change in the overall prediction accuracy of the model when we randomly permute the original values in a variable for each subsample, and then aggregate them across random subsamples to produce a single importance statistic for each variable (Friedman, Hastie and Tibshirani 2001). If a variable is important in a model, then after its permutation the model prediction should be less precise. Positive values of the permutation-based variable statistic indicate prediction accuracy declines when we predict an outcome using randomly permuted values of the variable instead of its actual values. A variable importance of 0 means no change in prediction error occurs when values of a variable is randomly permuted. Using the conventional measure in prior studies, we consider variables to be potentially important if their value was more than the absolute value of variable with the largest negative value (i.e., the least important moderator) (Brunwasser and Gillham 2018, Strobl, Malley and Tutz 2009).

RESULTS

Table 1 suggests that treatment and control groups are well balanced in the baseline covariates in our analyses, as expected from an experimental design. MTO participants are a socially and economically disadvantaged group with a majority of them being non-white (94%), about three quarters in the MTO sample being on welfare, Aid to Families with Dependent Children (AFDC) or Temporary Assistance for Needy Families (TANF) at the baseline, and almost half having neither a GED nor a high school diploma. Among youth developmental problems, school calls about youth behavior was the most common (30%), followed by youth having a learning problem (21%), youth having a health problem requiring special medicine (16%), youth having been suspended or expelled from school (14%), youth having health problems limiting activities (13%), and youth who weighed less than 6 pounds at birth (10%).

Table 1.

Descriptive Statistics for Moving to Opportunity Sample at Baseline (1994–1998), Overall and by Treatment Group

	Overall	Control	Treatment	F-Test
Observation	3300	1100	2100
Site				0.999
Baltimore	0.136	0.135	0.136
Boston	0.204	0.205	0.203
Chicago	0.206	0.205	0.206
Los Angeles	0.225	0.226	0.225
New York City	0.229	0.229	0.229
Household size				0.231
Less than 2	0.210	0.194	0.218
3	0.308	0.330	0.297
4	0.231	0.221	0.235
More than 5	0.251	0.255	0.250
On AFDC/TANF	0.757	0.765	0.753	0.477
Possession of a car	0.183	0.169	0.189	0.207
Disabled household member	0.151	0.146	0.153	0.637
No teenage (13–17 years) children	0.621	0.646	0.609	0.063
Household member victimized in past 6 months	0.419	0.412	0.422	0.634
Household head moved >3 times in past 5 years	0.096	0.107	0.091	0.178
Household head lived in neighborhood for ≥5 years	0.616	0.619	0.615	0.840
Education				0.176
Less than high school	0.456	0.440	0.464
GED	0.179	0.199	0.169
High school graduate or more	0.365	0.361	0.367
AD enrolled in school	0.161	0.162	0.160	0.934
AD Race				0.604
Black	0.657	0.670	0.651
White	0.072	0.068	0.074
Other	0.271	0.263	0.275
AD Hispanic origin	0.316	0.301	0.322	0.266
AD Male	0.018	0.022	0.016	0.314
Never married	0.638	0.648	0.634	0.459
AD teen parent	0.243	0.233	0.247	0.411
AD working for pay	0.254	0.236	0.262	0.136
AD Age on May 31, 1996 (years)	32.9	32.8	33	0.729
YT learning problem in past 2 years	0.211	0.209	0.212	0.850
YT suspended/expelled in past 2 years	0.141	0.130	0.147	0.246
School call about YT behavior in past 2 years	0.297	0.293	0.299	0.759
YT weighed less than 6 pounds at birth	0.099	0.114	0.093	0.094
YT in hospital before 1st birthday	0.146	0.164	0.138	0.078
YT health problem requiring special medicine	0.163	0.160	0.165	0.768
YT health problems limiting activities	0.129	0.124	0.131	0.597

Notes. All results were approved for release by the U.S. Census Bureau, authorization number CBDRB-FY21-ERD003-005. AFDC = Aid to Families with Dependent Children; TANF = Temporary Assistance for Needy Families; AD = Adult; YT = Youth.

Figure 1 presents mean residential trajectories of child opportunity indices, overall and by subdomain, by treatment and control groups with 95% confidence intervals at 1, 5, and10 years since randomization. There was a large neighborhood child opportunity gap one year after randomization, with the low poverty group experiencing a 12-point higher COI than the control group and 7-point higher COI than the traditional voucher group. The child opportunity gap narrowed over time. While the low poverty voucher group’s COI stayed relatively stable over time, the COI in the traditional voucher and control groups both increased so that by 10 years post-randomization, the two voucher groups had similar COI scores, and the control group was not far behind, although controls were statistically worse than the voucher groups.

Figure 1.

Mean Child Opportunity Scores of the Moving to Opportunity Sample (1994–2010) with 95% Confidence Interval by Study Groups and Years Since Randomization

Notes. We used a linear regression model with cluster-robust standard errors, which includes treatment groups interacted with yearly time dummies since randomization through to the final survey. All results were approved for release by the U.S. Census Bureau, authorization number CBDRB-FY21-ERD003-010.

Figure 2 shows the voucher treatment-control mean differences in the overall and domain-specific average post-treatment COIs with 95% confidence intervals, separately by low-poverty and traditional voucher group. Results suggest that over the 15 years of average follow-up period, both voucher groups had been exposed to higher neighborhood opportunity overall and across all domains of child opportunity compared to the control, with the exception of the traditional voucher group’s health and environmental opportunity. Low-poverty and traditional voucher groups experienced higher overall neighborhood opportunity than the control group by the child opportunity scores of 7.4 and 3.2 points, respectively. As suggested in Figure 1, the positive impacts of housing vouchers on neighborhood opportunity lasted more than a decade. Regarding domain-specific child opportunity, both the low-poverty and the traditional voucher group experienced higher social and economic opportunity than the control by 7.0 and 3.2 points, respectively. Both voucher groups also experienced higher educational opportunity than the control by 6.1 and 2.6 points. For health and environmental opportunity, only the low-poverty group experienced higher opportunity than the control, by 3.5 points. By examining the two groups of results in Figure 2, we see that the low-poverty group experienced significantly higher opportunity overall and across all the domains of child opportunity compared to the traditional voucher group. Additional analyses revealed all four to be significantly higher for the low-poverty group at the level of p<0.05.

Figure 2.

Treatment-Control mean difference in overall and domain-specific COIs for low-poverty and traditional voucher group in the Moving to Opportunity Sample (1994–2010)

Notes. We ran 4 separate regression models for the outcomes of overall and domain-specific child opportunity scores weighted by final survey weights. All results were approved for release by the U.S. Census Bureau, authorization number CBDRB-FY21-ERD003-010.

Figure 3 is a plot of variable importance from a model-based recursive partitioning with random forest of the low-poverty treatment effects on the average post-randomization COI. The x-axis indicates importance scores relative to the highest score. The red vertical line indicates the absolute value of the largest negative importance score, which is used as a cutoff to detect a potential effect modifier. The variables displayed at the top of the plot are the strongest effect modifiers. The results suggest that study site and whether the youth had a learning problem are most important, with the remaining variables having minimal relative importance in terms of differentiating treatment effects across subgroups. Figure 4 is a plot of variable importance from a model-based recursive partitioning with random forest of the traditional voucher treatment effects on the average post-treatment COI. The results suggest that study site is most important and disabled household member, youth weighed less than 6 pounds at birth, household had a car, and school called about youth behavior follow, as other important modifiers.

Figure 3.

Variable importance plot based on a model-based recursive partitioning with random forest of the low-poverty treatment effects on the average post-treatment Child Opportunity Score.

Notes. Data are from the Moving to Opportunity experiment (1994–2010). The sample is restricted to those who were randomly assigned to the low-poverty treatment or control group (n = 2,500). The x-axis indicates importance scores relative to the highest score. The red vertical line indicates the absolute value of the largest negative importance score, which is used as a cutoff to detect a potential effect modifier. The variables displayed at the top of the plot are the strongest effect modifiers. All results were approved for release by the U.S. Census Bureau, authorization number CBDRB-FY21-ERD003-010. AD = Adult; YT = Youth; HH = Household; AFDC = Aid to Families with Dependent Children; TANF = Temporary Assistance for Needy Families.

Figure 4.

Variable importance plot based on a model-based recursive partitioning with random forest of the traditional voucher treatment effects on the average post-treatment Child Opportunity Score.

Notes. Data are from the Moving to Opportunity experiment (1994–2010). The sample is restricted to those who were randomly assigned to the traditional voucher treatment or control group (n = 1,800). The x-axis indicates importance scores relative to the highest score. The red vertical line indicates the absolute value of the largest negative importance score, which is used as a cutoff to detect a potential effect modifier. The variables displayed at the top of the plot are the strongest effect modifiers. All results were approved for release by the U.S. Census Bureau, authorization number CBDRB-FY21-ERD003-010. AD = Adult; YT = Youth; HH = Household; AFDC = Aid to Families with Dependent Children; TANF = Temporary Assistance for Needy Families.

DISCUSSION

Tenant-based rental assistance has received much attention as a tool to ameliorate American poverty and income segregation (Briggs, Popkin and Goering 2010, Desmond 2016). Voucher-based rental assistance may mitigate the intergenerational reproduction of poverty by promoting residential moves to higher opportunity neighborhoods (Chetty, Hendren and Katz 2016). Our results suggest that the previously documented improved neighborhood opportunity for voucher recipients in the short term (Nguyen et al. 2017, Schmidt, Glymour and Osypuk 2021) was also sustained for MTO participants over the entire 10- to 15-year study period. Moreover, when compared to the findings from the final MTO report, our findings suggest that substantial, favorable long-term impacts of a housing voucher are not limited to just sociodemographic neighborhood characteristics, but also multiple subdomains of neighborhood opportunity, particularly relevant for child development. Importantly, families in the low-poverty voucher group experienced higher neighborhood child opportunity (overall and across domains) compared to the traditional voucher group, suggesting the necessity of supplementing a traditional voucher program with intensive housing counselling and support for housing search. Our results with the multidimensional measure of neighborhood child opportunity also suggest that a narrowly focused measure of neighborhood opportunity in policy design could miss an important dimension of neighborhood opportunity that influences child development. In particular, traditional voucher recipients did not experience improvement in health and environmental opportunity compared with families in the control group. Although HUD still uses a narrowly defined neighborhood opportunity (low poverty and minority concentration) to evaluate the performance of the mainstream housing voucher program in terms of the locational outcome of its recipients, the concept of multidimensional neighborhood opportunity is not new to HUD. As part of their efforts to implement Affirmatively Furthering Fair Housing (AFFH) Act, HUD has even developed assessment tools to measure disparity in neighborhood opportunity among program participants. Our results suggest that HUD needs to further extend this effort to incorporate the multidimensionality of neighborhood opportunity into the evaluation metric of a housing voucher program.

Moving beyond the question of whether a program works in the overall population, we conducted heterogeneous treatment effect (HTE) analyses using a machine learning approach to demonstrate its relative strengths and limitations to the traditional regression-based approach. Housing scholars have conducted HTE analysis to identify subgroups of people responding differently to housing program participation, which can inform housing policy makers in tailoring housing programs (Arcaya et al. 2016, Blumenberg and Pierce 2017, Jeon, Dawkins and Pendall 2018). There were limitations, however, with the traditional approach used in the previous studies. In contrast to the traditional approach of a priori specified stratification with single or at the most two variables, machine learning-based HTE analyses allow one to test differential treatment effects across subgroups that are defined not just with one, but with a combination of multiple variables: higher order interactions of treatment and potential effect modifiers (James et al. 2017). Using a machine-learning approach, we tested higher order interactions of theoretically informed 25 potential treatment effect modifiers and proposed a reduced set of variables that define subgroups responding differently to housing voucher treatment in terms of long-term child opportunity. Moreover, we also used a random forest technique incorporated within a machine learning approach, which randomly subsamples data and average results across subsamples to address the concern that a data-driven approach might capture sample specific noises rather than a true relationship. One of the challenges that scholars are actively working in the field of HTE analyses is the directionality of the association between modifiers and treatment effects. Currently available methods to obtain the directionality within a machine learning framework have not fully matured to accommodate a complex survey design such as MTO and often rely on strong assumptions (Molnar 2022). But scholars in the field of machine learning are developing a unified machine learning framework to examine not just variable importance as an effect modifier, but also its directionality (Alten et al. 2021, Lundberg and Lee 2017, Molnar 2022). Thus, applied housing researchers need to pay attention to the development of a streamlined machine learning framework for heterogeneous treatment effect analyses, which we believe will inform housing policy makers in tailoring housing programs.

Results from model-based recursive partitioning identified several potential modifiers of the MTO treatment-overall neighborhood opportunity association, including study sites, health and developmental problems of household members, and having vehicle access. Site effects could possibly reflect one common issue in a multi-site study: nonstandardized implementation of the treatment conditions across sites (Feaster, Mikulich-Gilbertson and Brincks 2011). A prior study documented that intensive housing counselling and housing search support provided by local non-profits for families in the low-poverty group differ across sites (Feins, McInnis and Popkin 1997), suggesting the possibility of generating larger treatment effects for the study site where implementation of the low-poverty treatment conditions was more successful. But effects of housing vouchers also differ across sites even among families in the traditional voucher group whose treatment conditions were more standardized, but not completely (see Kim (2022) for discussion on local implementation of the traditional voucher program), by federal regulations governing the traditional voucher program. Thus, nonstandardized implementation of treatment conditions might not fully explain site-specific effects of housing vouchers.

Another possible explanation is that site-level contextual factors have contributed to differential treatment effects on neighborhood opportunity across sites. Prior research suggests that cross-site differences in individual-level compositional characteristics that potentially modify treatment effects only partially explain heterogeneous treatment effects across sites, suggesting the importance of site-level contextual factors (Rudolph et al. 2018). However, since the MTO study only has five sites, we lack statistical power to model site-level contextual factors (Orr et al. 2003). Future studies need to examine whether site-level contextual factors, including an unequal distribution of rental units affordable to low-income families in opportunity neighborhoods across metropolitan areas (Acevedo-Garcia et al. 2020), contribute to long-term exposure to neighborhood opportunity among housing voucher recipients.

We also identified health problems as a potential modifier of the housing voucher effects on neighborhood opportunity. Literature on health and residential mobility decisions suggests that child health problems in a family can disrupt parental employment stability and cause financial hardship (Casey et al. 2004, Kuhlthau et al. 2005), which may explain the association of child health problems and neighborhood poverty over time (Arcaya et al. 2016). One prior study using the MTO interim data suggested that child health problems in a family not only lowers the chance of taking up a voucher, but also predicts selection into poor neighborhoods among treatment compliers in the low-poverty group (Arcaya et al. 2016). Our study suggests that health problems of a household member have a far-reaching impact on the long-term neighborhood opportunity across multiple domains, possibly taking a heavy toll on those with additional burden of taking care of household members with health problems. Also, we identified that having a car at the baseline could modify voucher treatment effects. Previous studies suggest that housing voucher holders without vehicle access are less likely to conduct housing searches in and make opportunity moves to neighborhoods located in suburban areas not densely populated enough to make mass transportation viable (Varady 2010). In particular, previous studies using the MTO data suggested that vehicle access improves exposure to neighborhood opportunity regardless of housing voucher treatment conditions (Blumenberg and Pierce 2017, Jeon, Dawkins and Pendall 2018). Our study further suggests that a lack of vehicle access could even compromise the favorable impact of housing vouchers on long-term neighborhood opportunity, possibly indicating a necessity to bundle a housing voucher with a transportation voucher to maximize beneficial neighborhood opportunity effects of housing vouchers.

It is also noteworthy that our result from model-based recursive partitioning suggests that the effects of housing vouchers on neighborhood opportunity do not significantly differ by race. Considering a large body of literature on the pervasive discrimination against and segregation of racial minorities in the U.S. housing markets (Acevedo-Garcia et al. 2020, Massey and Denton 1993), this result is somewhat contradictory to our expectation that doubly disadvantaged “racial and ethnic minority” “voucher” holders would be less able to make an opportunity move. The methodology we use is relatively conservative in terms of identifying HTE due to the use of cross-validation, where significant findings are only identified if replicated on subsets of the data. But the non-significant finding of race could possibly result from a lack of statistical power of the MTO study to conduct subgroup analyses defined by racial and ethnic status, whose majority of participants (97 percent of the sample) are racial and ethnic minorities. Also, it is critical to note that all the variables included in our analysis as a potential effect modifier are measured at the baseline. This was a deliberate decision not to make post-randomization variables break the strength of the randomized design. Considering our long-term follow up period, non-significance of some of the baseline variables does not necessary mean that they do not play a role in shaping neighborhood opportunity among voucher recipients; rather it could reflect their time-varying nature, particularly those associated with employment and household instability among low-income people, and their temporally restricted impacts on neighborhood opportunity, which could be diluted over the long-term follow up period.

This study has four primary limitations. First, data was gathered in five large U.S. cities between 1994 and 2010. Thus, due to temporal change and local variation in rental markets, it may not be generalizable to other periods or localities. However, the robust experimental design and long-term follow up illuminate the impact of vouchers, at least in large cities, in a way that few papers have. Second, COI was measured in 2010. Neighborhood environments are relatively stable over an extended period of time (Sampson 2012, Schmidt et al. 2014), but not all are wholly stable. Moreover, about 23% of the control group lived in severely distressed public housing at baseline that was demolished by the HOPE VI Program, which likely induced change in neighborhood opportunity. Thus the neighborhood opportunity MTO participants face in the early years of the MTO study might have been quite different than by the end of the study. But COI is the only publicly available, multidimensional measure of child opportunity, created and validated through a long-term, deliberate process (Acevedo-Garcia et al. 2014, Acevedo-Garcia et al. 2020, Noelke et al. 2020) and thus this paper sheds light on how housing policy affects child opportunity improving on prior measures.

Conclusion

This study is the first to examine the average and heterogeneous treatment effects of housing vouchers on long-term neighborhood opportunity using an innovative measure of neighborhood opportunity particularly relevant to children development and applying a machine learning approach. This study suggests that the traditional housing voucher program supplemented by housing counseling has a potential to ameliorate American poverty by increasing long-term exposure to neighborhood opportunity among low-income children. Along with support for making the housing voucher program to be an entitlement (Kim, Burgard and Seefeldt 2017), particularly to improve long-term neighborhood opportunity for those who are equally eligible, but not receiving any housing assistance, this study suggests that it is critical to further identify heterogeneous voucher treatment effects across subgroups to develop tailored housing intervention.