EVALUATING CONTRADICTORY EXPERIMENTAL AND NON-EXPERIMENTAL ESTIMATES OF NEIGHBORHOOD EFFECTS ON ECONOMIC OUTCOMES FOR ADULTS

Abstract

Although non-experimental studies find robust neighborhood effects on adults, such findings have been challenged by results from the Moving to Opportunity (MTO) residential mobility experiment. Using a within-study comparison design, this paper compares experimental and non-experimental estimates from MTO and a parallel analysis of the Panel Study of Income Dynamics (PSID). Striking similarities were found between non-experimental estimates based on MTO and PSID. No clear evidence was found that different estimates are related to duration of adult exposure to disadvantaged neighborhoods, non-linear effects of neighborhood conditions, magnitude of the change in neighborhood context, frequency of moves, treatment effect heterogeneity, or measurement, although uncertainty bands around our estimates were sometimes large. One other possibility is that MTO-induced moves might have been unusually disruptive, but results are inconsistent for that hypothesis. Taken together, the findings suggest that selection bias might account for evidence of neighborhood effects on adult economic outcomes in non-experimental studies.

INTRODUCTION

A prolific literature has generated many insights about the nature and magnitude of the effects of living in disadvantaged neighborhoods. That literature has examined outcomes in many domains—including the labor market, health, crime, fertility, and education—not only in the U.S. but throughout the world.¹ Although an emerging consensus suggests that neighborhoods affect children’s outcomes in multiple domains and also affect adults’ mental and physical health, a persistent unresolved question concerns the causal status of the association between neighborhood characteristics and the economic outcomes of adults. A large number of observational studies find robust associations between exposure to disadvantaged neighborhoods during adulthood and adults’ economic outcomes that are interpreted as causal effects after conditioning on individual and family characteristics. But these results have been called into question by contradictory results from the Moving to Opportunity (MTO) residential mobility experiment, which was able to evaluate the effects of random variation in neighborhood contexts created by random assignment of housing vouchers to public housing families in five U.S. cities in the late 1990s who were followed over the next ten to fifteen years (Orr et al. 2003; Sanbonmatsu et al. 2011).

Analyses of MTO data find persistent statistically significant differences in outcomes by neighborhood poverty in adult mental and physical health (Kessler et al. 2014; Ludwig et al. 2011, 2012, 2013; Sanbonmatsu et al. 2011) and find that children who moved through MTO when they are young (prior to adolescence) experienced long-term economic benefits including higher adult earnings and household incomes (Chetty, Hendren and Katz 2016). But MTO did not find detectable effects on the economic self-sufficiency of those who moved as adults (Ludwig et al. 2012; Orr et al. 2003; Sanbonmatsu et al. 2011). Why do neighborhood effects on economic outcomes for adults differ so markedly from the results in non-experimental studies? One possibility is that the difference is due to selection bias or other differences in study designs. This is an especially challenging problem in non-experimental studies because adults’ employment and earnings are directly related to the process of selection into disadvantaged neighborhoods. And as these unobserved variables also are determinants of future labor market outcomes, the causal effects of neighborhood disadvantage experienced in adulthood on adult employment and earnings may be difficult to detect because of these unobserved common causes in non-experimental studies.

In addition to selection bias, several other possible explanation exist for the seemingly contradictory results between the results from the MTO experiment and the results of non-experimental studies of neighborhood effects on the economic outcomes of adults. These include reliance on different outcome measures, neighborhood effect heterogeneity, differential duration of exposure to disadvantaged neighborhoods, non-linear neighborhood effects, the magnitude of changes in neighborhood context, measures of neighborhood context, and residential moves. However, we are unaware of any empirical studies that attempt to adjudicate between these explanations using both experimental and observational data.

The puzzle of contradictory results is important because neighborhood exposure effects are central to theories about the role of neighborhood context in economic and racial inequalities (Massey and Denton 1993; Wilson 1987). The puzzle is also important for public policy in light of the growth in U.S. residential segregation by income that has been occurring since 1970 (Jargowsky 1997, 2015; Kneebone, Nadeau, and Berube 2011; Reardon and Bischoff 2014; Reardon et al. 2018; Watson 2009). Uncertainty about the nature of neighborhood effects on low-income adults is an obstacle to the design of public policies targeting poverty and inequality. Income poverty is in many ways the central focus of social policy discussions in the U.S., and whether adult economic outcomes are affected by neighborhood environments is central to consideration of whether place-based policies—be they residential mobility or direct efforts to improve neighborhoods themselves—have the potential to reduce poverty.

Drawing inspiration from a long tradition of “within study comparison” research designs (e.g. Cook, Shadish, and Wong 2008; LaLonde 1986; Michalopoulos, Bloom, and Hill 2004), this paper sheds new light on the contradictions between MTO and non-experimental studies of neighborhood effects on the economic outcomes of adults through a non-experimental reanalysis of the MTO data compared with a parallel non-experimental analysis of data from the Panel Study of Income Dynamics (PSID). We choose the PSID because it is one of the most popular sources of data for observational studies of neighborhood effects. Our use of multiple samples, including a separate non-experimental analysis of the MTO sample, provides us with analytic leverage to test various hypotheses and propose more general explanations about the conditions under which non-experimental and experimental results in this domain are most likely to differ.

As detailed below, our results are inconsistent with a number of commonly offered hypotheses about neighborhood effect differences arising from MTO and PSID data. We find no clear evidence that differences in neighborhood effects on adult economic outcomes are due to the duration of exposure to different neighborhood contexts, non-linear effects, or to the magnitude of the change in neighborhood context produced by the MTO intervention. We also find little support for the idea that discrepant findings between MTO and observational studies are due to something about the MTO sample itself, since we can apply non-experimental methods to the MTO dataset (ignoring random assignment and instead focusing on within-randomized-group variation). This is a particularly useful comparison given the difficulty of drawing comparably disadvantaged families to MTO from within the PSID sample. We find the MTO non-experimental analysis typically yields estimates that are fairly similar to those from the PSID. Nor do we find strong evidence that the measurement of neighborhood context or labor market outcomes in MTO and observational studies can reconcile prior divergent estimates.

Our results narrow the set of possible explanations for study differences in the effects of adult neighborhood conditions on economic outcomes down to just two. One hypothesis, derived from qualitative research on housing mobility programs, is that the program timing and location constraints placed on MTO-based moves disrupt the lives of MTO families more than families observed to move in observational studies like the PSID. This hypothesis receives mixed support in further analyses reported below (i.e., it is consistent with some of the analyses we carry out, but not with others). The second hypothesis is essentially a residual explanation and cannot be tested directly with the data: that selection bias accounts for the differences in adult economic outcomes between MTO and observational studies.

We begin with a brief review of the nonexperimental literature on the effects of exposure to neighborhood disadvantage in adulthood on adult labor market outcomes. We then describe the key features of the MTO study design and its impacts on adult outcomes, highlighting similarities and differences between MTO findings and results from observational studies of neighborhood effects. Next we discuss the main hypotheses regarding the sources of these similarities and differences and how we test these hypotheses. After describing the MTO and PSID data and the subsamples we employ, we present our tests of the hypotheses. In the conclusion, we discuss remaining possible explanations for the discrepancy between MTO and PSID estimates and the implications of our findings for future studies.

NON-EXPERIMENTAL STUDIES OF NEIGHBORHOOD EFFECTS ON ECONOMIC OUTCOMES AMONG ADULTS

The empirical study of “neighborhood effects” dates back to the 17^th century (MacIntyre and Ellaway 2003). Hundreds of studies in sociology, epidemiology and other fields have regressed measures of individual outcomes against the attributes of the neighborhood in which the individual resides, adjusting for individual- or family-level background characteristics.² We focus our review here on evidence regarding the effects of adult neighborhood conditions on adult economic outcomes, because it is for adults that prior findings appear to be most divergent between MTO and observational studies.³ It is also the case that adult economic outcomes are a (if not the) dominant focus of social policy. For recent reviews of the effects of childhood exposure to disadvantaged neighborhoods on outcomes during childhood or adulthood, see Galster and Hedman (2013), Galster and Sharkey (2017), and Sharkey and Faber (2014).

Empirical investigations of neighborhood conditions and labor market outcomes during adulthood are typically motivated by one of three theories. The first is spatial mismatch, in which disadvantaged neighborhoods (particularly those in the central city) are thought to be located far from available job opportunities (Kain 1968; Wilson 1987, 1996). The second is the inability to use social networks for job search (Granovetter 1973). Residents of high poverty neighborhoods are socially isolated from mainstream institutions and so are less able to help one another secure employment (Wilson 1987, 1996). An alternative network-based explanation is that social distrust within high-poverty neighborhoods may make residents less willing to help each other find jobs (Smith 2007). The third theory is about social norms around work (Wilson 1987). In neighborhoods with high unemployment rates, formal work may become less normative and other strategies of income generation more socially acceptable. This last explanation is the most controversial (Anderson 1999; Harding 2010). Most empirical research on neighborhood effects on adult outcomes has either attempted to estimate an overall effect of disadvantaged neighborhoods on adult employment, or has focused specifically on testing the spatial mismatch or, more rarely, job-network theories.

Such observational studies have typically attempted to account for the selection of people or families into neighborhoods by including controls for individual and family characteristics or, if the data support it, by using individual or family fixed effects.⁴ These studies often find that more advantaged neighborhoods or lower levels of residential segregation are associated with greater likelihood of employment, higher earnings, and a decreased probability of welfare receipt (Casciano and Massey 2008; Collins and Margo 2000; Cutler and Glaeser 1997; Dawkins, Shen, and Sanchez 2005; Elliott 1999; Fauth, Leventhal and Brooks-Gunn 2004; Hoynes 2000; Parks 2004; Reingold, Van Ryzin, and Ronda 2001; Shang 2014; Vartanian 1997; Weinberg, Reagan, and Yankow 2004). Proximity to jobs is also associated with such outcomes (Allard and Danziger 2003; Howell-Moroney 2005; Thompson 1997; Weinberg, Reagan, and Yankow 2004). Greater neighborhood job referral opportunities are associated with greater employment, work hours, and earnings (Bayer, Ross, and Topa 2004). However, Bania, Coulton, and Leete (2003) find that neither neighborhood poverty nor job proximity is associated with employment and earnings among welfare recipients in one Ohio county, and Gurmu, Ihlanfeldt, and Smith (2008) find that geographic location is not associated with employment among Atlanta welfare recipients. Lens and Gabbe (2017) using the MTO data (discussed further below) find no relationship between neighborhood employment access and economic outcomes. Although most studies find links between adult exposure to disadvantaged neighborhoods and adult labor market outcomes, their research designs typically rely on strong assumptions about selection on observables (regression with controls) or selection on time-invariant characteristics (fixed effects), which make it difficult to dismiss selection bias as an alternative explanation.

Another set of studies has used quasi-experimental methods to address selection. A particularly influential study in this literature given its research design was Rosenbaum and Popkin’s (1991) quasi-experimental analysis of a survey of female household heads in Chicago’s Gautreaux Program (see also Rosenbaum [1995] and Rubinowitz and Rosenbaum [2000]). The Gautreaux intervention stemmed from litigation against the Chicago Housing Authority (CHA), which resulted in an agreement by the CHA to relocate African-American residents of public housing projects to new apartments located in either other parts of the city of Chicago or in low-poverty, predominantly white areas of the Chicago suburbs. Rosenbaum and Popkin found substantially higher employment rates (14 percentage points) for those who moved to the suburbs than for those who moved within the city.

The longer-term and more comprehensive follow-up analysis of Gautreaux Program impacts on low-income, black female household heads by Mendenhall, DeLuca, and Duncan (2006) finds little systematic impact on employment of moves to suburbs vs. the city, which was the initial contrast examined by Rosenbaum and Popkin. Their study also finds evidence that city versus suburban placement in Gautreaux was systematically related to the baseline attributes of the households, suggesting that their “natural experiment” study did not quite mimic the idealized design of a randomized experiment (see also Deluca et al. 2010; Votruba and Kling 2009). The long-term follow-up did find suggestive evidence of persistent positive employment effects of being initially placed in less-segregated (lower percent black) neighborhoods with greater neighborhood resources. Although other natural experiment strategies have been used to study the effects of childhood exposure to neighborhood disadvantage, such as the demolition of public housing (e.g. Jacob 2004; Galster and Santiago 2017), the work by Chyn (2018) is the only study we are aware of that applied that research design to examine the effects of adult exposure to neighborhood disadvantage on labor market outcomes. Chyn’s findings for adults are quite consistent with the experimental findings from MTO of large improvements in neighborhood quality but small and insignificant impacts on the employment and earnings of the displaced adults.

EXPERIMENTAL ESTIMATES FROM MOVING TO OPPORTUNITY

From 1994 to 1998 MTO enrolled 4,604 low-income public housing families living in high-poverty neighborhoods within five U.S. cities: Baltimore, Boston, Chicago, Los Angeles, and New York. Families were randomized into three groups: i) the low poverty voucher group, which received housing vouchers that subsidize private-market rents and could only be used in census tracts with 1990 poverty rates below 10 percent, and additional housing-mobility counseling; ii) the traditional voucher group, which received regular Section 8 housing vouchers without any MTO relocation constraint; and iii) a control group, which received no assistance through MTO. Some 48% of households assigned to the low-poverty voucher group and 63% of those assigned to the traditional voucher group moved using housing vouchers through MTO (the MTO “compliance rate”). We pay particular attention to the contrast between the low-poverty voucher and control groups, where the MTO-induced difference in neighborhood conditions (and hence power to detect neighborhood effects on adult outcomes) is greatest.

The MTO study sample is unusual within this literature partly because the baseline neighborhoods from which families were drawn were particularly disadvantaged. MTO enrolled families living in some of the most distressed public housing projects in the nation, including the Robert Taylor Homes in Chicago and Lafayette Courts in Baltimore. The average poverty rate in the census tracts in which MTO families lived at baseline was 53 percent. In the Chicago and Baltimore sites the baseline neighborhoods were almost entirely African American, while in the other three cities the neighborhoods were more evenly split between African-Americans and Hispanics. There were almost no non-Hispanic white families in these baseline neighborhoods, and hence virtually no white families in the MTO study sample. Given the high level of disadvantage of the baseline neighborhoods, it is not surprising that data from baseline surveys show that these families were quite economically disadvantaged when they applied for MTO.

In addition, MTO was a voluntary mobility program. The best available estimates from HUD housing records suggest that about one-quarter of eligible households that were offered the chance to sign up for the MTO lottery enrolled in it (Goering et al. 1999). These volunteers may differ systematically from other residents of the baseline neighborhoods, but that would not compromise the internal validity of MTO given the study’s experimental design. However, it could have implications for what other populations the experimental results generalize to (external validity), and therefore their comparability with results from other studies—an issue we return to below.

Follow-up data on MTO families show that the demonstration was successful in moving families into lower-poverty areas, particularly those families assigned to the low-poverty voucher group (see Table 2 and the discussion below). Over time the contrast between the neighborhood conditions of the low-poverty voucher and control groups attenuated, partly because the control group itself was moving into somewhat lower-poverty areas either on their own or as a result of other policy changes in the background (such as the demolitions of public housing projects carried out within the five MTO cities over this period under HUD’s HOPE VI program). Nevertheless, the effect of winning the lottery on duration-weighted tract poverty rates over the entire 10 to 15 year long-term MTO follow up was quite substantial—equal to 18 percentage points among those who moved with an MTO voucher.⁵ By comparison, over the course of the entire MTO study period control group families lived in census tracts that on average were 40 percent poor. In the national distribution of poverty rates, control group families lived in tracts at the 89^th percentile, on average. The effect of winning the lottery was a reduction of 21 percentiles, on average, among those who moved with an MTO voucher. Experimental group members who used their vouchers spent an average of over five more years in lower-poverty neighborhoods (census tracts with poverty rates below 25 percent) than they otherwise would have.

Table 2.

MTO Effects on the Characteristics of Participants’ Residential Census Tracts

	Control Mean	Experimental vs. Control
		ITT	TOT
Duration-weighted tract characteristics (over the 10- to 15- year follow-up period)
Tract share poor	0.400	−0.092	−0.188
		(0.006)	(0.012)
Tract share African-American	0.543	−0.034	−0.070
		(0.008)	(0.017)
Tract share minority	0.881	−0.062	−0.127
		(0.007)	(0.015)
Tract share receiving public assistance	0.187	−0.048	−0.099
		(0.004)	(0.008)
Tract share female-headed households	0.530	−0.074	−0.151
		(0.006)	(0.011)
Tract share unemployed	0.196	−0.039	−0.080
		(0.003)	(0.007)
Tract share residents under age 18	0.340	−0.024	−0.050
		(0.003)	(0.005)
Concentrated disadvantage index	1.855	−0.269	−0.551
		(0.021)	(0.042)
Tract share poor
1 year post-random assignment	0.482	−0.161	−0.330
		(0.008)	(0.015)
5 years post-random assignment	0.408	−0.102	−0.209
		(0.008)	(0.016)
10 years post-random assignment	0.339	−0.053	−0.108
		(0.007)	(0.015)
Number of years (over the 10- to 15-year follow-up period) in tracts with poverty rate less than…
10%	0.515	1.329	2.727
		(0.103)	(0.211)
25%	2.623	2.535	5.200
		(0.169)	(0.347)

Notes: This table presents the estimated effect of the MTO experimental (low-poverty voucher) treatment on a variety of neighborhood characteristics for female adults (N=2,550 with valid Census tract characteristics data). The concentrated disadvantage index is a weighted combination of census tract percent [i] in poverty, [ii] African-American, [iii] on welfare, [iv] unemployed, [v] female-headed family households, and [vi] under age 18, with loading factors developed using 2000 Census tracts in Chicago by Sampson, Sharkey, and Raudenbush (2008). Census tract characteristics are based on Census 2000 data. The tables includes both the intent-to-treat (ITT) and treatment-on-treated (TOT) estimates from an ordinary least squares regression where the outcome is regressed on the experimental (low-poverty voucher) flag as well as the standard set of MTO covariates and where the standard MTO weight is applied. Robust standard errors are in parentheses, and all effects are significant at the p<.01 level.

What does this imply about the magnitude of the MTO treatment “dosage”? On the one hand, by the standards of different social policy interventions that have been tried in the United States, MTO generates very large changes in social conditions for participating families. One year after randomization, the effect of receiving the housing voucher offer and subsequently using the voucher to move on tract poverty rates is about 2.8 standard deviations within the national census tract poverty-rate distribution. Over the entire MTO follow-up period the average family that moved with an experimental voucher lived in census tracts with poverty rates of about 20 percent, nearly 50 percent lower than that of the average control group family. As we will see below, only a small fraction of PSID families improved their neighborhood conditions by anything close to this amount.

MTO generated sizable changes in other measures of neighborhood economic disadvantage, and also led families to live in neighborhoods that were substantially safer, with seemingly higher levels of collective efficacy (Sampson, Raudenbush and Earls 1997), and with more social-network ties to more affluent neighborhoods, such as those with a college education (e.g. Sanbonmatsu et al. 2011). Despite these large impacts on neighborhood poverty, it is still possible that MTO created shorter-duration differences in neighborhood conditions compared with the variation in neighborhood conditions across individuals that is captured in observational datasets like the PSID.

Another possible way in which the MTO demonstration “dose” may be different from that of the PSID and other non-experimental datasets is that MTO families wound up living in lower-poverty neighborhoods that were still predominantly minority. The average control group family spent the MTO follow-up period in a census tract in which 88 percent of residents were members of a racial or ethnic minority group. While moving with a low-poverty voucher had a statistically significant effect in reducing tract share minority, equal to 12 percentage points, these families still lived in predominantly-minority neighborhoods (see also Clampet-Lundquist and Massey 2008; Sampson 2008). Put differently, the correlation between tract poverty and tract minority share for the MTO sample is quite different from what we see in national data like the PSID (see Appendix Table A1).

For MTO adults perhaps the most important puzzle is why observational studies often find evidence of gains in economic outcomes—employment, earnings, social-program participation, participation in job training or educational activities, etc.—but MTO does not. The absence of detectable MTO effects on the earnings or employment of those who moved during adulthood has been consistently documented in both the interim and long-term follow-ups of MTO using both survey and administrative data measures of economic outcomes (Orr et al. 2003; Sanbonmatsu et al. 2011; Ludwig et al. 2012).⁶ The same finding continues to hold in the longer-term follow-up of MTO participants using IRS data (Chetty, Hendren, and Katz 2016). There is evidence of MTO effects on important health outcomes such as depression, obesity and diabetes (Sanbonmatsu et al. 2011; Ludwig et al. 2011). But the discrepancy in findings for earnings remains a mystery, and a particularly important one given the special emphasis given to earnings in both social science and policy discussions.

HYPOTHESES

These puzzles motivate a number of hypotheses about the discrepancies between MTO findings and those of previous observational studies regarding neighborhood effects for adults on economic outcomes. We provide a brief description of each hypothesis and indicate how they will be tested.

Outcome Measurement:

MTO impacts on economic outcomes may differ substantially from effects found in observational studies because of the ways in which economic outcomes are measured. We will assess this with an examination of whether MTO experimental estimates differ from both MTO nonexperimental estimates and PSID estimates. Similar MTO experimental and nonexperimental estimates based on identical outcome measures would argue against this hypothesis.

Treatment Magnitude:

Differences may arise if changes in neighborhood poverty induced by the MTO intervention are smaller than the variation in neighborhood poverty used to estimate neighborhood effects in the PSID. If selecting and weighting the data to align neighborhood poverty comparisons in MTO and the PSID does not reduce discrepancies in estimates, then this hypothesis is unlikely to be correct.

Treatment Duration:

Differences in the duration of exposure to high-poverty neighborhoods in MTO and the PSID may explain differences in effects because, as we will see below, the MTO treatment groups tended to move back to somewhat higher poverty neighborhoods over time and the MTO control group experienced some improvements in neighborhood environments over the follow-up period. Weighting the data to align adult exposure to high poverty neighborhoods allows us to test this hypothesis. If such weighted analyses do not reduce the MTO-PSID discrepancy, this hypothesis is unlikely to be correct.

Non-linear effects:

Differences in neighborhood effects may arise because most nonexperimental studies of neighborhood effects assume a linear relationship between neighborhood poverty levels and outcomes. If the relationship is non-linear and different studies make comparisons across different ranges of neighborhood poverty, then different estimates will result. Again, selecting and weighting the data to align the range of neighborhood poverty across the different samples used to estimate neighborhood effects allows us to test this hypothesis. This hypothesis is unlikely to be correct if such weighting does not reduce the discrepancy between MTO and PSID estimates.

Treatment effect heterogeneity:

This hypothesis suggests that families with different characteristics have different capacities to take full advantage of moves to more advantaged neighborhoods. Because the MTO families are particularly disadvantaged relative to PSID families in high poverty neighborhoods (see Table 1 and our discussion below), we might expect smaller neighborhood effect estimates in MTO. By weighting the data to align observed family characteristics across the samples, we can examine whether effect heterogeneity related to those characteristics can account for the MTO-PSID discrepancies. If discrepancies persist even after family characteristics are aligned, that would suggest this hypothesis is incorrect. We caution, however, that we cannot examine effect heterogeneity by unobserved characteristics, including unobserved characteristics that may have impacted whether families volunteered for the MTO experiment in the first place.

Table 1.

Baseline Sample Characteristics and Neighborhood Poverty Exposures

	MTO Females			PSID Females
				No Weights			PSID Longitudinal Weights			PSID Cross-Sectional Weights
	Low-Poverty Voucher	Traditional Voucher	Control		Neighborhood Poverty	Neighborhood Poverty		Neighborhood Poverty	Neighborhood Poverty		Neighborhood Poverty	Neighborhood Poverty
	Group	Group	Group	All	< 25%	≥ 25%	All	< 25%	≥ 25%	All	< 25%	≥ 25%
Sample Size	1433	660	1114	4299	3722	577	4299	3722	577	4299	3722	577
Baseline Characteristics
Female head of household	0.930	0.957	0.940	0.324	0.290	0.551	0.345	0.332	0.492	0.271	0.259	0.441
Race & ethnicity
African-American Non-Hispanic	0.635	0.620	0.641	0.309	0.244	0.731	0.130	0.100	0.473	0.107	0.082	0.450
Hispanic	0.309	0.327	0.299	0.067	0.054	0.149	0.080	0.062	0.288	0.071	0.056	0.270
White	0.025	0.023	0.026	0.586	0.661	0.097	0.751	0.798	0.207	0.781	0.820	0.248
Age (mean in years)
Age at baseline	32.8	33.0	32.7	41.2	41.3	40.0	44.2	44.4	42.5	44.3	44.5	42.8
Age in 2009	45.5	46.0	45.5	53.3	53.5	52.1	56.3	56.5	54.6	56.5	56.6	54.9
Number of children (mean)	2.42	2.525	2.482	1.175	1.087	1.744	0.883	0.834	1.446	0.883	0.845	1.407
Graduated high school	0.382	0.349	0.362	0.792	0.827	0.563	0.812	0.837	0.529	0.829	0.849	0.550
Household Income
< 100% of Poverty Line	0.776	0.805	0.807	0.188	0.147	0.454	0.153	0.130	0.412	0.126	0.105	0.409
100–200% of Poverty Line	0.189	0.186	0.190	0.159	0.149	0.229	0.135	0.132	0.175	0.125	0.120	0.185
≥ 200% or Poverty Line	0.035	0.010	0.003	0.653	0.704	0.317	0.712	0.738	0.413	0.749	0.775	0.406
Working	0.261	0.255	0.232	0.659	0.680	0.525	0.657	0.670	0.504	0.653	0.664	0.498
Receiving AFDC/TANF	0.766	0.743	0.770	0.053	0.034	0.177	0.031	0.023	0.125	0.022	0.016	0.107
Region
North Central (Chicago)	0.204	0.213	0.207	0.253	0.260	0.215	0.266	0.272	0.199	0.233	0.238	0.175
Northeast (Boston/NYC)	0.430	0.443	0.435	0.152	0.158	0.113	0.202	0.205	0.169	0.206	0.209	0.171
South (Baltimore)	0.135	0.143	0.137	0.414	0.399	0.513	0.320	0.316	0.366	0.342	0.339	0.393
Baseline census tract
Share poor	0.494	0.497	0.502	0.143	0.115	0.325	0.120	0.103	0.319	0.114	0.100	0.312
Share African-American	0.556	0.568	0.563	0.225	0.173	0.563	0.122	0.100	0.375	0.111	0.093	0.357
Neighborhood Poverty Exposure 1998/1999 to 2008/2009
Mean	0.296	0.317	0.378	0.133	0.102	0.337	0.114	0.094	0.339	0.109	0.092	0.337
Median	0.281	0.311	0.378	0.105	0.091	0.317	0.090	0.083	0.318	0.086	0.081	0.318
25th Percentile	0.177	0.213	0.275	0.058	0.052	0.278	0.051	0.048	0.283	0.049	0.046	0.282
75th Percentile	0.403	0.409	0.477	0.183	0.145	0.379	0.152	0.131	0.383	0.144	0.129	0.377

Notes: The regression models controlled for age in 2009 as opposed to age at baseline (and also included a squared age term).

Neighborhood measures:

MTO families moved to lower poverty neighborhoods, which, given the high correlation of poverty with other components of composite neighborhood measures like the widely-used “neighborhood disadvantage index” of Sampson, Sharkey and Raudenbush (2008), we typically interpret as a summary proxy for neighborhood SES. But if these elements do not co-vary the same way within the MTO sample as national samples, we might expect to see few effects from moving to lower poverty neighborhoods that are still disadvantaged on other dimensions. To address this hypothesis, we weight the data to align not only neighborhood poverty but also the neighborhood disadvantage index. Since, as noted above, we already know the MTO intervention was less successful at promoting one specific dimension of that index, racial integration, which could be particularly important for outcomes (Massey and Denton, 1993, Sampson 2008), we pay special attention to that in our analysis. If aligning samples on these other measures of neighborhood SES does not reduce discrepancies between MTO and PSID, then this hypothesis is unlikely to be correct.

Residential Mobility:

To improve their neighborhood environments, MTO families also had to move, and residential mobility can itself be harmful if it removes individuals from social networks that support employment or provide job leads. Because the MTO research design cannot separate neighborhood effects from residential mobility effects, null effects on adult economic outcomes in MTO could be due to offsetting positive neighborhood effects and negative residential mobility effects.⁷ To test this hypothesis, we weight the data to align the number of residential moves between census tracts and the distances moved across the MTO and PSID samples. If doing so does not reduce MTO-PSID discrepancies, then this hypothesis is unlikely to be correct.

Selection bias:

Because neighborhood effects in MTO are estimated by leveraging random variation in neighborhood poverty generated by the random assignment of housing vouchers, it is unlikely that any effects estimated in MTO are due to unobserved differences between families living in different types of neighborhoods.⁸ In contrast, observational studies rely on observing and conditioning on all pre-existing differences between families living in high and low poverty neighborhoods, a potentially untenable assumption. Because this is a hypothesis about unobserved variables, we cannot directly test this hypothesis. That is, it is a residual explanation that is rendered more plausible by the failure of the other hypotheses above.

DATA AND METHODS

The goal of our analysis is to examine discrepancies in findings between MTO and previous studies. Such an effort to understand what estimation approaches or sample restrictions let us reproduce an experimental impact estimate using non-experimental comparisons has become known as a “within-study comparison.”⁹ We use two data sources: the survey of MTO participants conducted as part of the final follow-up of MTO households, which measured outcomes 10–15 years after baseline (Sanbonmatsu et al. 2011); and the Panel Study of Income Dynamics (PSID 2014), one of the most important and widely-used social science panel datasets in the literature on neighborhood effects. Our choice of the PSID reflects prior knowledge about requirements for effective within-study comparisons, including the availability of pre-treatment measures of the outcomes, the capacity to measure treatments and outcomes in the same way, and correspondence of geographic scope (Cook, Shadish, and Wong 2008). Although the correspondence between the five MTO cities and the nationwide scope of the PSID is far from perfect, the PSID provides the closest possible geographic correspondence among candidate studies that otherwise meet our need for neighborhood measurements, long-term follow-up, and measurement similarity.

MTO families completed a baseline survey at the time they enrolled in the MTO program between 1994 and 1998. Between 2008 and 2010, the Institute for Social Research (ISR) at the University of Michigan collected in-person survey data on 3,273 adult participants in MTO. ISR achieved an effective response rate of 90% for the adults, with similar response rates across treatment groups. In this paper, we focus exclusively on female adults because females comprised about 98% of the MTO adult sample.

The PSID is a longitudinal household survey which began in 1968 with a nationally representative sample of about 18,000 people—and by 2009 included over 24,000 people. In addition to having been used in numerous previous studies of neighborhood effects, the PSID also has the advantage for our purposes of including outcome measures and time periods that are comparable to MTO. We use 1997 as our “baseline” year of PSID data, examine neighborhood exposures over the 1999–2009 period, and look at adult outcomes for 2009. Restricting the PSID data to females who were household heads or spouses in both 2009 and 1997 yields a sample of 4,299 observations with neighborhood and outcome information.¹⁰ We link public-use PSID data on baseline characteristics and subsequent outcomes to the PSID restricted use data that includes residential census tract. The PSID survey conditional response rates in 2009 were about 95% for the main sample and 90% for the Latino/Immigrant refresher sample, although of course those families still in the PSID by the start of our study period and also eligible for the 2009 survey frame are a selected sample.¹¹

For our analysis of neighborhood characteristics, we link MTO address data from self-reports and passive tracking sources to census tract characteristics from the 2000 census. For both the MTO and PSID samples, we use biannual address data for the 1998/99 through 2008/09 time period. For PSID, follow-up surveys occurred biannually beginning in 1997, and we therefore use address data from odd years from 1999 to 2009. The MTO long-term survey included an address history where adults confirmed or reported addresses from their randomization date through their address as of the survey. To construct neighborhood measures parallel to the PSID, MTO address information was limited to biannual years between either 1998 and 2008 or 1999 and 2009.¹²

A key advantage of the randomized experimental design of MTO is that the low-poverty voucher and control groups have background characteristics that are very similar on average, with respect to both observed variables (which we can check) and unobserved variables (given the statistical implications of randomization). These two very similar groups then go on to experience very different neighborhood environments over the course of the post-program period (see Table 1 below). In contrast, in the PSID we can construct sub-samples of people that experience “MTO-like” differences in neighborhood poverty rates, but these high- and low-neighborhood-poverty groups differ substantially from one another with respect to their observed baseline characteristics (and hence potentially with respect to unobserved characteristics as well).

A key challenge in using the PSID for present purposes is that when we define PSID sub-samples that experience contrasts in neighborhood poverty rates that are similar to what we see in the MTO experiment, we end up with a set of PSID respondents whose background characteristics are not nearly as disadvantaged as those for the MTO study sample. This pattern motivates the concerns embedded in our “treatment heterogeneity” hypothesis—the MTO sample is very disadvantaged even compared to samples of people living in high poverty neighborhoods drawn from a nationally representative sample. If, instead, we try to identify a sub-sample of PSID respondents whose own level of household disadvantage is similar to that of the MTO study sample, the PSID group’s neighborhood poverty rates are not nearly as high as what we see among the MTO control group. In some sense this should not be surprising, since MTO limited eligibility to people living in public housing projects with poverty rates above 40 percent, and thereby selects only those households with disadvantaged observable characteristics who were initially living in very high-poverty areas. Whatever the underlying cause, this pattern helps to motivate our treatment magnitude and nonlinearity hypotheses. To overcome these challenges in creating comparable samples in MTO and PSID, we use weighting as described further below.

In our main models, we use the average neighborhood poverty over the follow-up period to construct our nonexperimental “treatment” and “control” groups. We define the groups such that the difference in poverty between the two groups is similar to the effect of the MTO intervention on neighborhood poverty. In addition to using average poverty, we perform analyses using other measures of neighborhood disadvantage including the duration of time spent in low poverty neighborhoods and the “concentrated disadvantage” index commonly used in neighborhood effects studies (e.g. Sampson, Sharkey and Raudenbush 2008).¹³

A final issue is our choice of outcome measures. The neighborhood effects literature draws on a large number of different data sources that include different specific measures of adult economic self-sufficiency. One common measure is current employment status, which captures the idea that job opportunities themselves may vary across neighborhoods. In principle one could look at current earnings instead, which creates the possibility of also incorporating information about the quality of jobs. However, relative to a binary employment indicator, earnings are harder for subjects to report and more prone to outlier values, so earnings could introduce not just new signal but also additional noise.

Our approach here is essentially to focus on both measures and use the composite “outcome indices” that have been used in many MTO analyses (Kling, Liebman, and Katz 2007; Ludwig et al. 2012, 2013). The use of a single index rather than all of its components reduces the risk of “false positives” from carrying out a large number of different hypothesis tests, improves measurement validity and reliability, and can also reduce the risk of “false negatives” by improving statistical power to detect effects (Kling, Liebman, and Katz 2007). This strategy views the index as a summary of information on self-sufficiency from multiple constructs rather than as a measure of a single latent construct. Because we are limited to variables measured in both MTO and PSID, our economic self-sufficiency index includes individual earnings (in 2009 dollars) and an indicator for current employment. The index is an average of these two measures, both standardized using the PSID female sample into Z-score form (subtracting off the PSID mean and dividing by the PSID standard deviation). It has a Cronbach’s alpha reliability of 0.70 for the MTO data and 0.77 for the PSID data.¹⁴ In the MTO data, the two-item index used here has a correlation of 0.91 with the index used in prior MTO studies. The appendix shows results separately for the individual measures of earnings and employment (Tables A8 and A9). Earnings for the PSID sample were top-coded at the 95th percentile, and a small number of outlier values were also top-coded in the MTO data. The results are qualitatively similar for each of the components of our labor market index when analyzed separately.

Statistical Models

We begin our analysis with a careful examination of how participation in MTO affected future exposure to disadvantaged neighborhoods. These estimates provide important background for understanding any neighborhood effects estimated with MTO, as they elucidate the changes in neighborhood context created by the MTO experiment, a key question in the debate over discrepancies between experimental and non-experimental studies of neighborhood effects on adults. We follow prior analyses of the MTO data by estimating MTO’s experimental impact in a multi-variate regression framework (Sanbonmatsu et al. 2011). We begin with MTO’s impact on neighborhood conditions over the follow-up period. Let D be an indicator variable for use of a voucher to move through the MTO program, or treatment compliance. Let Z be an indicator for being eligible for an MTO program subsidy, or randomized treatment group assignment. Let subsidy use be a function of a set of observed characteristics from the MTO baseline survey either known prior to randomization or retrospectively reported as existing prior to randomization (X) and other factors ε₁, as in (1).

$$\text{D}={\text{Z}\pi }_1+{\text{X}\beta }_1+{\epsilon }_1$$ (1)

The “Intention-To-Treat effect” (ITT) is captured by the estimate of the coefficient π₂ in a regression of some neighborhood measure (Y) on an indicator for assignment (Z) to a treatment group as in (2).

$$\text{Y}={\text{Z}\pi }_2+{\text{X}\beta }_2+{\epsilon }_2$$ (2)

We condition on baseline characteristics (X) to improve the precision of our estimates by accounting for residual variation in pre-existing characteristics.

This ITT parameter is an average of the causal effects of MTO randomization on future exposure to neighborhood disadvantage for those who do and do not take-up the “treatment” (that is, to relocate with a MTO voucher). Therefore, ITT estimates will understate the effects of actually moving to a lower-poverty neighborhood after baseline given that not all families “complied” with their MTO treatment assignment. Specifically 49 percent of families randomly assigned to the experimental group relocated to a low-poverty neighborhood through MTO, while 62 percent of those assigned to the regular Section 8 group relocated with a voucher through MTO.¹⁵ Note however that ITT estimates are not biased by self-selection in MTO treatment take-up within the treatment groups, because the control group is compared with all families assigned to a treatment group whether or not the latter decide to accept the invitation to participate in the voucher program (Ludwig et al. 2008).¹⁶

We then use the randomized treatment assignment as an instrumental variable to calculate the effect of moving with an MTO voucher on future neighborhood context, or what is commonly called “effect of the Treatment on the Treated” (TOT). Under assumptions that (a) treatment group assignment is random, (b) the Control group is prohibited from receiving MTO vouchers, and (c) the effect on outcomes of treatment assignment works entirely through making a subsidized move through MTO, TOT is equal to π₂/π₁, or ITT divided by the proportion receiving the treatment.¹⁷ The TOT provides an estimate of the effect of an MTO induced move among those who actually moved. We show ITT and TOT estimates on various neighborhood characteristics measured between the MTO baseline and the final follow-up.

Our analysis of economic self-sufficiency first estimates the MTO experimental TOT using the self-sufficiency scale at the end of long-term follow-up as the outcome. We then compare these MTO experimental estimates to neighborhood effects estimates from the PSID.¹⁸ If we let P represent some binary measure of neighborhood conditions, such as low poverty (1) and high poverty (0), we use the PSID to estimate the standard regression as in equation (3). We construct PSID high-poverty and low-poverty groups so the size of the difference in tract poverty rate is similar to the MTO TOT effect on average tract poverty rates over the 11 years leading up to and including the MTO follow-up year.¹⁹

$$\text{Y}={\text{P}\pi }_3+{\text{X}\beta }_3+{\epsilon }_3$$ (3)

The key scientific question of interest for our analysis is whether the MTO TOT estimates are distinguishable from estimates obtained by the PSID; that is, whether π₂/π₁= π_3.

We should note that our estimates for both parameters can be somewhat imprecise, given limits on the sample sizes available in both of our datasets—particularly when we weight samples to align them on baseline characteristics and neighborhood contrasts.²⁰ As a result, the confidence intervals around our test of the contrast between π₂/π₁ and π₃ can be sizable. A different way to think about comparing the MTO and PSID estimates would be to ask whether we would draw different inferences about the existence of neighborhood effects if we had relied on the MTO study rather than the PSID. In other words, are there situations where we would reject the null hypothesis that π₃=0 but would not reject the null hypothesis that π₂/π₁ =0, or vice versa? More generally we tend to focus on differences in point estimates even though (particularly when we use weights, discussed below) few of them would pass conventional statistical significance tests at p<.05.

To explore the question of whether different samples and treatment heterogeneity can help explain the difference in the MTO versus PSID findings, we also replicate the observational estimates in the PSID using the MTO data by treating the MTO data as non-experimental data. That is, we essentially discard the experimentally-induced variation in neighborhood conditions from the MTO experimental design and instead use only within-randomized-group variation in neighborhood conditions, which will be purely observational (non-experimental). For example, we use the MTO control group to estimate an equation like (4):

$$\text{Y}={\text{P}\pi }_4+{ \text{X}\beta }_4+{\epsilon }_4\hspace{0.17em}\text{for}\hspace{0.17em}\text{all}\hspace{0.17em}\text{observations}\hspace{0.17em}\text{with}\hspace{0.17em}\text{Z}=0$$ (4)

In practice we pool data from both the traditional voucher and control groups in estimating equation (4) to improve sample size and hence statistical power, adding an indicator for the traditional voucher group to the model as a control variable. Even with data from two MTO groups included in the regression, with the inclusion of the traditional voucher indicator all of the variation that is used to estimate the relationship between neighborhood poverty (P) and the outcome (Y) in equation (4) will be within-randomly-assigned-MTO group (that is, “non-experimental”). Generating non-experimental estimates using the MTO data also helps us determine whether something else about the MTO study, such as the way the data were collected or how the outcome measures were constructed, could also explain differences in results with datasets like the PSID.

We explore the degree to which treatment heterogeneity can explain differences in results between MTO and PSID by weighting the data so that the sample characteristics with respect to basic baseline attributes (Xs) are similar across samples. We weight the various groups to look similar to the PSID respondents who lived in high poverty neighborhoods (on average greater than 25% poor). Prior to weighting the different subgroups to look like the PSID high poverty group, we first trim the samples to exclude observations with little overlap with the MTO sample on basic demographics such as age, race, education, and being above or below the poverty line.²¹ We exclude observations with propensity scores greater than 0.9 from a model predicting whether the respondent is in the PSID vs. MTO low-poverty voucher sample using basic demographics. By using inverse probability weighting together with regression adjustment we are following in the spirit of the double-robust analysis recommended by Imbens (2015).

Finally, we examine how different types of neighborhood exposures, nonlinearities in neighborhood effects, or residential moves might explain differences between the MTO experimental estimates and those from PSID. We re-estimate equations (3) and (4) in different ways. First, we can construct our non-experimental “treatment” and “control” groups to both maintain the same contrast or “gap” in average poverty and the same levels. This allows us to examine if, say, a 17-percentage point reduction in average neighborhood poverty has the same effect whether the contrast is between 15% and 32% average poverty or between 20% and 37% poverty. See Appendix A for more details on the methods used to construct these contrasts. Second, we can use weighting to simultaneously bring the samples closer together on demographics and make the treatment magnitude (or “dosage”) on different neighborhood characteristics more similar to the MTO experimental analysis. We can match the treatment magnitude with respect to poverty, the concentrated disadvantage index, and duration of exposure to high poverty neighborhoods. To examine hypotheses related to the duration of the exposure and mobility, we can also bring the nonexperimental analyses into alignment with the experimental treatment magnitude on years in low poverty neighborhoods and number of moves across census tracts.

We use propensity models of the trimmed data to weight each lower-poverty group (<25% poverty) to look like the PSID higher-poverty group (> 25% poverty) on basic demographics.²² Weights are defined as p/(1-p) from a propensity model using a boosted Classification and Regression Tree implemented using the Toolkit for Weighting and Analysis of Nonequivalent Groups (Griffin et al. 2014). Using the weighted lower-poverty group, we then weight each higher-poverty group to look like the corresponding lower-poverty group on basic demographics and at the same time to have average poverty levels that are higher than the lower-poverty group’s average poverty rate by the magnitude of the TOT (using the weighted and trimmed data). The result of the weighting is to construct high and low neighborhood poverty groups that are similar on basic demographics and approximate the MTO treatment magnitude as well as the range of neighborhood poverty or disadvantage in which the gap occurs. See Appendix A for details on the propensity score trimming and weighting.

RESULTS

Comparison of MTO and PSID Samples

We begin by presenting descriptive analyses of the MTO and PSID samples we analyze. Recall that the PSID female sample we use is a subset of the larger PSID designed to be most comparable to MTO adults. These results serve to document the similarities and differences between the samples and to demonstrate the initial plausibility of some of the hypotheses will we analyze more rigorously below. Table 1 shows MTO and PSID sample characteristics at baseline and subsequent exposure to neighborhood poverty over the ten-year follow-up period. We show descriptive statistics for the PSID calculated without sample weights (which is how we carry out our regression analysis), and for completeness also show these means calculated using the PSID’s longitudinal sampling weights and cross-section sampling weights. Note that although the MTO baseline characteristics are stratified by the three arms of the MTO experiment (low-poverty voucher, traditional voucher, and control group), the average characteristics of these three groups are almost identical, which is a direct result of the MTO randomization.

One important result from Table 1 is that the MTO sample is considerably more disadvantaged than the PSID sample.²³ MTO respondents are more likely to be single mothers, to have not graduated from high school, to be not working, to have incomes below the poverty line, to be on welfare, and to have more children. MTO respondents are also living at baseline in poorer neighborhoods with more African-American residents than PSID respondents measured in the same year. These baseline differences motivate the treatment effect heterogeneity hypothesis, which argues that MTO respondents are so much more disadvantaged at baseline that they may be ill-prepared to take advantage of the features of low-poverty neighborhoods that could lead to improvements in employment or earnings.²⁴

Note also that there are important baseline differences within the PSID sample between respondents who will experience low poverty neighborhoods in the future (here, defined as mean yearly poverty rate less than 25% poor) and those who will experience high poverty neighborhoods in the future (greater than 25% poor). The latter group is more likely to be single mothers, has more children, is less likely to have graduated from high school, has lower incomes, and is more likely to be on welfare and less likely to be working. They are also living in neighborhoods with a higher proportion African American population at baseline. It is these observed differences that lend plausibility to the selection bias hypothesis. If observed differences between PSID mothers who experience low and high poverty neighborhoods are stark, then there are potentially unobserved differences as well. The question is whether such unobserved differences are both sufficiently independent of the observed covariates (control variables) and sufficiently important for future employment and earnings to impart serious biases to causal effects estimated from “selection on observables” methods like regression or matching.

The bottom section of Table 1 documents exposure to neighborhood poverty during the follow-up period. Focusing first on the differences across the randomized MTO groups in the left side of the table, we see that there are differences in neighborhood poverty, with the control group experiencing the poorest neighborhoods and the low-poverty voucher group experiencing the least poor neighborhoods. Note that these groups are based on the randomization, not on the actual neighborhoods experienced later, so they demonstrate that the randomization did indeed produce variation in neighborhood poverty. However, the variation looks relatively small, especially in comparison with the naturally occurring variation within the PSID sample. Note also that, in comparison to the PSID, all the MTO randomization groups have large numbers of mothers who experience very high poverty neighborhoods (the 75^th percentile for all groups is above 40% poor).

Numbers like these lend some plausibility to the treatment magnitude hypothesis, as they suggest that the change in neighborhood poverty induced by MTO may not be large enough to affect outcomes appreciably. Furthermore, they are also suggestive of the non-linear effects hypothesis because they indicate that changes in neighborhood poverty in the MTO sample occurred at higher levels of neighborhood poverty than the naturally occurring variation in neighborhood poverty in the PSID, where there are large numbers of PSID mothers in very low poverty neighborhoods (less than 10% poor), thus providing the opportunity to compare mothers who experience very low and very high poverty neighborhoods.

Table 2 provides more details on the changes in neighborhood conditions created by the MTO intervention by examining multiple neighborhood measures and how exposure to neighborhood poverty evolved over time. The mean duration-weighted neighborhood poverty rate in the control group was 40 percent.²⁵ In the experimental group, the mean neighborhood poverty rate was 9.2 percentage points lower (the ITT effect of the MTO experiment on mean neighborhood poverty), but when we use the randomization as an instrument (and isolate impacts on families initially moving to low-poverty neighborhoods), we see that there was an 18.8 percentage point drop, on average, among those who moved with a voucher (the TOT effect on mean neighborhood poverty).

We also examined the degree to which MTO changed other neighborhood characteristics because the “neighborhood measure” hypothesis argues that other measures of disadvantage that might be important for our outcomes were less affected by the MTO intervention. This appears to be the case, especially for percent African-American and percent minority. Percent African-American was reduced from 54.3 percent in the control group by seven percentage points while percent minority was reduced from 88.1 percent by 12.7 percentage points, meaning the average MTO mother in the experimental group lived in a neighborhood that was almost majority African-American and decidedly majority minority.

The concentrated disadvantage index shows similar results, falling from 1.855 standard deviations above the national mean by only about half a standard deviation, meaning low-poverty voucher group mothers still lived in neighborhoods far above the national mean on the concentrated disadvantage index after their initial program-related moves. This is consistent with past findings that MTO improved neighborhood poverty far more than neighborhood racial composition, in part due to the challenges of defying the powerful social forces that enforce neighborhood racial segregation (Sampson 2008, 2012). Reductions in neighborhood unemployment and public assistance receipt are similar to those for poverty, while reductions in percent youth and female headed households are similar to those for racial composition.²⁶ Below we will align the MTO and PSID on the concentrated disadvantage index to examine the neighborhood measure hypothesis directly.

The remainder of Table 2 focuses on how the neighborhood poverty rate evolved over time in MTO, showing mean neighborhood poverty rates at 1, 5, and 10 years following randomization and the number of years in tracts with poverty rates below various thresholds. Over time, the MTO induced reduction in neighborhood poverty fell from 33 percentage points one year after randomization to 10.8 percentage points ten years after randomization. This reduction reflects in part lower neighborhood poverty rates in the control group and in part higher poverty rates in the experimental group over time. Concordantly, the mean number of years in the lowest poverty neighborhoods (less than 10% poor) was increased by MTO by 2.7 years, from half a year (0.5) in the control group. The increase in years in neighborhoods with poverty below 25% was larger at 5.2 years, up from 2.6 in the control group. These results suggest the plausibility of the treatment duration hypothesis, which argues that the reduction in neighborhood poverty created by MTO could have been too short-lived for many participants to substantially improve adult economic outcomes (although little is known in practice about how much time is required in such a neighborhood to change outcomes). They also suggest a potential role for the non-linear effects hypothesis because they show the reductions in neighborhood poverty that did occur still left many MTO participants whose neighborhood environments did improve in neighborhoods where poverty rates were still fairly high, at 25 percent. (Appendix Table A10 provides parallel results using the percentile rank for each variable.)

Effects on Economic Self-Sufficiency

We now turn to the hypotheses that might explain discrepant results between the MTO experimental estimates and PSID observational estimates of effects of neighborhood poverty on adult economic outcomes (as measured by the economic index described above). Results are presented in Table 3 (parallel results for separate earnings and employment outcome variables are presented in Appendix Tables A8 and A9).²⁷ There is one row for each estimate of the effect of living on average in low poverty neighborhoods rather than high poverty neighborhoods during the follow-up period. That is, the independent variable is a binary indicator for low (1) vs. high (0) mean neighborhood poverty during follow-up. Each row shows the mean neighborhood poverty rate in the two groups, the difference between the neighborhood poverty rates, the mean outcome in the high neighborhood poverty group, and the estimated effect along with its standard error, p-value, and the sample size.²⁸ Positive effects indicate improvements in economic outcomes from living in less poor neighborhoods, which corresponds to the analysis framework in MTO of assessing the effects of an intervention to improve the neighborhood environments of participants.

Table 3.

Comparing experimental and nonexperimental estimates of the effects of low poverty neighborhoods on an economic index

	Census Tract Poverty Rates			Economic Index for High-Poverty	Estimated Effect of Low-Poverty Nbhd on Economic Index
	Low-Poverty Group Mean	High-Poverty Group Mean	Poverty Rate Diff.		Coeff. (SE)	P-Value	N
I. Prior to trimming and reweighting
A. Experimental estimate
(A1) MTO exp TOT (std wgts)	0.199	0.368	−0.169	−0.229	−0.012	0.863	2543
					(0.067)
B. Nonexperimental estimates (unweighted)
(B1) PSID nonexp estimate (< 25% poverty)	0.102	0.337	−0.235	−0.262	0.101	0.021	4299
					(0.044)
(B2) MTO nonexp est (< 25% poverty)	0.177	0.413	−0.236	−0.369	0.096	0.036	1770
					(0.046)
C. Nonexp estimate matching gap (unweighted)
(C1) PSID nonexp same poverty gap	0.148	0.319	−0.171	−0.250	0.107	0.018	2512
					(0.045)
(C2) Within study (MTO) nonexp same pov gap	0.199	0.370	−0.171	−0.368	0.053	0.295	1393
					(0.050)
II. Trimming and reweighting to make samples more demographically similar
D. Reweighted quasi-experimental estimate
(D1) MTO exp TOT (rewgtd)	0.200	0.369	−0.169	−0.111	−0.131	0.276	2532
					(0.120)
E. Reweighted non-experimental estimate matching poverty gap/level
(E1) PSID nonexp same poverty gap/level (rewgtd)	0.171	0.338	−0.167	−0.253	0.112	0.103	850
(< 25% poverty)					(0.068)
(E2) MTO nonexp same poverty gap/level (rewgtd)	0.183	0.353	−0.170	−0.256	0.069	0.316	1738
(< 25% poverty)					(0.069)
F. Reweighted non-experimental matching poverty gap/level & disadv gap
(F1) PSID nonexp same pov & disadv. gap/level	0.171	0.338	−0.166	−0.280	0.121	0.095	850
					(0.073)
(F2) MTO nonexp same pov & disadv. gap/level	0.183	0.353	−0.170	−0.249	0.091	0.188	1738
					(0.068)
G. Reweighted non-experimental matching poverty gap/level & disadv/years gap
(G1) PSID nonexp same poverty, disadv., and years	0.171	0.336	−0.165	−0.244	0.083	0.200	850
					(0.065)
(G2) MTO nonexp same poverty, disadv., and years	0.183	0.354	−0.171	−0.332	0.120	0.120	1738
					(0.077)
H. Reweighted non-experimental matching poverty gap/level & disadv/# of moves
(H1) PSID nonexp same poverty, disadv., and moves	0.171	0.337	−0.166	−0.247	0.079	0.222	850
					(0.065)
(H2) MTO nonexp same poverty, disadv., and moves	0.183	0.366	−0.183	−0.314	0.093	0.112	1738
					(0.058)

Notes: This table presents the estimated effects of living in low-poverty neighborhoods on an index of economic outcomes for female adults. The economic index is an average z-score of employment and earnings measures (standardized against the PSID female sample). The MTO TOT (row A1) compares the experimental (low poverty voucher) and control groups using a regression (with the standard MTO weights) and scaling up the intent-to-treat estimate by the inverse of the compliance rate. This table’s measure of neighborhood poverty exposure is constructed for the MTO sample to be parallel to the PSID so it uses only biannual addresses during the present study’s follow-up time. In Row A1, the Low-Poverty Group Mean is the experimental complier mean, the Poverty Rate Diff is the TOT for the effect on the mean neighborhood poverty rate during follow-up, and the High-Poverty Group Mean is the implied experimental control complier mean (the experimental complier mean minus the TOT). The non-experimental treatment estimates are generated by first creating high- and low- neighborhood poverty subsamples. Rows B1 and B2 divide the sample into high and low groups at 25% poverty. In rows C1 and C2, the high and low poverty groups have been constructed such that the difference between them in average poverty rate is similar to the size of the MTO experimental TOT of about 17 percentage points. The nonexperimental analyses regress the outcome on an indicator for the low-poverty group as well as the best-available controls for each sample. The MTO non-experimental treatment estimates use only the Section 8 (traditional voucher) and control groups and the model includes an indicator for the Section 8 group. In Panel I, the nonexperimental estimates are unweighted. In Panel II, the samples have been made more similar to each other on basic demographics by trimming and reweighting the data.

Here is some additional detail about how we construct the high and low nonexperimental groups for rows C1 and C2 in Panel I. The sample is initially split at 25% average neighborhood poverty. The MTO average neighborhood poverty (for the combined controls and traditional voucher group) is about 17.7% among those with average neighborhood poverty rates below 25 percent. To produce a contrast comparable to the MTO TOT, we create a “high poverty” group starting with respondents near 25 percent neighborhood poverty and incrementally expanding (by 0.5 percentage points) to encompass higher neighborhood poverty rates until we achieve a contrast of about 17 percentage points between the high and low poverty groups.

Row A1 shows the TOT estimate that compares the low-poverty voucher group with the MTO control group (while also conditioning on observed covariates).²⁹ Consistent with prior results on employment and earnings from MTO, this effect is very close to zero (slightly negative in fact) and not statistically significant. Row B1 shows estimates from the PSID data comparing low and high neighborhood poverty groups via a regression model controlling for covariates (see Appendix Tables A2 for covariates available in MTO and PSID; covariates used in the main text are dataset-specific covariates, but results are similar when covariates common to both datasets are used as shown in Appendix Table A7).

The PSID estimate is positive and statistically significant, with an estimated improvement in the economic index of about a tenth of a standard deviation. (Note that the mean outcome in the high neighborhood poverty group is very similar in the two samples, −0.229 vs. −0.262, so the difference in effect estimates is due to differences in outcomes between the PSID and MTO low-poverty groups). Researchers looking at these estimates would draw different conclusions from the MTO experimental estimates versus the PSID observational estimate about whether neighborhood poverty in adulthood matters for labor market outcomes.³⁰ The differences between these two estimates (A1 and B1) for the full sample and for key subgroups (discussed below) are the puzzles this paper attempts to address.

Row B2 presents the “within-study comparison” estimate from MTO. It uses the MTO traditional voucher group and the control group and analyzes the data as if it were observational data (ignoring the random assignment and simply comparing MTO respondents who lived in low and high poverty neighborhoods during follow-up). These “nonexperimental” MTO estimates are valuable because they rely on the same assumption as the PSID regarding selection on observables while preserving congruence with the MTO experimental estimates on other aspects of study design, including variable measurement, geography, and baseline characteristics. This approach produces an almost identical treatment effect estimate as the PSID, corresponding to a 0.096 standard deviation improvement in the economic index. We interpret the close similarity between the PSID and MTO nonexperimental estimates coupled with their divergence from the MTO experimental estimates to mean that the difference between the PSID and MTO experimental estimates cannot be due to differences in outcome measures.³¹

Note that the neighborhood poverty rate differences on which the effects are estimated are smaller in the MTO experimental sample than the PSID sample (16.9 percentage points vs. 23.5 percentage points). The difference in the MTO nonexperimental sample is similar at 23.6 percentage points. Rows C1 and C2 present PSID and MTO nonexperimental estimates from samples selected to achieve a gap similar to the MTO experimental sample, a step in the direction of creating a sort of “common support” on neighborhood conditions. (As is evident from the group mean neighborhood poverty rates, this was accomplished in the PSID sample by selecting low neighborhood poverty cases with higher neighborhood poverty rates and accomplished in the MTO nonexperimental sample by selecting both low neighborhood poverty cases with slightly higher neighborhood poverty and high neighborhood poverty cases with slightly lower neighborhood poverty.³²) Doing so does not entirely remove the discrepancy between the MTO experimental estimates and the two nonexperimental estimates, but it does make the discrepancy between the MTO experimental and MTO nonexperimental smaller than the MTO experimental-PSID discrepancy (and results in neighborhood effect estimates that are no longer significantly different from zero). We interpret these findings to mean that the treatment magnitude hypothesis cannot fully explain the discrepancy between the magnitude of the MTO and PSID estimates (particularly since the discrepancy re-emerges when we weight the data below). Because the MTO samples are also estimating neighborhood effects in almost the exact same range of neighborhood poverty, these results also suggest that the non-linear effects hypothesis is incorrect.

The second panel of Table 3 presents estimates from trimming and weighting the samples. We weight the MTO treatment groups from the experiment and the MTO and PSID high and low poverty groups to look like a common reference group: the PSID high poverty group. We choose the PSID high-poverty group as our common reference group, rather than choosing the MTO sample as the reference group, because as noted above the MTO sample is unusually disadvantaged and so is less nationally representative. As also noted above, we do some trimming to avoid problems that arise with people with propensity scores that are very close to 0 or 1, which can yield extreme weights. This process allows us to align the baseline characteristics in the PSID and MTO samples, thereby addressing the effect heterogeneity hypothesis. Furthermore, it will allow us to align the samples on other neighborhood characteristics and the number of residential moves in order to examine additional hypotheses. Appendix Tables A3 and A4 show the balance on baseline covariates both before and after trimming and weighting. The weighting procedure has resulted in good balance on key covariates across samples. The weighting procedure has also increased the standard errors on all effect estimates substantially (as weights always do).

Row D1 of Table 3 shows reweighted MTO experimental estimates and rows E1 and E2 show weighted PSID and MTO nonexperimental estimates, respectively. The neighborhood poverty gaps and levels continue to be closely aligned between the three samples, but the effect estimates are now even more different.³³ The weighting increased somewhat the PSID and MTO nonexperimental effect estimates, bringing them back closer to the levels in Rows B1 and B2. More importantly, the MTO experimental estimate is now negative (but not statistically significant due to the impact of weighting on the standard errors). This is a product of the weighting on baseline covariates, which resulted in upweighting of more advantaged MTO participants to make the MTO experimental sample more similar to the PSID sample. Note that the high poverty group’s neighborhood poverty mean is the same pre- and post-weighting (compare rows A1 and D1), but the mean outcome in the high neighborhood poverty group is now higher (better) and the implied mean in the low poverty group is unchanged (add the mean outcome in the high poverty group to the effect estimate).³⁴

Below we return to the question of why the MTO experimental estimate is negative when the sample is weighted to align more closely with the more advantaged PSID sample. Understanding the source of this negative effect estimate may prove important to understanding why the MTO experimental estimate differs from the MTO nonexperimental and PSID estimates. Before we do so, we need to first examine whether the other hypotheses can account for the discrepancy.

The neighborhood measures hypothesis suggests that the reason MTO did not find effects on adult employment or earnings was that MTO improved neighborhood poverty but had only a small effect on other dimensions of neighborhood SES. To examine whether this can explain the divergent outcomes, we added the concentrated disadvantage index to the weighting procedure to align the disadvantage index in the PSID and MTO nonexperimental samples with that in the MTO experimental sample (Rows F1 and F2 in Appendix Table A5 show that this procedure was successful at aligning the gap on the disadvantage index). Rows F1 and F2 in Table 3 show that this did not appreciably change the estimates in either sample. To the extent that racial isolation may be particularly important for outcomes and varies in a way that is different within MTO compared to national samples, matching on concentrated disadvantage overall in MTO need not match to racial isolation specifically in the PSID. That said, the effect on racial isolation in the MTO experimental versus MTO non-experimental analyses are quite similar (with a percentage point change in tract share black of −0.080 versus −0.069), yet the impact on adult labor market outcomes are quite different and the MTO non-experimental analysis remains similar to what we see in the PSID. So we conclude that the neighborhood measures hypothesis seems unlikely to explain divergent effect estimates.

The treatment duration hypothesis argues that the MTO experiment produced a change in the duration of exposure to low poverty neighborhoods that was too small to lead to improved outcomes for the MTO adults. To examine this hypothesis, we added the number of years in a low poverty neighborhood to the weighting procedure to align that measure in the PSID and MTO nonexperimental samples with that in the MTO experimental sample. However, this procedure fully aligned the gap only in the MTO nonexperimental sample (See rows G1 and G2 in Appendix Table A5). Doing so produces only a small change in the effect estimates (rows G1 and G2 in Table 3), suggesting that the treatment duration hypothesis is unlikely to explain the difference between the MTO experimental and non-experimental results. For the PSID, we cannot fully test this hypothesis because we cannot fully align the years in a low poverty neighborhood between the PSID and MTO experimental results.

Finally, the residential mobility hypothesis argues that the increased residential mobility experienced by the MTO participants who moved to better neighborhoods had the potential to counteract any gains from living in a lower-poverty neighborhood. To examine this hypothesis, we added number of residential moves between census tracts to our weighting procedure to align the number of residential moves in the PSID and MTO nonexperimental samples with that in the MTO experimental sample. The weighting procedure was able to fully align the gap only for the PSID sample (See rows H1 in Appendix Table A5). Because we still see a positive point estimate in the PSID sample (Row H1 in Table 3) even when the gaps in the number of moves between the high neighborhood poverty and low neighborhood poverty groups are the same in the MTO experimental and PSID samples, we conclude that the residential mobility hypothesis cannot explain the divergent estimates between the MTO experiment and the PSID.³⁵

DISCUSSION

While there are limits to the statistical power of our estimates, the analysis above suggests that none of the commonly offered hypotheses that seek to explain the discrepancies in the effects of improvements in neighborhood poverty on adult economic outcomes between existing experimental and non-experimental studies—including outcome measures, treatment magnitude, nonlinear effects, treatment effect heterogeneity, neighborhood measures, treatment duration, and residential mobility—can account for differences in estimates between experimental and non-experimental estimates. When we hold these factors constant, different point estimates of effects remain.³⁶ We next discuss some remaining possible explanations for the discrepancy in estimates between the MTO experimental estimates and the PSID and MTO non-experimental results.

We begin by returning to an anomalous result in the above analysis that offers a possible clue as to the source of the discrepancy between MTO experimental estimates and the observational estimates from the PSID and MTO nonexperimental analysis. Recall that when we weighted the MTO experimental sample to be similar to the more advantaged PSID sample, the point estimate from the MTO experiment changed from essentially zero (Table 3, row A1) to large and negative (Table 3, row D1). Further inspection of the data shows that this weighting increased the weight given to those employed at baseline in the analysis (although we continue to trim large weights to avoid a small number of observations unduly influencing the results).

In an exploratory analysis of treatment effect heterogeneity by baseline employment and education in the MTO and PSID (Table 4), we see suggestive—but far from definitive—evidence that among individuals working at baseline, the TOT effects on the economic index are negative (but not statistically significant). These negative effects are largest among those working at baseline who were not high school graduates, individuals who might have been most precariously employed at baseline due to their low skill levels and the types of industries and occupations available to them (on precarious employment in the low-skill labor market, see for example Kalleberg [2009]). Notably, these negative effects occur for both the low poverty and traditional voucher groups, both of whom received Section 8 vouchers. In contrast, those with high school diplomas who were not working at baseline appear to benefit economically from the MTO intervention, although the effects are smaller in magnitude, are not statistically significant at conventional levels, and could be due to a “floor effects” problem. These results suggest that the standard MTO effect estimates could be a combination of negative effects among those working at baseline and positive effects among those not working at baseline, resulting in an average effect close to zero. In other words, it may be that the negative effects among those employed at baseline are driving any differences in results between MTO and observational studies (although these are just 25% of the MTO study sample).

Table 4.

Comparison of MTO experimental TOT and PSID non-experimental effects on self-sufficiency index, by subgroup

	MTO Sample (N = 2543)				PSID Sample (N = 7520)
	Exper Complier Mean	Exp TOT		N	Low Poverty Group Mean	Estimated Effect of Low-Poverty Nbhd on Economic Index		N
Working at baseline
Working	0.135	−0.117		633	0.317	0.241	***	2845
		(0.126)				(.034)
Not working	−0.362	0.041		1910	−0.494	.046	*	1957
		(0.080)				(.027)
African-American
African-American	−0.246	0.021		1710	−0.025	.124	***	1576
		(0.081)				(.028)
Not African-American	−0.192	−0.044		833	−0.008	.117	***	3226
		(0.123)				(.038)
High school graduate at baseline
High school graduate	−0.287	0.219	*	977	0.111	.173	***	3542
		(0.112)				(.032)
Not a high school graduate	−0.197	−0.135		1566	−0.363	.073	**	1260
		(0.084)				(.029)
Child under age 5 at baseline
Child under age 5	−0.196	0.033		1322
		(0.093)
No children under age 5	−0.265	−0.038		1221
		(0.099)
Interacting baseline education and work
Not h.s. grad and not working	−0.338	−0.069		1245	−0.557	.058	*	801
		(0.100)				(.033)
Not h.s. grad and working	0.221	−0.322	*	321	−0.025	.168	***	459
		(0.158)				(.055)
High school grad but not working	−0.408	0.235	*	665	−0.449	.076	*	1156
		(0.132)				(.043)
High school grad and working	0.016	0.153		312	0.382	.236	***	2386
		(0.203)				(.043)
Interacting baseline education and race
Not h.s. grad and not African-American	−0.172	−0.101		569	−0.388	.072		676
		(0.150)				(.045)
Not h.s. grad and African-American	−0.212	−0.153		997	−0.335	.077	**	584
		(0.101)				(.038)
High school grad and not African-American	−0.231	0.068		264	0.093	.172	***	2550
		(0.210)				(.057)
High school grad and African-American	−0.309	0.280	*	713	0.158	.170	***	992
		(0.132)				(.038)
Interacting baseline work and race
Not working and not Afr. American	−0.383	0.076		644	−0.553	.046		1218
		(0.152)				(.043)
Not working and African-American	−0.354	0.026		1266	−0.395	.058		739
		(0.094)				(.036)
Working and not African-American	0.244	−0.330		189	0.323	.231	***	2008
		(0.204)				(.059)
Working and African-American	0.073	0.012		444	0.302	.232	***	837
		(0.159)				(.043)
Interacting baseline kids under 5 and work
No kids under 5 and not working	−0.396	−0.044		858
		(0.121)
No kids under 5 and working	0.014	−0.029		363
		(0.171)
Kids under 5 and not working	−0.336	0.108		1052
		(0.107)
Kids under 5 and working	0.284	−0.224		270
		(0.185)

Note: MTO ITT estimates are from an OLS model predicting the economic index on an indicator for treatment group assignment interacted with the different subgroups and controlling for baseline covariates. The MTO TOT was calculated by dividing the ITT by the compliance rate. The PSID estimates are from an OLS model interacting an indicator for average poverty below 25% with the subgroups and controlling for baseline covariates.

^***p < .01;

^**p < .05;

^*p < .1

It is impossible to know whether the negative economic effects of the MTO intervention we see among those employed at baseline are unique to the particular sample that participated in MTO or even whether they are unique to the time and place in which MTO was implemented, but it is worth considering how it might have come about and why we do not see similar effects in the PSID (Table 4). We see two possible explanations. The first is simply selection bias. It may be that the inability of observational studies to condition on unobserved covariates that play a role in both sorting individuals and families into low or high poverty neighborhoods and effect adult economic outcomes generates a positive bias in the PSID and MTO nonexperimental estimates among those employed at baseline. It seems likely that such selection bias is more severe for individuals employed at baseline, as such individuals may have, ex ante, more neighborhood options. The PSID estimates are consistent with this explanation, as positive estimates of moving to more advantaged neighborhoods are concentrated among those employed at baseline, who may have been on a better economic trajectory irrespective of residential moves to more advantaged neighborhoods. Because the MTO experimental estimates are based on a randomized experiment with a large sample, selection bias is highly unlikely to bias the MTO estimates, even in a subgroup that is only a part of the entire sample. However, subgroup analyses are considerably noisier due to smaller sample sizes.

Another possible explanation is that the MTO voucher moves were particularly disruptive for MTO respondents who were employed at baseline. In other words, although the above analysis tries to align the number of residential moves, it may also be that there is a qualitative difference in the type of moves created by the MTO intervention and residential moves experienced by PSID respondents or the MTO control group. Although residential moves by poor families are often involuntary and disruptive (Desmond 2016), moving with an MTO voucher puts particular constraints on the timeframe for moving without losing the voucher, and the low-poverty voucher restricts the neighborhoods in which the voucher can be used.

Qualitative evidence from MTO and other studies of housing voucher programs suggest that “leasing up” with a voucher can be difficult because many landlords do not want to accept Section 8 vouchers. This seems to be particularly true in low-poverty neighborhoods, where landlords have more potential tenants and where housing unit sizes that can accommodate families are in shorter supply (Edin, DeLuca, and Owens 2012; Rosenblatt and DeLuca 2012; Small et al. 2013). One consequence of these constraints is that leasing up with a Section 8 voucher can be very time-consuming. Somewhat counterintuitively, those who are working may face a particularly daunting task, as they will not have the time to engage with these challenges (e.g. Pashup et al. 2005) and may already have stable childcare arrangements that allow them to work. As a result, it is possible that MTO participants working at baseline were forced to accept housing units in locations that were not otherwise ideal for maintaining their employment in the long term. This suggests one possible explanation regarding qualitative differences between moves experienced by members of the low-poverty voucher and traditional voucher groups compared to PSID respondents: for individuals employed at baseline, their more constrained moves may be disruptive to their employment. They may be unable to lease-up in neighborhoods from which they can continue to commute to their jobs or from which they can access the social supports they need to maintain employment, such as childcare.

However, we also uncovered some evidence that was inconsistent with the disruptive voucher moves explanation. For example, one group that might provide an alternative test of the disruptive voucher moves explanation is MTO control group respondents who lived in housing developments that were scheduled for demolition, since they likely experienced voucher moves similar to those of the low-poverty voucher group. They were likely awarded Section 8 vouchers with a limited timeframe in which to move (although they were unlikely to have been required to move to low poverty neighborhoods as the low-poverty voucher group was). Yet our exploratory analysis of effects of living in a low poverty neighborhood in this subsample exhibit the largest positive coefficients among those who were employed at baseline, the opposite of what the disruptive moves explanation would predict. (We caution, however, that this group is particularly small and concentrated in one MTO city, Chicago.) Moreover, when we examined effect heterogeneity in the MTO experimental sample by the presence of children under age five in the household, who are too young to attend school and therefore might present the most challenges in terms of child care, our exploratory analysis finds no differences in point estimates. And while the disruptive voucher moves explanation would suggest that effects should be largest in the period after MTO randomization, this does not seem to be the case. Finally, for the MTO sample we are examining long-term employment outcomes measured 10–15 years after baseline, which raises the question of whether any disruptive effects of constrained moves might actually persist that long. Experimental effects on the economic index among people working at baseline were smaller 4 to 7 years after random assignment than 10 to 15 years after random assignment. Oreopoulous (2003) studied neighborhood effects on adult earnings of people whose families were assigned to different public housing units when they were youths—a research design that was like that of MTO in many ways except that it involved a comparison between living in higher- and lower-poverty public housing among people who all moved into public housing. The fact that neither the MTO experiment nor Oreopoulous’s quasi-experiment produced evidence of economic effects when a key difference between them was that Oreopoulous’s quasi-experiment involved changes in neighborhood quality with more moving suggests that the association between neighborhood quality and moving is of limited importance. Further research on housing mobility programs might investigate the possibility of potential disruptive effects more thoroughly than we are able to with these data.

CONCLUSION

The apparent discrepancy in findings between the MTO randomized residential-mobility experiment and previous non-experimental studies of the effects of exposure to poor neighborhoods during adulthood on adult labor market outcomes represents a puzzle for social science with important implications for social policy. This paper has improved our understanding of how to reconcile these estimates by replicating the MTO experimental estimate through application of non-experimental methods to both the PSID data and the MTO data.

While our statistical analysis frequently suffers from limited statistical power, our primary contribution is to at least cast doubt on many of the hypothesized explanations for the discrepancy between MTO and non-experimental studies. Because our PSID and MTO nonexperimental point estimates are so similar to each other but different from MTO experimental estimates, we view hypotheses about differences in outcome measurement and sample composition as unlikely. We do not find clear evidence that aligning the MTO experimental sample with the PSID and MTO non-experimental samples on observed baseline characteristics, neighborhood poverty gaps, years in high poverty neighborhoods, neighborhood poverty ranges, the concentrated disadvantage index, and number of moves across census tracts removes the discrepancies in neighborhood effects across samples. We therefore conclude that hypotheses related to effect heterogeneity, treatment magnitude, treatment duration, neighborhood effect nonlinearities, residential mobility, and neighborhood measures are unlikely to account for differences between experimental and non-experimental estimates of neighborhood effects on adult economic outcomes, recognizing again the caveat about the limits to our statistical power.

We also find suggestive evidence that null effects in the MTO experimental analyses could be due to a combination of positive effects among those not employed at baseline and negative effects among those employed at baseline. This would narrow the source of the discrepancy between MTO and PSID in particular to those employed at baseline. One possible explanation for the discrepancy between MTO experimental estimates and PSID observational estimates is that MTO voucher moves were particularly disruptive to women employed at baseline. Another is that selection bias is particularly important in this subgroup. To the extent that selection bias with respect to unobserved determinants of economic outcomes is a key explanation for why MTO and non-experimental studies diverge in their findings, future research must address selection more rigorously in order to both test theory and evaluate policy. Previous within-study comparisons in other domains of social policy suggest it may be particularly important to focus data collection on those variables associated with “treatment selection” (Cook, Steiner, and Pohl 2009; Pohl et al. 2009). Our analyses suggest there may be particular value to focusing such data collection on the detailed factors that help determine economic outcomes and that are often hard to fully proxy with what is available in typical datasets.

Given that true randomized field experiments in the social sciences are practically difficult, expensive, and rare, our findings also suggest that there would be great value to supporting data collection built around “natural experiments” in which policy changes generate exogenous differences in neighborhood environments across similar families. Examples include housing program lotteries to allocate scarce program slots (Jacob and Ludwig 2012; Jacob, Kapustin, and Ludwig 2015), public housing demolitions (Chyn 2018, Jacob 2004), and quasi-random relocation of people displaced by war or natural disasters (Kirk 2009). In principle this could also include examination of natural experiments that leave people in place but change the neighborhood around them.

To the extent that residential moves with housing choice vouchers are disruptive to employment among low-income women with children, effective housing mobility policy would require voucher program implementation that avoids such unintended consequences. Moreover, social science research would need to further understand the sources of these effects. With regard to the latter, the role of social and institutional supports for maintaining employment in an increasingly precarious and unstable low-wage labor market is an important topic of future research. So too is the geography of opportunity in the rental housing market, particularly as it intersects with child care, transportation and neighborhood quality.

One remaining question is why there are discrepancies between MTO and non-experimental studies in adult economic outcomes but not in other adult outcomes or in child outcomes. Our suggestive findings that the discrepancies for adult economic outcomes might be concentrated among adults employed at baseline may shed some light on this question. We have argued that either selection bias among this subsample of adults in non-experimental studies or the particularly disruptive effects of housing voucher moves on employment in this subsample in MTO may be responsible for the discrepancies. Both of these explanations should lead us to think more carefully about baseline characteristics related to employment, which is likely to be closely related to both selection into neighborhoods and to future adult economic outcomes. Because baseline adult employment is likely to be more closely related to future adult economic outcomes than to either child outcomes or adult health outcomes, both the selection bias and the voucher move disruption explanations should be more important for analyses of adult economic outcomes.

We conclude by reminding the reader of the limitations of this study. First, we cannot account for possible differences between MTO and PSID introduced by differential selection into those studies based on unobserved characteristics, including the decision to volunteer for the MTO randomization or the decision to participate in the PSID survey. Similarly, we cannot account for effect heterogeneity by unobserved characteristics that are independent of observed characteristics. However, the general similarity in non-experimental estimates that we see in MTO and the PSID suggest this may not be a major source of bias in practice. Second and more importantly, the sample sizes available in both the MTO and PSID datasets limit our ability to precisely determine when and under what conditions experimental estimates diverge from those derived using non-experimental approaches. In addition, with only five demonstration cities in MTO, we do not have the statistical power to understand how neighborhood effects vary across different parts of the country. Third, this research focuses on people moving among neighborhoods in the upper half of the distribution of tract poverty rates and does not examine effects that might occur in the lower half. Fourth, given the nature of the MTO data it is also not possible to fully explore the role played by the neighborhoods that surrounds one’s own neighborhood. Previous research suggests that there may be important differences across race and ethnic groups in the U.S. in the characteristics of neighborhoods around where one lives, even conditional on one’s immediate neighborhood characteristics, and raises the possibility that those contiguous neighborhoods could independently exert influence over people’s life outcomes (Crowder and South 2011; Sharkey 2014). Finally, MTO by its nature is not well-suited to understanding the effects of having neighborhoods differ for very long amounts of time, including across generations (Sharkey 2013).