Efficient Targeting of Homelessness Prevention Services for Families

Abstract

Objectives. We developed and evaluated a model to target homelessness prevention services to families more efficiently.

Methods. We followed 11 105 families who applied for community-based services to prevent homelessness in New York City from October 1, 2004, to June 30, 2008, through administrative records, using Cox regression to predict shelter entry.

Results. Over 3 years, 12.8% of applicants entered shelter. Both the complete Cox regression and a short screening model based on 15 risk factors derived from it were superior to worker judgments, with substantially higher hit rates at the same level of false alarms. We found no evidence that some families were too risky to be helped or that specific risk factors were particularly amenable to amelioration.

Conclusions. Despite some limitations, an empirical risk model can increase the efficiency of homelessness prevention services. Serving the same proportion of applicants but selecting those at highest risk according to the model would have increased correct targeting of families entering shelter by 26% and reduced misses by almost two thirds. Parallel models could be developed elsewhere.

Efforts to prevent people from becoming homeless have increased dramatically since 2009, when the federal government distributed $1.5 billion for the Homeless Prevention and Rapid Rehousing Program as part of the American Recovery and Reinvestment Act.¹ The National Alliance to End Homelessness credits this spending with reducing homelessness by 1% between 2009 and 2011,² when the economic downturn might otherwise have led to its burgeoning. Even earlier, 24 of 25 cities surveyed by the US Conference of Mayors³ had programs to prevent homelessness among families facing eviction. However, evidence that particular prevention efforts reduce homelessness remains sparse.

Burt et al.⁴ suggested that prevention strategies must be both effective (i.e., they must stop people from becoming homeless) and efficient (i.e., they must target help to people who would become homeless without it). Efficiency—getting services to the right people—may be the harder problem. Analysis of the American Community Survey by the Joint Center for Housing Studies has shown that 20.2 million households (18% of all US households) were severely cost burdened in 2010, paying more than half of their income for housing. This figure increased by 6.4 million households in the decade from 2001 to 2010.^5(p27) The number and percentage of households who doubled up (shared a housing unit with another household) increased over the course of the recession, with 21.8 million households, or 18.3%, doubling up in 2011.⁶ Yet despite widespread risk, most households avoid entering shelter. In this article, we develop a method for predicting which family households are most likely to become homeless in the absence of preventive services.

Practitioners frequently confound good services with bad targeting, deeming prevention successful if participants do not later come to shelter³; by this criterion, services could be made to appear even more effective by giving them only to millionaires. Existing targeting models are based on the accumulated wisdom of service providers but often lack empirical foundation. Many cities use a 1-factor model: eviction.³ Hennepin County, Minnesota, whose more sophisticated targeting model for preventing family homelessness has been widely copied, recently returned to the drawing board after finding that families given prevention services differed sharply from families who became homeless. For example, 40% of prevention recipients, but 94% of homeless families, had incomes less than $1000 per month; 1% versus 33% had a household head aged younger than 22 years.⁷

Ensuring that families who receive prevention services resemble families in shelter is also insufficient. For example, many families in shelter are headed by single mothers, both because single-parent families tend to be poor and because shelters in many jurisdictions exclude men. Among poor families, however, those who are single are no more likely to become homeless^8–10 or have repeat episodes.^11,12 Prior case-control studies have examined predictors of shelter entry for particular groups,^10,13–17 but only a few researchers^9,18,19 have examined the adequacy of the resulting models.

In other fields, a large literature spanning more than 50 years has suggested that actuarial predictions based on statistical models are more accurate than professional or clinical judgments.^20,21 More recent reviews have confirmed the superiority of mechanical models for prediction in the majority of cases across medicine, mental health, personality, and education^22,23 but have suggested somewhat more variability. In particular, logically rather than empirically derived rules are not necessarily better than clinical judgment,²³ which in the domain of homelessness means that rules for distributing service on the basis of judgments and experience, rather than empirical models, may not be better than caseworker judgments in individual cases.

With this study, we helped New York City develop an empirical targeting model to enhance the efficiency of its HomeBase prevention program for families. The program, which was shown to be effective in experimental and quasi-experimental evaluations,^24,25 provides customized services including case management, eviction prevention, landlord mediation, short-term emergency funding, and assistance in obtaining employment and public benefits to families at imminent risk of homelessness.

We evaluated the relationship of risk factors collected from applicants to later shelter entry, whether some families are too risky to be helped (so that a triage model might work better than one that gives services to families at highest risk), and whether some risk factors are particularly amenable to services. We developed a screening instrument the city has begun to use to target services and evaluated the efficiency of different models.

METHODS

Our report is based on city records of 11 105 families with incomes less than 200% of the federal poverty level²⁶ who applied for HomeBase prevention services from the New York City Department of Homeless Services between October 1, 2004, and June 30, 2008. We followed families through city records for 3 years to determine whether they entered the family shelter system. Applicants were defined as families if the household included at least 1 adult and at least 1 child or a pregnant woman. Workers interviewed applicants about potential risk factors for homelessness based on previous studies of risk^9,12 and then used their judgment to determine eligibility for services.

Measures

Predictors of shelter entry came from both Department of Homeless Services administrative records of past contact with the shelter system and from intake interviews conducted by HomeBase workers. Predictors in the domains of demographic variables, human capital, housing conditions, disability, interpersonal discord, childhood experiences, and shelter history by both self-report and administrative records are shown in Table 1.

TABLE 1—

Descriptive Data, Adjusted Hazard Ratios, and Confidence Intervals for Variables Predicting Shelter Entry in Cox Regression (n = 11 105): New York City; October 1, 2004–June 30, 2008

Predictora	No Shelter (n = 9686), % or Meanc	Shelter (n = 1149), % or Meanc	HR (95% CI)
Demographics
Female	90.2	93.3	1.281* (1.005, 1.633)
African American	51.9	56.3	1.351 (0.895, 2.040)
Hispanicb	45.5	41.3	1.074 (0.713, 1.619)
English speaker	77.7	86.3	1.099 (0.893, 1.354)
Age, yc	33.7	30.1	0.983*** (0.975, 0.990)
Child aged < 2 y	29.3	37.4	1.138* (1.008, 1.286)
No. of childrenc	1.91	1.97	1.043 (0.995, 1.092)
Pregnant	13.7	19.8	1.242** (1.078, 1.431)
Married or partner	13.7	13.5	1.090 (0.906, 1.311)
Veteran	0.7	0.6	1.119 (0.536, 2.338)
Human capital
High school or GED	55.7	44.7	0.849* (0.749, 0.963)
Currently employed	51.6	43.6	0.812** (0.712, . 926)
Currently receiving public assistance	26.9	37.5	1.297*** (1.131, 1.488)
Lost benefits in past y	14.3	19.9	1.140 (0.964, 1.349)
Housing conditions
Name on lease	38.3	30.0	0.816* (0.677, 0.983)
Overcrowding or discordd	39.2	54.0	1.021 (0.871, 1.196)
Doubled up	47.2	63.6	1.137 (0.934, 1.384)
Eviction threate	55.3	66.1	1.196* (1.039, 1.376)
Rent > 50% income	33.5	28.6	0.925 (0.789, 1.084)
Arrears, $c	1507	1163	1.000 (1.000, 1.000)
Unsafe conditions	9.1	11.3	0.880 (0.714, 1.084)
Level of disrepairc	2.22	2.36	1.020 (0.991, 1.051)
Moves in past yc	0.95	1.27	1.156*** (1.076, 1.241)
Currently receiving subsidy	10.3	8.5	0.851 (0.676, 1.072)
Disability and criminal justice
Chronic health problem or hospitalization	42.2	44.9	1.100 (0.958, 1.262)
Mental illness or hospitalization	12.4	13.9	0.823 (0.665, 1.018)
Substance problem or treatment	7.0	11.2	1.219 (0.953, 1.558)
Criminal justice involvementf	11.9	17.4	1.112 (0.924, 1.338)
Interpersonal discord
Domestic violenceg	24.7	29.3	0.869 (0.729, 1.036)
Legal involvementh	4.9	7.8	0.977 (0.747, 1.277)
Protective services involvementi	8.7	16.1	1.367*** (1.128, 1.658)
Discord ratingcj	2.26	2.70	1.089*** (1.046, 1.134)
Childhood experiences
Adolescent mother	22.3	33.6	0.947 (0.814, 1.102)
Adversity indexc, k	0.63	1.00	1.147*** (1.077, 1.221)
Shelter history by self-report
Shelter history as adult	24.4	47.0	1.425*** (1.221, 1.664)
Shelter application last 3 mo	3.1	11.0	1.628*** (1.309, 2.024)
Reintegrating into community	6.2	11.5	1.294* (1.056, 1.585)
Shelter history by administrative data
Previous shelter	10.8	25.0	1.153 (0.888, 1.498)
No. of previous shelter applicationsc	0.26	0.69	1.184*** (1.082, 1.296)
Found eligible previously	2.4	8.0	1.099 (0.846, 1.427)
Exit to subsidy	2.9	5.7	0.955 (0.734, 1.243)

Note. 95% CI = confidence interval; GED = general equivalency diploma; HR = hazard ratio.

^aCommunity district also controlled.

^bOmitted race/ethnicity category is “all other.”

^cContinuous variable.

^dOvercrowding and discord were combined in the original data set.

^eIncludes being evicted or told to leave by the landlord or leaseholder.

^fAny family member ever incarcerated or respondent on probation or parole.

^gEver experience domestic violence or violence in past year.

^hPolice called, charges filed, or order of protection received.

ⁱAdministration for Children and Families investigation in past year, open case, child ever in foster care, currently in protective care.

^jDiscord rating (9-point scale) with landlord, leaseholder, or household members.

^kCount of 5 experiences in childhood: family receipt of public assistance, abuse, shelter, foster care, 4 or more residential moves.

*P ≤ .05; **P ≤ .01; ***P ≤ .001.

Analyses

Variables considered here had an average of 34.7% of values missing. To complete missing data regarding pregnancy and presence of young children, the city matched nearly 80% of applicants in welfare case records. Following the literature, we imputed the remaining missing data using Stata’s imputation by chained equations multiple imputation program,^27,28 including auxiliary variables. We estimated a predictive model using Cox regression, including all families who did and did not receive services, as recommended in the forecasting literature.²⁹ Analyses are based on 50 imputed data sets, with robust standard errors corrected by Stata version 12 (StataCorp LP, College Station, TX).

RESULTS

Workers deemed 66.5% of families eligible for service (excluding those in the wrong community district or who refused services); 12.8% of families entered shelter over the next 3 years. Table 1 shows the demographic characteristics and risk factors for families who did and did not enter shelter for all variables considered in the Cox regression (before imputation). It also shows the adjusted hazard ratio and 95% confidence interval (CI) for each predictor, controlling for all others. The hazard ratio is the amount by which the rate of shelter entry is multiplied for people having a characteristic or the amount by which it is multiplied for each increment in continuous variables. Analyses controlled for community of residence.

Consistent with the literature, several variables were associated with elevated risk of shelter entry (or reentry), including young age, being pregnant or having a child aged younger than 2 years, facing an eviction threat, frequent moves in the past year, not holding a lease, childhood adversity or disruptions, current protective services involvement, and shelter history.^9,12 Self-reported shelter history or recent application was a stronger predictor than administrative data. Also consistent with the literature, a number of factors were shown to have no statistically significant effect, including self-reported poor building conditions, adolescent motherhood, indicators for mental illness, substance abuse and health problems, and history of criminal justice involvement.^9,12

Some results diverged from those of past studies, notably the finding of no effect for doubling up and receipt of housing subsidies. Doubling up was positively associated with interpersonal discord (averaged across the landlord, leaseholder, and household), which was a risk factor, and negatively associated with being a leaseholder, which was protective. The result for housing subsidies may be a result of measurement, as a subsidy was recorded only if the applicant reported the amount as income. Domestic violence, which did not predict shelter entry here, has had inconsistent relationships to homelessness in past studies.^8,9,12,30-36

Alternative Models

The model just described predicts overall risk of shelter entry among all families who applied for HomeBase services. If HomeBase is more effective in combating some risk factors than others, however, it might make sense to direct services to families with particular risks. If some families are at such high risk that services make no difference, a triage model might be more efficient. To explore the first possibility, we reestimated predictive models separately for families deemed eligible for services and for families deemed ineligible. Overall, the 2 models were similar. We conducted post hoc tests for statistical interactions between eligibility and each of 5 variables for which services seemed to make the most difference, that is, for which hazard ratios were lower for individuals who received services than for those who did not. These variables were pregnancy, (lack of) a high school diploma or general equivalency diploma, eviction, mental illness, and child protective services. None of the interactions made significant contributions to the model (P < .05).

To evaluate the possibility that some families were too vulnerable to help, we examined shelter entry by risk level for families who were deemed eligible and ineligible for services. (We calculated a risk score for each family by averaging predicted values from the model in Table 1 across the 50 imputed data sets.) Figure 1 shows that, ignoring eligibility status, the probability of shelter entry increased from 1% for families in the lowest decile of risk to 37% for families in the highest decile. Unsurprisingly, workers were more likely to give services to families deemed by the model to be at high rather than low risk; nevertheless, approximately 50% of families in the bottom half of the risk distribution were served. It does not appear that services helped low-risk families. Rates of shelter entry in deciles 1 through 5 were no lower for those who received services than for those who did not. Services did matter for families in the top half of the risk distribution, and the spread between rates of shelter entry for eligible and ineligible families increased with risk decile. No level of risk was too high for families to benefit from services and, indeed, even in the highest decile of measured risk, a majority of families avoided shelter.

FIGURE 1—

Proportion of HomeBase family applicants entering shelter by risk level and eligibility (n = 11 044), excluding families with eligibility pending: New York City; October 1, 2004–June 30, 2008.

Although eligibility did not interact with particular risk factors in the prediction of shelter entry, Figure 1 suggests that eligibility did interact with overall risk level. When risk (scored as 1 for the top 5 deciles and 0 for the bottom 5 deciles), eligibility, and a risk × eligibility interaction were added to the equation with all variables in Table 1, the parameter estimates and 95% CIs were as follows: eligibility, 1.257 (95% CI = 0.953, 1.66); risk dichotomy, 2.904 (95% CI = 2.242, 3.760); risk × eligibility interaction, 0.636 (95% CI = 0.472, 0.857); all with P values of less than .003.

Screening Instrument

The length of the HomeBase intake interview probably contributed to the amount of missing data, and the model derived from it is overfitted: variable weights capitalize on specific sample characteristics. Thus, we created a screening model that required less data collection and did not depend on the precise variable weights. We pruned the model by means of backward elimination of nonsignificant predictors, checking that no eliminated variable made a contribution to the final model, and assigned from 1 to 3 points to each variable on the basis of regression weights and examination of shelter entry at different levels of continuous predictors. The resulting screening model is shown in Table 2. It has 15 variables, and families can theoretically receive from 0 to 25 points. The empirical distribution ranged from 0 to 23 points, with higher scores indicating greater risk.

TABLE 2—

Screening Model for Receipt of HomeBase Services: New York City; October 1, 2004–June 30, 2008

Variables	Points Assigneda
Pregnancy	1
Child aged < 2 y	1
No high school or GED	1
Not currently employed	1
Not leaseholder	1
Reintegrating into community from any placementb	1
Currently receiving public assistance	2
Any involvement with protective servicesc	2
Reports being evicted or asked to leave by landlord or leaseholder	2
Reports applying for shelter in past 3 mo	2
Reports having been in shelter as an adult	3
Age, y
23–28	1
≤ 22	2
Moves in past y
1–3	1
≥ 4	2
Disruptive experiences in childhoodd
1–2	1
≥ 3	2
Discord with landlord, leaseholder, or within household
Moderatee	1
Severef	2

Note. GED = graduate equivalency degree.

^aPoints were assigned on the basis of regression rates and examination of shelter entry at different levels of continuous predictors. Families could theoretically receive from 0 to 25 points on the 15 variables.

^bShelter, jail, or treatment program.

^cIncluding investigation in past year, protective care, an open case, or a child in foster care.

^dFoster care, shelter, reported abuse, family received public assistance, or family moved 4 or more times.

^eOperationalized as 4.0–5.59 on a 9-point scale.

^fOperationalized as 5.6–9.0 on a 9-point scale.

Evaluation of Model Efficiency

Figure 2 shows receiver operating characteristic plots of hit rates (sensitivity, or correct predictions of shelter entry among families entering shelter) versus false alarm rates (1 – specificity, or false predictions among families who did not enter shelter) for the full model and the screening model (based on a single imputed data set). Any score on these models can be chosen as a cutoff, with families scoring higher than that cutoff predicted to enter shelter. As the risk level required for a positive prediction decreases, both hit rates and false alarm rates increase; administrators can attain higher hit rates if they are willing to tolerate high false alarm rates. The figure also shows point estimates for worker eligibility decisions (worker), a model based on any form of eviction by landlord or leaseholder (eviction), a model based on administrative records of any prior contact with the shelter system (prior contact), and the screening model serving the same percentage of families as the worker model (constant percent).

FIGURE 2—

Receiver operating characteristic curves for competing models (n = 10 410), excluding families outside appropriate community district and who refused services: New York City; October 1, 2004–June 30, 2008.

Note. constant percent = screening model serving the same percentage of families as the worker model; eviction = model based on any form of eviction by landlord or leaseholder; prior contact = model based on administrative records of any prior contact with the shelter system; worker = model based on worker eligibility decision.

Several conclusions are evident. The screening model was nearly as efficient as the full model. The prior contact model had the lowest rates of hits and false alarms (25.5% and 11.4%, respectively); it did about as well as the screening model at the same false alarm rate but provided no information for the 87% of families with no prior contact with the shelter system. The eviction model had higher rates (67.4% hits; 57.4% false alarms). The worker model had the highest rates (71.6% hits; 65.7% false alarms). At an equivalent false alarm rate, the screening model (with a cutoff of 5 or more points) had a hit rate of 91.9%. If the proportion of families served was held constant rather than the proportion of false alarms (constant percent), the screening model increased hits by 26% and reduced misses (among families who later entered shelter) by almost two thirds. Alternatively, the screening model (with a cutoff of 7 or more points) could yield a higher hit rate (74.7%) with a much lower false alarm rate (36.6%).

To check the validity of the screening model, we evaluated how well it predicted shelter entry across reasons for ineligibility. Using the model to target 66.5% of families who applied for HomeBase services (constant percent), we correctly identified 89% of the 427 families who entered shelter despite being classified as ineligible for services. This percentage includes 88% of 49 families thought to have insufficient housing risk, 87% of 125 families deemed eligible for a more appropriate program, 88% of 134 families who did not comply with the intake process, 100% of 9 families who refused services, 92% of 48 families who lived outside of the community district, and 89% of 62 families deemed ineligible for other or unspecified reasons.

DISCUSSION

Our analyses show that adoption of an empirical model for deciding which families to serve can make homelessness prevention more efficient. Because eligibility did not interact with particular risk factors in the prediction of shelter entry, and because services mattered most for those at highest risk, we conclude that the most efficient approach to prevention is to offer services to families at highest empirical risk.

The model permits choices about cutoff scores that represent trade-offs between correct identification of families entering shelter and false alarm rates. However, the optimal cutoff is not simply an empirical decision. It depends on the costs of homelessness—to the families as well as to the city, the effectiveness of prevention and the savings that accrue, and competing uses for public funds. That is, political and moral as well as empirical considerations can and should affect decisions.

This study has a number of limitations, even in its geographical and temporal contexts. Although the HomeBase intake form was based on prior research and included variables from a variety of domains, omitted-variable bias is likely. For example, we did not have an accurate income figure, which might add to the predictive power of the model. We also did not have a pure measure of crowding, only a dichotomous variable, crowding or discord, rated by intake workers without further definition. A housing subsidy was recorded only if the applicant reported the amount as income. Crowding and subsidies have both been found to be predictive in earlier work in the same city.⁹

Data are based on self-reports, which may be biased. For example, we cannot assert that substance problems have no influence on homelessness, simply that self-reported substance problems did not predict shelter entry for families in the context of other predictors. However, future data collected in the same way should have equivalent predictive power.

Missing data required imputation. Inaccuracies in data or imputations would attenuate prediction for all models, but especially the empirical ones. The extent to which services were effective further attenuated prediction for all models, but especially the implicit worker model. It is notable that a model based on people who applied for services but who were deemed ineligible was very similar to our full model (no evidence of interactions was found). Nonetheless, ineligible families may differ from eligible families in their probability of shelter entry, based on unobserved characteristics.

In the face of these limitations, we are encouraged by how well the model did in predicting shelter entry both in the overall data and the subsamples, who were ineligible for various reasons. The model is far from perfect—only 47.8% of families in the highest decile of risk who were not served entered shelter—but, consistent with the forecasting literature, it is better than worker judgments.

The superiority of statistical models depends on a stable state for prediction, and our model might not hold if we went beyond the period in which the data were collected. The people who applied for HomeBase services depended on both the communities served and the nature of outreach—they were not a random sample of poor New Yorkers. If the sample were to change, as it no doubt has as services have been extended to new communities and different outreach strategies have been adopted, risk factors would be likely to change somewhat. The policy environment has also changed, with the ending of a subsidy program for families exiting shelter (because of withdrawal of state funds), which is likely to change the characteristics of families applying for services going forward. It is thus important to continually test and revise models as circumstances change. The screening model is likely more robust to such changes than that based on precise regression weights, so it is encouraging that it did almost as well, even for the current sample for whom regression weights were optimal. Many of the variables found important in this study were also important predictors in earlier studies of initial shelter entry⁹ and returns to shelter¹² in the same city, suggesting some degree of temporal stability.

A final criticism of this approach is that it only works for people who apply for prevention services. In the study period, 13% of all shelter entrants from the 6 largest HomeBase service areas first applied for preventive services; this percentage would have been higher had services not reduced entries.^24,25 Prevention services that reach only a small proportion of families at risk may be worthwhile if they reduce homelessness among those families. Shelter is costly, and homelessness among families has other disruptive effects on children’s schooling³⁷ and involvement with child protective services.^38,39 Although good reason exists to expand a successful program if feasible, little reason exists to abandon it because it does not reach everyone.

On the basis of these analyses showing that the HomeBase program can be made more efficient by using an empirical model to decide which families to serve, New York City has adopted the model. It is allowing workers to override model decisions with explanations and approval in a limited number of cases. Allowing overrides permits workers to adapt to changing populations and conditions and to react to unique circumstances; it may also reduce workers’ resistance to empirical approaches. Analysis of reasons for overrides will allow evaluation of factors that the model might miss.

Should this model or method be adopted elsewhere? Models are quite likely to depend on the populations served, the services offered, and the local homeless service system. Thus, we advocate our approach rather than our model, at least until it can be compared with other models. Such models could be derived in jurisdictions that adopted formal assessments to determine eligibility for homelessness prevention services funded under the American Recovery and Reinvestment Act¹ and can link these to records of shelter entry in Homeless Management Information Systems. The forecasting literature has suggested that optimal approaches for estimation depend on the number of observations per predictor variable and the multiple correlation of the predictors and the outcome.²⁹ However, the precise method is not critical because models based on the same predictors will be highly correlated and offer nearly the same predictions.⁴⁰

Jurisdictions without data on homelessness may be able to model workers’ decision-making process. Models of expert (worker) decisions are likely to be superior to the actual decisions of the experts.⁴¹ For jurisdictions with no data at all, our model might be a reasonable starting point.

Data from New York City’s HomeBase prevention program suggest that formal models of risk can offer a substantial bonus in efficiency in homelessness prevention services, getting services to people most likely to benefit from them. Continuing to serve the same percentage of families but selecting them via a model rather than caseworker judgment would have increased the correct targeting of families entering shelter by 26% and reduced misses of those families by almost two thirds during the study period. If services remain equally effective for higher risk families, improving efficiency is as important as improving effectiveness in preventing homelessness. If services are even more effective for higher-risk families, as our data suggest, the gains from better targeting would be even greater. Similar strategies could enhance the efficiency of prevention services elsewhere.