Impact of Supported Housing on Clinical Outcomes Analysis of a Randomized Trial Using Multiple Imputation Technique

Abstract

In 1992, the US Department of Housing and Urban Development (HUD) and the US Department of Veterans Affairs (VA) established the HUD-VA Supported Housing (HUD-VASH) Program to provide integrated clinical and housing services to homeless veterans with psychiatric and/or substance abuse disorders at 19 sites. At four sites, 460 subjects were randomly assigned to one of the three groups: (1) HUD-VASH, with both Section 8 vouchers and intensive case management; (2) case management only; and (3) standard VA care. A previous publication found HUD-VASH resulted in superior housing outcomes but yielded no benefits on clinical outcomes. Since many participants missed prescheduled visits during the follow-up period and follow-up rates were quite different across the groups, we reanalyzed these data using multiple imputation statistical methods to account for the missing observations. Significant benefits were found for HUD-VASH in drug and alcohol abuse outcomes that had not previously been identified.

Delivery of effective services to homeless people with serious psychiatric and/or addictive disorders has been a major challenge, in large part because of the need to combine services from multiple agencies to address housing, psychiatric, and substance abuse problems and provide income support and social and vocational rehabilitation (Goldman and Morrissey, 1985).

Recently, experimental studies have demonstrated superior outcomes for homeless persons with mental illness with diverse interventions, typically described as supported housing in which case management and housing resources are combined. These interventions appear to generate benefits for housing outcomes but not for clinical status (Morse, 1999; Rosenheck et al., 2003; Tsemberis et al., 2004). In 1992, the US Department of Housing and Urban Development (HUD) and the US Department of Veterans Affairs (VA) established the HUD-VA Supported Housing (HUD-VASH) program in which VA intensive case management were linked to Section 8 housing vouchers, which provide rent subsidies to low-income individuals with disabilities. Participants at four of the 19 HUD-VASH sites agreed to participate in an experimental evaluation through which they were randomly assigned to (1) HUD-VASH; (2) intensive case management without special access to Section 8 vouchers, in which case managers were to provide the same intensity of services as in HUD-VASH and were encouraged to use whatever housing resources could be obtained for their clients; or (3) standard VA homeless services, which consisted of short-term broker case management linking clients to VA and community services. Because of limited case manager resources, one participant was randomized to group 2 for every two participants randomized to groups 1 and 3.

It was hypothesized that supported housing would facilitate exit from homelessness and result in the improved housing outcomes but would also result in improved clinical outcomes because housed clients would have better access to services. In the initial report from this evaluation, Rosenheck et al. (2003) showed that HUD-VASH did facilitate exit from homelessness and increased use of mental health services, but that there was no greater improvement in psychiatric or substance abuse outcomes with HUD-VASH, and no greater benefits with intensive case management than with standard short-term care.

However, many participants missed scheduled follow-up assessment visits, and follow-up rates were highest for the HUD-VASH group (77.8%); lower among clients who received case management but not vouchers (63.4%), and lowest among those who received short-term case management (55.0%; Table 1). The evaluation of the intervention effects would not be biased and the missing follow-up observations would pose no problem if the missed observations were missing completely at random (Little and Rubin, 2002), i.e., if those who missed a follow-up visit were in no way different from those who were interviewed. However, it seems more likely that those missed follow-up visits were doing less well than others and that evaluation results were influenced as much by problems with missed observations as by the differential effects of the interventions. For example, further analysis of the HUD-VASH data reveals that veterans with more days of intoxication at baseline were more likely to miss a follow-up visit than those with fewer days of intoxication. The actual number of days intoxicated may thus be much larger than the observed rates, especially in groups with more extensive loss of follow-up data. Analytic methods such as multiple imputation (Rubin, 1987, 1996) and inverse-weighting (Lin et al., 2004; Robins et al., 1995) have been developed precisely to address the evaluation of intervention effects in experimental longitudinal studies with substantial missing data. These methods all rest on the assumption that data are missing at random (MAR; Little and Rubin, 2002), i.e., the likelihood of missing a follow-up visit can be predicted by available baseline and/or past observed data. Practical analytic methods are usually based on the plausible assumption that data are MAR, i.e., that previous observations can be used to impute subsequent missing data. The analyses presented in this paper are based on such a MAR assumption. A more serious missing data situation occurs when data are nonignorable (Little and Rubin, 2002), i.e., when the likelihood of missing a follow-up visit cannot be predicted from previous observations. In such circumstances, imputation cannot be implemented without making unverifiable assumptions. In this study, we use multiple imputation to re-examine the results of the HUD-VASH evaluation, addressing the potential biases introduced in the original publication by differential loss of data between treatment groups.

TABLE 1.

Follow-Up Rates at 6-Month Intervals for Three Intervention Groups

Months After Randomization	Group 1 (N = 182)	Group 2 (N = 90)	Group 3 (N = 188)
0	100	100	100
6	70.7	65.6	67.0
12	83.5	70.0	66.5
18	83.5	70.0	60.6
24	78.0	75.6	56.9
30	75.8	60.0	45.7
36	77.8	63.4	55.0
Median number of imputation values	1	1.5	3
Percentage of complete case	41.21%	26.67%	18.62%

METHODS

HUD-VASH Program, Participants, and Measures

In 1992, HUD and VA allocated funds for approximately 1000 housing vouchers and integrated case management assistance for homeless veterans with psychiatric and/or substance abuse problems at 19 sites (Kasprow et al., 2000). These vouchers authorize payment of a standardized local fair market rent (established by HUD using surveys of local rents) less 30% of the individual beneficiary's income.

The study took place at four of 19 participating VA medical centers, located in San Francisco, California (N = 107); San Diego, California (N = 91); New Orleans, Louisiana (N = 165); and Cleveland, Ohio (N = 97). Veterans were eligible if they had been homeless for 1 month or more (i.e., living in a homeless shelter or on the streets) and had received a diagnosis of a major psychiatric disorder (schizophrenia, bipolar disorder, major affective disorder, PTSD) and/or an alcohol or drug abuse disorder. Altogether, 460 veterans gave written informed consent to participate in the study. Further details of the study design and demographic and clinical characteristics of the participants have been presented previously (Rosenheck et al., 2003).

The primary outcome measures were the number of nights housed in the previous 90 at the time of each assessment interview (i.e., sleeping in an apartment, room, or house of one's own or of a family member or of a friend) and the number of nights homeless (i.e., sleeping in an emergency shelter, substandard single room occupancy hotel, or outdoors). The residual housing category documented nights in institutions (e.g., hospitals, halfway houses, jails, and so forth). Among those who were housed, the quality of the residence was further assessed using two scales developed for the Robert Wood Johnson Program on Chronic Mental Illness (Newman et al., 1994): one that addressed positive characteristics of the residence (e.g., safety, near shopping, big enough, private enough, affordable) and the other measuring problematic characteristics (e.g., pests, broken windows, neighborhood crime, plumbing problems).

Specific items and composite scores from the Addiction Severity Index (ASI; McLellan et al., 1980) were used to assess alcohol, drug, and medical problems. The Brief Symptom Inventory (Derogatis and Spencer, 1982) was used to measure psychological distress. Diagnoses were based on the working clinical diagnoses of the case management teams.

Subscales from the Lehman (1988) Quality of Life Interview were used to evaluate overall subjective quality of life and satisfaction with current housing, family relationships, social relationships, health care, and finances.

Social support was measured in three ways: by the average number of types of people who would help with a loan or transportation or in an emotional crisis (Vaux and Athanassopulou, 1987), the number of people in nine different categories to whom the veteran reported feeling close, and an index of the total frequency of contact with these people (Lam and Rosenheck, 1998).

Data Analysis

As in the original publication, we compared outcomes across the three intervention groups to determine whether HUD-VASH (housing subsidies and case management together) were associated with superior outcomes to either case management alone or standard care, and whether intensive case management was superior to standard care. The follow-up periods selected for analysis were baseline and 6, 12, 18, 24, 30, and 36 months, and all interviews conducted during each interval were included. Because we planned to compare the three treatment groups during five intervals following the baseline assessment, we used generalized linear models for repeated measures. For the comparison of the three groups over the 3-year follow-up period, we calculated the area under the estimated response curve (AUC). AUC represents estimated average cumulative status during the entire 3-year study period.

Due to the fact that some participants had missing observations at various time intervals, we first used the multiple imputation method developed by Rubin (1987, 1996) to impute missing responses. To impute a missing outcome of type k at time interval j for subject i (where i = 1, 2, …, n represents a participant i; j = 0, 6, 12, …, 36 represents a time point j; and k represents an outcome measure k), we used linear and quadratic terms to represent time for each intervention group (i.e., interaction terms between time and group were created) and the most recent nonmissing type k outcome measure from subject i was included as a covariate. The maximum number of imputed outcomes is 6 (in the cases for which only baseline visits were available, N = 40 cases) and the maximum number of imputed outcomes was 5 in cases for which only baseline and the 6-month visits were available (N = 30 cases). Specifically, we use the following imputation model for j > 0 since there are not missing observations at baseline:

$$Y_{\mathit{ijk}}={\alpha }_0+{\alpha }_{1Z_i}{t}_j+{\alpha }_{2Z_i}{t}_j^2+{\alpha }_3^T{X}_i+{\alpha }_4{Y}_{\mathit{ik},j-1}+e_{\mathit{ijk}}$$ (1)

where Y_ijk denotes the outcome type k response measure for subject i at time point j; trichotomous variable Z_i represents the intervention group assignment for subject i; i.e., Z_i = 1, 2, or 3 indicates whether subject i is in group 1, 2, or 3; α₀ is a common intercept; (α_1Zi, α_2Zi) are group-specific linear and quadratic terms representing time for the intervention group to which subject i belongs; while α₃ is a vector of regression coefficients associated with baseline covariates X_i; and α₄ is the coefficient associated with Y_{ik, j–1} which is the most recent outcome measure of type k observed for subject i. The error term e_ijk is normally distributed with a mean value of 0 and variance ${\theta }_k^2$. Covariates X are selected among those that significantly predict the outcome in question using general estimation equations (SAS Proc Genmod) with a step-down approach. Based on the fitted regression coefficients from the above imputation model, a new regression model is simulated from the posterior predictive distribution of the parameters and is used to impute the missing values for each variable (Rubin, 1987, pp. 166–167). The imputation is implemented using monotone regression method in SAS Proc MI. Twenty imputed data sets are generated for the analysis of each outcome that is described below.

After imputation, generalized linear models for repeated measures (Zeger and Liang, 1986) were used to model the trajectory of each outcome. The time points were treated as discrete variables, and the specific trajectory over time was modeled for each intervention group. Specifically, we use the following repeated-measures analysis model:

$$Y_{\mathit{ijk}}={\beta }_{{\mathrm{ijZ}}_i}+{\epsilon }_{\mathit{ijk}}$$ (2)

where the mean outcome response of type k at time j for the intervention group to which subject i is assigned is therefore modeled by β_jkZi and the error term ε_ijk is normally distributed with mean 0 and variance ${\sigma }_k^2$. The correlation across different time points within same participants is assumed to be ρ_jj'k for j ≠ j', and observations across different types of outcome or across different participants are assumed to be independent. The model was fitted with SAS Proc Mixed using the Model and Repeated statements. The estimates β_jkZi's allow us to plot the mean trajectory of an outcome k for an intervention group over the follow-up period. Since β_jkZi are estimated 20 times from 20 complete imputed data sets, SEs for the final β_jkZi are estimated by incorporating the variability associated with estimating β_jkZi in a single imputed data set and the variability across the 20 imputed data sets. This is achieved by SAS Proc Mianalyze.

For the comparison of the three groups over the entire follow-up period, area under the mean response trajectory curve is calculated for each intervention group using the final estimates β_jkZi. The AUC represents the estimated mean cumulative sum of the outcome response during entire 3-year study period. Differences between paired intervention groups were considered statistically significant if the p value was smaller than the prespecified α = 0.05.

RESULTS

As expected, there were no significant differences among the three intervention groups at baseline in any of the 16 central outcomes (Rosenheck et al., 2003). Comparison of results using standard repeated-measures methods and using multiple imputation methods revealed differences in statistically significant results on six of 16 outcome measures, five of which involved measures of substance abuse and one of which involved housing outcomes (Table 2). We did not adjust the level of α for multiple comparisons because days housed was the primary outcome measure, and the analysis of the other outcomes is exploratory. All the tests were two-tailed.

TABLE 2.

Statistical Evaluation of the Outcome Differences Across the Treatment Groups

	Estimated AUCs Over 3 Years
	Without Multiple Imputation				After Multiple Imputation
	Group Label				Group Label
OUTCOME	(1)	(2)	(3)	Summary	(1)	(2)	(3)	Summary
Days houseda	850.7	755.2	694.4	(1) > (2), (3)	854.4	739.0	687.5	(1) > (2), (3)
Days intoxicateda	40.3	41.40	46.83		68.7	94.77	101.54	(1) < (3)
Days drank alcohola	86.3	132.3	110.28	(1) < (2)	122.9	178.3	174.2	(1) < (2), (3)
Days using drug in communitya	69.8	70.3	72.4		100.2	117.3	128.9	(1) < (3)
Days in institutionsa	168.3	182.9	218.3	(1) < (3)	205.6	254.7	292.4	(1) < (2), (3)
Days employeda	256.6	264.9	272.6		271.4	280.8	310.8
Alcohol index scoreb	.1096	.1391	.1104		.1334	.1706	.1506	(1) > (2)
Drug index scoreb	.00585	.0644	.0591		.0727	.0850	.0875	(1) > (3)
Psychiatric index scoreb	.2467	.2583	.2383		.2717	.3067	.2750
Medical index scoreb	.2633	.2633	.2683		.2867	.3033	.3050
Expenditure on substance abuseb	72.25	71.83	61.33		111.3	149.40	153.43	(1) < (3)
Positive housing characteristicsb	4.03	3.87	3.73	(1) > (3)	3.83	3.72	3.52	(1) > (3)
Negative housing characteristicsb	1.80	2.65	2.68	(1) < (2), (3)	2.30	3.00	3.13	(1) < (2), (3)
Social helpb	7.83	6.80	7.47		7.90	7.07	7.52
Social network sizeb	11.60	9.75	10.90		11.85	9.92	11.01
Social contactsb	38.97	32.12	39.15		40.90	34.46	40.50
Overall QOLb	40.67	37.15	39.93	(1), (3) < (2)	41.15	37.27	40.38	(1), (3) < (2)

^aCumulative days over 3 years.

^bAverage monthly measure over 3 years.

While only one marginally significant difference was found between groups on substance abuse outcomes using ordinary repeated-measures analyses (HUD-VASH veterans used alcohol on fewer days than intensive case management controls; p value = 0.046), with multiple imputation analysis the HUD-VASH group had substantially and significantly fewer days of alcohol use than the standard care (i.e., short-term case management) group (p value = 0.0047); fewer days on which they drank to intoxication (p value = 0.0053); and fewer days of drug use (p value = 0.028); as well as lower scores on the ASI composite drug problem index (p value = 0.015; Table 2).

Figure 1 plots the results for days of intoxication, using standard methods, while Figure 2 shows the results with multiple imputation. HUD-VASH clients were also found to have had lower scores than the intensive case management control group (group 2) on the ASI composite alcohol problems index (p value = 0.018).

FIGURE 1.

Estimated days intoxicated over 3 years: repeated-measures analysis without multiple imputation. Days intoxicated for past 30 days at each visit time point (baseline = 0, 6, 12, 18, 24, 30, 36 months) were estimated and plotted against time point for three groups before multiple imputation. By inspecting this plot, we cannot see the significant different of days intoxicated among three groups since the lines are tangled together at some time points.

FIGURE 2.

Days intoxicated over 3 years by treatment group: repeated-measures analysis following multiple imputation. After the multiple imputation for missing values, we plotted the estimated days intoxicated against the seven visiting time points. From this plot, we can tell that the HUD-VASH group has less days intoxicated compared with the other two groups, and this finding is supported statistically (p value= 0.018).

The only nonsubstance abuse measure for which results changed with multiple imputation concerned nights slept in institutions. In the original analysis, the HUD-VASH group had fewer nights in institutions than the standard care (i.e., short-term case management) group (p value = 0.021). With multiple imputation, HUD-VASH veterans were also found to have spent fewer days in institutions than the intensive case management without vouchers group (p value = 0.030).

While there were no significant differences in money spent on alcohol and drugs among any of the three intervention groups in the original analysis, with multiple imputation, the HUD-VASH group was also found to have significantly lower expenditures on alcohol and drugs (p = 0.048) than the standard care group.

We further explore the analysis of the number of days of drinking to intoxication to understand better the difference in results between the original analysis and the analysis with multiple imputations. As noted, the analysis with multiple imputations, but not the standard analysis, showed that the HUD-VASH group had fewer days of intoxication than both the intensive case management only group and the short-term case management group. The explanation for this difference is that subjects with more days of intoxication were significantly more likely to miss follow-up visits. The values restored by the imputation method are thus most likely to be those pertaining to observations that are expected to have reported more days of intoxication. Since data loss was greatest in the intensive case management only and short-term case management groups, they had more imputed observations that reported days of intoxication than the HUD-VASH group. Differences between the results with multiple imputation analysis and the original analysis on the other outcomes have similar explanations.

DISCUSSION

We studied differences in psychiatric, substance abuse, and community adjustment outcomes among the participants who were randomized to a supported housing intervention with intensive case management or to one of two comparison groups in a population of homeless people with mental illness and/or addictive disorders. The original publication from this study found improved housing outcomes for the HUD-VASH intervention group (Rosenheck et al., 2003). In our reanalysis with multiple imputation of missing data, we demonstrated that the HUD-VASH intervention was also associated with improved substance abuse outcomes. The result of our re-analysis yielded similar conclusions to the original analysis regarding the effectiveness of intensive case management without housing vouchers—that there were no statistically significant advantages over standard VA care in housing, psychiatric, substance abuse, or community adjustment outcomes. Overall quality of life was improved for the participants in the experimental intervention as shown in both the original analysis and in the current analysis (results not presented here); however, the difference between group 1 and group 3 did not reach statistical significance.

Our reanalysis using multiple imputation technique addressed a principal limitation of the previous publication on the HUD-VASH evaluation, the differential follow-up attrition across the treatment groups. However, it should be noted that the underlying assumption for multiple imputation analysis is that data are MAR, which means the missingness in an outcome measure can be predicted by observed data and does not require additional unobserved data. If this assumption is violated—that is, if the missingness depends on unobserved outcome responses—then the multiple imputation method used here may not be valid. Mixed effects models can provide useful inferences about an observed repeated outcome by integrating the missing outcome data under the assumption of MAR and normality of the outcome; however, it does not unambiguously support causal inferences of intervention effects, although it is consistent with such inferences.

The overall between-group differences during the 3-year period were assessed using AUC. Since repeated-measures analysis was used with time as a categorical variable, the outcome trajectories can also be captured. Groups with similar AUCs may have quite different between-group trajectories, and vice versa. This is clearly the advantage of using AUC in repeated-measures analysis.

In addition to demonstrating the value of multiple imputations in the analysis of data from clinical trials involving homeless persons with mental illness, this study also has substantive importance for the evaluation of supported housing, because it suggests that providing intensive community services with housing subsidies may facilitate improvement both in housing and in some clinical outcomes. It has long been speculated that improving the housing status of homeless people with addictive disorders might stimulate a virtuous cycle in which clients would be more motivated to pursue recovery goals and reduce their substance abuse. Additional analysis showed that improved housing status was associated with reduced alcohol and drug use. However, after including housing status as a time-varying covariate, improvement in substance abuse outcomes was still significantly associated with assignment to HUD-VASH, suggesting that these benefits may have reflected the delivery of more intensive case management services in this intervention. Our reanalysis of data from the HUD-VASH provides the most supportive data to this hypothesis yet available.

CONCLUSION

Using multiple imputation to restore missing outcome data from the HUD-VASH experimental evaluation, we successfully identified hitherto unrecognized statistically significant differences in clinical outcomes between the three intervention groups and also confirmed superior housing outcome as reported previously. In these analyses, significantly less drug and alcohol abuse was found in the HUDVASH group than in the intensive case management group and standard VA care groups. We also found that the HUDVASH group spent significantly fewer days in institutions than the other two groups. Under the plausible assumption of MAR, our results confirmed the original hypothesis that access to the housing subsidies and case management services together would result in improvements in both housing and substance abuse outcomes, although not in improvement of psychiatric outcomes.