Abstract
Objectives
Veterans are overrepresented within the homeless population compared with their non-veteran counterparts, particularly when controlling for poverty. The U.S. Department of Veterans Affairs (VA) aims to prevent new episodes of homelessness by targeting households at greatest risk; however, there are no instruments that systematically assess veterans' risk of homelessness. We developed and tested a brief screening instrument to identify imminent risk of homelessness among veterans accessing VA health care.
Methods
The study team developed initial assessment items, conducted cognitive interviews with veterans experiencing homelessness, refined pilot items based on veterans' and experts' feedback and results of psychometric analyses, and assigned weights to items in the final instrument to indicate a measure of homelessness risk.
Results
One-third of veterans who responded to the field instrument reported imminent risk of homelessness (i.e., housing instability in the previous 90 days or expected in the next 90 days). The reliability coefficient for the instrument was 0.85, indicating good internal consistency. Veterans who had a recent change in income, had unpaid housing expenses, were living temporarily with family and friends, needed help to get or keep housing, and had poor rental and credit histories were more likely to report a risk of homelessness than those who did not.
Conclusion
This study provides the field with an instrument to identify individuals and households at risk of or experiencing homelessness, which is necessary to prevent and end homelessness. In addition, it supports VA's investment in homelessness prevention and rapid rehousing services for veterans who are experiencing or are at risk for homelessness.
On one day in January 2013, 57,849 veterans in the United States were homeless (i.e., in shelters, transitional housing, and places not meant for human habitation).1 Veterans comprise 12% of homeless U.S. adults, are overrepresented within the homeless population, and are at a particularly high risk of homelessness compared with individuals living in poverty.2 The U.S. Department of Veterans Affairs (VA) is focusing efforts on ending chronic homelessness and preventing new episodes of homelessness. VA has allocated more than $1 billion toward homelessness prevention services through the Supportive Services for Veteran Families (SSVF) program during federal fiscal years 2011–2014.
SSVF provides homelessness prevention and rapid rehousing services for veterans who are at imminent risk of or have recently experienced homelessness. To ensure the efficient use of limited resources, efforts must target veteran households most likely to become homeless. To target resources accurately, service providers must have an understanding of (1) how to identify at-risk individuals and households, (2) where these households come from, and (3) what causes their homelessness.3 There is little evidence regarding the most efficient way to target households for a homelessness prevention intervention; such decisions are often associated with a high false-positive rate, making homelessness prevention relatively inefficient.4 While communities across the U.S. use a number of measures to determine if a household is at imminent risk—including barriers to housing, eviction notice, and household's level of self-sufficiency—few of these assessments have been tested for their effectiveness, and research has suggested that providing services based, at least in part, on empirical evidence can increase the efficiency of homelessness prevention.5
We describe the process by which VA developed and evaluated the reliability and validity of a brief screening and risk assessment instrument to (1) identify veterans accessing VA health-care services who are at imminent risk of homelessness and (2) quantify their level of risk to determine the appropriate level of care within VA Homeless Programs.
METHODS
The study team constructed and tested a pilot instrument, which was administered by social workers as part of regular operations. Using administrative data, the study team analyzed responses to the instrument and refined it to be psychometrically reliable, valid, and appropriate for use as a universal screen in varied health-care settings. The study took place at the Philadelphia VA Medical Center during 2012.
Development of a pilot screening instrument
Item development.
The study team gathered a pool of potential items to assess homelessness risk based on a literature review and items that homelessness prevention programs have implemented with some success. We identified measures assessing a number of constructs: current living situation,6 residential history,7 types of living environments,8,9 history of homelessness6,7,10 and residential transience,6,7,9–11 as well as a variety of barriers to housing stability.10,12 A panel including veterans as well as researchers and clinicians with expertise in homelessness prevention and intervention reviewed the item pool and developed new questions to ascertain imminent risk of homelessness.
Cognitive interviews.
Members of the study team conducted interviews with a small sample of veterans recruited from a community drop-in center serving male veterans (n=3) and a transitional housing program for female veterans (n=5) in Philadelphia. These interviews assessed if veterans responding to the instrument comprehended the questions (i.e., terminology used as well as question intent) and if their understanding was consistent with the instrument's goal.
Item refinement.
The study team used findings from the panel of researchers and cognitive interviews with veterans to refine the items included in the pilot instrument, which were organized into two stages. Stage I of the pilot instrument (screening) assessed whether or not the veteran was at risk of imminent homelessness by asking the following questions:
(1) Do you have a home of your own that is safe and where you have lived for the past 90 days? (“No” indicates positive for imminent risk of homelessness.)
(2) Are you worried that you may not have a home of your own that is safe and where you can live for the next 90 days? (“Yes” indicates positive for imminent risk of homelessness.)
While Stage I was intended to screen or identify veterans at imminent risk of homelessness, Stage II (risk assessment) assessed risk factors for homelessness to determine the degree or severity of homelessness risk. Each Stage II item required a yes/no response other than the number of moves, which was an ordinal coded variable. Table 1 lists each of the 37 items included in the initial pilot instrument.
Table 1.
Prevalence of Stage II items by screening disposition and odds of veterans screening positive for homelessness risk as a function of Stage II items: Pilot Homelessness Risk Screener, Pennsylvania and Delaware, 2012 (n=379)
Note: All Stage II items were binary variables (yes/no, with no as the reference category), with the exception of number of moves, which was ordinal.
aIndicates items that were not included in final instrument
bEstimates could not be calculated because one of the groups did not have variation in that item.
c≥2 moves vs. ≤1 move
OR = odds ratio
CI = confidence interval
NC = not calculated
HIV = human immunodeficiency virus
AIDS = acquired immunodeficiency syndrome
Field test
Social workers administered the pilot instrument from December 2011 to February 2012 as part of regular operations in social work, mental health, and substance use outpatient clinics at four locations in Pennsylvania and Delaware. The social worker at each location administered the instrument to the first 100 veterans who presented for outpatient services at their particular clinic; one site administered the instrument to 68 veterans, two sites to 101 veterans each, and the fourth site to 109 veterans, for a total sample of 379. Social workers administered both stages of the pilot instrument to each veteran by reading the items and then entering the veteran's response into his/her electronic medical record, requiring approximately 5 minutes of time. Table 2 includes a summary of the demographic characteristics of the study sample. The study team provided a detailed overview of the instrument's intent, instructions on how to complete the instrument in the veteran's medical record, and a document outlining, for each item, the rationale for inclusion, definitions, and response options. Members of the study team were available to address specific questions.
Table 2.
Characteristics of veterans responding to a pilot instrument assessing homelessness risk: Delaware and Pennsylvania, 2012 (n=379)
SD = standard deviation
The study team accessed veterans' responses to the instrument from the VA Corporate Data Warehouse.13 We estimated the internal consistency reliability of the instrument for the two stages separately, as well as for the entire instrument, using the Kuder-Richardson Formula 20 (KR-20) statistic, a derivative of the Cronbach formula for dichotomous variables.14 We assessed the convergent validity—how well two variables or constructs are related, given that they are theoretically assumed to be related (e.g., one's previous experience of homelessness is theoretically related to one's risk of homelessness)—of the pilot version of the screening instrument using responses to both Stage I and Stage II items. We compared the convergent validity of Stage I with Stage II items by computing unadjusted odds ratios from simple logistic regression analyses, with results of Stage I screening (positive or negative) as the outcome and each Stage II item as a predictor in separate models. We hypothesized that those who screened positive on Stage I would also report higher levels of risk as measured by Stage II items. Statistical assumptions for the logistic regression analyses were met.
Development of final screening instrument
Based on the initial psychometric analyses of the pilot instrument, we reduced the items and amended several questions. Specifically, we removed some of the questions due primarily to low positive response rates (i.e., the items were not endorsed, yielding low variability) (Table 1). For example, we combined two items regarding recent experience of homelessness into one question, collapsed categories for number of moves during the past year, collapsed physical disability to include chronic health conditions and human immunodeficiency virus/acquired immunodeficiency syndrome, and removed needing reliable/affordable childcare. We then reestimated the internal consistency reliability as well as the convergent validity of the items in the final instrument. The final instrument included 25 items; these items and results of the psychometric testing of the instrument are included in Table 3.
Table 3.
Prevalence of Stage II items by screening disposition, odds of veterans screening positive for homelessness risk as a function of Stage II items, and suggested weighting scheme for Stage II items: Final Homelessness Risk Screener, Pennsylvania and Delaware (n=379; KR-20=0.85)
Note: All Stage II items are binary variables (yes/no, with no as the reference category).
aWeighting scheme to determine severity of homelessness risk: 1 = low, 2 = moderate, and 3 = high.
KR-20 = Kuder-Richardson 20 statistic for internal consistency reliability
OR = odds ratio
CI = confidence interval
HIV = human immunodeficiency virus
AIDS = acquired immunodeficiency syndrome
Development of guidelines for level of care
To quantify risk based on Stage II items, we convened three subject-matter experts with more than 40 years of combined experience conducting homelessness research and program evaluations to assign weights for endorsement of each item in Stage II: 1 = low risk of homelessness, 2 = moderate risk of homelessness, and 3 = severe risk of homelessness (Table 3). In instances of disagreement, discussions were held until a majority (n=2) agreed upon the weighting value. Agreement was reached on all items, lending a high degree of content validity—the extent to which a measure represents all facets of a given social construct—to the final set of weights. We then summed the endorsed weighted items to create a risk severity score; all Stage II items were binary scaled (yes/no), so the weighted value of endorsed items contributed to the risk severity score while unendorsed items contributed nothing. We used the risk severity score to assign individuals to risk severity groups, which could be used to guide their referral to more or less intensive services (Figure). We also computed point-biserial correlations between the risk severity score and the veterans' responses to Stage I items to establish the convergent validity of the total risk severity score and created receiver operator characteristic (ROC) curves, with the risk severity score as the test variable and veterans' responses to Stage I as the state variable to estimate the discriminant ability of the risk severity score.
Figure.
Severity of homelessness risk based on veterans' responses to Stage II of an instrument to assess homelessness risk: Pennsylvania and Delaware, 2012 (n=379)
RESULTS
The pilot instrument was administered to 379 veterans who were presenting for outpatient services. Of these veterans, 33% screened positive for imminent risk of homelessness by indicating that they had experienced housing instability in the previous 90 days or expected housing instability in the next 90 days. More than 90% of the respondents were male. The majority of the respondents were veterans of the Vietnam War (34%) or the Persian Gulf War (36%), with an average age of approximately 51 years. Table 2 provides a summary of the sample characteristics.
Psychometric analyses of pilot instrument
The internal consistency reliability of items from Stage I was 0.61 and from Stage II was 0.82. The internal consistency reliability of the entire instrument was 0.85. The lower reliability of the Stage I portion of the instrument is largely explained by the small number of binary-scored items (n=2); therefore, the reliability coefficient for the entire instrument is the better indicator of the internal consistency reliability (data not shown).
Table 1 shows the results of the assessment of convergent validity of Stage II items, indicating a relationship between veterans' endorsement of imminent risk of homelessness and both housing barriers and risk factors for homelessness. With the exception of four items—living in unsubsidized housing, paying for housing expenses, having friends or family the household could live with, and having a physical disability—a positive response to each of the Stage II items increased the likelihood that a veteran would screen positive for homelessness risk. Large effect sizes were observed for some risk factors: veterans living in a sheltered homeless situation were 15 times as likely to screen positive rather than negative for homelessness risk, those living with family or friends on a temporary basis were almost 21 times as likely to screen positive rather than negative for homelessness risk, and veterans who needed help to get or keep housing were almost 26 times as likely to screen positive rather than negative for homelessness risk.
Psychometric analyses of final instrument
The internal consistency reliability of the final instrument was the same as the pilot instrument (0.61 for Stage I, 0.82 for Stage II, and 0.85 total), indicating that revisions did not diminish the internal consistency reliability and resulted in a simpler instrument. Because several of the predictors (items) did not change between the pilot and final versions of the instrument, their odds ratios remained the same in both versions. Combining the two items regarding recent experience of homelessness into one item reduced the number of questions required by the instrument but maintained a strong and significant relationship between risk of homelessness and recent experience of homelessness (Table 3).
Homelessness risk severity score
For the entire sample, the summed risk severity score from Stage II items—calculated by adding the weights for items with a positive response—ranged from 0 (no risk) to 32 (high risk), with a mean of 5.47 (standard deviation [SD] = 6.72). Point-biserial correlations between the risk severity score and Stage I items were 0.56 (p<0.001) for Stage I item 1 (currently have own home), 0.69 (p<0.001) for Stage I item 2 (worried about having own home), and 0.63 (p<0.001) for the result of both Stage I items (positive or negative). The results of the ROC curve analyses with risk severity score as the test variable and results of Stage I items indicated significant discriminative ability (all p<0.001) for each of the following: Stage I item 1 (currently have own home) was 0.83 (95% confidence interval [CI] 0.77, 0.88), Stage I item 2 (worried about having own home) was 0.89 (95% CI 0.85, 0.94), and Stage I result (positive or negative) was 0.84 (95% CI 0.80, 0.89) (data not shown).
Sensitivity and specificity were both maximized when the risk severity score was 6 for Stage I item 1 (0.76 and 0.77, respectively) and item 2 (0.83 and 0.86, respectively), and 5 for the result of Stage I screening (positive or negative) (0.72 and 0.73, respectively). In other words, approximately 80% of participants were classified as at risk for homelessness (true positives) with risk severity scores >5; conversely, about 80% of participants were not classified as at risk for homelessness (true negative) with risk severity scores <5. Similarly, 72% of participants were classified with a Stage I positive screen (true positive) with risk severity scores >6; conversely, about 73% of participants were not classified as positive screens (true negative) with risk severity scores <6 (data not shown).
For the subset of participants who screened positive for risk of homelessness (n=125), the mean risk severity score was 11.47 (SD=7.84, range 0–32). The Figure shows the distribution of risk severity scores for those who screened positive: 46% of veterans were at low risk, 44% were at moderate risk, and 10% were at high risk for homelessness. We developed the cut points based on the theory that individuals with the highest risk require the most intensive services and are the smallest proportion of the population at risk (10%).15,16
DISCUSSION
The field of homelessness prevention has long been in need of a valid and reliable instrument to assist providers in determining which households would become homeless but for assistance and the level of assistance that the households require. We described the process by which we developed, refined, and tested a homelessness risk screener. The expert panel's review contributed to the instrument's overall content validity, while the cognitive interviews contributed to the face validity. The psychometric testing indicated that it demonstrates good internal consistency reliability (KR-20=0.85). The tests for convergent validity demonstrated strong and significant associations between veterans' report of imminent risk of homelessness and risk factors for homelessness.
The preliminary development of a homelessness risk severity score is promising when considered along with clinicians' assessment of households' needs. The results of analyses that associate screening items with weighted risk scores indicate that Stage I item 2 had a higher degree of convergent validity than Stage I item 1, suggesting its sensitivity to risk for future homelessness. Additional work will determine the utility of risk scores in assigning or referring individuals to varying intensities of intervention, providing an empirically based guide for practitioners. Various scoring mechanisms could be implemented and tested as well as the relationship between a household's homelessness risk severity score and intervention dosage. Further, risk severity scores may enable VA to identify veterans who are at the most imminent risk of homelessness, representing a significant improvement over prior research, where systematic efforts to quantify risk and predict homelessness through the use of such instruments are almost nonexistent. Further examination of the antecedents of homelessness and severity of risk among veterans and disparities in homelessness risk would further support VA in establishing eligibility for prevention services and prioritizing the allocation of such resources.
Additional next steps for this study include testing more predictors of homelessness (e.g., factors related to children and family, such as childcare costs) and further psychometric testing, including a further evaluation of predictive validity to determine if responses and risk severity score in fact predict veterans' future experience of homelessness. It is important to note, however, that estimates of predictive validity may be confounded by the fact that veterans experiencing homelessness risk who are identified within VA would be referred for some intervention, although not all may be served due to personal preference, capacity issues, or lack of specific programming within a particular jurisdiction. Additional analyses would also explore the sensitivity of specific items (e.g., if severe housing burden is associated with perceived risk of housing instability in the next 90 days).
Limitations
This study was subject to several limitations. First, the study did not include a pretesting phase for the initial instrument prior to deploying it more broadly in the field. Second, pilot data were collected among a convenience sample of veterans accessing VA health-care services with few specified exclusion or inclusion criteria. The sample was slightly younger and included more females than the general veteran population. Because the sample comprised mostly men, it is uncertain if results are generalizeable to women. Although we did not find differences in our results when the sample was restricted to men only, this finding does not imply that our findings hold equally well for males and females. Future research will more thoroughly examine the psychometrics of the risk assessment instrument with females. Third, data have yet to be collected or analyzed to determine if veterans who indicated a risk of imminent homelessness later experienced homelessness. Finally, the findings presented in this study are relevant only to veterans accessing VA outpatient health care. Future work—including testing the instrument in other contexts and with other populations—is necessary to support its widespread use.
CONCLUSION
This homelessness risk assessment instrument demonstrates strong psychometric qualities and has potential as a screening and referral instrument for both VA and non-VA clientele. Users of this instrument will benefit from the ease of its administration in a variety of settings, and the information gleaned will be useful for prioritizing resources and targeting appropriate interventions.
Footnotes
This study was approved by the Institutional Review Board of the Philadelphia Veterans Affairs Medical Center. The contents of this article do not necessarily represent the views of the U.S. Department of Veterans Affairs or the U.S. government.
REFERENCES
- 1.Henry M, Cortes A, Morris S. Washington: Department of Housing and Urban Development (US); 2013. The 2013 Annual Homeless Assessment Report (AHAR) to Congress. Part 1: point-in-time estimates of homelessness. [Google Scholar]
- 2.Fargo J, Metraux S, Byrne T, Munley E, Montgomery AE, Jones H, et al. Prevalence and risk of homelessness among U.S. veterans: a multisite investigation. Prev Chronic Dis. 2012;9:110–2. [PMC free article] [PubMed] [Google Scholar]
- 3.Lindblom EN. Toward a comprehensive homelessness-prevention strategy. Housing Policy Debate. 1991;2:957–1025. [Google Scholar]
- 4.Shinn M, Baumohl J, Hopper K. The prevention of homelessness revisited. Anal Soc Issues Public Policy. 2001;1:95–127. [Google Scholar]
- 5.Shinn M, Greer AL, Bainbridge J, Kwon J, Zuiderveen S. Efficient targeting of homelessness prevention services for families. Am J Public Health. 2013;103(Suppl 2):S324–30. doi: 10.2105/AJPH.2013.301468. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Aidala AA, Lee G, Abramson DM, Messeri P, Siegler A. Housing need, housing assistance, and connection to HIV medical care. AIDS Behav. 2007;11(6 Suppl 1):S101–15. doi: 10.1007/s10461-007-9276-x. [DOI] [PubMed] [Google Scholar]
- 7.North CS, Eyrich KM, Pollio DE, Foster DA, Cottler LB, Spitznagel EL. The Homeless Supplement to the Diagnostic Interview Schedule: test-retest analyses. Int J Methods Psychiatr Res. 2004;13:184–91. doi: 10.1002/mpr.174. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Milby JB, Wallace D, Ward CL, Schumacher JE, Michael M. Towards a more sensitive assessment of homelessness: the Homelessness Severity Scale. J Social Distress Homeless. 2005;14:151–70. [Google Scholar]
- 9.Tsemberis S, McHugo G, Williams V, Hanrahan P, Stefancic A. Measuring homelessness and residential stability: the residential time-line follow-back inventory. J Community Psych. 2007;35:29–42. [Google Scholar]
- 10.Pavao J, Alvarez J, Baumrind N, Induni M, Kimerling R. Intimate partner violence and housing instability. Am J Prev Med. 2007;32:143–6. doi: 10.1016/j.amepre.2006.10.008. [DOI] [PubMed] [Google Scholar]
- 11.Burt MR. Washington: Department of Housing and Urban Development (US), Office of Policy Development and Research; 2010. Life after transitional housing for homeless families. [Google Scholar]
- 12.Minnesota Homeless Management Information System. Barriers to housing stability assessment. 2008 [cited 2012 Aug 14] Available from: URL: http://www.hmismn.org/forms/docs/BarriersToHousing.doc.
- 13.Maynard C, Chapko MK. Data resources in the Department of Veterans Affairs. Diabetes Care. 2004;27(Suppl 2):B22–6. doi: 10.2337/diacare.27.suppl_2.b22. [DOI] [PubMed] [Google Scholar]
- 14.Kuder GF, Richardson MW. The theory of the estimation of test reliability. Psychometrika. 1937;2:151–60. [Google Scholar]
- 15.Culhane DP, Metraux S, Byrne T. A prevention-centered approach to homelessness assistance: a paradigm shift? Housing Policy Debate. 2011;21:295–315. [Google Scholar]
- 16.Montgomery AE, Metraux S, Culhane DP. Rethinking homelessness prevention among persons with serious mental illness. Soc Issues Policy Rev. 2013;7:58–82. [Google Scholar]