Validation of Self-Reported Veteran Status Among Two Sheltered Homeless Populations

Abstract

Objectives

We assessed the accuracy of self-reported veteran status among sheltered homeless adults to assess the reliability of using self-report to determine the number of veterans in homeless populations and examine whether there are demographic correlates to inaccurate reporting of veteran status.

Methods

Records on 5,860 sheltered adults from Columbus, Ohio, and 16,346 sheltered adults from New York City (NYC) were matched with U.S. Department of Veterans Affairs (VA) records. We analyzed the agreement between veteran self-reporting and official records using descriptive measures, diagnostic tests, and logistic regression.

Results

The degree of concordance was moderate. Using VA records rather than self-report data to determine veteran status increased homeless veteran prevalence rates by 27% in Columbus and 39% in NYC. Veterans with discordant veteran status (i.e., false positive or false negative) showed lower levels of services use in the VA (both cities) and in the municipal shelter system (NYC only). Younger veterans and women were at higher risk of not being identified as veterans.

Conclusion

Administrative records can help to more accurately identify homeless veterans and to connect them to available services and benefits.

The U.S. Department of Veterans Affairs (VA) has committed to ending homelessness among veterans by 2015. In measuring progress toward this goal, researchers have used data on self-reported veteran status collected from adults in the homeless population through surveys and administrative data on homeless services use.^1–3 However, these self-report data have not been cross-referenced with military or veteran records, and the validity of using these data for estimating the homeless population size is unknown.⁴

Although survey responses from homeless people have been shown as sufficiently accurate and reliable for use in research studies,^5–8 various factors may lead to either under- or overreporting veteran status. Homeless people may embellish or fabricate claims of military service for reasons linked to social desirability effects.⁹ Survey respondents may find the term “veteran” to be misleading. Women, National Guard and Reserve members, and those without combat experience are more likely to incorrectly believe they are not veterans, and people whose military service was not extensive enough to qualify them as veterans (from the perspective of veterans' organizations) may see themselves as veterans. Veterans with a bad conduct discharge or other reasons for concealing military service may deny veteran status to interviewers.¹⁰ Incorrect self-report may also come from inaccurate recording. For example, as veterans are more likely to be male and older than the non-veteran population,¹¹ those collecting homeless management information system (HMIS) data may assume that women or younger adults would not be veterans and, thus, incorrectly denote the status of veterans in these subgroups.

Only a few studies have examined the accuracy of self-reported veteran status. These studies, which have occurred in non-homeless settings¹² or in previous eras,⁹ have found a high level of concordance between self-reported veteran status and administrative records, with inconsistencies consisting mainly of false positives, where the self-reported affirmative veteran status was uncorroborated by the records. We used records from homeless services providers in New York City (NYC) and Columbus, Ohio, and compared them with administrative records of military service and VA enrollment to assess the sensitivity and specificity of the self-reported veteran status and whether there are demographic correlates to inaccurate reporting of veteran status.

METHODS

The study population comprised all adults with records of shelter use in 2008 in either NYC or Columbus. NYC's Department of Homeless Services (DHS) and Columbus's Community Shelter Board (CSB) each maintain databases for more than 85% of all shelter beds within their jurisdictions. Records on adult shelter users in 2008 from these databases, collectively known as HMIS,¹³ were matched with two databases obtained from the VA—one containing records on all people who separated from military service and the other on people who had accessed health-care services, including homeless services, through the VA. The NYC HMIS contained records on 16,346 single adults (veteran status for adults in homeless families was unavailable). The Columbus HMIS contained 5,860 records of adults who were homeless both individually and as part of families. Veteran status was self-reported in both cities and, in NYC, was verified in some cases by DHS through external sources.^14,15 In NYC, veteran status was either marked affirmative or left blank; missing data presumed non-veteran status. In Columbus, veteran status was indicated with either a “yes” or “no,” and the 59 records with missing values (1% of total) were coded non-veteran.

The first VA dataset came from the VA/Department of Defense Identity Repository (VADIR), a VA-based data repository that contains a comprehensive set of records for veterans, people in the military, and their dependents dating back to 1981 (and less complete records prior to 1981). Identifying information in these records was validated with Social Security Administration records. The second VA dataset, coming from the Veterans Health Administration Support Service Center (VSSC), was a roster of veterans who were enrolled for VA health benefits. Inclusion in the VSSC database indicated access to VA health (including homeless) services.

HMIS and VADIR records were linked based on deterministic (exact) and probabilistic matching criteria involving multiple personal identifiers (e.g., Social Security number, date of birth, name, and sex).¹⁶ HMIS records were also matched with VSSC records through a deterministic match of Social Security numbers. Because the VSSC match was less rigorous, matches were only retained if the record in question also indicated a match with either HMIS veteran or VADIR records. The resulting, integrated dataset contained three measures of veteran status: self-report (through HMIS), administrative record (through VADIR), and VA-enrolled (through VSSC). A match with a VADIR record was the gold standard for determining veteran status.

Descriptive statistics on the numbers of veterans identified through the three veteran measures are reported, and diagnostic tests (i.e., sensitivity, -specificity, and positive and negative predictive values) were performed to determine the consistency of veteran classification across datasets. Finally, we used logistic regression to determine differences by virtue of age or gender associated with the likelihood of accurately reporting veteran status. All data management, matching, and analyses were performed using SAS® version 9.3.¹⁷

RESULTS

Table 1 shows the frequencies of veteran status based on HMIS (i.e., self-report) and VADIR (i.e., military record matches) for both Columbus and NYC. By either measure, the proportion of the homeless population constituting veterans was higher in Columbus than in NYC. Altogether, 92.3% and 95.3% of the responses in Columbus and NYC, respectively, concordantly indicated either veteran or non-veteran status in both VADIR and HMIS. This finding led to respectable Kappa statistic scores (measuring agreement between HMIS and VADIR) of 0.66 for Columbus and 0.73 for NYC (data not shown).

Table 1.

Veteran status based on self-report (HMIS) and military records (VADIR) for homeless adults in community shelters: Columbus, Ohio, and New York City, 2008

HMIS = homeless management information system

VADIR = Veterans Affairs/U.S. Department of Defense Identity Repository

However, Table 1 also shows the discordance between HMIS and VADIR to be notable. Among the inconsistent responses in both cities, those who self-reported non-veteran status yet matched a military record (i.e., false negatives) substantially outnumbered those who self-reported as a veteran but had no military record (i.e., false positives). Higher proportions of veterans were identified through VADIR (14.3% and 11.1% for Columbus and NYC, respectively) than through HMIS (11.3% and 8.0% for Columbus and NYC, respectively). This finding means that, if veteran status were based on military record instead of self-report, the rates of veterans among these homeless populations would increase by 27% in Columbus and 39% in NYC. In other words, only 64% of those identified as veterans through VADIR had an affirmative veteran status in the combined HMIS records of the two cities (data not shown).

Table 2 shows the sensitivity measure (i.e., the rate of positively identifying veteran status among actual veterans) to be moderate at best for both cities. Likewise, the positive predictive value (i.e., the rate by which HMIS self-report correctly reflected veteran status) was moderate (79.3%) for Columbus and higher (89.9%) for NYC. The corresponding measures for non-veteran status, specificity, and negative predictive value were substantially higher in both cities despite the higher numbers of false negatives.

Table 2.

Diagnostic tests measuring consistency of veteran classification across self-report (HMIS) and military record (VADIR) data for sheltered homeless populations in Columbus, Ohio, and New York City, 2008

HMIS = homeless management information system

VADIR = Veterans Affairs/U.S. Department of Defense Identity Repository

Logistic regression models using the records from both cities (full-model results not shown) were run using only the discordant observations. Among those with VADIR military service records (n=2,657), both female sex (adjusted odds ratio [AOR] = 2.77; 95% confidence interval [CI] 1.95, 3.94) and younger age (AOR=0.60; 95% CI 0.55, 0.65 for each increasing decade of age) were significantly associated with increased odds of false-negative status, and location (NYC or Columbus) was non-significant. Among those who self-reported being a veteran (n=1,970), neither age nor sex was significantly associated with a false-positive status. Those with self-reported veteran status in Columbus were more than twice as likely as their NYC counterparts (AOR=2.31; 95% CI 0.33, 0.56) to lack a corresponding military service record (data not shown).

Shelter use records, which were available only for the veterans in NYC, showed that those with discordant veteran status (false negative and false positive), on average, had accumulated significantly fewer shelter days (15%–20% fewer days) than veterans with concordant indicators (data not shown). There was no significant difference between those with false-positive and false-negative indications.

The final analysis of this study concerned the extent to which homeless veterans were eligible for VA health and homeless services. Among veterans with military records, in Columbus, 80.5% of those who self-reported being a veteran and 40.7% of those who did not disclose being a veteran had records in the VA health-care system. In NYC, the results were very similar, with 83.1% and 39.6% of those who self-identified as veterans and non-veterans, respectively, showing a matching VA record (data not shown).

DISCUSSION

Setting a goal of ending homelessness among veterans necessitates an accurate means to assess progress toward this goal. However, the current means of assessing the number of veterans among the homeless population relies primarily on self-report of veteran status and is largely unvalidated. The results of this study indicate that reliance on self-report is a potential cause for concern, as the use of self-report led to underassessing the proportion of veterans in the sheltered homeless population by 27% in Columbus and 39% in NYC. Cases of undisclosed military histories (false negatives) were much more commonplace than unconfirmed military histories (false positives) in both cities.

The underspecification of veteran status found in both cities was most pronounced among women and those of younger age. The potential for homelessness among veterans in both of these demographic subgroups has received increased attention, even as little research currently exists on the dynamics and risk factors associated with their homelessness.^18,19 It is unclear from the results of this study why veteran status among these subgroups was more likely to be missed, or even whether underreporting among these subgroups was more a product of veterans not affirmatively disclosing their status or recorders falsely indicating their status.

Limitations

Having generally consistent data from two disparate cities, with different data-collection structures and reporting protocol, enhances the likelihood that similar results would be seen in other jurisdictions. Nonetheless, given that these results represented only two cities, and that there may have been changes in how veteran self-report data have been collected since 2008 (when these HMIS data were collected), this study may not be generalizable to other cities.

CONCLUSION

These results support the widespread use of administrative records to help more accurately identify and reach out to veterans in homeless populations. As more communities have functional HMISs covering their homeless services, there arises more potential for using data matches similar to the one performed in this study to identify homeless veterans, target outreach efforts, and facilitate veterans' access to services and benefits. Several jurisdictions have proposed such matches as part of their 10-year plans to end homelessness,^20,21 and the U.S. Government Accountability Office has proposed such a matching process for monitoring the placement of homeless veterans into U.S. Department of Housing and Urban Development—Veterans Affairs Supportive Housing Program buildings.²² Data access logistics and confidentiality safeguards are concerns with such data matches.²³ However, as automated data systems become more commonplace, implementing data matches should occur on a more routine basis, and the need for relying on self-report data should decrease. Greater use of multiple data sources to monitor the composition and trends of homeless populations, and, specifically, veterans among these populations, stands to play a key role in the VA's goal of ending homelessness among veterans.