Abstract
Objective
To assess the accuracy of self‐reported ambulatory care visits, emergency department (ED) encounters, and overnight hospitalizations in a population‐based sample of homeless adults.
Data Source
Self‐report survey data and administrative health care utilization databases.
Study Design
Self‐reported health care use in the past 12 months was compared to administrative encounter records among 1,163 homeless adults recruited in 2004–2005 from shelters and meal programs in Toronto, Ontario.
Data Extraction Methods
Self‐reported health care use was assessed using a structured face‐to‐face survey. Each participant was linked to administrative databases using a unique personal health number or their first name, last name, sex, and date of birth.
Principal Findings
The sensitivity of self‐report for ambulatory care visits, ED encounters, and overnight hospitalizations was 89, 80, and 73 percent, respectively; specificity was 37, 83, and 91 percent. The mean difference between self‐reported and documented number of encounters in the past 12 months was +1.6 for ambulatory care visits (95 percent CI = 0.4, 2.8), −0.6 for ED encounters (95 percent CI = −0.8, −0.4), and 0.0 for hospitalizations (95 percent CI = 0.0, 0.1).
Conclusions
Adults experiencing homelessness are quite accurate reporters of their use of health care, especially for ED encounters and hospitalizations.
Keywords: Homeless persons, health care utilization, self‐report, administrative data
Homeless individuals are sometimes very high users of health care services (Chambers et al. 2013). Health care systems may therefore have an interest in determining the level of health care utilization among individuals experiencing homelessness and facilitating their appropriate use of health services. Researchers have often used self‐reports to assess health care utilization among people who are homeless (Padgett et al. 1995; O'Toole et al. 1999; Nyamathi, Leake, and Gelberg 2000; Kushel, Vittinghoff, and Haas 2001; Kushel et al. 2002), but the accuracy of these self‐reports is uncertain. Certain conditions that occur at higher than average rates among people experiencing homelessness, such as mental illness, alcohol use, and illicit drug use, may adversely affect their ability to accurately recall their recent health care use.
Confirmation of homeless individuals' self‐reported health care encounters through chart review is difficult when homeless people seek health care at multiple sites that are widely dispersed. The use of administrative data to assess the accuracy of self‐reports has been problematic because many homeless individuals in the United States lack health insurance and thus are not captured in private insurers' databases or Medicare records (Kushel, Vittinghoff, and Haas 2001). Consequently, previous studies of the accuracy of self‐reported health care use among homeless persons have had substantial methodological limitations. Some studies have compared self‐reports to information obtained from key informant service providers (Calsyn et al. 1993, 1997), or from review of medical records at a single clinic or a small number of health care organizations (Gelberg and Siecke 1997; O'Toole et al. 1999; Kee et al. 2007). Other studies have used administrative data to assess the accuracy of self‐reported health care utilization in a selected subset of homeless persons with mental illness or substance use disorders (Bonin et al. 2007; Clifasefi et al. 2011).
A comprehensive assessment of the accuracy of self‐reported use of health care services by people experiencing homelessness would be of substantial benefit for clinicians and researchers who work with this population. In particular, it is useful to determine if individuals are high or low users of ambulatory care, emergency departments (EDs), and inpatient care, as intervention strategies would vary according to the individual's pattern of utilization across these different health care delivery sites. An opportunity to conduct such an assessment arises in Ontario, Canada, where almost all residents are insured by a publicly funded single‐payer health insurance system, and essentially all health care encounters within the province are captured in population‐wide administrative databases. The primary goal of this study was to assess the accuracy of self‐reported ambulatory care visits, ED encounters, and hospitalizations during the past 12 months using comprehensive administrative databases in a population‐based sample of homeless adults in Toronto. A secondary goal was to identify individual characteristics associated with under‐ or overreporting of health care use in this population, as this information would indicate when self‐report data from homeless adults should be used with particular caution. Predictors of under‐ or overreporting were selected according to the Behavioral Model of Health Services Utilization for Vulnerable Populations (Gelberg, Andersen, and Leake 2000). According to this framework, determinants of health care use include the demographic and social structural attributes that predispose individuals to use health services or affect their propensity to use services; the personal, family, and community resources that facilitate or enable the use of services; and the symptoms or health conditions that precipitate the need for health services.
Methods
Study Participants
Toronto (population 2.6 million) is the largest city in Canada; approximately 5,000 people are homeless in Toronto on any given night, and more than 17,000 different individuals use homeless shelters in the city each year (City of Toronto, Shelter, Support, and Housing Administration 2015). A random sample of homeless participants was selected from shelters and meal programs in Toronto over 12 consecutive months in 2004–2005. Recruitment and sampling methods for this study have been described previously (Hwang et al. 2008, 2010; Chiu et al. 2009).
Recruitment was stratified to obtain a 2:1:1 ratio of single adult men (i.e., men without dependent children), single adult women (i.e., women without dependent children), and family adults (i.e., men or women accompanied by a partner and/or dependent children) (Khandor and Mason 2007). Based on a pilot study, we determined that about 90 percent of homeless people in Toronto slept at shelters, while 10 percent did not use shelters but used meal programs (Hwang et al. 2005). We therefore recruited 90 percent of our sample at shelters and the remaining 10 percent at meal programs. Meal program users were eligible if they were homeless but had not used a shelter in the past 7 days.
Homelessness was defined as living within the last 7 days at a shelter, public place, vehicle, abandoned building, or someone else's home, and not having a home of one's own. Participants were excluded if they did not meet our definition of homelessness, were unable to communicate in English, or were unable to provide informed consent. Participants were also excluded if they did not have a valid Ontario health insurance number, as this information was required for linkage to administrative data. All participants provided written informed consent and received $15 for their participation. The Research Ethics Board at St. Michael's Hospital in Toronto provided ethics approval for this study.
For the purposes of recruitment, families were considered as units. In instances where two adults of the same family unit were present, we randomly selected one adult for inclusion in our analysis. Of the 2,516 single adults and family units who were screened, 882 (35 percent) were ineligible to participate, and an additional 443 (18 percent) individuals declined to participate and two were excluded due to duplicate or invalid data. In total, 1,189 unique adults were included in the study, corresponding to a response rate of 73 percent.
Self‐Reported Health Care Use
Self‐reported health care use was assessed using a structured face‐to‐face interview. Participants were asked “In the past 12 months, have you gotten health care from any of the following places?” Response options were: hospital emergency room; hospital where you stayed ≥1 night, not counting staying overnight in the ED; hospital outpatient clinic; doctor or nurse in a shelter, soup kitchen, bus, or other program; community health centres (CHCs); walk‐in clinic; or private doctor's office. Participants were also asked how many times in the past 12 months they obtained health care from each of these sources or if any of these encounters occurred outside of Ontario. As the administrative databases used in this study are limited to Ontario, the self‐reported number of visits for each type of health care was corrected for the number of times visits occurred outside of the province by subtracting the number of self‐reported visits that occurred outside of Ontario from the total number of self‐reported visits for each type of health care. For the purposes of this analysis, outpatient clinics, shelter and other outreach‐based health care, CHCs, walk‐in clinics, and private doctor's offices were all considered forms of ambulatory care.
Documented Health Care Use
To determine documented health care use for the 12 months period prior to each participant's interview, participants were linked to administrative databases using their unique 10‐digit personal health number (94 percent of sample) or, in instances where the participant's health number could not be obtained or was not valid, using their first name, last name, sex, and date of birth (4 percent of sample). In total, linkage was achieved for 1,165 (98 percent) participants. Administrative data were accessed through the Institute for Clinical and Evaluative Services (ICES), an independent, nonprofit organization funded by the Ontario Ministry of Health and Long‐Term Care.
Records of ambulatory care visits with physicians were obtained from the Ontario Health Insurance Plan (OHIP) claims database of services provided by fee‐for‐service physicians to individuals covered under the provincial insurance plan. These data do not capture ambulatory care visits with physicians remunerated under alternative payment plans (Canadian Institute for Health Information 2008). For this reason, we conducted an electronic search of patient encounter databases at five CHCs in Toronto that are specially mandated to provide low‐barrier primary health care services for homeless individuals in their catchment area. These records were combined with fee‐for‐service records from OHIP to construct an ambulatory care dataset. As multiple claims are sometimes submitted for different services provided at a single encounter, records with identical service dates, patient identifiers, and physician identifiers (or CHC facility codes) were counted as single visits.
ED encounters were obtained from the National Ambulatory Care Reporting System, which captures all ED visits in Ontario. Planned or scheduled visits to the ED were excluded, as were duplicate records. Hospitalization records were obtained from the Canadian Institute for Health Information Discharge Abstract Database (CIHI‐DAD) and the Ontario Mental Health Reporting System (OMHRS), which together capture all hospitalizations in Ontario. OMHRS was created in October 2005 to record all adult inpatient admissions to mental health beds in the province of Ontario. Institutions with designated adult inpatient mental health beds that had previously reported to CIHI‐DAD were required to report to both CIHI‐DAD and OMHRS during a dual reporting period from October 1, 2005 to March 31, 2006. These duplicate hospitalization records were identified in CIHI‐DAD using an ICES‐derived key variable and were excluded from the merged hospitalizations dataset.
Predictors of Under‐ or Overreporting Health Care Use
Predisposing, enabling, and need factors were assessed using structured, in‐person interviews at baseline within the framework of the Behavioral Model of Health Services Utilization for Vulnerable Populations (Gelberg, Andersen, and Leake 2000). Predisposing factors included demographic (e.g., age, family status) and social structural attributes that affect the propensity to use services. Enabling factors included personal, family, and community factors (e.g., perceived barriers to care) that impede or facilitate health service use. Need factors included symptoms or conditions (e.g., physical or mental health status) that precipitate service use. While the Behavioral Model was designed to identify factors associated with the use of health care services, it also provides a logical organizing framework of characteristics that may be associated with under‐ or overreporting of health care use.
The presence of alcohol, drug, and mental health problems was assessed using the Addiction Severity Index (ASI) (McGahan et al. 1986; McLellan, Kushner, and Metzger 1992). ASI scores were dichotomized for each subscale (≥0.17 for alcohol problems, ≥0.10 for drug problems, and ≥0.25 for mental health problems) using cut‐off scores for homeless persons (Burt et al. 1999). Propensity to underseek care was assessed on a four‐point scale for seeking health care for: (1) weight loss of more than 10 pounds in a month when not dieting; (2) shortness of breath with light exercise or light work; (3) chest pain when exercising; (4) loss of consciousness, fainting, or passing out; or (5) bleeding other than nosebleeds and not caused by accident or injury (Bindman, Grumbach, and Osmond 1995). One point was assigned for each symptom rated as “a little important” or “not at all important.” Competing priorities were based on difficulty in meeting shelter, food, clothing, washing, or bathroom needs over the past 30 days using a four‐point scale (Gelberg et al. 1997). Participants were classified as having competing priorities if they responded “usually” to any of the five items.
Perceived health status was measured using the validated 12‐item Short Form (SF‐12) Health Survey (Ware, Kosinski, and Keller 1995; Larson 2002). SF‐12 physical component summary (PCS) and mental component summary (MCS) scores were calculated according to the publishers' specifications and were standardized to the general U.S. population (mean = 50, standard deviation = 10), with higher scores representing better overall health status (Ware, Kosinski, and Keller 1995). Chronic health conditions were based items from the National Survey of Homeless Assistance Providers and Clients and included diabetes, anemia, high blood pressure, heart disease or stroke, liver problems including hepatitis, arthritis or joint problems, cancer, problems walking, lost limb or other handicap, and HIV infection or AIDS (Burt et al. 1999).
Statistical Analyses
Two (0.2 percent) participants were excluded because they were missing all data on self‐reported health care use. Two (0.2 percent) additional participants were missing data on the self‐reported number of ED encounters or hospitalizations and were excluded from analyses based on counts of health care utilization.
Paired t‐tests were used to compare the mean difference between the number of self‐reported and documented health care visits to the hypothesized mean population difference of zero for each type of health care. The Spearman rank test was used to assess the nonparametric correlation between self‐reported and documented counts of number of visits. The sensitivity and specificity of self‐report were calculated for each type of health care. Cohen's kappa coefficient was used to determine the strength of the agreement between binary outcomes, according to Landis and Koch (1977).
For ED encounters and hospitalizations, participants were considered to be accurate reporters if their self‐reported number of visits equaled their documented number of visits. For ambulatory care visits, a slightly more lenient definition was used, with accurate reporting defined as an absolute difference between self‐reported and documented use of no more than one visit. Separate logistic regression models were constructed for each type of health care use and for underreporting and overreporting of use. The comparison group was participants who were accurate reporters within each type of health care. Backward stepwise selection was used to identify independent variables associated with underreporting or overreporting, using p = .10 as the significance level for entry into the model and p = .05 as the significance level for removal. Independent variables were assessed for multicollinearity, and no problems were detected. All analyses were performed using SAS 9.2 statistical analysis software (SAS Institute, Cary, NC, USA).
Results
A total of 1,163 homeless participants were included in the analyses, of whom 50.4 percent were single adult men, 25.4 percent were single adult women, and 24.3 percent were family adults. The mean age (±SD) of participants was 36.1 (±12.4) years. Over one‐half (55.7 percent) of homeless participants identified as white race/ethnicity; black (22.4 percent) and Aboriginal (8.2 percent) participants were overrepresented in our sample of homeless adults, while other racial/ethnic minorities (13.7 percent) were underrepresented compared to the general population of Toronto (Chiu et al. 2009). The median self‐reported lifetime duration of homelessness was 2 years.
During the 12‐month period preceding the interview, 976 (83.9 percent) participants self‐reported having ≥1 ambulatory care visit, compared to 941 (80.9 percent) participants whose visits could be confirmed using administrative databases (Table 1). For ED encounters, the corresponding values were 580 (49.9 percent) and 605 (52.0 percent), respectively, and for overnight hospitalizations, 225 (19.4 percent) and 195 (16.8 percent). On average, participants reported a mean of 11.2 ambulatory care visits (standard deviation [SD] = 19.5 visits) in the past 12 months compared to 9.6 visits (SD = 13.9) according to the administrative databases (Table 1). For ED encounters, the corresponding values were 1.4 (SD = 2.8) and 2.0 (SD = 4.5) visits, respectively, and for overnight hospitalizations, 0.3 (SD = 1.1) and 0.3 (SD = 0.9). Study participants reported a significantly higher number of ambulatory care visits than those confirmed by administrative data (mean difference = +1.6; 95 percent confidence interval [CI] = 0.4, 2.8) and a significantly lower number of ED encounters (mean difference = −0.6; 95 percent CI = −0.8, −0.4) (Table 1). The mean difference between the self‐reported and documented number of overnight hospitalizations was nonsignificant at 0.0 (95 percent CI = 0.0, 0.1).
Table 1.
Summary Statistics of Self‐Reported and Documented Health Care Utilization in the Past 12 Months among Homeless Adults
| Self‐Reported Utilization | Documented Utilization | Difference (Self‐Reported Minus Documented) | 95% CI for Mean Difference | Spearman's Rho | |
|---|---|---|---|---|---|
| Ambulatory care visits | |||||
| N (%) | 976 (83.9) | 941 (80.9) | |||
| Number of visits | |||||
| Mean (SD) | 11.2 (19.5) | 9.6 (13.9) | 1.6 (20.1) | 0.4, 2.8a | 0.50b |
| Median (IQR) | 4.0 (11.0) | 5.0 (11.0) | 0.0 (6.0) | ||
| Range | 0–221 | 0–113 | −110 to 197 | ||
| Emergency department encounters | |||||
| N (%) | 580 (49.9) | 605 (52.0) | |||
| Number of visits | |||||
| Mean (SD) | 1.4 (2.8) | 2.0 (4.5) | −0.6 (4.1) | −0.8, −0.4a | 0.70b |
| Median (IQR) | 0.0 (2.0) | 1.0 (2.0) | 0.0 (1.0) | ||
| Range | 0–35 | 0–64 | −54 to 59 | ||
| Hospitalizations | |||||
| N (%) | 225 (19.4) | 195 (16.8) | |||
| Number of visits | |||||
| Mean (SD) | 0.3 (1.1) | 0.3 (0.9) | 0.0 (1.1) | 0.0, 0.1 | 0.60b |
| Median (IQR) | 0.0 (0.0) | 0.0 (0.0) | 0.0 (0.0) | ||
| Range | 0–15 | 0–11 | −10 to 10 | ||
Difference is statistically significantly different from zero according to paired t‐test (p < .05).
Correlation between self‐reported and documented counts of number of health care visits is statistically significant according to nonparametric Spearman rank test (p < .001).
CI, confidence interval; IQR, interquartile range; SD, standard deviation.
Self‐report had good sensitivity (89.0 percent) but poor specificity (37.4 percent) for the presence of any ambulatory care visits (Table 2). Self‐report had higher specificity for ED encounters (83.2 percent) and overnight hospitalizations (91.4 percent) compared to ambulatory care but lower sensitivity (80.3 and 72.8 percent, respectively). The kappa coefficients indicated fair agreement between self‐report and administrative data for the presence of any ambulatory care visits, and substantial agreement for the presence of any ED encounters and hospitalizations (Table 2).
Table 2.
Sensitivity and Specificity of Self‐Report for Any Use of Three Types of Health Care Services in the Past 12 Months among Homeless Adults
| Ambulatory Care Visits | Emergency Department Encounters | Hospitalizations | |
|---|---|---|---|
| Sensitivity (%) | 89.0 | 80.3 | 72.8 |
| Specificity (%) | 37.4 | 83.2 | 91.4 |
| Kappa coefficient (95% CI)a | 0.28 (0.21, 0.35) | 0.63 (0.59, 0.68) | 0.61 (0.55, 0.67) |
| Strength of agreementb | Fair | Substantial | Substantial |
Agreement between self‐reported and documented health care utilization, dichotomized as yes/no.
Defined based on kappa statistic ranges for strength of agreement provided in Landis and Koch (1977).
CI, confidence interval.
The percent of participants with concordance/discordance between their self‐reported and documented number of health care encounters are shown in Table 3 and were used to define accuracy of reporting by type of health care. Overall, 361 (31.0 percent) participants were considered to have accurately reported their number of ambulatory care visits, while 398 (34.2 percent) were considered to have underreported and 404 (34.7 percent) were considered to have overreported their number of ambulatory care visits. For ED encounters, 641 (55.2 percent) were considered accurate reporters and 321 (27.6 percent) and 200 (17.2 percent) were considered under‐ and overreporters, respectively. For hospitalizations, 972 (83.6 percent) were considered accurate reporters and 83 (7.1 percent) and 107 (9.2 percent) were considered under‐ and overreporters, respectively. Overall, 5 (0.4 percent) participants were accurate reporters of both ambulatory care visits and ED encounters, 85 (7.3 percent) were accurate reporters of both ambulatory care visits and hospitalizations, 364 (31.4 percent) of participants were accurate reporters of both ED encounters and hospitalizations, and 247 (21.3 percent) were accurate reporters of all three types of health care. There were 136 (11.7 percent) participants who did not accurately report any type of health care, of whom 22 (16.1 percent) were overreporters and 40 (29.4 percent) were underreporters for all three types of health care.
Table 3.
Percent of Participants by Self‐Reported and Documented Counts of Health Care Encounters in the Past 12 Months among Homeless Adults, for Three Types of Health Carea. (a) Ambulatory Care Visits; (b) Emergency Department Encounters; (c) Hospitalizations
| (a) | Documented Number of Health Care Encounters (%) | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | ≥10 | ||
| Self‐reported number of health care encounters | 0 | 7 | 2 | 1 | 1 | 1 | <1 | 1 | 1 | <1 | <1 | 2 |
| 1 | 4 | 2 | 1 | <1 | <1 | <1 | <1 | <1 | <1 | <1 | 1 | |
| 2 | 1 | 2 | 2 | 1 | 1 | 1 | <1 | 0 | <1 | <1 | 1 | |
| 3 | 2 | 1 | 1 | 1 | 1 | 1 | <1 | <1 | <1 | <1 | 1 | |
| 4 | 1 | 1 | 1 | 1 | <1 | 1 | <1 | <1 | <1 | <1 | 2 | |
| 5 | 1 | <1 | <1 | 1 | <1 | 1 | <1 | 1 | <1 | <1 | 1 | |
| 6 | 1 | <1 | <1 | 1 | <1 | 1 | <1 | <1 | <1 | 1 | 1 | |
| 7 | <1 | <1 | <1 | <1 | <1 | 0 | <1 | <1 | <1 | 0 | 1 | |
| 8 | <1 | <1 | <1 | 0 | <1 | <1 | <1 | 0 | <1 | <1 | 7 | |
| 9 | <1 | <1 | <1 | <1 | <1 | 0 | 0 | 0 | <1 | 0 | 1 | |
| ≥10 | 2 | 2 | 1 | 2 | 1 | 1 | 1 | 1 | 2 | 1 | 19 | |
| (b) | Documented Number of Health Care Encounters (%) | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | ≥10 | ||
| Self‐reported number of health care encounters | 0 | 40 | 7 | 1 | 1 | <1 | <1 | <1 | 0 | <1 | <1 | <1 |
| 1 | 5 | 11 | 2 | 1 | <1 | <1 | <1 | 0 | 0 | 0 | <1 | |
| 2 | 2 | 3 | 3 | 2 | 1 | <1 | <1 | <1 | <1 | <1 | <1 | |
| 3 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | <1 | <1 | <1 | 1 | |
| 4 | 1 | <1 | <1 | <1 | <1 | 1 | <1 | 0 | <1 | <1 | 1 | |
| 5 | 0 | <1 | <1 | <1 | <1 | <1 | <1 | 0 | <1 | <1 | <1 | |
| 6 | <1 | 0 | <1 | <1 | 0 | 0 | 0 | <1 | 0 | <1 | <1 | |
| 7 | 0 | 0 | 0 | <1 | <1 | 0 | 0 | 0 | <1 | 0 | <1 | |
| 8 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | <1 | <1 | <1 | |
| 9 | <1 | 0 | 0 | 0 | 0 | 0 | <1 | 0 | 0 | 0 | 0 | |
| ≥10 | <1 | <1 | 0 | <1 | <1 | <1 | <1 | 0 | <1 | <1 | 1 | |
| (c) | Documented Number of Health Care Encounters (%) | ||||||
|---|---|---|---|---|---|---|---|
| 0 | 1 | 2 | 3 | 4 | ≥5 | ||
| Self‐reported number of health care encounters | 0 | 76 | 3 | 1 | <1 | <1 | <1 |
| 1 | 4 | 7 | 1 | <1 | <1 | <1 | |
| 2 | 1 | 1 | 1 | <1 | <1 | <1 | |
| 3 | 1 | <1 | <1 | <1 | <1 | 0 | |
| 4 | <1 | <1 | 0 | <1 | 0 | <1 | |
| ≥5 | 1 | 0 | <1 | <1 | 0 | <1 | |
Percent of participants with concordance between their self‐reported and documented number of health care encounters are shown on the horizontal in gray shading and were used to define accurate reporters by type of health care. Percent of participants with discordance between their self‐reported and documented counts of health care encounters are shown without shading; those above the horizontal line were considered underreporters of their health care use, while those below the horizontal line were considered overreporters.
Characteristics Associated with Underreporting and Overreporting
Certain characteristics were independently associated with both underreporting and overreporting ambulatory care visits in the final multivariable logistic regression model: greater number of chronic health conditions, mental health problem in the past 30 days, and having a primary care provider (Table 4). Family adult status and competing priorities for subsistence needs were also independently associated with underreporting ambulatory care visits, while lower SF‐12 PCS scores were independently associated with overreporting ambulatory care visits.
Table 4.
Characteristics Associated with Underreporting and Overreporting of Use of Three Types of Health Care Services in the Past 12 Months among Homeless Adults
| Characteristics | Ambulatory Care Visits | Emergency Department Encounters | Hospitalizations | |||
|---|---|---|---|---|---|---|
| Underreport† (N = 398) | Overreport† (N = 404) | Underreport‡ (N = 321) | Overreport‡ (N = 200) | Underreport§ (N = 83) | Overreport§ (N = 107) | |
| AOR (95% CI) | AOR (95% CI) | AOR (95% CI) | AOR (95% CI) | AOR (95% CI) | AOR (95% CI) | |
| Predisposing factors | ||||||
| Demographic group | ||||||
| Single adult male (ref) | 1.00 | ns | ns | ns | ns | ns |
| Single adult female | 1.28 (0.88–1.86) | |||||
| Family adult | 1.82 (1.25–2.64)** | |||||
| Age (years) | ns | ns | ns | ns | ns | ns |
| Lifetime duration of homelessness ≥2 years | ns | ns | 1.37 (1.02–1.84)* | ns | ns | 1.84 (1.17–2.90)** |
| Race/ethnicity | ||||||
| White (ref) | ns | ns | ns | ns | ns | ns |
| Black | ||||||
| Aboriginal | ||||||
| Other visible minority | ||||||
| Born in Canada | ns | ns | 1.68 (1.21–2.33)** | ns | ns | ns |
| Highest level of education | ||||||
| Less than high school | ns | ns | ns | ns | ns | ns |
| High school or equivalent | ||||||
| College/vocational training | ||||||
| History of traumatic brain injury | ns | ns | ns | ns | ns | ns |
| Physical assault in past 12 months | ns | ns | 1.68 (1.22–2.30)** | 2.67 (1.86–3.83)** | ns | 1.98 (1.26–3.13)** |
| Sexual assault in past 12 months | ns | ns | ns | ns | ns | 3.06 (1.59–5.90)** |
| Alcohol problem in past 30 days | ns | ns | ns | ns | 0.55 (0.31–0.97)* | ns |
| Drug problem in past 30 days | ns | ns | ns | ns | ns | ns |
| Mental health problem in past 30 days | 1.55 (1.13–2.14)** | 1.60 (1.16–2.20)** | ns | ns | ns | 1.94 (1.24–3.02)** |
| Propensity to underseek care score | ns | ns | ns | ns | ns | ns |
| Enabling factors | ||||||
| Monthly income | ||||||
| <$500 | ns | ns | ns | ns | ns | ns |
| $500–999 | ||||||
| ≥$1,000 | ||||||
| Has a primary care provider | 1.45 (1.03–2.03)* | 1.77 (1.26–2.49)** | ns | ns | ns | ns |
| Self‐perceived unmet needs for health care | ns | ns | ns | ns | ns | ns |
| Self‐perceived unmet needs for mental health care | ns | ns | ns | ns | ns | ns |
| Has competing priorities for subsistence needs | 3.48 (1.55–7.81)** | ns | ns | ns | ns | ns |
| Need factors | ||||||
| SF‐12 physical component summary score | ns | 0.98 (0.97–1.00)* | 0.96 (0.95–0.98)** | 0.96 (0.94–0.97)** | 0.97 (0.95–0.99)** | ns |
| SF‐12 mental component summary score | ns | ns | 0.99 (0.98–1.00)* | 0.99 (0.97–1.00)* | ns | ns |
| No. of chronic health conditions | ||||||
| None (ref) | 1.00 | 1.00 | ns | ns | ns | 1.00 |
| 1 | 1.77 (1.23–2.55)** | 1.73 (1.20–2.49)** | 0.76 (0.41–1.40) | |||
| 2 | 2.65 (1.69–4.18)** | 2.25 (1.40–3.63)** | 1.04 (0.55–1.98) | |||
| 3 or more | 2.52 (1.54–4.13)** | 2.04 (1.19–3.51)** | 2.51 (1.42–4.44)** | |||
†As compared to reference group of homeless participants who accurately reported number of ambulatory care visits (n = 361).
‡As compared to reference group of homeless participants who accurately reported number of emergency department encounters (n = 641); one participant had missing data on the self‐reported number of emergency department encounters and was excluded from logistic regression analyses.
§As compared to reference group of homeless participants who accurately reported number of hospitalizations (n = 972); one participant had missing data on the self‐reported number of hospitalization and was excluded from logistic regression analyses.
AOR = adjusted odds ratio; ns = not significant (p > .05) in final multivariable logistic regression model.
*p < .05; **p < .01.
Characteristics independently associated with both underreporting and overreporting ED encounters were lower SF‐12 physical and MCS scores and being a victim of physical assault in the past 12 months. A lifetime duration of homelessness ≥2 years and birth in Canada were independently associated with underreporting ED encounters.
For hospitalizations, a lower SF‐12 PCS score and alcohol problem in the past 30 days were independently associated with underreporting hospitalizations. In contrast, a lifetime duration of homelessness of ≥2 years, a greater number of chronic health conditions, being a victim of physical or sexual assault in the past 12 months, and having a mental health problem in the past 30 days were independently associated with overreporting.
Discussion
This study determined the accuracy of self‐reported health care use in a population‐based sample of 1,163 homeless adults as compared to comprehensive administrative databases within a publicly funded system of universal health insurance. Overall, adults experiencing homelessness were quite accurate reporters of their use of health care, suggesting that clinicians and researchers may use self‐reported health care utilization data with relative confidence. The sensitivity of self‐reported health care in the past 12 months was highest for ambulatory care visits and lowest for hospitalizations, while the specificity was highest for hospitalizations and lowest for ambulatory care visits. These findings suggest a gradient effect, such that more frequent and less salient forms of health care (e.g., ambulatory care visits) were associated with higher sensitivity but lower specificity for self‐reported health care use, while less frequent and more salient forms of health care (e.g., hospitalizations) were associated with lower sensitivity but higher specificity (Bhandari and Wagner 2006). While homeless adults self‐reported significantly more ambulatory care visits and fewer ED encounters than were documented in administrative databases, the magnitudes of these differences across all participants, on average, were quite small. The low specificity of self‐report for the presence of any ambulatory care visits, as well as the significantly higher number of self‐reported ambulatory care visits than were documented in administrative databases, may reflect the extent to which homeless individuals in Toronto receive health care from health care providers whose encounters are not captured in the administrative databases used in this study, such as nurses who work at shelters and other outreach settings or physicians who are remunerated under alternative payment plans.
The association of certain individual characteristics with underreporting or overreporting of ED encounters and hospitalizations may be relevant to researchers who are studying specific homeless populations. For example, studies of homeless people who have experienced a recent physical or sexual assault, who have an active mental health problem, or who have been homeless for ≥2 years should take into account the tendency of these individuals to overreport their number of hospitalizations in the past 12 months. Similarly, homeless people who have experienced a recent physical assault appear to provide less accurate self‐reports of their ED use, both in terms of under‐ and overreporting their health care use. In general, factors independently associated with underreporting or overreporting were consistent within each type of health care and across the three types of health care examined in this study. This finding suggests that there is considerable overlap in the reference group of participants considered accurate reporters across the different types of health care. Over 60 percent of study participants were considered accurate reporters for more than one type of health care. Given the degree of overlap between participants considered accurate reporters of health care use across the three types of health care, regression models are more likely identifying factors associated with accurate reporting, rather than underreporting or overreporting of one specific type of health care per se. In general, those participants who were homeless for a shorter duration of time and who had better physical and mental health status were less likely to be under‐ or overreporters of their health care use. Conversely, those who experienced a recent physical or sexual assault or who have an active mental health problem were more likely to be under‐ or overreporters of their health care use.
A number of factors are known to influence accurate health care reporting in the general population, including nonmodifiable factors such age, sex, and cognitive impairment, and modifiable factors such as recall timeframe, survey design, data collection method, and the use of memory aids and probes (Bhandari and Wagner 2006). A systematic review of 42 studies that evaluated the accuracy of self‐reported health care utilization data found that the accuracy of self‐reports improved with shorter recall periods, more salient types of health care such as hospitalizations, and types of health care with less frequent visits (Bhandari and Wagner 2006). These studies were largely based on samples from the general population; however, their findings seem applicable to our population of homeless adults.
Our findings are consistent with the results of the few previous studies that have examined the accuracy of homeless individuals' self‐reported use of health care use, although methodological differences prevent direct comparison. In a sample of 349 homeless adults in Los Angeles who reported at least one visit in the past 12 months to a single specific clinic serving this population, records confirming a visit were obtained for 78 percent of respondents (Gelberg and Siecke 1997). In a study of 373 homeless and housed low‐income adults in Pittsburgh who reported receiving health care in the previous 6 months, researchers were able to obtain medical records documenting receipt of that care in 88 percent of participants (O'Toole et al. 1999). A comparison of self‐reported health service use among 161 homeless adults with chronic medical conditions to medical records at two hospital sites found a sensitivity of 86 percent for ED encounters and 97 percent for hospitalizations (Kee et al. 2007), with the latter figure being substantially higher than that observed in our study. Among 134 homeless individuals with severe alcohol problems, a comparison of self‐reported and documented hospitalizations in the past 30 days revealed only fair agreement and a tendency toward overreporting (Clifasefi et al. 2011). Finally, a study of clients who were homeless and had mental or substance use disorders in Quebec, Canada, found that self‐reported health care use was generally valid when compared to administrative data, but the correlation was greater for psychiatric hospitalizations than for medical hospitalizations (Bonin et al. 2007).
This study has certain limitations. The sampling strategy excluded homeless individuals who use neither shelters nor meal programs; however, the number of such individuals in the homeless population in Toronto is very small (City of Toronto, Shelter, Support, and Housing Administration 2009). Study participants were required to have an Ontario health insurance number, which resulted in the exclusion of individuals who were refugees or refugee claimants, had no legal status in Canada, or had arrived in Ontario within the past 3 months. Our study utilized a recall period of 12 months (one of the most common recall time periods); however, substantially different results might have been obtained using a shorter recall period (Bhandari and Wagner 2006; Clifasefi et al. 2011). Our survey instrument did not differentiate between overnight hospitalizations for medical versus psychiatric reasons; consequently, we were unable to examine accuracy of health care reporting separately for medical and psychiatric hospitalizations. Finally, our administrative database was only able to identify encounters with physicians who submit claims under a fee‐for‐service schedule or who provide services at a select number of CHCs in Toronto, and this may have contributed to an apparent overreporting of ambulatory care visits by study participants.
Conclusions
In conclusion, this study determined the accuracy of self‐reported health care use in a large community‐based sample of homeless adults using comprehensive administrative databases within a publicly funded system of universal health insurance. Although a small number of individuals may have greatly underreported or overreported their use of health care, most individuals experiencing homelessness were quite accurate reporters, particularly for more salient forms of health care, such as ED encounters and hospitalizations. These findings support the use of self‐reports in research studies of health care utilization among homeless individuals and in clinical settings where the goal is to identify high users of ED and inpatient care.
Supporting information
Appendix SA1: Author Matrix.
Acknowledgments
Joint Acknowledgment/Disclosure Statement: This project was supported by operating grants from the Agency for Healthcare Research and Quality (1 R01 HS014129‐01) and the Canadian Institutes of Health Research (MOP‐62736). This study was supported by the Institute for Clinical Evaluative Sciences (ICES), which is funded by an annual grant from the Ontario Ministry of Health and Long‐Term Care. The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the manuscript. The opinions, results, and conclusions reported in this paper are the views of the authors and do not necessarily reflect the views of any of the above named organizations. No endorsement by the Institute for Clinical Evaluative Sciences or the Ontario Ministry of Health and Long‐Term Care should be inferred.
Disclosures: None.
Disclaimers: None.
References
- Bhandari, A. , and Wagner T.. 2006. “Self‐Reported Utilization of Health Care Services: Improving Measurement and Accuracy.” Medical Care Research and Review 63 (2): 217–35. [DOI] [PubMed] [Google Scholar]
- Bindman, A. B. , Grumbach K., and Osmond D.. 1995. “Preventable Hospitalizations and Access to Health Care.” Journal of the American Medical Association 274 (4): 305–11. [PubMed] [Google Scholar]
- Bonin, J. P. , Fournier L., Blais R., Perreault M., and White N. D.. 2007. “Are the Responses of Clients with Psychiatric and Addiction Disorders Using Services for the Homeless Valid?” Canadian Journal of Psychiatry 52 (12): 798–802. [DOI] [PubMed] [Google Scholar]
- Burt, M. R. , Aron L. Y., Douglas T., Valente J., Lee E., and Iwen B.. 1999. Homelessness: Programs and the People They Serve. Findings of the National Survey of Homeless Assistance Providers and Clients (Rep. No. 6‐2‐6‐3). Washington, DC: Interagency Council on the Homeless. [Google Scholar]
- Calsyn, R. J. , Allen G., Morse G. A., Smith R., and Tempelhoff B.. 1993. “Can You Trust Self‐report Data Provided by Homeless Mentally Ill Individuals?” Evaluation Review 17 (3): 353–66. [Google Scholar]
- Calsyn, R. J. , Morse G. A., Klinkenberg W. D., and Trusty M. L.. 1997. “Reliability and Validity of Self‐Report Data of Homeless Mentally Ill Individuals.” Evaluation and Program Planning 20 (1): 47–54. [Google Scholar]
- Canadian Institute for Health Information . 2008. Physicians in Canada: The Status of Alternative Payment Programs 2005–2006. Ottawa: Canadian Institute for Health Information. [Google Scholar]
- Chambers, C. , Chiu S., Katic M., Kiss A., Redelmeier D. A., Levinson W., and Hwang S. W.. 2013. “High Utilizers of Emergency Health Services in a Population‐Based Sample of Homeless Adults.” American Journal of Public Health 103 (S2): S302–10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Chiu, S. , Redelmeier D. A., Tolomiczenko G., Kiss A., and Hwang S. W.. 2009. “The Health of Homeless Immigrants.” Journal of Epidemiology and Community Health 63 (11): 943–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- City of Toronto, Shelter, Support, and Housing Administration . 2009. “Street Needs Assessment Results 2009” [accessed on September 15, 2013]. Available at http://www.toronto.ca/legdocs/mmis/2010/cd/bgrd/backgroundfile-29123.pdf
- City of Toronto, Shelter, Support, and Housing Administration . 2015. “Quick Facts about Homelessness and Social Housing in Toronto” [accessed on May 5, 2015]. Available at http://www1.toronto.ca/city_of_toronto/shelter_support__housing_administration/files/pdf/quickfactsinfographic.pdf
- Clifasefi, S. L. , Collins S. E., Tanzer K., Burlingham B., Hoang S. E., and Larimer M. E.. 2011. “Agreement between Self‐report and Archival Public Service Utilization Data among Chronically Homeless Individuals with Severe Alcohol Problems.” Journal of Community Psychology 39 (6): 631–44. [Google Scholar]
- Gelberg, L. , Andersen R. M., and Leake B. D.. 2000. “The Behavioral Model for Vulnerable Populations: Application to Medical Care Use and Outcomes for Homeless People.” Health Services Research 34 (6): 1273–302. [PMC free article] [PubMed] [Google Scholar]
- Gelberg, L. , and Siecke N.. 1997. “Accuracy of Homeless Adults' Self‐Reports.” Medical Care 35 (3): 287–90. [DOI] [PubMed] [Google Scholar]
- Gelberg, L. , Gallagher T. C., Andersen R. M., and Koegel P.. 1997. “Competing Priorities as a Barrier to Medical Care among Homeless Adults in Los Angeles.” American Journal of Public Health 87 (2): 217–20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hwang, S. W. , Chiu S., Kiss A., Katic M., Tolomiczenko G., Levinson W., Redelmeier D., Szalai J. P., and Cowan L.. 2005. “Use of Meal Programs and Shelters by Homeless People in Toronto [abstract].” Journal of Urban Health 82: ii46. [Google Scholar]
- Hwang, S. W. , Colantonio A., Chiu S., Tolomiczenko G., Kiss A., Cowan L., Redelmeier D. A., and Levinson W.. 2008. “The Effect of Traumatic Brain Injury on the Health of Homeless People.” Canadian Medical Association Journal 179 (8): 779–84. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hwang, S. W. , Ueng J. J., Chiu S., Kiss A., Tolomiczenko G., Cowan L., Levinson W., and Redelmeier D. A.. 2010. “Universal Health Insurance and Health Care Access for Homeless Persons.” American Journal of Public Health 100 (8): 1454–61. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kee, R. , Sadowski L., Buchanan D., and Garcia D.. 2007. “Validity of Self‐Reported Health Service Utilization Data in the Homeless” [abstract]. 135th APHA Annual Meeting & Exposition, Washington, DC, November 3–7, 2007.
- Khandor, E. , and Mason K.. 2007. The Street Health Report 2007. Toronto: Street Health. [Google Scholar]
- Kushel, M. B. , Vittinghoff E., and Haas J. S.. 2001. “Factors Associated with the Health Care Utilization of Homeless Persons.” Journal of the American Medical Association 285 (2): 200–6. [DOI] [PubMed] [Google Scholar]
- Kushel, M. B. , Perry S., Bangsberg D., Clark R., and Moss A. R.. 2002. “Emergency Department Use among the Homeless and Marginally Housed: Results from a Community‐Based Study.” American Journal of Public Health 92 (5): 778–84. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Landis, J. R. , and Koch G. G.. 1977. “The Measurement of Observer Agreement for Categorical Data.” Biometrics 33 (1): 159–74. [PubMed] [Google Scholar]
- Larson, C. O. 2002. “Use of the SF‐12 Instrument for Measuring the Health of Homeless Persons.” Health Services Research 37 (3): 733–50. [DOI] [PMC free article] [PubMed] [Google Scholar]
- McGahan, P. L. , Griffith J. A., Parente R., and McLellan A. T.. 1986. Addiction Severity Index: Composite Scores Manual. Philadelphia: The University of Pennsylvania/Veterans Administration Centre for Studies of Addiction. [Google Scholar]
- McLellan, A. T. , Kushner H., and Metzger D.. 1992. “The Fifth Edition of the Addiction Severity Index.” Journal of Substance Abuse Treatment 9 (3): 199–213. [DOI] [PubMed] [Google Scholar]
- Nyamathi, A. M. , Leake B., and Gelberg L.. 2000. “Sheltered versus Nonsheltered Homeless Women.” Journal of General Internal Medicine 15 (8): 565–72. [DOI] [PMC free article] [PubMed] [Google Scholar]
- O'Toole, T. P. , Gibbon J. L., Hanusa B. H., and Fine M. J.. 1999. “Utilization of Health Care Services among Subgroups of Urban Homeless and Housed Poor.” Journal of Health Politics, Policy and Law 24 (1): 91–114. [DOI] [PubMed] [Google Scholar]
- Padgett, D. K. , Struening E. L., Andrews H., and Pittman J.. 1995. “Predictors of Emergency Room Use by Homeless Adults in New York City: The Influence of Predisposing, Enabling and Need Factors.” Social Science & Medicine 41 (4): 547–56. [DOI] [PubMed] [Google Scholar]
- Ware, J. E. , Kosinski M., and Keller D.. 1995. SF‐12: How to Score the SF‐12 Physical and Mental Health Summary Scales, Vol. 2. Boston: The Health Institute, New England Medical Center. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Appendix SA1: Author Matrix.