Skip to main content
VA Author Manuscripts logoLink to VA Author Manuscripts
. Author manuscript; available in PMC: 2023 Feb 21.
Published in final edited form as: Health Soc Care Community. 2022 Jul 22;30(6):e5027–e5037. doi: 10.1111/hsc.13918

What do Veterans with homeless experience want us to know that we are not asking? A qualitative content analysis of comments from a national survey of health care experience

Allyson L Varley 1, April Hoge 1, Kevin R Riggs 1,2, Aerin deRussy 1, Audrey L Jones 3,4, Erika L Austin 5, Sonya Gabrielian 6,7, Lillian Gelberg 6,7, Adam J Gordon 3,4, John R Blosnich 8,9, Ann Elizabeth Montgomery 1,5, Stefan G Kertesz 1,2
PMCID: PMC9942008  NIHMSID: NIHMS1864917  PMID: 35866310

Abstract

Surveys of people who experience homelessness can portray their life and healthcare experiences with a level of statistical precision; however, few have explored how the very same surveys can deliver qualitative insights as well. In responding to surveys, people experiencing homelessness can use the margins to highlight health and social concerns that investigators failed to anticipate that standard question batteries miss. This study describes the unprompted comments of a large national survey of Veterans with homeless experiences. The Primary Care Quality-Homeless Services Tailoring (PCQ-HOST) survey presented 85 close-ended items to solicit social and psychological experiences, health conditions, and patient ratings of primary care. Among 5377 Veterans responding to the paper survey, 657 (12%) offered 1933 unprompted comments across nearly all domains queried. Using a team-based content analysis approach, we coded and organized survey comments by survey domain, and identified emergent themes. Respondents used comments for many purposes. They noted when questions called for more nuanced responses than those allowed, especially “sometimes” or “not applicable” on sensitive questions, such as substance use, where recovery status was not queried. On such matters, the options of “no” and “yes” failed to capture important contextual and historical information that mattered to respondents. Respondents also elaborated on negative and positive care experiences, often naming specific clinics or clinicians. This study highlights the degree to which members of vulnerable populations, who participate in survey research, want researchers to know the reasons behind their responses and topics (like chronic pain and substance use disorders) that could benefit from open-ended response options.

Keywords: homelessness, primary care, qualitative methods, survey research

Background

Homelessness has major implications for public health, community well-being, and the health of the individuals experiencing it. Studies have demonstrated that homelessness is associated with higher mortality, increased hospital use, and poorer overall health (Baggett et al., 2013; Baggett et al., 2010; Kushel et al., 2001; Moore & Rosenheck, 2016; Roncarati et al., 2018; Zhang et al., 2018). Individuals experiencing homelessness are increasingly older and have multiple, often longstanding, medical, psychological, and symptom-related needs (Beijer et al., 2012; Patanwala et al., 2017). In turn, they face unique challenges in obtaining appropriate primary care: perceived stigma, lack of trust, negative experiences with care coordination, as well as logistical barriers such as payment barriers, lack of transportation, and pressure from competing priorities (Gelberg et al., 1997; Gruenewald et al., 2018; Jones et al., 2017; Ramsay et al., 2019; Wen et al., 2007).

Patients experiencing homelessness often report poor experiences with health care (Baggett et al., 2010; Jego et al., 2018; Kushel et al., 2001; Riggs et al., 2020). Recent research focuses on whether tailoring services for homeless experience can engage and improve care for this population (Gabrielian et al., 2017; O’Toole et al., 2016). Such efforts to tailor care include a Veterans Health Administration (VHA) clinic paradigm, called Homeless Patient Aligned Care Teams (H-PACTs), which may improve patient experience and satisfaction (Kertesz et al., 2021). The VHA is the largest integrated healthcare system in the U.S., providing health and social services for 9 million Veteran enrollees across 179 medical centers and 1,245 outpatient clinics.

Understanding whether any new service makes a difference to a historically vulnerable population (e.g., persons with limited material and social resources; medical comorbidity) depends on learning not just about their experiences with the service but also about the health and social concerns of the population itself. This can be done qualitatively, through semi-structured or cognitive interviews, or through quantitative surveys (McCallum et al., 2019; Montgomery et al., 2014; Ramsay et al., 2019; Sestito et al., 2018; Varley et al., 2020). The statistical precision afforded by validated quantitative surveys comes with a notable risk: investigators may fail to query important aspects of life experience, or cardinal determinants of care experience. Careful study of unprompted qualitative responses to a widely distributed survey can mitigate the risks of oversight and deliver a more holistic understanding of concepts queried in the survey (Creswell, 2013; Teddlie & Tashakkori, 2009). What patients write into survey margins can highlight shortfalls of the survey and direct future inquiry (Clayton et al., 1999; Huppertz & Smith, 2014). Such commentary may even force a reconsideration of the questions that service leaders or researchers propose to study or how to study these questions.

Prior studies that explored unprompted qualitative comments on patient satisfaction surveys found that respondents offered additional comments on negative care experiences (Fifolt et al., 2021). To our knowledge, however, there has not previously been a qualitative analysis of unprompted responses from a large sample of respondents with homeless experiences. This work will explore and describe the content of unprompted comments offered by Veterans with homeless experience in the margins of an 85-item survey that covered personal and social history, as well as health care experiences (deRussy et al., 2021).

Methods

Ethical considerations

This project was approved by the VHA Central Institutional Review Board and VHA Research and Development committees at participating centers.

Setting

The setting for this study was a survey of clients of VHA’s H-PACTs, one that queried many health and social factors, along with ratings of primary care. The H-PACTs represent a central VHA effort to remediate health care challenges faced by Veterans with homeless experience (O’Toole et al., 2016). As of 2021, there are over 60 H-PACTs in operation. Though H-PACTs vary in aspects of care design and delivery, they typically provide open-access/walk-in care and one-stop coordination of health and social services. The parent study for this research, the Primary Care Quality-Homeless Services Tailoring (PCQ-HOST), sought to compare experiences of Veterans with homeless experience receiving primary care in the VHA’s mainstream primary care settings to those using H-PACTs through mailed and telephone-administered surveys.

Data Source

The present analysis focuses on 5,337 Veterans who responded by mail to a paper-based survey and therefore had opportunity to annotate their responses. The survey’s 85 items contained the Patient Care Quality-Homelessness (PCQ-H) measure, a 33-item patient care experience questionnaire tailored to homeless-specific needs, other validated measures related to physical and mental health, housing, social support, substance use, criminal justice status, and demographic information, as well as select questions developed by the research team (Kertesz et al., 2014; Varley et al., 2020). See Table 1 for survey topics and number of questions.

Table 1.

PCQ-H Survey Domains

Survey Domain Number of Items Topic of Items with Written Comments*
A. Where and how often you get primary care (Primary Care) 12 Primary care
B. Current and past living situation 8 Homelessness
C. Primary care experience (PCQ-H) 33 Primary care
D. Health and quality of life 15 Health / quality of life
E. Mental health, social environment, and criminal justice involvement 16 Health / quality of life
Social environment
F. Social Support 8 Social environment
G. Background/demographic questions 10 Sociodemographics

Analysis

We copied survey respondents’ unprompted written comments verbatim into NVIVO 12. Words that were not legible were noted in the database. Using a content analysis approach, we organized the database by survey domain first (i.e., survey sections that covered similar topics like demographics, substance use, primary care experience, etc.), and then by individual item (Bengtsson, 2016; Erlingsson & Brysiewicz, 2017). We enumerated comment frequency by survey domain. Because of the large number of comments, our content analysis focused on survey items that drew at least 20 comments. We then reviewed all comments by survey domain, identified emergent recurring themes, and assigned a theme to each comment through a team coding approach. Coding was done by the team weekly and all data were coded by at least three team members collaboratively. Each comment was discussed until consensus was met for an appropriate code or theme. We then created and reviewed a coding matrix for each survey domain across the codebook to identify the most frequently occurring themes (Brooks et al., 2015; Gale et al., 2019). Baseline descriptive quantitative data (means, standard deviations, frequencies, and percentages) were calculated for individuals that left a comment (commenters) and individuals that did not leave any unprompted comments (non-commenters) on the mail-based survey.

Results

Of the 5,337 respondents, 657 (12%) offered a total of 1,933 comments. Table 2 describes the characteristics of the respondents who left unprompted comments compared to those who did not. Among the respondents who left unprompted comments, the average age was 59, 17.1% reported current homelessness at the time of survey administration, and half (50.2%) self-reported severe chronic pain. When compared to non-commentors, commentors were more likely to report homelessness (current, over time, and unsheltered experience) and severe chronic pain. 18 of 85 survey items had at least 20 comments and were included in the content analysis. Figure 1 depicts the study team’s coding scheme. The domains and survey questions most frequently commented upon by participants are outlined in Table 3 and described below.

Table 2.

Characteristics of Commenters and Non-Commenters

Characteristics* Overall
N=5337
Commenters
N=657
Non-Commenters
N=4680

No. (%) No. (%) No. (%)

Demographics

Age, M(SD) 58.9 (10.7) 59.4 (10.9) 58.9 (10.7)
Gender
Male 4757 (90.5) 564 (87.9) 4193 (90.8)
Female 479 (9.1) 75 (11.7) 404 (8.8)
Other 23 (0.4) 3 (0.5) 20 (0.4)
Hispanic or Latinx 551 (10.6) 75 (12.0) 476 (10.4)
Race
White 2224 (45.4) 300 (51.6) 1924 (44.5)
African American 2096 (42.8) 179 (30.8) 1917 (44.4)
American Indian or Alaska 69 (1.6)
Native 89 (1.8) 20 (3.4)
Asian Pacific Islander 80 (1.6) 9 (1.6) 71 (1.6)
Multiple Race 412 (8.4) 73 (12.6) 339 (7.9)

Housing Status

Currently homeless 677 (12.7) 112 (17.1) 565 (12.1)
Chronic homelessness 933 (17.5) 159 (24.2) 774 (16.5)
Unsheltered experience, last 6 months 755 (14.2) 116 (17.7) 639 (13.7)
Income <$1000/month 2113 (40.9) 272 (43.4) 1841 (40.5)

Substance Use and Pain

Drug problem§ 726 (13.9) 97 (15.2) 629 (13.7)
Alcohol problem§ 1511 (28.9) 175 (27.4) 1336 (29.1)
Personal overdose experience in last 3 yearsǁ 355 (6.7) 63 (9.8) 292 (6.3)
Severe chronic pain 2040 (38.2) 330 (50.2) 1710 (36.5)

Notes

*

Missing information from survey response: gender 78, ethnicity 136, race 436, income 164, drug problem 107, alcohol problem 107, overdose 71

Chronic homelessness: If a participant affirms either (a) having 4 separate instances of homelessness in the last 3 years or (b) affirming current homelessness with longest episode >1 year.

Unsheltered experience: Participant spent 1 or more nights on the street, in a car, in an abandoned building in the 6 months prior to the survey

§

Alcohol or drug problem is based on the Two-Item Conjoint Screening test, pertaining to the last 12 months.

ǁ

Overdose: Affirming overdose on alcohol or drugs that necessitated immediate medical care in the 3 years prior to survey completion

Severe Chronic Pain: Participants who reported having bodily pain that of more than 3 months duration coupled with current pain ≥7 on a 0–10 scale

Figure 1.

Figure 1.

Displays the overall coding scheme used to categorize text. On the inside of the figure, we list the themes that were developed based on the existing survey domains. On the top of the figure, we list themes that emerged across survey domains.

Table 3.

Questions with most comments by section and examples

Survey Topic Survey Item Or Section Number of comments

Primary Care Does the distance you have to travel make it difficult to receive care at your clinic? 61
Introduction to primary care questions and PCQ-H 40
People often see many care providers in the VA. In the last 6 months, have you seen any of the following PCP, Mental Health, Specialist, Women’s health, Housing, Group SUD MH, Other? 29

Substance Use Have you felt you wanted or needed to cut down on your drinking or drug use in the last 12 months? If yes, what substance? 108
In the last 12 months, have you ever drunk alcohol or used drugs more than you meant to? If yes, what substance? 41
In the last 3 years, have you had an overdose where you needed to go to the emergency room or get medical care right away. If yes, what were you taking at the time? 33

Health and Quality of Life In the last 30 days, have you had a significant period of time in which you have been prescribed medication for any psychological or emotional problems? 31
Do you currently have bodily pain that has lasted for more than 3 months? 29
Please circle the number that best describes your pain on average in the past week. 28

Homelessness Are you currently homeless?
47
Thinking about the last 3 years, how many times have you been homeless? 41
Thinking about the last 3 years, how many times have you been homeless? 36

Social Environment In general, how would you rate your satisfaction with your social activities and relationships? 34
I have someone who I can borrow $20 from if I need it. 29
I feel isolated from others.
25
Sociodemographics How hard is it for you to pay for the very basics like food and heating? 36
In the last 6 months, what has been your typical monthly income from all sources before taxes? 36
What was the highest level of school you completed? 30

General Comments 95

The results below summarize comments primarily by survey domain. The last four sections concern themes that were emergent and crossed domains of the survey, including two that reflected a degree of frustration with the survey itself: response options of sometimes and not applicable.

Deductive Themes Based on Survey Domains

Primary Care

Most comments in the Primary Care survey domain provided extra detail on what clinical team and/or who respondents were referring to in their responses. Other comments focused on care access and experience. Within this topic, the most frequently commented item was, “Does the distance you have to travel make it difficult to receive care at [CLINIC NAME]?” Participants frequently noted the type of transportation they used to see their doctor. Veterans who responded “no” to problems with travel listed both personal and public transportation methods to explain why they had no problems related to transportation. For example, one respondent wrote, “I use veterans medical transportation service.”

Substance Use

Survey respondents frequently commented on three questions related to substance use. Respondents wrote out the specific substances they had used, including cannabis, specific alcohol types (such as beer), spice, methamphetamine, and tobacco. Other comments appeared to explain why the participant used alcohol or drugs, including for pain control (n=10) in response to care changes. For example, one Veteran commented, “Marijuana to ease with my pain. Because they took my pain meds.” Respondents frequently emphasized that they did not use drugs or alcohol: with responses like “I don’t consume alcohol or use drugs.” Those that made comments about previously stopping substance use often explained when they had stopped, suggesting a value placed on recovery, which the survey items did not explicitly query. For example, one comment said, “I been clean and sober for over 13 years.”

Health and Quality of Life

For survey items focused on Health and Quality of Life, respondents commented most on the question, “In the last 30 days, have you had a significant period of time in which you have been prescribed medication for any psychological or emotional problems?” derived from the Addiction Severity Index (McLellan et al., 1992). In response to this item, respondents volunteered their diagnosis and medication types. For example, “depression” and “anxiety” were frequently written out next to this question. Survey respondents also provided time references like, “Paxil for 15 years.”

Pain

Questions related to pain drew comments from 65 respondents. On one item assessing the presence of chronic pain—“Do you currently have bodily pain that has lasted for more than 3 months?”—survey respondents often commented on the reasons for pain: “Sciatica,” “Screw in knee while in US Airforce,” and “arthritis in hands.” They inserted additional detail on the length of time past 3 months ranging from 1 to 46 years. When asked to rate the degree to which pain had interfered with their enjoyment in life on a scale from 0 to 10, some Veterans wrote in the activities they were unable to do, like sleeping or jogging. Many commented that their pain exceeded the maximum on the provided scale (e.g., “plus 4” and “11+”).

Housing

Respondents offered a great deal of descriptive information on questions related to Homelessness as well. Although duration ranges (no nights, between 1 and 6 nights, or 7 or more nights) were offered as response options, 21 Veterans reported a specific duration, ranging from 2 months to 11 years. Next to one item that asked, “are you currently homeless?”, written comments mentioned current housing, such as “in transitional housing” and “HUD-VASH” (referencing a permanent supportive housing program). Other comments suggested concerns about their housing status changing “soon.” For instance, one Veteran wrote, “Almost soon if they do not help me” in response to the question about current homelessness.

Social Environment

For survey items assessing the Social Environment, comments often elaborated sources of social support. Some reported sources the investigative team had not anticipated, like dogs, God, and their VHA health care providers. Other respondents left comments like, “I like being alone” and reported satisfaction with their social support.

Sociodemographics

On Sociodemographic items, respondents added relevant details. When asked about the ability to pay bills, many explained why they could not (e.g., “because of car insurance and car payment”). When asked about education, where the provided response options included high school and/or college, respondents commented on specific degrees, trade certifications, and the police academy. When asked about income and financial support, they detailed programs and benefits through both VHA and non-VHA sources and, if unemployed, many noted why they were unable to work (e.g., “disability”).

Overarching Emergent Themes

Concerns and Complaints

The study team recorded many (n= 95) general comments not related to any survey item, most commonly at the beginning or end of the survey. Compliments or complaints about the VHA hospital or clinic where they received care were common. Some complaints reflected broad perceptions of the VHA, while others profiled specific situations or needs. For example, when asked about primary care experiences, some Veterans declared concerns about certain clinics like, “I don’t think [community clinic] and [hospital] have very good communication.” Others described specific situations or difficulties they were currently experiencing:

“I have been trying to contact my health care pro for over a week now. 7 days ago I filled out a request form to get a call or talk to her and I am still waiting, the nurse says it is on her desk but nothing. That’s how the service is around here.”

Many comments coupled compliments with complaints. For example, respondents expressed their satisfaction with their primary care clinic, but had complaints about specialty care clinicians or clinics. For example:

“My primary care is great, but the specialty clinics suck. When I lived in [location], the VA’s worked on my lower back. I’ve been requesting surgery, but they won’t do it? However, once they recommended bariatric surgery, I got a consult within a week. BS!”

Compliments and Praise

Respondents also used the blank space on the paper survey to share compliments about their clinicians and care experiences. They expressed gratitude for their care such as, “my care has been excellent” and “Thank you H-PACT staff!!” Many comments represented were general compliments about the health system or medical center. Other compliments were more specific. For example, one Veteran described multiple services: “Thank you! The women Vets outreach/social workers HUD/VASH & Mental Health have helped me very much to be as secure as I now am.” Respondents also wrote out compliments specific to their primary care clinician:

“To Whom it May Concern, [clinician name] is the best doctor I have ever had. It would be a shame if the VA was to ever lose her. She cares very deeply about her patients and is very devoted to meeting their needs. Please do whatever it takes to keep her on staff at the VA.”

Two Veterans commented that they were so satisfied with their care, they wished to return the $1 initial survey incentive. One comment said, “I filled this out for [clinician name] and [clinician name]. BEST EVER. Sending back your dollar” (referring to the incentive payment enclosed with the original survey).

Not Applicable

“Not applicable” was one of the most frequent comments. This response was provided most often in response to questions related to substance use, even though “no” was a provided response option. For one item from the Two-Item Conjoint Screener (Brown et al., 2001)—“Have you felt you wanted or needed to cut down on your drinking or drug use in the last 12 months?”—95 of 108 written comments declared the item not applicable. Other comments (n= 24) affirmed the respondent to be in recovery or expressed pride in recovery: one Veteran wrote, “I am in A.A. 4 years no drinking or drugs for 30 years. I think about alcohol every day, and what can happen if I start drinking again or pick up that first drink.” It was common for respondents to describe the duration of time they had been in recovery, varying from a few months to 35 years. Two additional survey items referred to substance use: “In the last 12 months, have you ever drunk alcohol or used drugs more than you meant to?” and “In the last 3 years, have you had an overdose where you needed to go to the emergency room or get medical care right away?” Although, “no” was an option, respondents often commented that the questions were “not applicable.” Across all three of these questions, some “not applicable” comments to substance use items (n=55 comments) declared that they did not use substances at all. This pattern continued when asked about the use of psychiatric medications in the past 30 days; 12 comments stated, “Not applicable”, even when checking “No” was an option.

Sometimes

Like “not applicable,” “sometimes” was frequently written next to yes/no questions, suggesting discomfort with static accounts of well-being, care experiences or access to resources. For example, in a question about problems with transportation for care, the comment, “sometimes,” appeared 26 times next to this question, signaling that the response options of “yes” and “no” failed to capture irregularities in access to transportation. The invocation of “sometimes” appeared 34 times across four survey items related to Social Environment, suggesting that social support may vary over time. Respondents also commented, “at times,” to describe fluctuations in health states. When asked to rate their health in general, some respondents wanted to select more than one option. For example, one Veteran endorsed “fair,” but then commented, “but at times my health is poor.”

Discussion

In a mail-based survey of 5,337 Veterans with homeless experience covering health and social functioning as well as health care experiences, 657 (1 in 8) offered unprompted comments in the margins of the paper questionnaire. The high frequency of these comments and their content suggested that standard, validated survey components are not only poised to miss matters of importance to this population but also at risk of obscuring important experiences, particularly for a very vulnerable patient group. Overall, respondents used comments to tell the research team when questions called for more nuanced response options for aspects of social experience, health status, substance use history, and pain: hence, they inserted “sometimes” or “not applicable” for yes/no questions. Binary response options may do an injustice to complex experiences, particularly when matters of both pain and substance use are subject to stigma. Querying such sensitive matters, without providing survey respondents the opportunity to explain, may result in incomplete information, missing responses, or even distress for the respondent.

Among comments that conveyed a degree of passion or intensity, some stood out. When delivering feedback on their experience of care, Veterans specified which clinician they had in mind, even though the PCQ-H care questionnaire did not ask. Patients often see multiple providers in team-based clinics. They know who they are thinking about, even when survey designers choose not to ask, as was the case for our survey and is also the case for the widely used Consumer Assessment of Health Plans (CAHPS). Questions about care experiences should take this into consideration.

A large proportion of the comments on substance use (95 of 108) underscored that “yes” and “no” failed to capture the nuances of remission, particularly where reduced use may designate viable recovery, although understandings of recovery remain contested (Witkiewitz et al., 2021). Similar issues have been reported in other surveys like the AUDIT-C (Broyles et al., 2011). Brief, validated assessments of substance use, although convenient for survey research, fail to ask about recovery and remission. This is a problem, as reduced use and remission represent the most common outcome for substance use disorder, a stigmatized condition (Brown et al., 2001; Kelly et al., 2017). Previous research has demonstrated the importance of language around recovery (Botticelli & Koh, 2016). Surveys for stigmatized populations, including persons who are homeless, may invite respondents’ frustration, however inadvertently, by asking about substance use without allowing response options that reflect non-use or recovery.

Though our survey did not query pain-related health care, many Veterans volunteered comments concerning both their pain and its care. Sometimes comments appeared next to questions about pain, but others appeared elsewhere, such as near the substance use items. However, the frequency of these comments suggests that future surveys should investigate the experience of pain and pain-related care experiences more deeply; this aligns with research suggesting that aspects of the care environment bear heavily on subjective pain experience (Benintendi et al., 2021; Penn et al., 2020).

Respondents’ frequent insertion of the word “sometimes” or its equivalent signified a problem with surveys’ use of “yes/no” questions. A desire to reduce respondent burden led us to “yes/no” response options for some items. For many sensitive topics, particularly in a population that may feel unheard, the binary choices invited frustration. We believe that additional pilot testing or cognitive interviews may help improve survey validity and reliability, as well as the experiential quality of survey participation (Boness & Sher, 2020).

The present study also reinforces a potential need for researchers, particularly those who work with disadvantaged populations, to consider co-production of research tools with prospective participants. A community-based participatory research approach can help to ensure surveys adequately capture the patient experience. It can also ensure that analyses and dissemination of research data are relevant and patient-driven (Jull et al., 2017). This is particularly important for persons with homeless experience, who often have unique healthcare and social needs (Baggett et al., 2010; Cusack & Montgomery, 2018; Desai & Rosenheck, 2005; Kertesz et al., 2009; Kiser & Hulton, 2018; Midboe et al., 2019; Wen et al., 2007).

Our findings align with other studies exploring unprompted written comments on quantitative care experience surveys in general outpatients; we are unaware of other studies of written comments on large surveys on the social and health care experiences of vulnerable populations. For example, reports about comments on the CAHPS survey also found descriptions of positive or negative experiences with specific clinicians (Fifolt et al., 2021; Huppertz & Smith, 2014). Substance use, a topic where there may be a remission or reduction, drew many comments. This finding echoes, in some ways, a study of sexual functioning among patients with prostate cancer, where questions on sexual functioning drew comments, often from persons for whom sexual activity had ceased (Talvitie et al., 2022). Comments like these can help survey researchers identify questions that require further refinement.

This study has limitations. The 429 respondents who completed the survey via telephone did not have an opportunity to leave unprompted comments. Also, the mailed survey packets did not solicit the written comments. More people likely would have registered concerns, clarifications, and elaborations had such a solicitation been made. Not requesting such comments could have biased the types of clarifications we received, potentially toward those with stronger emotional content. Additionally, there were differences in characteristics between commentors and non-commentors. Thus, these insights may not reflect important concerns for persons who did not offer qualitative comments on this survey.

Conclusions and implications

This study offers lessons learned for researchers engaged with vulnerable populations and social determinants of health. First, even within the confines of an 85-item survey and a very small financial incentive, participants wanted to ensure that their experiences were not unduly distilled by standardized survey response options. Strong and sometimes impassioned commentary from respondents reinforced a concern that the validated survey questions offered by investigators may not be the right ones. Respondents might tell us more if given the chance.

The present study also highlights the degree to which people with social disadvantage who participate in health systems research would like researchers to know the reasons behind their responses. For people experiencing homelessness, responding “yes” or “no” is often not enough. Qualitative analysis of comments on quantitative surveys can help identify questions that need improvement. Moreover, adding open-ended text response option, or conducting qualitative interviews with a subset of respondents, could help improve future surveys by giving respondents space to provide context to their survey responses.

What is known about this topic?

  1. Understanding patient perspectives can help improve care.

  2. Quantitative data from surveys can provide statistical precision but may miss key patient perspectives.

  3. The content that patients write into survey margins can highlight shortfalls of a survey and point toward future areas of inquiry.

What this paper adds:

  1. Veterans with homeless experience want to provide additional detail about their lives and care experiences in ways that transcend the boundaries of close-ended survey questions.

  2. Questions on substance use proved especially likely to draw comments that went beyond the permitted response options, often to declare that the respondent was in recovery.

  3. Respondents frequently clarified aspects of their care experiences related to pain, pain care, transportation, and experiences of homelessness.

Acknowledgements

Thank you to the Veterans that participated in the survey.

Funding

This material is based upon work supported by the Department of Veterans Affairs (VA) Health Services Research and Development (IIR 15-095). Dr. Jones is supported by a Career Development Award from VA Health Services Research & Development (CDA 19-233, Award No IK2HX003090). Dr. Gabrielian is supported in part by a Career Development Award from VA Health Services Research & Development (CDA 15-074). Infrastructure support was provided to Drs. Jones and Gordon from Informatics, Decision-Enhancement and Analytic Sciences (IDEAS) Center (CIN 13-414) and the Vulnerable Veteran Innovative PACT (VIP) Initiative. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government.

Footnotes

Disclosures

All authors were financially affiliated with the U.S. Department of Veterans Affairs through employment or contract during the preparation of this manuscript. Views expressed in this article are those of the authors alone and do not represent formal positions of the U.S. Department of Veterans Affairs. There are no other conflicts of interest relevant to the work under consideration. Outside of the submitted work, ALV has a financial relationship with Heart Rhythm Clinical Research Solutions, LLC; AJG and SGK have a financial relationship with UpToDate, Inc. SGK owns stock in Thermo Fisher Scientific and Zimmer Biomet.

Data Availability

Research data are not shared.

References

  1. Baggett TP, Hwang SW, O’Connell JJ, Porneala BC, Stringfellow EJ, Orav EJ, Singer DE, & Rigotti NA (2013). Mortality among homeless adults in Boston: shifts in causes of death over a 15-year period. JAMA Internal Medicine, 173(3), 189–195. 10.1001/jamainternmed.2013.1604 [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Baggett TP, O’Connell JJ, Singer DE, & Rigotti NA (2010). The unmet health care needs of homeless adults: a national study American Journal of Public Health, 100(7), 1326–1333. 10.2105/AJPH.2009.180109 [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Beijer U, Wolf A, & Fazel S (2012). Prevalence of tuberculosis, hepatitis C virus, and HIV in homeless people: a systematic review and meta-analysis. Lancet Infectious Diseases, 12(11), 859–870. 10.1016/s1473-3099(12)70177-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Bengtsson M (2016). How to plan and perform a qualitative study using content analysis. NursingPlus Open, 2, 8–14. 10.1016/j.npls.2016.01.001 [DOI] [Google Scholar]
  5. Benintendi A, Kosakowski S, Lagisetty P, Larochelle M, Bohnert ASB, & Bazzi AR (2021). “I felt like I had a scarlet letter”: Recurring experiences of structural stigma surrounding opioid tapers among patients with chronic, non-cancer pain. Drug and Alcohol Dependence, 108664. 10.1016/j.drugalcdep.2021.108664 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Boness CL, & Sher KJ (2020). The case for cognitive interviewing in survey item validation: a useful approach for improving the measurement and assessment of substance use disorders. Journal of Studies on Alcohol and Drugs, 81(4), 401–404. 10.15288/jsad.2020.81.401 [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Botticelli MP, & Koh HK (2016). Changing the language of addiction. JAMA, 316(13), 1361–1362. 10.1001/jama.2016.11874 [DOI] [PubMed] [Google Scholar]
  8. Brooks J, McCluskey S, Turley E, & King N (2015). The utility of template analysis in qualitative psychology. Qualitative Research in Psychology, 12(2), 202–222. 10.1080/14780887.2014.955224 [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Brown RL, Leonard T, Saunders LA, & Papasouliotis O (2001). A two-item conjoint screen for alcohol and other drug problems. Journal of the American Board of Family Practice, 14(2), 95–106. [PubMed] [Google Scholar]
  10. Broyles LM, Gordon AJ, Sereika SM, Ryan CM, & Erlen JA (2011). Do words matter? Incongruent responses to inconsistently worded AUDIT-C alcohol screening instruments. Substance Abuse, 32(4), 202–209. 10.1080/08897077.2011.600673 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Clayton DK, Rogers S, & Stuifbergen A (1999). Answers to unasked questions: writing in the margins. Research in Nursing and Health, 22(6), 512–522. [DOI] [PubMed] [Google Scholar]
  12. Creswell J (2013). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. SAGE Publications. [Google Scholar]
  13. Cusack M, & Montgomery AE (2018). Barriers and facilitators to housing access and maintenance in HUD-VASH: Participant and staff perspectives. Social Work in Health Care, 57(6), 422–439. 10.1080/00981389.2018.1441213 [DOI] [PubMed] [Google Scholar]
  14. deRussy AJ, Jones AL, Austin EL, Gordon AJ, Gelberg L, Gabrielian SE, Riggs KR, Blosnich JR, Montgomery AE, Holmes SK, Varley AL, Hoge AE, & Kertesz SG (2021). Insights for conducting large-scale surveys with veterans who have experienced homelessness. Journal of Social Distress and Homelessness, 1–12. 10.1080/10530789.2021.2013013 [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Desai MM, & Rosenheck RA (2005). Unmet need for medical care among homeless adults with serious mental illness. General Hospital Psychiatry, 27(6), 418–425. [DOI] [PubMed] [Google Scholar]
  16. Erlingsson C, & Brysiewicz P (2017). A hands-on guide to doing content analysis. African Journal of Emergency Medicine, 7(3), 93–99. 10.1016/j.afjem.2017.08.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Fifolt M, Patel K, Rucks A, & Ford EW (2021). A review of unsolicited comments on the CAHPS5.0 Health Plan Survey. Journal of Patient Experience, 8, 23743735211048056. 10.1177/23743735211048056 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Gabrielian S, Chen JC, Minhaj BP, Manchanda R, Altman L, Koosis E, & Gelberg L (2017). Feasibility and acceptability of a co-located homesless-tailored primary care clinic and emergency department. Department. Journal of Primary Care and Community Health, 8(4), 338–344. 10.1177/2150131917699751 [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Gale RC, Wu J, Erhardt T, Bounthavong M, Reardon CM, Damschroder LJ, & Midboe AM (2019). Comparison of rapid vs in-depth qualitative analytic methods from a process evaluation of academic detailing in the Veterans Health Administration. Implementation Science, 14(1), 11. 10.1186/s13012-019-0853-y [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Gelberg L, Gallagher TC, Andersen RM, & Koegel P (1997). Competing priorities as a barrier to medical care among homeless adults in Los Angeles. American Journal of Public Health, 87(2), 217–220. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Gruenewald DA, Doan D, Poppe A, Jones J, & Hutt E (2018). “Meet me where I am”: removing barriers to end-of-life care for homeless Veterans and Vertans without stable housing. American Journal of Hospice and Palliative Care, 35(12), 1483–1489. 10.1177/1049909118784008 [DOI] [PubMed] [Google Scholar]
  22. Huppertz JW, & Smith R (2014). The value of patients’ handwritten comments on HCAHPS surveys. Journal of healthcare management 59(1), 31–47. http://europepmc.org/abstract/MED/24611424 [PubMed] [Google Scholar]
  23. Jego M, Abcaya J, Ștefan D-E, Calvet-Montredon C, & Gentile S (2018). Improving health care management in primary care for homeless people: A literature review. International journal of environmental research and public health, 15(2), 309. 10.3390/ijerph15020309 [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Jones AL, Hausmann LRM, Haas GL, Mor MK, Cashy JP, Schaefer JH, & Gordon AJ (2017). A national evaluation of homeless and nonhomeless veterans’ experiences with primary care. Psychological Services, 14(2), 174–183. 10.1037/ser0000116 [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Jull J, Giles A, & Graham ID (2017). Community-based participatory research and integrated knowledge translation: advancing the co-creation of knowledge. Implement Science, 12(1), 150. 10.1186/s13012-017-0696-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Kelly JF, Bergman B, Hoeppner BB, Vilsaint C, & White WL (2017). Prevalence and pathways of recovery from drug and alcohol problems in the United States population: Implications for practice, research, and policy. Drug and Alcohol Dependence, 181, 162–169. 10.1016/j.drugalcdep.2017.09.028 [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Kertesz SG, deRussy AJ, Kim Y. i., Hoge AE, Austin EL, Gordon AJ, Gelberg L, Gabrielian SE, Riggs KR, Blosnich JR, Montgomery AE, Holmes SK, Varley AL, Pollio DE, Gundlapalli AV, & Jones AL (2021). Comparison of patient experience between primary care settings tailored for homeless clientele and mainstream settings. Medical Care, Publish Ahead of Print. 10.1097/mlr.0000000000001548 [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Kertesz SG, Hwang SW, Irwin J, Ritchey FJ, & Lagory ME (2009). Rising inability to obtain neede health care among homeless persons in Birmingham, Alabama (1995–2005). Journal of General Internal Medicine, 24(7), 841–847. 10.1007/s11606-009-0990-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Kertesz SG, Pollio DE, Jones RN, Steward J, Stringfellow EJ, Gordon AJ, Johnson NK, Kim TA, Daigle SG, Austin EL, Young AS, Chrystal JG, Davis LL, Roth DL, & Holt CL (2014). Development of the Primary Care Quality-Homeless (PCQ-H) instrument: a practical survey of homeless patients’ experiences in primary care. Medical Care, 52(8), 734–742. 10.1097/MLR.0000000000000160 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Kiser T, & Hulton L (2018). Addressing health care needs in the homesless population: a new approach using participatory action research. SAGE Open, 8(3). 10.1177/2158244018789750 [DOI] [Google Scholar]
  31. Kushel MB, Vittingoff E, & Haas JS (2001). Factors associated with the health care utilization of homeless persons. JAMA, 285(2), 200–206. [DOI] [PubMed] [Google Scholar]
  32. McCallum R, Medved MI, Hiebert-Murphy D, Distasio J, Sareen J, & Chateau D (2019). Fixed nodes of transience: narratives of homelessness and emergency department use. Qualitative Health Research, doi: 10.1177/1049732319862532 [Epub ahead of print]. 10.1177/1049732319862532 [DOI] [PubMed] [Google Scholar]
  33. McLellan AT, Kushner H, Metzger D, Peters R, Smith I, Grissom G, Pettinati H, & Argeriou M (1992). The fifth edition of the Addiction Severity Index. Journal of Substance Abuse Treatment, 9(3), 199–213. 10.1016/0740-5472(92)90062-s [DOI] [PubMed] [Google Scholar]
  34. Midboe AM, Byrne T, Smelson D, Jasuja G, McInnes K, & Troszak LK (2019). The opioid epidemicin Veterans who were homeless or unstably housed. Health Affairs (Millwood), 38(8), 1289–1297. 10.1377/hlthaff.2019.00281 [DOI] [PubMed] [Google Scholar]
  35. Montgomery AE, Fargo JD, Kane V, & Culhane DP (2014). Development and validation of an instrument to assess imminent risk of homelessness among veterans. Public Health Reports, 129(5), 428–436. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Moore DT, & Rosenheck RA (2016). Factors affecting emergency department useby a chronically homeless population. Psychiatric Services, 67(12), 1340–1347. 10.1176/appi.ps.201500526 [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. O’Toole TP, Johnson EE, Aiello R, Kane V, & Pape L (2016). Tailoring care to vulnerable populations by incorporating social determinants of health: the Veteran’s Health Administration’s “Homeless Patient Aligned Care Team” program. Preventing Chronic Disease, 13, E44. 10.5888/pcd13.150567 [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Patanwala M, Tieu L, Ponath C, Guzman D, Ritchie CS, & Kushel M (2017). Physical, psychological, social, and existential symptoms in older homeless-experienced adults: observational study of he home cohort. Journal of General Internal Medicine. 10.1007/s11606-017-4229-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Penn TM, Overstreet DS, Aroke EN, Rumble DD, Sims AM, Kehrer CV, Michl AN, Hasan FN, Quinn TL, Long DL, Trost Z, Morris MC, & Goodin BR (2020). Perceived injustice helps explain the association between chronic pain stigma and movement-evoked pain in adults with nonspecific chronic low back pain. Pain Medicine, 21(11), 3161–3171. 10.1093/pm/pnaa095 [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Ramsay N, Hossain R, Moore M, Milo M, & Brown A (2019). Health care while homeless: barriers, facilitators, and the lived experiences of homeless individuals accessing health care in a Canadian regional municipality. Qualitative Health Research, 1049732319829434. 10.1177/1049732319829434 [DOI] [PubMed] [Google Scholar]
  41. Riggs KR, Hoge AE, DeRussy AJ, Montgomery AE, Holmes SK, Austin EL, Pollio DE, Kim YI, Varley AL, Gelberg L, Gabrielian SE, Blosnich JR, Merlin J, Gundlapalli AV, Jones AL, Gordon AJ, & Kertesz SG (2020). Prevalence of and risk factors associated with nonfatal overdose among Veterans who have experienced homelessness. JAMA Network Open, 3(3), e201190. 10.1001/jamanetworkopen.2020.1190 [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Roncarati JS, Baggett TP, O’Connell JJ, Hwang SW, Cook EF, Krieger N, & Sorensen G (2018). Mortality among unsheltered homeless adults in Boston, Massachusetts, 200–2009. JAMA Internal Medicine, 178(9), 1242–1248. 10.1001/jamainternmed.2018.2924 [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Sestito SF, Rodriguez KL, Hruska KL, Conley J, Mitchell MA, & Gordon AJ (2018). A photo-elicitation study of homeless and marginally housed Veterans’ experiences with patient-centered care. Patient Experience Journal, 5(3), 123–148. 10.35680/2372-0247.1279 [DOI] [Google Scholar]
  44. Talvitie AM, Ojala H, Tammela T, & Pietila I (2022). The expanded prostate cancer index composite short form (EPIC-26) for measuring health-related quality of life: content analysis of patients’ spontaneous comments written in survey margins. Qualty of Life Research, 31(3), 855–864. 10.1007/s11136-021-02940-z [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Teddlie A, & Tashakkori C (2009). Foundations of mixed-methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Sage. [Google Scholar]
  46. Varley AL, Montgomery AE, Steward J, Stringfellow E, Austin EL, Gordon AJ, Pollio D, deRussy A, Hoge A, Gelberg L, Riggs K, Kim TW, Rubens SL, & Kertesz SG (2020). Exploring quality of primary care for patients who experience homelessness and the clinicians who serve them: what are their aspirations? Qualitative Health Research, 30(6), 865–879. 10.1177/1049732319895252 [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Wen CK, Hudak PL, & Hwang SW (2007). Homeless people’s perceptions of welcomeness and unwelcomeness in healthcare encounters. Journal General Internal Medicine, 22(7), 1011–1017. 10.1007/s11606-007-0183-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Witkiewitz K, Wilson AD, Roos CR, Swan JE, Votaw VR, Stein ER, Pearson MR, Edwards KA, Tonigan JS, Hallgren KA, Montes KS, Maisto SA, & Tucker JA (2021). Can individuals with alcohol use disorder sustain non-abstinent recovery? Non-abstinent outcomes 10 years after alcohol use disorder treatment. Journal of Addiction Medicine, 15(4), 303–310. 10.1097/adm.0000000000000760 [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Zhang L, Norena M, Gadermann A, Hubley A, Russell L, Aubry T, To MJ, Farrell S, Hwang S, & Palepu A (2018). Concurrent Disorders and Health Care Utilization Among Homeless and Vulnerably Housed Persons in Canada. Journal of Dual Diagnosis, 14(1), 21–31. 10.1080/15504263.2017.1392055 [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

Research data are not shared.

RESOURCES