Non-response bias in social risk factor screening among adult emergency department patients

Abstract

Healthcare organizations increasingly use screening questionnaires to assess patients’ social factors, but non-response may contribute to selection bias. This study assessed differences between respondents and those refusing participation in a social factor screening. We used a cross-sectional approach with logistic regression models to measure the association between subject characteristics and social factor screening questionnaire participation. The study subjects were patients from a mid-western state safety-net hospital’s emergency department. Subjects’ inclusion criteria were: 1) ≥ 18 years old, 2) spoke English or Spanish, and 3) able to complete a self-administered questionnaire. We classified subjects that consented and answered the screening questionnaire in full as respondents. All others were non-respondents. Using natural language processing, we linked all subjects’ participation status to demographic characteristics, clinical data, an area-level deprivation measure, and social risk factors extracted from clinical notes. We found that nearly 6 out of every 10 subjects approached (59.9%), consented, and completed the questionnaire. Subjects with prior documentation of financial insecurity were 22% less likely to respond to the screening questionnaire (marginal effect = −22.40; 95% confidence interval (CI) = −41.16, −3.63; p = 0.019). No other factors were significantly associated with response. This study uniquely contributes to the growing social determinants of health literature by confirming that selection bias may exist within social factor screening practices and research studies.

Introduction

Social factors encompass numerous patients’ nonclinical, economic, contextual, and psychosocial characteristics^1–3. Thus, social factors are important drivers of morbidity, mortality, disparities, healthcare utilization, and costs^4,5. Subsequently, healthcare organizations are increasingly screening patients to ascertain the presence of multiple social factors, mostly with questionnaires^6,7. Available screening questionnaires include those developed by healthcare organizations⁸, collaborative organizations^9,10, payers¹¹, or electronic health record (EHR) vendors¹². Questionnaire-based social factor screening has the advantage of flexible deployment: clinical staff can ask for patient responses, patients can self-complete questionnaires on paper, or patients can complete them electronically through EHR patient portals¹³. Moreover, once collected, social factor screening results can be guide referrals to the social services¹⁴ or for population health management activities¹⁵.

While healthcare organizations, payers, and researchers may want patient-level social factor data, such screening is not risk-free¹⁶. Patients have multiple reasons for not completing social factor screening questionnaires. For example, patients may fear that disclosure of social risks can change the nature of the treatments, lead to stigmatization, or be inappropriately disclosed to other parties^17,18. Additionally, patients may not view social factors as relevant to healthcare and therefore opt not to answer screening questions¹⁹. Importantly, patient views of the appropriateness of social factor screening may vary by race and ethnicity ²⁰. Patient concerns about social risk screening may be particularly salient among underrepresented groups with a long history of mistreatment by healthcare organizations²¹. Thus, some patients may be less likely to complete screening questionnaires and become non-respondents, contributing to selection bias in the data collection process²².

The longstanding epidemiological assumption is that respondents to questionnaires are likely different than non-respondents²³. Differences due to non-response bias may affect care delivery and population health management initiatives. First, non-response may prevent social factor screening interventions from effectively identifying and referring all patients with social needs. Patients may refuse screening out of a desire not to disclose a sensitive topic ²⁴. Thus, they will not be offered services or referrals designed to address their social needs. Second, incomplete identification of social needs can undermine the population-based risk stratification modeling²⁵. If such selection biases are left unaccounted for, risk stratification approaches exacerbate or create further inequities in the healthcare delivery ²⁶. Finally, non-response can produce inaccurate estimates of the prevalence of social needs among the patients, complicating needs assessment planning for hospitals and health systems.

In light of the potential challenges posed by non-responses and the lack of quantifiable information on those refusing social factor screening, this study sought to assess the difference in patient characteristics between respondents and those refusing participation in a social factor screening study conducted in the emergency department (ED) of one safety-net hospital located in a mid-western state. Identification of screening response rates and potential differences between respondents and nonrespondents will help set organizational expectations for the effectiveness of social screening programs and identify risks of bias in using social factor screening data for analyses.

METHODS

Within a larger survey study on social risk factors, we nested a cross-sectional study to compare adult ED patients that responded or refused to complete a social factor screening questionnaire. This analysis is part of the larger project aimed at developing risk stratification models in the ED.

Setting & subjects

We recruited adults who sought care at Eskenazi Health’s ED between July and November 2021. Eskenazi Health is a 315-bed safety-net hospital in a mid-western state. Potential subjects were approached for social factor screening if they met the following criteria: 1) ≥ 18 years old, 2) spoke English or Spanish, and 3) were able to complete a self-administered social factor screening questionnaire. Patients with altered mental states, trauma, and acute behavioral health issues were excluded. We did not recruit confirmed or suspected COVID-19 patients out of safety concerns. We approached 429 eligible ED patients for participation between June and November 2021. The sample size was not pre-determined, but based on the pragmatic choice to evaluate potential response bias within the first 6 months of data collection.

Recruitment

During various shifts during days, nights, and weekends, trained data collection staff reviewed the ED’s EHR for eligible study subjects seeking care in the ED. Next, a research staff invited all eligible subjects to complete a survey about ED patients’ social risks. Research staff explained to eligible subjects the purpose of the study to better understand the occurrence of social risks to help healthcare organizations implement more effective referral services. Recruitment occurred within patient rooms or pre-discharge waiting areas. All subjects were offered a $5 gift card as an incentive for completing the survey. The social factor screening questionnaire was self-administered on a tablet ^27,28. The Indiana University IRB approved this study (2011558232).

Participation measures

Subjects that consented to study inclusion and answered the social factor screening questionnaire in full were classified as respondents. Non-respondents included subjects who declined participation and those who verbally agreed to participate in the study but subsequently declined to sign consent forms or failed to complete the questionnaire after consent.

Data & covariates

We linked participation status to demographic and clinical structured data from the safety-net hospital’s EHR for all subjects. These included subjects: age, race/ethnicity, gender, preferred language, date & time of visit, insurance status, and discharge diagnosis. We grouped primary discharge diagnoses using AHRQ’s Clinical Classification Software²⁹. The NYU-ED algorithm identified non-emergency visits³⁰.

We combined two additional data sources to assess differences in social factors between respondents and non-respondents comprehensively. First, we characterized the subjects’ zip code of residence using a modified Townsend Index³¹, an overall summary measure of socioeconomic disadvantage. We categorized this continuous measure into quartiles. Second, we identified subjects with any mention of housing instability, income insecurity, and unemployment in clinical textual documents (e.g., clinical notes and reports) in the prior 12 months using natural language processing (NLP). The development and validation of the NLP algorithm have been reported previously and included notes from ED visits, inpatient admissions, and outpatient visits³².

Analyses

We described the sample using percentages. We compared subjects’ characteristics by participation status using Chi-square tests. We measured the association between subject characteristics and social factor screening questionnaire participation using logistic regression models controlling for all covariates. To facilitate interpretation, we expressed regression coefficients as marginal effects, i.e., the changes in the predicted probability of being a respondent for each characteristic.

RESULTS

Reflective of an ED patient population at a safety-net hospital, the overall sample was diverse (Table 1). Briefly, more than half (59.2%) of approached subjects were females and were on Medicaid (56.4%). Subjects aged 45–66 (38.7%) and Black non-Hispanic (45.2%) were the largest groups compared to other age and race/ethnicity groups. Most subjects were English speaking (80.4%). The most common diagnoses were related to pain and headache. Nearly 6 out of every 10 subjects approached (59.9%) consented and completed the social factor screening questionnaire. Subjects spent, on average, eight minutes (mean = 8.1, sd=5.2) to complete the questionnaire.

Table 1.

Comparison of subjects approached for a social risk factor screening questionnaire by response status.

Demographics	Respondents (N=257; 59.9 %) %	Non-respondents (N=172; 40.1%) %	Total (N= 429) N (%)	P-value
Gender				0.867
Male	40.5	41.3	175 (40.8)
Female	59.5	58.7	254 (59.2)
Age				0.080
18–34	31.5	32.0	136 (31.7)
35–44	21.8	19.8	90 (21.0)
45–64	40.9	35.5	166 (38.7)
>65	5.8	12.8	37 (8.6)
Race and ethnicity				0.503
White non-Hispanic	28.4	26.2	118 (27.5)
Black non-Hispanic	42.4	49.4	194 (45.2)
Hispanic	26.5	21.5	105 (24.5)
Other / unknown	2.7	2.9	12 (2.8)
Preferred language
English	79.4	82.0	345 (80.4)	0.506
Other than English	20.6	18.0	84 (19.6)
Insurance status				0.320
Commercial	12.8	12.8	55 (12.8)
Medicare	15.2	29.3	75 (17.5)
Medicaid	56.4	56.4	242 (56.4)
Self-pay	14.4	8.7	52 (12.1)
Other/Unknown	1.2	1.2	5 (1.2)
Discharge diagnosis				0.707
Pain & headache1	35.4	39.0	158 (36.8)
Injury	12.1	13.4	54 (12.6)
Urinary tract infection	5.1	3.5	19 (4.4)
Nausea & vomiting	3.5	1.7	12 (2.8)
Respiratory illness & conditions	7.4	5.2	28 (6.5)
Other	36.6	37.2	158 (36.8)
Non-emergency condition	19.8	21.5	88 (20.5)	0.675
Time of visit				0.229
AM	20.6	25.6	97 (22.6)
PM	79.4	74.4	332 (77.4)
Day of visit				0.447
Weekend	16.3	19.2	75 (17.5)
Weekday	83.7	80.8	354 (82.5)
Area level deprivation
1 (least disadvantaged)	28.2	27.8	119 (28.1)	0.979
2	23.1	21.9	96 (22.6)
3	26.7	28.4	116 (27.4)
4 (most disadvantaged)	22.0	21.9	93 (21.9)
Documented in clinical notes 2
Housing instability	6.2	4.7	24 (5.6)	0.487
Financial insecurity	4.3	9.9	28 (6.5)	0.021
Unemployed	16.0	22.7	80 (18.7)	0.080

¹Includes: abdominal, musculoskeletal, spondylopathies & spondyloarthropathy, chest pain, headache

²In the past 12 months

Subject participation in the social factor screening questionnaire did not vary significantly by structured data elements in the EHR, such as age, gender, race/ethnicity, payer status, preferred language, discharge diagnosis, the timing of the visit, and whether the visit was related to the non-emergency condition (see Table 1). Likewise, the area-level social determinants of health measures did not vary significantly by participation status. However, financial insecurity documentation in clinical notes varied by participation: 9.9% of non-respondents had such documentation in the past 12 months, whereas it was present in 4.3% of those that completed the screening questionnaire.

In the fully adjusted model (Table 2), subjects with prior documentation of financial insecurity were 23% less likely to respond to the social factor screening questionnaire than those without (marginal effect = −22.90; 95% confidence interval = −41.76, −4.05; p = 0.017). No other factors were significantly associated with response.

Table 2.

Association between patient characteristics and nonresponse to social factor screening (N= 424 ED patients)¹

Demographics	Non-response2 (Marginal effect; 95% Confidence Interval)	P-value
Gender
Male	Reference
Female	3.64 (-6.15, 13.42)	0.467
Age
18–34	Reference
35–44	7.67 (−5.48, 20.82)	0.253
45–64	9.50 (−2.2, 21.2)	0.112
>65	−13.77 (−35.36, 7.83)	0.211
Race and ethnicity
White non-Hispanic	Reference
Black non-Hispanic	−3.43 (−14.89, 8.04)	0.558
Hispanic	6.51 (−14.6, 27.62)	0.546
Other / unknown	1.99 (−26.91, 30.9)	0.893
Preferred language
English	Reference
Other than English	−5.83 (−27.73, 16.06)	0.602
Insurance status
Commercial	Reference
Medicare	3.68 (−15.88, 23.24)	0.712
Medicaid	3.26 (−11.78, 18.29)	0.671
Self-pay	14.89 (−4.12, 33.91)	0.125
Other/Unknown	7.48 (−35.25, 50.21)	0.732
Primary diagnosis
Pain & headache3	−2.38 (−13.12, 8.36)	0.664
Injury	−2.52 (−17.94, 12.89)	0.748
Urinary tract infection	7.54 (−14.79, 29.88)	0.508
Nausea & vomiting	16.74 (−8.44, 41.92)	0.193
Respiratory illness & conditions	5.20 (−14.41, 24.81)	0.603
All others	Reference
Non-emergency condition	−5.06 (−17.66, 7.53)	0.431
Time of visit
AM	Reference
PM	9.58 (−1.91, 21.07)	0.102
Day of visit
Weekday	Reference
Weekend	−6.37 (−18.87, 6.13)	0.318
Area level deprivation 4
1 (least disadvantaged)	Reference
2	3.82 (−9.26, 16.9)	0.567
3	−2.70 (−15.66, 10.26)	0.683
4 (most disadvantaged)	1.88 (−11.38, 15.14)	0.781
Documented in clinical notes 5
Housing instability	14.47 (−3.72, 32.66)	0.119
Financial insecurity	−22.90 (−41.76, −4.05)	0.017
Unemployed	−8.90 (−21.77, 3.97)	0.175

¹Reference category is responded to survey;

²Adjusted for gender, age, race & ethnicity, preferred language, and insurance type

³Includes: abdominal, musculoskeletal, spondylopathies & spondyloarthropathy, chest pain, headache;

⁴Subject zip code

⁵In the past 12 months

DISCUSSION

In an adult ED patient population, nearly 6 out of 10 approached subjects responded to a social factor screening questionnaire. Our study had a typical response rate for ED social risk screening efforts^17,21. While respondents and non-respondents were similar in demographic characteristics, they did differ in the history of documented financial insecurity. Our finding that respondents and non-respondents differ by risk factor is expected, as it is well-established in the literature that reporting social needs is not risk-free for patients¹⁶. This study uniquely contributes to the growing social determinants of health literature by confirming that selection bias may exist within social factor screening practices and research studies.

A critical concern from both the practice and research perspectives is that patients who refuse screening participation are at higher risk. We observed that those with prior documented financial insecurity were less likely to respond to the social factor screening questionnaire. As financial insecurity is a critical determinant of health and an underlying driver of other social risks^33–35, this finding indicates a potentially significant difference between respondents and nonrespondents. The history of documented financial insecurity may reflect more acute social needs because a provider thought it warranted documentation. Consistent with our findings, research among pediatric caregivers reported that those who declined social screening were potentially at higher social risk³⁶.

We can speculate about potential reasons for the observed differences in documented financial insecurity. Screening questionnaires and clinical documentation represent two very different methods of data collection and contexts, which may drive the observed differences. For example, sharing personal social information may be easier with a healthcare provider with an assumed or historical level of trust^19,37. A such trust might explain why a patient that divulged financial difficulties in the past would not respond to a research survey. Additionally, our questionnaire administration was not part of the clinical care process. Patients may have refused to fill out our survey because they needed to see how social information disclosure was relevant or potentially beneficial. Organizations may face similar nonresponse challenges if employing modalities that are outside the patient encounter, such as sending social factors screening questionnaires through patient portals or personal health records^8,38.

Critically, the differences in prior financial insecurity documentation stand in stark contrast to the absence of any significant difference in measured and observable characteristics, including area-level social determinants of the health measure. Checking for differences in response rates in patient demographics or area-level social determinants of health measures may not be sufficient to assess the presence of bias for any social factor screening. Thus, referral programs to social service interventions and population health analytics efforts should consider alternative approaches for dealing with unscreened patients. Additionally, the absence of differences in common structured data elements may hinder analytic approaches to address missing information. For example, a recent review on EHR-based social factor data quality recommended leveraging imputation methods to deal with missingness³⁹. While imputation can be achieved using numerous techniques, these findings indicate that such efforts may require additional data sources or advanced information extraction methods like NLP.

Social factor screening is most effective when consistently applied to the entire target population⁴⁰. Whether that target population should be all patients or the subset of patients with known risk factors is an open question^41,42. Still, multiple healthcare organizations have attempted to screen all their patients^14,43, CMS’ Accountable Health Communities model requires universal screening⁴⁴, and screening all patients avoids perpetuating biases and stigmatization¹⁶. Currently, screening is often occurring selectively and inconsistently. The screening is often based on the patient’s appearance^18,45 or demographic characteristics ⁴⁶; different screening questionnaires are used in the organization concurrently⁶; or patients may be excluded because data are collected in a single language⁴⁶. Closing the gap between current response rates and nearer-universal screening will require integration within EHR portals for out-of-office data collection, mining clinical notes using NLP⁴⁷, or risk stratification modeling to identify those most likely in need of additional screening⁴⁸.

Limitations

The generalizability of our findings to clinical practice warrants further exploration. The social factor screening questionnaire was administered as part of a research study, which included interaction with nonclinical staff, written informed consent, and incentives. Interest in responding to social factor screening may be different if administered by the patient’s healthcare provider in the context of care. These findings may not generalize to other care settings, non-adult patients, or provider types. The current study focused on the response to a social factor screening questionnaire. Still, prior work on social screening among pediatric caregivers has also identified substantial differences among non-response to individual questionnaire items as well³⁶. Finally, these data were collected during the COVID-19 pandemic (after vaccinations became widely available). Overall, the stresses felt by patients and the healthcare environment may have influenced recruitment.

CONCLUSIONS

Respondents and non-respondents to a social factor screening questionnaire differed in a history of prior financial insecurity. Selection bias may exist within social factor screening.

AVAILABLITY OF DATA AND MATERIALS

The data for this study are not available.