Psychometric Evaluation of the Screener for Intensifying Community Referrals for Health

Abstract

This study examined the psychometric properties of the Screener for Intensifying Community Referrals for Health (SINCERE), a 10-item, low-literacy screening tool developed to elicit social needs (e.g. transportation, housing) impacting patients’ ability to engage in health-related activities. Patients seeking care in a tertiary care emergency department (ED) were invited to complete the SINCERE as part of registration processes, and were asked about their desire for follow-up by a partnering service provider offering low- and no-cost community resource referrals. A total of 5081 patients completed screenings were included in this sample. Exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) identified and verified one-factor structure, suggesting that the SINCERE’s 10-items are homogenous and measure one construct. The reliability of Cronbach’s alpha and McDonald’s Omega were 0.89. Item Response Theory (IRT) suggested the SINCERE can effectively identify patients wishing referrals, or who have social needs. Moreover, patients who had two or more social needs were those willing to receive referrals after discharge. The SINCERE is a valid and reliable tool for measuring social needs for health, and should be considered as a screening option for practice interventions seeking to address social needs.

INTRODUCTION

It is widely acknowledged that the environments in which people live affect health outcomes, contributing to health disparities (United States Department of Health and Human Services, 2020). Social determinants of health (SDOH) include many variables representing environmental conditions, material attributes, and patterns of social engagement (Centers for Disease Control, 2009, 2020). While there is a long history of health systems accounting for SDOH in payment structures (e.g., risk adjustment) (Maroko et al., 2016; Phillips et al., 2016), a new recognition of how SDOH place individual patients at risk for poor health outcomes (Joynt Maddox et al., 2019) have led policy and payer groups to focus on identifying the stated needs of patients subject to clinical intervention, and to recommend that clinical systems screen for “social needs” such as food and housing insecurity; financial strain; transportation, childcare, education, employment, and mental health needs; exposure to violence; and social isolation (Artiga & Hinton, 2018; Gottlieb et al., 2013; Institute of Medicine, 2016; Robert Wood Johnson Foundation, 2016; Woolf & Braveman, 2011). However, a recent review (Henrikson et al., 2019) revealed that, while there is enthusiasm for implementing social risk screening in clinical settings, there are scant data regarding both the pragmatic (e.g., ease and costs of administration) (Glasgow, 2013; Lewis et al., 2018) and psychometric properties of such assessments (Andermann, 2018; Pai et al., 2016; Stanick et al., 2018). As clinical interventions addressing social risk are rapidly adopted, research regarding the measurement properties of social needs screeners is needed (Henrikson et al., 2019).

One reason for the lack of rigorous psychometric testing of social risk assessments may rest with the fact that the majority of limited “social needs” assessments adopted into clinical settings have been developed with the sole purpose of eliciting patient-stated needs for clinical intervention. While latent constructs such as depression, anxiety, or quality of life may follow from patients’ stated social needs, patient stated needs are largely made explicit through screening, and it is difficult to validate stated needs with objective measures (Henrikson et al., 2019). Consequently, it is difficult to establish the validity of social needs screeners by applying standard concepts such as construct, content, and criterion validity. Further, because addressing social needs likely involves multi-level intervention, behavior change, and resources beyond the clinical environment over time, typical analyses seeking to associate clinical interventions and screening results with changes in distal health outcomes such as quality of life are, at best, tenuous. In fact, interventions that refer patients to community resources have documented less than 20% of patients with needs ultimately receive services (Lindau et al., 2019), and the first barrier limiting intervention reach is overcoming patients’ desire for assistance (Fiori et al., 2020; Schickedanz et al., 2019).

In this paper, we propose that exploring the psychometric value of social needs screeners requires recognizing the place of screening in the landscape of interventions that, subsequently, require engagement and action on the part of patients. As such, we have identified that receptivity to, and engagement with, an intervention for addressing social needs is a critical proximal outcome from which all other benefits must stem. We focus on a new 10-item screening tool – the Screener for Intensifying Community Referrals for Health (SINCERE) – inquiring about needs related to mortgage or rent, utilities, food, household items, unstable housing, medical expenses, transportation, employment, medication expenses, or childcare or eldercare (Table 1). A particular focus during the development of the SINCERE was to identify needs subject to follow-up and service referrals after emergency department (ED) discharge, and was born of methods that engaged community and clinical stakeholders to develop, adapt, refine, and iteratively test a community service referral process using existing resources in the ED setting (Wallace et al., 2020). The process used to develop and test the early iterations of the SINCERE are described elsewhere and addressed key pragmatic measures for implementation research (Glasgow, 2013; Stanick et al., 2018). We surmise that patients with motivation for outreach and referrals represent those who are, indeed, most impacted by unmet needs. Further, we believe our patient-centered approach of focusing on patient receptivity to outreach is responsive to concerns regarding the ethics of universal screening for social determinants that may include marginalizing vulnerable populations. In our previous work (Wallace et al., 2020), we addressed key aspects of the Psychometric and Pragmatic Evidence Rating Scale (PAPERS), including acceptability, relative advantage over existing methods, ease of completion, compatibility, organizational activities, informing clinical or organizational decision-making, cost, language accessibility, and assessor burden.

Table 1.

Items in the SINCERE

It takes a lot to be healthy. I’m going to ask you questions about things everyone needs to care for themselves at home, like food, housing, or transportation. We have a partnership with United Way 2-1-1, a free service, to give you information related to your needs. There is no cost for this. These questions are not a part of your medical records. Here are the questions:		Response
In the last month…
Was there a time when you were not able to pay your mortgage or rent?		Yes No
Was there a time when you were not able to pay your utility bills?		Yes No
Did you feel there was not enough money for food?		Yes No
Did you feel there was not enough money for items like clothing or furniture?		Yes No
Have you slept outside, in a shelter, in a car, or any place not meant for sleeping?		Yes No
Have you needed to see a doctor but could not because it costs too much?		Yes No
Have you not seen a doctor because you didn’t have a way to get to the clinic or hospital?		Yes No
Have you been unemployed and looking for work?		Yes No
Did you not take medications to save money?		Yes No
Have problems getting child care or elder care made it difficult for you to work or get to appointments?		Yes No

The purpose of the SINCERE was to efficiently and effectively elicit patient-reported social needs perceived as impacting their ability to engage in health-related activities. We aimed to develop a reliable and valid tool for assessing individuals’ social needs for health. This paper applies Classical Test Theory (i.e., Exploratory Factor Analysis, EFA, and Confirmatory Factor Analysis, CFA) and Item Response Theory (IRT) to examine the SINCERE item and psychometric properties in a larger population sample of ED patients, using receptivity to service referrals as the primary outcome of interest. These analyses are part of a larger parent study evaluating the technical and operational feasibility and acceptability of implementing a health information technology (HIT)-delivered social needs assessment and community service referral process referral system during routine emergency department service delivery.

METHODS

Study design

A cross-sectional survey design was used to evaluate the psychometric properties of the SINCERE with a sample of patients seeking care in a tertiary care emergency department. The data were collected between January 2019 and September 2020.

Survey instrument

Our previous work established that, on average, patients are able to complete the 10 dichotomous yes-no questions in less than 80 seconds (Wallace et al., 2020). Immediately following the screening questions, patients were asked to share their preferred contact information if they wished to be contacted by a United Way of Utah 211 (UW-211) information specialist within 48 hours of ED discharge. As part of our referral intervention, patients were given a written (via the touchscreen screener) and verbal (via registration staff) introduction to UW-211, a free service providing referrals to low- and no-cost community resources for needs such as transportation, food, housing, and medications.

Data collection and sample

This study was conducted with adult patients seen in the University of Utah (UHealth) University Hospital ED, an academic health sciences Level I Trauma Center servicing 50,000 patients annually. ED registration staff administered the SINCERE using touchpads and the Research Electronic Data Capture (REDCap) online system (Harris et al., 2019; Harris et al., 2009) as part of their standard workflow. The ED registration staff were trained in administering the SINCERE before the data collection started; the quality of data collection process was assured by giving a clear instruction with scripts to the registration staff. Patients were allowed to either self- or verbally-respond to the questions. While registration staff were asked to complete the screening with all patients, patients who were unable to complete the screening due to cognitive impairment, trauma, language other than English, or those who were residents of skilled nursing facilities were omitted. This study was reviewed and approved by the University of Utah’s Institutional Review Board.

Statistical analysis

Using the Statistical Package for Social Sciences (SPSS version 24) random number generator, the total sample was randomly divided into three groups for EFA, CFA, and IRT to explore the underlying latent structure and item characteristics of SINCERE. Due to the dichotomous nature of the response format of the SINCERE with “yes” coded as 1 and “no” coded as 0, we chose the structure based on the tetrachoric correlation matrix for both EFA and CFA. The number of dimensions, factor loadings, and communality (h²) were evaluated in EFA; factor loadings greater than 0.33 with significant at p <.05, and communality greater than 0.2 indicated a good item fit in a specific factor, or dimension (Tabachnick & Fidell, 2013; Watkins, 2018). Factor analysis was used to assess the reproducibility of the factor structure derived from the EFA and to check whether the factor structure was a good fit to the data. Acceptable model fit was evaluated based on comparative fit index (CFI; >0.95), Tucker-Lewis index (TLI; >0.95), and the root mean square error of approximation (RMSEA; <0.06) (Bentler, 1990; Hu & Bentler, 1998). Modification indices were examined to determine whether additional paths to the item error terms should be added to free parameters to improve the model fit.

IRT was used for modeling the relationship between an individual’s underlying latent trait level (individuals’ social needs for health) and item responses (“yes” or “no” for each item). To choose the optimal model to estimate item parameters, we examined both one-parameter logistic (1-PL; estimating one item parameter with item difficulty, or threshold) and two-parameter logistic (2-PL; estimating two-item parameters including item threshold and item discrimination) models by using model fit statistics. The lower values of the Akaike information criterion (AIC) and the Bayesian information criterion (BIC) indicate a better model. Moreover, the likelihood ratio test (LRT) was used to determine which model statistically fit the data. The performance of the SINCERE and individual items were further examined by item parameter estimates and test information function (TIF). A TIF estimates the level of latent trait at which the SINCERE is most effective.

A cutpoint analysis was conducted to determine the number of social needs from the SINCERE associated with individuals’ desire for assistance for social needs. In this cutpoint analysis, we used the SINCERE and the variable of willingness to be referred to UW-211 (“Yes” coded as 1, “No” coded as 0). The optimal cutpoint was estimated with the corresponding measures of classification accuracy; the corresponding area under the curve (AUC), sensitivity, and specificity were calculated. Descriptive statistics analysis was performed with SPSS version 24.0 (IBM, Armonk, NY, USA). EFA, CFA, IRT, and cutpoint analysis were conducted using R version 3.6.3 with the ltm, and lavaan, and cutpointr packages.

RESULTS

Data were collected between January 14, 2019 and September 23, 2020. A total of 8313 individual patients were approached; of those, 5490 agreed to respond to the SINCERE. Of those who agreed to respond, 5081 (92.6%) completed all 10 items were eligible for data analysis and were randomly assigned into three groups for EFA (n = 1693), CFA (n = 1693), and IRT (n = 1695) analyses, respectively. Demographic data were not statistically significantly different among these three groups (Table 2). Positive answers for each SINCERE item ranged from a high of 22% for mortgage/rent and utilities to 7% for childcare or eldercare (Table 3).

Table 2.

Demographic Profile of Sample

	Total N = 5081	EFA n =1693	CFA n =1693	IRT n =1695	χ2 (df)	p
	n (%)	n (%)	n (%)	n (%)
Sex					3.4 (2)	.18
Female	1333 (54.5)	434 (52.2)	459 (56.7)	440 (54.9)
Male	1111 (45.5)	398 (47.8)	351 (43.3)	362 (45.1)
Ethnicity					1.7 (2)	.42
Non-Hispanic	2112 (87.5)	705 (86.3)	709 (88.1)	698 (88.2)
Hispanic/Latino	301 (12.5)	112 (13.7)	96 (11.9)	93 (11.8)
Race					7.9 (10)	.64
White	1952 (80.8)	673 (82)	656 (81.5)	623 (79)
Black	102 (4.2)	29 (3.5)	36 (4.5)	37 (4.7)
Asian	48 (2.0)	15 (1.8)	16 (2.0)	17 (2.2)
Native Hawaiian and other Pacific Islander	34 (1.4)	8 (1.0)	10 (1.2)	16 (2.0)
American Indian and Alaska Native	25 (1.0)	11 (1.3)	8 (1.0)	6 (0.8)
Other or more than one race	254 (10.5)	85 (10.4)	79 (9.8)	90 (11.4)
SINCERE score					15.8 (20)	.79
0	2940 (57.9)	963 (56.9)	975 (57.6)	1002 (59.1)
1	491 (9.7)	171 (10.1)	154 (9.1)	166 (9.8)
2	359 (7.1)	118 (7.0)	115 (6.8)	126 (7.4)
3	260 (5.1)	90 (5.3)	93 (5.5)	77 (4.5)
4	222 (4.4)	70 (4.1)	70 (4.1)	82 (4.8)
5	186 (3.7)	70 (4.1)	65 (3.8)	51 (3.0)
6	176 (3.5)	68 (4.0)	56 (3.3)	52 (3.1)
7	173 (3.4)	51 (3.0)	64 (3.8)	58 (3.4)
8	118 (2.3)	40 (2.4)	44 (2.6)	34 (2.0)
9	77 (1.5)	26 (1.5)	25 (1.5)	26 (1.5)
10	79 (1.6)	26 (1.5)	32 (1.9)	21 (1.2)
	M (SD, Range)	M (SD, Range)	M (SD, Range)	M (SD, Range)	F (df)	p
Age	44.6 (17.6, 18.1–103.0)	43.7 (17.3, 18.3–97.4)	44.3 (17.5, 18.1–92.4)	45.7 (18.0, 18.1–103.0)	2.7 (2, 2441)	.07

CFA, confirmatory factor analysis; EFA, exploratory factor analysis; IRT, item response theory.

Table 3.

Results of EFA, CFA, and IRT for the SINCERE

	Total N = 5081		EFA n = 1693				CFA n = 1693			IRT (2-PL) n = 1695
Item	M (SD)		M (SD)	Factor loading	h 2		M (SD)	Factor loading		M (SD)	b (SE)	a (SE)
Mortgage or rent	0.22 (0.41)		0.22 (0.42)	0.76	0.58		0.22 (0.41)	0.79		0.21 (0.41)	0.83 (0.05)	4.93 (0.54)
Utilities	0.22 (0.41)		0.23 (0.42)	0.75	0.56		0.22 (0.41)	0.78		0.21 (0.41)	0.83 (0.05)	4.27 (0.36)
Food	0.19 (0.39)		0.19 (0.39)	0.77	0.59		0.20 (0.40)	0.77		0.18 (0.39)	0.94 (0.05)	3.92 (0.33)
Household items	0.20 (0.40)		0.21 (0.41)	0.76	0.58		0.21 (0.41)	0.77		0.19 (0.39)	0.92 (0.05)	3.83 (0.34)
Unstable housing	0.15 (0.36)		0.15 (0.35)	0.68	0.46		0.16 (0.36)	0.66		0.14 (0.35)	1.21 (0.06)	2.97 (0.22)
Medical expenses	0.18 (0.39)		0.19 (0.39)	0.64	0.41		0.19 (0.40)	0.66		0.17 (0.38)	1.13 (0.06)	2.46 (0.18)
Transpor-tation for medical care	0.15 (0.36)		0.15 (0.36)	0.58	0.34		0.16 (0.37)	0.63		0.14 (0.35)	1.35 (0.07)	2.14 (0.16)
Lack of job	0.19 (0.39)		0.20 (0.40)	0.62	0.38		0.19 (0.39)	0.59		0.18 (0.38)	1.14 (0.06)	2.23 (0.16)
Medication expenses	0.10 (0.30)		0.10 (0.30)	0.59	0.35		0.11 (0.32)	0.59		0.10 (0.30)	1.68 (0.09)	2.00 (0.17)
Childcare or eldercare	0.07 (0.25)		0.07 (0.25)	0.45	0.20		0.07 (0.26)	0.47		0.06 (0.24)	2.10 (0.13)	1.86 (0.18)

2-PL, two-parameter logistic; a, discrimination parameter; b, threshold parameter; CFA, confirmatory factor analysis; EFA, exploratory factor analysis; h², communality; IRT, item response theory; SE, standard error.

EFA and CFA of the SINCERE

The EFA, via a principal component factoring solution with varimax rotation, resulted in unidimensional model accounting for 44.3% of the total variance based on the parallel analysis and the scree plot that presented only one component with an eigenvalue greater than one. The value of Kaiser-Meyer-Olkin was 0.89 and the Bartlett’s Test of Sphericity was significant (p <.001), indicating adequacy of data for EFA. No multicollinearity was found as no correlation coefficients were greater than 0.80 between items. The range of the communalities were between 0.20 and 0.59, which were above the minimum suggested value of 0.20 (Table 3). All 10 items had significant factor loadings between 0.45 and 0.77, which were above the minimum suggested value of 0.33. Overall, the EFA results indicated there was a suitable amount of variance in each item, and all 10 items were essential for the SINCERE (Table 3). The reliability values of both Cronbach’s alpha and McDonald’s Omega were 0.89, showing a high internal consistency and satisfactory reliability in the SINCERE.

To verify the one-factor, or unidimensional model suggested by the EFA findings, CFA was conducted using the weighted least squares means and variance adjusted (WLSMV) estimation method as is appropriate for dichotomous data. The model fit indices of CFA indicated structural validity (CFI = 1.00, TLI = 0.99, RMSEA=0.06). The modification indices suggested adding three additional paths to the error terms of the items: mortgage or rent/utilities; food/household items; medical expenses/medication expenses. Adding these correlation terms resulted in an even better fit (GFI =1.00, TLI = 1.00, RMSEA = 0.01). These pairs, while closely related to the primary domain concept, are viewed as providing meaningful, specific information to guide unique community service referrals by our community partners and, thus, we believed there was justification for including these paths.

IRT and Cutpoint for the SINCERE

Prior to examining the IRT parameters, we first compared 1-PL and 2-PL IRT models. In the model comparison, a 2-PL IRT model with a satisfactory fit to the sample (CFI = 0.97, TLI = 0.97, RMSEA = 0.078) was chosen based on the lower AIC and BIC values comparing with 1-PL model. Moreover, the LRT also suggested that the 2-PL model was significantly better than the 1-PL model (p <.001).

The results of the IRT analysis showed that threshold parameters for all 10 items were above average level (the value of 0), ranged from 0.83 to 2.10. All 10 items showed substantial discrimination parameters with values greater than 1 (Table 3). Items of “mortgage”, “utilities,” “food”, and “household items” had lower values of threshold parameters and higher values of discrimination parameters compared to the other items. This indicated that these four items likely present more fundamental social needs and perform well at distinguishing people who have fundamental needs and specific needs. The “childcare or eldercare” item had the highest value of the threshold parameter indicating it is a more specific social needs for health. According to the TIF, a graphic representation of the total quantity of information yielded by a set of items at the latent trait level (i.e., social needs for health), the maximum value is evident around 1 with a range between 0 and 2, indicating the SINCERE is most precise at estimating the latent trait (i.e., social needs for health) at the slightly-higher-than average value of 0. The curve also shows that there is less information on lower-end value (< 0) which means that the SINCERE items lack precision in measuring social needs for health beyond the lower-end value.

Cutpoint for the SINCERE

Using the sample of IRT analysis (n = 1695), we identified the optimal cutpoint for those who wish to be referred. The results showed that individuals answering “yes” for two or more SINCERE items were more likely willing to be referred for community resources for their social needs for health. This suggested cutpoint had a good performance and clinical applicability with an accuracy of 80.1%, a sensitivity of 81.6%, a specificity of 80.7%, and an AUC of 0.862.

DISCUSSION

This paper presents psychometric analyses of the English version of SINCERE’s 10-items in a population of patients seeking care in a tertiary academic health sciences center ED. These data add important psychometric information to the pragmatic factors addressed in our previously published pilot work (Wallace et al., 2020).

A novel aspect of this study is that we examined the psychometric properties of the screener through the lens of clinical intervention delivery, recognizing that the most proximal outcome of screening is receptivity to social needs intervention, or intervention engagement. While there are multiple tools to assess social needs with pragmatic properties (Social Interventions Research and Evaluation Network, 2019), to our knowledge, this is the first known evaluation to be approached using intervention receptivity as the outcome of interest. We believe this is responsive to research in the field suggesting that a key factor in interventions aiming to address social needs is to, first, overcome patient barriers to receiving and acting upon service referrals (Fiori et al., 2020; Schickedanz et al., 2019). As interventions aiming to address social needs are rapidly adopted, adapted, and refined, understanding and addressing patient motivation for social needs outreach will likely emerge as a particularly important component in interventions aiming to implement and scale universal social needs screening.

While establishing reliability and validity of SINCERE helps ensure that the screening results reflect constructs of interest, in this case, social needs, it is done so with the understanding that it is difficult to establish construct validity for topics such as housing insecurity. Some have suggested that construct validity for some social needs, such as income or housing, may be possible (Henrikson et al., 2019). However, we challenge this notion: subjective experiences of need are unlikely to align with objective information and honoring a patient-centered approach to potentially stigmatizing and sensitive questions requires that needs reflect patient experiences vs. outside information. While social needs may not represent many properties of latent psychological constructs, it is still a subjective experience with many contributing factors. We place the SINCERE measure in the landscape of clinical tools derived from pragmatic considerations and do not suggest abandoning efforts to more comprehensively understand the experience of social needs.

This work successfully addressed two aspects related to validity critical to effective social needs screening – known-groups (the ability of a measure to identify groups with differing characteristics) and predictive (the degree to which a measure can predict or correlate with an outcome of interest (Henrikson et al., 2019; Lin & Yao, 2014; Strauss & Smith, 2009). We were unable to address measure responsiveness or clinically important changes over time. Further, our analysis suggests three correlated error terms rooted in financial concerns: “mortgage or rent” and “utilities”; “food” and “household items”; “medical expenses” and “medication expenses”. However, these additional paths in the model could be optional in the model due to a good model fit in CFA. Moreover, all of these items are used individually by our partnering service providers (i.e., UW-211) to provide distinct resources. For example, there are unique assistance programs or community resources for rent vs. for utilities.

Screening data were collected from a sample that was relatively homogenous with respect to race/ethnicity, and patients who are not native English speakers may need help to comprehend this English version of SINCERE. Therefore, efforts are currently being undertaken to more rigorously validate the SINCERE into other languages such as Spanish. While we believe that our sample size helps protect from selection bias, these analyses only superficially explored reasons for why patients were not administered the screening tool; it is possible that patients who refused outreach may not have done such in-person referrals vs. telephone contact after discharge.

The 10-item SINCERE measure is a pragmatic and psychometrically sound tool to be adopted in clinical settings seeking to implement social needs screening and referral interventions. However, caution is needed to generalize the findings to other clinical systems because of the convenient sample from the ED of a single medical center in this study and future studies are needed to evaluate the use of SINCERE in other clinical settings. We believe that by focusing on SINCERE factors contributing to willingness to have needs addressed may expand the reach of clinical interventions in clinical settings and, ultimately, offer means of comparing the effectiveness of population health interventions aiming to address social determinants of health during routine clinical service delivery.