Clinical and Sociobehavioral Prediction Model of 30-Day Hospital Readmissions Among People with HIV and Substance Use Disorder: Beyond Electronic Health Record Data

Abstract

Background:

Under the Affordable Care Act, hospitals receive reduced reimbursements for excessive 30-day readmissions. However, the Centers for Medicare and Medicaid Services does not consider social and behavioral variables in expected readmission rate calculations, which may unfairly penalize systems caring for socially disadvantaged patients, including patients with HIV.

Setting:

Randomized controlled trial of patient navigation with or without financial incentives in HIV-positive substance users recruited from the inpatient setting at 11 US hospitals.

Methods:

External validation of an existing 30-day readmission prediction model, using variables available in the electronic health record (EHR-only model), in a new multicenter cohort of HIV-positive substance users was assessed by C-statistic and Hosmer-Lemeshow testing. A second model evaluated sociobehavioral factors in improving the prediction model (EHR-plus model) using multivariable regression and C-statistic with cross-validation.

Results:

The mean age of the cohort was 44.1, and participants were predominantly male (67.4%), non-white (88.0%), and poor (62.8%, <$20,000/year). Overall, 17.5% individuals had a hospital readmission within 30 days of initial hospital discharge. The EHR-only model resulted in a C-statistic of 0.65 (95% CI: 0.60 – 0.70). Inclusion of additional sociobehavioral variables, food insecurity and readiness for substance use treatment, in the EHR-plus model resulted in a C-statistic of 0.74 (0.71 after cross-validation, 95% CI 0.64–0.77).

Conclusion:

Incorporation of detailed social and behavioral variables substantially improved the performance of a 30-day readmission prediction model for hospitalized HIV-positive substance users. Our findings highlight the importance of social determinants in readmission risk and the need to ask about, adjust for, and address them.

Introduction

Thirty-day hospital readmissions are a key quality metric under the Centers for Medicare and Medicaid Services, which limits reimbursements for hospitals with excess readmissions. Under this metric, expected readmission rates are solely adjusted for age, sex, discharge diagnosis and recent medical diagnoses.¹ Concerns have been raised that this limited adjustment unfairly penalizes hospitals serving a disproportionate number of socially disadvantaged patients.

We previously published a readmission prediction model for patients with HIV which uses electronic health record (EHR) variables that are available in real-time at the time of hospital admission. We found that a combination of variables that indicate medical illness (e.g. CD4 count, abnormal creatinine) and socio-economic status (e.g. Medicaid insurance, homelessness) provided the best performing prediction model.² Similarly, various studies have found that social and behavioral factors such as number of address changes in past year, residing in socio-economically deprived areas, cocaine use, smoking and being unmarried are associated with readmissions among different medical conditions, including congestive heart failure, pneumonia, cirrhosis and traumatic injury.^3–7 In response to these concerns, the National Academy of Medicine was commissioned to identify social risk factors that impact clinical outcomes and identified the following priority domains: socioeconomic position, race/ethnicity, gender, social relationships and residential and community context.^8,9 However, many of these factors are not readily available from the EHR, and further efforts are needed to determine how to effectively incorporate these factors and assess the impact of these adjustments on expected readmission rates.

In the context of a multicenter prospective study of inpatients with HIV, in which detailed clinical and social data were collected, we have a unique opportunity to evaluate the contribution of these factors to 30-day readmissions. Patients with HIV who were hospitalized and had a history of substance and/or heavy alcohol use were recruited from the inpatient setting and randomized to one of three groups: patient navigation, patient navigation with financial incentives or treatment as usual. In addition to demographic and clinical variables, extensive data were collected on social and behavioral factors, including those at an individual level (self-efficacy, perceived health status, psychological distress), interpersonal level (social support, intimate partner violence, patient provider relationship) and household/community-level (housing stability, food insecurity), which may be associated with readmissions.

People living with HIV (PLWH) are at high risk for readmission within 30 days of hospital discharge (rates of 19–25%),^2,10,11 which may be related to a high level of social need, medical complexity, or both. HIV disproportionately affects racial/ethnic minorities and socially disadvantaged patients, and less than one-half of PLWH have an undetectable HIV viral load (VL).¹² PLWH are frequently admitted to safety net hospitals in large urban centers, which are 30% more likely to have readmissions above the national average.¹³ By identifying key clinical and social risk factors involved in hospital readmissions among PLWH, we will not only improve our ability to predict 30-day readmissions, refining a clinical prediction tool, but also provide empirical evidence for more equitable adjustment of this national quality metric.¹⁴

In this study, we seek to: (a) externally validate a published, EMR-enabled readmission risk prediction model for PLWH² (EHR-only model) in a new and multicenter sample; (b) assess the contributions of social and behavioral risk factors to 30-day readmissions in this cohort of hospitalized patients with HIV and substance use; and (c) refine and validate a best-performing and efficient prediction model (EHR-plus model) for 30-day readmissions among PLWH.

Methods

This study analyzed data from a 3-arm randomized clinical trial in hospitalized patients with HIV and substance use. Participants were recruited from 11 hospitals across the US (Atlanta, Georgia; Baltimore, Maryland; Boston, Massachusetts; Birmingham, Alabama; Chicago, Illinois; Dallas, Texas; Los Angeles, California; Miami, Florida; New York, New York; and Philadelphia and Pittsburgh, Pennsylvania) from July 2012 to January 2014. This study was approved by the institutional review boards at all participating sites. The checklist according to the TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis method) statement¹⁵ was followed for this study.

Study description

Participants were eligible if they were HIV-infected, hospitalized, ≥18 years old, able to communicate in English, lived in the vicinity, provided detailed locator information, completed the baseline assessment, signed a medical record release, had a Karnofsky performance score of ≥60, reported or had medical records documenting opioid, stimulant (cocaine, ecstasy, or amphetamines), or heavy alcohol use as determined by the Alcohol Use Disorders Identification Test (AUDIT-C) within the past 12 months and met ≥ 1 clinical criteria indicating uncontrolled HIV as described elsewhere.¹⁶ After enrollment, social/behavioral assessments and a blood draw were completed. Participants were then randomly assigned in 1:1:1 ratio to receive either (1) patient navigation, (2) patient navigation plus financial incentives, or 3) treatment as usual. Details about the interventions have been described previously.¹⁶ All participants enrolled in the main study were included in this ancillary study.

Measures

Variables collected at baseline include demographics (age, race, ethnicity, gender, marital status) and socio-economic variables (income, housing, employment, education, prior incarceration). In addition, alcohol/substance use (Addiction Severity Index, Drug Abuse Screening Test (DAST-10) and AUDIT) were collected. Severe substance use was defined as a DAST-10 score ≥6, and/or AUDIT ≥6 (for women) or AUDIT ≥7 (for men). Clinical variables (CD4 count, HIV VL, hepatitis C antibody), HIV medication adherence (AIDS Clinical Trials Group questionnaire),¹⁷ primary diagnosis from index admission (using CCS classification system)¹⁸ and healthcare utilization (clinic visits, emergency department (ED) use, hospitalizations) were obtained. All remaining variables from the original EHR prediction model (e.g. creatinine, anion gap, history of AIDS-defining illness)¹⁹ were collected from the medical record for all participants. Additional sociobehavioral factors measured at baseline include: adherence self-efficacy,²⁰ medical mistrust,²¹ patient provider relationship,²² access to care,²³ smoking,²⁴ health literacy (3 item scale),²⁵ HIV-related cognitive problems (HIV dementia scale),²⁶ perceived health status (SF-12),²⁷ food insecurity (household food insecurity access scale),²⁸ readiness for substance use treatment (readiness and negative attitudes scales),²⁹ interpersonal violence,³⁰ housing stability,³¹ psychological distress (brief symptom inventory, BSI-18),³² and social support (short social support and conflictual social interaction scales).³³

Primary outcome

The primary outcome for this study is 30-day readmission (binary variable of whether or not an individual was re-hospitalized within 30 days of the initial hospital discharge date). Only the initial admission during which the patient was recruited and enrolled was considered as an index admission. A readmission could be to any hospital and if the participant was admitted to a hospital different from the index hospitalization, medical records were requested and reviewed by the research team.

Data analyses

Data analysis was conducted in two parts. The first part involved the application of an existing model² using new data (“EHR-only”) to validate its performance and the second part was to develop a new model (“EHR-plus”) utilizing the data from this trial. The EHR-only model is an external validation in a new sample (HIV-positive substance users at 11 US hospitals) of our prior EHR prediction model of 30-day readmissions (developed in a single center using data from all HIV inpatients). All variables included in the initial published model were entered into the EHR-only model.

For the EHR-only model, the same variables were entered as in the original model, with previously estimated regression coefficients, including: history of AIDS-defining illness, CD4 <92 cells/μL, absolute lymphocyte count ≤0.33 (x10⁹/L), creatinine ≤0.55 (mg/dL), creatinine > 1.77 (mg/dL), HCO₃≤18 (mmol/L), ALT or AST >35 (U/L), hematocrit ≤28.3 or >48.8 (%), p0₂ 57–133 (mmHg), Anion gap > 12 or missing (mmol/L), Medicaid insurance, lives > 13 miles from hospital (distance between individual and hospital zip codes using zipcityfunction in SAS), number of inpatient admissions in past 6 months, number of ED visits in past 12 months, and homelessness.

The second model, “EHR-plus”, is a de novo parsimonious prediction model that incorporates additional clinical and socio-behavioral variables available from the study. First, we examined the univariate relationship between readmission and each variable through logistic regression at threshold significance of p≤ 0.20. Continuous variables were divided into categories based on published cut-off values or standard clinical thresholds. Variables with theoretic overlap were assessed for collinearity. Multivariate logistic regression using a stepwise variable selection procedure, with a threshold significance of p<0.05 to remain in the model, was used to fit the final model. There were relatively few missing values except in two variables, hepatitis C antibody and patient provider relationship (around 30%, with revised N indicated in the baseline characteristics table), neither of which were significantly associated with 30-day readmission in univariate analyses. Since these two variables were not included in the multivariate model, final variable selection was based on complete data.

Both models were validated using several approaches. Model discrimination was assessed by the C-statistic. Model calibration was conducted using the Hosmer-Lemeshow χ² goodness-of-fit test. For the EHR-only model, a survival analysis of readmission-free time up to 30 days was performed comparing predicted readmission risk groups (high, medium and low). In addition, five risk categories were created based on quintiles of predicted risk and graphically represented.

As the EHR-plus model was a de novo model, cross-validation was performed by randomly splitting the dataset into 2/3 derivation and 1/3 validation datasets. The final model was refit on the derivation set and then the fitted model was applied to the validation set, obtaining a C-statistic, also known as a concordance statistic, which is equal to the area under the receiver operating curve. This operation was repeated 1000 times, and from these 1000 C-statistics we obtained the mean and 95% CI of C-statistics. A graph of the Hosmer-Lemeshow goodness-of-fit test result was created by comparing predicted versus observed risk for 30-day readmission. All statistical analyses were performed using SAS 9.4 (SAS Institute, Cary, NC).

Results

A total of 801 participants were included. The mean age was 44.1 years, and the majority of participants were male (67.4%), African-American (75.3%), never married (66.2%) and earned <$20,000/year (62.8%). In addition, 59.2% reported food insecurity and 77.6% had a history of incarceration. All participants had a history of alcohol use (58.5%) and/or substance use (97.4%), with 69.4% reporting stimulant use and 21.5% reporting opiate use in the past 12 months. There were low self-reported rates of medication adherence (17% reported taking >85% of pills in past month), and high rates of psychological distress (53.8%), and interpersonal violence (67.7%) (Table 1).

Table 1.

Baseline Characteristics of Study Population

Variable	30-day readmission N=140	No 30-day readmission N=661	Total N=801
Demographics
Age, mean years (range)	44.8 (22–67)	44 (18–68)	44.1 (18–68)
Male gender	91 (65%)	449 (67.9%)	540 (67.4%)
Race/ethnicity  Non-Hispanic white  Non-Hispanic black  Hispanic  Other	19 (13.8%) 108 (77.1%) 10 (7.1%) 3 (2.1%)	77 (11.7%) 495 (74.9%) 78 (11.8%) 11 (1.7%)	96 (12.0%) 603 (75.3%) 88 (11.0%) 14 (1.7%)
HIV risk factor  IDU  MSM+IDU  MSM  Heterosexual	41 (29.3%) 2 (1.4%) 17 (12.1%) 80 (57.1%)	175 (26.5%) 41 (6.4%) 129 (19.5%) 315 (47.7%)	216 (26.8%) 44 (5.4%) 146 (18.2%) 395 (49.3%)
Marital status  Married  Widowed  Divorced  Separated  Never married  Living with partner	10 (7.1%) 7 (5%) 16 (11.4%) 9 (6.4%) 90 (64%) 8 (5.7%)	36 (5.5%) 21 (3.2%) 76 (11.5%) 52 (7.9%) 440(66.6%) 36 (5.5%)	46 (5.7%) 28 (3.5%) 92 (11.5%) 61 (7.6%) 530 (66.2%) 44 (5.5%)
Socio-economic
Education  < High school  High school/GED  >High school	54 (38.6%) 48 (34.3%) 38 (27.1%)	265 (40.1%) 223 (37.7%) 173 (26.2%)	319 (39.8%) 271 (33.8%) 211 (26.3%)
Income, annual  $0  $1–5000  $5001–10,000  $10,001–20,000  >$20,000  Missing	19 (13.6%) 18 (12.9%) 44 (31.4%) 12 (8.6%) 5 (3.6%) 42 (30%)	60 (9.1%) 65 (9.8%) 208(31.5%) 76 (11.5%) 58 (8.8%) 194(29.4%)	79 (9.9%) 83 (10.4%) 252 (31.5%) 88 (11.0%) 63 (7.9%) 236 (29.5%)
Insurance  Any  Medicaid  Medicare  Dual  Private  Other	102 (73.4%) 60 (42.9%) 5 (3.6%) 14 (10.0%) 1 (1%) 22 (15.7%)	432 (65.9%) 211 (31.9%) 46 (7.0%) 38 (5.6%) 21 (4.9%) 116 (17.6%)	534 (67.2%) 271 (33.8%) 51 (6.4%) 52 (6.5%) 22 (4%) 138 (17.2%)
Employment  Working  Unemployed  Disabled  Other	9 (6.4%) 47 (33.5%) 75 (53.6%) 9 (6.4%)	84 (12.7%) 234 (35.4%) 324 (49.0%) 19 (2.9%)	93 (11.6%) 281 (35.1%) 399 (49.8%) 28 (3.5%)
Food insecurity  Any  Mild  Moderate  Severe	93 (66.4%) 10 (7.1%) 25 (17.9%) 58 (41.4%)	381 (57.6%) 80 (12.1%) 88 (13.3%) 209 (31.7%)	474 (59.2%) 90 (11.3%) 113 (14.1%) 267 (33.4%)
Housing (N=791)*  Homeless/shelter  Transitional housing/group home  With friends/family  Rent/own apartment/home	22 (15.8%) 12 (8.6%) 43 (30.9%) 62 (44.6%)	101(15.5%) 60 (9.2%) 175(26.8%) 316(48.5%)	123 (15.6%) 72 (9.1%) 218 (27.6%) 378 (47.8%)
Socio-behavioral
Prior Incarceration  Lifetime  Past 6 months	111 (79.9%) 20 (14.4%)	509 (77.1%) 101 (15.3%)	620 (77.6%) 121 (15.1%)
Substance use (past 12 months) Stimulants (crack, cocaine, meth)  Crack  Powder cocaine  Methamphetamines Opiates (heroin, pain pills)  Heroin  Pain pills Marijuana Ecstasy	101 (72.1%) 69 (49.3%) 48 (34.3%) 4 (2.9%) 30 (21.4%) 25 (17.9%) 10 (7.1%) 57 (40.7%) 6 (4.3%)	455 (68.8%) 302 (45.7%) 205 (31.0%) 65 (9.8%) 142 (21.5%) 18 (17.9%) 38 (5.8%) 301 (45.5%) 31 (4.7%)	556 (69.4%) 371 (46.3%) 253 (31.6%) 69 (8.6%) 172 (21.5%) 143 (17.9%) 48 (6.0%) 358 (44.7%) 37 (4.6%)
Substance use  AUDIT (>7 for men, >6 for women)  DAST-10 (≥6)  Severe substance use (AUDIT or DAST-10)	62 (44.3%) 59 (41.6%) 92 (66.7%)	287 (43.4%) 275 (42.1%) 441 (65.7%)	349 (43.6%) 334 (41.7%) 553 (66.5%)
Substance use treatment attitudes, mean (SD)  Readiness  Negative attitude	13.6 (4.5) 13.1 (3.6)	14.3 (4.1) 13.3 (3.5)	14.2 (4.2) 13.3 (3.5)
Smoking history	38 (27.1%)	203 (30.7%)	241 (30.1%)
Adherence to HIV medications in last month  Taking >85% of pills  ART pills taken in last mo,% (mean,SD)	26 (18.6%) 56.7 (36.6)	110 (16.6%) 57.3 (37.6)	136 (17.0%) 57.2 (37.3)
Adherence self efficacy (mean, SD)  Integration  Perseverance  Total score	66.8 (22.8) 22.1 (8.4) 88.8 (29.8)	68.1 (21.4) 22.6 (7.7) 90.7 (28.0)	67.9 (21.7) 22.5 (7.8) 90.4 (28.3)
Patient provider relationship (N=548)* (mean, SD)  Satisfaction  Trust  Total	13.9 (4.0) 24.2 (5.0) 108.2 (27.8)	13.8 (4.0) 24.4 (4.7) 107.6 (26.1)	13.8 (4.0) 24.3 (4.8) 107.7 (26.4)
Medical mistrust (mean, SD)	29.3 (7.8)	28.5 (7.8)	28.7 (7.8)
Health literacy (mean, SD)	8.6 (3.2)	9.1 (3.1)	9.0 (3.1)
Perceived health status (mean, SD)	32.3 (9.5)	34.0 (9.0)	33.7 (9.2)
Interpersonal violence, any	103 (73.6%)	439 (66.4%)	542 (67.7%)
Psychological distress (BSI)  Anxiety/panic  Somatization  Depression  Total score	68 (48.6%) 81 (57.9%) 68 (48.6%) 84 (60%)	298 (45.0%) 385 (58.2%) 271 (41.0%) 347 (52.5%)	366 (45.7%) 466 (58.2%) 339 (42.3%) 431 (53.8%)
Social support/ conflictual interaction	22.0 (7.7)	22.0 (7.5)	22.0 (7.5)
Access to care (mean, SD)	6.3 (4.7)	6.2 (4.8)	6.2 (4.8)
Distance from hospital, miles (mean, SD) Median miles, IQR	5.8 (7.4) 4.5 (2.2, 7.6)	12.2 (53.8) 4.1 (2.2,7.2)	11.1 (49.0) 4.2 (2.2,7.5)
Hospitalization details
Length of hospital stay, days (mean SD)	6.5 (5.7)	6.5 (7.1)	6.5 (6.9)
Primary diagnosis on admission  AIDS defining illness  Infection, non AIDS defining   CD4<200  Cardiovascular  Endocrine/Nutrition/Electrolytes/Other  Gastrointestinal/Liver  Hematology/Oncology  Injury  Neurology  Psychiatry  Pulmonary  Renal  Substance use	15 (10.7%) 47 (33.6%) 40 (28.6%) 18 (12.9%) 7 (5.0%) 22 (15.7%) 2 (1.4%) 0 (0%) 7 (5.0%) 5 (3.7%) 11 (7.9%) 3 (2.1%) 3 (2.1%)	61 (9.3%) 247 (37.4%) 167 (25.3%) 46 (7.0%) 30 (4.5%) 59 (8.9%) 13 (2.0%) 38 (5.8%) 41 (6.2%) 14 (2.1%) 87 (13.2%) 15 (2.3%) 9 (1.4%)	76 (9.5%) 294 (36.8%) 207 (25.8%) 64 (8.0%) 37 (4.6%) 81 (10.1%) 15 (1.9%) 38 (4.8%) 48 (6.0%) 19 (2.4%) 98 (12.3%) 18 (2.3%) 12 (1.5%)
On ART prior to admission	62 (44.3%)	247 (37.4%)	390 (38.6%)
Started on ART during admission	25 (17.9%)	99 (15.0%)	124 (15.5%)
Discharged with supply of ART	23 (16.4%)	120 (18.3%)	143 (18.0%)
Referrals received as inpatient  Case management  Drug treatment  Housing  Mental health  Outpatient HIV clinic  Any referral	23 (16.4) 18 (12.9%) 14 (10.0%) 13 (9.3%) 78 (55.7%) 91 (65%)	146 (22.3%) 109 (16.6%) 92 (14.0%) 56 (8.5%) 375 (57.2%) 448 (67.8%)	169 (21.2%) 127 (16.0%) 106 (13.3%) 69 (8.7%) 453 (56.9%) 539 (67.3%)
Laboratory values/illness severity
CD4 count  ≤50  51–200  >200	62 (44.3%) 37 (26.4%) 41 (29.3%)	207 (31.3%) 224 (33.9%) 230 (34.8%)	269 (33.6%) 261 (32.6%) 271(33.8%)
HIV viral load  Median (IQR), x1000 copies/mL  Undetectable ≤200 copies/mL  ≤1000  1001–10,000  10,001–100,000  >100,000	56.5 (3.8–229.6) 18 (12.9%) 28 (20.0%) 18 (12.9%) 37 (26.4%) 57 (40.7%)	52.4 (6.6–194.8) 69 (10.4%) 110 (16.6%) 76 (11.5%) 227 (34.3%) 248 (37.5%)	52.8 (5.2–199.2) 87 (10.9%) 138 (17.2%) 94 (11.7%) 264 (33.0%) 305 (38.1%)
HCV Ab positive (N=559)*	35 (37.6%)	168 (36.1%)	203 (36.3%)
Laboratory values  WBC < 4.2 × 10(9)/L  Absolute lymphocytes < 1.3 × 10(9)/L  HCT <39.6% (men), <34.1% (women)  Creatinine <0.67(mg/dL)  Creatinine >1.17 (mg/dL)  AST > 50 U/L (men) or >35 U/L (women)  ALT > 50 U/L (men) or >35 U/L (women)	128 (91.4%) 106 (75.7%) 108 (77.1%) 21 (15.0%) 43 (30.7%) 56 (40%) 35 (25%)	562 (85.0%) 466 (70.5%) 455 (68.8%) 82 (12.4%) 144 (21.8%) 224 (33.9%) 132 (20.0%)	690 (86.1%) 572 (71.4%) 563 (70.3%) 103 (12.9%) 187 (23.4%) 280 (35.0%) 167 (20.9%)
HIV-dementia scale (≤ 10)	44 (31.4%)	162 (24.5%)	206 (25.7%)
Karnofsky ≥ 60 and < 80	69 (49.3%)	265 (40.1%)	334 (41.7%)
Healthcare utilization
HIV primary care visit in past 6 months	66 (47.1%)	299 (45.2%)	365 (45.6%)
ED visits in past 12 months  ≤1  2–4  ≥ 5	31 (22.1%) 66 (47.1%) 43 (30.7%)	289 (43.7%) 272 (41.1%) 100 (15.1%)	320 (40.0%) 338 (42.2%) 143 (17.6%)
Hospitalizations in past 6 months  0  1  ≥2	34 (24.3%) 42 (30.0%) 64 (45.7%)	321 (48.6%) 172 (26.0%) 168 (25.4%)	355 (44.3%) 214 (26.7%) 232 (29.0%)

^*for missing data, revised N is noted

Categorical measures are depicted as n (%).

IDU= injection drug use; MSM= men who have sex with men; GED= General Education Diploma; AUDIT= alcohol use disorders identification test; DAST-10= drug abuse screening test; SD= standard deviation; BSI=brief symptom inventory; IQR= interquartile range; AIDS= acquired immune deficiency syndrome; ART= antiretroviral therapy; HCV Ab =hepatitis C virus antibody; WBC= white blood cell count; HCT= hematocrit; AST= aspartate aminotransferase; ALT= alanine aminotransferase; ED= emergency department

Overall, 140/801(17.5%) individuals had a hospital readmission within 30 days of the initial admission. Mean time to readmission was 15.1 days. The most common diagnoses at the time of index admission were infections that were non-AIDS defining (294, 36.8%), such as pneumonia, skin and soft tissue infections and bacteremia, though 70% (207/294) of this group met the clinical definition of AIDS (CD4<200). Other common diagnostic categories were pulmonary complaints (12.3%, including shortness of breath, chronic obstructive pulmonary disease), Gastro-intestinal (GI) illness (10.1%, including liver disease, GI bleeding, abdominal pain) and AIDS-defining illnesses (9.5%, such as candida esophagitis, pneumocystis pneumonia). Only 38.6% reported receiving antiretroviral therapy (ART) prior to admission and 15.5% were started on ART during the index admission (Table 1).

External validation of EHR-only risk prediction model

The previously published prediction model using this new dataset resulted in a C-statistic of 0.65 (95% CI: 0.60 – 0.70). Some but not all variables were significant in this model (see Table 2). Survival analyses (Figure 1a) show the difference in time to readmission in the first 30 days after discharge in the three risk groups--high, medium and low- with the high-risk group experiencing higher numbers of readmissions sooner after discharge. Figure 1b further demonstrates the calibration of the EHR-only model by stratifying 30-day readmission rate by risk group in quintiles, from very low risk group (8.8% readmitted) to very high risk group (28.8% readmitted).

Table 2.

External Validation of Multivariate Prediction Model for 30-day Hospital Readmissions Among HIV-Infected Individuals

Variable	aOR	95% CI	p value
Clinical
History of AIDS defining illness	0.87	0.58–1.32	0.52
CD4 <92 (cells/μL)	1.61	1.07–2.42	0.02
Absolute lymphocyte count <=0.33 (x 109/L)	0.93	0.61–1.44	0.75
Creatinine ≤ 0.55 (mg/dL)	1.02	0.39–2.68	0.96
Creatinine > 1.77(mg/dL)	2.35	1.28–4.34	0.006
HCO3 ≤18 (mmol/L)	0.95	0.51–1.78	0.86
AST or ALT >35 (U/L)	1.34	0.91–1.98	0.14
HCT ≤ 28.3 or >48.8 (%)	0.85	0.53–1.41	0.53
PO2 (57–113) (mmHg)	1.07	0.23–5.01	0.94
Anion gap >12 or missing (mmol/L)	0.56	0.33–0.96	0.04
Nonclinical
Medicaid insurance	1.44	0.97–2.15	0.07
Lives >13 miles from hospital	0.77	0.33–1.41	0.23
Number prior inpatient admits in past 6 mo	1.99	1.47–2.71	<0.001
Number prior ED visits in past 12 mo	2.08	1.30–3.30	0.002
Homeless	0.79	0.46–1.36	0.40

C statistic=0.646

AIDS= acquired immune deficiency syndrome; HC03= Bicarbonate; AST= aspartate aminotransferase; ALT= alanine aminotransferase; HCT= hematocrit; P02= partial pressure of Oxygen; ED= emergency department

Figure 1a.

Proportion of Participants Remaining Readmission-free up to 30 days, by EHR-only Model Predicted Readmission Risk Group

Figure 1b.

Thirty-Day Readmission Rate by EHR-only Model Predicted Readmission Risk Category

EHR-plus prediction model

Variables that were associated with 30-day readmission in univariate analyses include HIV risk factor, low income, unemployment, Medicaid insurance, food insecurity, low readiness for substance use treatment, low health literacy, low perceived health status, interpersonal violence, depression, CD4 count, low Karnofsky score, high HIV dementia score, abnormal complete blood counts, and abnormal kidney function, admission diagnoses related to GI and cardiovascular diseases and prior ED visits and hospitalizations (Table 3).

Table 3.

Predictors of 30-day Readmissions: Univariate Analyses

Variable	Unadjusted OR (95% CI)	P value
Demographics
Age	1.01 (0.99, 1.03)	0.37
Male gender	0.88 (0.60, 1.29)	0.50
Race/ethnicity  Hispanic/other v non-Hispanic white  Non-Hispanic Black v non-Hispanic white	0.59 (0.27,1.28) 0.88 (0.51, 1.52)	0.18 0.66
HIV Risk factor  IDU v MSM  Heterosexual v MSM	1.50 (0.82, 2.75) 1.93 (1.10, 3.38)	0.18* 0.02*
Marital status  Married/living together v other	1.21 (0.69, 2.10)	0.50
Treatment group  PN-CM v. PN  TAU v. PN	0.74 (0.48,1.15) 0.76 (0.49, 1.19)	0.18 0.23
Socio-economic
Education  High school or greater	1.07 (0.75, 1.55)	0.74
Income, annual  <$10,000/year v other**	1.84 (1.07, 3.16)	0.03*
Insurance  Any  Medicaid^	1.43 (0.95, 2.15) 1.86 (1.29, 2.68)	0.09* 0.001*
Employment  Disabled v working  Other v working	2.16 (1.04, 4.49) 2.07 (0.98, 4.36)	0.04* 0.06*
Food insecurity,  Moderate-severe v none/mild	1.79 (1.23, 2.59)	0.002*
Housing  Homelessness	1.03 (0.63, 1.71)	0.89
Socio-behavioral
Prior Incarceration, any  Past 6 months	1.18 (0.75, 1.85) 0.93 (0.55, 1.56)	0.48 0.78
Substance use (past 12 months)  Simulants  Opiates  Polysubstance use (≥2)	1.17 (0.78, 1.76) 1.00 (0.64, 1.55) 0.83 (0.58, 1.20)	0.44 0.99 0.33
Substance use  AUDIT  DAST-10  Severe substance use (AUDIT or DAST-10)	1.04 (0.72, 1.50) 1.02 (0.71, 1.48) 0.96 (0.65, 1.40)	0.85 0.91 0.82
Substance use treatment attitudes  Readiness  Negative attitude	0.97 (0.94, 1.00) 0.98 (0.93, 1.03)	0.06* 0.51
Smoking	0.84 (0.56, 1.26)	0.40
Adherence to antiretrovirals  Taking >85%	1.14 (0.71, 1.83)	0.58
Adherence self-efficacy  Integration  Perseverance  Total score	0.99 (0.99, 1.00) 0.99 (0.97, 1.02) 0.99 (0.99, 1.00)	0.54 0.46 0.46
Patient provider relationship (N=548)	1.00 (0.99,1.01)	0.85
Medical mistrust	1.01 (0.99, 1.04)	0.30
Health literacy	0.96 (0.91, 1.01)	0.12*
Perceived health status	0.98 (0.96, 1.00)	0.06*
Interpersonal violence	1.41 (0.94, 2.12)	0.10
Psychological distress (BSI)  Anxiety/panic  Somatization  Depression  Any	1.15 (0.80, 1.66) 0.98 (0.68, 1.42) 1.36 (0.94, 1.96) 1.36 (0.94, 1.97)	0.45 0.93 0.10 0.11
Social support	1.00 (0.98, 1.03)	1.00
Access to care	1.00 (0.97, 1.05)	0.73
Distance from hospital  >13 miles from hospital	0.99 (0.97, 1.01) 0.63 (0.32, 1.25)	0.27 0.19
Hospitalization details
Length of stay	1.03 (0.97, 1.03)	0.97
Primary diagnosis on admission  AIDS defining illness  Infection, non AIDS defining  Cardiovascular  Gastrointestinal/Liver  Neurology  Psychiatry  Pulmonary  Substance use	1.18 (0.65, 2.14) 0.85 (0.58, 1.24) 1.97 (1.11, 3.52) 1.90 (1.22, 3.23) 0.80 (0.35, 1.81) 1.71 (0.61, 4.83) 0.56 (0.29, 1.08) 1.59 (0.43, 5.94)	0.59 0.40 0.02* 0.02* 0.59 0.31 0.09* 0.49
On ART prior to admission	1.33 (0.92, 1.93)	0.13
Started on ART during admission	1.23 (0.76, 2.00)	0.39
Discharged with supply of ART	0.88 (0.54, 1.43)	0.60
Referrals received as inpatient  Case management  Drug treatment  Housing  Mental health  Outpatient HIV clinic  Any referral	0.68 (0.43, 1.11) 0.74 (0.43, 1.27) 0.68 (0.38, 1.24) 1.10 (0.58, 2.07) 0.94 (0.65, 1.36) 0.88 (0.60, 1.30)	0.12* 0.27 0.21 0.77 0.75 0.52
Laboratory values/illness severity
CD4 count, cells/uL  <50 v. >200  50–200 v. >200	1.68 (1.09, 2.60) 0.93 (0.57, 1.50)	0.02* 0.76
HIV viral load (log)  Undetectable <200 v other	0.99 (0.94, 1.05) 1.27 (0.73, 2.20)	0.81 0.40
HCV Ab status (N=559)	1.07 (0.68, 1.70)	0.77
Laboratory values  WBC < 4.2 × 10(9)/L  Absolute lymphocytes < 1.3 × 10 (9)/L  HCT <39.6% (men), <34.1% (women)  Creatinine <0.67(mg/dL)  Creatinine> 1.17 (mg/dL)  AST > 50 U/L (men) or >35 U/L (women)  ALT > 50 U/L (men) or >35 U/L (women)	1.88 (1.00, 3.53) 0. 69 (0.50, 0.97) 0.96 (0.93, 0.99) 1.25 (0.74, 2.09) 1.59 (1.06, 2.38) 1.00 (0.99, 1.00) 1.00 (0.99, 1.00)	0.05* 0.03* 0.01* 0.41 0.02* 0.99 0.86
HIV-dementia scale (<= 10)	1.41 (0.95, 2.10)	0.09*
Karnofsky <80	1.45 (1.01, 2.09)	0.05*
Healthcare utilization
HIV primary care visit in past 6 months	1.08 (0.75, 1.56)	0.68
ED visits in past 12 months  ≤1  2–4  >= 5	Ref 2.26 (1.43, 3.58) 4.01 (2.40, 6.71)	-- <0.001* <0.001*
Hospitalizations in past 6 months  0  1  ≥2	Ref 2.3 1(1.41, 3.76) 3.60 (2.28, 5.67)	-- <0.001* <0.001*

^**missing included in low income category

^{^}includes dual insurance (Medicare + Medicaid)

^*p value ≤ 0.20

IDU= injection drug use; MSM= men who have sex with men; PN-CM= patient navigation with contingency management; PN= patient navigation; TAU= treatment as usual;AUDIT= alcohol use disorders identification test; DAST-10= drug abuse screening test; SD= standard deviation; BSI=brief symptom inventory; AIDS= acquired immune deficiency syndrome; ART= antiretroviral therapy; HCV Ab=hepatitis C virus antibody; WBC= white blood cell count; HCT= hematocrit; AST= aspartate aminotransferase; ALT= alanine aminotransferase; ED= emergency department

In multivariate modeling, significant independent predictors of 30-day readmissions included variables from multiple categories, including socio-economic (food insecurity), socio-behavioral (readiness for substance use treatment), clinical illness severity (CD4 category (<50 compared to >200 cells/μL), elevated creatinine (>1.17mg/dL), primary admitting diagnosis of cardiovascular or gastrointestinal disease), and healthcare utilization (ED visits (≥2 in past 12 months) and hospitalizations (number in past 6 months)) (Table 4). Model discrimination of the EHR-plus as measured by the C-statistic was 0.74. Model calibration, using cross-validation, resulted in a C-statistic of 0.71 (95% CI 0.64–0.77). The comparison of predicted versus observed readmission, as per the Hosmer-Lemeshow method is graphed in Supplemental Figure 1.

Table 4.

Multivariate Model of Predictors of 30-day Readmissions Among HIV-positive Individuals, EHR-plus model

Variable	aOR	95% CI	p value
Clinical
CD4 category, cells/μL  <50  51–200  >200	1.69  0.73  Ref	1.06–2.71 0.43–1.23 --	0.002
Creatinine >1.17 (mg/dL)	1.83	1.02–3.27	0.04
Primary admitting diagnosis  Cardiovascular disease (v. other)  Gastrointestinal disease (v. other)	2.07  1.87	1.09–3.92  1.04–3.35	0.03  0.04
Nonclinical
Medicaid insurance	1.39	0.93–2.08	0.11
ED visits in past 12 months, ≥ 2 (v. <2)	1.84	1.15–2.95	0.01
Number of hospital admissions in past 6 months	1.22	1.12–1.34	<0.01
Food Insecurity (moderate-severe v. mild/none)	1.67	1.11–2.49	0.01
Readiness for substance use treatment	0.94	0.90–0.99	0.01

EHR= electronic health record; ED= emergency department

Discussion

Our external validation of a published prediction model for 30-day readmissions among PLWH in a multicenter population of HIV-positive individuals with substance use disorder performed moderately well, with a C-statistic of 0.65 (95%CI 0.60–070). In comparison to this EHR-only model, the EHR-plus model, which included detailed socio-economic and socio-behavioral variables from patient interviews, resulted in a substantially stronger prediction model, with a C-statistic of 0.74 (0.71 with cross validation, 95% CI 0.64–0.77). The addition of two social predictors of readmission, food insecurity and readiness for substance use treatment, resulted in considerable improvement in prognostic capacity for 30-day readmission. The EHR-plus model also contained previously identified predictors of readmission (CD4 count, renal dysfunction, prior acute care utilization), and highlighted the role of cardiovascular and GI diseases in readmissions among this population.

The independent predictive value of these sociobehavioral variables to 30-day readmissions, which are not typically available in the EHR, underscore the impact of key social determinants influencing readmission risk and the need to ask about and address them, especially in safety net populations. Frequently cited prediction models, such as LACE³⁴ and HOSPITAL,³⁵ do not include any social or behavioral predictors, which may limit their use in safety net populations (where LACE index had a C-statistic of 0.56 among congestive heart failure patients).³⁶ In two different systematic reviews of readmission prediction models, very few models included variables involving social determinants of health.^37,38 A 2011 review of 26 models (average C statistic= 0.66) cited 2 models which included social determinants of health,³⁸ and a 2018 review (range of C statistics was 0.21–0.88) identified a small proportion of models which included factors such as living arrangement, marital status and substance use.³⁷ Given mounting evidence that social and behavioral factors are strong predictors of readmissions (as well as morbidity and mortality), hospital systems will be incentivized to measure these domains as part of routine clinical care or to employ tools such as natural language processing to extract key factors from the EHR.³⁹ The National Academy of Medicine has identified social and behavioral domains related to health outcomes (including alcohol use and financial strain for acquiring food), with a focus on systematic and efficient incorporation of these factors into the EHR, creating an opportunity for future automated EHR-plus prediction models.⁴⁰

Food insecurity and its role in HIV outcomes is well documented, as inadequate access to quality nutrition has been associated with decreased adherence to ART and lower virologic suppression rates.^41,42,43 In addition, and particularly relevant to this study population, there may be synergistic effects of food insecurity and substance use that impede adherence to ART in people with HIV.^44,45 These associations persist after adjusting for neighborhood poverty, transportation and housing, indicating that food insecurity is uniquely associated with adherence. Similarly, we found that food insecurity was not collinear with low income or Medicaid status, indicating that these more commonly available variables may inadequately estimate the impact of food insecurity. In our study, food insecurity may follow similar pathways to those noted in other studies, leading to ART nonadherence which contributes to readmission. Food assistance for PLWH has been shown to have a positive impact on ART adherence⁴⁶ and others have proposed addressing food insecurity to reduce hospital readmissions in different populations.⁴⁷ Access to safe and adequate nutrition is not routinely assessed during hospitalizations, but may be a sensitive indicator of a patient’s capacity to prioritize healthcare needs over other survival needs.

Readiness for substance use treatment was independently associated with decreased 30-day readmissions in this study population of individuals who were either heavy alcohol or substance users or both. Substance use disorders among hospitalized patients has been associated with increased subsequent acute care hospital utilization in other urban populations.⁴⁸ In addition, engagement in substance use treatment from the inpatient setting may decrease ED utilization and increase ambulatory visits.⁴⁹ Therefore, readiness for substance use treatment may indicate a willingness to change behavior, including adherence to medications or visits. Hospitalization itself may serve as a catalyst for behavior change among substance users. In a study assessing readiness to change substance use behaviors among inpatients, tested at baseline and every three days thereafter, 43.6% of subjects increased to a higher stage or remained in the action stage of behavior change.⁵⁰ These findings and our own underscore the potential impact of direct linkage to substance use treatment services from the inpatient setting on subsequent healthcare utilization.

Our study has several implications for policies and providers. It provides empiric evidence for the importance of measuring and addressing social determinants of health during hospital admissions, and specifically for assessing food insecurity and substance use treatment readiness. Our findings may inform current readmission metrics and their equitable application to different hospital settings including safety net hospitals. In addition, our findings have direct implications for interventions, such as rapid ART initiation for those with low CD4 counts, addressing cardiovascular and GI co-morbidities, food assistance programs for the food insecure, and direct linkage to substance use services for those expressing readiness for treatment.

Several limitations are worth noting. First, the study population is comprised of hospitalized HIV-positive substance users, a group that is majority non-white, socially vulnerable and who mainly accesses care at safety net hospitals. Though this may appear to limit the generalizability of our findings, a large proportion of hospitalized PLWH in our current era share these characteristics^10,11,51 and may benefit from readmission reduction interventions.⁵² Second, given the sample size, we did not adjust for study site in our analyses, and therefore were unable to adjust for hospital-specific effects, though the multicenter nature of the study does enhance its relevance to other U.S. hospitals. Third, the study population is a select group who was willing to participate in a randomized trial and received extensive follow-up for intervention purposes (care navigation and financial incentives for 6 months in intervention arms) as well as ongoing contact and outreach for study retention, which may not be generalizable to the general population. Lastly, we had a modest number of events, with only 140 of 801 participants experiencing 30-day readmissions; therefore we may not be powered to detect weak predictors of readmissions in our study. However, our readmission rate (17.5%, allowing only one readmission per individual) is comparable to this same metric in other published studies involving PLWH, 14.6% and 15%.^53,10

Conclusions

In sum, we present the results of an external validation of an EHR-based readmission prediction model for HIV-positive individuals in a new multicenter population. In addition, by incorporating extensive social and behavioral factors into a new prediction model, we have improved our prediction model performance, and have identified two additional independent sociobehavioral predictors of readmission: food insecurity and low readiness for substance use treatment. In addition to providing important potential targets for future interventions among PLWH, these findings have broader implications for healthcare systems. In an era of value-based care, where hospital systems need to optimize their use of medical informatics and are simultaneously compelled to recognize the critical (and costly) impact of social determinants on health outcomes, the collection and integration of social and behavioral variables into the EHR is imperative to improving outcomes.

Supplementary Material

Supplemental Figure 1

Supplemental Figure 1 Calibration of EHR-plus model based on Hosmer-Lemeshow goodness-of-fit test. Circles represent observed versus predicted 30-day readmissions within each decile of risk. Dashed line represents situation in which predicted and observed readmission rates are identical.