Abstract
Background
Driven by quality outcomes and economic incentives, predicting 30-day hospital readmissions remains important for healthcare systems. The Cleveland Clinic Health System (CCHS) implemented an internally validated readmission risk score in the electronic medical record (EMR).
Objective
We evaluated the predictive accuracy of the readmission risk score across CCHS hospitals, across primary discharge diagnosis categories, between surgical/medical specialties, and by race and ethnicity.
Design
Retrospective cohort study.
Participants
Adult patients discharged from a CCHS hospital April 2017–September 2020.
Main Measures
Data was obtained from the CCHS EMR and billing databases. All patients discharged from a CCHS hospital were included except those from Oncology and Labor/Delivery, patients with hospice orders, or patients who died during admission. Discharges were categorized as surgical if from a surgical department or surgery was performed. Primary discharge diagnoses were classified per Agency for Healthcare Research and Quality Clinical Classifications Software Level 1 categories. Discrimination performance predicting 30-day readmission is reported using the c-statistic.
Results
The final cohort included 600,872 discharges from 11 Northeast Ohio and Florida CCHS hospitals. The readmission risk score for the cohort had a c-statistic of 0.6875 with consistent yearly performance. The c-statistic for hospital sites ranged from 0.6762, CI [0.6634, 0.6876], to 0.7023, CI [0.6903, 0.7132]. Medical and surgical discharges showed consistent performance with c-statistics of 0.6923, CI [0.6807, 0.7045], and 0.6802, CI [0.6681, 0.6925], respectively. Primary discharge diagnosis showed variation, with lower performance for congenital anomalies and neoplasms. COVID-19 had a c-statistic of 0.6387. Subgroup analyses showed c-statistics of > 0.65 across race and ethnicity categories.
Conclusions
The CCHS readmission risk score showed good performance across diverse hospitals, across diagnosis categories, between surgical/medical specialties, and by patient race and ethnicity categories for 3 years after implementation, including during COVID-19. Evaluating clinical decision-making tools post-implementation is crucial to determine their continued relevance, identify opportunities to improve performance, and guide their appropriate use.
Supplementary Information
The online version contains supplementary material available at 10.1007/s11606-021-07277-4.
KEY WORDS: hospital readmission, electronic medical record, decision support model
Driven by quality outcomes and economic incentives, predicting 30-day hospital readmissions remains an important focus for healthcare systems. Several models to predict hospital readmission have been previously published.1–7 The ultimate goal of predicting readmission risk is to improve the quality of care provided to patients at high risk for worsening health upon discharge from the hospital, and efforts to reduce 30-day readmissions are also linked to payment for hospitals.8 However, readmissions may vary in different contexts such as by hospital or patient characteristics9–14 and risk prediction models that are initially validated and utilized in a health system may need to be reevaluated once implemented to assess longer-term performance15 and specifically across different groups. The Cleveland Clinic Health System (CCHS) created and validated a 30-day readmission risk score and implemented the score reporting in our electronic medical record (EMR). The goals of this study were to confirm that (1) over time the readmission risk score model was performing acceptably overall (2) to assess whether there was variation in performance in particular contexts. Thus, we sought to report the performance of the readmission risk score implementation for predicting all-cause readmissions over 3 years across diverse hospitals in the CCHS; medical and surgical specialties; primary discharge diagnosis categories, including the time frame of the 2020 COVID-19 pandemic; and race and ethnicity.
METHODS
Initial Readmission Risk Score Development
The CCHS is a large, integrated health system with national and international locations. For the current analysis, Northeast Ohio and Florida hospitals were included. A model to predict 30-day hospital readmissions was created in the CCHS in 2017 for operational purposes using EMR data from all hospital admissions in the CCHS in 2015 and validated using a cohort from January to June 2016, including observation admissions and excluding Oncology and Labor and Delivery admissions given their unique context.
Model Variables
Model variables were chosen based upon evidence from published literature and input from clinical experts. The final model contains 18 variables available in the EMR (Appendix Table 3) including previous healthcare utilization in the CCHS (number of admissions in the past year, total number of emergency department [ED] visits in the past 6 months, and whether there was an ED visit in the past 6 months); admission class (whether admission was an observation type); discharge disposition (home, home with home care, hospice, other, skilled nursing facility); comorbidities including history of chronic kidney disease, chronic lung disease, and anemia; drug abuse; medications (number of active medications, whether there were any anticoagulation medications); laboratory values (hemoglobin and calcium at admission; blood urea nitrogen, albumin, and sodium at discharge); insurance status (Medicaid, Medicare, private, self-pay, other); and a variable to capture language barrier, poor health literacy, or impaired cognition.
Model Development
The model was created using a multiple logistic regression and was reduced to find the best-fitting model with the fewest number of variables. Starting with over 200 candidate variables, model reduction was performed using a “stepdown” model reduction process that ranks each variable on the reduction of the R2 from smallest to largest. Variables were removed from the model according to their rank, and the process was stopped when the change in the area under the receiver operating characteristic curve (AUC) or c-statistic was less than 0.01. Validation on the 2016 data was done with an imputation of missing variables using the mean of that variable from the cohort. This imputation process was a simple imputation that could be applied to the automation of the model within the medical record. The odds ratios for the initial validation are shown in Appendix Table 3. The model obtained a c-statistic of 0.723 when validated on the 2016 cohort. The performance on the validation cohort compared favorably to that of the previously published HOSPITAL2 and LACE7 models (c-statistic 0.65 and 0.69, respectively).
Identification of High-Risk Readmission Score
A risk score of 0–100 to predict readmission was generated from the model with a high risk score identified as ≥ 40 calculated to capture the top 5% of patients at risk for 30-day readmission. Identifying the top 5% of patients at risk for readmission also allowed for feasibility of implementing transitional care interventions. When tested on the 2016 validation cohort, this cutoff score had a sensitivity of 16%, specificity of 97%, a positive predictive value of 52%, and a negative predictive value of 85%.
Model Implementation in EMR
The readmission risk score model was implemented in the Epic EMR system16 in April 2017 using configurable rule logic records and assigned a score to all applicable patients daily during the hospital admission with a final score on the day of discharge.
Some modifications to variable definitions were made to accommodate performance considerations, such as limiting lookback to 1 year of history, and to use existing fields and categories available as input to Epic rules. Additional modifications were made to incorporate missing data or data not yet present upon admission. The population mean (continuous variables), or population mode (categorical variables), was imputed for missing data. If the variable became available during the admission, e.g., new lab result, then the daily score would update using the patient’s specific data. The score can also be calculated in real time when viewing a patient record. The score and the patient variables contributing to the score can be included as columns in patient lists. A readmission report available on each patient record shows a line chart of the history of the patient’s daily risk scores, a description of how the score is calculated, and the specific variables contributing to the score, both describing factors that may increase the patient’s readmission risk and factors protective against readmission. The data for the specific variables contributing to the score is stored as variable names and numeric values of each named variable. For patients with a risk score of at least 40, a red banner appears identifying the patient as high risk.
Implementation Analysis
This retrospective cohort study utilized data obtained from CCHS EMR and billing databases from April 1, 2017, through September 3, 2020. All patient demographic information including race and ethnicity was obtained as recorded in the EMR. All patients aged ≥ 18 years discharged from a CCHS hospital were included except patients discharged from Oncology or Labor and Delivery departments or with attending providers specializing in cancer care, patients with hospice orders, or patients who died during the admission. Discharges were categorized as surgical when identified in the record as a surgical department or if surgery was performed during admission. All other admissions were categorized as medical. Primary discharge diagnoses were categorized according to Agency for Healthcare Research and Quality (AHRQ) Clinical Classifications Software (CCS) Level 1 categories.17 Primary discharge diagnoses are available as International Classification of Diseases (ICD) 10 codes in each patient’s billing data. ICD 10 codes were categorized according to AHRQ CCS ICD-10-CM beta version level 1 categories. The beta version of the CCS includes ICD-10-CM codes valid through September 2019. As our study time frame was through September 3, 2020, any ICD-10 codes newer than September 2019 were categorized by our research team in the same level 1 category as pre-existing codes from the same ICD-10 range if possible, or manually categorized if not.
Every admission in this analysis has a binary outcome value calculated for whether the patient was readmitted to any CCHS hospital within 30 days. The final recorded risk score was used to assess readmission risk score performance of observed vs. expected readmissions, assessing separately the variation according to the hospital site, medical vs. surgical specialty, and a categorization of the primary discharge diagnosis. For each category, the risk score’s discrimination performance in predicting all-cause 30-day readmission is reported using the c-statistic as well as the Brier score for prediction accuracy. A 95% confidence interval for the c-statistic was calculated from 1000 bootstrap samples. Variation in performance was assessed by comparison of each category’s c-statistic and confidence interval to the overall cohort c-statistic. In addition, given that performance of models may differ based upon race or ethnicity, 18,19 we performed subgroup analyses to assess performance of the readmission risk score by race and ethnicity. Calibration is presented in visual plots by dividing the observations into bins with an even number of data points per bin. The mean predicted risk is calculated for each bin and compared to the proportion of observed readmissions. In a perfectly calibrated model, the predicted and observed readmissions would be equal, represented by a line at a 45° angle. A sensitivity analysis was done assessing readmission rates and model performance excluding the COVID-19 pandemic period from March 1 to September 3, 2020. To assess the frequency of contributions of the specific variables contributing to the readmission risk score, we extracted for the year 2019—which is the last full year of data in our study period prior to the COVID-19 pandemic—the section of the discharge summary containing the individual variables following the header “Patient variables contributing to increased readmission risk.” Regular expressions were used to extract the variable names and numeric values of each named variable. Python version 3.7 was used for all analyses. This project was reviewed and classified as exempt by the CCHS Institutional Review Board.
RESULTS
The final retrospective cohort included 600,872 discharges for 321,470 unique patients from 11 acute care hospital sites in the CCHS including 10 in Northeast Ohio and 1 in Florida (Appendix Fig. 4, Table 1). The mean readmission score was 18.2 [standard deviation (SD) 11.0]. The mean age of the patients was 62.8 years (SD 17.7); there were 52.5% females; race categories included less than 1% American Indian or Alaskan Native or Asian, 21.8% Black, less than 1% Native Hawaiian or Pacific Islander, 63.5% White, 1.5% multiracial, less than 1% other race, and 12.4% unknown race and with 4.5% of Hispanic ethnicity and 1.8% of unknown ethnicity. There were 60.2% medical discharges. The most frequent CCS diagnosis categories included diseases of the circulatory system (25.3%), diseases of the digestive system (12.9%), injury and poisoning (11.1%), and diseases of the respiratory system (9.7%).
Table 1.
Characteristics of All Cleveland Clinic Health System Discharges by Readmission Status April 2017–September 2020
| Missing | Overall | Not readmitted | Readmitted | ||
|---|---|---|---|---|---|
| Number | 600,872 | 505,375 | 95,497 | ||
| Recorded score, mean (SD) | 0 | 18.2 (11.0) | 16.9 (9.9) | 24.7 (13.7) | |
| Age, mean (SD) | 0 | 62.8 (17.7) | 62.5 (17.8) | 64.3 (17.1) | |
| Length_of_stay, mean (SD) | 1566 | 3.8 (5.8) | 3.6 (5.6) | 5.1 (6.9) | |
| Number_past_all_admissions, mean (SD) | 1566 | 1.2 (2.4) | 1.0 (1.8) | 2.6 (4.2) | |
| Gender, N (%) | Female | 1566 | 314,757 (52.5) | 265,041 (52.6) | 49,716 (52.2) |
| Male | 284,548 (47.5) | 239,021 (47.4) | 45,527 (47.8) | ||
| NA | 1 (0.0) | 1 (0.0) | |||
| Race, N (%) | American Indian/Alaska Native | 0 | 700 (0.1) | 569 (0.1) | 131 (0.1) |
| Asian | 3198 (0.5) | 2760 (0.5) | 438 (0.5) | ||
| Black | 131,170 (21.8) | 106,233 (21.0) | 24,937 (26.1) | ||
| Multiracial | 9176 (1.5) | 7501 (1.5) | 1675 (1.8) | ||
| Native Hawaii/Pacific Islander | 288 (0.0) | 246 (0.0) | 42 (0.0) | ||
| Other | 2 (0.0) | 2 (0.0) | |||
| Unknown | 74,622 (12.4) | 65,479 (13.0) | 9143 (9.6) | ||
| White | 381,716 (63.5) | 322,585 (63.8) | 59,131 (61.9) | ||
| Ethnicity, N (%) | Hispanic | 0 | 27,039 (4.5) | 23,025 (4.6) | 4014 (4.2) |
| Not Hispanic | 562,732 (93.7) | 472,697 (93.5) | 90,035 (94.3) | ||
| Unknown | 11,101 (1.8) | 9653 (1.9) | 1448 (1.5) | ||
| CCS_Level_1, N (%) | COVID-19 | 0 | 2123 (0.4) | 1817 (0.4) | 306 (0.3) |
| Congenital anomalies | 2873 (0.5) | 2585 (0.5) | 288 (0.3) | ||
| Diseases of the blood and blood-forming organs | 8139 (1.4) | 6111 (1.2) | 2028 (2.1) | ||
| Diseases of the circulatory system | 151,803 (25.3) | 128,389 (25.4) | 23,414 (24.5) | ||
| Diseases of the digestive system | 77,425 (12.9) | 64,120 (12.7) | 13,305 (13.9) | ||
| Diseases of the genitourinary system | 36,238 (6.0) | 29,852 (5.9) | 6386 (6.7) | ||
| Diseases of the musculoskeletal system and connective tissue | 41,196 (6.9) | 37,794 (7.5) | 3402 (3.6) | ||
| Diseases of the nervous system and sense organs | 31,115 (5.2) | 27,548 (5.5) | 3567 (3.7) | ||
| Diseases of the respiratory system | 58,196 (9.7) | 47,706 (9.4) | 10,490 (11.0) | ||
| Diseases of the skin and subcutaneous tissue | 12,415 (2.1) | 10,917 (2.2) | 1498 (1.6) | ||
| Endocrine, nutritional, and metabolic diseases and immunity disorders | 32,133 (5.3) | 25,845 (5.1) | 6288 (6.6) | ||
| Infectious and parasitic diseases | 29,738 (4.9) | 23,783 (4.7) | 5955 (6.2) | ||
| Injury and poisoning | 66,628 (11.1) | 55,102 (10.9) | 11,526 (12.1) | ||
| Mental illness | 20,423 (3.4) | 17,498 (3.5) | 2925 (3.1) | ||
| Neoplasms | 3973 (0.7) | 3516 (0.7) | 457 (0.5) | ||
| Residual codes; unclassified; all E codes [259. and 260.] | 3782 (0.6) | 3266 (0.6) | 516 (0.5) | ||
| Symptoms, signs, and ill-defined conditions and factors influencing health status | 22,672 (3.8) | 19,526 (3.9) | 3146 (3.3) | ||
| Medical/surgical, N (%) | Medical | 0 | 361,660 (60.2) | 303,643 (60.1) | 58,017 (60.8) |
| Surgical | 239,212 (39.8) | 201,732 (39.9) | 37,480 (39.2) | ||
| Hospital site, N (%) | Hospital A | 0 | 29,418 (4.9) | 25,082 (5.0) | 4336 (4.5) |
| Hospital B | 48,576 (8.1) | 41,834 (8.3) | 6742 (7.1) | ||
| Hospital C | 33,228 (5.5) | 28,293 (5.6) | 4935 (5.2) | ||
| Hospital D | 69,721 (11.6) | 59,371 (11.7) | 10,350 (10.8) | ||
| Hospital E | 23,778 (4.0) | 20,782 (4.1) | 2996 (3.1) | ||
| Hospital F | 31,159 (5.2) | 25,146 (5.0) | 6013 (6.3) | ||
| Hospital G | 153,831 (25.6) | 128,259 (25.4) | 25,572 (26.8) | ||
| Hospital H | 28,892 (4.8) | 24,263 (4.8) | 4629 (4.8) | ||
| Hospital I | 79,657 (13.3) | 65,788 (13.0) | 13,869 (14.5) | ||
| Hospital J | 72,766 (12.1) | 61,766 (12.2) | 11,000 (11.5) | ||
| Hospital K | 29,846 (5.0) | 24,791 (4.9) | 5055 (5.3) | ||
The overall readmission rate was 15.9% (N = 95,497) and readmission rates per year of our study were 16.1% in 2017 (N = 19,708) and 2018 (N = 28,494), 15.1% in 2019 (N = 27,862), and 16.5% through September 2020 (N = 19,433). There were similar readmission rates of ~ 16% for medical and surgical discharges, with variation in readmission rates by hospital (range 12.6–19.3%) and by CCS diagnosis categories from 8.3% for diseases of the musculoskeletal system and connective tissue to 24.92% for diseases of the blood and blood-forming organs. The readmission rate for the COVID-19 diagnosis group in 2020 was 14.41% (306/2123) (Appendix Table 4).
The performance of the risk score evaluated for the entire cohort from 2017 to 2020 had a c-statistic of 0.6875. Medical and surgical specialties showed similar consistent performance in discrimination and calibration, with c-statistics of 0.6923, 95% CI [0.6807, 0.7045], and 0.6802, 95% CI [0.6681, 0.6925], respectively. Both contained the overall c-statistic of 0.6875 within their 95% confidence intervals. The calibration was optimized at lower readmission risk scores (Table 2, Fig. 1) with a trend toward over-prediction at higher scores. The c-statistic for hospital sites ranged from 0.6762, 95% CI [0.6634, 0.6876], at hospital G to 0.7023, 95% CI [0.6903, 0.7132], at hospital D. Except for hospital D, the 95% confidence intervals for the hospital sites included the overall c-statistic of 0.6875 (Table 2). The pattern of over-prediction of risk for patients with higher risk scores was demonstrated across sites, but some sites, such as hospital E, showed moderate over-prediction for their entire cohort (Fig. 2). CCS level 1 categories of the primary discharge diagnosis showed more variation, with several categories having 95% confidence intervals that did not include the cohort total c-statistic. Some of the lower-performing categories with c-statistic < 0.65 include COVID-19 (c-statistic 0.6387), infectious and parasitic diseases (c-statistic 0.6398), benign neoplasms (c-statistic 0.6153), or admissions related to congenital anomalies (c-statistic 0.5403) (Table 2, Fig. 3). Subgroup analyses by race and ethnicity categories showed c-statistics of > 0.65 across race and ethnicity categories (Appendix Table 5) while calibration (Appendix Figs. 5 and 6) showed over-prediction using the readmission risk score for patients of White and unknown race compared to Black and multiracial patients while there was over-prediction overall for ethnicity for all categories (Hispanic, non-Hispanic, and unknown ethnicity). The sensitivity analysis of readmission rates and overall model performance excluding the COVID-19 pandemic period from March 1 through September 3, 2020, showed the total number of discharges was 513,862 with a c-statistic for overall performance of 0.688, 95% CI [0.6766–0.6997], and Brier score of 0.12. There were 80,987 readmissions or 15.8% compared to 15.9% including the pandemic period. Readmission by year was 16.0% for 2020 similar to the yearly rate for 2020 of 16.5% including the pandemic time period.
Table 2.
Risk Score Performance by Cleveland Clinic Health System Hospital Site, Medical/Surgical, and Diagnosis Categories
| Group | Count | c-stat | 95 min c-stat | 95max c-stat | Brier score |
|---|---|---|---|---|---|
| Hospital A | 29,418 | 0.6933 | 0.6813 | 0.7043 | 0.12 |
| Hospital B | 48,576 | 0.6801 | 0.6675 | 0.6918 | 0.11 |
| Hospital C | 33,228 | 0.6872 | 0.6750 | 0.6992 | 0.12 |
| Hospital D | 69,721 | 0.7023 | 0.6903 | 0.7132 | 0.12 |
| Hospital E | 23,778 | 0.6910 | 0.6777 | 0.7045 | 0.11 |
| Hospital F | 31,159 | 0.6951 | 0.6851 | 0.7061 | 0.15 |
| Hospital G | 153,831 | 0.6762 | 0.6634 | 0.6876 | 0.13 |
| Hospital H | 28,892 | 0.6988 | 0.6865 | 0.7101 | 0.13 |
| Hospital I | 79,657 | 0.6961 | 0.6851 | 0.7072 | 0.13 |
| Hospital J | 72,766 | 0.6802 | 0.6680 | 0.6927 | 0.12 |
| Hospital K | 29,846 | 0.6963 | 0.6849 | 0.7073 | 0.13 |
| Medical | 361,660 | 0.6923 | 0.6807 | 0.7045 | 0.13 |
| Surgical | 239,212 | 0.6802 | 0.6681 | 0.6925 | 0.13 |
| Congenital anomalies | 2873 | 0.5403 | 0.5235 | 0.5578 | 0.09 |
| COVID-19 | 2123 | 0.6387 | 0.6253 | 0.6514 | 0.12 |
| Diseases of the blood and blood-forming organs | 8139 | 0.6777 | 0.6674 | 0.6877 | 0.17 |
| Diseases of the circulatory system | 151,803 | 0.6923 | 0.6803 | 0.7044 | 0.12 |
| Diseases of the digestive system | 77,425 | 0.6798 | 0.6681 | 0.6910 | 0.13 |
| Diseases of the genitourinary system | 36,238 | 0.6646 | 0.6525 | 0.6766 | 0.14 |
| Diseases of the musculoskeletal system and connective tissue | 41,196 | 0.6900 | 0.6721 | 0.7064 | 0.07 |
| Diseases of the nervous system and sense organs | 31,115 | 0.6800 | 0.6654 | 0.6937 | 0.10 |
| Diseases of the respiratory system | 58,196 | 0.6762 | 0.6652 | 0.6884 | 0.14 |
| Diseases of the skin and subcutaneous tissue | 12,415 | 0.6942 | 0.6815 | 0.7080 | 0.10 |
| Endocrine; nutritional; and metabolic diseases and immunity disorders | 32,133 | 0.6875 | 0.6767 | 0.6989 | 0.15 |
| Infectious and parasitic diseases | 29,738 | 0.6398 | 0.6288 | 0.6508 | 0.16 |
| Injury and poisoning | 66,628 | 0.6745 | 0.6635 | 0.6851 | 0.14 |
| Mental Illness | 20,423 | 0.6754 | 0.6628 | 0.6884 | 0.12 |
| Neoplasms | 3973 | 0.6153 | 0.6001 | 0.6290 | 0.10 |
| Residual codes; unclassified; all E codes [259. and 260.] | 3782 | 0.7134 | 0.7006 | 0.7254 | 0.11 |
| Symptoms; signs; and ill-defined conditions and factors influencing health status | 22,672 | 0.6912 | 0.6782 | 0.7042 | 0.11 |
| All | 600,872 | 0.6875 | 0.6756 | 0.6998 | 0.13 |
Figure 1.
Calibration plot: observed vs. mean predicted readmission risk for Cleveland Clinic Health System medical and surgical discharges April 2017–September 2020.
Figure 2.
Calibration plot: observed vs. mean predicted readmission risk for Cleveland Clinic Health System hospital sites April 2017–September 2020.
Figure 3.
Calibration plot: observed vs. mean predicted readmission risk for Cleveland Clinic Health system discharge diagnosis categories April 2017–September 2020.
The subgroup analysis of 2019 discharges of readmission risk score variables identified as contributing to an increased score included 126,269 discharges. The following categories of variables were listed (Appendix Table 6): utilization (N = 261,204), laboratory results (N = 246,679), comorbidities and medications (N = 244,799), insurance (N = 125,142), discharge disposition to home (N = 67,093), health literacy (N = 14,025), and history of drug abuse (N = 7120).
DISCUSSION
Our work highlights the importance of continued study of risk models that are implemented in practice to determine their sustained relevance, to identify opportunities to capture additional contextual data for model improvement, and to determine the appropriate role of model-based risk assessment after initial implementation. It is critical to regularly evaluate the performance of risk models utilized in healthcare to ensure acceptable performance on potentially evolving patient populations compared to the population on which the model was initially developed as well as to assess for model performance among diverse populations to ensure equity.20
Our retrospective analysis of the implementation of the CCHS readmission risk score shows overall good performance across medical and surgical patients and diverse health system hospital sites over 3 years of implementation in our health system EMR. The model is generally well-calibrated but shows some over-prediction of risk for patients with higher readmission risk scores overall and certain diagnosis groups. COVID-19 is a new category that would not have been present in the training data for the risk score but for which the readmission risk score had acceptable performance.
Readmission risk prediction had generally consistent performance by hospital site but with clear outliers (hospitals B and E) with under- and over-prediction. These are sites that require evaluation of implementation context to further understand the variable performance including possible differences in additional patient characteristics not captured by the prediction model or other site-specific issues. Regarding the primary diagnosis categories, there was also variation in performance which is consistent with previous work at our health system suggesting the impact of the primary diagnosis on outcomes related to hospitalization21 and may impact post-discharge planning specific to diagnosis categories.
The value of readmission risk prediction is in the potential to improve care delivery.22,23 Our subgroup analysis of variables identified as contributing to a higher risk score suggests potentially modifiable variables including discharge disposition to home and health literacy. Future transitional care efforts could focus on these variables in addition to assessing for variation in hospital sites and primary diagnoses. For example, we previously reported the association with subsequent healthcare utilization of a CCHS transitional care intervention program of post-discharge home visits that had been successfully implemented utilizing a high-risk CCHS readmission risk score as one criterion for program offering.24 Further efforts to provide support at home post discharge may thus be promising.
Regarding transitional care program eligibility, an interesting finding of our current study is that while model performance is overall similar across race categories, there is over-prediction of readmission risk for White patients compared to Black patients at higher readmission risk scores. Since the readmission risk score is utilized for risk stratification in the offering of transitional care programs, this could potentially lead to offering of these programs to patients with actually lower risk for readmission. This would also be the case for the over-prediction at higher risk scores overall and for certain hospital sites or diagnosis categories. It is important to note that clinician assessment of readmission risk may be a valuable addition to readmission risk prediction25,26 and the readmission risk score is currently only one criterion used for program offering, as our health system programs are also open to referral by our clinicians and nurse care coordinators regardless of risk score, mitigating to some extent the concern of potentially exacerbating disparities in program eligibility or offering based upon race, site of care, or diagnosis category using the risk score alone.
Even with appropriate program offering to patients and available reimbursement with recent increased national trends of utilization,27 the acceptance of transitional care programs may vary24,28,29 and addressing both medical and social needs30 after discharge will be required to have impact on longer-term outcomes. Future work will require focus not only on utilizing the risk score to inform resource allocation for post-discharge initiatives and subsequent assessment of clinical outcomes but on implementation strategies to improve uptake of the programs offered. For example, recent experience suggests telehealth may be promising to deliver transitional care interventions.31
Limitations of our analysis include that the readmission risk score is calculated based upon available CCHS EMR data and the outcome of 30-day hospital readmission is captured only if the readmission was in the CCHS. Changes to the patient’s medical record, such as the discharge disposition, one of the variables in the risk score, could be made after the last readmission risk score is filed. While we were able to capture in a subgroup analysis the frequency of the predictive variables contributing to an elevated risk score displayed to the provider, we could not assess their effect on provider actions at the time of discharge.
Our results suggest that our readmission risk score continues to perform well overall and calls for further study of variation in performance by hospital site and primary discharge diagnosis, and continued assessment of performance by race and ethnicity categories in order to optimally inform our ongoing transitional care efforts.
Supplementary Information
(DOCX 949 kb)
Acknowledgements
We thank James B. Young, MD, for his critical review of the manuscript.
Funding
This project was funded internally by the Healthcare Delivery and Implementation Science Center at Cleveland Clinic.
Declarations
Conflict of Interest
The authors report no conflict of interest. Dr. Misra-Hebert receives funding from the Agency for Healthcare Research and Quality grant # K08HS024128 and reports grants from the National Heart Lung and Blood Institute, National Human Genome Research Institute, Novo Nordisk, Inc., Merck Inc., and Boehringer Ingelheim Pharmaceuticals, Inc., outside the submitted work. Mr. Milinovich receives grants from Novo Nordisk Inc., Merck Inc., Boehringer Ingelheim Pharmaceuticals Inc., National Institute on Aging, and NFL Players Association outside the submitted work. Dr. Kattan receives grant funding from Novo Nordisk Inc., Merck Inc., and Boehringer Ingelheim Pharmaceuticals, Inc., and is a consultant for Glaxo Smith Kline and RenalytixAI, all outside of the submitted work. Dr. Alberts has received personal compensation related to consulting for Qr8 Health and received research funding from NIH, Department of Defense, and the Michael J. Fox Foundation.
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Kansagara D, Englander H, Salanitro A, et al. Risk Prediction Models for Hospital Readmission: A Systematic Review. JAMA J Am Med Assoc. 2011;306(15):1688–1698. doi: 10.1001/jama.2011.1515. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Donzé J, Aujesky D, Williams D, Schnipper JL. Potentially Avoidable 30-Day Hospital Readmissions in Medical Patients: Derivation and Validation of a Prediction Model. JAMA Intern Med. 2013;173(8):632–638. doi: 10.1001/jamainternmed.2013.3023. [DOI] [PubMed] [Google Scholar]
- 3.Tsui E, Au SY, Wong CP, Cheung A, Lam P. Development of an automated model to predict the risk of elderly emergency medical admissions within a month following an index hospital visit: a Hong Kong experience. Health Informatics J. 2015;21(1):46–56. doi: 10.1177/1460458213501095. [DOI] [PubMed] [Google Scholar]
- 4.Gildersleeve R, Cooper P. Development of an automated, real time surveillance tool for predicting readmissions at a community hospital. Appl Clin Inform. 2013;4(2):153–169. doi: 10.4338/ACI-2012-12-RA-0058. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Uhlmann M, Lécureux E, Griesser A-C, Duong HD, Lamy O. Prediction of potentially avoidable readmission risk in a division of general internal medicine. Swiss Med Wkly. 2017;147:w14470. doi: 10.4414/smw.2017.14470. [DOI] [PubMed] [Google Scholar]
- 6.Zapatero A, Barba R, Marco J, et al. Predictive model of readmission to internal medicine wards. Eur J Intern Med. 2012;23(5):451–456. doi: 10.1016/j.ejim.2012.01.005. [DOI] [PubMed] [Google Scholar]
- 7.Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community | CMAJ. Accessed April 14, 2021. https://www.cmaj.ca/content/182/6/551 [DOI] [PMC free article] [PubMed]
- 8.Hospital Readmissions Reduction Program (HRRP) | CMS. Accessed April 14, 2021. https://www.cms.gov/medicare/medicare-fee-for-service-payment/acuteinpatientpps/readmissions-reduction-program
- 9.Tsai TC, Joynt KE, Orav EJ, Gawande AA, Jha AK. Variation in Surgical-Readmission Rates and Quality of Hospital Care. N Engl J Med. 2013;369(12):1134–1142. doi: 10.1056/NEJMsa1303118. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Haneuse S, Dominici F, Normand S-L, Schrag D. Assessment of Between-Hospital Variation in Readmission and Mortality After Cancer Surgical Procedures. JAMA Netw Open. 2018;1(6):e183038–e183038. doi: 10.1001/jamanetworkopen.2018.3038. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Singh S, Lin Y-L, Kuo Y-F, Nattinger AB, Goodwin JS. Variation in the risk of readmission among hospitals: the relative contribution of patient, hospital and inpatient provider characteristics. J Gen Intern Med. 2014;29(4):572–578. doi: 10.1007/s11606-013-2723-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Yu S, Farooq F, van Esbroeck A, Fung G, Anand V, Krishnapuram B. Predicting readmission risk with institution-specific prediction models. Artif Intell Med. 2015;65(2):89–96. doi: 10.1016/j.artmed.2015.08.005. [DOI] [PubMed] [Google Scholar]
- 13.Yeh RW, Rosenfield K, Zelevinsky K, et al. Sources of hospital variation in short-term readmission rates after percutaneous coronary intervention. Circ Cardiovasc Interv. 2012;5(2):227–236. doi: 10.1161/CIRCINTERVENTIONS.111.967638. [DOI] [PubMed] [Google Scholar]
- 14.Hekkert K, Kool RB, Rake E, et al. To what degree can variations in readmission rates be explained on the level of the hospital? a multilevel study using a large Dutch database. BMC Health Serv Res. 2018;18(1):999. doi: 10.1186/s12913-018-3761-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Gallagher D, Zhao C, Brucker A, et al. Implementation and Continuous Monitoring of an Electronic Health Record Embedded Readmissions Clinical Decision Support Tool. J Pers Med. 2020;10(3):E103. doi: 10.3390/jpm10030103. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Epic Systems, Inc., Verona, WI. Accessed May 26, 2021. https://www.epic.com/
- 17.Clinical Classifications Software (CCS) for ICD-10-PCS (beta version). Accessed April 14, 2021. https://www.hcup-us.ahrq.gov/toolssoftware/ccs10/ccs10.jsp
- 18.Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019;366(6464):447–453. doi: 10.1126/science.aax2342. [DOI] [PubMed] [Google Scholar]
- 19.Coley RY, Johnson E, Simon GE, Cruz M, Shortreed SM. Racial/Ethnic Disparities in the Performance of Prediction Models for Death by Suicide After Mental Health Visits. JAMA Psychiatry. 2021;78(7):726–734. doi: 10.1001/jamapsychiatry.2021.0493. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Matheny ME, Whicher D, Thadaney IS. Artificial Intelligence in Health Care: A Report From the National Academy of Medicine. JAMA. 2020;323(6):509–510. doi: 10.1001/jama.2019.21579. [DOI] [PubMed] [Google Scholar]
- 21.Hilton CB, Milinovich A, Felix C, et al. Personalized predictions of patient outcomes during and after hospitalization using artificial intelligence. Npj Digit Med. 2020;3(1):1–8. doi: 10.1038/s41746-020-0249-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Struja T, Baechli C, Koch D, et al. What Are They Worth? Six 30-Day Readmission Risk Scores for Medical Inpatients Externally Validated in a Swiss Cohort. J Gen Intern Med. 2020;35(7):2017–2024. doi: 10.1007/s11606-020-05638-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Leppin AL, Gionfriddo MR, Kessler M, et al. Preventing 30-Day Hospital Readmissions: A Systematic Review and Meta-analysis of Randomized Trials. JAMA Intern Med. 2014;174(7):1095. doi: 10.1001/jamainternmed.2014.1608. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Misra-Hebert AD, Rothberg MB, Fox J, et al. Healthcare utilization and patient and provider experience with a home visit program for patients discharged from the hospital at high risk for readmission. Healthc Amst Neth. 2021;9(1):100518. doi: 10.1016/j.hjdsi.2020.100518. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Nguyen OK, Washington C, Clark CR, et al. Man vs. Machine: Comparing Physician vs. Electronic Health Record–Based Model Predictions for 30-Day Hospital Readmissions. J Gen Intern Med. 2021;36(9):2555–2562. doi: 10.1007/s11606-020-06355-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Flaks-Manov N, Srulovici E, Yahalom R, Perry-Mezre H, Balicer R, Shadmi E. Preventing Hospital Readmissions: Healthcare Providers’ Perspectives on “Impactibility” Beyond EHR 30-Day Readmission Risk Prediction. J Gen Intern Med. 2020;35(5):1484–1489. doi: 10.1007/s11606-020-05739-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Marcotte LM, Reddy A, Zhou L, Miller SC, Hudelson C, Liao JM. Trends in Utilization of Transitional Care Management in the United States. JAMA Netw Open. 2020;3(1):e1919571–e1919571. doi: 10.1001/jamanetworkopen.2019.19571. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Hoyer EH, Golden B, Dougherty G, et al. The Paradox of Readmission Prevention Interventions: Missing Those Most in Need. Am J Med. Published online May 7, 2021. 10.1016/j.amjmed.2021.04.006 [DOI] [PubMed]
- 29.Fakha A, Groenvynck L, de Boer B, van Achterberg T, Hamers J, Verbeek H. A myriad of factors influencing the implementation of transitional care innovations: a scoping review. Implement Sci. 2021;16(1):21. doi: 10.1186/s13012-021-01087-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Bensken WP, Alberti PM, Koroukian SM. Health-Related Social Needs and Increased Readmission Rates: Findings from the Nationwide Readmissions Database. J Gen Intern Med. 2021;36(5):1173–1180. doi: 10.1007/s11606-021-06646-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Anderson TS, O’Donoghue AL, Dechen T, Herzig SJ, Stevens JP. Trends in telehealth and in-person transitional care management visits during the COVID-19 pandemic. J Am Geriatr Soc. n/a(n/a). 10.1111/jgs.17329 [DOI] [PMC free article] [PubMed]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
(DOCX 949 kb)