Abstract
Objective
The primary objective is to develop a prediction model of 30-day hospital readmission among adults with diabetes mellitus (DM) whose index admission was DM-related. The secondary aims are to internally and externally validate the prediction model and compare its performance with 2 existing models.
Research Design and Setting
Data of inpatients aged ≥ 18 years from 2008 to 2015 were extracted from the electronic medical record system of the National University Hospital, Singapore. Unplanned readmission within 30 days was calculated from the discharge date of the index hospitalization. Multivariable logistic regression and 10-fold cross-validation were performed. For external validation, simulations based on prevalence of 30-day readmission, and the regression coefficients provided by referenced papers were conducted.
Results
Eleven percent of 2355 patients reported 30-day readmission. The prediction model included 4 predictors: length of stay, ischemic heart disease, peripheral vascular disease, and number of drugs. C-statistics for the prediction model and 10-fold cross-validation were 0.68 (95% CI 0.66, 0.70) and 0.67 (95% CI 0.63 to 0.70), respectively. Those for the 3 simulated external validation data sets ranged from 0.64 to 0.68.
Conclusion
The prediction model performs well with good internal and external validity for identifying patients with DM at risk of unplanned 30-day readmission.
Keywords: 30-day readmission, diabetes, statistical model, validation study
Readmission is defined as unplanned hospital admission within a prespecified period (eg, 30 days) (1, 2). Thirty-day hospital readmission rates are increasingly used for both quality improvement and cost control (3). Adult patients with diabetes mellitus (DM) represent 10% to 25% of all 30-day unplanned hospital readmissions (4). DM-related readmission is common in patients with DM (5, 6); in addition, patients with DM-related index admission diagnosis also have a higher readmission rate compared to those with other diagnoses (7). However, predictors of readmission among patients with DM whose index admission is DM-related are not well-studied; 30-day readmission prediction models (5, 8, 9) often included other types of index admission diagnoses (4, 5). Demographic, socioeconomic, inpatient factors, and comorbidities may be different in patients whose index admission diagnosis is DM-related (4, 5). Poorly controlled DM is commonly associated with ischemic heart disease (IHD) (10), peripheral vascular disease (PVD) (11), and renal failure (12). It may be appropriate to determine the individual effect of each comorbidity on 30-day readmission among DM patients with index admission diagnosis related to DM; a previous study has attempted to predict readmission based on consolidated Charlson Comorbidity Index (13) (CCI).
Routine inpatient medical records can be used to identify patients at risk of hospital readmission and develop prediction models (14, 15). Although prediction models can objectively support healthcare professionals to make clinical decisions and interventions (16), their performance should be evaluated using independent data (17). When independent data are not available, simulated data may be generated to compare the results with the prediction model (18). LACE (13) and PCi (19) models included 4 [length of stay (LOS), acuity of the hospital admission (emergency vs nonemergency), CCI score, and number of visits to emergency department in the past 6 months] and 2 (polypharmacy and CCI score) variables, respectively: neither model has been externally validated in DM patients whose index admission is DM-related.
The primary objective of this study is to develop a prediction model of 30-day unplanned hospital readmission among adult patients with DM with a DM-related index admission. Secondary aims are to (1) internally validate the prediction model using 10-fold cross-validation; (2) externally validate the prediction model by simulating 30-day readmission rates and patient characteristics based on results of published studies (8, 13, 20); and (3) compare the performance of the prediction model with LACE (13) and PCi (19).
Method
Administrative data of the National University Hospital, Singapore, from January 2008 to December 2015 were retrospectively extracted from the registration system, electronic health records, and pharmacy database. Index admission was defined as the first admission during the study period. The sample consisted of inpatients aged ≥ 18 years at the time of index admission with hospital stay more than 24 hours and survived to hospital discharge. Patients were included if the primary or secondary diagnosis of index admission was DM-related based on the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM, from 2007 to 2011) and International Classification of Diseases, Tenth Revision (ICD-10AM, 2012 onward). These included but are not limited to diabetic ketoacidosis, hypoglycemia, impaired glucose regulation, abnormal glucose tolerance with previous history of DM, elevated blood glucose level with previous history of DM, and preexisting DM in pregnancy. Exclusion criteria included DM arising during pregnancy, diabetes insipidus, death during index admission, and missing data on comorbidities and number of drugs administered.
Dependent Variable
Being discharged and readmitted within 30 days without a planned second admission was the outcome of interest. Readmission within 30 days was calculated from the discharge date of index hospitalization.
Independent Variables
Demographic characteristics included sex, age, race, and residential status. Medical history included number of surgical operations, emergency department visits in the past 12 months prior to the index admission, and number of drugs on discharge. Admission and discharge information included LOS (1-3 days vs ≥4 days) in the hospital, LOS in the intensive care unit, type of hospital admission (emergency or elective), hospital ward subsidy (as a marker of socioeconomic status), discharge to home or stepdown care facilities, and comorbidities listed in the CCI score (21).
Predictors in LACE and PCi Models
In the LACE (13) model, logarithmic and square root transformations were performed for LOS and number of visits to emergency department during the previous 6 months, respectively. In the PCi model (19), polypharmacy was defined as having ≥6 medications vs 1 to 5 medications, and the CCI score was coded as a binary variable (≥5 vs 0-4).
Statistical Analysis
Chi-square tests were used to compare 30-day readmission status for categorical variables. Mann-Whitney U tests were used to compare 30-day readmission status for CCI and number of drugs, which were not normally distributed. The initial multivariable logistic model included all significant variables in the bivariate analysis after considerations of collinearity and linearity. Nested models were compared using the likelihood ratio test, and variable selection of the final model was based on the principle of parsimony. The Hosmer-Lemeshow test was used to determine the goodness-of-fit of the prediction model. The area under the receiver operating characteristic curve (AUROC) based on 10-fold (22) cross-validation was generated to internally validate the prediction model (17).
External validation of the prediction model involved generating simulated data with 20 000 observations for 3 different scenarios mimicking real-life data. The first external validation data were simulated from patient characteristics and regression coefficients from the prediction model. LOS was simulated using a zero-truncated negative binomial distribution with parameter 4.0 and probability 0.5 to give a median LOS of 4. Number of drugs was simulated assuming a Poisson distribution with mean 8.7 and dispersion parameter 2.0. Binary IHD and PVD were generated assuming binomial distributions, both with probability of 0.05. The second external validation data were simulated assuming the prevalence of readmission, patient characteristics, and regression coefficients reported by Enomoto et al (20). As information on the number of drugs was not available from the published study, its distribution was simulated as in the prediction model. The third external validation data were simulated using a combination results from 2 studies—the distribution for 3 of the predictors were simulated based on LACE (13) while that for PVD was simulated from the information provided by Eby et al (8). Readmission rates for the 3 external validation data sets 1, 2, and 3 (Table 1) were 11.8%, 20.4%, and 10.3%, respectively, corresponding to those of the prediction data by Enomoto et al (20) and Eby et al (8).
Table 1.
Characteristics of predictors from source and simulated data
| Model | Median LOS | % IHD | % PVD | Mean number of drugs | 30-day readmission rate |
|---|---|---|---|---|---|
| External validation data 1 | 4 (4) | 4.1 (4.9) | 6.0 (5.0) | 8.8 (8.7) | 11.0 (11.8) |
| External validation data 2 | 4.8a (5.7) | 24.1 (24.2) | 8.7 (8.6) | 8.8 (8.7b) | 18.0 (20.4) |
| External validation data 3 | 5c (6) | 9.0c (8.9) | 2.1d (2.2) | 4c (5.5) | 10.0$ (10.3) |
Data are the summary statistics from the selected studies, while those of the simulated data are presented in parenthesis.
Abbreviations: IHD, ischemic heart disease; LOS, length of stay; PVD, peripheral vascular disease.
aEnomoto et al (20) summarized LOS in terms of mean. In the simulation, a mean LOS of 5.64 was assumed.
bThis is simulated based on number of drugs from the prediction model, whereas the other variables are simulated based on results of Enomoto et al (20).
cSimulated based on the results of LACE (13).
dSimulated based on results of Eby et al (8).
Calibration plots of observed vs predicted probability were generated with points shown at the deciles of predicted probability for each plot; the corresponding observed probability was calculated as the proportion readmitted at each decile cutoff (23).
The calibration-in-the-large statistic was presented to determine whether the average predicted probability overestimated or underestimated the average observed probability (17). The calibration slope was estimated to determine whether the predicted probabilities showed the same variation as the observed probabilities (17). C-statistics of the simulated data were estimated to determine whether observations who were readmitted had higher predicted probabilities than those who were not readmitted (17).
The distribution of the predicted risks of the prediction model was also compared with those of the 3 simulated validation data by generating membership regression models with dependent variable coded 1 to denote individual participants from the prediction model, and 0 to denote individual participants from the simulated validation data (17). The ratio of SD of the linear predictor (LP) as a measure of discriminative ability and difference in mean LP as a measure of the difference in predicted outcome frequency derived from the membership models were compared. In addition, the C-statistics of the membership models were presented to distinguish the prediction model from the individual simulated data.
The performance of the prediction model was compared with LACE (13) and PCi (19) using AUROC, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio (LR+), negative likelihood ratio (LR−), and F-score.
All statistical analyses were performed using Stata version 16 with the exception of the zero-truncated negative binomial distribution which was simulated using R. The level of significance was set at 0.05 assuming a 2-sided test. The study followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) statement, a 22-item checklist for reporting (24).
Results
Descriptive Statistics of Patient Characteristics
Demographic characteristics of the study cohort involving 2355 patients (Fig. 1) were summarized by 30-day readmission status (Table 2). Overall, 11.0% of the patients had unplanned readmission within 30 days. Malay (25.5%) and Indian (15.4%) patients were overrepresented in the study cohort compared to the local adult population census, where the percentages were 13.5% and 9.0%, respectively, in 2020 (25). Ethnic distributions were similar in not readmitted vs readmitted patients. Most patients were Singapore residents, with at least 1 emergency department visit in the past 12 months and had stayed in government-subsidized wards. Their most common comorbidities were diabetes-related chronic complications (67.2%), followed by renal disease (23.3%) (Table 3).
Figure 1.
Selection of study cohort.
Table 2.
Characteristics of 2355 participants by 30-day readmission status
| Demographic characteristics | Total (n = 2355, 100%) |
Not readmitted (n = 2095, 89.0%) |
Readmitted (n = 260, 11.0%) |
P-value |
|---|---|---|---|---|
| Sex | 0.623 | |||
| Female | 1116 (47.4) | 989 (47.2) | 127 (48.9) | |
| Male | 1239 (52.6) | 1106 (52.8) | 133 (51.1) | |
| Age, years | <0.007 | |||
| ≤64 | 1529 (64.9) | 1380 (65.8) | 149 (57.3) | |
| ≥65 | 826 (35.1) | 715 (34.2) | 111 (42.7) | |
| Ethnicity | 0.484 | |||
| Chinese | 1171 (49.7) | 1034 (49.3) | 137 (52.7) | |
| Malay | 601 (25.5) | 545 (26.00) | 56 (21.5) | |
| Indian | 362 (15.4) | 320 (15.3) | 42 (16.2) | |
| Others | 221 (9.4) | 196 (9.4) | 25 (9.6) | |
| Residential status | 0.712 | |||
| Nonresident | 214 (9.1) | 192 (9.2) | 22 (8.5) | |
| Resident | 2141 (90.9) | 1903 (90.8) | 238 (91.5) | |
| Medical history | ||||
| Surgical operation, n | <0.001 | |||
| 0 | 1755 (74.5) | 1593 (76.1) | 162 (62.3) | |
| ≥1 | 600 (25.5) | 502 (23.9) | 98 (37.7) | |
| Emergency department visits in the past 12 months, n | 0.248 | |||
| 0 | 255 (10.9) | 223 (10.7) | 32 (12.3) | |
| 1 | 1808 (76.7) | 1619 (77.2) | 189 (72.7) | |
| ≥2 | 292 (12.4) | 253 (12.1) | 39 (15.0) | |
| Number of drugs, median (IQR) | 8 (5, 11) | 8 (5, 11) | 11 (8, 14) | <0.001 |
| Admission and discharge information | ||||
| Length of stay, days | <0.001 | |||
| 1-3 | 1141 (48.4) | 1059 (50.5) | 82 (31.5) | |
| ≥4 | 1214 (51.6) | 1036 (49.5) | 178 (68.5) | |
| ICU length of stay, days | 0.003 | |||
| 0 | 2255 (95.8) | 2015 (96.2) | 240 (92.3) | |
| ≥1 | 100 (4.2) | 80 (3.8) | 20 (7.7) | |
| Type of hospital admission | 0.880 | |||
| Nonemergency | 149 (6.3) | 132 (6.3) | 17 (6.5) | |
| Emergency | 2206 (93.7) | 1963 (93.7) | 243 (93.5) | |
| Type of wards accommodation | 0.322 | |||
| Private | 357 (15.2) | 323 (15.4) | 34 (13.1) | |
| Subsidized | 1998 (84.8) | 1772 (84.6) | 226 (86.9) | |
| Discharge type | 0.281 | |||
| Discharged home/discharged to home with day rehab or medical appointment | 2,25 (94.4) | 1984 (94.6) | 241 (92.7) | |
| Discharged to other hospitals or nursing homes/discharged against medical advice/absconded | 130 (5.6) | 111 (5.4) | 19 (7.3) |
Unless otherwise indicated, data are given as n (%).
Abbreviations: ICU, intensive care unit; IQR, interquartile range.
Table 3.
Comorbidities of participants by 30-day readmission status
| Comorbidity | Total n (%) | Not readmitted (n = 2095, 89.0%) |
Readmitted (n = 260, 11.0%) |
P-value |
|---|---|---|---|---|
| CCI, median (IQR) | 3 (1, 4 ) | 3 (1, 4) | 3 (2, 4) | <0.001 |
| Diabetes chronic complication | 1582 (67.2) | 1379 (65.8) | 203 (78.1) | <0.001 |
| Renal disease | 550 (23.3) | 464 (22.1) | 86 (33.1) | <0.001 |
| Heart failure | 162 (6.9) | 140 (6.7) | 22 (8.5) | 0.299 |
| Peripheral vascular disease | 142 (6.0) | 112 (5.3) | 30 (11.5) | <0.001 |
| Ischemic heart disease | 97 (4.1) | 67 (3.2) | 30 (11.5) | <0.001 |
| Liver disease | 58 (2.5) | 50 (2.4) | 8 (3.1) | 0.497 |
| Dementia | 34 (1.4) | 28 (1.3) | 6 (2.3) | 0.215 |
| Chronic obstructive pulmonary disease | 21 (0.9) | 18 (0.9) | 3 (1.1) | 0.633 |
| Peptic ulcer disease | 21 (0.9) | 15 (0.7) | 6 (2.3) | 0.010 |
| Any tumor | 19 (0.8) | 15 (0.7) | 4 (1.50) | 0.149 |
| Cerebrovascular disease hemiplegia | 12 (0.5) | 11 (0.5) | 1 (0.4) | 1.000 |
| Metastatic tumour | 6 (0.30) | 4 (0.20) | 2 (0.8) | 0.135 |
| Connective tissue disease | 5 (0.2) | 5 (0.2) | 0 (0.0) | 1.000 |
| Acquired immune deficiency syndrome | 2 (0.1) | 2 (0.1) | 0 (0.0) | 1.000 |
Unless otherwise indicated, data are given as n (%).
Abbreviations: CCI, Charlson comorbidity index; IQR, interquartile range.
Bivariate analysis identified nine significant risk factors (age, LOS, number of surgical operations, number of drugs, diabetes chronic complications, IHD, PVD, peptic ulcer disease, and renal disease) of 30-day unplanned hospital readmission. These were included in the initial logistic regression model.
Prediction Model
The final prediction model (LIPiD) included 4 significant predictors: LOS, IHD, PVD, and number of drugs (Table 4) after removing the insignificant predictors in multivariable regression. Those with LOS ≥ 4 days had a 45% increase in odds of 30-day readmission [odds ratio (OR) 1.45 (95% CI 1.07, 1.95)] as compared to patients with LOS 1 to 3 days. Patients who had IHD had more than twice the odds [OR 2.31 (95% CI 1.43, 3.72)] of being readmitted than those without IHD. Patients with PVD had 58% (OR 1.58 (95% CI 1.01, 2.48)] higher odds of being readmitted as compared to those without PVD. Number of drugs was also positively associated with readmission [OR 1.09 (95% CI 1.06, 1.12)]. The Hosmer-Lemeshow test for LIPiD suggested a good fit (P = 0.37). The 10-fold cross-validation (mean AUROC: 0.67, bootstrap bias corrected 95% CI 0.63, 0.70) demonstrated a relatively reasonable performance (26) of the prediction model (Fig. 2).
Table 4.
Significant risk factors of 30-day unplanned hospital readmission in the bivariate and multivariable analyses
| Predictors | Bivariate analysis | Multivariable LIPiD model (n = 2355) | ||||
|---|---|---|---|---|---|---|
| OR | 95% CI | P-value | OR | 95% CI | P-value | |
| Age (years) | ||||||
| ≤64 | Reference | |||||
| ≥65 | 1.44 | 1.11, 1.87 | 0.007 | |||
| Surgical operations, n | ||||||
| 0 | Reference | |||||
| ≥1 | 1.92 | 1.47, 2.52 | <0.001 | |||
| Number of drugs, median (IQR) | 1.12 | 1.09, 1.15 | <0.001 | 1.09 | 1.06, 1.12 | 0.001 |
| Length of stay, days | ||||||
| 1-3 | Reference | Reference | ||||
| ≥4 | 2.22 | 1.69, 2.92 | <0.001 | 1.45 | 1.07, 1.96 | 0.016 |
| Diabetes chronic complication | 1.85 | 1.36, 2.51 | <0.001 | |||
| Renal disease | 1.73 | 1.32, 2.29 | <0.001 | |||
| Peripheral vascular disease | 2.31 | 1.51, 3.53 | <0.001 | 1.58 | 1.01, 2.47 | 0.042 |
| Ischemic heart disease | 3.94 | 2.51, 6.20 | 0.001 | 2.31 | 1.43, 3.72 | 0.001 |
| Peptic ulcer disease | 3.28 | 1.26, 8.52 | 0.015 |
Abbreviation: IQR, interquartile range.
Figure 2.
Area under the receiver operating characteristic curve of LIPiD model based on 10-fold cross-validation. Mean Cross-Validated Area Under the ROC Curve (cvAUC) (solid curve) and k-fold receiver operating characteristic curves (dashed curves).
Results on External Validation of LIPiD Using Simulated Data
The estimated regression coefficients from external validation data 1 were similar to those of the prediction model although the 95% CIs were narrower due to the increased simulated sample size (Table 5). For external validation data 1, LOS ≥ 4 days [log-odds (95% CI 0.37 0.28, 0.46)], IHD [log-odds 1.29 (95% CI 1.10, 1.47)], PVD [log-odds 0.37 (95% CI 0.20, 0.55)], and number of drugs [log-odds 0.17 (95% CI 0.16, 0.18)] were significantly associated with 30-day readmission. The 4 predictors of external validation data 2 were also significant. However, as expected, the estimated regression coefficients of IHD [0.10 (95% CI 0.02, 0.18)] and PVD [0.14 (95% CI 0.02, 0.26)] were weaker as compared to those in our prediction model, since external validation data 2 was simulated based on the findings of Enomoto et al (20), which also reported lower ORs for both IHD and PVD but higher incidence of 30-day readmission. The regression coefficient for IHD was also lower for external validation data 3 where we simulated its prevalence based on information provided for acute coronary syndrome and atrial fibrillation in the supplementary materials (Appendix A) of the LACE (13) study. Interestingly, neither acute coronary syndrome nor atrial fibrillation were significant predictors of readmission in the LACE study (13). Thus, this suggest that the prediction model was robust in a variety of clinical settings, including instances where the prevalence rates of IHD (20) or 30-day readmission (20) were higher than the source data as well as when the PVD rate was lower (8).
Table 5.
Estimated coefficients and 95% CIs of the prediction and external validation data
| Predictors | Prediction model (95% CI) | External validation data 1 (95% CI) | External validation data 2 (95% CI) | External validation data 3 (95% CI) |
|---|---|---|---|---|
| Length of stay | 0.37 (0.06, 0.67) |
0.37 (0.28, 0.46) |
0.40 (0.33, 0.47) |
0.37 (0.28, 0.47) |
| Ischemic heart disease | 0.83 (0.35, 1.31) |
1.29 (1.10, 1.47) |
0.10 (0.02, 0.18) |
0.19 (0.04, 0.35) |
| Peripheral vascular disease | 0.46 (0.15, 0.90) |
0.37 (0.20, 0.55) |
0.14 (0.02, 0.26) |
0.40 (0.12, 0.68) |
| Number of drugs | 0.08 (0.05, 0.11) |
0.17 (0.16, 0.18) |
0.09 (0.08, 0.10) |
0.15 (0.14, 0.16) |
| Constant (model intercept) | −3.25 (−3.58, −2.92) |
−3.96 (−4.11, −3.81) |
−2.51 (−2.61, −2.41) |
−3.40 (−3.51, −3.28) |
Points on the validation plots were relatively close to the origin, indicating low observed and predicted probabilities of readmission in most instances (Fig. 3). When the LIPiD model was applied to the prediction sample, several observed frequencies per decile of predicted probabilities (as indicated by circles) were on the line of equality indicating good agreement (23). However, there were a few points below the line of equality suggesting the predictions of 30-day readmission were slightly higher at the extreme (23). The observed and predicted probabilities for external validation data model 1, external validation data model 2 [data simulated from Enomoto et al (20)], and external validation data model 3 [data simulated from published results of LACE (13) and Eby et al (8)] were also in agreement, and hence close to line of equality especially for earlier deciles, while those for the later deciles tended to have higher predictive probabilities. The 3 plots suggest a relatively good performance of the LIPiD model when validated against a variety of real-life clinical settings.
Figure 3.
Observed vs predicted probabilities of 30-day readmission of the prediction model and 3 external validation data models. a = calibration-in-the-large, b = calibration slope, and c = c-statistic. Prediction model (LIPiD), a = −0.21, b = 0.98, and c = 0.68. External validation 1: a = −0.24, b = 0.97, and c = 0.68. External validation 2: a = −0.14, b = 0.97, and c = 0.64. External validation 3: a = −0.31, b = 0.93, and c = 0.68.
The calibration-in-the-large statistic (Figure 3) was −0.21 for the prediction model, indicating that LIPiD slightly overestimated 30-day readmission (17). This was also true for the 3 external validation data models. The calibration slope for the prediction model and all the external validation data models were close to 1, suggesting that the predicted risks were proportionally accurate (17). The C-statistics of the prediction model and external validation data 1 and 3 were 0.68 and slightly higher than those for external validation data 2 (Fig. 3).
The SD of the LP for external validation data model 2 showed slight variations from that of the prediction model while those for external validation data models 1 and 3 were more similar (Fig. 4A). This suggested external validation models 1 and 3 had the same discriminating ability as the prediction model (17). The mean LP for external validation models 1 and 3 were also similar to that of the prediction model (Fig. 4B), suggesting similar model performance in terms of predicted outcome frequency (17). The C-statistic of the membership model implied that the samples for prediction and external validation data models 1 were similar (17).
Figure 4.
(A) Relative difference in standard deviation of the linear predictor comparing validation vs prediction sample. (B) Difference in mean of the linear predictor comparing validation vs prediction sample.
Model Comparison: LIPiD, LACE, and PCi Using Prediction Data
The discrimination performance of LIPiD [AUROC 0.68 (95% CI 0.66, 0.70)] was significantly better than LACE [AUROC 0.65 (95% CI 0.63, 0.67), P = 0.041] and PCi [AUROC 0.60 (95% CI 0.58, 0.62), P < 0.001], respectively (Table 6). At a readmission rate of 11%, the sensitivity and specificity of LIPiD were 62% (95% CI 55%, 67%) and 65% (95% CI 63%, 67%), respectively. LIPiD yielded higher values than PCi in terms of specificity, PPV, LR+, LR−, and F-score. Although the sensitivity of LIPiD was lower than PCi, its specificity was more than twice that of PCi. The 3 models did not yield a high PPV owing to the low prevalence of readmission.
Table 6.
Performance indicators of prediction models at 30-day readmission rate of 11%
| Model | AUROC (95%CI) |
Sensitivity, % (95%CI) | Specificity, % (95%CI) |
PPV, % (95%CI) |
NPV, % (95%CI) |
LR+ | LR− | F-score |
|---|---|---|---|---|---|---|---|---|
| LIPiD | 0.68 (0.66, 0.70) |
62 (55, 67) |
65 (63, 67) |
18 (16, 19) |
93 (92, 94) |
1.81 | 0.58 | 0.28 |
| LACE | 0.65 (0.63, 0.67) |
57 (51to 63) |
64 (61, 65) |
16 (14, 18) |
92 (91, 93) |
1.59 | 0.66 | 0.25 |
| PCi | 0.60 (0.58, 0.62) |
89 (84, 92) |
28 (26, 30) |
13 (12, 14) |
95 (93, 96) |
1.24 | 0.38 | 0.23 |
Abbreviations: AUROC, area under the receiver operating characteristic curve; LR+, positive predictive value; LR−, negative predictive value; NPV, negative predictive value; PPV, positive predictive value
Discussion
The LIPiD prediction model identified 4 predictors of unplanned 30-day readmission among DM patients with DM-related index admission: LOS, IHD, PVD, and number of drugs. The finding that LOS was a significant predictor corroborates with that observed from 2 prediction models for privately insured (8) and Medicare patients with DM (9). The 2 publications included all patients with type 2 DM regardless of diagnoses at index admission. Thus, LOS appeared to be an important risk factor for 30-day readmission in DM patients irrespective of the actual diagnosis at initial admission, case-mix, or type of DM. It is also 2 of the 4 predictors of the LACE model (13).
Consistent with the reviews by Robbins et al (27) and Soh et al (4), who reported numerous comorbidities as independent predictors of 30-day unplanned hospital readmission in patients with DM, we found IHD and PVD to be important predictors. The combination of hyperglycemia, insulin resistance, and free fatty acid excess can possibly lead to the development of IHD (28). Thus, it is necessary to ensure patients with DM and IHD adequately control their glycemia and lipids. The UK Prospective Diabetes Study had shown that hyperglycemia (as indicated by hemoglobin A1c) was independently associated with an increased risk for PVD (29), thus connecting overall glycemic control to the development of PVD. A further study reported that PVD increased the risk of 30-day unplanned hospital readmission among patients with DM and was associated with prolonged LOS (20). Consistent glucose monitoring was associated with improvements in glycemic status (30), which may thus prevent the development of PVD. The odds of readmission were found to increase with number of drugs in this study, which may point to the overall disease burden. A local study demonstrated the number of medications and poor compliance to be predictors of readmission (31). Patients on multiple drugs who are noncompliant may not receive the desired therapeutic benefits resulting in disease progression and readmission. Number of drugs is also a predictor in the PCi model (19), although the variable was classified differently from the LIPiD model.
The internal validity of the LIPiD model was acceptable and reproducible. Our external validation used 3 sets of simulated data reflecting a variety of real-life clinical scenarios. The 4 predictors of 30-day unplanned hospital readmission identified by LIPiD remained significant in all models with the C-statistic ranging from 0.64 to 0.68. LIPiD performs well not only for the simulated data with readmission rate of around 10% (external validation data 1 and 3) but also when the readmission rate was doubled in external validation data 2, as well as with varying ranges of IHD from 9% to 24% and PVD from 2% to 9%.
We considered individual comorbidity as possible predictor of 30-day readmission rather than CCI as in the LACE and PCi models. In the derivation of CCI, more weight is allocated to more severe comorbidities such as AIDS and metastatic tumor, which are not known predictors of 30-day readmission among patients with DM, although these have been demonstrated to be predictors of 10-year survival (21). Conversely, CCI underweights IHD and PVD, which are associated with DM. Hence, the use of CCI may be less relevant for identifying 30-day unplanned hospital readmission among DM patients whose index admission was attributed to DM. In addition, local studies by Low et al highlighted the limitation of LACE in predicting 30-day readmission (32, 33). Both studies yielded higher AUROCs than the LACE model after including blood test results (32) and markers of hospitalization severity (33).
Electronic hospital records may provide a rich source of secondary data for research but may not contain all the information needed for a particular study. We acknowledge the following limitations in our study. First, data on insulin therapy, a possible predictor of 30-day unplanned hospital readmission (4), were not available for analysis. Second, the diagnosis at admission was too diverse for more meaningful subgroup analysis. Other limitations of this study included the lack of information on medication compliance and blood test results such as hemoglobin A1c prior to index admission, which are known predictors of 30-day readmission (34, 35). Other relevant risk factors such as psychosocial characteristics (36) including history of depression or anxiety and social support were also unavailable, as were data on continuity of care after discharge (37). In addition, the prediction model did not yield a high PPV possibly due to the low prevalence of readmission (11%). Thirty-day readmission rate among patients with DM varies from country to country and usually hovers between 10% and 20% (8, 38). Thus, the model may be applicable to diverse clinical settings as demonstrated by our external validation, which assumed 30-day readmission rates of between 10% and 20% and prevalences of IHD and PVD ranging from 5% to 24% and 2% to 9%, respectively. Future studies may be conducted to explore the clinical utility of the prediction model for readmission > 20%.
Conclusion
A validated prediction model, LIPiD, demonstrated that unplanned 30-day readmission among patients with DM whose index diagnosis was DM-related was associated with LOS, IHD, PVD, and number of drugs administered. The model has good internal and reasonable external validity, and the findings may help healthcare providers identify patients at high risk of readmission.
Acknowledgment
We would like to thank Chen Zhaojin and Gu Tianyuan for their assistance in generating the 10-fold cross-validation and simulation of the length of stay based on zero-truncated negative binomial distribution, respectively. We wish to show our appreciation to Ms Winnie Chong Lin Siew for her tremendous effort in data extraction and management.
Contributor Information
Jade Gek Sang Soh, Saw Swee Hock School of Public Health, National University of Singapore, Singapore; Health and Social Sciences, Singapore Institute of Technology, Singapore.
Amartya Mukhopadhyay, Respiratory and Critical Care Medicine, National University Hospital, Singapore; Yong Loo Lin School of Medicine Singapore, National University Singapore, Singapore; Medical Affairs – Research Innovation & Enterprise, Alexandra Hospital, National University Health System, Singapore.
Bhuvaneshwari Mohankumar, Medical Affairs – Clinical Governance, National University Health System, Singapore.
Swee Chye Quek, Division of Cardiology, Department of Pediatrics, National University Hospital, Singapore.
Bee Choo Tai, Saw Swee Hock School of Public Health, National University of Singapore, Singapore.
Funding
The research is supported by the National University Health System Centre for Health Services and Policy Research Seed Grant.
Author Contributions
J.S. drafted the manuscript and performed the statistical analysis. T.B.C. conceived the research idea and advised and guided the writing of manuscript and statistical analysis. B.M. was responsible for the administrative and medical data extraction. Q.S.C., T.B.C., and A.M. provided intellectual inputs on the research and development. All authors provided inputs on the critical revision of the manuscript. Q.S.C. was the principal investigator of the grant supporting the research for which the electronic medical record data were based. Q.S.C. and T.B.C. contributed equally as co-last authors.
Disclosures
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Data Availability
Some or all data generated or analyzed during this study are included in this published article or in the data repositories listed in the references.
References
- 1. Chin DL, Bang H, Manickam RN, Romano PS. Rethinking Thirty-day hospital readmissions: shorter intervals might be better indicators of quality of care. Health affairs (Millwood, VA). 2016;35(10):1867-1875. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Gerhardt G, Yemane A, Hickman P, Oelschlaeger A, Rollins E, Brennan N. Medicare readmission rates showed meaningful decline in 2012. Medicare Medicaid Res Rev. 2013;3(2). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Fischer C, Lingsma HF, Marang-van de Mheen PJ, Kringos DS, Klazinga NS, Steyerberg EW. Is the readmission rate a valid quality indicator? A review of the evidence. PLoS One. 2014;9(11):e112282. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Soh JGS, Wong WP, Mukhopadhyay A, Quek SC, Tai BC. Predictors of 30-day unplanned hospital readmission among adult patients with diabetes mellitus: a systematic review with meta-analysis. BMJ Open Diabetes Res Care. 2020;8(1):e001227. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Rubin DJ, Handorf EA, Golden SH, Nelson DB, McDonnell ME, Zhao H. Development and validation of a novel tool to predict hospital readmission risk among patients with diabetes. Endocr Pract. 2016;22(10):1204-1215. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Rubin DJ, Recco D, Turchin A, Zhao H, Golden SH. External validation of the diabetes early re-admission risk indicator (derriTM). Endocr Pract. 2018;24(6):527-541. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Ostling S, Wyckoff J, Ciarkowski SL, et al. The relationship between diabetes mellitus and 30-day readmission rates. Clin Diabetes Endocrinol.2017;3(1):3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Eby E, Hardwick C, Yu M, et al. Predictors of 30 day hospital readmission in patients with type 2 diabetes: a retrospective, case-control, database study. Curr Med Res Opin.2015;31(1):107-114. [DOI] [PubMed] [Google Scholar]
- 9. Collins J, Abbass IM, Harvey R, et al. Predictors of all-cause 30-day readmission among Medicare patients with type 2 diabetes. Curr Med Res Opin.2017;33(8):1517-1523. [DOI] [PubMed] [Google Scholar]
- 10. Leon BM, Maddox TM. Diabetes and cardiovascular disease: epidemiology, biological mechanisms, treatment recommendations and future research. World J Diabetes. 2015;6(13):1246-1258. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Nativel M, Potier L, Alexandre L, et al. Lower extremity arterial disease in patients with diabetes: a contemporary narrative review. Cardiovasc Diabetol. 2018;17(1):138. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Nasri H, Rafieian-Kopaei M. Diabetes mellitus and renal failure: prevention and management. J Res Med Sci. 2015;20(11):1112-1120. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. van Walraven C, Dhalla IA, Bell C, et al. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ. 2010;182(6):551-557. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Png ME, Yoong J, Chen C, et al. Risk factors and direct medical cost of early versus late unplanned readmissions among diabetes patients at a tertiary hospital in Singapore. Curr Med Res Opin. 2018;34(6):1071-1080. [DOI] [PubMed] [Google Scholar]
- 15. Howell S, Coory M, Martin J, Duckett S. Using routine inpatient data to identify patients at risk of hospital readmission. BMC Health Serv Res. 2009;9(1):96. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Shipe ME, Deppen SA, Farjah F, Grogan EL. Developing prediction models for clinical use using logistic regression: an overview. J Thorac Dis. 2019;11(suppl 4):S574-S584. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Debray TP, Vergouwe Y, Koffijberg H, Nieboer D, Steyerberg EW, Moons KG. A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol. 2015;68(3):279-289. [DOI] [PubMed] [Google Scholar]
- 18. Sargent RG. Verification and validation of simulation models. J Simulation. 2013;7(1):12-24. [Google Scholar]
- 19. Logue E, Smucker W, Regan C. Admission data predict high hospital readmission risk. J Am Board Fam Med. 2016;29(1):50-59. [DOI] [PubMed] [Google Scholar]
- 20. Enomoto LM, Shrestha DP, Rosenthal MB, Hollenbeak CS, Gabbay RA. Risk factors associated with 30-day readmission and length of stay in patients with type 2 diabetes. J Diabetes Complications. 2017;31(1):122-127. [DOI] [PubMed] [Google Scholar]
- 21. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40(5):373-383. [DOI] [PubMed] [Google Scholar]
- 22. Rodríguez J, Pérez A, Lozano J. Sensitivity analysis of k-fold cross validation in prediction error estimation. IEEE Trans Pattern Anal Mach Intell. 2010;32(3):569-575. [DOI] [PubMed] [Google Scholar]
- 23. Vergouwe Y, Steyerberg EW, Eijkemans MJ, Habbema JD. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol. 2005;58(5):475-483. [DOI] [PubMed] [Google Scholar]
- 24. Moons KG, Altman DG, Reitsma JB, et al. Transparent reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162(1):W1-73. [DOI] [PubMed] [Google Scholar]
- 25.Singapore Department of Statistics. Population trends 2020. September 2020. https://www.singstat.gov.sg/-/media/files/publications/population/population2020.pdf
- 26. Yang S, Berdine G. The receiver operating characteristic (ROC) curve. Southwest Respir Crit Care Chron. 2017;5(19):34-36. [Google Scholar]
- 27. Robbins TD, Lim Choi Keung SN, Sankar S, Randeva H, Arvanitis TN. Risk factors for readmission of inpatients with diabetes: a systematic review. J Diabetes Complications. 2019;33(5):398-405. [DOI] [PubMed] [Google Scholar]
- 28. Severino P, D’Amato A, Netti L, et al. Diabetes mellitus and ischemic heart disease: the role of ion channels. Int J Mol Sci. 2018;19(3):802. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Adler AI, Stevens RJ, Neil A, Stratton IM, Boulton AJM, Holman RR. UKPDS 59: hyperglycemia and other potentially modifiable risk factors for peripheral vascular disease in type 2 diabetes. Diabetes Care. 2002;25(5):894-899. [DOI] [PubMed] [Google Scholar]
- 30. Smith MB, Albanese-O’Neill A, Macieira TGR, et al. Human factors associated with continuous glucose monitor use in patients with diabetes: a systematic review. Diabetes Technol Ther. 2019;21(10):589-601. [DOI] [PubMed] [Google Scholar]
- 31. Toh MR, Teo V, Kwan YH, Raaj S, Tan SY, Tan JZ. Association between number of doses per day, number of medications and patient’s non-compliance, and frequency of readmissions in a multi-ethnic Asian population. Prev Med Rep. 2014;1:43-47. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32. Low LL, Lee KH, Hock Ong ME, et al. Predicting 30-day readmissions: performance of the LACE index compared with a regression model among general medicine patients in Singapore. Biomed Res Int. 2015;2015:169870. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. Low LL, Liu N, Wang S, Thumboo J, Ong ME, Lee KH. Predicting 30-day readmissions in an Asian population: building a predictive model by incorporating markers of hospitalization severity. PLoS One. 2016;11(12):e0167413. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34. Hsieh CJ. High glucose variability increases 30-day readmission rates in patients with type 2 diabetes hospitalized in department of surgery. Sci Rep. 2019;9(1):14240. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35. Rosen OZ, Fridman R, Rosen BT, Shane R, Pevnick JM. Medication adherence as a predictor of 30-day hospital readmissions. Patient Prefer Adherence. 2017;11:801-810. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36. Alavi M, Baharlooei O, AdelMehraban M. Do psychosocial factors predict readmission among diabetic elderly patients? Iran J Nurs Midwifery Res. 2017;22(6):460-464. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37. Wu EQ, Zhou S, Yu A, et al. Outcomes associated with post-discharge insulin continuity in US patients with type 2 diabetes mellitus initiating insulin in the hospital. Hosp Pract (1995). 2012;40(4):40-48. [DOI] [PubMed] [Google Scholar]
- 38. Shaka H, Edigin E, Akuna E, et al. Rates and predictors of 30-day readmission in adults with type 1 diabetes hospitalized for diabetic ketoacidosis in the US: a nationwide study. J Endocr Soc. 2021;5(suppl 1):A449-A449. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
Some or all data generated or analyzed during this study are included in this published article or in the data repositories listed in the references.
