Abstract
Background
To reduce readmissions, it may be cost-effective to consider risk stratification, with targeting intervention programs to patients at high risk of readmissions. In this study, we aimed to derive and validate a prediction model including several novel markers of hospitalization severity, and compare the model with the LACE index (Length of stay, Acuity of admission, Charlson comorbidity index, Emergency department visits in past 6 months), an established risk stratification tool.
Method
This was a retrospective cohort study of all patients ≥ 21 years of age, who were admitted to a tertiary hospital in Singapore from January 1, 2013 through May 31, 2015. Data were extracted from the hospital’s electronic health records. The outcome was defined as unplanned readmissions within 30 days of discharge from the index hospitalization. Candidate predictive variables were broadly grouped into five categories: Patient demographics, social determinants of health, past healthcare utilization, medical comorbidities, and markers of hospitalization severity. Multivariable logistic regression was used to predict the outcome, and receiver operating characteristic analysis was performed to compare our model with the LACE index.
Results
74,102 cases were enrolled for analysis. Of these, 11,492 patient cases (15.5%) were readmitted within 30 days of discharge. A total of fifteen predictive variables were strongly associated with the risk of 30-day readmissions, including number of emergency department visits in the past 6 months, Charlson Comorbidity Index, markers of hospitalization severity such as ‘requiring inpatient dialysis during index admission, and ‘treatment with intravenous furosemide 40 milligrams or more’ during index admission. Our predictive model outperformed the LACE index by achieving larger area under the curve values: 0.78 (95% confidence interval [CI]: 0.77–0.79) versus 0.70 (95% CI: 0.69–0.71).
Conclusion
Several factors are important for the risk of 30-day readmissions, including proxy markers of hospitalization severity.
Introduction
Preventing unnecessary readmissions is one of the principal challenges facing health systems worldwide. The biggest driver of healthcare cost remains inpatient cost and the average cost of a bed day in Singapore in 2009 was USD 623[1]. The potential cost savings are huge if readmission reduction programs are successful. It has been suggested that reducing unnecessary hospital readmissions first require adequate risk stratification to identify patients at highest risk for readmission, followed by effective intervention programs targeted at modifiable risk factors [2]. It is also widely suggested that readmissions are most preventable in the immediate period after discharge [3]. Therefore, most efforts have concentrated on predicting 30-day readmission risk and developing transitional care programs focused on reducing this risk. Since 2012, hospitals in the United States with excessive 30-day readmission rates for acute myocardial infarction, pneumonia and heart failure are penalized by the Centre for Medicare and Medicaid [4].
For risk stratification, many countries and health systems have developed 30-day readmission models specific to their settings [5], although there is less literature on successful implementation in clinical settings. The more notable prediction scores include PARR-30 developed in the United Kingdom [6], HOSPITAL score [7] in the United States for potentially avoidable readmissions and the LACE index (Length of stay, Acuity of admission, Charlson comorbidity index, Emergency department visits in past 6 months) developed in Ontario, Canada [8]. With the exception of the LACE index, most of these models have limited generalizability to other health systems due to the unique socio-demographic variables [6,9] or have limited clinical utility due to the complexity of the model [5,9,10]. Moreover, the LACE index had moderate discriminative ability c-statistic 0.7 despite its simplicity while only four out of 25 other predictive models reviewed by Kansagara et al performed better [5]. The most distinctive advantage of the LACE index is its replicability that allows external validation in populations beyond its original derivation setting. However, to date, the LACE index performed poorly when externally validated in an older UK population with a c-statistic 0.55 [11] and in Denmark c-statistic 0.648 [12].
Singapore is a city state in South-east Asia with a multi-ethnic population of 5.6 million people. It’s population is one of the most rapidly ageing in Asia with an increasing chronic disease burden [13]. In 2010, the all-cause 30-day readmission rate was 11.6%, but rises to 19.0% in the elderly 65 years and older [14]. This rate is only slightly lower than the 19.6% in the United States [15]. Although Bloomberg rated Singapore as the most efficient healthcare system, healthcare expenditure is expected to triple from S$4 billion in 2011 to S$12 billion in 2020 with 10,000 additional hospital beds required to meet the needs of an aging population. There is great interest to develop a sustainable healthcare system that is future proof for the aging population. Several transitional care programs are already successful in reducing readmission rates [1,16], but these are more criteria driven in patient selection rather than targeted at patients at highest risk of readmission. Consequently, there is a need to identify predictors of readmission risk to derive a predictive model that can guide patient selection for these resource intensive programs.
Suggested predictors of 30-day readmission risk from previous studies include age, Charlson comorbidity index, high-risk medications on discharge, prior healthcare utilization patterns and social determinants of health [5,17–19]. In Kansagara’s review, 14 out of 26 models were derived from retrospective administrative data only and were limited in clinical predictors of illness severity. Only three models looked at real-time automated data and the most promising was Amarasingham et al, who utilized 17 laboratory and vital sign variables. However, missing laboratory and vital sign values would have limited its clinical utility in our setting, especially applied to a general hospital population. In an updated review by Zhou et al on 73 risk predictive models from 2011 to 2015, the most common variables included in predictive models were comorbidities, demographics / social, length of stay and number of previous admissions [20]. A significant gap remained on using markers of hospitalization severity to discriminate for higher risk patients to predict for early 30-day readmissions. Therefore, a potential area of interest in readmission prediction is the markers of hospitalization severity that are discriminative and simple to collect. Any risk score should be available to clinicians and case managers before patient discharge as interventions targeting both the hospitalization phase and post-discharge phase had been found to be most effective [21].
Leveraging on the electronic health record, our primary objective was to systematically investigate known and potential predictors of 30-day readmission risk, including markers of hospitalization severity. The performance of our prediction model will be measured using receiver operating characteristic (ROC) analysis, sensitivity, specificity and precision and compared with the LACE index. We hypothesize that a prediction model incorporating additional markers of hospitalization severity will outperform an established risk stratification tool (the LACE index) at predicting readmissions within 30 days of a patient’s discharge from hospital.
Methods
Study design and subjects
This was a retrospective cohort study performed at the Singapore General Hospital (SGH), aiming to compare the LACE index with a predictive model incorporating markers of hospitalization severity. SGH is the largest tertiary hospital in Singapore with 1597 beds, accounting for about 20% of the total public acute hospital beds. SGH is wholly owned by the Ministry of Health Holdings, Singapore. With a workforce numbering above 10,000, SGH admits over 1 million patients every year at its wards, emergency department and outpatient specialist clinics [22].
All admitted adult patients 21 years of age or older from January 1, 2013 through May 31, 2015 at the Singapore General Hospital were enrolled. Patients were eligible for inclusion if they were residents of Singapore. Patients were excluded if they died during the hospitalization or if the admission specialty was obstetrics, emergency medicine, dentistry or ophthalmology. We excluded admissions to the emergency department as these are observation ward stays up to 24 hours duration. We excluded patient cases admitted to obstetrics as these admissions are pregnancy related; and admissions to dentistry and ophthalmology are usually elective in nature. In addition, patients who died during the inpatient admission were excluded.
Outcome and predictive variables
Clinical and administrative data were extracted from SingHealth’s electronic health records (EHR) system, Electronic Health Intelligence System (eHINTS), which is an enterprise data repository that integrates information from multiple sources, including administrative, clinical and ancillary.
The outcome was defined as unplanned readmissions within 30 days of discharge from the index hospitalization. Readmissions to our hospital were captured by the EHR.
Candidate variables were identified a priori and based on a survey of literatures [5,22–24], and were broadly grouped into five categories: (1) Patient demographics; (2) Social determinants of health; (3) Past healthcare utilization; (4) Medical comorbidities; and (5) Markers of hospitalization severity. Patient demographics included age, gender, and ethnicity. Social determinants of health included the requirement of financial assistance using Medifund (an endowment fund of Singapore government to help needy Singapore citizens to pay the remainder of their hospital bills after receiving government subsidies and drawing on other means of payment including insurance) and admission to a subsidized hospital ward. Past healthcare utilization consisted of number of emergency department (ED) visits in the past six months, number of specialist clinic visits in the past year, and number of hospital admissions in the past year, all before the index admission. For medical comorbidities, chronic diseases such as heart failure, chronic obstructive pulmonary disease, cerebrovascular accident, peripheral vascular disease among other major diseases listed under the Charlson Comorbidity Index, Elixhauser comorbidities and Singapore Ministry of Health Chronic Diseases Program were extracted [25–27]. These diseases were extracted using International Classification of Diseases (ICD) -10 codes of primary and secondary discharge diagnoses dating back to seven years. This is most comprehensive among published literature and would account for potential lapses in diagnostic coding [28]. We also calculated the Charlson Comorbidity Index (CCI) for each patient using the comorbidities.icd10 package in R version 3.2.3 (R Foundation, Vienna, Austria). CCI is also a validated marker for mortality within one year [25].
The psychological state of patients is a well-known factor in coping with illness. We used treatment with anti-depressants in the past one year as a proxy marker of debilitating mood disorders. This is likely to be a more accurate proxy marker than ICD-codes as many mood disorders are managed as outpatients and the use of pharmacy records is recommended to improve validity of ICD coding [29]. For markers of hospitalization severity, we included length of stay of index admission, ‘treatment with intravenous furosemide 40 milligrams or more’; ‘treatment with second line intravenous antibiotics’ (defined as Piperacillin-Tazobactam, Vancomycin, Meropenem and Moxifloxacin); ‘admission to isolation ward’ and ‘required dialysis’. Finally, the LACE index was also computed for each patient [8]. It included four variables, namely length of stay, acute admission, Charlson comorbidity index (CCI), and number of ED visits in the past six months.
Statistical analysis
We compared the characteristics of patients who were readmitted within 30 days with patients who were not readmitted, using t-test for continuous variables and χ2 test or Fisher’s exact test for dichotomous variables. We used both univariable and multivariable logistic regressions to examine the associations between candidate predictors and 30-day readmission. The statistical significance was set at p<0.05. In evaluating the predictive models, we adopted 10-fold cross-validation scheme. In 10-fold cross-validation, the whole dataset was first evenly distributed into 10 non-overlapped subsets. This was followed by building a predictive model with nine subsets of data and validated on the remaining one subset. This process was repeated another nine times so that all subsets are tested.
Receiver operating characteristic (ROC) analysis was performed to compare area under the ROC curve (AUC) between our derived regression model and the LACE index. Additionally, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were compared and reported. In this study, descriptive analysis and statistical modeling were performed in R version 3.2.3 and the ROC analysis was done in MATLAB R2014b (Mathworks, Natick, MA).
The study was approved by the Singapore Health Services (SingHealth) Centralized Institutional Review Board with waiver of informed consent (CIRB 2015/2696).
Results
During the two and a half-year study period, there were 206,699 patient admissions. Of these, 132,597 (64.2%) patient cases were excluded from analysis (Fig 1). The remaining 74,102 patient cases were enrolled with a 30-day readmission rate of 15.5% (n = 11,492). Table 1 presents baseline demographics, prior healthcare utilization, clinical, and comorbidity characteristics of the study population. Patient cases who were readmitted within 30 days after discharge were generally older than patient cases who were not readmitted, with the mean age of 65 (SD = 15) versus 60 (SD = 17). In comparing the readmission group with the non-readmission group, most demographics, clinical and prior healthcare utilization variables were statistically significant except for ‘treatment with second line intravenous antibiotics’ during the inpatient stay (p = 0.947).
Fig 1. Study flow chart showing number of included, excluded, and readmitted patients.
Table 1. Baseline characteristics by 30-day hospital readmission status.
| Variable | All patients (n = 74102) | Readmitted patients (n = 11492) | No readmitted patients (n = 62610) | p-value |
|---|---|---|---|---|
| Patient Demographics | ||||
| Age, Mean (SD) | 61.27 (17.18) | 65.27 (15.57) | 60.53 (17.36) | <0.001 |
| Gender, Male (%) | 38641 (52.1%) | 6374 (55.5%) | 32267 (51.5%) | <0.001 |
| Ethnicity | ||||
| Chinese (%) | 54162 (73.1%) | 8667 (75.4%) | 45495 (72.7%) | <0.001 |
| Indian (%) | 7184 (9.7%) | 1091 (9.5%) | 6093 (9.7%) | 0.438 |
| Malay (%) | 9200 (12.4%) | 1387 (12.1%) | 7813 (12.5%) | 0.227 |
| Others (%) | 3556 (4.8%) | 347 (3%) | 3209 (5.1%) | <0.001 |
| Required financial assistance using Medifund (%) | 1907 (2.6%) | 636 (5.5%) | 1271 (2%) | <0.001 |
| Past Healthcare Utilization | ||||
| ED visits (6 month before index admission), Mean (SD) | 1.02 (2.25) | 2.48 (4.24) | 0.75 (1.49) | <0.001 |
| Specialist Clinic visits (1 year before index admission), Mean (SD) | 3.93 (9.19) | 6.93 (15.24) | 3.37 (7.43) | <0.001 |
| Hospital admissions (1 year before index admission), Mean (SD) | 1.58 (3.14) | 4.01 (5.88) | 1.13 (2.02) | <0.001 |
| Index Admission Variables | ||||
| Index admission was urgent (%) | 57017 (76.9%) | 9763 (85%) | 47254 (75.5%) | <0.001 |
| Index admission was planned (%) | 13471 (18.2%) | 1325 (11.5%) | 12146 (19.4%) | <0.001 |
| Stayed in a subsidized ward during index admission (%) | 60463 (81.6%) | 10286 (89.5%) | 50177 (80.1%) | <0.001 |
| Required second line antibiotics during index admission (%) | 5312 (7.2%) | 826 (7.2%) | 4486 (7.2%) | 0.947 |
| Required inpatient dialysis during index admission (%) | 4861 (6.6%) | 1233 (10.7%) | 3628 (5.8%) | <0.001 |
| Required intravenous Furosemide 40mg and above during index admission (%) | 5826 (7.9%) | 1268 (11%) | 4558 (7.3%) | <0.001 |
| Required isolation during index admission (%) | 1422 (1.9%) | 300 (2.6%) | 1122 (1.8%) | <0.001 |
| Length of stay of index admission, Mean (SD) | 6.09 (10.37) | 5.29 (5.38) | 6.23 (11.04) | <0.001 |
| Medical Comorbidities (ICD codes past 7 years) | ||||
| Depression (%) | 11554 (15.6%) | 3396 (29.6%) | 8158 (13%) | <0.001 |
| Stroke (%) | 955 (1.3%) | 146 (1.3%) | 809 (1.3%) | 0.885 |
| Metastatic Disease (%) | 7543 (10.2%) | 1244 (10.8%) | 6299 (10.1%) | 0.013 |
| Non-metastatic malignancy (%) | 11810 (15.9%) | 1895 (16.5%) | 9915 (15.8%) | 0.081 |
| Peripheral Vascular Disease (%) | 2484 (3.4%) | 416 (3.6%) | 2068 (3.3%) | 0.088 |
| Heart Failure or Fluid Overload (%) | 7864 (10.6%) | 1236 (10.8%) | 6628 (10.6%) | 0.6 |
| Pressure Ulcer (%) | 1907 (2.6%) | 312 (2.7%) | 1595 (2.5%) | 0.313 |
| Thromboembolism (%) | 3896 (5.3%) | 628 (5.5%) | 3268 (5.2%) | 0.29 |
| Spine Fracture (%) | 1718 (2.3%) | 298 (2.6%) | 1420 (2.3%) | 0.036 |
| Coronary Heart Disease or Myocardial Infarction (%) | 8922 (12%) | 1454 (12.7%) | 7468 (11.9%) | 0.029 |
| Hip Fracture (%) | 1199 (1.6%) | 185 (1.6%) | 1014 (1.6%) | 0.971 |
| Atrial Fibrillation (%) | 4378 (5.9%) | 675 (5.9%) | 3703 (5.9%) | 0.882 |
| Epilepsy (%) | 788 (1.1%) | 131 (1.1%) | 657 (1%) | 0.412 |
| Parkinsonism (%) | 840 (1.1%) | 123 (1.1%) | 717 (1.1%) | 0.516 |
| Anxiety (%) | 734 (1%) | 132 (1.1%) | 602 (1%) | 0.07 |
| Bipolar Disorder (%) | 178 (0.2%) | 33 (0.3%) | 145 (0.2%) | 0.31 |
| Collagen Vascular Disease (%) | 1152 (1.6%) | 179 (1.6%) | 973 (1.6%) | 1 |
| Dementia (%) | 1840 (2.5%) | 256 (2.2%) | 1584 (2.5%) | 0.06 |
| Hypothyroidism (%) | 1323 (1.8%) | 216 (1.9%) | 1107 (1.8%) | 0.429 |
| Chronic Kidney Disease, Stages 1–4 (%) | 11266 (15.2%) | 1804 (15.7%) | 9462 (15.1%) | 0.111 |
| Chronic Obstructive Pulmonary Disease (%) | 1589 (2.1%) | 248 (2.2%) | 1341 (2.1%) | 0.94 |
| Osteoarthritis (%) | 4947 (6.7%) | 821 (7.1%) | 4126 (6.6%) | 0.03 |
| Benign Prostatic Hypertrophy (%) | 2651 (3.6%) | 382 (3.3%) | 2269 (3.6%) | 0.118 |
| Asthma (%) | 2186 (2.9%) | 321 (2.8%) | 1865 (3%) | 0.294 |
| Hyperlipidemia (%) | 17722 (23.9%) | 2851 (24.8%) | 14871 (23.8%) | 0.015 |
| Hypertension (%) | 24835 (33.5%) | 3925 (34.2%) | 20910 (33.4%) | 0.117 |
| Chronic Kidney Disease Stage 5 or End Stage Renal Failure (%) | 8866(12%) | 1441(12.5%) | 7425 (11.9%) | 0.04 |
| Diabetes (%) | 16201 (21.9%) | 2614 (22.7%) | 13587 (21.7%) | 0.013 |
| History of Alcoholism (%) | 1372 (1.9%) | 316 (2.7%) | 1056 (1.7%) | <0.001 |
| Charlson Comorbidity Index, Mean (SD) | 2.27 (2.85) | 3.83 (3.15) | 1.98 (2.69) | <0.001 |
Readmitted patient cases had used healthcare services more frequently compared with non-readmitted patient cases, in terms of number of ED visits in the past six months (mean = 2.48 [SD = 4.24] versus mean = 0.75 [SD = 1.49]), number of specialist clinic visits in the past year before index admission (mean = 6.93 [SD = 15.24] versus mean = 3.37 [SD = 7.43]), and number of hospital admissions in the past year before index admission (mean = 4.01 [SD = 5.88] versus mean = 1.13 [SD = 2.02]). Medical comorbidities including metastatic disease, spine fracture, coronary heart disease or myocardial infarction, osteoarthritis, hyperlipidemia, chronic kidney disease stage 5 or end stage renal failure, diabetes, history of alcoholism and treatment with anti-depressants in the past one year were statistically significant. Furthermore, readmitted patient cases had a mean CCI of 3.83 and non-readmitted patients had a mean CCI of 1.98.
Univariable and multivariable logistic regressions were performed and the results were shown in Table 2. Several variables were statistically significant in univariable logistic regression but became insignificant in multivariable logistic regression, including number of specialist clinic visits in the past year before index admission, ‘admission to an isolation ward’ during inpatient stay, and the following medical comorbidities (metastatic disease, coronary heart disease or myocardial infarction, hyperlipidemia, chronic kidney disease stage 5 or end stage renal failure, and diabetes).
Table 2. Univariable and multivariable logistic regression.
| Variable | Unadjusted OR (95% CI) | p-value | Adjusted OR (95% CI) | p-value |
|---|---|---|---|---|
| Patient Demographics | ||||
| Age | 1.02 (1.02, 1.02) | <0.001 | 1.01 (1.01, 1.01) | <0.001 |
| Gender (Male) | 1.17 (1.13, 1.22) | <0.001 | 1.09 (1.05, 1.14) | <0.001 |
| Ethnicity | ||||
| Others | Baseline | |||
| Chinese | 1.15 (1.10, 1.21) | <0.001 | 1.15 (1.02, 1.31) | 0.025 |
| Indian | 0.97 (0.91, 1.04) | 0.428 | 0.98 (0.86, 1.14) | 0.832 |
| Malay | 0.96 (0.91, 1.02) | 0.221 | 1.10 (0.96, 1.26) | 0.175 |
| Required financial assistance using Medifund | 2.83 (2.56, 3.12) | <0.001 | 1.24 (1.10, 1.39) | <0.001 |
| Past Healthcare Utilization | ||||
| ED visits (6 month before index admission) | 1.35(1.34, 1.37) | <0.001 | 1.07 (1.06, 1.08) | <0.001 |
| Specialist Clinic visits (1 year before index admission) | 1.04(1.04, 1.04) | <0.001 | 1.00 (1.00, 1.00) | 0.953 |
| Hospital admissions (1 year before index admission) | 1.30(1.29, 1.31) | <0.001 | 1.19 (1.17, 1.20) | <0.001 |
| Index Admission Variables | ||||
| Index admission was urgent | 1.83 (1.74, 1.94) | <0.001 | 1.49 (1.4, 1.58) | <0.001 |
| Stayed in a subsidized ward during index admission | 2.11 (1.99, 2.25) | <0.001 | 1.43 (1.33, 1.54) | <0.001 |
| Required second line antibiotics during index admission | 1.00 (0.93, 1.08) | 0.931 | 1.00 (0.92, 1.10) | 0.921 |
| Required inpatient dialysis during index admission | 1.95 (1.83, 2.09) | <0.001 | 1.19 (1.10, 1.29) | <0.001 |
| Required intravenous Furosemide 40mg and above during index admission | 1.58 (1.48, 1.69) | <0.001 | 1.24 (1.15, 1.34) | <0.001 |
| Required isolation during index admission | 1.47 (1.29, 1.67) | <0.001 | 1.15 (0.99, 1.34) | 0.069 |
| Length of stay of index admission | 0.99 (0.99, 0.99) | <0.001 | 0.96 (0.96, 0.96) | <0.001 |
| Medical Comorbidities (ICD codes past 7 years) | ||||
| Depression | 2.80 (2.67, 2.93) | <0.001 | 1.57 (1.49, 1.66) | <0.001 |
| Stroke | 0.98 (0.82, 1.17) | 0.85 | 0.92 (0.75, 1.12) | 0.432 |
| Metastatic Disease | 1.09 (1.02, 1.16) | 0.013 | 1.08 (0.99, 1.18) | 0.071 |
| Non-metastatic malignancy | 1.10 (0.99, 1.22) | 0.083 | 1.06 (0.93, 1.20) | 0.382 |
| Peripheral Vascular Disease | 1.02 (0.95, 1.09) | 0.588 | 0.96 (0.87, 1.05) | 0.352 |
| Heart Failure or Fluid Overload | 1.07 (0.94, 1.21) | 0.298 | 1.09 (0.94, 1.25) | 0.237 |
| Pressure Ulcer | 1.05 (0.96, 1.15) | 0.279 | 1.01 (0.91, 1.12) | 0.859 |
| Thromboembolism | 1.05 (0.99, 1.11) | 0.078 | 1.01 (0.94, 1.09) | 0.727 |
| Spine Fracture | 1.15 (1.01, 1.30) | 0.033 | 1.17 (1.02, 1.35) | 0.025 |
| Coronary Heart Disease or Myocardial Infarction | 1.07 (1.01, 1.14) | 0.028 | 1.04 (0.96, 1.14) | 0.311 |
| Hip Fracture | 0.99 (0.85, 1.16) | 0.939 | 0.96 (0.81, 1.14) | 0.668 |
| Atrial Fibrillation | 0.99 (0.91, 1.08) | 0.865 | 0.98 (0.88, 1.08) | 0.628 |
| Epilepsy | 1.09 (0.90, 1.31) | 0.384 | 0.97 (0.78, 1.20) | 0.794 |
| Parkinsonism | 0.93 (0.77, 1.13) | 0.486 | 0.92 (0.74, 1.14) | 0.452 |
| Anxiety | 1.20 (0.99, 1.44) | 0.063 | 1.17 (0.94, 1.44) | 0.147 |
| Bipolar Disorder | 1.24 (0.84, 1.79) | 0.264 | 1.31 (0.84, 1.97) | 0.217 |
| Collagen Vascular Disease | 1.00 (0.85, 1.17) | 0.978 | 0.95 (0.79, 1.13) | 0.567 |
| Dementia | 0.88 (0.77, 1.00) | 0.056 | 0.89 (0.76, 1.03) | 0.117 |
| Hypothyroidism | 1.06 (0.92, 1.23) | 0.407 | 1.06 (0.90, 1.26) | 0.459 |
| Chronic Kidney Disease, Stages 1–4 | 1.05 (0.99, 1.11) | 0.108 | 0.98 (0.88, 1.09) | 0.652 |
| Chronic Obstructive Pulmonary Disease | 1.01 (0.88, 1.15) | 0.912 | 1.00 (0.85, 1.16) | 0.961 |
| Osteoarthritis | 1.09 (1.01, 1.18) | 0.029 | 1.11 (1.02, 1.22) | 0.019 |
| Benign Prostatic Hypertrophy | 0.91 (0.82, 1.02) | 0.112 | 0.95 (0.84, 1.07) | 0.417 |
| Asthma | 0.94 (0.83, 1.05) | 0.28 | 0.87 (0.76, 1.00) | 0.055 |
| Hyperlipidemia | 1.06 (1.01, 1.11) | 0.015 | 1.03 (0.96, 1.11) | 0.395 |
| Hypertension | 1.03 (0.99, 1.08) | 0.114 | 0.98 (0.92, 1.06) | 0.662 |
| Chronic Kidney Disease Stage 5 or End Stage Renal Failure | 1.07 (1.00, 1.13) | 0.039 | 1.04 (0.92, 1.16) | 0.551 |
| Diabetes | 1.06 (1.01, 1.11) | 0.013 | 1.04 (0.97, 1.12) | 0.295 |
| History of Alcoholism | 1.65 (1.45, 1.87) | <0.001 | 1.22 (1.05, 1.42) | 0.008 |
| Charlson Comorbidity Index | 1.21 (1.20, 1.22) | <0.001 | 1.14 (1.13,1.15) | <0.001 |
A total of fifteen predictive variables were strongly associated with the risk of 30-day readmissions (Table 2). We built the multivariable logistic regression model using the following statistically significant variables: age, gender, required financial assistance using Medifund, number of visits to the emergency department in the past 6 months, number of hospital admissions in the past year before index admission, whether or not the index admission was urgent, whether or not staying in a subsidized ward during index admission, required inpatient dialysis during index admission, ‘treatment with intravenous furosemide 40 milligrams or more’ during index admission, length of stay of index admission, comorbidities (depression, spine fracture, osteoarthritis, and history of alcoholism), and the Charlson Comorbidity Index.
Fig 2 depicted the ROC curves produced by our logistic regression model and the LACE index where the results were based on 10-fold cross-validation. Our model outperformed the LACE index by achieving larger area under the curve (AUC) values: 0.78 (95% confidence interval [CI]: 0.77–0.79) versus 0.70 (95% CI: 0.69–0.71). Moreover, sensitivity, specificity, PPV and NPV were compared where optimal cutoffs were chosen (Table 3). The optimum was determined by the point on the ROC curve, which was nearest to the upper left corner. At the cutoff of 0.14, our logistic regression model obtained sensitivity of 74.3% (95% CI: 73.5%-75.1%) and specificity of 67.3% (95% CI: 66.9%-67.7%). At the cutoff of 8, the LACE index achieved sensitivity of 72.0% (95% CI: 71.2%-72.8%) and specificity of 60.3% (95% CI: 59.9%-60.7%).
Fig 2. ROC curves of the LACE index and our logistic regression model.
Table 3. Prediction results by the multivariable logistic regression (LR) model using receiver operating characteristic (ROC) analysis.
| AUC (95% CI) | Cut-off | Sensitivity (95% CI) | Specificity (95% CI) | PPV (95% CI) | NPV (95% CI) | |
|---|---|---|---|---|---|---|
| LR | 0.78 (0.77–0.79) | 0.14 | 74.3% (73.5% - 75.1%) | 67.3% (66.9% - 67.7%) | 29.4% (28.9% - 30.0%) | 93.4% (93.2% - 93.7%) |
| LACE | 0.70 (0.69–0.71) | 8 | 72.0% (71.2% - 72.8%) | 60.3% (59.9% - 60.7%) | 25.0% (24.5% - 25.4%) | 92.1% (91.9% - 92.4%) |
AUC: area under the ROC curve; CI: confidence interval; PPV: positive predictive value; NPV: negative predictive value; LR: logistic regression
Furthermore, the calibration plots (Fig 3) demonstrated that our model achieved stronger agreement between observed and predicted probabilities when compared to the LACE index. In the bin with class probabilities ranging from 20% to 30%, the observed percentage of events for the LACE index was 5.7%, lower than the percentage in our logistic regression model (23.4%). In the bin with class probabilities ranging from 90% to 100%, the observed percentage of events for the LACE index was 23.7%, far lower than the percentage in our model (91.1%).
Fig 3. Comparison of the calibration plots between our logistic regression model and the LACE index.
The calibration curves were generated using “caret” package in R version 3.2.3 (R Foundation, Vienna, Austria).
Discussion
In this retrospective observational study of 74,102 eligible adult cases, we have identified 15 predictors associated with readmission within 30 days of hospital discharge in Singapore. In this cohort, our predictive model incorporating markers of hospitalization severity and social determinants of health significantly outperformed the LACE index in terms of AUC (0.78 versus 0.70, p<0.001), sensitivity, specificity, positive predictive value and negative predictive value. Compared to published models where majority of the models had an AUC<0.7, our final regression model had good discriminatory power with an AUC of 0.78 and the advantage of being available before patient discharge. We intend to further develop these predictors into a risk score and externally validate the risk score in other health systems. Surprisingly, the performance of the LACE index in our study cohort was comparable to the Ontario population that it was first derived and performed better than the UK and Danish populations that the index was externally validated on [11,12].
Although we started deriving our risk score with 44 variables, we found that 15 factors were important in predicting 30-day readmissions. Our findings that increasing age, prior hospital admissions and emergency department visits and social determinants of health increased 30-day readmission are consistent with existing literature [5,23]. Interestingly, treatment with anti-depressants is the strongest predictor for 30-day readmission risk in our patient cohort. Pederson et al found that one-third of patients discharged from medical wards were depressed [30], while one-sixth of our cohort were treated with anti-depressants. In their review, depressed patients had a 73% higher risk to be readmitted within 30 days compared to 57% in our cohort. Our findings also have implications that depression as a potentially modifiable risk factor could have been under-recognized in risk stratification and under-treated in intervention programs. Similarly, patients with a history of alcoholism may be a surrogate marker of existing or recent social instability. Further investigation on the reasons and causes of 30-day readmissions in these patients would inform the healthcare system to develop focused models of care targeted at these risk factors. Interestingly, increasing length of stay (LOS) was associated with a protective effect for 30-day readmission in our cohort. This is contrary to the findings of Walraven et al and the LACE index. In a review on hospital length of stay and 30-day readmission rate, Kaboli et al found that hospitals with mean risk-adjusted LOS that was lower than expected had a 6% increase in readmission rate for each day lower than expected. These findings and ours suggest that there is a modest trade-off between hospital LOS and readmission, and further investigation is require to examine which patient subgroups are at highest risk of premature discharge and increased risk of adverse outcomes.
Our study is also novel in exploring proxy markers of hospitalization severity on 30-day readmission risk. To our knowledge, no previous studies have evaluated the impact of high risk medications used during the hospitalization stay but instead focused on high risk medications at discharge [17]. Intravenous furosemide is used when prompt and effective diuresis is required and second line antibiotics are used to treat severe hospital acquired infections [31]. Similarly, patients receiving inpatient dialysis could have suffered from acute renal failure or are existing end stage renal failure (ESRF) patients. Although our data does not allow us to differentiate between the two, it is noteworthy that patients with Stage 5 chronic kidney disease or ESRF were not at higher risk of 30-day readmission. We found that ESRF patients requiring inpatient dialysis and patients who received intravenous furosemide 40mg and above were 19% and 24% more likely to be readmitted in the following 30 days. It is possible that undergoing inpatient dialysis and intravenous furosemide better reflect current illness burden than pre-existing medical comorbidities.
In the final multivariable logistic regression model, with the exception of depression, alcoholism and Charlson Comorbidity Index (CCI), the rest of chronic disease comorbidities were not independently associated with 30-day readmission risk. A likely explanation is that comorbidities are confounded by CCI and utilization in past one year. The CCI is a validated measure of one year mortality [25] and covered 11 of the 28 comorbidities investigated in our study. Another explanation is that a patient’s past comorbidities may have lesser significance on his current risk than recent events. Another three (alcoholism, hypothyroidism and parkinsonism) are part of the elixhauser comorbidity index [32], while the rest are from the Singapore chronic disease management program [27]. Our study findings also have implications that tedious retrieval of individual Charlson and Elixhauser comorbidities may not be yielding in determining 30-day risk.
In Kansagara’s review of readmission risk prediction models, only 11 out of the 26 unique models considered social determinants of health [33–35]. Of these 11 models, social determinants of health were defined but not restricted to income, employment status, insurance status, education level and marital status. Social determinants are often unique to the population or country and less generalizable across settings. We selected admission to subsidized ward and requiring medifund for hospitalization bill settlement as social determinants of health in our study, as these information are readily available in our setting. Patients requiring Medifund are among the neediest in Singapore. Medifund is a safety net by the government to reimburse public hospitals for treating patients who would otherwise not afford the hospitalization bill. This ensures no citizen is denied healthcare. Only 2.6% in our patient cohort required financial assistance with Medifund and these patients had a 24% higher risk of 30-day readmission after adjustment for other predictors. Finally, the inclusion or exclusion of these social determinants of health in a risk score should take into consideration the impact on model performance, generalizability and its planned application.
It is noteworthy that many risk predictive models focused on predicting readmissions in cardiovascular-related disease including pneumonia [20,36–41]. As our setting was in a general hospital, we did not restrict our study population to patients with cardiovascular-related disease or pneumonia. In future studies, it would be interesting to validate our findings in this group of patients while incorporating more specific markers of cardiovascular-related disease severity. Unsurprisingly, patients who discharged against medical advice were at higher risk for readmissions in medical and cardiovascular disease related discharges [42,43]. While this indicator is available too late in the admission to include for risk prediction, this group of high-risk patients require attention for post-discharge surveillance.
There is potential to explore other variables of interest in future. These include health literacy, functional status, caregiver availability and markers of social instability [5,44]. At the moment, these data are not routinely collected in most health systems although we have an intention to do so as part of a population database in our hospital. In the interim, we have focused on identifying predictors that can be easily retrieved from a patient’s medical records or can potentially be automated through the EHR. Therefore, we have overlooked variables that require additional collection by healthcare workers currently. Taha et al [45] explored polypharmacy and problem medications such as anti-coagulants and opioids on discharge. While these are available in our EHR system, the discharge prescription is among the finalized documents given to a patient on discharge. Delays in obtaining this information would have limited its usefulness in deriving a risk score for the case managers and clinicians. In deriving our predictors, we have intentionally selected variables that are readily available in the electronic health records (EHR) for a predictive score to be automated or can be easily retrieved from patient medical records and entered into an online spreadsheet or smartphone/tablet application to facilitate clinical use.
Limitations
Although our study was carefully prepared, several limitations must be considered. Firstly, variables in our dataset are restricted to those routinely collected in the EHR and administrative databases. As such, the granularity of social determinant variables is restricted to ward class and requirement for financial assistance. Functional status, caregiver availability and degree of social support were not routinely collected in our healthcare setting. Secondly, due to the retrospective nature of the study, we were unable to confirm a causal association between the predictor variables and frequent hospital admissions. After our predictive model has identified patients at high risk for 30-day readmission, intervention programs would have to identify potentially modifiable risk factors. Thirdly, we did not exclude patients who might have deceased after index hospital discharge. We felt that that would have biased the prediction model as these patients could have been readmitted before death or died during the readmission. Although data on the 30-day post-discharge mortality rate is not available in Singapore, this outcome only occurred in 0.81% of cases in the Ontario study [8]. Finally, although our study was conducted at the largest health system in Singapore, we were unable to account for readmissions to other health systems. We minimized such bias by excluding patients who are not Singapore citizens or permanent residents.
Conclusions
In this study we have shown that our predictive model incorporating markers of hospitalization severity and social determinants of health significantly outperformed the LACE index in the ROC analysis. We identified 15 variables that were associated with the risk of 30-day readmission. Among these variables, treatment with anti-depressants was found the strongest predictor, which may have been under-recognized in readmission risk stratification. Furthermore, we explored the use of proxy markers of hospitalization severity in predictive modeling.
Data Availability
All data used in this paper are owned by the Singapore General Hospital. Data are available from the Singapore General Hospital for researchers who meet the criteria for access to confidential data. We confirm that all interested researchers can access the data by the same means the authors accessed it. To access the data, interested researchers will need to sign a research collaboration agreement with the Singapore General Hospital. They may contact the Singapore General Hospital Division of Research (Email: division.research@sgh.com.sg). Authors Lian Leng Low, Nan Liu, Julian Thumboo, Marcus Eng Hock Ong, and Kheng Hock Lee are affiliated with the Singapore General Hospital.
Funding Statement
This study was supported by Singapore Ministry of Health (MOH) Health Services Research New Investigator Grant HSRNIG14nov002 and the SingHealth Foundation Health Services Research (Ageing) Grant SHF/HSRAg004/2015. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
References
- 1.Wee SL, Loke CK, Liang C, Ganesan G, Wong LM, Cheah J (2014) Effectiveness of a national transitional care program in reducing acute care use. J Am Geriatr Soc 62: 747–753. 10.1111/jgs.12750 [DOI] [PubMed] [Google Scholar]
- 2.Kripalani S, Theobald CN, Anctil B, Vasilevskis EE (2014) Reducing hospital readmission rates: current strategies and future directions. Annu Rev Med 65: 471–485. 10.1146/annurev-med-022613-090415 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Graham KL, Wilker EH, Howell MD, Davis RB, Marcantonio ER (2015) Differences between early and late readmissions among patients: a cohort study. Ann Intern Med 162: 741–749. 10.7326/M14-2159 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Centers for Medicare and Medicaid Services (2014) Readmissions Reduction Program.
- 5.Kansagara D, Englander H, Salanitro A, Kagen D, Theobald C, Freeman M, et al. (2011) Risk prediction models for hospital readmission: a systematic review. JAMA 306: 1688–1698. 10.1001/jama.2011.1515 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Billings J, Blunt I, Steventon A, Georghiou T, Lewis G, Bardsley M (2012) Development of a predictive model to identify inpatients at risk of re-admission within 30 days of discharge (PARR-30). BMJ Open 2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Donze J, Aujesky D, Williams D, Schnipper JL (2013) Potentially avoidable 30-day hospital readmissions in medical patients: derivation and validation of a prediction model. JAMA Intern Med 173: 632–638. 10.1001/jamainternmed.2013.3023 [DOI] [PubMed] [Google Scholar]
- 8.van Walraven C, Dhalla IA, Bell C, Etchells E, Stiell IG, Zarnke K, et al. (2010) Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ 182: 551–557. 10.1503/cmaj.091117 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Choudhry SA, Li J, Davis D, Erdmann C, Sikka R, Sutariya B (2013) A public-private partnership develops and externally validates a 30-day hospital readmission risk prediction model. Online J Public Health Inform 5: 219 10.5210/ojphi.v5i2.4726 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Bottle A, Aylin P, Majeed A (2006) Identifying patients at high risk of emergency hospital admissions: a logistic regression analysis. J R Soc Med 99: 406–414. 10.1258/jrsm.99.8.406 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Cotter PE, Bhalla VK, Wallis SJ, Biram RW (2012) Predicting readmissions: poor performance of the LACE index in an older UK population. Age Ageing 41: 784–789. 10.1093/ageing/afs073 [DOI] [PubMed] [Google Scholar]
- 12.Cooksley T, Nanayakkara PW, Nickel CH, Subbe CP, Kellett J, Kidney R, et al. (2015) Readmissions of medical patients: an external validation of two existing prediction scores. QJM. [DOI] [PubMed] [Google Scholar]
- 13.Ministry of Health Singapore (2012) Health facts Singapore Healthcare Institution Statistics 2012. www.moh.gov.sg/content/moh_web/home/statistics/healthcare_institutionstatistics.html.
- 14.Lim E, Matthew N, Mok W, Chowdhury S, Lee D (2011) Using hospital readmission rates to track the quality of care in public hospitals in Singapore. BMC Health Serv Res 11: A16. [Google Scholar]
- 15.Jencks SF, Williams MV, Coleman EA (2009) Rehospitalizations among patients in the Medicare fee-for-service program. N Engl J Med 360: 1418–1428. 10.1056/NEJMsa0803563 [DOI] [PubMed] [Google Scholar]
- 16.Low LL, Vasanwala FF, Ng LB, Chen C, Lee KH, Tan SY (2015) Effectiveness of a transitional home care program in reducing acute hospital utilization: a quasi-experimental study. BMC Health Serv Res 15: 100 10.1186/s12913-015-0750-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Allaudeen N, Vidyarthi A, Maselli J, Auerbach A (2011) Redefining readmission risk factors for general medicine patients. J Hosp Med 6: 54–60. 10.1002/jhm.805 [DOI] [PubMed] [Google Scholar]
- 18.Arbaje AI, Wolff JL, Yu Q, Powe NR, Anderson GF, Boult C (2008) Postdischarge environmental and socioeconomic factors and the likelihood of early hospital readmission among community-dwelling Medicare beneficiaries. Gerontologist 48: 495–504. [DOI] [PubMed] [Google Scholar]
- 19.Hu J, Gonsahn MD, Nerenz DR (2014) Socioeconomic status and readmissions: evidence from an urban teaching hospital. Health Aff (Millwood) 33: 778–785. [DOI] [PubMed] [Google Scholar]
- 20.Zhou H, Della PR, Roberts P, Goh L, Dhaliwal SS (2016) Utility of models to predict 28-day or 30-day unplanned hospital readmissions: an updated systematic review. BMJ Open 6: e011060 10.1136/bmjopen-2016-011060 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Hansen LO, Young RS, Hinami K, Leung A, Williams MV (2011) Interventions to reduce 30-day rehospitalization: a systematic review. Ann Intern Med 155: 520–528. 10.7326/0003-4819-155-8-201110180-00008 [DOI] [PubMed] [Google Scholar]
- 22.Tan SY, Low LL, Yang Y, Lee KH (2013) Applicability of a previously validated readmission predictive index in medical patients in Singapore: a retrospective study. BMC Health Serv Res 13: 366 10.1186/1472-6963-13-366 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Low LL, Lee KH, Hock Ong ME, Wang S, Tan SY, Thumboo J, et al. (2015) Predicting 30-Day Readmissions: Performance of the LACE Index Compared with a Regression Model among General Medicine Patients in Singapore. Biomed Res Int 2015: 169870 10.1155/2015/169870 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Shadmi E, Flaks-Manov N, Hoshen M, Goldman O, Bitterman H, Balicer RD (2015) Predicting 30-day readmissions with preadmission electronic health record data. Med Care 53: 283–289. 10.1097/MLR.0000000000000315 [DOI] [PubMed] [Google Scholar]
- 25.Deyo RA, Cherkin DC, Ciol MA (1992) Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol 45: 613–619. [DOI] [PubMed] [Google Scholar]
- 26.Quan H, Sundararajan V, Halfon P, Fong A, Burnand B, Luthi JC, et al. (2005) Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care 43: 1130–1139. [DOI] [PubMed] [Google Scholar]
- 27.Ministry of Health (2010) Chronic diseases. www.moh.gov.sg/content/moh_web/home/costs_and_financing/schemes_subsidies/medisave/Chronic_Diseases.html.
- 28.Howell S, Coory M, Martin J, Duckett S (2009) Using routine inpatient data to identify patients at risk of hospital readmission. BMC Health Serv Res 9: 96 10.1186/1472-6963-9-96 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Fiest KM, Jette N, Quan H, St. Germaine-Smith C, Metcalfe A, Patten SB, et al. (2014) Systematic review and assessment of validated case definitions for depression in administrative data. BMC Psychiatry 14: 1–11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Pederson JL, Warkentin LM, Majumdar SR, McAlister FA (2016) Depressive symptoms are associated with higher rates of readmission or mortality after medical hospitalization: A systematic review and meta-analysis. J Hosp Med. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Peleg AY, Hooper DC (2010) Hospital-acquired infections due to gram-negative bacteria. N Engl J Med 362: 1804–1813. 10.1056/NEJMra0904124 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Elixhauser A, Steiner C, Harris DR, Coffey RM (1998) Comorbidity measures for use with administrative data. Med Care 36: 8–27. [DOI] [PubMed] [Google Scholar]
- 33.Billings J, Dixon J, Mijanovich T, Wennberg D (2006) Case finding for patients at risk of readmission to hospital: development of algorithm to identify high risk patients. Bmj 333: 327 10.1136/bmj.38870.657917.AE [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Smith DM, Katz BP, Huster GA, Fitzgerald JF, Martin DK, Freedman JA (1996) Risk factors for nonelective hospital readmissions. J Gen Intern Med 11: 762–764. [DOI] [PubMed] [Google Scholar]
- 35.Hasan O, Meltzer DO, Shaykevich SA, Bell CM, Kaboli PJ, Auerbach AD, et al. (2010) Hospital readmission in general medicine patients: a prediction model. J Gen Intern Med 25: 211–219. 10.1007/s11606-009-1196-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Luthi JC, Lund MJ, Sampietro-Colom L, Kleinbaum DG, Ballard DJ, McClellan WM (2003) Readmissions and the quality of care in patients hospitalized with heart failure. Int J Qual Health Care 15: 413–421. [DOI] [PubMed] [Google Scholar]
- 37.Keenan PS, Normand SL, Lin Z, Drye EE, Bhat KR, Ross JS, et al. (2008) An administrative claims measure suitable for profiling hospital performance on the basis of 30-day all-cause readmission rates among patients with heart failure. Circ Cardiovasc Qual Outcomes 1: 29–37. 10.1161/CIRCOUTCOMES.108.802686 [DOI] [PubMed] [Google Scholar]
- 38.Jasti H, Mortensen EM, Obrosky DS, Kapoor WN, Fine MJ (2008) Causes and risk factors for rehospitalization of patients hospitalized with community-acquired pneumonia. Clin Infect Dis 46: 550–556. 10.1086/526526 [DOI] [PubMed] [Google Scholar]
- 39.Luthi JC, Burnand B, McClellan WM, Pitts SR, Flanders WD (2004) Is readmission to hospital an indicator of poor process of care for patients with heart failure? Qual Saf Health Care 13: 46–51. 10.1136/qshc.2003.006999 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Krumholz HM, Merrill AR, Schone EM, Schreiner GC, Chen J, Bradley EH, et al. (2009) Patterns of hospital performance in acute myocardial infarction and heart failure 30-day mortality and readmission. Circ Cardiovasc Qual Outcomes 2: 407–413. 10.1161/CIRCOUTCOMES.109.883256 [DOI] [PubMed] [Google Scholar]
- 41.Neupane B, Walter SD, Krueger P, Marrie T, Loeb M (2010) Predictors of inhospital mortality and re-hospitalization in older adults with community-acquired pneumonia: a prospective cohort study. BMC Geriatr 10: 22 10.1186/1471-2318-10-22 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Onukwugha E, Mullins CD, Loh FE, Saunders E, Shaya FT, Weir MR (2011) Readmissions after unauthorized discharges in the cardiovascular setting. Med Care 49: 215–224. 10.1097/MLR.0b013e31820192a5 [DOI] [PubMed] [Google Scholar]
- 43.Glasgow JM, Vaughn-Sarrazin M, Kaboli PJ (2010) Leaving against medical advice (AMA): risk of 30-day mortality and hospital readmission. J Gen Intern Med 25: 926–929. 10.1007/s11606-010-1371-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Amarasingham R, Moore BJ, Tabak YP, Drazner MH, Clark CA, Zhang S, et al. (2010) An automated model to identify heart failure patients at risk for 30-day readmission or death using electronic medical record data. Med Care 48: 981–988. 10.1097/MLR.0b013e3181ef60d9 [DOI] [PubMed] [Google Scholar]
- 45.Taha M, Pal A, Mahnken JD, Rigler SK (2014) Derivation and validation of a formula to estimate risk for 30-day readmission in medical patients. Int J Qual Health Care 26: 271–277. 10.1093/intqhc/mzu038 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
All data used in this paper are owned by the Singapore General Hospital. Data are available from the Singapore General Hospital for researchers who meet the criteria for access to confidential data. We confirm that all interested researchers can access the data by the same means the authors accessed it. To access the data, interested researchers will need to sign a research collaboration agreement with the Singapore General Hospital. They may contact the Singapore General Hospital Division of Research (Email: division.research@sgh.com.sg). Authors Lian Leng Low, Nan Liu, Julian Thumboo, Marcus Eng Hock Ong, and Kheng Hock Lee are affiliated with the Singapore General Hospital.