The derivation and validation of a simple model for predicting in-hospital mortality of acutely admitted patients to internal medicine wards

Supplemental Digital Content is available in the text

Abstract

Limited information is available about clinical predictors of in-hospital mortality in acute unselected medical admissions. Such information could assist medical decision-making.

To develop a clinical model for predicting in-hospital mortality in unselected acute medical admissions and to test the impact of secondary conditions on hospital mortality.

This is an analysis of the medical records of patients admitted to internal medicine wards at one university-affiliated hospital. Data obtained from the years 2013 to 2014 were used as a derivation dataset for creating a prediction model, while data from 2015 was used as a validation dataset to test the performance of the model. For each admission, a set of clinical and epidemiological variables was obtained. The main diagnosis at hospitalization was recorded, and all additional or secondary conditions that coexisted at hospital admission or that developed during hospital stay were considered secondary conditions.

The derivation and validation datasets included 7268 and 7843 patients, respectively. The in-hospital mortality rate averaged 7.2%. The following variables entered the final model; age, body mass index, mean arterial pressure on admission, prior admission within 3 months, background morbidity of heart failure and active malignancy, and chronic use of statins and antiplatelet agents. The c-statistic (ROC-AUC) of the prediction model was 80.5% without adjustment for main or secondary conditions, 84.5%, with adjustment for the main diagnosis, and 89.5% with adjustment for the main diagnosis and secondary conditions. The accuracy of the predictive model reached 81% on the validation dataset.

A prediction model based on clinical data with adjustment for secondary conditions exhibited a high degree of prediction accuracy. We provide a proof of concept that there is an added value for incorporating secondary conditions while predicting probabilities of in-hospital mortality. Further improvement of the model performance and validation in other cohorts are needed to aid hospitalists in predicting health outcomes.

1. Introduction

Predicting in-hospital mortality has been the focus of numerous studies aimed to provide clinicians with simple and reproducible risk assessment models. The Charlson comorbidity index,^[1] which was originally formulated to predict 1-year mortality based on coexistent comorbidities, has been adapted to predict various outcomes such as in-hospital mortality, disability, hospital readmissions, higher hospital costs, and length of stay.^[2,3] Numerous studies have consistently shown that comorbidity is one of the main factors associated with adverse outcomes among hospitalized patients.^[4,5] To date, nearly all prediction model studies that investigated the association between various variables and in-hospital mortality focused on specific patient populations such as patients with pneumonia,^[6] congestive heart failure (CHF),^[7] acute myocardial infarction,^[8] chronic obstructive pulmonary disease (COPD),^[9] stroke,^[10] infective endocarditis,^[11] and cancer.^[12] In addition, these studies mainly addressed associations between the main diagnosis at hospital discharge notes and various variables, without adjustment for secondary diagnoses or concomitant conditions that coexisted at the time of admission or that developed during hospital stay. Such conditions could have a substantial impact on the predicted probability of in-hospital mortality and are not always accounted for during data analysis.

Our objective was to develop a simple and reliable model that relies only on clinical characteristics to predict in-hospital mortality among patients admitted to internal medicine wards and to examine the impact of secondary conditions on overall hospital mortality.

2. Materials and methods

We conducted a retrospective cohort study of patients who were admitted to one of the 5 internal medicine wards at Emek Medical Center over a 3-year period. Data obtained from the first 2 years (2013–2014) were used as a derivation dataset for creating a prediction model, while data from 2015 were used as a validation dataset to test the performance of the model.

Emek Medical Center is a university-affiliated hospital located in northeastern Israel, with an annual 130,000 visits to the emergency departments. Our hospital policy for the past 15 years dictates the review of all discharge notes of hospitalized patients by specially trained administrative staff. This is to ensure the completeness and the compatibility of the main and any secondary diagnosis with the medical notes registered during the patient's hospitalization. Data for the present study were gathered using 2 separate datasets, Chameleon Medical Record (Elad Health, Tel-Aviv, Israel), a web-based electronic medical record information system that includes inpatient administrative and clinical information and SAP (systems–applications–products) Business Objects-Business Intelligence (BI) platform (SAP, Walldorf, Germany), a suite of front-end applications that assembles clinical and administrative data from both in-hospital and outpatient sources. Admissions that lasted for at least 24 hours were eligible for inclusion in the study. Cohort assembly was carried out while ensuring that only one admission per patient was included. Thus, for patients who had multiple admissions during the study period (2013–2014) and survived, one admission was randomly selected. For patients who had multiple admissions and died we chose the admission that ended with death, we did not include any admission before that. If the patient died during the study period (2013–2014), had multiple admissions, but did not die during hospital stay, we chose only the last admission before his death. For each patient's admission the following variables were obtained: age, gender, body mass index (BMI), mean arterial pressure (MAP) on admission, date of admission, date of discharge, date of death, length of stay, month of admission, weekend or weekday admission, admission within 3 months before index admission, and time of admission according to nurses’ shifts (08:00–15:59, 16:00–23:59, and 24:00–07:59).

The Chameleon Medical Record system uses the International Classification of Diseases—Ninth Revision (ICD-9) for assigning medical diagnosis and requires differentiation between the main diagnosis, secondary conditions, and background morbidity. For each patient, the main diagnosis was recorded and all other conditions that coexisted at hospital admission or that developed during hospital stay were considered to be secondary conditions. The main diagnosis and secondary conditions listed in the discharge notes were grouped into 13 categories, matching the ICD-9 classification with some modifications (see Supplementary file S1). Conditions that could not be grouped into any of the 13 categories, or that had less than 5% frequency, were grouped under “other diagnosis.” This grouping of diagnoses was intended to decrease misclassifications and increase the utility, as well as feasibility, of the prediction model.

From the outpatient datasets the following variables were recorded: the adjusted clinical groups (ACG) score^[13] (in the year before index admission), which measures morbidity burden based on disease patterns, age and gender as a constellation of morbidities, not as individual diseases. The ACG system automatically collapses the full set of ACG categories into 6 simplified morbidity categories (nonusers, healthy-users; and low, moderate, high, and very high morbidity). For each ACG, a relative weight was determined, which is the ratio of the mean ambulatory cost for each ACG to the mean ambulatory cost for the entire population. In addition, the number and type of specific chronic comorbidities was noted; CHF, diabetes mellitus, COPD, hypertension, chronic renal failure, cerebrovascular disease (CVD), and active malignancy excluding localized basal or squamous cell skin cancer and cervical carcinoma in situ; and the number and type of chronic medications: diuretics, β blockers, calcium channel blockers (CCBs), statins, antiplatelet therapy, antithrombotic therapy, angiotensin-converting enzyme inhibitors, angiotensin II receptor blockers, and proton pump inhibitors.

Patients were excluded from the analysis according to the following criteria: short admissions (<24 hours), admissions classified under symptoms, signs, and ill-defined conditions (ICD-9 codes 780–799), and admissions classified under “observation for” (ICD-9 codes V71, V71.2, V29.0, V29.1, V29.2, V29.8, and V29.9).

2.1. Statistical analysis

We used the Chi-square test (or Fisher exact test) to investigate the association between categorical variables and death. Continuous variables were examined using the Student t test (or Wilcoxon 2 sample test). The effect of the independent risk factors on the odds of death was estimated by implementing multivariate stepwise logistic regression. The stepwise algorithm was used after looking for confounders and interactions in the stratified analyses. The threshold probability for entering variables into the model was P < .10. Removal threshold was P > .05. The model accuracy and goodness of fit were estimated using receiver operating characteristic (ROC) and area under the curve (AUC), and the Hosmer and Lemeshow Goodness-of-Fit Test.

To explore the impact of main and secondary conditions on the accuracy of the model, 3 multivariate models were compared: without main or secondary diagnoses (model A); with main diagnoses only (model B), with main and secondary conditions (model C). The models were compared using the AUC as well as the integrated discrimination improvement (IDI),^[14] and the net reclassification index (NRI).^[15] The application of NRI and IDI is intended to assess the added discrimination offered by the addition of a marker to a prediction model. The NRI uses risk categories to quantify the reclassification improvement of the new model over the other. The IDI assesses the improvement in sensitivity without sacrificing average specificity. The model was validated using the validation dataset; for each patient a predicted probability for in-hospital mortality was obtained, using the final model coefficients. To assess the accuracy of the prediction model, ROC curve was constructed and AUC was calculated. The data management and statistical analyses were performed using the SAS (version 9.4) software.

The study was approved by the hospital's institutional review board (IRB; approval number: EMC-14-0109).

3. Results

In 2013 to 2014, 12,499 patients were admitted to the 5 participating internal medicine wards, resulting in 21,794 admissions. Readmission episodes ranged from 2 to 33. The in-hospital mortality rate was 7.1% (n  =  882). After excluding patients who did not meet study inclusion criteria (n  =  4723), and patients with missing discharge diagnoses (n  =  508), the 7268 eligible patients who remained comprised the derivation dataset (Table 1). Altogether, nearly 80% of the main diagnoses fit the 13 categories, and more than 93% of the diagnoses of patients who died fit these categories. The most frequent main diagnoses were pneumonia, heart failure, and CVD (Table 2). The most common main diagnoses among the deceased in decreasing frequency were pneumonia, sepsis and septicemia, malignant neoplasms, and heart failure (Table 2).

Table 1.

Clinical and epidemiological characteristics according to survival, together with crude odds ratios for mortality of each covariate (derivation dataset, n  =  7268).

Table 2.

Main diagnoses from hospital discharge notes according to survival, classified into 13 categories, with crude odds ratio for mortality.

Univariate analysis based on the derivation dataset resulted in 22 variables that were considered candidates for the logistic regression (Table 1), with 8 remaining in the final model (Table 3). These included age, BMI, MAP on admission, previous admission within 3 months before the index admission, background diagnoses of heart failure and active malignancy, and chronic use of statins and antiplatelet agents. In addition, 6 secondary conditions entered the final model; these included heart failure, pneumonia, sepsis and septicemia, renal failure, malignant neoplasm, and acute coronary syndrome (Table 3, and Supplementary file-S2). The analysis was executed again without patients with missing BMI values (n  =  939), no significant differences were found from the original cohort (data not shown).

Table 3.

Multivariate stepwise logistic regression to assess the contribution of variables to mortality.

A prediction model was created based on estimates from the final model. The c-statistic (ROC-AUC) of the prediction model was 80.5% without adjustment for main or secondary conditions, 84.5%, with adjustment for the main diagnosis, and 89.5% with adjustment for the main diagnosis and secondary conditions (Fig. 1). Similarly, both IDI and NRI measures showed statistically significant discrimination ability for the addition of secondary conditions (Table 4).

Figure 1.

Receiver operating characteristic (ROC) curves and relevant area under curve (AUC) applied for 3 possible models using standardized weighting coefficients.

Table 4.

Summary statistics comparing the discrimination ability of the different risk prediction models.

The validation dataset consisted of 7843 patients with 11,508 admissions; 6323 patients were admitted only once. Readmission episodes ranged from 2 to 13. In-hospital mortality rate was 7.4% (n  =  582). For each patient, the predictive probability for in-hospital death was calculated according to the established model. The accuracy of the developed predictive model reached 81%.

4. Discussion

The present study showed that a simple clinically based model can reasonably predict the risk of in-hospital mortality of acutely admitted medical patients. Our objective was to integrate secondary conditions coexistent at hospital admission or developing during hospital stay and to examine their impact on the prediction accuracy of in-hospital mortality. The addition of secondary conditions increased the c-statistic of the model form 84.5% to 89.5%. The derived model includes 8 variables, 13 possible main diagnoses, and 6 secondary conditions. Testing the accuracy of the prediction model on a separate cohort revealed a c-statistic of 81%.

Our prediction model would be easy for clinicians to use as it relies on basic variables that include age, BMI, MAP on admission, history of prior admission (3 months), chronic morbidity (heart failure and malignancy), and the use of certain chronic medications (statins and antiplatelet agents). Previous publications have described models for predicting in-hospital mortality with clinical and laboratory variables using large datasets in the United States.^[16–19] However, these studies did not evaluate the role of additional clinical data, such as vital signs, in predicting mortality.^[16,17] Some targeted specific patient populations (male patients admitted to intensive care units),^[17] or limited their analysis to 6 common clinical conditions.^[18] A recent study described an automated disease-specific risk adjustment system using clinical data,^[19] while incorporating a wide spectrum of clinical conditions, 2 dozen numerical laboratory tests, and administrative data. The average c-statistic for the automated clinical models was 0.83. Nevertheless, none of these studies adjusted their analyses for secondary conditions. The c-statistic of our model was 89.5%, indicating an excellent correlation with in-hospital mortality, and an excellent predictive accuracy of 81%. Compared with other predictive models for in-hospital mortality, our simple model provides better, or at least comparable, predictive accuracy.^[16,18–20]

A study published 3 decades ago estimated that nearly 17% of patients suffer from new complications during hospitalization.^[21] A more recent study reported that in-hospital medical complications developed among 25% of patients admitted with stroke.^[22] A limitation of the present study is that we could not determine whether the secondary conditions listed in the computerized discharge notes were complications that developed during hospital stay or were present at hospital admission. Several recent studies evaluated rates of specific medical complications among patients admitted with some common medical conditions. For example, rates of cardiac events among patients admitted with community acquired pneumonia ranged between 8% and 19%.^[23,24] Likewise, rates of cardiac dysrhythmias among patients with sepsis have been estimated to range between 8% and 46%.^[25] Urinary tract infection (UTI) and pneumonia developed in 15% and 9%, respectively, of patients with stroke, according to a Danish study^[22]; and cardiac arrest developed in 3.9% of patients with acute ischemic stroke in a recently published study.^[26]

Heart failure was the most common diagnoses among the survivors of the present study, in both the derivation and validation cohorts, but less so among the deceased. This was also the most common diagnosis among readmitted patients. Heart failure diagnosis entered the final model in 2 different categories, as a background chronic morbid condition, and as a secondary condition. However, heart failure as the main diagnosis was not associated with increased probability of in-hospital mortality. This confusing observation could imply that admission criteria for patients with heart failure in our institution may not have been properly implemented and that some of these patients could have been treated in an outpatient setting.

Data from the derivation and validation datasets imply that prior use of statins, and antiplatelet agents are associated with lower rates of in-hospital mortality. These observations may reflect healthier user effect, yet are to be interpreted cautiously. Further investigations are needed to extend these observations to additional patients. Nevertheless, recent studies suggested that prior use of statins are associated with lower in-hospital mortality among patients admitted with acute ischemic stroke,^[27] intracerebral hemorrhage,^[28] and sepsis.^[29] In addition, the use of statins has been suggested to be associated with declining in-hospital mortality from acute myocardial infarction^[30] and heart failure,^[28] and a modest reduction in pneumonia mortality in wards outside of intensive care.^[29] Finally, a recent systematic review and meta-analysis of observational studies that evaluated the effect of statins use on mortality in cancer patients concluded that statins use may be beneficial for overall survival and cancer-specific survival.^[31] Similarly, prior use of antiplatelet agents has been shown to be associated with lower in-hospital mortality in critically ill^[32] and septic patients.^[33]

Risk prediction using data from electronic health records (EHRs) has become popular in the past 20 years with the increased availability of EHRs in hospitals and other healthcare providers.^[34,35] Clinical research using EHRs is typically carried out using either association analysis^[36] or prediction analysis,^[37] while combining both, as implemented in the present study, is uncommon. Our design was impacted by our goal to provide a proof of concept that there is an added value of incorporating secondary conditions while predicting probabilities of in-hospital mortality. The selection of covariates in present study was carried out using multivariate stepwise logistic regression, this method, though criticized in some publications^[38–41] and expert opinion^[42] due to possible bias, model over-fitting, and lack of generalizability, was the most popular method of selecting covariates in epidemiological studies published in 2008.^[42] Our decision to use multivariate stepwise logistic regression was driven by its simplicity, ease of use, reproducibility, and to make our model accessible and understood to the readers. In this regard, logistic regression with regularization is considered the preferred method of statistical analysis in epidemiological studies.^[42–45] Regularized logistic regression has several advantages over standard logistic regression. First, it helps prevent the model from over-fitting the data, second, it makes the first step of analysis (association tests) unnecessary since it allows automatic selection of the most informative covariates, and last, it has better model generalizability. One technique of regularized logistic regression is the least absolute shrinkage and selection operator (LASSO),^[46] also named “shrinkage with selection.” This technique corrects the extremes in the distribution of all variables and thus shrinks very unstable estimates toward zero. This effectively excludes some variables without the need for formal statistical testing. Despite its powerful and important features, LASSO and similar methods, were not applied in any study of 171 selected articles from 4 leading epidemiological journals in 2008.^[42] Perhaps there are several reasons for that. One may be that implementing stepwise methods is much simpler than the modern techniques (LASSO and other shrinkage models). Another is the lack of familiarity of medical researchers with these methods, and third, it is possible that statisticians may have not adequately promoted and addressed the method's feasibility.^[42]

Our study has a number of limitations; first, the retrospective study design confers limitations, including the potential for misclassification and incomplete data. Second, the lack of accuracy and uniformity in hospital discharge notes may have impacted data analysis as main diagnoses may have been mistakenly classified as secondary and vice versa. However, this misclassification is nondifferential and is expected to bias our results toward the null. The high proportion of diagnoses that fits the 13 established categories, and the particularly high proportion that fits the categories of those who died, indicate good, though not complete, categorization of the diagnoses recorded on discharge notes. Third, it could be argued that the utility of a prediction model based on discharge diagnoses may not be ideal for patients at the start of their admission, due to the discrepancy between admission and discharge diagnoses or due to the occurrence of discharge diagnoses toward the end of a patient's admission. We believe that the impact of such bias is limited. In support of our view, a study of adults admitted to general internal medicine wards of a large medical center showed that the 10 most common admitting diagnoses that did not match the principal discharge diagnosis were classified as ill-defined conditions (ICD-9 codes 780–799)^[47]; all these conditions were excluded from the current analysis. Fourth, our cohort lacked data concerning smoking and alcohol drinking habits which could influence the predictive probability of in-hospital death. Fifth, the generalizability of the proposed model could be limited for other populations as the stepwise algorithm may produce irreproducible estimates.^[42] And last, our study is from a single-institution which could limit generalizability of our findings.

Despite these limitations, our study has several strengths. First, this is a first-ever study to provide a clinical model for predicting in-hospital mortality of unselected acute medical admissions while incorporating secondary conditions. Second, the model requires simple and readily available measures. Third, it performs equally well to other more complicated models. Further improvement of the model's performance and validation in other larger cohorts are needed to aid hospitalists in predicting health outcomes.

5. Conclusion

A prediction model based on clinical and epidemiological data with adjustment for secondary conditions exhibited 81% prediction accuracy of in-hospital mortality among unselected acute medical admissions. We provide a proof of concept that there is an added value for incorporating secondary conditions while predicting probabilities of in-hospital mortality. Further improvement of the model performance and validation in other cohorts are needed to aid hospitalists in predicting health outcomes. The calculator for predicting in-hospital mortality is available in the Supplementary file (S2).