Using Machine Learning Methods to Predict In-hospital Mortality through the Elixhauser Index: a Medicare Data Analysis

Abstract

Accurate in-hospital mortality prediction can reflect the prognosis of patients, help guide allocation of clinical resources, and help clinicians make the right care decisions. There are limitations to using traditional logistic regression models when assessing the model performance of comorbidity measures to predict in-hospital mortality. Meanwhile, the use of novel machine learning methods is growing rapidly. In 2021, the Agency for Healthcare Research and Quality published new guidelines for using the Present-on-Admission (POA) indicator from the International Classification of Diseases, Tenth Revision, for coding comorbidities to predict in-hospital mortality from the Elixhauser’s comorbidity measurement method. We compared the model performance of logistic regression, elastic net model, and artificial neural network (ANN) to predict in-hospital mortality from the Elixhauser’s measures under the updated POA guidelines. In this retrospective analysis, 1,810,106 adult Medicare inpatient admissions from six U.S. states admitted after Sep 23rd, 2017 and discharged before April 11th, 2019 were extracted from the Centers for Medicare and Medicaid Services data warehouse. The POA indicator was used to distinguish pre-existing comorbidities from complications that occurred during the hospitalization. All models performed well (C-statistics >.77). Elastic net method generated a parsimonious model, in which there were five fewer comorbidities selected to predict in-hospital mortality with similar predictive power compared to the logistic regression model. ANN had the highest C-statistics compared to the other two models (0.800 versus 0.791 and 0.791). Elastic net model and AAN can be applied successfully to predict in-hospital mortality.

1 |. INTRODUCTION

To describe and compare various patient populations and the overall quality of health care services, it is necessary to measure and, more importantly, predict in-hospital mortality risk. Accurate prediction of in-hospital mortality has critical implications for practice, policy, and research. In clinical practice, accurate prediction of in-hospital mortality can appropriately reflect the severity of disease or the prognosis of patients, thereby aiding in the reasonable allocation of clinical resources and ultimately helping clinicians make the right decisions (Liu et al., 2019). From a policy standpoint, value-based payment models are determined by quality measures, and in-hospital mortality prediction is critical to conduct and fairly interpret the benchmarking of organizations and provider performance (Goldman et al., 2015). In conducting research on the mortality of inpatient populations, there are challenges posed by confounding factors generated by comorbid conditions, which can significantly increase mortality (Charlson, et al., 1987). Therefore, the ability to better predict in-hospital mortality helps researchers create study designs that limit such confounding.

While both the Charlson Comorbidity Index and Elixhauser Comorbidity Index are widely used tools to estimate mortality risk, substantially higher predictive power has been reported from the Elixhauser Comorbidity Index (Liu et al., 2019; Quan et al., 2005; Sharma et al., 2021). The original version of the Elixhauser Comorbidity Index, which included 30 binary diagnoses, was created in 1998 to predict hospital resource use and in-hospital mortality (Elixhauseret al., 1998). When assessing predictive power (i.e., the ability to generate a testable prediction) and selecting significant comorbidities from a comorbidity measurement tool such as the Elixhauser Comorbidity Index to predict in-hospital mortality, researchers have typically relied on traditional logistic regression models (e.g., Quan et al., 2011; Southern et al., 2004; Thompson et al., 2015). Yet, there are several limitations to using traditional logistic regression with Elixhauser’s approach including highly correlated comorbidities (van Walraven et al., 2009). Correlation between independent variables in traditional multiple regression modelling can have far-reaching impacts on its results, such as interpretation, accuracy, and limited adaptability due to model overfitting, leading to less reliable findings and a lack of generalizability of results (Belsley et al., 2005). When the degree of collinearity increases, the traditional multiple regression becomes more unstable and its regression coefficients become more sensitive to small changes in the data; thus, the interpretation of its regression coefficient becomes more challenging since variations in one variable are associated with dramatic shifts in another variable (Fox, 2015). In addition, with infinitely large sample sizes being examined in research in the new era of “big data”, significant differences between any variables can be produced using traditional statistical significance tests, stressing the importance for the development of new and alternative methods to measure in-patient mortality using the Elixhauser Comorbidity Index.

1.1 |. Machine Learning Models to Improve In-Hospital Mortality Prediction

The advancement of machine learning, a type of artificial intelligence in which systems can learn from data, identify patterns, and make decisions with minimal human intervention (Mitchell, 1997), presents alternative options to model building in this era of big data. Machine learning techniques can be used to automate model building to improve both the efficiency in obtaining research results and the overall accuracy and generalizability of findings, and its application in healthcare research is growing rapidly. There are numerous benefits to the use of machine learning models with Elixhauer’s Comorbidity Index to predict in-patient mortality.

1.1.1 |. Elastic Net Method

The elastic net method is a type of machine learning that uses a nuanced approach to variable selection with both the LASSO and ridge regularization methods to help generalize models with highly complex relationships (Zou & Hastie, 2005). Instead of selecting variables based on absolute statistical significance cutoffs (as is done in logistic regression), LASSO regularization offers a different approach by selecting variables through soft thresholding, while ridge regularization is used to handle correlated variables in models. In short, elastic net models are able to select a reduced set of correlated covariates for use in a model with competitive prediction accuracy compared to traditional logistic regression (Tibshirani, 1996; Zou & Hastie, 2005). Overly complex models can use too many covariates and lead to issues of models overfitted to unique datasets, providing poor prediction with new data. There are at least five benefits of effective parsimonious models: fewer data requirements, reduced computational complexity, and improved system representation, transparency, and insightfulness (Daganzo, et al., 2012). Thus, a simpler model with fewer parameters is favored over more complex models with more parameters, i.e., parsimony, which is critical in model development for data adaptability (Vandekerckhove, 2015).

1.1.2 |. Artificial Neural Network

The artificial neural network (ANN) is another machine learning approach which is particularly useful with complex datasets since it naturally includes both linear and nonlinear relationships (Ripley, 1996). Another advantage of ANN is that it does not impose any restrictions on the input and residual distributions, whereas the use of logistic regressions including elastic nets are constrained with normal distribution assumptions of the residuals. Other advantages of ANN include its ability to model and extract unseen features; it can also better model heteroskedasticity. Due to this broad applicability, multiple studies for predicting adverse patient outcomes (e.g., hospital acquired infection, longer length of stay, patients experiencing trauma) have demonstrated promising results showing that ANN yields superior model performance when compared to logistic regression models (Lisboa, 2002; Tsai et al., 2016; Zachariah et al., 2020; Hassanipour, et al., 2019).

1.2 |. Present on Admission Indicator in Predicting In-Patient Mortality

Regardless of which analytic approach is used to predict in-patient mortality, incorporating the timing of a comorbid diagnosis into mortality risk is critical. Since October 2007, the Centers for Medicare & Medicaid Services (CMS) has mandated the use of the Present on Admission (POA) indicator to signify whether a patient had the condition at the time of admission for all Medicare inpatient acute care claims (CMS, 2019). The POA indicator is critical as it helps distinguish comorbidities that were present upon admission from complications of care (e.g., hospital-acquired renal insufficiency) that occurred during the hospitalization. Thus, it provides a more accurate reflection of hospital performance or patients’ initial severity of illness by accounting for how sick a patient is upon admission (Goldman, et al., 2015). In 2021, the Agency for Healthcare Research and Quality (AHRQ) published new guidelines for the use of the POA indicator to distinguish pre-existing conditions from in-hospital complications when coding Elixhauser comorbidities from the International Classifications of Diseases, 10^th Revision, Clinical Modification (ICD-10-CM) administrative data to predict in-hospital mortality (AHRQ, 2022). Under these guidelines, while the POA indicator is required for diagnoses such as neurological disorders and congestive heart failure, it is not needed for certain diagnoses such as diabetes, cancer, and AIDS which can be assumed to be pre-existing and not the result of hospital care. Prior to AHRQ’s publication of its new guidelines, previous studies either applied the POA indicator to each comorbid diagnosis which led to under-estimating the prevalence of a comorbidity (Liu et al., 2019; van Walraven et al., 2009) or did not use the POA indicator at all, causing an over-estimating of the prevalence of a comorbidity (Thompson et al., 2015). Along with the evolution of disease prevalence, treatment, and management over the years, this update to the POA guidelines highlights the need to re-evaluate the performance of the Elixhauser method to predict in-hospital mortality.

1.3 |. Study Aims

To date, little is known about the use of novel machine learning models to predict in-hospital mortality from the Elixhauser comorbidity measurement method, particularly under the newest POA guidelines. To fill this gap and address limitations of the traditional logistic regression model, the aims of this study were to (1) explore the suitability of two different machine learning models, the elastic net model and the ANN approach, to predict in-hospital mortality using measures derived from the Elixhauser method coded from ICD-10 administrative data and (2) compare their performance (i.e., parsimony, discrimination ability, and calibration assessment) to the logistic regression model. Importantly, to distinguish our approach and fill a current gap in the literature, we tested the models using the POA indicator based on the 2021 AHRQ guidelines.

2 |. METHODS

2.1 |. Data Source and Variables

We conducted a secondary analysis of data from CMS, which were obtained as part of another study (for published protocol, see [Harrison, Germack, Poghosyan, D’Aunno, & Martsolf, 2021]). In summary, our sample was limited to Medicare beneficiaries receiving care in primary care practices with at least one nurse practitioner (NP) between 2017 and 2019. Six U.S. states with varying NP scope of practice (SOP) regulations were included in the study (Arizona and Washington [full SOP], New Jersey and Pennsylvania [reduced SOP], and California and Texas [restricted SOP]).

We included data on all hospitalizations (n=1,810,106) for adult (≥ 18 years) Medicare beneficiaries hospitalized after September 23, 2017 and discharged before April 11, 2019. Medicare is available for patients aged 65+ and younger patients with disabilities or with end stage renal disease (HHS, 2021). Beneficiaries’ demographic information was extracted from the CMS Resdac master beneficiary summary file. Part B inpatient claims data provided information on patients’ admission source (i.e., emergency, elective, urgent, trauma, etc.), admission date, discharge date, discharge status (i.e., in-hospital mortality or alive), and ICD-10-CM codes (inclusive of primary and secondary diagnoses) (ResDac, 2021a). The outcome variable was in-hospital mortality extracted from discharge status information.

2.2 |. Elixhauser Comorbidity Measures

Consistent with the respective methodologies, 31 binary diagnoses were coded for the current version of the Elixhauser method based on Quan et al.’s widely used ICD-10-CM coding algorithm (Quan et al., 2005). There are appealing advantages to using composite scores rather than multiple binary comorbidities, such as being able to provide an overall description of the population, reducing the risk of overfitting in small datasets, and lessening computational requirements in large datasets (Austin et al., 2015; van Walraven et al., 2009). Therefore, we also created the VW Elixhauser composite score, one of the most widely used composite scores of the Elixhauser method, as a weighted composite score of multiple binary diagnoses ranging from −19 to 89 (van Walraven et al., 2009). In addition, we assessed whether including additional covariates such as age, race, and admission source to the multiple binary comorbidities would improve the predictive power.

To ensure the face-validity of the Elixhauser Comorbidity Index in predicting in-hospital mortality, only comorbidities identified by the POA indicator were used to calculate the Elixhauser comorbidity measures to distinguish POA comorbidities from complications that occurred during the hospitalization. The POA indicator was collected from hospital discharge data for both primary and secondary ICD-10-CM codes (Resdac, 2021b). The CMS and the National Center for Health Statistics have published guidelines for reporting the POA indicator in the ICD-10-CM coding (CMS, 2019). POA is defined as present at the time of inpatient admission, including both principal and secondary diagnoses. There are four reporting definitions in POA: Y = present at the time of inpatient admission, N = not present at the time of inpatient admission, U = documentation is insufficient to determine if condition was POA, and W = provider is unable to clinically determine whether the condition was POA or not. In this study, category Y was coded as POA, and all other categories were coded as non-POA. Traditionally, the POA indicator is applied to each of the 31 comorbid diagnoses, and only diagnoses with a “Y” value in the indicator are classified as comorbidities. Per the 2021 AHRQ guidelines, the use of POA is not needed for 18 comorbid diagnoses out of the 31 Elixhauser comorbidities assessed in the study (e.g., diabetes, cancer, and HIV/AIDS) as these conditions can be assumed to be pre-existing and not the result of hospital care. However, for other diagnoses such as pulmonary circulation disorders and paralysis, the POA indicator is needed to distinguish pre-existing conditions from complications resulting from hospital care (see footnotes of Table 2 and Appendix Table 1 for the list of comorbidity diagnoses for which the POA is needed based on the AHRQ guidelines; see Appendix Table 1 for comorbidity frequency comparison among various ways of using the POA indicator).

Table 2.

Prevalence of individual comorbidities from the Elixhauser method by patient discharge status (N = 1,810,106)

	In-hospital mortality
	Survived ((N = 1,761,878)	Death in hospital (N = 48,228)	Total (N = 1,810,106)	Odds ratio of in-hospital mortality
	n (%)	n (%)	n (%)
Congestive heart failurea	516,933 (29.34)	23,542 (48.81)	540,475 (29.86)	2.297**
Peripheral vascular disease	184,158 (10.45)	6,342 (13.15)	190,500 (10.52)	1.298**
Chronic pulmonary disease	502,252 (28.51)	15,702 (32.56)	517,954 (28.61)	1.211**
Diabetes without chronic conditions	421,038 (23.90)	10,587 (21.95)	431,625 (23.85)	0.896**`
Diabetes with chronic conditions	372,632 (21.15)	12,274 (25.45)	384,906 (21.26)	1.273**
Metastatic solid tumor	69,347 (03.94)	5,309 (11.01)	74,656 (04.12)	1.1050**
AIDS/HIV	5,810 (00.33)	173 (00.36)	5,983 (00.33)	1.092
Cardiac arrhythmia	619,856 (35.18)	26,389 (54.72)	646,245 (35.70)	2.226**
Valvular Diseasea	162,354 (09.21)	5,658 (11.73)	168,012 (09.28)	1.310**
Pulmonary circulation disordersa	116,831 (06.63)	5,714 (11.85)	122,545 (06.77)	1.893**
Hypertension uncomplicated	665,232 (37.76)	10,575 (21.93)	675,807 (37.34)	0.463**
Hypertension complicated	701,253 (39.80)	25,995 (53.90)	727,248 (40.18)	1.768**
Paralysisa	50,131 (02.85)	2,208 (04.58)	52,339 (02.89)	1.638**
Other neurological disordersa	286,550 (16.26)	15,685 (32.52)	302,235 (16.70)	2.482**
Hypothyroidism	364,087 (20.66)	8,903 (18.46)	372,990 (20.61)	0.869**
Renal failurea	500,647 (28.42)	19,409 (40.24)	520,056 (28.73)	1.697**
Liver Diseasea	94,503 (05.36)	5,754 (11.93)	100,257 (05.54)	2.390**
Peptic ulcer disease excluding bleedinga	16,697 (00.95)	365 (00.76)	17,062 (00.94)	0.797**
Lymphoma	29,644 (01.68)	1,412 (02.93)	31,056 (01.72)	1.763**
Solid tumor without metastasis	131,815 (07.48)	7,203 (14.94)	139,018 (07.68)	2.172**
Rheumatoid arthritis/collagen vascular diseases	84,297 (04.78)	1,996 (04.14)	86,293 (04.77)	0.859**
Coagulopathya	124,266 (07.05)	8,436 (17.49)	132,702 (07.33)	2.794**
Obesity	273,489 (15.52)	4,945 (10.25)	278,434 (15.38)	0.622**
Weight lossa	177,760 (10.09)	13,392 (27.77)	191,152 (10.56)	3.426**
Fluid and electrolyte disorders	672,519 (38.17)	33,215 (68.87)	705,734 (38.99)	3.584**
Blood loss anemiaa	20,107 (01.14)	558 (01.16)	20,665 (01.14)	1.014
Deficiency anemiaa	84,405 (04.79)	2,141 (04.44)	86,546 (04.78)	0.923**
Alcohol abuse	75,054 (04.26)	1,768 (03.67)	76,822 (04.24)	0.855**
Drug abuse	79,960 (04.54)	910 (01.89)	80,870 (04.47)	0.405**
Psychosesa	77,619 (04.41)	876 (01.82)	78,495 (04.34)	0.402**
Depression	305,786 (17.36)	5,119 (10.61)	310,905 (17.18)	0.565**

^aThe present on admission (POA) indicator accompanied with the diagnosis was used to identify the comorbidity based on the 2021 guidelines from Agency for Healthcare Research and Quality. For all other comorbidity diagnoses, the non-POA diagnoses are assumed to be pre-existing and not the result of hospital care;

^**p-values of unadjusted odds ratio were lower than 0.01 from bivariate logistic regression models.

2.3 |. Data Analyses

Characteristics of patient admissions using descriptive statistics are presented in Table 1. We compared the prevalence of the 31 individual Elixhauser comorbidities by patient hospital discharge status (i.e., death in hospital versus survival) and calculated the unadjusted odds ratio (UOR) of in-hospital mortality for each comorbidity using bivariate logistic regression. We then compared the associations of individual comorbidities with in-hospital mortality using a logistic regression model versus an elastic net model. Adjusted odds ratios (AORs) and associated 95% confidence intervals (CIs) were reported to assess the strength and direction of associations in logistic regression. However, researchers generally agree that traditional p-values and significance tests are not directly available in regularized regression models, including elastic net models, and reporting traditional significance testing in regularized regression models is typically not recommended (Lockhart et al., 2014). Instead, elastic net models choose variables based on soft thresholding in which variables that have substantial associations with the outcome are selected by forcing the sum of the absolute value of the regression coefficients to be less than a fixed value, which forces certain coefficients to zero, excluding them from impacting prediction. ANN is a black-box method, and we do not know the associations of individual predictors on the outcome; thus, no regression coefficients or ORs from ANN were reported.

Table 1.

Sample characteristics (N = 1,810,106 patient admissions)

Characteristics	Value
Female sex – no. (%)	964,201 (53.3)
Race – no. (%)
American Indian/Alaska Native	16,118 (0.89)
Asian	49,024 (2.71)
Black	185,326 (10.24)
Hispanic	204,493 (11.3)
Non-Hispanic White	1,328,167(73.38)
Other/Unknown	26,978 (1.49)
Age—mean years (SD)	72.89 (13.75)
Hospital length of stay, median (Q1-Q3)	5 (4-9)
VW Elixhauser composite score, mean (SD)	10.57 (9.51)
Admission source-no. (%)
Emergency	1,137,523 (62.84)
Elective	361,753 (19.99)
Urgent	281,829 (15.57)
Trauma Center	26,479 (1.46)
All other	2,522 (0.14)
State—no. (%)
Arizona	117,848 (6.51)
California	357,446 (19.75)
New Jersey	235,270 (13.00)
Pennsylvania	320,131 (17.69)
Texas	520,105 (28.73)
Washington	138,878 (7.67)
All other*	120,428 (6.65)
Patient Outcomes—no. (%)
In-hospital mortality	48,228 (2.67)

Note. SD=Standard Deviation.

^*Some beneficiaries could be hospitalized in other states, although their primary care practices were located in the six survey states.

Finally, we compared the predictive power of three measures derived from the Elixhauser method (i.e., the VW Elxhauser composite score [E1], the 31 binary comorbidity indicators [E2], and the 31 binary comorbidity indicators with other covariates [E3]) to predict in-hospital mortality using three competing methods (i.e., traditional logistic regression models, elastic net models, and ANN). Prior to model development, we randomly divided the study sample into two sections with 70% (n=1,267,467 admissions) for model training and the remaining 30% for model validation (n=528,704 admissions). To avoid potential bias, after splitting data into training and validation datasets, we used a 10-fold cross-validation design through partitioning the training and validation datasets into 10 random equal sized subsamples, and the result is the average of all test results (Stone, 1974). For each model, the area under the receiver operating characteristic curve, also called the C-statistic, was obtained as model discrimination index. The C-statistic ranges from 0.5 to 1, where a 0.5 indicates that the model is no better than random chance in predicting the outcome, and a 1 indicates that the model has perfect discriminatory ability. Models are typically considered good when the C-statistic is greater than 0.7 and strong when it is greater than 0.8 (Hosmer & Lemeshow, 2000). In addition, we evaluated model calibration measurements using mean absolute error (MAE), Hosmer–Lemeshow (H-L) test, and Cox’s intercept and slope (Van Calster et al., 2019; Huang et al., 2020). Decile calibration plots were used to visualize the calibration assessment (Austin & Steyerberg, 2014).

We chose the elastic net models as the ones through changing the alpha value, which is a value representing different trade-offs between the ridge and LASSO regularizations, from 0 to 1.0 with an increment of 0.10. The shrinkage parameter that gives the most regularized model (or the best lambda) in which the cross-validated error is within one standard error of the minimum was selected for the elastic net model. The number of epochs in ANN was selected to generate convergent results, and appropriate batch size was selected to maximize the C-statistic in the validation data. All data analyses were performed using R (R Core Team, 2020). Package caret was used for the traditional logistic regression analysis (Kuhn, 2008), package glmnet for the elastic net model (Friedman et al., 2010), and package kera with tensorflow for ANN (Kalinowski, et al., 2023). SAS software version 9.4 (SAS Institute Inc, 2013) was used to generate the calibration plots.

3|. RESULTS

Characteristics of the sample and outcomes are shown in Table 1. The sample was 53.3% female and the average age at admission was 72.9 years (standard deviation [SD]: 13.8; range: 18-113), with 79.9% of patients being ≥ 65 years. In-hospital mortality in this cohort was 2.7%. The average VW Elixhauser composite score was 10.57 (SD: 9.51; range: −16-66), and the median length of stay was 5 days (interquartile range: 4-9).

3.1 |. Prevalence of individual comorbidity by patient hospital discharge status

Table 2 presents the prevalence of the 31 individual comorbidities by patient hospital discharge status (i.e., those who survived versus those who died) and the UORs of in-hospital mortality for each comorbidity. Based on results from Chi-square analysis, there were significant differences for all comorbidities between the hospital discharge statuses with the exception of HIV/AIDS and blood loss anemia. For example, the prevalence of congestive heart failure was higher in the sample that died in hospital compared to the sample who survived (48.81% vs 29.34%; UOR=2.297; p-value<0.001). Additionally, the prevalence of diabetes without chronic conditions was lower in the sample who died in the hospital compared to the sample who survived (21.95% versus 23.90%; UOR=0.896; p-value<0.001).

3.2 |. Association of individual comorbidity with in-hospital mortality from logistic regression versus elastic net models.

After initial variable selection, all other binary comorbidity indicators (p-value<0.20) except HIV/AIDS and blood loss anemia were selected into the multivariable logistic regression model and elastic net model. All remaining binary comorbidities except diabetes with chronic conditions were significant (p-value<0.05) in the multivariable logistic regression model, whereas an additional five comorbidities (i.e., diabetes without chronic conditions, valvular disease, hypertension complicated, peptic ulcer disease excluding bleeding, and rheumatoid/arthritis/collagen) were not selected as substantial factors associated with in-hospital mortality in the elastic net model (Table 3). Results from the elastic net model indicate that fifteen comorbidities were associated with higher risk of in-hospital mortality (i.e., congestive heart failure [OR=1.55], metastatic solid tumor [OR=2.05], neurological disorders other than paralysis [OR=2.03], liver disease [OR=1.72], weight loss (OR=2.14), etc.), and eight comorbidities were associated with decreased risk of in-hospital mortality (i.e., hypertension uncomplicated [OR=0.78], hypothyroidism [OR=0.93], obesity [OR=0.80], deficiency anemia [OR=0.82], alcohol abuse [OR=0.88], etc.). Directions and strength of the associations in logistic regression and elastic net model results were similar when there were significant (i.e., logistic regression) or substantial (elastic net model) associations in both models. The elastic net model chooses variables that have substantial, rather than statistically significant, regression coefficients with the outcome.

Table 3.

The association of each individual comorbidity with in-hospital mortality ^a

	Logistic regression model				Elastic net model b
	Odds Ratio	95% CI		P-value	Odds Ratio
Congestive heart failure	1.743	1.691	1.796	<.0001	1.55
Peripheral vascular disease	1.098	1.062	1.136	<.0001	1.01
Chronic pulmonary disease	1.131	1.104	1.158	<.0001	1.03
Diabetes without chronic conditions	0.961	0.935	0.988	0.005	NA
Diabetes with chronic conditions	1.023	0.993	1.055	0.135	NA
Metastatic solid tumor	2.110	2.013	2.211	<.0001	2.05
Cardiac arrhythmia	1.797	1.755	1.84	<.0001	1.70
Valvular disease	0.898	0.866	0.932	<.0001	NA
Pulmonary circulation disorders	1.294	1.247	1.342	<.0001	1.19
Hypertension uncomplicated	0.689	0.667	0.712	<.0001	0.78
Hypertension complicated	0.765	0.739	0.793	<.0001	NA
Paralysis	1.598	1.513	1.687	<.0001	1.37
Other neurological disorders	2.17	2.117	2.223	<.0001	2.03
Hypothyroidism	0.853	0.829	0.878	<.0001	0.93
Renal failure	1.186	1.15	1.224	<.0001	1.05
Liver disease	1.944	1.873	2.018	<.0001	1.72
Peptic ulcer disease excluding bleeding	0.72	0.635	0.816	<.0001	NA
Lymphoma	1.542	1.443	1.648	<.0001	1.31
Solid Tumor without metastasis	1.454	1.396	1.515	<.0001	1.38
Rheumatoid arthritis/ collagen vascular diseases	0.921	0.871	0.973	0.0035	NA
Coagulopathy	1.779	1.725	1.835	<.0001	1.72
Obesity	0.708	0.682	0.735	<.0001	0.80
Weight loss	2.174	2.118	2.232	<.0001	2.14
Fluid and electrolyte disorders	2.599	2.537	2.663	<.0001	2.44
Deficiency anemia	0.686	0.65	0.724	<.0001	0.82
Alcohol abuse	0.697	0.656	0.741	<.0001	0.88
Drug abuse	0.525	0.484	0.57	<.0001	0.72
Psychoses	0.606	0.559	0.658	<.0001	0.82
Depression	0.655	0.632	0.678	<.0001	0.73

^aModels developed from the n=1,267,467 patient admissions training dataset.

^btraditional p-values are not directly available in elastic net model; CI=Confidence Interval; NA=Not available when elastic net model did not select the comorbidity as a substantial factor for predicting in-hospital mortality.

3.3 |. Predictive power among traditional logistic regression, elastic net models, and AAN to predict in-hospital mortality

Each model predicted in-hospital mortality well: all C-statistics were higher than 0.77. C-statistics of each measure (E1, E2, and E3) were similar between the logistic regression and elastic net models. For measure E3, the C-statistic was higher (0.800) in ANN compared to results from the logistic regression model (C-statistics [0.791]) and the elastic net model (C-statistics [0.791]). There were no overlapping 95% CIs for the C-statistics between ANNs and the other two models, indicating that ANN predicted in-hospital mortality with significantly higher discrimination performance for measure E3 (Table 4). For measures E1 and E2, ANN performed equally to the other two models as evidenced by the overlapping CIs of the C-statistics. Figure 1 depicts the pattern that ANNs performed better for more complex models. Specifically, for the single composite score (Measure E1), all three models performed equally. For more complex models (Measure E3), ANN performed better than the other two models.

Table 4.

Model discrimination performances of predicting in-hospital mortality using Elixhauser measures from three different models based on the 2021 AHRQ POA guidelines^a

Measure	Traditional logistic regression		Elastic net model		Artificial neural network
	C statistics		C statistics		C statistics
	Measure	95% CI	Measure	95% CI	Measure	95% CI
E1: VW Elixhauser composite score	0.77	0.766-0.773	0.77	0.766-0.773	0.77	0.766-0.773
E2: 31 individual Elixhauser comorbidities	0.786	0.782-0.790	0.786	0.781-0.790	0.794	0.789-0.800
E3: 31 individual Elixhauser comorbidities plus other patient characteristics	0.791	0.787-0.794	0.791	0.787-0.794	0.8	0.797-0.810

^aModels were built on training data (n=1,267,467 patient admissions) and model discrimination performance results were from testing data (n=528,704 patient admissions); AHRQ=agency for healthcare research and quality; POA=present on admission; CI=confidence interval; MAE= mean absolute error.

Figure 1.

The area under the receiver operating characteristic curve of using the VW composite score, multiple binary comorbidities, and binary comorbidities with other covariates to predict in-hospital mortality using three models.

3.4 |. Calibration measures among traditional logistic regression, elastic net models, and AAN to predict in-hospital mortality

ANNs had slightly smaller MAE values compared to Logistic regression and elastic net models (Table 5). H-L tests indicate good model fit for all models (p-value >0.05). The Cox intercepts were close to 0, and slopes were close to 1.0, indicating proper fit for all models. The decile calibration plots visually assessing the agreement between predictions and observations in different deciles of the predicted values also indicate proper calibration (Figure 2) for all models, especially for low values of the predicted probabilities. For the ANN models of measures E1-E3, the models were accurate from low values of the predicted probabilities to high values of the predicted probabilities.

Table 5.

Model calibration assessment of predicting in-hospital mortality using Elixhauser measures from three different models based on the 2021 AHRQ POA guidelines^a

Model	Traditional logistic regression			Elastic net model			Artificial neural network
Measure	E1	E2	E3	E1	E2	E3	E1	E2	E3
MAE	0.050	0.050	0.050	0.050	0.050	0.050	0.048	0.048	0.047
H-L statistics	1.070	0.648	0.667	1.070	0.531	0.583	7.601	2.912	1.942
H-L p-value	0.998	0.999	0.999	0.998	0.999	0.999	0.473	0.940	0.980
Cox intercept	−0.005	−0.019	−0.031	−0.005	−0.018	−0.029	−0.003	−0.011	0.007
Cox slope	1.001	0.996	0.992	1.001	0.996	0.993	0.996	0.991	1.002

^aModels were built on training data (n=1,267,467 patient admissions), and model discrimination performance results were from testing data (n=528,704 patient admissions); E1: VW Elixhauser composite score; E2: 31 individual Elixhauser comorbidities; E3: 31 individual Elixhauser comorbidities plus other patient characteristics; MAE=Mean absolute error; H-L= Hosmer–Lemeshow.

Figure 2.

Decile calibration plots of measures E1-E3 to predict in-hospital mortality using logistic regression, elastic net models, and artificial neural networks.

4 |. DISCUSSION

To our knowledge, we conducted the first study demonstrating that, under the AHRQ’s 2021 guidelines for the use of the POA indicator to identify comorbidities, the Elixhauser method performed well in predicting in-hospital mortality using different machine learning methods (elastic net and AAN) as well as traditional logistic regression. All models performed well with good to strong discrimination performance (C-statistics: 0.77 - 0.80) and proper model calibration demonstrated by small values of MAE, non-significant H-L test results, and Cox intercept values close to 0 and slope close to 1. The elastic net method generates a parsimonious model, in which there were five fewer comorbidities selected from Measure E2 to predict in-hospital mortality with similar predictive discrimination and accuracy performance compared to the logistic regression model. For the measure E3, which includes the greatest number of input variables, ANN had the highest discrimination performance as compared to the other two models (C-statistics: 0.800 versus 0.791 and 0.791), and it was accurate from low values of the predicted probabilities to high values of the predicted probabilities.

Results from the elastic net model indicate that weight loss (OR=2.14), neurological disorders other than paralysis (OR=2.03), and metastatic solid tumor (OR=2.05) were associated with the highest increased risk of death in hospital, consistent with the findings of our logistic regression model and previous studies (van Walraven et al., 2009; Thompson et al., 2015). For comorbidities associated with a decreased risk of in-hospital mortality, Elixhauser hypothesized that the negative associations of some comorbidities with hospital death could be a result of bias in coding (Elixhauser et al., 1998). For example, in some cases, it is possible that seriously ill patients have so many diagnoses that acutely immaterial diagnoses are not coded, whereas patients in the absence of other deadly diseases are more likely to have nonthreatening conditions coded. Thus, the presence of codes for nonthreatening diseases can be an indicator for a relatively healthy patient with a low risk of hospital death. These findings highlight the limitations of using administrative databases to code patients’ diagnostic condition.

A strength of our study is that we used the elastic net model and ANN, both advanced machine learning methods. Building on the findings of previous research, this study shows that the elastic net regression is superior at generating parsimonious models with competitive predictive power as compared to traditional logistic regression (Jang, Neto, Guinney, Friend, & Margolin, 2014). Five fewer comorbidities were selected from the elastic net regression model to predict death in hospital as compared to the logistic regression model, improving the parsimony of the model. Achieving a parsimonious model is critical as excessively complex models can be overfitted to specific datasets and thus provide poor predictions when applied to other data. Our results are in alignment with the conclusions of Vandekerckhove et al., “parsimony is essential because it helps discriminate the signal from the noise, allowing better prediction and generalization to new data” (Vandekerckhove, 2015). Previous studies have demonstrated the superior performance of ANN in predicting hospital-associated infection (Zachariah et al., 2020). The model performance of ANN in our study demonstrated higher C-statistics and lower MAEs, and thus were better than the logistic regression model and elastic net model when more variables were included (Measure E3 in Tables 4). These findings are consistent with empirical evidence showing that ANN has the potential to outperform logistic analyses when handling complex data with many covariates (Tsai et al., 2016; Zachariah et al., 2020). However, ANN is a black-box method, meaning that a final model is recommended without a clear indication of what testing produced the final decisions, which may limit its application in the field.

We used the most updated AHRQ guidelines for the use of POA indicator accompanying each ICD-10-CM code to distinguish whether a code represents a comorbidity (i.e., a condition present at hospital admission) or a complication (i.e., a condition that occurs after hospital admission). The use of the POA diagnoses ensures the face-validity of the Elixhauser measures to predict in-hospital mortality. However, whether to use the POA indicator to identify comorbidities should be based on the objectives of each study. For example, in predicting long-term patient outcomes such as patient mortality inclusive of death after discharge (i.e., one-year or 60-day mortality), it is recommended to include all diagnoses, inclusive of complications developed after admission, since complications in the hospital represent comorbidities over time. However, if the study only covers a short period of time (i.e., to predict in-hospital mortality or hospital length of stay), then including complications as comorbidities is not appropriate.

In addition, it is worth noting that the results demonstrated that the predictive power of the VW composite score, which was created based on the original POA guidelines from logistic regression, was lower compared to using multiple binary comorbidities. This highlights the needs to create a newer, updated Elixhauser composite score based on the AHRQ’s 2021 POA guidelines using novel analytic methods such as elastic net model. At the same time, results from all three models (i.e., logistic regression, elastic net models, and ANN) show that adding other factors like patient demographic information and admission source into the comorbidity measure did not improve the models’ predictive power or accuracy, indicating that there is no need to add those factors into the Elixhauser comorbidity measure to predict in-hospital mortality.

Our findings have important clinical practice implications. Since hospitals are mandated to submit standardized quality measures, it is critical to achieve accurate in-hospital mortality prediction to conduct and fairly interpret the benchmarking of organizations and provider performance. The ability to predict in-hospital mortality is also important in developing a plan of care. In-hospital mortality prediction profiles generated in this study can help clinicians prioritize tasks and develop personalized care based on the individual patient risk score and needs. Healthcare managers and administrators can also use this information to allocate patient care resources in order to provide support services where they are most needed.

4.1 |. Limitations and Future Directions

The study is a secondary data analysis utilizing CMS claims data, precluding the measurement of other variables such as functional capacity and lab results, which may have significant associations with patient outcomes (Blanco et al., 2020; Inouye et al., 1998). Future research studies should include these factors when possible to critically assess both the strength of applying ANN and the predictive values for other outcomes such as hospital length of stay, readmission, or cost. Another limitation is that this study examined a non-selective, general Medicare population with 1.8 million adult inpatient admissions; therefore, we did not limit our sample to a specific group or illness, which would have improved study generalizability. However, the use of a more generalizable sample would result in greater heterogeneity of undocumented comorbidities, compounding the risk of unobservable confounders, ultimately compromising the improvements of the predictive power of the models we assessed. Another limitation is that our findings are not generalizable to pediatric populations as the diagnoses included in these indices, such as chronic pulmonary disease and congestive heart failure, are not commonly found in children. Finally, though model calibration assessment indicates proper model fit for all models, recalibration methods such as plat scaling or isotonic regression might be useful to improve the calibration in a future study to develop newer weights for the Elixhauser composite score using elastic net methods.

5 |. CONCLUSION

This study demonstrates that under the newest AHRQ guidelines for the POA indicator, the Elixhauser comorbidity measurement method performed well in predicting in-hospital mortality using the traditional logistic regression as well as machine learning methods of elastic net model and ANN. Moreover, the elastic net model is superior at variable selection as it outperformed the traditional logistic regression model in terms of model parsimony while maintaining competitive predictive power. Though we only noted slight improvements in model performance using the ANN when including all 31 individual comorbidities and other covariates into the models, ANN has the potential to have better predictive power compared to traditional logistic regression for complex data (Edwards et al., 1999; Tsai et al., 2016; Zachariah et al., 2020). Our results demonstrate that the elastic net model and AAN can be applied successfully to predict in-hospital mortality from comorbidity measures, enabling further studies using these methods to improve predictive accuracy and model adaptability in the era of big data.

Appendix Table 1.

Prevalence of binary comorbidities when ignoring the POA indicator, using traditional POA guidelines, and the AHRQ’s 2021 guidelines (N = 1,810,106 admissions)

	Prevalence of individual comorbidity
	Ignoring the POA indicator		Traditional POA guidelines		AHRQ’s 2021 POA guidelines
	N	%	N	%	N	%
Congestive heart failurea	557,882	30.8	540,475	29.9	540,475	29.9
Peripheral vascular disease	190,500	10.5	134,824	7.5	190,500	10.5
Chronic pulmonary disease	517,954	28.6	497,555	27.5	517,954	28.6
Diabetes without chronic conditions	431,625	23.9	405,397	22.4	431,625	23.9
Diabetes with chronic conditions	384,906	21.3	374,770	20.7	384,906	21.3
Metastatic solid tumor	74,656	4.1	73,637	4.1	74,656	4.1
AIDS/HIV	5,983	0.3	5,813	0.3	5,983	0.3
Cardiac arrhythmia	646,245	35.7	551,101	30.5	646,245	35.7
Valvular diseasea	211,302	11.7	168,012	9.3	168,012	9.3
Pulmonary circulation disordersa	127,422	7.0	122,545	6.8	122,545	6.8
Hypertension uncomplicated	675,807	37.3	648,206	35.8	675,807	37.3
Hypertension complicated	727,248	40.2	708,106	39.1	727,248	40.2
Paralysisa	56,927	3.1	52,339	2.9	52,339	2.9
Other neurological disordersa	335,346	18.5	302,235	16.7	302,235	16.7
Hypothyroidism	372,990	20.6	359,396	19.9	372,990	20.6
Renal failurea	541,312	29.9	520,056	28.7	520,056	28.7
Liver diseasea	108,536	6.0	100,257	5.5	100,257	5.5
Peptic ulcer disease excluding bleedinga	17,797	1.0	17,062	0.9	17,062	0.9
Lymphoma	31,056	1.7	30,436	1.7	31,056	1.7
Solid tumor without metastasis	139,018	7.7	136,405	7.5	139,018	7.7
Rheumatoid arthritis/ collagen vascular diseases	86,293	4.8	83,343	4.6	86,293	4.8
Coagulopathya	153,240	8.5	132,702	7.3	132,702	7.3
Obesity	278,434	15.4	268,733	14.9	278,434	15.4
Weight lossa	203,534	11.2	191,152	10.6	191,152	10.6
Fluid and electrolyte disorders	705,734	39.0	604,789	33.4	705,734	39.0
Blood loss anemiaa	22,542	1.3	20,665	1.1	20,665	1.1
Deficiency anemiaa	89,878	5.0	86,546	4.8	86,546	4.8
Alcohol abuse	76,822	4.2	72,951	4.0	76,822	4.2
Drug abuse	80,870	4.5	77,129	4.3	80,870	4.5
Psychosesa	86,074	4.8	78,495	4.3	78,495	4.3
Depression	310,905	17.2	290,862	16.1	310,905	17.2

Note. AHRQ= Agency for Healthcare Research and Quality; POA=present on admission.

^aThe POA indicator accompanied with the diagnosis was used to identify the comorbidity diagnoses based on the 2021 guidelines from the AHRQ. For all other comorbidity diagnoses, the non-POA diagnoses are assumed to be pre-existing and not the result of hospital care.