A Machine Learning Model for Prediction of Amputation in Diabetics

Abstract

Background:

Diabetic foot ulcer (DFU) and the resulting lower extremity amputation are associated with a poor survival prognosis. The objective of this study is to generate a model for predicting the probability of major amputation in hospitalized patients with DFU.

Methods:

The National Inpatient Sample (NIS) database from 2008 to 2014 was used to select patients with DFU, who were then further divided by major amputation status. International Classification of Diseases, Ninth Edition, Clinical Modification (ICD-9-CM) and Agency for Healthcare Research and Quality (AHRQ) comorbidity codes were used to compare patient characteristics. For the descriptive statistics, the Student t test, the χ² test, and the Spearman correlation were utilized. The five most predictive variables were identified. A decision tree model (CTREE) based on conditional inference framework algorithm and a random forest model were used to develop the algorithm.

Results:

A total of 326 853 inpatients with DFU were identified, and 5.9% underwent major amputation. The top five contributory variables (all with P < .001) were gangrene (odds ratio [OR] = 11.8, 95% confidence interval [CI] = 11.5-12.2), peripheral vascular disease (OR = 2.9, 95% CI = 2.8-3.0), weight loss (OR = 2.6, 95% CI = 2.5-2.8), systemic infection (OR = 2.5, 95% CI = 2.4-2.53), and osteomyelitis (OR = 1.7, 95% CI = 1.6-1.73). The model performance of the training data was 77.7% (76.1% sensitivity and 79.3% specificity) and of the testing data was 77.8% (76.2% sensitivity and 79.4% specificity). The model was further validated with boosting and random forest models which demonstrated similar performance and area under the curve (AUC) (0.84, 95% CI = 0.83-0.85).

Conclusion:

Utilizing machine learning methods, we have developed a clinical algorithm that predicts the risk of major lower extremity amputation for inpatients with diabetes with 77.8% accuracy.

Introduction

Diabetes and its associated complications pose a daunting and rising health burden in the United States. About 30.3 million (9.4%) of Americans were affected by the disease in 2015 with an additional 1.5 million adults diagnosed annually. ¹ The financial costs of diabetes have also been steadily increasing. From 2012 to 2017, the total estimated cost of diabetes increased from $245 billion to $327 billion with a $61 billion increase incurred in direct medical costs and a $21 billion increase in lost productivity. ² Among patients with diabetes, diabetic foot ulcer (DFU) is one of the leading causes for hospitalization and precedes about 80% to 85% of lower extremity amputations.^3,4 The lifetime incidence of developing a foot ulcer in patients with diabetes has been estimated to be between 15% and 25%.^{5 -8} Moreover, a DFU diagnosis is associated with a poor survival prognosis; the five-year mortality rate has been reported to be between 43% and 55% for patients with DFU and up to 74% for those with lower extremity amputations.^{9 -11} In addition to DFU’s considerable impact on mortality, DFU is also associated with devastating financial, emotional, and psychological burden.^{11 -14}

The profound impact that DFUs have on patients and society underscores the need to develop more effective diagnosis and treatment tools. In 2015, the Society for Vascular Surgery addressed this need by proposing lower extremity wound classification system known as the Wound, Ischemia, and foot Infection (WIfI). ¹⁵ Some studies have since demonstrated the effectiveness of this classification system at predicting wound healing in patients with DFU and amputation in patients with lower extremity ischemia.^{16 -18} However, others have shown that the WIfI classification system is not predictive of major amputation specifically in patients with DFU, a high-risk group for lower extremity amputations. ¹⁶ Furthermore, the prognostic value of the WIfI system for amputation among patients with DFU is intrinsically limited because it the system is built through expert consensus and not from clinical data.

Recent advances in machine learning techniques offer a new opportunity to generate prediction algorithms from large set of clinical data. ¹⁹ Such algorithms have already been put into use to predict a variety of clinical outcomes such as sepsis, dementia, readmission, and mortality post-chemotherapy.^{20 -23} However, the prognostic potential of machine learning techniques, such as deep learning, have not been explored regarding the inpatient population with DFU. Thus, we aim to apply deep learning techniques to analyze a large inpatient database and identify the risk factors that predict major lower extremity amputations in patients with DFU. These factors would then be incorporated into a prognostic model that could aid in the early identification of high-risk patients.

Methods

Data Collection

We utilized the Nation Inpatient Sample (NIS) of the Healthcare Cost and Utilization Project (HCUP) to select for patients who were admitted with a diagnosis of DFU between 2008 and 2014. International Classification of Diseases, Ninth Edition, Clinical Modification (ICD-9-CM) diagnosis and procedure codes, and the Agency for Healthcare Research and Quality (AHRQ) comorbidity measures were utilized to determine the patients’ comorbidities, outcomes, and demographic information. The data set was arranged into categorical, continuous, or ordinal variables. The specific codes used for each of these parameters are provided in Supplemental Table 1.

Data Cleaning

We focused on analyzing major lower extremity amputation in hospitalized adult patients with DFU. Records with key missing values were excluded from analysis. A total of 326 853 patients with DFU, age 18 years and older, who did not expire during their inpatient admission were selected as case candidates. A combination of literature review and clinician expertise generated a list of candidate variables with potential correlation to amputation along with their corresponding ICD-9 and AHRQ codes (Supplemental Table 1). The complete list of variables, with ICD-9 and AHRQ codes, is listed in Supplementary table.

Statistical Analysis

All analyses and figures were prepared using Statistical Package for the Social Sciences (SPSS) and RStudio. Specifically, multiple R packages were utilized in this study, including “foreign,” “dplyr,” “sqldf,” “party,” “partykit,” “grid,” “mvtnorm,” “modeltools,” “stats4,” “strucchange,” “randomForest,” “glmnet,” “cowplot,” and “ggplot2.” For descriptive statistics, the P value was calculated using Student t test for the difference of numerical variables; the chi-square test was used for the correlations of binary variables and also for independence of multilevel variables, with P < .05 deemed to be significant.

Calculation

The data set was randomly divided by randomly assigning 70% to training and 30% to testing data sets. As the major amputation cases are highly imbalanced within the NIS database, a decision tree model (CTREE) was performed. CTREE is an unbiased recursive partitioning method with a permutation test performed at each pruning step. The nodes are split when the P value of the Bonferroni test is less than .05. The predicted probability of each unique variable combination was then generated using this model. Lasso regression was used to select the final variables. The dependent variable was set as binomial, and the penalty parameter alpha as 1 for the generalized linear regression. Five variables were extracted after the variable selection and parameter elimination process. The top 10 most contributory factors were also selected based on multiple regression, with forward and backward selection, and performance was compared with the five-variable model. The best probability cutoff point was determined using the average of sensitivity and specificity because of the highly unbalanced nature of the data. For the testing data set, all cases were used for the evaluation of the model’s performance. The NIS data set, a stratified sample, represents about one fifth of all discharges from community hospitals, excluding rehabilitation and long-term acute care hospitals in the United States. ²⁴ In addition to the original data set, a boosted data set was also analyzed to improve the probability accuracy. The boosted data set consists of 10 unique 5× boosted data sets. The median value probability of each unique combination was assigned as the predicted probability of major amputation. Last, the same training and testing data were also analyzed using a random forest model, and we then compared the performance between the two models.

Results

We identified a total of 326 853 patients in the NIS database between 2008 and 2014 who were admitted with DFU. 5.9% of patients with DFU underwent major amputation. The majority of patients within the study were White (63.3%), followed by Black (18.9%) and Hispanic (12.8%). Medicare was the largest payer for patients with DFU accounting for 59.1% of patients, followed by Private Insurance (19.1%), Medicaid (13.7%), and self-pay (5.0%). Patients who underwent major amputations were older (mean age 64.0 ± 13.2 years old, P < .001), had a higher incidence of end-stage renal disease (21.3% vs 12.8%, P < .001), experienced a longer length of stay (mean difference 5.6 days, P < .001), and incurred greater total cost (mean difference charges $59 042, P < .001). Median household income based on zip code was also strongly negatively correlated with risk of amputation (1, P < .001) (Table 1).

Table 1.

Demographics.

Characteristics	No amputation (n = 307 509)	Major amputation (n = 19 344)	P value
Age—years (mean ± SD)	62.3 ± 14.1	64.0 ± 13.2	<.001
Women, %	111 982 (36.4)	6450 (33.4)	<.001
Race—no. (%)			<.001
White	179 036 (63.7)	9896 (56.2)
Black	51 983 (18.5)	4448 (25.3)
Hispanic	35 845 (12.8)	2437 (13.9)
Asian	3421 (1.2)	225 (1.3)
Native American	3065 (1.1)	207 (1.1)
Other	7633 (2.7)	388 (2.2)
Comorbidities—no. (%)
End-stage renal disease	39 286 (12.8)	4122 (21.3)	<.001
Smoking	72 781 (23.67)	4458 (23.05)	.049
Hypertension	229 190 (74.5)	15 254 (78.9)	<.001
Congestive heart failure	76 147 (24.8)	4917 (25.4)	.04
BMI			.017
(0,18)	1291 (3.7)	194 (9.7)
[18,25)	1710 (4.9)	194 (9.7)
[25,30)	2566 (7.3)	200 (10.0)
[30,40)	12 811 (36.4)	641 (32.1)
[40+)	16 815 (47.8)	767 (38.4)
Median household income based on zip code—no. (%)			<.001
$1-$39 999	102 473 (34.2)	7274 (38.5)
$40 000-$50 999	79 523 (26.5)	5092 (27.0)
$51 000-$65 999	67 762 (22.6)	3882 (20.6)
$66 000 or more	49 935 (16.7)	2638 (14.0)
Patient insurance—no. (%)			< .001
Medicare	180 541 (58.9)	12 452 (65.5)
Medicaid	41 979 (13.7)	2607 (13.5)
Private	59 170 (19.3)	2976 (15.4)
Not insured	15 520 (5.1)	746 (3.9)
Other	1868 (0.6)	66 (0.3)
No charge	7824 (2.6)	464 (2.4)
Hospital bed size—no. (%)
Small	44 309 (14.5)	2094 (10.9)	<.001
Medium	80 991 (26.5)	4713 (24.6)
Large	180 557 (59.0)	12 373 (64.5)
Geographic area—no. (%)			<.001
Metropolitan nonteaching	120 125 (39.3)	6874 (35.8)
Metropolitan teaching	151 091 (49.4)	10 410 (54.3)
Rural	34 641 (11.3)	1896 (9.9)
Hospital type—no. (%)			<.001
Government, nonfederal	36 499 (11.9)	2611 (13.6)
Private, not-profit	223 856 (73.2)	13 938 (72.7)
Private, invest-own	45 502 (14.9)	2631 (13.7)
Total charges—US$ (mean ± SD) a	72 325 ± 89 381	131 367 ± 145 688	<.001
Length of stay—days (mean ± SD)	7.3 ± 7.5	12.9 ± 10.5	<.001

Abbreviations: SD, standard deviation; BMI, body mass index; USD, United States Dollar.

^aCharges have been adjusted to consumer price of 2019 by the consumer price index inpatient hospital services annual rate.

Selected possible contributing factors were accumulated from previous publications and expert opinions. The individual odds ratios (ORs) were calculated for all the binary factors and presented in Table 2. Gangrene was the single most associated factor with risk of major amputation (OR = 11.8, P < .001). Interestingly, smoking (OR = 1.0, P = .05) was not found to be a very contributory risk factor in predicting major amputation from DFU.

Table 2.

Odds Ratio of Binary Variables.

Binary variables	OR	95% CI	P value
Gangrene	11.9	11.5-12.3	<.001
Septic shock	3.1	2.9-3.4	<.001
Peripheral vascular disease	3.1	3.0-3.2	<.001
Weight loss	2.6	2.5-2.8	<.001
Septicemia	2.6	2.5-2.7	<.001
Systematic infection	2.5	2.4-2.5	<.001
Anemia	1.8	1.7-1.8	<.001
Age ≥ 40	1.7	1.6-1.9	<.001
Bacteremia	1.7	1.6-1.9	<.001
Elective procedure	1.7	1.6-1.8	<.001
Osteomyelitis	1.7	1.6-1.7	<.001
Leukocytosis	1.6	1.5-1.8	<.001
Paralysis	1.6	1.5-1.8	<.001
Renal failure	1.5	1.5-1.6	<.001
Depression	1.1	1.0-1.2	<.001
Male	1.1	1.1-1.2	<.001
Congestive heart failure	1.0	1.0-1.1	.04
Hypertension	1.3	1.2-1.3	<.001
Urinary tract infection	1.0	1.0-1.1	.33
Smoking	1.0	0.9-1.0	.05

Abbreviations: OR, odds ratio; CI, confidence interval.

The top five contributing variables to the predictive model (all with P < .001) were gangrene (covariant = 2.3), osteomyelitis (covariant = 0.8), systemic infection (covariant = 0.7), peripheral vascular disease (covariant = 0.6), and weight loss (covariant = 0.5). The unboosted five-variable model’s performance of the training data was 77.7% (76.1% sensitivity and 79.3% specificity) and of the testing data was 77.8% (76.2% sensitivity and 79.4% specificity). The resulting area under the curve (AUC) of the five-variable unboosted model was 0.84 (95% confidence interval [CI] = 0.83-0.85). The results for the 10-variable model were nearly identical to the five-variable model as shown in the performance column of Table 3. Furthermore, the boosted results of each model did not significantly differ. For internal validation, analysis with random forest was also performed and yielded a similar AUC of 0.83 and performance of 77.8%.

Table 3.

Top Five and Top 10 Model Performance.

	Sensitivity (%)	Specificity (%)	Performance (%)	AUC
10 variables
Testing	76.7	79.0	77.8	0.84
Training	77.0	78.9	77.9	NA
Boosted	77.0	78.8	77.9	0.84
5 variables
Testing	76.2	79.4	77.8	0.84
Training	76.1	79.3	77.7	NA
Boosted	76.2	79.4	77.8	0.84
Random forest
Training	75.5	79.8	77.7	NA
Boosted	75.7	79.8	77.8	0.83

Abbreviations: AUC, area under the curve; NA, not applicable.

The AUC was almost the same at 0.84 for both models, boosted and nonboosted. The resulting receiver operating characteristics curve of the unboosted five-variable model is presented in Figure 1.

Figure 1.

Model performance receiver operating characteristics curve.

Receiver operating characteristic curve using the five-factor model

The decision tree model utilizing the top five most predicted factors is presented in Figure 2.

Figure 2.

Decision tree model (CTREE).

The resulting CTREE model in Figure 2 has five levels of depth, 24 inner nodes, and 25 terminal nodes. The final model is an interactive scoring algorithm that generates a percent risk of amputation and also a risk multiplier of amputation when compared with a general patient admitted with DFU. The interactive model can be accessed through the link provided: https://grenut.shinyapps.io/amputation/

Discussion

Our model is able to predict major amputation with 77.8% accuracy and has an AUC of 0.84 (95% CI = 0.83-0.85) using only five variables: gangrene, osteomyelitis, peripheral vascular disease, systemic infection, and weight loss. Gangrene was the single most important symptom associated with undergoing major lower extremity amputation. With gangrene alone, and none of the other variables, a patient has a 232.0% increased risk of having a major amputation. A recent meta-analysis on the risk of lower extremity amputations in patients with DFU also identified gangrene as the highest risk factor with an OR of 9.9. ²⁵ Moreover, three out of top five factors used in our model represent signs of infection, demonstrating that the presence of infections are powerful risk factors for amputation and provide another meaningful point of intervention for clinicians. The role of active infection in lower extremity amputation is well supported by literature.^{25 -27} Unsurprisingly, the presence of peripheral vascular disease (PVD) was associated with major amputation, which has also been identified in a multicenter observation study by Ugwu et al ²⁶ Our model also found weight loss, defined by the AHRQ as ICD-9 260-263.9, to have high prognostic value. This is notable because weight loss and the associated nutrition deficiencies are modifiable factors and offers opportunities to slow disease progression. Last, our model’s AUC of 0.84 is within the range of AUC (0.67-0.94) for published machine learning-based prediction models.^{21 -23,28}

Although the application of machine learning to clinical medicine is in its infancy, many algorithms have been developed to predict clinical outcomes such as sepsis, dementia, readmission, and mortality postchemotherapy.^{20 -23} To the best of our knowledge, this study represents the first attempt to use machine learning techniques to produce a prediction model of lower extremity amputation for patients with DFU based on clinical data from a national database. No dedicated prediction model for the risk of major lower extremity amputation of inpatients with diabetes exists. Previous wound classification systems are the closest modality clinicians have for predicting the risk of amputation. The current wound classification systems are based on expert consensus regarding symptoms and severity of atherosclerotic disease instead of robust support from clinical data. Furthermore, among the array of classification systems associated with lower extremity amputation such as the Fontaine, Rutherford, and Bolinger, only the WIfI and Graziani classification systems were developed specifically for patients with diabetes.^15,29,30 Although these classification systems categorize the severity of wounds, they have limited prognostic value. A recent meta-analysis consisting of 2669 patients with critical limb ischemia showed positive correlation between the WIfI stage and the risk an amputation after one year. However, when only patients with diabetes were evaluated, Mathioudakis et al ¹⁶ did not find WIfI to be prognostic for amputation.

Many machine learning prediction models are black boxes that do not offer an explanation for their results, which can lead unintended consequences for high-stake fields like criminal justice and healthcare. ³¹ Thus, the decision tree model, a well-known classification method, was selected for this study because of its ability to provide transparency and allows clinicians to interpret the result themselves. In addition, the decision tree allows analysis of data with a mixture of real-valued and categorical features as well as with data that may be missing some variables. Furthermore, the training of the decision tree model does not require compensating for imbalanced data samples by using techniques such as propensity score matching or oversampling. This characteristic made it ideal for the NIS database, which is an imbalanced binary data set. Thus, all the information within data set is retained without manipulations, and the resulting model is based on the overall sample probability and not a regression. Specifically, the CTREE model, a type of decision tree, was selected because it has been shown to reduce overfitting and selection bias toward covariates with many possible splits better than the traditional decision tree models (CHAID, ID3, C4.5, CART, and QUEST).^32,33 Last, Boosting and random forest analysis were performed as a measure of internal validation of the data set. Whether using 10 variables, five variables with boosting, or a random forest analysis, the AUC remained the same.

This prediction algorithm is not only clinically relevant, but it is also easily accessible. The final five-variable model offers similar accuracy as the 10-variable model but has the added benefit of using a smaller number of predictive variables. By using less variables, the concise five-variable model allows for easier use in the clinical setting and reduces the chance of input errors. In addition, we have provided access to the model on the web. Any clinician can answer the five short questions for a particular patient in seconds and immediately receive the calculated probability of a major amputation for that patient as compared with the average inpatient with a DFU. Unfiltered access to the algorithm will also allow individual clinicians to verify the predicted risk with real clinical outcomes.

During this study, we encountered some of the inherent limitations of machine learning. The quality of the database determines the quality of the subsequent analysis. We utilized the NIS database in this study because it is currently the most comprehensive publicly accessible patient database. However, the NIS database is dependent on manual reporting of patient information and diagnoses, and thus can be biased by human errors regarding ICD code entry. Furthermore, the resulting model is limited to only including factors that have a code within the NIS database. Thus, we may be excluding variables that lack a specific code but that are potentially great risk factors for lower extremity amputation, such as anatomic depth of wounds. In addition, the NIS database only contains a snapshot of a single hospital admission; therefore, it may overemphasize the contribution of acute diseases process, such as infection, over chronic illness. This lack of long-term patient follow-up data limits the sensitivity, specificity, and overall accuracy of the resulting prediction model. Last, machine learning models could suffer from overfitting when they are trained using a single database. Given our model’s high AUC of 0.84, there was a concern of overfitting as a potential problem. We attempted to mitigate this issue by using the CTREE model and by comparing the performance of randomly divided training and testing data.

Another inherent limitation of machine learning is the Big Data Paradox. This puts forth that inaccurate conclusions of power and significance can be drawn due to both the large sample size (n) of big data and the loss of probabilistic sampling. ³⁴ This is an inherent problem for all machine learning research. Dr Meng, editor of the Harvard Data Science Review, concisely summarizes this limitation as “the bigger the data, the surer we fool ourselves.” ³⁴ The recommended solution to this problem is cross-validation, which we plan to do with other databases such as the Veterans Health Administration inpatient care database and the Surgical Outcomes Collaborative. Cross-validation will also address the potential problem of overfitting that can occur with the use of a single database. Last, because the accuracy of machine learning analysis improves with the size of the training data, the low prevalence of amputation among the inpatient population limit the quality of the resulting algorithm. As only 5.9% inpatients within the NIS database underwent lower extremity amputations, it is important to continue training the algorithm on bigger data sets to reduce the risk of overfitting to a particular database and increase the quality of the predictive model.

With the adaptation of electronic medical records and natural language processing, the amount of clinical data available is increasing at an exponential rate. The quantity of data being put forth provides an exciting opportunity to further our understanding of clinical outcomes and enact interventions that greatly improve the health of our society. We have presented a novel algorithm derived from machine learning that predicts lower extremity major amputation in hospitalized patients with diabetes with 77.8% accuracy based on a national clinical database. These findings represent an important step in transforming medical decision-making from being rooted in expert consensus to grounded in clinical data. The accuracy and applicability of machine learning in medicine will only improve as the quantity and quality of clinical data grow in the coming years.

Conclusion

Diabetic foot ulcer and the resulting lower extremity amputation are associated with considerable financial, emotional, and psychological burden in addition to poor survival prognosis. Thus, it is critical to identify high-risk groups and provide early intervention to reduce rates of complications. Utilizing machine learning methods, we have developed an algorithm based on an extensive national clinical database to predict risks of significant lower extremity amputation for inpatients with diabetes. This study represents the first attempt to leverage the power of machine learning in the treatment of vascular patients and highlight its potential to improve patient care.