A Model to Predict Limb Salvage in Severe Combat-related Open Calcaneus Fractures

Abstract

Background

Open calcaneus fractures can be limb threatening and almost universally result in some measure of long-term disability. A major goal of initial management in patients with these injuries is setting appropriate expectations and discussing the likelihood of limb salvage, yet there are few tools that assist in predicting the outcome of this difficult fracture pattern.

Questions/purposes

We developed two decision support tools, an artificial neural network and a logistic regression model, based on presenting data from severe combat-related open calcaneus fractures. We then determined which model more accurately estimated the likelihood of amputation and which was better suited for clinical use.

Methods

Injury-specific data were collected from wounded active-duty service members who sustained combat-related open calcaneus fractures between 2003 and 2012. One-hundred fifty-five open calcaneus fractures met inclusion criteria. Median followup was 3.5 years (interquartile range: 1.5, 5.1 years), and amputation rate was 44%. We developed an artificial neural network designed to estimate the likelihood of amputation, using information available on presentation. For comparison, a conventional logistic regression model was developed with variables identified on univariate analysis. We determined which model more accurately estimated the likelihood of amputation using receiver operating characteristic analysis. Decision curve analysis was then performed to determine each model’s clinical utility.

Results

An artificial neural network that contained eight presenting features resulted in smaller error. The eight features that contributed to the most predictive model were American Society of Anesthesiologist grade, plantar sensation, fracture treatment before arrival, Gustilo-Anderson fracture type, Sanders fracture classification, vascular injury, male sex, and dismounted blast mechanism. The artificial neural network was 30% more accurate, with an area under the curve of 0.8 (compared to 0.65 for logistic regression). Decision curve analysis indicated the artificial neural network resulted in higher benefit across the broadest range of threshold probabilities compared to the logistic regression model and is perhaps better suited for clinical use.

Conclusions

This report demonstrates an artificial neural network was capable of accurately estimating the likelihood of amputation. Furthermore, decision curve analysis suggested the artificial neural network is better suited for clinical use than logistic regression. Once properly validated, this may provide a tool for surgeons and patients faced with combat-related open calcaneus fractures in which decisions between limb salvage and amputation remain difficult.

Level of Evidence

Level IV, prognostic study. See Instructions for Authors for a complete description of levels of evidence.

Introduction

Open calcaneus fractures are limb threatening and complex and frequently result in long-term morbidity [12, 14, 18, 23]. Reports in the civilian literature have documented high complication rates (up to 78% of patients treated for these injuries [3, 6, 7, 14, 18, 20, 24, 30]). We previously reported a series of combat-related open calcaneus fractures having an exceedingly high complication rate, with more than 42% undergoing amputations for failed limb salvage at final followup [12]. Though variables including the size of the open wound, ipsilateral forefoot fractures, and Gustilo-Anderson fracture type were independently associated with failed limb salvage and eventual amputation by logistic regression [12], this method has not been validated. As such, it is unknown whether accurate, individualized estimates of the likelihood of successful limb salvage can be derived to guide patient treatment.

The goal of treating open calcaneus fractures is to maximize function and quality of life while setting appropriate expectations for outcome [5, 18, 22, 34]. Limb salvage frequently requires multiple surgeries and entails significant perioperative morbidity, pain, high complication rates, and lengthy hospital stays [5, 12, 23, 34]. Furthermore, because functional results of “late” amputations may not be as good as primary amputations [10, 19], a method that is able to estimate the likelihood of successful limb salvage at or soon after presentation could prove extremely valuable.

Artificial neural networks are statistical programs that can be used to identify relationships in data sets that are often not evident using traditional frequentist statistics. As a machine learning technique, the artificial neural network assesses relationships between input variables or “features” to arrive at a predetermined output—in this case, eventual amputation or successful limb salvage. Artificial neural networks have a vast number of applications as bioinformatics tools and have been successfully used for estimating the likelihood of various oncologic outcomes including survival, diagnosis, and staging [2, 15, 29, 32, 35].

When developing prognostic models, emphasis is usually placed on maximizing accuracy. However, this alone does not necessarily translate into how well a model might perform in a clinical setting because traditional means of assessing accuracy such as receiver operator characteristic (ROC) analysis do not weigh the clinical consequences of a falsely positive or negative result [11, 15, 31]. Decision curve analysis [31] is therefore required to characterize consequences of wrong answers generated by the model(s) and help determine whether the model is suited for clinical use. This stipulation is particularly important to consider in the context of this study, since overtreatment of the condition would lead to amputation, which is irreversible.

We therefore developed and validated a clinical decision support tool for severe open calcaneus fractures sustained in combat. We developed two models, an artificial neural network and a logistic regression model, using information available during the initial débridements and asked the following research questions: (1) Which model more accurately estimated the likelihood of eventual amputation on ROC analysis? And (2) which model, if any, is better suited for clinical use using decision curve analysis [31]?

Patients and Methods

After institutional review board approval, we searched our institution’s electronic medical record for all patients treated for a combat-related open calcaneus fracture between March 2003 and March 2012. Information was gleaned from several sources, including the Armed Forces Health Longitudinal Technology Application, which is the electronic medical record and coding system for the Military Health System; the Joint Theater Trauma Registry, which is a database of medical treatment information for casualties injured in combat operations abroad; and the local Surgical Scheduling System, which is an electronic operative case log of all surgical procedures at our institution. All databases were queried to identify patients with open or closed calcaneus fractures (ICD-9 code 825.0, Current Procedural Terminology codes 28400, 28415, 28420, 28406, and keyword [“fracture,” AND [“foot,” OR “calcaneus”]). The collection of study subjects for this current analysis represents an expansion on a data set from our institution that has been reported previously [12].

We included patients who sustained open calcaneus fractures during combat operations and subsequently received treatment at a single military tertiary referral hospital. Patients with a calcaneus fracture that underwent amputation within the first 24 hours of injury were excluded. Thus, we designed this study to include patients who did not require immediate amputation for a grossly unsalvageable limb or to preserve life. Furthermore, all patients in this study received the option of limb salvage versus amputation and were counseled by a multidisciplinary team of trauma surgeons, peer counselors from limb salvage and amputation groups, psychiatrists, and physiatrists. To be included in the final statistical analysis, patients were required to have initially chosen a limb salvage treatment plan.

One hundred fifty-five open calcaneus fractures in 134 patients met inclusion criteria (Table 1). The large majority of these patients had sustained blast injuries (81%, or 125 of 155 fractures), and the population consisted of a correspondingly severe range of injury patterns (67%, or 104 of 155 fractures were Gustilo-Anderson Type III; 58%, or 90 of 155 fractures were Sanders Type IV fractures). The median followup was 3.5 years (interquartile range: 1.5, 5.1 years), at which time 87 patients had maintained limb salvage, yielding an amputation rate of 44%.

Table 1.

Patient information

Variable	Value
Number of patients	134
Number of open calcaneus fractures	155
Age (years)*	24 (22, 28)
Male sex (number of patients)	130 (97%)
Blast mechanism (number of fractures)	125 (80.6%)
Time to presentation in United States (days)*	5 (4, 6)
ISS (points)*	18 (16, 34)
ASA grade*	2 (1, 2)
Followup (years)*	3.5 (1.5, 5.1)
Number of procedures*	4 (2, 8)
Gustilo-Anderson Type III (number of fractures)	104 (67%)
Sanders Type IV (number of fractures)	90 (58%)
Diminished/absent plantar sensation (number of fractures)	82 (65%)
Bilateral calcaneus fractures (number of fractures)	65 (32%)
Infection (number of fractures)	56 (36%)
Amputation (number of fractures)	67 (44%)

* Values are expressed as median, with interquartile range in parentheses; ISS = Injury Severity Score; ASA = American Society of Anesthesiologists Physical Status Classification.

Abstracted data included patient demographics, mechanism of injury, wound size and location, fracture types according to Gustilo and Anderson [16, 17] and Sanders [27, 28] classifications, interval and definitive treatment procedures, adjunctive procedures (rotational or free tissue transfer, skin graft, and neurovascular procedures), and ipsilateral and contralateral orthopaedic injuries. The definitive treatment was defined as the procedure after injury that, after healing, would allow reduced fracture union and eventual weightbearing. The primary outcome recorded was need for amputation after a failed limb salvage attempt. Amputation type and reason for unsuccessful limb salvage were collected.

Descriptive statistics regarding patient’s mechanism of injury, overall severity of injury, and fracture-specific variables were collected on the study subjects. Using data points believed to impact the decision between limb salvage and amputation, an artificial neural network was developed from the patient data set to predict successful limb salvage in open calcaneus fractures. We selected 26 features with a proven or theoretical association with successful or unsuccessful limb salvage as candidates for inclusion into the model. Each was identifiable at the patient’s initial presentation to the treating tertiary care center in the continental United States (Table 2). The artificial neural network was produced using the Oncogenomics Online Artificial Neural Network Analysis system from the National Cancer Institute [1], which uses feed-forward resilient back-propagation multilayer perceptron methodology. The network first establishes connections between the features (inputs) and minimizes error through two primary steps: a forward activation to produce an estimated likelihood of an outcome (amputation or limb salvage) and a backward propagation in response to an error, if applicable [2]. This feed-forward back-propagation process is repeated over and over again during a “training” phase of artificial neural network development. In brief, the weights of the input features are adjusted to minimize the error between the output produced by the artificial neural network and the actual “real-world” result, which we defined as amputation or successful limb salvage.

Table 2.

Patient variables predicting successful limb salvage

Variable for artifical neural network development	Definition
Sex	Male/female
Rank	Military rank (officer/enlisted)
Weight (kg)
Age (years)
ISS	Injury Severity Score
ASA grade	American Society of Anesthesiologists Physical Status Classification: Grade 1–4
Gustilo-Anderson type	Open fracture severity: I–III
Sanders classification	Degree of comminution through posterior facet: 1–4
OTA classification	Location of fracture and degree of comminution
Tscherne classification	Extent of soft tissue injury: Grade 1–4
Injury mechanism	Blast, gunshot wound, motor vehicle accident, fall
Number of procedures	Total number of surgical procedures
Number of débridements	Total number of surgical débridement procedures
Number of open fractures	Total number of open fractures on presentation
ICU treatment	Intensive care unit care on admission: yes/no
Wound size (cm2)
Wound location	Medial/lateral/dorsal/plantar/combined
Plantar sensation	Intact/diminished/absent
Free flap coverage	Wound covered by free flap: yes/no
Fracture treatment before arrival in United States	External fixator/pin fixation/splint/débridement/screw fixation
Vascular injury	Injury to major vessel supplying extremity: femoral/popliteal/posterior tibial/anterior tibial/peroneal/dorsalis pedis
Tobacco use	Smoking on presentation to United States: yes/no
Mounted vs dismounted	Injury occurred while inside vehicle: yes/no
Bilateral calcaneus injury	Bilateral open fractures: yes/no
Procedures before arrival in United States	Total number of procedures before presentation to United States
War	Iraq/Afghanistan

We first performed principle component analysis on the 26 patient-specific/injury-specific features selected for model development to identify the top 10 uncorrelated features with the largest variance. This step was designed to simplify the computational analysis and avoid “overfitting” the model to the training data. The artificial neural network was composed of three layers: an input later consisting of the top 10 principle components identified above, a hidden layer with five nodes, and an output layer to yield a decision between the two possible primary outcomes (amputation or limb salvage). During the error-minimizing process, the minimum number of features was identified that resulted in the least amount of error. To establish which input features were most important for the artificial neural network’s arrival at the final outcome (amputation versus salvage), a leave-one-out process was used during which one input variable at a time is removed from the overall input variable list. The artificial neural network then determines the negative impact on network accuracy that was caused by removal of the particular variable. Intuitively, features that have a greater negative impact on network accuracy are more important for arrival at final outcome.

For comparison, we developed a conventional logistic regression model using variables identified to be potentially significant on univariate analysis. These have been reported previously and included the mechanism of injury, presence of a plantar wound, wound size, and Gustilo-Anderson fracture classification [12]. For the current analysis, the logistic regression model was repeated using the new expanded data set. We used Proc Logistic of SAS version 9.3 for the analysis.

To assess accuracy, we performed internal validation using 10-fold cross-validation. ROC curve analysis was performed in which true-positive results were plotted against false-positive results to graphically describe the relative accuracy of each model. The area under the ROC curve (AUC) serves as a metric of overall accuracy, where 0.5 represents pure chance and 1.0 represents a perfect model. Second, decision curve analysis [31] was performed in a manner previously described [15]. This analysis is designed to calculate the clinical utility of prediction models. Using decision curve analysis, the relative impact of false-positive and false-negative results produced by the prediction model is measured to yield the “net benefit” for the model. In other words, net benefit is defined as one injury duly receiving a treatment commensurate with its estimated likelihood of success. The threshold probability is the probability at which a surgeon is indecisive about which treatment to offer and is surgeon, patient, and situation dependent. When displayed graphically, the resulting curves illustrate the net benefit across all possible threshold probabilities (0–1) through weighing the relative harm of a false-positive or false-negative result to the benefit of a true-positive or true-negative result. As an additional assessment of clinical utility, the decision curve analysis curves of each model were also compared to two other theoretical scenarios: one in which every patient undergoes amputation and one in which no patient undergoes amputation, regardless of the probability of amputation.

Results

An artificial neural network model containing eight features resulted in the least amount of error. The eight features identified were determined by the artificial neural network to have the most influence on the outcome of the limb at final followup. These features in decreasing order of importance were high American Society of Anesthesiologists (ASA) grade, plantar sensation, interval fracture treatment before arrival in the United States, fracture severity according to Gustilo-Anderson and Sanders classifications, presence of a vascular injury, male sex, and dismounted injury mechanism (Table 3). Using these eight patient-specific/injury-specific features, the artificial neural network was able to accurately predict which patients had an amputation within the cohort. Stated more plainly, a lower ASA grade, intact plantar sensation, less treatment before arrival, lower Gustilo-Anderson fracture type, lower Sanders classification, absence of vascular injury, female sex, and a lack of a dismounted mechanism suggested successful salvage, while a higher ASA grade, lack of plantar sensation, complex treatment before arrival, higher Gustilo-Anderson fracture type, high Sanders classification, presence of vascular injury, male sex, and a dismounted blast mechanism favored amputation. The balance (18) of the original 26 features did not improve the accuracy of the artificial neural network and are considered irrelevant within the context of this particular model.

Table 3.

Features associated with successful limb salvage after principal component analysis, ranked from largest to smallest variance

Rank	Variable
1	ASA grade
2	Plantar sensation
3	Fracture treatment before arrival
4	Gustilo-Anderson open fracture type
5	Sanders calcaneus fracture type
6	Vascular injury
7	Male sex
8	Dismounted blast injury mechanism

ASA = American Society of Anesthesiologists Physical Status Classification.

The artificial neural network was found to be more accurate and better suited for clinical use than the logistic regression model. Both models produced validation data suitable for ROC and decision curve analysis. On internal validation, the artificial neural network model was more accurate with an AUC of 0.8 (95% CI: 0.77, 0.82), which demonstrated a sensitivity of 64%, a specificity of 91%, a positive predictive value of 86%, and a negative predictive value of 74%. The AUC for the logistic regression model was 0.65 (95% CI: 0.56, 0.74), which demonstrated a sensitivity of 55%, a specificity of 61%, a positive predictive value of 50%, and a negative predictive value of 60%. Decision curve analysis revealed that both models could be used clinically, rather than recommend all patients or no patients undergo amputation (Fig. 1). However, the artificial neural network model resulted in higher net positive benefit across the broadest range of threshold probabilities, indicating that it is likely better suited for clinical use than the logistic regression model. The higher net benefit indicates that the artificial neural network yielded a higher number of true-positive/true-negative results relative to the number of false-positive/false-negative results in this particular patient population.

Fig. 1.

Decision curves for open calcaneus fractures represent the following four scenarios: artificial neural network, logistic regression, all patients treated with amputation (all negative), and all patients treated with limb salvage (all positive). Note that the artificial neural network has the most positive net benefit when the threshold probability for amputation is between 0.10 and 0.75.

Discussion

Advances in initial medical care performed in combat and improved methods of limb reconstruction now enable the preservation of limbs previously destined for amputation [4, 23, 25, 26, 33]. Despite this, amputation rates after severe blast injury to the lower extremity remain high [12]. Conversely, salvage of open calcaneus fractures often calls for multiple operations and lengthy convalescent periods and is associated with an exceedingly high complication rate [3, 6, 7, 12, 14, 18, 20, 23, 24, 30]. A model capable of classifying the eventual fate of the limb in open calcaneus fractures would provide the surgeon the ability to focus surgical interventions, avoid unnecessary treatments, and counsel patients more appropriately earlier in the course of their treatments using objective criteria. Using information available early in the treatment of open calcaneus fractures, we developed a network that arrives at one of two possible outcomes (amputation or limb salvage) and compared it to a previously published logistic regression model [12] by asking the following questions: (1) Which model more accurately estimated the likelihood of eventual amputation? And (2) which model, if any, was better suited for clinical use?

Our study has several limitations. First, this series does not include amputations performed within 24 hours of injury, as many of these acute treatments were performed in a combat environment. As such, a complete characterization of those injuries was not possible. Furthermore, these early or immediate amputations were intentionally excluded because we chose to focus only injuries that were deemed to be, in the opinion of the treating surgeons, potentially salvageable. Amputations from open calcaneus fractures in the civilian sector often occur within the first 24 hours of presentation. This represents an important distinction between the treatment algorithms of this injury in the civilian sector versus the military combat wounded. When injury occurs in combat, every attempt is made to salvage the extremity until further assessment can be made outside of the hostile zone. Secondary to this, injuries that may be deemed unsalvageable on presentation to a civilian trauma center and amputated within 24 hours are salvaged in theater. A second limitation is that the model was derived using data from combat-injured patients treated at one institution. Similarly, patients included in this series represent a small subset of all patients with open calcaneus fractures, namely patients who sustained high-energy injuries in a combat environment, large soft tissue wounds, and damage to other organ systems. Therefore, this particular model may not apply to other centers treating combat-related or lower-energy civilian trauma, and we do not recommend extrapolating our results to other centers with differing patient populations and treatment philosophies. External validation of this model is therefore absolutely necessary before widespread clinical use. Additionally, there are other clinical factors (infection, wound size, ipsilateral fractures) that are important to consider in this setting that were not included by this machine learning technique. The absence of these features from the present model does not necessarily mean that they are unrelated to the outcomes chosen for this experiment but rather did not improve the model’s performance, given the availability of the eight features listed above. In addition, the logistic regression model used for this exercise [12] did not include potential interactions between variables, which may be a disadvantage when one considers the interdependent nature of wound-specific and systemic considerations in this patient population. Related to this, we considered bilateral fractures as independent; we recognize that there may be disagreement about the appropriateness of doing so, but we believe this is nonetheless appropriate because clinically, each open fracture is different; ie, the treatment of one does not necessarily depend on the presence and/or treatment of the other. Furthermore, each calcaneus fracture is subject to its own wound microenvironment driven largely by its vascular supply, bioburden, and zone of injury. Finally, since patients with bilateral injuries were relatively common, omitting them would limit the use of this model in the future.

Despite having no missing data, this study is also limited by a relatively small sample size (n = 155) compared to the number of features examined (26). By extension, only two statistical techniques were evaluated. It is entirely possible that other statistical techniques, such as classification and regression trees, for example, may outperform the approaches used in the present study. In addition, complete reliance on computational methods that have been optimized for a particular patient population can be problematic if overfit to the training data. As such, models may suffer from lack of generalizability when applied to other patient populations. We attempted to minimize this effect by performing principle component analysis and cross-validation; however, the importance of external validation studies cannot be overstated. It is also important to mention that models such as this one should be considered “works in progress.” During the external validation phase, it is possible that additional features may be collected that add to the accuracy/net benefit of this model. As such, prognostic tools such as the one presented in this article have the potential to improve over time as treatment methods evolve and our ability to collect population-based data improves.

Using the current series of 155 combat-related open calcaneus fractures, we designed an artificial neural network to estimate the likelihood of successful limb salvage, using patient and injury data available on initial presentation. The model was accurate, correctly classifying the fate of the extremity in the majority of cases with an AUC of 0.8 (95% CI: 0.77, 0.82) and a specificity of 0.92. Furthermore, the accuracy of the artificial neural network was found to be 30% higher than that of the previously published logistic regression model [12]. To date, artificial neural networks have been only sparingly applied to orthopaedic trauma. In 2010, a study comparing an artificial neural network to a logistic regression model found the artificial neural network to more accurately predict mortality after geriatric hip fracture [21]. More commonly, artificial neural networks have been used in the medical literature for estimating the likelihood of survival and characterization of cancer. A report investigating 5-year mortality rates in colon cancer illustrated an artificial neural network that made more accurate survival predictions than experienced oncologists [9]. The model developed in our study identified eight features that accurately predicted eventual amputation in combat-related open calcaneus fractures. These features, ASA grade, plantar sensation, treatment before arrival, Gustilo-Anderson type, Sanders type, vascular injury, male sex, and injury mechanism, all identifiable on the patients’ initial presentation to the treating tertiary care center, provide clinically useful and prognostic information on the fate of the extremity.

In addition to being more accurate than the logistic regression model, our analysis suggested that the artificial neural network was perhaps better suited for clinical use. When the probability of amputation from an open calcaneus fracture is high (approaching 1.0), the surgeon will likely recommend amputation. On the other hand, when the probability of amputation is low (approaching 0.0), the surgeon is likely to recommend limb salvage. However, there exists a large and variable middle range in which the surgeon and patient are indecisive, representing the threshold probability (p_t). Our decision curve (Fig. 1) illustrates the net benefit of the artificial neural network and logistic regression across a range of threshold probabilities. Though both models could be used, rather than assume all patients or no patients undergo amputation, the artificial neural network yielded a higher net benefit than the logistic regression model across a broad range of p_t (Fig. 1). A clinical example of this conclusion is as follows: an active-duty service member presents with an open calcaneus fracture after a combat-related blast injury. Based on the followup data in our cohort, his risk of amputation is approximately 40% given his particular injury. When deciding whether to proceed with limb salvage or to perform an amputation, the treating surgeon may choose to use a clinical decision support tool to estimate the individual’s risk of amputation. To determine which tool to use, the surgeon determines his/her p_t, which is surgeon, patient, and situation dependent. Using the decision curve, the model that results in the highest net benefit at that particular p_t, would be used to generate a patient-specific estimate of the likelihood of eventual amputation. If the p_t estimated by the surgeon was 0.5, for instance, the artificial neural network would be used. These results are similar to those previously reported in which artificial neural network clinical decision support models applied to orthopaedic outcomes outperformed models derived using frequentist and Bayesian techniques [15].

The differences in functional outcome of lower-extremity amputees compared to those patients undergoing limb salvage secondary to severe lower-extremity trauma remains a debated topic. Results from the Lower Extremity Assessment Project suggest that the outcomes between these two patient groups are similar and universally poor [8]. Conversely, the results of the Military Extremity Trauma Amputation/Limb Salvage study indicate that, within the military population, amputees function at a higher level compared to those with salvaged extremities [13]. Given this, the ability to predict which patients will receive amputation on presentation would obviate prolonged limb salvage that may not provide functional benefit. This report demonstrated an artificial neural network model, developed using the largest series of open calcaneus fractures of which we are aware, that is capable of accurately estimating the likelihood of eventual amputation. The amputation rate in this series was 42% at final followup, much higher than what has been reported in civilian literature. The high-energy mechanism, massive soft tissue damage, and additional trauma to other organ systems are the most plausible explanations for this distinction. In this unique cohort, decision curve analysis suggested that the artificial neural network was better suited for clinical use than a logistic regression model based on the same data set. However, external, prospective validation is necessary, and planned, in additional institutions with differing patient populations and treatment philosophies, before recommending this model for clinical use. If successful, the model described here may provide a useful tool for surgeons, patients, and families faced with severe open calcaneus fractures, in which decisions to embark on limb salvage or undergo amputation remain difficult.