Machine Learning Without Borders? An Adaptable Tool to Optimize Mortality Prediction in Diverse Clinical Settings

Abstract

Background:

Mortality prediction aids clinical decision-making and is necessary for quality improvement initiatives. Validated metrics rely on pre-specified variables and often require advanced diagnostics which are unfeasible in resource-constrained contexts. We hypothesize that machine learning will generate superior mortality prediction in both high-income and low and middle-income country cohorts.

Methods:

SuperLearner (SL), an ensemble machine-learning algorithm, was applied to data from three prospective trauma cohorts: a highest-activation cohort in the United States (US), a high-volume center cohort in South Africa (SA), and a multicenter registry in Cameroon. Cross-validation was used to assess model discrimination of discharge mortality by site using receiver operating characteristic curves. SL discrimination was compared with standard scoring methods. Clinical variables driving SL prediction at each site were evaluated.

Results:

Data from 28,212 injured patients were used to generate prediction. Discharge mortality was 17%, 1.3%, and 1.7% among US, SA, and Cameroonian cohorts. SL delivered superior prediction of discharge mortality in the US (AUC 94–97%) and vastly superior prediction in Cameroon (AUC 90–94%) compared to conventional scoring algorithms. It provided similar prediction to standard scores in the SA cohort (AUC 90–95%). Context-specific variables (partial thromboplastin time in the US and hospital distance in Cameroon) were prime drivers of predicted mortality in their respective cohorts, while severe brain injury predicted mortality across sites.

Conclusions:

Machine learning provides excellent discrimination of injury mortality in diverse settings. Unlike traditional scores, data-adaptive methods are well-suited to optimizing precise site-specific prediction regardless of diagnostic capabilities or dataset inclusion allowing for individualized decision-making and expanded access to quality improvement programming.

Level of Evidence:

Level III

Study Type:

Prognostic and Therapeutic

BACKGROUND

Mortality prediction after injury enables patient risk stratification, aids in clinical decision-making, and is a necessary benchmark for the development of quality improvement initiatives (1, 2). Over the past 60 years, the search for superior, widely-applicable trauma prediction has led to the development of dozens of severity algorithms. These models have incorporated a variety of different clinical variables including indicators of anatomic severity (3–5), physiologic predictors (6–8), and combinations of both anatomic and physiologic parameters (9, 10).

Despite these many approaches, no single score has been successful in generating ideal prediction over time and across diverse clinical contexts (11). While most scores are initially published with high reported predictive capacity, historically, performance has degraded over time, particularly when applied to different trauma populations. This, in turn, fuels the development of still more severity scoring systems and modifications.

One substantial problem has been the application of severity scores to trauma populations in low- and middle-income countries (LMICs). Scoring metrics developed in the US and other high-income countries routinely rely upon advanced diagnostic imaging or operative exploration for accurate anatomic assessment, making utilization unfeasible in resource-constrained contexts. Even physiologic parameters, such as vital signs, are often sparse in settings with extreme staffing and equipment limitations. Scoring systems designed to generate prediction in LMICs, such as the Kampala Trauma Score, have performed inconsistently in different datasets (12–14).

In other fields, machine learning algorithms have been utilized to generate excellent prediction of complex outcomes by using a data-driven approach to model selection with many potential predictors (15, 16). Due to the fact that these methods do not rely on a static variable set, they have the potential to actually adapt prediction to different clinical contexts (17, 18). Application of one such machine learning algorithm called SuperLearner (SL) has previously been demonstrated to improve prediction of complex outcomes in human immunodeficiency virus (15, 16) and in a single-center cohort SL has demonstrated superior discrimination of death after injury compared to conventional statistical approaches (19, 20).

However, the promise of machine learning is wide applicability of the precision medicine approach. To date, the predictive capacity of machine learning in diverse trauma datasets remains unknown. In this study, we hypothesize that machine learning, specifically so-called ensemble learning using all available admission variables will generate superior prediction of death in both high-income country and LMIC trauma cohorts. Identification of a translatable method of generating reliable prediction in diverse contexts would enable setting-specific tailored severity estimation for improved resource triage and clinical decisionmaking.

METHODS

SuperLearner (SL), an ensemble machine-learning algorithm, was applied to all admission data from three large prospective trauma cohorts: a highest-level activation urban trauma center in the United States (US), a high-volume trauma center in South Africa (SA), and three referral hospitals in Cameroon.

US COHORT

The US data utilized was extracted from prospective data collected on a severely-injured patient cohort presenting to San Francisco General Hospital between February 2005 and April 2015. Specifically, the Activation of Coagulation and Inflammation in Trauma (ACIT), which has been described in detail in other studies, was a single-center prospective cohort study which followed severely injured trauma patients from emergency department admission through hospitalization (21, 22). All adult trauma patients presenting to San Francisco General Hospital between February 2005 and April 2015 who met criteria for highest triage activation level were included into the US cohort. Exclusion criteria included patient age less than 15 years, pregnancy, incarceration, thermal mechanism, and transfer from outside hospital. Baseline data collected included patient demographics, past medical history, substance use, and injury characteristics and physiologic variables including vital signs, laboratory monitoring including an extensive panel of coagulation and inflammation markers, ventilator parameters, input/output data, and fluid, colloid, blood product, and medication administration. Outcomes data including hospital mortality were collected.

Although ACIT collected longitudinal data from admission throughout the first 28 days of hospitalization, for the purposes of this study, only admission variables were used for SL prediction. The study was carried out with the approval of the University of California San Francisco Institutional Review Board.

SOUTH AFRICAN COHORT

The South African data utilized was extracted from prospective data collected on injured patients presenting to Groote Schuur Hospital between April 2014 and November 2016. South Africa is an upper-middle-income country with high rates of injury deaths due to homicide and vehicular trauma (23). Groote Schuur Hospital is a government-funded, high-volume, tertiary teaching hospital in Cape Town, South Africa with an estimated census of 9000 trauma patients per year (24). Between April 2014 and November 2016 prospective data were collected on injured patients presenting to Groote Schuur Hospital using a clinician-entered point-of-care mobile electronic Trauma Health Record (eTHR) system. This system and cohort has been described in detail in prior publications (24). Briefly, all admitted patients older than 12 years of age with primary blunt or penetrating trauma mechanisms were included in the cohort and followed from admission through hospitalization. Patients were excluded if they were determined to be dead on arrival, presented with thermal injuries, were discharged home or transferred to a step-down facility from the emergency department, or if the patient’s discharge and/or hospital disposition was unknown. Concurrent with patient care clinicians entered data on demographic and injury characteristics, clinical parameters on admission, and patient outcomes including disposition and death.

For the purposes of this study only admission variables were used for SL prediction. Ethical approval for this study was obtained from the Human Research Ethics Committee of the University of Cape Town and Institutional Review Board of the Department of Surgery, Groote Schuur.

CAMEROONIAN COHORT

The Cameroon data utilized was extracted from the Cameroon Trauma Registry which collected prospective data on injured patients between July 2015 and January 2017. Cameroon is a primarily francophone lower-middle income country in central Africa with among the lowest life expectancies on earth. In 2015 a Cameroon Trauma Registry was established to collect prospective data on all injured patients presenting to three referral hospitals: Laquintinie Hospital of Douala, a high-volume urban hospital in Cameroon’s largest city; Limbe Regional Hospital, a regional medical center with mixed urban/rural catchment in the Anglophone Southwest Region of Cameroon; and the Catholic Hospital of Pouma, a hospital on the highly-trafficked road between Douala and Yaoundé, Cameroon’s capital. Between July 2015 and January 2017 all injured patients presenting to these three hospitals for trauma were followed from emergency admission to hospital discharge. Specifically, trained Cameroonian trauma registrars recorded patient data at the time of patient admission using paper trauma registry forms (collecting data regarding demographics, mechanism of injury, past medical and surgical history, injuries and injury severity, vital signs, radiology studies, resuscitation measures, treatments, and disposition) and subsequently on paper follow-up up forms (collecting data regarding operative interventions, laboratory and radiology studies obtained, medications and treatments administered, complications, and hospital outcomes). The paper forms were initially stored in secured lock boxes on-site at each hospital and then were subsequently entered by trauma registrars into a REDCap database hosted by the University of California San Francisco server. All patients presenting with injury mechanisms were included in the cohort. No exclusions were made on the basis of injury severity, trauma mechanism, hospital disposition, or age.

For the purposes of this analysis, only hospital indicators available at presentation were utilized for SL prediction. Ethical clearance was obtained from the University of California, San Francisco Committee on Human Research and the National Ethics Committee of the Republic of Cameroon.

SUPERLEARNER PREDICTION

SL is a previously-validated ensemble machine learning algorithm which has been described in detail in prior publications (18). SL can be downloaded as a package within the R coding language (25). Both the R software and the SL package are open-source and can be accessed without charge to the user (https://cran.r-project.org/ and https://cran.r-project.org/web/packages/SuperLearner/index.html, respectively). This means that this technology is widely accessible to clinicians and researchers in both high-income and LMIC contexts.

Rather than pre-specifying a single statistical approach, SL simultaneously investigates multiple algorithms ranging from simple logistic regression to highly complex machine learning (e.g., neural nets) in order to optimally predict outcomes of interest from complex datasets. SL uses cross-validation to tailor a weighted (convex) combination of learners to optimize prediction on new data from the same data-generating distribution. Embedded cross-validation eliminates the risk of over-fitting (18).

In this study SL was applied to all admission variables of the US, SA, and Cameroonian cohorts to generate setting- specific prediction of hospital mortality. We used a set of algorithms including: logistic and linear regression, generalized additive models with various levels of smoothing (26), random forest (27), lasso (28) and systems-based on sieves of parametric models (e.g. polyclass). To report the ability of the resulting SL fit to future data, we estimated the cross-validated area under the curve (AUC) of receiver-operator characteristic curves (ROC) as well as using cv-AUC as the objective function, so the procedure optimizes prediction towards more clinically-relevant measures of performance (in this case, a function of specificity and sensitivity). SL prediction and cross-validated risk was used to evaluate performance of each model (29).

COMPARISON OF SUPERLEARNER AND STANDARD SEVERITY SCORING METRICS

Feasibility, calculated by present missingness in each dataset, was calculated for standard severity scoring metrics, including Trauma and Injury Severity Score (TRISS) (9), Kampala Trauma Score (KTS) (8), Revised Trauma Score (RTS) (30), Glasgow Coma Scale, Age, Pressure Score (GAP) (6), Mechanism, Age, and Pressure Score (MGAP) (31), and the Glasgow Coma Score (GCS) (32).

Discrimination of hospital mortality using SL was compared with these standard scoring metrics (TRISS, KTS, RTS, GAP, MGAP, and GCS). Given differences in the baseline variables, not all scores could be calculated for each dataset. Specifically, TRISS was only calculated in the US cohort whereas KTS was only calculated in the Cameroonian cohort.

VARIABLE IMPORTANCE MEASURES

Many choices are available for assessing the relative importance of predictor variables in the overall ability of the final model to predict. One such technique is available as a byproduct of another ensemble-type machine learning algorithm called random forest. In this case an average of regression trees is used as the final predictor. Specifically, random forest variable importance is measured as the change in the out-of-sample (samples not used in training) fit of the model when each variable is randomly permuted (and thus made independent of the outcome). We report the five most important predictor variables within each study as chosen by this metric (27, 33). Top influential predictors for hospital mortality after injury were compared across clinical settings.

RESULTS:

DATA AND COHORT CHARACTERISTICS

Data from 28,212 injured patients were evaluated and used to generate prediction. (Table 1) Available admission variables differed by site, with the quantity of variables used for prediction ranging from 133 (Cameroon) to 212 (US).

Table 1:

Dataset and Trauma Cohort Characteristics

	US Cohort	SA Cohort	Cameroon Cohort
	N=1,494	N=18,821	N=7,897
Data Characteristics
Data Collection (years)	2005–2015	2014–2016	2015–2017
Medical Centers (n)	1	1	3
Admission Predictor Variables (n)	212	144	131
Observations (n)	853,074	17,416,200	2,574,422
Cohort Characteristics
Age (years, IQR)	36 (25– 52)	30 (24 −32)	29 (22–40)
Pediatric Patients (%)	0	5.1	14.2
Male Patients (%)	82.0	75.6	72.4
Blunt Mechanism (%)	57.0	45.6	82.8
Discharge Mortality (%)	17.4	1.28	1.74

Age is presented as median (IQR), all other variables reported as specified. US, United States Cohort; SA, South African Cohort.

Cohort composition also varied by dataset (Table 1); however, all three cohorts were predominantly young (median age 29–36) and male (72.4 −82.0%). Rates of blunt trauma were lower for the SA and US cohorts (45.6% and 57%) than for the Cameroonian cohort (82.8%). Discharge mortality ranged from 1.28% in the SA cohort to 17.4% in the US Cohort.

FEASIBILITY OF CONVENTIONAL SEVERITY SCORES

Compared to the SL algorithm, which was able to utilize data from all patients at each site, all conventional scores were limited by missing composite patient data, which ranged from 0.94% for patient age in the US cohort to 100% for injury severity score in the Cameroonian cohort and KTS in the US cohort. Overall data missingness for conventional scores was higher for resource-constrained settings compared to the US site. (Table 2)

Table 2:

Data Missingness by Trauma Severity Score and Cohort (%)

	US Cohort	SA Cohort	Cameroon Cohort
	N=1,494	N=18, 821	N=7,897
SuperLearner	0%	0%	0%
Kampala Trauma Score	100%	0%	46.6%
Revised Trauma Score	6.2%	6.8%	46.5%
Trauma and Injury Severity Score	9.4%	0%	100%
GCS, Age, Pressure Score	3.4%	8.3%	20.6%
Mechanism, GCS, Age, Pressure Score	3.7%	9.4%	24.2%
Glasgow Coma Score	1.4%	4.7%	3.9%

US, United States Cohort; SA, South African Cohort; GCS, Glasgow Coma Score.

SL DISCRIMINATION OF MORTALITY

SL demonstrated excellent prediction of discharge mortality in the US cohort (AUC 94–97%) with improved discrimination compared to TRISS and far superior discrimination compared to physiologic scores, including the GAP, MGAP, GCS, and RTS. (Figure 1) KTS could not be calculated in this cohort due to lack of composite score recording of the Glasgow Coma Scale.

Figure 1: Comparative prediction of mortality in a US trauma cohort. (N=1,494).

Receiver operating characteristic curves demonstrating discrimination of mortality using a machine learning algorithm (SuperLearner) compared with conventional severity scoring in a single center, highest activation US trauma cohort. Data in parentheses represent 95% confidence intervals. AUC, Area Under the Curve; US, United States; SL, SuperLearner; TRISS, Trauma and Injury Severity Score; RTS, Revised Trauma Score; GAP Glasgow Coma Scale, Age, Pressure Score; MGAP, Mechanism, Glasgow Coma Scale, Age, and Pressure Score; and GCS, Glasgow Coma Scale.

Among the SA cohort, SL generated prediction with similar discrimination to conventional scoring using GAP and RTS (AUC 90–95%). (Figure 2) KTS and TRISS could not be calculated for this cohort with the available variables.

Figure 2: Comparative prediction of mortality in a South African trauma cohort. (N=18,821).

Receiver operating characteristic curves demonstrating discrimination of mortality using a machine learning algorithm (SuperLearner) compared with conventional severity scoring in a high-volume urban South African trauma cohort. Data in parentheses represent 95% confidence intervals. AUC, Area Under the Curve; SA, South Africa; SL, SuperLearner; RTS, Revised Trauma Score; GAP Glasgow Coma Scale, Age, Pressure Score; MGAP, Mechanism, Glasgow Coma Scale, Age, and Pressure Score; and GCS, Glasgow Coma Scale.

SL demonstrated far superior prediction of mortality in the Cameroonian cohort (AUC 90–94%) compared to KTS, GAP, MGAP, GCS, and RTS. (Figure 3) Anatomic scoring was not available for this cohort given limitations of diagnostic imaging, record-keeping, and operative capacity in this setting.

Figure 3: Comparative prediction of mortality in a multisite Cameroonian trauma cohort. (N=7,897).

Receiver operating characteristic curves demonstrating discrimination of mortality using a machine learning algorithm (SuperLearner) compared with conventional severity scoring in a multi-center Cameroonian trauma cohort. Data in parentheses represent 95% confidence intervals. AUC, Area Under the Curve; SL, SuperLearner; KTS, Kampala Trauma Score; RTS, Revised Trauma Score; GAP, Glasgow Coma Scale, Age, Pressure Score; MGAP, Mechanism, Glasgow Coma Scale, Age, and Pressure Score; and GCS, Glasgow Coma Scale.

VARIABLE IMPORTANCE MEASURES

For each cohort, influential drivers of prediction were identified. (Figure 4) Unique, context-specific variables found to drive prediction of mortality in each cohort included admission partial thromboplastin time in the US cohort and distance to the hospital and patient education level in the Cameroonian cohort. Regardless of site differences, indicators of severe head injury, including admission GCS (US, SA) and image-guided or clinician-estimated head injury severity (US, Cameroon), portend high risk of mortality after injury. Advanced patient age was also found to drive prediction of trauma mortality in diverse clinical contexts (US and Cameroon).

Figure 4: Context-specific Clinical Admission Variables Driving SuperLearner Prediction of Mortality.

Top variable importance measures by context using random forest estimation. Estimated severity, ordinal ranking of perceived injury severity by provider at patient presentation (1–6); AIS, Abbreviated Injury Score.

DISCUSSION

In this study, we demonstrate that machine learning generates excellent prediction of trauma mortality across extremely diverse clinical settings. Unlike conventional statistical approaches, which rely on relatively complete collection of pre-specified variables, machine learning was able to discriminate mortality on the basis of different baseline admission variables. Additionally, the feasibility of conventional algorithms in all settings was limited by variable missingness. We further demonstrate that variables driving prediction can lend important insights into context-specific and potentially modifiable targets for clinical intervention. These capabilities of machine learning address many of the challenges that have plagued prior attempts at severity scoring and suggest that machine learning could be an adaptable benchmark for establishing optimal prediction to aid clinical decision making and evaluate trauma systems in diverse contexts.

The datasets utilized in this study reflect trauma care in areas with substantial differences in resource availability, patient population, injury patterns, prehospital systems, and trauma practice standard of care. Prior attempts to apply algorithms developed in high-income contexts to LMIC contexts have been limited by feasibility constraints and these underlying differences in patient trajectory in different health systems (34). Lack of reliable scoring has hindered efforts to generate quality improvement programming in LMICs. In high-income countries, clinicians have yet to harness the full power of the vast amounts of patient data available on modern trauma patients.

Machine learning offers potential solutions to many of these obstacles by approaching the problem in a novel fashion. Traditional modeling has attempted to generate a single model capable of assessing all trauma patients. It can be argued that in addition to patient physiology, injured patient outcomes are a reflection of the overall fitness of a trauma system. Moderate injury may confer minimal mortality risk in the US, but it confers a much higher risk in contexts where resources such as blood transfusions, prehospital care, critical care, and other therapeutic interventions are not available. Machine learning does not attempt to find a single model for all patients but rather to identify the risk to a given patient in their given context. Tailored prediction is achievable because machine learning, unlike conventional scoring, is not constrained by the need for algorithm simplicity and is able to adjust for the complex relationships between many more potential predictors. These features largely explain why SL prediction remains robust despite differences in care.

Although excellent prediction was generated at all sites, the discrimination margin between SL and conventional scoring narrows where mortality prediction is already very good, as seen in the SA cohort. This is because SL will predict future outcomes using the same data-generating distribution as well (asymptotically) or better than the best single algorithm in the library of learners (18). If one uses a broad suite of learners, from very smooth, traditional modeling approaches (e.g., logistic regression) to very flexible, nonparametric routines (like random forest), then the cross-validated estimate of fit can provide a standard by which to compare other, simpler competing approaches. In contrast, where traditional prediction is less robust, either because of underfitting when a large number of physiologic parameters are available (US cohort), or where feasibility of advanced imaging and laboratory measurements has limited application (Cameroon), machine learning offers substantial improvements in discrimination with important implications on health systems development.

Importantly, the embedded use of cross-validation within the SL algorithm is the essential component in the generation of reliable context-tailored prediction. It is possible that simplistic models may report discrimination higher than that generated by SL, but as these models do not inherently embed cross-validation, they are naturally prone to overfit (predicting the building data well), and require calibration on a prospective hold-out sample to better gauge performance.

Comparative variable importance reveals conserved and context-variable predictors driving prediction of mortality in different settings. While it is not possible to definitively determine why this is the case, we would hypothesize that this is not due simply to differences in what variables are being measured but also in part due to underlying differences in the patient population, health system, and in care and treatment practices. Distance to the hospital is a top variable importance measure in Cameroon where there is no formal pre-hospital emergency response system, catchment areas are large and treatment delays are often substantial, whereas this measure is not predictive of outcome in the US cohort as transit times within the catchment region of the center analyzed were negligible. Conversely, markers of trauma-induced coagulopathy rank highly in the US cohort. We would expect that, if measured, these markers would also predict mortality in LMIC cohorts. This highlights how, in addition to revealing potentially modifiable context-specific targets for intervention, variable importance generates an evidence-based means of triaging variable inclusion which could potentially add predictive power in resource-constrained settings.

Although this approach appears promising, there are several notable limitations that should be considered. First, machine learning is most appropriately utilized with large and complex datasets. Without adequate size and variability, predictive capacity will be limited. With simple predictor sets, SL may be effective, but may not be necessary and more traditional statistical approaches may yield similar results. Although machine learning can maximize data utilization and address variable missingness, it will not be able to compensate for poor data quality. It is likely that some prediction-driving variables in any context remain unmeasured, and these will limit the prediction capacity of any prospective algorithm, including SL. Particularly among LMIC settings, acquisition of high quality data inputs will likely remain the most significant barrier to use. Continued progression toward ubiquitous trauma registry data collection on all injured patients will be essential to facilitate improvements in trauma care. Finally, despite being open source, the authors anticipate that trauma clinicians may not routinely have time to download the software and learn a coding language in order to use SL to aid clinical decision making. With this in mind, in order to facilitate point-of-care use in the clinical setting it will be important to develop open-source cloud-based applications of the SL software which allow clinicians to simply upload their trauma dataset in order to obtain context-tailored prediction without needing advanced coding skills.

Currently, the surgical field remains skeptical of black-box approaches like machine learning, as most clinicians and scientists have been trained under a strict paradigm of hypothesis testing based on clinical relevance; data mining, therefore, can seem unfamiliar. Despite their novelty, clinically agnostic approaches offer several other potential advantages. First, SL and similar programs allow for context-tailored prediction and target identification with minimal risk of overfitting. Additionally, these programs can easily be translated into user-friendly open-source applications with the potential for rapid, point-of-care use in diverse contexts. The algorithms are adaptable over time; in contexts with longitudinal and streaming data capacity, they could establish a foundation for real-time individualized decision support. Finally, it is important to note that this method can be readily adapted for clinically-relevant outcomes other than trauma mortality, providing great potential for augmenting clinical decision to optimize performance.

CONCLUSIONS

Machine learning provides superior prediction of mortality after injury in diverse clinical contexts offering discrimination without the need for sophisticated diagnostic data or a common variable set. It is readily scalable and can be used to identify site-specific factors that drive prediction, showing potential as a benchmark for outcomes scoring and risk stratification to improve injury care.