Utilizing Community Level Factors to Improve Prediction of Out of Hospital Cardiac Arrest Outcome using Machine Learning

Abstract

Objectives:

To evaluate the impact of community level information on the predictability of out-of-hospital cardiac arrest (OHCA) survival.

Methods:

We used the Cardiac Arrest Registry to Enhance Survival (CARES) to geocode 9,595 Chicago incidents from 2014–2019 into community areas. Community variables including crime, healthcare, and economic factors from public data were merged with CARES. The merged data were used to develop ML models for OHCA survival. Models were evaluated using Area Under the Receiver Operating Characteristic curve (AUROC) and features were analyzed using SHapley Additive exPansion (SHAP) values.

Results:

Baseline results using CARES data achieved an AUROC of 84%. The final model utilizing community variables increased the AUROC to 88%. A SHAP analysis between high and low performing community area clusters showed the high performing cluster is positively impacted by good health related features and good community safety features positively impact the low performing cluster.

Conclusion:

Utilizing community variables helps predict neurologic outcomes with better performance than only CARES data. Future studies will use this model to perform simulations to identify interventions to improve OHCA survival.

Introduction

Out-of-Hospital-Cardiac-Arrest (OHCA) occurs when cardiac mechanical activity stops outside of the hospital and is confirmed by the absence of circulation¹. OHCA is a severe public health problem affecting over 400,000 people in the USA each year, with just 10% of them surviving². Presenting rhythm, age, and whether bystander cardiopulmonary resuscitation (CPR), targeted temperature management (TTM), coronary angiography, and a cardiac stent were provided are used to predict likelihood of survival with high accuracy³. However, some of the most significant factors influencing OHCA outcomes are the social determinants of health (e.g., neighborhood crime rates, access to healthcare, and housing)⁴. When designing and implementing strategies to achieve equity in OHCA it is important to consider community contextual factors and their influence OHCA incidence, etiology, treatment decisions, and subsequent outcomes.

Machine learning (ML) is a technique for learning underlying patterns in data and performing a desired task⁵. ML is extensively employed in major technological businesses with abundant data and computing capabilities. Researchers can use ML to uncover healthcare patterns using data-rich information such as electronic health records (EHR)⁶. While traditional statistical analyses are the gold standard for investigating patient and treatment interventions, ML algorithms can outperform formal statistical analyses and provide better insight into underlying data patterns in various clinical settings⁷ including disease outbreak prediction⁸, monitoring of vaccine information⁹, and hospital appointment scheduling¹⁰.

The disproportionately high incidence of OHCA and low survival rates in minority and lower income urban neighborhoods are associated with community-level factors such as lower bystander CPR rates, traffic congestion, and safety and liability concerns on the part of OHCA witnesses and first responders compared to suburban more affluent neighborhoods¹¹. As we consider prioritizing interventions to improve OHCA survival, we must understand the association of community level contextual factors. EHRs can easily be integrated with community data as time stamps and location information are readily available. ML models that merge patient-level data with features of the geographic location where OHCA occurs may be an important cross-sectoral approach to addressing disparities in survival. Including publicly available community-level data may increase the performance of the ML models, with new information allowing the models to better predict the outcome. SHapley Additive exPansion (SHAP) values allow analysis of individual features contribution to the outcome of interest in the modeling process¹². SHAP values tools used in the ML modeling post-analysis that quantifies the contribution of each input feature. The use of SHAP values to explain complex ML models has proven effective in other diseases including diabetes¹³, depression¹⁴, and COVID-19¹⁵.

The objective of this study is to determine whether incorporating community variables into a ML model of OHCA can increase predictive accuracy of survival with functional neurologic outcome. The use of SHAP values obtained utilizing the proposed ML model highlights the contributions of community level information to the survival prediction. Understanding how the location of an OHCA influences survival will allow community leaders, public health departments, and healthcare providers to tailor policies and practices to improve OHCA survival at the local level.

Methods

This study was approved by the Office for the Protection of Research Subjects of the University of Illinois at Chicago Protocol # 2017–0637

Study setting

Chicago has approximately three million residents with a diverse demographic makeup (31.4% white, 28.7% African American, 29.9% Hispanic, 6.9% Asian) and stark disparities across neighborhoods in median income and chronic disease.¹⁶ The Chicago Fire Department (CFD) is the single emergency medical services (EMS) agency in the city and provides emergency response and transport to approximately 3,000 OHCA incidents annually¹⁷. The Chicago EMS system has 33 receiving hospitals, of which 24 are ST-elevation myocardial infarction (STEMI) centers of care with interventional cardiology and TTM capabilities¹⁷.

Data Collection

Cardiac Arrest Registry to Enhance Survival (CARES), a multicenter coordinated effort by the Centers for Disease Control and Prevention (CDC) and Emory University, is the largest cardiac arrest registry in the USA and collects data from over 1,800 EMS agencies and 2,200 hospitals¹⁸. This study utilizes CARES data for the City of Chicago as the primary source of OHCA data for modeling.

This study utilized publicly available data sources including the Chicago Health Atlas (CHA)¹⁹, American Community Survey (ACS)²⁰, Chicago Data Portal (CDP)²¹, Weather data²², Flu statistics²³, and Opioid Use statistics²⁴. Once community data is located online, it is collected through webscrappers, which crawl internet sources and store information in tabular forms²⁵. We collected data at the smallest available geographic and time unit for each source. For instance, the CHA provides community data from 77 community areas in Chicago and reports annual statistics, whereas weather information can be extracted daily and is reported based on specific weather stations across Chicago. Throughout this study, data obtained through the webscrapping process is referred to as community data.

The CFD EMS treated 12,839 non-traumatic OHCA incidents between January 2014 and December 2019. This study utilizes the 9,595 OHCA incidents in Chicago outside of Nursing and Healthcare facilities. Figure 1 shows how the Chicago CARES data is preprocessed and split into model development sets. The information used for modeling includes 24 potential input features from CARES. Community data is merged with CARES data for each instance to explore the impact of community characteristics in the model. Community data is merged with the constraints that the data needs a known time and location component and that the information must be known before modeling. As a single feature example of this merging process using the CHA data source, an OHCA incident that occurs in 2018 in the Austin community area can be connected using information about Adult Binge Drinking rates from 2015–2017.

Figure 1:

Data Flow for Machine Learning Modeling Subsets

The objective of this model is to see which community features help the ML algorithms learn the underlying patterns of the OHCA incident and survival outcome. Survival was defined by Cerebral Performance Category (CPC) score: 1 denotes mild or no neurological disability, 2 reflects moderate neurological disability, 3 indicates severe neurological disability, 4 is persistent coma or vegetative state, and 5 indicates death. We collapsed the neurologic outcome into a binary classification where Class 0 are individuals who survived with functional neurological outcomes (CPC1/2), and Class 1 are patients with non-functional neurological outcomes (CPC3/4/5).

Machine Learning Modeling

Figure 1 subsets the 9,595 OHCA instances for this study into three sets: training, validation, and testing. The modeling sets are split randomly across the data timeframe on the assumption that decision-making policies are consistent during the timeframe. The training set consists of 5750 events (60%) and constructs the ML models. The validation set consists of 1445 events (15%) used for model parameter optimization and comparison. The testing set consists of 2400 events (25%) and evaluates models on completely unseen data. Table S1 provides demographic data for each set.

To explore the use of community data when modeling OHCA survival outcome, we start by developing a model using only CARES data (Table S2). We utilized an Embedded Fully Convolutional Network (EFCN)²⁶ which is a neural network architecture that takes as input categorical data and aims to classify the target label. To incorporate community data, an iterative modeling process is used to search for new data features and optimize the model based on the new features. Figure 2 illustrates the modeling process. We start by merging CARES and community data, initially using a random subset of community features, and as the iterative process becomes targeted over time keeping the high impact community features. The result of the data merge is the model data which is fed to the model optimization component. Model optimization aims to develop the best model for the new features by exploring a variety of neural network connections (illustrated in Figure S1) and the network parameters. After optimization, the developed model is compared to the existing best model where the initial best is the base model developed using only CARES data. The model evaluation and comparison give insight into how impactful to the outcome are new community features. Based on the evaluation, some community features are retained in the next iteration, and some are removed/replaced. This iterative process continues until there is no improvement in model comparison process for 20 loops. During the exploration of community data, the process starts by just exploring the CHA data source. After the iterative process is complete for CHA, additional data is added in the next run of the iterative modeling process. This allows the process to significantly reduce the number of feature combinations that the iterative modeling has searched for and therefore reduces the computational time.

Figure 2:

Iterative Modeling Process

The exploration of community data and the optimization of parameters results in training ML models that are complex. While they have better performance, these complex models are generally not interpretable²⁷. To combat the complexity of these models in deep learning, knowledge distillation can be utilized²⁸ which is a model compression method that involves teaching a smaller model to mimic a larger model. The smaller model is then trained to learn the exact behavior of the bigger model by aiming to replicate its outputs at every level (not just the final loss).

Feature Analysis

The individual input features can be more easily explored using the models developed through the knowledge distillation process. For the binary class of neurological outcome that we are purposed, the SHAP process outputs a vector of values corresponding to each input feature to the optimal model¹². The goal of SHAP is to explain the prediction of an instance by calculating the contribution of each feature to the final prediction. These values are normalized to explain the expected outcome, where positive values correspond to the feature contributing to the patient being Class 0 (CPC1/2) and negative values towards Class 1. The summation of the SHAP vector leads to the final prediction of the instance. Each instance can have positive and negative features in the vector, and the final summation is the expected output class. The values in the SHAP vectors are generally small in magnitude where the ratio of the contribution is the most important aspect when comparing values. As an example, a patient who survives with CPC1/2 can come from a community with low opioid use (SHAP value = 0.0396) and a high rate of stroke-related deaths (SHAP value = −0.0649). Overall, this patient is expected to survive with CPC1/2 because the summation of all input features is positive. Given the SHAP values for all the instances in the testing set, input features can be analyzed to explore the impact of community context on the OHCA survival rates.

Results

The training set consists of 5750 OHCA patients. Of these, 416 have a neurological outcome of Class 0. The training set is used to develop the ML models. Each model utilizes the training data to learn the underlying patterns in the data to perform the classification task. During the iterative modeling process, the training instances remain the same. The only difference is that the input features are searched to find the best combination of community features.

The validation set consists of 1445 cardiac arrest patients, of which 107 have a neurological outcome of Class 0. Table S3 provides detailed information about the optimized hyperparameters for each model. Hyperparameters provide the details of the algorithm for model reproducibility. Table 1 presents the Area Under the Receiver Operating Characteristic curve (AUROC) for the key steps in the iterative modeling process. The base model utilizing only CARES data achieves a validation AUROC of 87.9. During iterative modeling, the search objective is to maximize the validation AUROC while minimizing the number of new features included. The final model from the modeling process achieves a validation AUROC of 90.8.

Table 1:

Results on Validation and Test Set in terms of AUROC with 95% Confidence Intervals

Iteration	Model Data	Validation	Test	Knowledge Distillation on Test
Base	CARES [18]	87.898 +/− 2.691	84.498 +/− 3.089	--
1	CARES [18] CHA [19]	88.676 +/− 2.543	86.798 +/− 3.128	86.122 +/− 3.233
2	CARES [18] CHA [19] Weather [22]	88.978 +/− 2.613	86.998 +/− 3.023	86.232 +/− 3.256
3	Cares [18] CHA [19] ACS [20]	89.568 +/− 2.689	87.244 +/− 3.002	86.879 +/− 3.301
4	Cares [18] CHA [19] ACS [20] Flu [23] Opioid [24]	90.766 +/− 2.589	88.128 +/− 3.032	87.912 +/− 3.289

The testing set consists of 2400 cardiac arrest patients, of which 147 have a Class 0 neurological outcome. The AUROC of the base model is 84.5. The best modeled set from the validation process results in an AUROC of 88.1. This best model utilizes community data from the CHA, flu, and opioid sources. We then use the knowledge distillation process to generate a smaller model that mimics the best model’s behavior. This smaller model results in an AUROC of 87.9.

Utilizing the knowledge distilled model from Iteration 4, community area level SHAP analysis is performed on the 2400 cardiac arrest patients in the testing set. Our SHAP analysis focuses on the difference in collective values for community area clusters in Chicago. The criteria for comparison are that two clusters must provide EMS level interventions in a statistically similar manner and that the crossover between destination hospitals of instances in both clusters must be greater than 80%. In addition to these criteria, we aim to compare clusters that have significantly different survival outcomes. These criteria allow for the analysis to focus on community-level features and not the specifics of EMS and hospital discrepancies. Cluster 1 consists of three community areas with 118 instances in the testing set and a CPC1/2 rate of 10.1%. Cluster 2 consists of four community areas with 144 instances in the testing set and a CPC1/2 rate of 5.6%. Table 2 provides the top three most impactful features towards both classes of neurological outcomes. Positive contributions toward survival in Cluster 1 are features related to the health of the individuals (Low Food Related Deaths and Low Flu Rates). In comparison, Cluster 2 is positively impacted when the community is in a situation associated with community safety (Low Homicide Rates and Low Opioid Related Deaths). When looking at Cluster 1, rates of survival with CPC 1/2 are negatively affected by indicators of poor health (Lung Cancer Death Rate and High Stroke Rates) whereas Cluster 2 is negatively impacted when the health/nutrition of the community is poor (Low Fruit and Vegetable Consumption and High Flu Rates).

Table 2:

SHAP Values for the Two Cluster Analysis

Cluster 1 CPC1/2 Rate: 10.1%		Cluster 2 CPC1/2 Rate: 5.6%
Feature & Value	Average SHAP Value	Feature & Value	Average SHAP Value
Diet Related Deaths [Low]	0.0513	Household Property [Low]	0.0498
Household Property [Low]	0.0456	Homicides [Low]	0.0356
Flu Rate Prior 1 Week [Low]	0.0398	Opioid-Related Deaths 1 Month Prior [Low]	0.0267
Opioid-Related Deaths 1 Month Prior [High]	−0.0421	Violent Crime [High]	−0.0512
Lung Cancer Deaths [Mid]	−0.0347	Adult Fruit and Vegetable Serving [Low]	−0.0398
Stroke Deaths [High]	−0.025	Flu Rate Prior 1 Week [High]	−0.0345

Discussion

ML is a useful tool in OHCA survival prediction using individual-level data^26,29. Our work builds upon previous ML studies for OHCA survival prediction by adding community-level health and demographic data into the predictions. We demonstrated that ML models that include community and CARES patient-level data can predict OHCA survival more accurately than models based on CARES data alone. By combining CARES data elements with Chicago community-level data, our final survival model achieves a testing AUROC of 88.1, a significant increase in AUROC of 3.63 over the base model. The testing and validation AUROCs for all steps in the iterative modeling process are similar, suggesting that these models will generalize well to new data.

To understand how social determinants of health are associated with OHCA outcomes, previous studies have investigated the relationship between community level characteristics, such as predominant race/ethnicity and socioeconomic status, and bystander CPR rates and subsequent survival from OHCA^{30, 31} Our study stands out from previously published work by including communities’ clinical co-prevalence in ML model development (chronic comorbidity and acute illness surveillance data). Our SHAP analysis looks into the contribution of community variables for high performing and low performing community clusters as illustrated by our example comparison of Cluster 1 and Cluster 2. This work has the potential to inform policy makers about the risk associated with a community.

Community comorbidity prevalence and social determinants of health have been utilized to predict incidence and outcomes from multiple cardiovascular diseases including hypertension, heart failure, and stroke³². This type of work has become especially relevant during the COVID19 pandemic. For example, the COVID-19 Community Vulnerability Index (CCVI) uses statistical modeling to link community-level socioeconomic features, epidemiologic risk factors, community factors, and healthcare system factors to assess community resilience to the impact of COVID-19³³. These risk prediction models can be powerful tools to promote health equity as they can be leveraged by public health officials and accountable care organizations to justify the prioritization of public health interventions to communities at high risk³⁴.

Limitations

The CHA, CDP, flu and opioid data sources are specific to Chicago. However, similar information is collected by municipalities in other surveillance databases to replicate our analyses. For example, data from the ACS and weather data sources are readily available for all EMS agencies across the USA. The CDC also reports model-based, population-level analysis and community estimates of health measures from communities across the USA.³⁵

Although the ACS and CHA data are reflective of the diversity of individuals who live in Chicago, both are subject to non-sampling error. These data sources serve as great primary exploratory sources when we started the iterative process. However, the model would benefit from other sources that provide similar information collected on a monthly or weekly basis.

Public Health Implications

A novel aspect of our work is the use of SHAP values to identify the most significant community factors for predicting OHCA outcomes throughout the entire city of Chicago and within each community. Identifying the most impactful model features can help drive the discussion to address the most efficient and effective community interventions to eliminate disparities in OHCA outcomes. The analysis shown in Table 2 has the potential to reveal features that may be responsive to community-level interventions to increase survival rates within a given community. After knowing these features, simulation can determine other communities within Chicago that would be most responsive to community level interventions. As a motivating example, we see that the two clusters in Table 2 are both strongly impacted by the flu rates. When we simulate 10,000 instances of OHCA for Cluster 2 with high and low flu rates we see that the expected CPC1/2 rate goes from 5.5% to 5.8%. Future work will aim to develop a comprehensive simulation framework that allows key policymakers to understand how changes in community factors can potentially influence the outcomes of OHCA patients throughout the cardiac arrest system of care.

A ML model that integrates patient-level data from the largest EMS agency reporting to CARES and community-level features from publicly available datasets improved predictive of variance of OHCA survival outcomes over existing ML models. Moreover, the use of SHAP values can identify the most impactful features and identify the most efficient and effective community interventions that will result in improved survival and eliminate disparities in OHCA outcomes. This work represents an important cross-sectoral approach to addressing disparities in OHCA survival.