Abstract
Aims
The out-of-hospital cardiac arrest (OHCA) outcomes project is a national research registry. One of its aims is to explore sources of variation in OHCA survival outcomes. This study reports the development and validation of risk prediction models for return of spontaneous circulation (ROSC) at hospital handover and survival to hospital discharge.
Methods and results
The study included OHCA patients who were treated during 2014 and 2015 by emergency medical services (EMS) from seven English National Health Service ambulance services. The 2014 data were used to identify important variables and to develop the risk prediction models, which were validated using the 2015 data. Model prediction was measured by area under the curve (AUC), Hosmer–Lemeshow test, Cox calibration regression, and Brier score. All analyses were conducted using mixed-effects logistic regression models. Important factors included age, gender, witness/bystander cardiopulmonary resuscitation (CPR) combined, aetiology, and initial rhythm. Interaction effects between witness/bystander CPR with gender, aetiology and initial rhythm and between aetiology and initial rhythm were significant in both models. The survival model achieved better discrimination and overall accuracy compared with the ROSC model (AUC = 0.86 vs. 0.67, Brier score = 0.072 vs. 0.194, respectively). Calibration tests showed over- and under-estimation for the ROSC and survival models, respectively. A sensitivity analysis individually assessing Index of Multiple Deprivation scores and location in the final models substantially improved overall accuracy with inconsistent impact on discrimination.
Conclusion
Our risk prediction models identified and quantified important pre-EMS intervention factors determining survival outcomes in England. The survival model had excellent discrimination.
Keywords: Cardiac arrest, Emergency medical services, Out-of-hospital cardiac arrest, Resuscitation, Predictive model
Introduction
Out-of-hospital cardiac arrest (OHCA) is a leading cause of cardiac -related death in developed countries; only 1 in 10 patients survive to hospital discharge.1,2 However, in recent years several countries and regions have made major advances in the improvement of OHCA survival rates: 25% of patients in Stavanger, Norway survived to hospital discharge , 3 21% in North Holland,4 and 24% in Seattle, USA.5
In England, there are approximately 80 000 OHCA incidents annually, with resuscitation attempted in less than half by emergency medical services (EMS).6 However, survival outcomes have shown limited improvement, with data from English ambulance services indicating one in four patients have return of spontaneous circulation (ROSC) sustained to hospital handover, while the survival to hospital discharge rate is still around 10%,6,7 with regional variation reported between 2% and 12%.8 A validated risk adjustment model would aid understanding of regional variations, enabling unbiased comparisons between ambulance services for survival outcomes.9 Risk adjustment models are an important element to support healthcare quality improvement e.g. for in-hospital cardiac arrest (IHCA).10 Improving the management of OHCA is part of the National Health Service (NHS) Long Term plan11 and the British Heart Foundation, Resuscitation Council (UK) and NHS England are committed to improving survival from OHCA in England.12,13
Recent studies have recognized a range of OHCA patient case-mix and pre-EMS intervention factors in non-UK populations that are associated with survival. These include: OHCA location; 14,15 patient age, gender; 14,16–19 witnessed status; bystander cardiopulmonary resuscitation (CPR); 14,16,19,20 initial rhythm; 14,17,19 aetiology; 14 public access defibrillator (PAD) use; 16 and socioeconomic status.21,22
The relative contribution of each of these factors to survival varies between countries. Only a few studies have assessed the possible interaction between pre-EMS intervention factors and OHCA outcomes in a systematic process through the development and validation of risk prediction models.23,24 However, such models also include EMS intervention factors. Whilst validated models to risk-adjust survival rates for IHCA exist , 10 no such models have currently been developed for OHCA outcomes in England.
The aim of this analysis was to (i) identify key factors associated with two survival outcomes, ROSC at hospital handover and survival to hospital discharge; (ii) develop and evaluate models to predict both outcomes in England; and (iii) support quality improvement activities in the UK to improve OHCA outcomes.
Data and methods
Data source
The Out-of-hospital Cardiac Arrest Outcomes (OHCAO) registry, hosted by the University of Warwick, is a national research database developed in accordance with Utstein-style guidelines.25 UK ambulance services collect and contribute OHCA cases where there is a resuscitation attempt by EMS. Patient case-mix data, process variables, structure data, and survival outcomes are collected. Details of the project have been published elsewhere.26
Study population
The study population included OHCA cases from 1 January 2014 to 31 December 2015 treated by 7 out of 10 English NHS ambulance services providing data to the registry. Patients of all age groups were included except those with a Do Not Attempt Resuscitation order in place or who achieved ROSC before EMS arrival. The 2014 data were used to identify important factors of the outcomes and develop the prediction models. The 2015 data were used for model validation.
Data management
Variables assessed in the analysis included age, gender, witnessed status, bystander CPR, aetiology, and initial rhythm (Table 1). Age was used as a continuous variable in the analysis. For aetiology, unknown cause was presumed as medical according to the Utstein definitions.25 We merged categories with few cases to enable better estimation. The aetiology was re-defined as (i) medical; (ii) trauma and exsanguination; (iii) overdose, asphyxia, and submersion; (iv) other. Similarly, the rhythm for modelling included (i) ventricular fibrillation (VF) or pulseless ventricular tachycardia (VT); (ii) asystole; (iii) pulseless electrical activity (PEA) and bradycardia. Cases that had one or more missing assessed variables were excluded from the analysis.
Table 1.
Characteristics of out-of-hospital cardiac arrest and survival outcomes in the development and validation sets
| 2014 (N = 17 528) | 2015 (N = 17 078) | |
|---|---|---|
| Age—mean (SD) | 68.9 (19.0) | 68.0 (19.3) |
| Gender | ||
| Male | 10 933 (62.4%) | 10 770 (63.1%) |
| Female | 6595 (37.6%) | 6308 (36.9%) |
| IMD 2010 score—mean (SD) | 22.5 (14.9) | 24.2 (16.2) |
| Missing | 11 054 (63.1%) | 7533 (44.1%) |
| IMD 2015 score—mean (SD) | 22.8 (14.9) | 24.3 (16.3) |
| Missing | 11 054 (63.1%) | 7533 (44.1%) |
| Witnessed by | ||
| Unwitnessed | 6569 (37.5%) | 5851 (34.3%) |
| EMS | 2835 (16.2%) | 3035 (17.8%) |
| Layperson | 8124 (46.3%) | 8192 (48.0%) |
| Bystander CPR | ||
| Yes | 8272 (56.3%) | 7919 (56.4%) |
| No | 6421 (43.7%) | 6124 (43.6%) |
| Not applicable (EMS witnessed) | 2835 | 3035 |
| Aetiology | ||
| Medical | 15 486 (88.4%) | 14 505 (84.9%) |
| Trauma and Exsanguination | 527 (3.0%) | 502 (2.9%) |
| Trauma | 527 (3.0%) | 501 (2.9%) |
| Exsanguination | 0 (0.0%) | 1 (0.0%) |
| Submersion, overdose, asphyxia and respiratory | 637 (3.6%) | 695 (4.1%) |
| Submersion | 38 (0.2%) | 35 (0.2%) |
| Drug overdose | 65 (0.4%) | 232 (1.4%) |
| Asphyxia | 534 (3.0%) | 428 (2.5%) |
| Respiratory | 0 (0.0%) | 0 (0.0%) |
| Other (non-cardiac) | 878 (5.0%) | 1376 (8.1%) |
| Initial rhythm | ||
| VF/VT | 4691 (26.8%) | 4143 (24.3%) |
| Asystole | 9067 (51.7%) | 8776 (51.4%) |
| PEA and bradycardia | 3770 (21.5%) | 4159 (24.4%) |
| PEA | 3660 (20.9%) | 4069 (23.8%) |
| Bradycardia | 110 (0.6%) | 90 (0.5%) |
| Location | ||
| Home | 6145 (35.1%) | 1621 (9.5%) |
| Non-home | 1689 (9.6%) | 512 (3.0%) |
| Missing | 9694 (55.3%) | 14 945 (87.5%) |
| Survival outcomes | ||
| ROSC at hospital handover | ||
| Yes | 4696 (26.8%) | 5194 (30.4%) |
| No | 11 774 (67.2%) | 11 125 (65.1%) |
| Missing | 1058 (6.0%) | 759 (4.4%) |
| Survival to hospital discharge | ||
| Yes | 1427 (8.1%) | 1484 (8.7%) |
| No | 9221 (52.6%) | 12 202 (71.4%) |
| Missing | 6880 (39.3%) | 3392 (19.9%) |
Percentages were calculated using the total N in each year as the denominator and may not added up to 100% due to rounding errors.
CPR, cardiopulmonary resuscitation; EMS, emergency medical services; IMD, Index of Multiple Deprivation; PEA, pulseless electrical activity; ROSC, return of spontaneous circulation; SD, standard deviation; VF, ventricular fibrillation; VT, ventricular tachycardia.
Bystander CPR has been widely accepted as a key factor for improving survival.12,27 In our data, patients witnessed by EMS are treated as not receiving bystander CPR. However, they are different from the non-EMS witnessed patients in terms of the time to start of CPR and advanced treatments, and therefore in their chance of survival.28 Consequently, the combination of these witnessed cases may cancel out the bystander CPR effect. Therefore, the witness and bystander CPR were assessed as an interaction using the following categories: (i) unwitnessed and no bystander CPR; (ii) unwitnessed but bystander CPR given; (iii) EMS witnessed; (iv) bystander witnessed but no CPR given; (v) bystander witnessed and CPR given. Witnessed cases with unspecified type (by bystander or EMS) were excluded from the analysis to improve the accuracy of the analysis.
Because of the heterogeneity in data collection methods across the UK, ambulance services provided data of varying quality. Particularly, post-hospital transfer data collection is complex and expensive, leading to variation in levels of missing data. Large amount of missing data in a number of the assessed variables and outcomes resulted in excluding 4 of the 10 ambulance services from model development. However, only three of the four ambulance services were excluded from model validation for the same reason. Ethnicity and PAD use were not included in the analysis due to consistently poor data quality across the registry. The Index of Multiple Deprivation (IMD) scores were extracted from the English indices of deprivation data via the linkage to the patients’ home postcode. The OHCA location data were converted from addresses to postcodes before being recoded to home or non-home by comparing the location postcode with home postcode. The 2014 location data had less missing data due to a data quality improvement project in collaboration with Public Health England, which was not available for 2015 data collection. The IMD and location were assessed in the final models as a sensitivity analysis to explore their contribution to the prediction improvement.
Statistical methods
Factors and outcomes were summarized in the following way: frequency and proportion for categorical variables and mean and standard deviation (SD) for continuous variables. Assessment and estimation were carried out using mixed- effects logistic regression models. Ambulance services were fitted as a random effect in all models to account for the potential heterogeneity of patient and event characteristics across ambulance services. The key factors were determined if they showed statistical significance (P < 0.05) when individually tested in a mixed- effects model.
We employed the fractional polynomial (FP) method to explore the best fitting functional form of continuous factors using linear and polynomial functions.29 As the method does not take into account the random effect, we decided that the best form would only be used if it also improved model fit in the mixed- effects models. Otherwise, only the linear form was included.
The development of the prediction models for each outcome was conducted in three stages (Figure 1).
Figure 1.
Model development flowchart. *Non-significant main effect or interaction is removed unless the removal led to poorer model performance.
Model performance was measured by model fit as well as prediction performance (discrimination and calibration). How well the model fit the data was measured by Akaike information criterion (AIC) value. Akaike information criterion estimates the relative amount of information lost by a given model: the lower the AIC value, the less information a model loses, the higher the quality of that model. Model discrimination (the ability of the model to separate individuals who do and do not achieve sustained ROSC or survive) was quantified by the area under the receiver operating characteristic curve (AUC) with 95% confidence interval, which measures how well the prediction can discriminate positive and negative outcomes.30 The following categories were used to interpret AUC: (i) ≥0.9 = outstanding; (ii) 0.8–0.9 = excellent; (iii) 0.7–0.8 = acceptable, (iv) <0.7 = poor.31 A range of measures were taken to evaluate model calibration (i.e. the agreement between observed and predicted risk). (i) Hosmer–Lemeshow test compares the agreement between observed and predicted values. However, the test is sensitive to the sample size.31 Hence, the observed proportion of positive outcomes (e.g. survival) was plotted against the deciles of the predicted values for a virtual inspection. (ii) Cox calibration fits a line between the outcome and log odds of the prediction using logistic regression.32 A line with an intercept of 0 and slope of 1 indicates perfect agreement. (iii) Overall accuracy was measured by Brier’s score, the mean squared error of prediction with 0 indicating the best prediction.33 The developed models were applied in the validation data to obtain the predictions and the model performance measurements.
A P-value <0.05 was considered statistically significant. The FP method was applied in Stata v15 (StataCorp). Other analyses were carried out in SAS v9.4 (SAS Institute Inc., Cary, NC, USA).
Results
There were 17 528 and 17 078 eligible OHCA cases in 2014 and 2015, respectively. By excluding missing outcome data, 16 470 and 10 648 cases were used for the model development of ROSC at handover and survival to discharge, respectively; and 16 319 and 13 686 cases were used to validate models, respectively. Patient -level characteristics and study outcomes in both development and validation sets are summarized in Table 1.
Key factors of outcomes
Individual assessment of candidate factors is summarized in Table 1. Gender, witness/bystander CPR, aetiology, and initial rhythm were significantly associated with both outcomes. Age was significantly associated only with survival to hospital discharge. The IMD scores and location data were available in less than half of the development data. Only location was significantly associated with both outcomes (Table 2).
Table 2.
Association between the included factors and survival outcomes
| ROSC at hospital handover (N = 16 470) |
Survival to hospital discharge (N = 10 648) |
|||
|---|---|---|---|---|
| Variables | Odds ratio (95% CI) | P-value | Odds ratio (95% CI) | P-value |
| Age | 1.002 (0.999–1.004) | 0.068 | 0.984 (0.981–0.987) | <0.001 |
| Gender | ||||
| Male | 1 | 0.014 | 1 | <0.001 |
| Female | 1.091 (1.018–1.170) | 0.606 (0.534–0.687) | ||
| Witness/bystander CPR | ||||
| Unwitnessed and no bystander CPR | 1 | <0.001 | 1 | <0.001 |
| Unwitnessed but bystander CPR given | 0.846 (0.742–0.965) | 1.162 (0.870–1.553) | ||
| EMS witnessed | 2.309 (2.036–2.618) | 4.822 (3.754–6.195) | ||
| Bystander witnessed but no CPR given | 1.764 (1.552–2.005) | 2.000 (1.520–2.630) | ||
| Bystander witnessed and CPR given | 2.112 (1.884–2.367) | 3.001 (2.345–3.841) | ||
| Aetiology | ||||
| Medical | 1 | 0.006 | 1 | <0.001 |
| Trauma and Exsanguination | 0.786 (0.639–0.965) | 0.470 (0.303–0.730) | ||
| Submersion, overdose, asphyxia, and respiratory | 1.181 (0.991–1.407) | 0.745 (0.541–1.027) | ||
| Other | 0.857 (0.729–1.008) | 0.507 (0.356–0.722) | ||
| Initial rhythm | ||||
| VF/VT | 1 | <0.001 | 1 | <0.001 |
| Asystole | 0.857 (0.729–1.008) | 0.507 (0.356–0.722) | ||
| PEA and bradycardia | 0.247 (0.228–0.268) | 0.068 (0.057–0.082) | ||
| IMD 2010 scorea | 1.002 (0.998–1.006) | 0.438 | 0.999 (0.993–1.005) | 0.689 |
| IMD 2015 scorea | 1.001 (0.997–1.005) | 0.747 | 1.000 (0.994–1.006) | 0.912 |
| Locationb | ||||
| Non-home | 1 | <0.001 | 1 | <0.001 |
| Home | 0.793 (0.704–0.893) | 0.510 (0.429–0.607) | ||
Only 6383 and 4214 cases included in the ROSC and survival analysis, respectively.
Only 7107 and 3996 cases included in the ROSC and survival analysis, respectively.
Model development
At Stage 1, all individually significant factors remained in Model S1 for ROSC at handover and survival to discharge (Table 3). These full models had the best model fit and prediction performance at this stage (Table 4).
Table 3.
Model specification for Stages 1, 2, and 3
| Model | Outcome | Factors | |
|---|---|---|---|
| Stage 1 | S1 (ROSC) | ROSC at hospital handover | Gender + witness/bystander CPR + aetiology + initial rhythm |
| S1 (Survival) | Survival to hospital discharge | Age + gender + witness/bystander CPR + aetiology + initial rhythm | |
| Stage 2 | S2 (ROSC) | ROSC at hospital handover | Gender + witness/bystander CPR + aetiology + initial rhythm + witness/bystander CPR*gender + witness/bystander CPR*aetiology + witness/bystander CPR*initial rhythm + aetiology*initial rhythm |
| S2 (Survival) | Survival to hospital discharge | Age + gender + witness/bystander CPR + aetiology + initial rhythm + witness/bystander CPR*gender + witness/bystander CPR*aetiology + witness/bystander CPR*initial rhythm + aetiology*initial rhythm | |
| Stage 3 | S3 (ROSC) | ROSC at hospital handover | Gender + witness/bystander CPR + aetiology + initial rhythm + witness/bystander CPR*gender + witness/bystander CPR*aetiology + witness/bystander CPR*initial rhythm + aetiology*initial rhythm |
| S3 (Survival) | Survival to hospital discharge | Age + gender + witness/bystander CPR + aetiology + initial rhythm + witness/bystander CPR*gender + witness/bystander CPR*aetiology + witness/bystander CPR*initial rhythm + aetiology*initial rhythm | |
Interaction of two effects. There was no model term dropped from Stage 2 to 3.
Table 4.
Discrimination and calibration of the predictive models for return of spontaneous circulation at handover and survival to hospital discharge
| Model fit | Discrimination | Calibration |
||||||
|---|---|---|---|---|---|---|---|---|
| AIC | AUC (95% CI) | HL P-value | Cox calibration |
Brier score | ||||
| Intercept | Slope | |||||||
| ROSC | Model S1 | 2014 | 18 220.0 | 0.69 (0.68–0.70) | <0.001 | 0.00 (−0.05 to 0.06) | 1.00 (0.95–1.06) | 0.185 |
| 2015 | — | 0.66 (0.66–0.67) | <0.001 | 0.12 (0.06 to 0.17) | 0.91 (0.85–0.96) | 0.197 | ||
| Model S3 | 2014 | 18 050.2 | 0.70 (0.69–0.71) | <0.001 | 0.00 (−0.05 to 0.06) | 1.00 (0.95–1.05) | 0.182 | |
| 2015 | — | 0.67 (0.66–0.68) | <0.001 | 0.09 (0.03–0.14) | 0.88 (0.83–0.93) | 0.194 | ||
| Survival | Model S1 | 2014 | 6256.1 | 0.85 (0.84–0.86) | <0.001 | 0.00 (−0.09 to 0.10) | 1.00 (0.95–1.05) | 0.087 |
| 2015 | — | 0.86 (0.85–0.87) | <0.001 | −0.10 (−0.19 to −0.01) | 1.01 (0.97–1.06) | 0.072 | ||
| Model S3 | 2014 | 6208.8 | 0.85 (0.84–0.86) | <0.001 | 0.00 (−0.09 to 0.10) | 1.00 (0.95–1.05) | 0.086 | |
| 2015 | — | 0.87 (0.86–0.88) | <0.001 | −0.13 (−0.22 to −0.05) | 1.00 (0.96–1.05) | 0.072 | ||
AIC, Akaike information criterion (not reported in the validation): a lower value indicates a better fit. AUC, area under the curve: (i) ≥0.9 = outstanding; (ii) 0.8–0.9 = excellent; (iii) 0.7–0.8 = acceptable, and (iv) <0.7 = poor. HL, Hosmer–Lemeshow: P < 0.05 suggests poor calibration. Cox calibration: a line with an intercept of 0 and slope of 1 indicates perfect agreement. Brier score is the mean squared error of the predictions and a value of 0 indicates the best prediction. Model S2 included the same main and interaction effects as Model S3 for both outcomes. Hence, the prediction performance summary for Model S2 was omitted.
Fewer interaction terms were evaluated at Stage 2 for ROSC compared with survival to hospital discharge (Table 3). Four interaction terms with P-value < 0.1 were included for further analysis in Model S2 for both outcomes: witness/bystander CPR with gender, aetiology and initial rhythm, and aetiology with initial rhythm.
For both outcomes, all included interactions and main effects from the corresponding Model S2 remained significant and produced the best prediction performance at Stage 3. Therefore, Model S3 is identical to Model S2 for both outcomes. In the ROSC Model S3, the main effect of witness/bystander CPR was not significant but remained in the final model mainly due to its clinical importance. Additionally, the removal of this term led to a negligible decrease in the model fit (AIC from 18 050.2 to 18 051.0) and had no benefit gained regarding the AUC and brier score (Appendix Table A1). Likewise, gender was kept in the survival Model S3 for the same reason (AIC changed from 6208.8 to 6208.9 after removal). The Model S3 for both outcomes was decided as the final models. The full specification of each model is shown in Table 3.
Model validation
The results of the validation of Model S3 for both OHCA outcomes are shown in Table 4. The AUC values showed that the survival model produced good prediction (AUC = 0.85 in development and 0.87 in validation) while the ROSC model was less so (AUC = 0.70 in development and 0.67 in validation). The Cox calibration regression produced a positive intercept and a slope < 1 in the validation data for the ROSC model, indicating a decreasing underestimation. The survival model, however, had a negative intercept, indicating a consistent overestimation of the survival rate. These trends were also supported by the calibration plots (Figure 2).
Figure 2.
Hosmer–Lemeshow calibration plot of the development (top) and validation data (bottom) for return of spontaneous circulation at handover and survival to discharge. (Circles and bars are the observed proportions and 95% confidence intervals of achieved return of spontaneous circulation and survived to discharge in the deciles of predictions.)
Sensitivity analysis
The IMD 2010 and 2015 scores substantially improved the AUC in the development (0.69 –0.77 for ROSC at handover, 0.85 –0.89 for survival at discharge) but not in the validation data (0.66– 0.67 for ROSC at handover, 0.87 –0.83 for survival at discharge). Consistent improvement was gained in terms of the overall accuracy for both outcomes in the development and validation data. However, the Cox calibration intercept and slope indicated the inclusion of these variables led to more over-/under-estimations for individual cardiac arrest patients. The impact of location was inconclusive as both models had worse performance. It may be caused by more missing data in the validation data but requires further analysis.
Discussion
In this analysis, we developed and validated risk prediction models for ROSC at hospital handover and survival to hospital discharge using the 2014 and 2015 OHCAO data, respectively, of seven English ambulance services. Gender, aetiology, witness, bystander CPR, and initial rhythm were used as the key factors for both outcomes, together with their first-order interactions. Age was an additional factor included in the survival model. The survival model had an excellent predictive performance with an AUC of 0.87 in the validation data, while more improvement is needed for the ROSC model with an AUC of 0.67.
Our model development was based on the case-mix and event data prior to EMS intervention. The data depict the initial status of an OHCA patient. Case-mix data are commonly used to account for the variability of outcomes in healthcare research.34 Early recognition and access to EMS service, as well as early CPR, are fundamental to and the strongest links35 of improving OHCA outcomes in the chain of survival.36
Age had a less significant effect on ROSC at hospital handover than on survival to hospital discharge. Previous studies have demonstrated a conflicting result of the age effect. Some recent studies showed a significant association14,37 but others reported the opposite result.38,39 In addition, in a study of non-trauma Welsh OHCA patients, Barnard et al.40 identified a significant quadratic (i.e. squared) effect but not linear effect of age.
The survival model had better discrimination and overall accuracy compared with the ROSC model. Event location and socioeconomic status, measured by IMD 2010 and 2015 score, are associated with survival outcomes.14,15,21,22 However, the inclusion of these data in the sensitivity analysis only yielded a small improvement in AUC and Brier score for both models and did not reduce the model performance gap. The ROSC after cardiac arrest (RACA) score, which was developed in a similar way to our Model S1 with the inclusion of the time to EMS arrival, achieved an AUC of 0.73.14 The model discrimination was relatively improved but is still not as good as our survival model. The large prediction discrepancy between ROSC and survival models suggested that the pre-EMS intervention variables may not be the best candidates for predicting ROSC. Sustained ROSC at hospital transfer is a short-term survival event. It is associated with EMS intervention factors, such as CPR quality , 41 airway management,42 and drug treatment.43 Inclusion of EMS intervention factors in the model may further improve performance.
Our survival model had excellent predictive performance when compared with the existing models for non-UK populations. Valenzuela et al. developed a model using age, bystander CPR, time to CPR, time to defibrillation, and time -related interactions in the US patients.44 The model made poor predictions of survival (AUC = 0.66). A Dutch study also assessed survival prediction using three cumulative sets of data related to bystander, first responder, and paramedics. The first responder model also looked at pre-EMS intervention data and had similar model discrimination (AUC = 0.85) to that of our model. The other two, involving less and more predictors, had an AUC of 0.76 and 0.90, respectively.45 The French OHCA score was developed using rhythm, time to CPR, CPR duration, and laboratory data.46 This tool produced an AUC of 0.88 when restricting the prediction of survival with good neurological outcome in a small group of patients achieving ROSC. It was further developed as the Cardiac Arrest Hospital Prognosis (CAHP) score which achieved an AUC of 0.85 and 0.91 in two larger validation sets.15 However, both scores were inferior to the Swedish risk score in terms of AUC (0.75 for OHCA and CAHP vs. 0.84) when three models were validated using the Target Temperature Management trial data.47
Implications
Improving OHCA management and survival in England is advocated in key national documents.11,13 Our risk prediction models identified important pre-EMS intervention factors of survival outcomes in England. The models could provide case-mix adjusted performance evaluation for the NHS ambulance services. More importantly, we aim to use the models to identify at-risk patients in the English population, help ambulance services and health authorities develop health strategies in different communities, and ultimately improve survival rates and reduce the health burden of OHCA.
Strengths and limitations
This is the first analysis to develop and evaluate risk prediction models for OHCA survival outcomes in the UK. It gained strength by using the OHCAO registry data. This national registry collects and standardizes a key sub-set of Utstein elements48 from the participating UK ambulance services. Thus, the models offer comparable results across the ambulance services and can be applicable to other registries built on the same guidelines. In addition, we used mixed -effect logistic regression to carry out the analysis. The random effect model takes into account the heterogeneity of patient and event characteristics across ambulance services and provides more precise estimations.
This analysis also has several limitations. We did not include several pre- and peri-EMS intervention variables that were used as factors of OHCA survival outcomes in other models due to the perspective of our analysis and the limited Utstein elements. For example, prodromal symptoms18 and biomarkers49 were not collected in the OHCAO registry. However, in the future data linkage may enable us to evaluate their contribution in a more comprehensive prediction model. Our datasets contained data of varying quality because of the heterogeneous data collection processes across ambulance services. Some important variables, such as OHCA location and socioeconomic status, had more missing data and were restricted to the sensitivity analysis to avoid compromising the model interpretation and prediction. However, some improvement has been observed, such as less missing survival data from 2014 to 2015. Ongoing work to improve the data quality should enable improvement in model performance. In addition, location did not improve the model performance in the sensitivity analysis. The effect could have become irrelevant after the inclusion of other prediction terms. The improvement of data quality may allow a better assessment of the variable.
Conclusion
In conclusion, our risk prediction models identified and quantified important pre-EMS intervention factors determining survival outcomes in England. The survival model had excellent discrimination.
Acknowledgements
Out-of-Hospital Cardiac Arrest Outcomes Steering Committee. OHCAO Collaborators: Theresa Foster, East of England Ambulance Service NHS Trust; Frank Mersom, East of England Ambulance Service NHS Trust; Gurkamal Francis, London Ambulance Service NHS Trust; Michelle O’Rourke, North East Ambulance Service NHS Trust; Clare Bradley, North West Ambulance Service NHS Trust; Philip King, South Central Ambulance Service NHS Trust; Ed England, South Central Ambulance Service NHS Trust; Patricia Bucher, South East Coast Ambulance Service NHS Trust; Jessica Lynde, South Western Ambulance Service NHS Trust; Nancy Loughlin, South Western Ambulance Service NHS Trust; Jenny Lumley-Holmes, West Midlands Ambulance Service NHS Trust; Dr Julian Mark, Yorkshire Ambulance Service NHS Trust.
Funding
Research grants from the British Heart Foundation and Resuscitation Council (UK); G.D.P. is supported as NIHR Senior Investigator and Director of Research for the Intensive Care Foundation.
Conflict of interest: C.J., T.B., S.B., C.H., and G.D.P. are employed by the University of Warwick, which receives grants from the British Heart Foundation and the Resuscitation Council (UK) for the conduct of the OHCAO project. Other authors report no conflicts of interest.
Appendix
Table A1.
Calibration, discrimination, and overall accuracy of the sensitivity analysis
| AUC (95% CI) | HL P-value | Cox calibration |
Brier score | ||||
|---|---|---|---|---|---|---|---|
| Intercept | Slope | ||||||
| ROSC | Model S3 + IMD 2010 | 2014 | 0.77 (0.76–0.78) | 0.001 | 0.00 (−0.07 to 0.08) | 1.00 (0.94–1.06) | 0.066 |
| 2015 | 0.68 (0.66–0.69) | 0.891 | 0.50 (0.43 to 0.57) | 0.60 (0.54–0.66) | 0.074 | ||
| Model S3 + IMD 2015 | 2014 | 0.77 (0.76–0.78) | 0.001 | 0.00 (−0.07 to 0.08) | 1.00 (0.94–1.06) | 0.066 | |
| 2015 | 0.68 (0.66–0.69) | 0.953 | 0.50 (0.43 to 0.57) | 0.60 (0.54–0.66) | 0.074 | ||
| Model S3 + location | 2014 | 0.71 (0.70–0.72) | <0.001 | 0.00 (−0.07 to 0.08) | 1.00 (0.93–1.08) | 0.082 | |
| 2015 | 0.65 (0.63–0.68) | 0.001 | 0.51 (0.34 to 0.68) | 0.74 (0.61–0.87) | 0.032 | ||
| Survival | Model S3 + IMD 2010 | 2014 | 0.89 (0.87–0.90) | <0.001 | 0.00 (−0.12 to 0.13) | 1.00 (0.93–1.07) | 0.035 |
| 2015 | 0.83 (0.82–0.85) | <0.001 | −0.19 (−0.29 to −0.08) | 0.82 (0.76–0.88) | 0.034 | ||
| Model S3 + IMD 2015 | 2014 | 0.89 (0.87–0.90) | <0.001 | 0.00 (−0.12 to 0.13) | 1.00 (0.93–1.07) | 0.035 | |
| 2015 | 0.83 (0.82–0.85) | 0.881 | −0.18 (−0.29 to −0.08) | 0.82 (0.76–0.89) | 0.034 | ||
| Model S3 + location | 2014 | 0.85 (0.84–0.87) | <0.001 | 0.00 (−0.12 to 0.13) | 1.00 (0.93–1.08) | 0.039 | |
| 2015 | 0.84 (0.81–0.87) | 0.030 | −0.76 (−0.95 to −0.57) | 0.88 (0.75–1.02) | 0.009 | ||
Contributor Information
OHCAO Collaborators:
Theresa Foster, Frank Mersom, Gurkamal Francis, Michelle O’Rourke, Clare Bradley, Philip King, Patricia Bucher, Jessica Lynde, Jenny Lumley-Holmes, and Dr Julian Mark
References
- 1. Girotra S, van Diepen S, Nallamothu BK, Carrel M, Vellano K, Anderson ML et al.; CARES Surveillance Group and the HeartRescue Project. Regional variation in out-of-hospital cardiac arrest survival in the United States. Circulation 2016;133:2159–2168. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Grasner JT, Lefering R, Koster RW, Masterson S, Bottiger BW, Herlitz J et al.; EuReCa ONE Collaborators. EuReCa ONE-27 Nations, ONE Europe, ONE Registry: a prospective one month analysis of out-of-hospital cardiac arrest outcomes in 27 countries in Europe. Resuscitation 2016;105:188–195. [DOI] [PubMed] [Google Scholar]
- 3. Lindner TW, Soreide E, Nilsen OB, Torunn MW, Lossius HM. Good outcome in every fourth resuscitation attempt is achievable—an Utstein template report from the Stavanger region. Resuscitation 2011;82:1508–1513. [DOI] [PubMed] [Google Scholar]
- 4. Berdowski J, Berg RA, Tijssen JG, Koster RW. Global incidences of out-of-hospital cardiac arrest and survival rates: systematic review of 67 prospective studies. Resuscitation 2010;81:1479–1487. [DOI] [PubMed] [Google Scholar]
- 5. Public Health – Seattle and King County Division of Emergency Medical Services 2013 Annual Report to the King County Council. 2017. https://www.kingcounty.gov/depts/health/emergency-medical-services/~/media/depts/health/emergency-medical-services/documents/reports/2017-Annual-Report.ashx (25 February 2020).
- 6.NHS England. Ambulance Quality Indicators. https://www.england.nhs.uk/statistics/statistical-work-areas/ambulance-quality-indicators/ (31 May 2019).
- 7. Hawkes C, Booth S, Ji C, Brace-McDonnell SJ, Whittington A, Mapstone J, Cooke MW et al. Epidemiology and outcomes from out-of-hospital cardiac arrests in England. Resuscitation 2017;110:133–140. [DOI] [PubMed] [Google Scholar]
- 8. Perkins GD, Cooke MW. Variability in cardiac arrest survival: the NHS Ambulance Service Quality Indicators. Emerg Med J 2012;29:3–5. [DOI] [PubMed] [Google Scholar]
- 9. Krumholz HM, Brindis RG, Brush JE, Cohen DJ, Epstein AJ, Furie K et al. American Heart A, Quality of C, Outcomes Research Interdisciplinary Writing G, Council on E, Prevention, Stroke C, American College of Cardiology F. Standards for statistical models used for public reporting of health outcomes: an American Heart Association Scientific Statement from the Quality of Care and Outcomes Research Interdisciplinary Writing Group: cosponsored by the Council on Epidemiology and Prevention and the Stroke Council. Endorsed by the American College of Cardiology Foundation. Circulation 2006;113:456–462. [DOI] [PubMed] [Google Scholar]
- 10. Harrison DA, Patel K, Nixon E, Soar J, Smith GB, Gwinnutt C et al. Development and validation of risk models to predict outcomes following in-hospital cardiac arrest attended by a hospital-based resuscitation team. Resuscitation 2014;85:993–1000. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.National Health Service. The NHS Long Term Plan. 2019. https://www.longtermplan.nhs.uk/wp-content/uploads/2019/01/nhs-long-term-plan-june-2019.pdf (25 February 2020).
- 12. Perkins GD, Lockey AS, de Belder MA, Moore F, Weissberg P, Gray H; Community Resuscitation Group. National initiatives to improve outcomes from out-of-hospital cardiac arrest in England. Emerg Med J 2016;33:448–451. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.OHCA Steering Group. Resuscitation to Recovery: A National Framework to Improve Care of People with Out-of-Hospital Cardiac Arrest (OHCA) in England. 2017. https://www.resus.org.uk/publications/resuscitation-to-recovery/ (25 February 2020).
- 14. Grasner JT, Meybohm P, Lefering R, Wnent J, Bahr J, Messelken M et al. German Resuscitation Registry Study G. ROSC after cardiac arrest—the RACA score to predict outcome after out-of-hospital cardiac arrest. Eur Heart J 2011;32:1649–1656. [DOI] [PubMed] [Google Scholar]
- 15. Maupain C, Bougouin W, Lamhaut L, Deye N, Diehl JL, Geri G et al. The CAHP (Cardiac Arrest Hospital Prognosis) score: a tool for risk stratification after out-of-hospital cardiac arrest. Eur Heart J 2016;37:3222–3228. [DOI] [PubMed] [Google Scholar]
- 16. Abrams HC, McNally B, Ong M, Moyer PH, Dyer KS. A composite model of survival from out-of-hospital cardiac arrest using the Cardiac Arrest Registry to Enhance Survival (CARES). Resuscitation 2013;84:1093–1098. [DOI] [PubMed] [Google Scholar]
- 17. Fridman M, Barnes V, Whyman A, Currell A, Bernard S, Walker T et al. A model of survival following pre-hospital cardiac arrest based on the Victorian Ambulance Cardiac Arrest Register. Resuscitation 2007;75:311–322. [DOI] [PubMed] [Google Scholar]
- 18. Nehme Z, Andrew E, Bray JE, Cameron P, Bernard S, Meredith IT et al. The significance of pre-arrest factors in out-of-hospital cardiac arrests witnessed by emergency medical services: a report from the Victorian Ambulance Cardiac Arrest Registry. Resuscitation 2015;88:35–42. [DOI] [PubMed] [Google Scholar]
- 19. Rea TD, Cook AJ, Stiell IG, Powell J, Bigham B, Callaway CW et al. Resuscitation Outcomes Consortium I. Predicting survival after out-of-hospital cardiac arrest: role of the Utstein data elements. Ann Emerg Med 2010;55:249–257. [DOI] [PubMed] [Google Scholar]
- 20. Sasson C, Rogers MA, Dahl J, Kellermann AL. Predictors of survival from out-of-hospital cardiac arrest: a systematic review and meta-analysis. Circ Cardiovasc Qual Outcomes 2010;3:63–81. [DOI] [PubMed] [Google Scholar]
- 21. Vaillancourt C, Lui A, De Maio VJ, Wells GA, Stiell IG. Socioeconomic status influences bystander CPR and survival rates for out-of-hospital cardiac arrest victims. Resuscitation 2008;79:417–423.18951678 [Google Scholar]
- 22. Wells DM, White LL, Fahrenbruch CE, Rea TD. Socioeconomic status and survival from ventricular fibrillation out-of-hospital cardiac arrest. Ann Epidemiol 2016;26:418–423 e1. [DOI] [PubMed] [Google Scholar]
- 23. Aschauer S, Dorffner G, Sterz F, Erdogmus A, Laggner A. A prediction tool for initial out-of-hospital cardiac arrest survivors. Resuscitation 2014;85:1225–1231. [DOI] [PubMed] [Google Scholar]
- 24. Grasner JT, Herlitz J, Koster RW, Rosell-Ortiz F, Stamatakis L, Bossaert L. Quality management in resuscitation—towards a European cardiac arrest registry (EuReCa). Resuscitation 2011;82:989–994. [DOI] [PubMed] [Google Scholar]
- 25. Jacobs I, Nadkarni V, Bahr J, Berg RA, Billi JE, Bossaert L et al.; International Liaison Committee on Resuscitation; American Heart Association; European Resuscitation Council; Australian Resuscitation Council; New Zealand Resuscitation Council; Heart and Stroke Foundation of Canada; InterAmerican Heart Foundation; Resuscitation Councils of Southern Africa; ILCOR Task Force on Cardiac Arrest and Cardiopulmonary Resuscitation Outcomes. Cardiac arrest and cardiopulmonary resuscitation outcome reports: update and simplification of the Utstein templates for resuscitation registries: a statement for healthcare professionals from a task force of the International Liaison Committee on Resuscitation (American Heart Association, European Resuscitation Council, Australian Resuscitation Council, New Zealand Resuscitation Council, Heart and Stroke Foundation of Canada, InterAmerican Heart Foundation, Resuscitation Councils of Southern Africa). Circulation 2004;110:3385–3397. [DOI] [PubMed] [Google Scholar]
- 26. Perkins GD, Brace-McDonnell SJ; OHCAO Project Group. The UK Out of Hospital Cardiac Arrest Outcome (OHCAO) project. BMJ Open 2015;5:e008736. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. Hasselqvist-Ax I, Riva G, Herlitz J, Rosenqvist M, Hollenberg J, Nordberg P et al. Early cardiopulmonary resuscitation in out-of-hospital cardiac arrest. N Engl J Med 2015;372:2307–2315. [DOI] [PubMed] [Google Scholar]
- 28. Hostler D, Thomas EG, Emerson SS, Christenson J, Stiell IG, Rittenberger JC et al.; Resuscitation Outcomes Consortium Investigators. Increased survival after EMS witnessed cardiac arrest. Observations from the Resuscitation Outcomes Consortium (ROC) Epistry-Cardiac arrest. Resuscitation 2010;81:826–830. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Royston P, Ambler G, Sauerbrei W. The use of fractional polynomials to model continuous risk variables in epidemiology. Int J Epidemiol 1999;28:964–974. [DOI] [PubMed] [Google Scholar]
- 30. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010;21:128–138. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Hosmer DW Jr,, Lemeshow S, Sturdivant RX. Applied Logistic Regression. 3rd ed. New York: Wiley; 2013. p160–164. [Google Scholar]
- 32. Cox DR. Two further applications of a model for binary regression. Biometrika 1958;45:562–565. [Google Scholar]
- 33. Brier GW. Verification of forecasts expressed in terms of probability. Mon Weather Rev 1950;78:1–3. [Google Scholar]
- 34. Lilford R, Mohammed MA, Spiegelhalter D, Thomson R. Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma. Lancet 2004;363:1147–1154. [DOI] [PubMed] [Google Scholar]
- 35. Deakin CD. The chain of survival: not all links are equal. Resuscitation 2018;126:80–82. [DOI] [PubMed] [Google Scholar]
- 36. Nolan J, Soar J, Eikeland H. The chain of survival. Resuscitation 2006;71:270–271. [DOI] [PubMed] [Google Scholar]
- 37. Deasy C, Bray JE, Smith K, Harriss LR, Bernard SA, Cameron P. Out-of-hospital cardiac arrests in the older age groups in Melbourne, Australia. Resuscitation 2011;82:398–403. [DOI] [PubMed] [Google Scholar]
- 38. Weston CF, Wilson RJ, Jones SD. Predicting survival from out-of-hospital cardiac arrest: a multivariate analysis. Resuscitation 1997;34:27–34. [DOI] [PubMed] [Google Scholar]
- 39. Wuerz RC, Holliman CJ, Meador SA, Swope GE, Balogh R. Effect of age on prehospital cardiac resuscitation outcome. Am J Emerg Med 1995;13:389–391. [DOI] [PubMed] [Google Scholar]
- 40. Barnard EBG, Sandbach DD, Nicholls TL, Wilson AW, Ercole A. Prehospital determinants of successful resuscitation after traumatic and non-traumatic out-of-hospital cardiac arrest. Emerg Med J 2019;36:333–339. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41. Meaney PA, Bobrow BJ, Mancini ME, Christenson J, de Caen AR, Bhanji F et al. CPR Quality Summit Investigators; the American Heart Association Emergency Cardiovascular Care Committee; the Council on Cardiopulmonary Critical Care, Perioperative and, Resuscitation. Cardiopulmonary resuscitation quality: [corrected] improving cardiac resuscitation outcomes both inside and outside the hospital: a consensus statement from the American Heart Association. Circulation 2013;128:417–435. [DOI] [PubMed] [Google Scholar]
- 42. McMullan J, Gerecht R, Bonomo J, Robb R, McNally B, Donnelly J et al. Airway management and out-of-hospital cardiac arrest outcome in the CARES registry. Resuscitation 2014;85:617–622. [DOI] [PubMed] [Google Scholar]
- 43. Perkins GD, Ji C, Deakin CD, Quinn T, Nolan JP, Scomparin C et al.; PARAMEDIC2 Collaborators. A randomized trial of epinephrine in out-of-hospital cardiac arrest. N Engl J Med 2018;379:711–721. [DOI] [PubMed] [Google Scholar]
- 44. Valenzuela TD, Roe DJ, Cretin S, Spaite DW, Larsen MP. Estimating effectiveness of cardiac arrest interventions: a logistic regression survival model. Circulation 1997;96:3308–3313. [DOI] [PubMed] [Google Scholar]
- 45. Waalewijn RA, de Vos R, Tijssen JG, Koster RW. Survival models for out-of-hospital cardiopulmonary resuscitation from the perspectives of the bystander, the first responder, and the paramedic. Resuscitation 2001;51:113–122. [DOI] [PubMed] [Google Scholar]
- 46. Adrie C, Cariou A, Mourvillier B, Laurent I, Dabbane H, Hantala F et al. Predicting survival with good neurological recovery at hospital admission after successful resuscitation of out-of-hospital cardiac arrest: the OHCA score. Eur Heart J 2006;27:2840–2845. [DOI] [PubMed] [Google Scholar]
- 47. Martinell L, Nielsen N, Herlitz J, Karlsson T, Horn J, Wise MP et al. Early predictors of poor outcome after out-of-hospital cardiac arrest. Crit Care 2017;21:96. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48. Perkins GD, Jacobs IG, Nadkarni VM, Berg RA, Bhanji F, Biarent D et al. Cardiac arrest and cardiopulmonary resuscitation outcome reports: update of the Utstein Resuscitation Registry Templates for Out-of-Hospital Cardiac Arrest: a statement for healthcare professionals from a task force of the International Liaison Committee on Resuscitation (American Heart Association, European Resuscitation Council, Australian and New Zealand Council on Resuscitation, Heart and Stroke Foundation of Canada, InterAmerican Heart Foundation, Resuscitation Council of Southern Africa, Resuscitation Council of Asia); and the American Heart Association Emergency Cardiovascular Care Committee and the Council on Cardiopulmonary, Critical Care, Perioperative and Resuscitation. Resuscitation 2015;96:328–340. [DOI] [PubMed] [Google Scholar]
- 49. Huang CH, Tsai MS, Chien KL, Chang WT, Wang TD, Chen SC, Ma MH et al. Predicting the outcomes for out-of-hospital cardiac arrest patients using multiple biomarkers and suspension microarray assays. Sci Rep 2016;6:27187. [DOI] [PMC free article] [PubMed] [Google Scholar]