Abstract
Background
The aim of this study was to validate and compare the performance of statistical (Utstein‐Based Return of Spontaneous Circulation and Shockable Rhythm–Witness–Age–pH) and machine learning–based (Prehospital Return of Spontaneous Circulation and Swedish Cardiac Arrest Risk Score) models in predicting the outcomes following out‐of‐hospital cardiac arrest and to assess the impact of the COVID‐19 pandemic on the models' performance.
Methods and Results
This retrospective analysis included adult patients with out‐of‐hospital cardiac arrest treated at 3 academic hospitals between 2015 and 2023. The primary outcome was neurological outcomes at hospital discharge. Patients were divided into pre‐ (2015–2019) and post‐2020 (2020–2023) subgroups to examine the effect of the COVID‐19 pandemic on out‐of‐hospital cardiac arrest outcome prediction. The models' performance was evaluated using the area under the receiver operating characteristic curve and compared by the DeLong test. The analysis included 2161 patients, 1241 (57.4%) of whom were resuscitated after 2020. The cohort had a median age of 69.2 years, and 1399 patients (64.7%) were men. Overall, 69 patients (3.2%) had neurologically intact survival. The area under the receiver operating characteristic curves for predicting neurological outcomes were 0.85 (95% CI, 0.83–0.87) for the Utstein‐Based Return of Spontaneous Circulation score, 0.82 (95% CI, 0.81–0.84) for the Shockable Rhythm–Witness–Age–pH score, 0.79 (95% CI, 0.78–0.81) for the Prehospital Return of Spontaneous Circulation score, and 0.79 (95% CI, 0.77–0.81) for the Swedish Cardiac Arrest Risk Score model. The Utstein‐Based Return of Spontaneous Circulation score significantly outperformed both the Prehospital Return of Spontaneous Circulation score (P<0.001) and the Swedish Cardiac Arrest Risk Score model (P=0.007). Subgroup analysis indicated no significant difference in predictive performance for patients resuscitated before versus after 2020.
Conclusions
In this external validation, both statistical and machine learning–based models demonstrated excellent and fair performance, respectively, in predicting neurological outcomes despite different model architectures. The predictive performance of all evaluated clinical scoring systems was not significantly influenced by the COVID‐19 pandemic.
Keywords: cardiopulmonary resuscitation, COVID‐19, machine learning, out‐of‐hospital cardiac arrest, prediction model, prognostication
Subject Categories: Cardiopulmonary Resuscitation and Emergency Cardiac Care
Nonstandard Abbreviations and Acronyms
- ML
machine learning
- NTUH
National Taiwan University Hospital
- OHCA
out‐of‐hospital cardiac arrest
- P‐ROSC
Prehospital Return of Spontaneous Circulation
- ROSC
return of spontaneous circulation
- SCARS
Swedish Cardiac Arrest Risk Score
- SWAP
Shockable Rhythm–Witness–Age–pH
- UB‐ROSC
Utstein‐Based Return of Spontaneous Circulation
Clinical Perspective.
What Is New?
For the intended purpose, the Shockable Rhythm–Witness–Age–pH score and the Swedish Cardiac Arrest Risk Score model demonstrated excellent and fair accuracy, respectively, in predicting neurological outcomes (area under the receiver operating characteristic curve, 0.82) and survival at hospital discharge (area under the receiver operating characteristic curve, 0.70), similar to the values shown in their development studies.
All included clinical scoring systems, including the Utstein‐Based Return of Spontaneous Circulation, Shockable Rhythm–Witness–Age–pH score, Prehospital Return of Spontaneous Circulation score, and Swedish Cardiac Arrest Risk Score model, can predict neurological outcomes with fair to excellent accuracy.
What Are the Clinical Implications?
Given a predefined context and prespecified threshold of sensitivity or specificity, these 4 well‐validated scoring systems can be applied in clinical care to assist in decision making regarding therapeutic strategies, such as transfer from the scene to the hospital, implementation of extracorporeal cardiopulmonary resuscitation, or termination of resuscitation.
Depending on the availability of clinical information, clinicians may choose from these scoring systems to assess outcomes following out‐of‐hospital cardiac arrest, as these scoring systems demonstrate similar predictive performance.
In Asia, emergency medical services (EMS) respond to ≈59.4 out‐of‐hospital cardiac arrest (OHCA) incidents per 100 000 person‐years. 1 Among these patients, only 5.4% survive to hospital discharge, and a scant 2.8% achieve favorable neurological outcomes. 2
The ability to accurately predict the prognosis of patients experiencing OHCA plays a crucial role in guiding clinical decision making. Most risk‐stratification scores for OHCA were developed for patients achieving return of spontaneous circulation (ROSC). 3 Based on variables obtainable before ROSC, most clinical scoring systems were developed for termination of resuscitation, 4 with only a few scoring systems 5 developed to predict OHCA outcomes before ROSC was achieved.
Using statistical methods, Baldi et al 6 designed the Utstein‐Based Return of Spontaneous Circulation (UB‐ROSC) score by logistic regression models, drawing on registry data from Italy and Switzerland in 2015 to 2017, to predict sustained ROSC upon arrival at the emergency department (ED). Similarly, Shih et al 7 developed the Shockable Rhythm–Witness–Age–pH (SWAP) score by logistic regression models using Taiwanese data from the same time frame to forecast neurological outcomes at hospital discharge.
In contrast, using machine learning (ML) techniques, Liu et al. 8 derived the Prehospital Return of Spontaneous Circulation (P‐ROSC) score by random forest–based models from Asian multinational registry data spanning 2009 to 2018, estimating the likelihood of prehospital ROSC before ED admission. Likewise, Hessulf et al 9 created the Swedish Cardiac Arrest Risk Score (SCARS) model by extreme gradient boosting algorithms, using Swedish national data from 2010 to 2020, to predict 30‐day survival following OHCA.
This study first seeks to validate these clinical scoring systems for their respective intended outcomes within a multicenter ED‐based cohort. Subsequently, the performance of these models in predicting neurological outcomes is compared. Finally, considering their development before the COVID‐19 pandemic, a subgroup analysis will assess the pandemic's impact on the models' predictive accuracy.
METHODS
The data that support the findings of this study are available upon reasonable request to the corresponding authors. This study was a secondary analysis of a prospectively collected database on OHCA cases registered at the EDs of National Taiwan University Hospital (NTUH), NTUH Yunlin Branch, and Far Eastern Memorial Hospital (FEMH). The study was conducted in accordance with the Declaration of Helsinki and its later amendments. Ethical approval for this research was granted by the Research Ethics Committee of NTUH (reference number: 202301117RIND), which also waived the requirement for informed consent. The reporting of results conforms to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis statement. 10
Study Setting
NTUH, NTUH Yunlin Branch, and Far Eastern Memorial Hospital are academic medical centers located in different regions of Taiwan, featuring ≈2600, 400, and 1200 inpatient beds, respectively. Annually, these centers manage around 100 000, 50 000, and 130 000 ED visits.
The EMS in the areas surrounding these hospitals operate through a 2‐tiered, fire‐based system, 11 comprising teams for basic life support with defibrillation and advanced life support. The basic life support with defibrillation team is capable of performing defibrillation and placing a laryngeal mask airway. The advanced life support team providers are authorized to perform tracheal tube insertion and intravenous injections of resuscitation medications, such as epinephrine and amiodarone, as per protocol. 12 Cardiopulmonary resuscitation (CPR) is performed in alignment with resuscitation guidelines, 13 , 14 incorporating mechanical CPR devices such as LUCAS or AutoPulse. The World Health Organization declared the COVID‐19 outbreak as a pandemic on March 11, 2020. 15 To reduce aerosol generation, 16 the protocols for airway management were adapted from that time 17 until May 2023. In Taiwan, except for certain situations, EMS personnel are legally prohibited from declaring a patient deceased, 2 so a “scoop‐and‐run” approach is adopted for OHCA cases. This entails that, after both basic life support with defibrillation and advanced life support teams have arrived on the scene and initiated mechanical CPR, they transport the patient to an ED for further resuscitation efforts.
The EDs at the study's hospitals follow the Advanced Cardiac Life Support collaborative framework, 13 , 14 , 15 placing significant emphasis on teamwork. 18 , 19 A blood gas analysis was performed upon patient arrival at the ED. The decision to administer sodium bicarbonate therapy was contingent upon these analysis results. In the absence of a do‐not‐resuscitate order, it is typical in Taiwan to continue CPR for patients with OHCA who do not achieve ROSC for at least 30 minutes in EDs.
Study Population
This study evaluated patients who experienced OHCA and underwent CPR between January 1, 2015, and December 31, 2023. The inclusion criteria for this analysis were (1) individuals aged ≥18 years, (2) those who received resuscitation and were transported by EMS, and (3) those who received ongoing CPR upon ED arrival. Patients were excluded (1) if data on any variables required for calculating UB‐ROSC, SWAP, and P‐ROSC scores, or SCARS model were missing; or (2) if the cardiac arrest was attributable to situational causes, such as trauma, hanging, drowning, or asphyxia.
Data Collection, Variable Definitions, and Outcome Measures
All OHCA incidents were documented using the standardized Utstein‐style template. 20 , 21 Details of prehospital resuscitation were collected by EMS, whereas in‐hospital resuscitation processes, critical care interventions, and outcomes were extracted from electronic medical records. This task was undertaken by clinical abstractors specifically trained for this purpose and blinded to the study's hypothesis.
The time to EMS arrival was determined as the period from EMS dispatch to their arrival at the scene. Medications administered before hospital arrival included epinephrine and amiodarone. The pH value obtained within 2 minutes of ED arrival was analyzed. The duration of CPR within the ED was defined as the time from ED arrival to cessation of resuscitation efforts, whether due to the achievement of ROSC, the initiation of extracorporeal membrane oxygenation, or the patient's death.
The calculations of UB‐ROSC, 6 SWAP, 7 and P‐ROSC 8 scores were conducted following the methodologies described in the original studies. For the SCARS model, 9 the probability of 30‐day survival was computed using the designated web application (https://scars‐1.streamlit.app/). The variables used for each scoring model are provided in Figure 1.
Figure 1. Variables used in each clinical scoring system.
The yellow‐shaded rounded rectangle indicates statistical models, while the orange‐shaded rounded rectangle indicates machine learning–based models. The components of each scoring system are represented by rectangles in different colors, with the same variables depicted in the same colors, except for the black rectangles. The black rectangles indicate the variables not shared by any other scoring system. CPR indicates cardiopulmonary resuscitation; EMS, emergency medical service; P‐ROSC, Prehospital Return of Spontaneous Circulation; pVT, pulseless ventricular tachycardia; SCARS, Swedish Cardiac Arrest Risk Score; SWAP, Shockable Rhythm–Witness–Age–pH; VF, ventricular fibrillation; and UB‐ROSC, Utstein‐Based Return of Spontaneous Circulation.
The primary outcome was favorable neurological recovery at hospital discharge, defined by a Cerebral Performance Category of 1 or 2. 20 , 21 Secondary outcomes included sustained ROSC and survival to hospital discharge. Sustained ROSC was defined as ROSC that persisted until the patient's arrival at the ED and the transfer of care to the hospital's medical staff. 20 , 21
Statistical Analysis
Categorical variables are presented as counts and proportions, whereas continuous variables are summarized using medians and interquartile ranges. Categorical variables were analyzed using the χ2 test, while continuous variables were compared using the Mann–Whitney U test. The comparisons between patients stratified by neurological outcomes were expressed with standardized differences. 22 The predictive performance of the UB‐ROSC, SWAP, and P‐ROSC scores, and the SCARS model was assessed using the area under the receiver operating characteristic (ROC) curve (AUC), with comparisons made using the DeLong test for correlated ROC curves. 23 An AUC of <0.7 is considered poor, 0.7 to 0.8 is considered fair, 0.8 to 0.9 is considered excellent, and >0.9 is considered outstanding. 24 Other performance metrics, including sensitivity, specificity, positive predictive value, and negative predictive value, were calculated on the basis of maintaining the lower boundary of the 95% CI of sensitivity >0.9. 4 To examine the impact of the COVID‐19 pandemic on the models' predictive accuracy, patients were divided into 2 subgroups on the basis of the year of resuscitation: before (2015–2019) or after 2020 (2020–2023). The predictive performance was compared between these 2 subgroups using the DeLong test for independent ROC curves. 23 The overall comparisons of the AUC were conducted by STATA 18.0 (StataCorp LLC, College Station, TX). All other statistical analyses were performed using MedCalc Statistical Software version 20.218 (MedCalc Software Ltd., Ostend, Belgium). A 2‐tailed P value of <0.05 was considered statistically significant. To accommodate multiple comparisons (6 pairwise comparisons among the 4 models), the post hoc P value threshold was adjusted to 0.008 (ie, 0.05/6).
RESULTS
The patient selection process (Figure 2) yielded a final cohort of 2161 patients, with 1011 patients enrolled from NTUH, 969 from Far Eastern Memorial Hospital, and 181 from NTUH Yunlin Branch. As summarized in Table 1, the median age of the cohort was 69.2 years, with 1399 (64.7%) of the patients being male. Post‐2020 resuscitation accounted for 1241 patients (57.4%). The majority of OHCA events (n=1530 [70.8%]) occurred at home. Bystanders or EMS witnessed 940 cardiac arrests (43.5%), and bystander CPR was administered in 1210 cases (56.0%). Initial shockable rhythms at the scene were documented in 595 patients (28.0%), and 510 patients (23.6%) received defibrillation during prehospital CPR. During prehospital resuscitation, epinephrine and amiodarone were administered to 770 (35.6%) and 66 patients (3.1%), respectively. The median blood gas pH value was 6.91 (interquartile range, 6.88–7.03), and the median CPR duration in the ED was 28.0 minutes. A total of 1440 patients (66.6%) achieved sustained ROSC, 156 patients (7.2%) survived to hospital discharge, and 69 patients (3.2%) experienced favorable neurological recovery.
Figure 2. Patient inclusion flowchart.
CPR indicates cardiopulmonary resuscitation; EMS, emergency medical services; OHCA, out‐of‐hospital cardiac arrest; and ROSC, return of spontaneous circulation.
Table 1.
Characteristics of Patients Stratified by Favorable Neurological Recovery at Hospital Discharge
| Variables | All patients (n=2161) | Patients without favorable neurological recovery at hospital discharge (n=2092) | Patients with favorable neurological recovery at hospital discharge (n=69) | Standardized difference |
|---|---|---|---|---|
| Basic demographics | ||||
| Age, y | 69.2 (59.0 to 80.8) | 69.9 (59.3 to 80.9) | 59.9 (51.7 to 66.2) | 0.670 |
| Male sex, n (%) | 1399 (64.7) | 1344 (64.2) | 55 (79.7) | −0.350 |
| Post‐2020 resuscitation | 1241 (57.4) | 1218 (58.2) | 23 (33.3) | 0.516 |
| Cardiac arrest locations | 1.155 | |||
| At home | 1530 (70.8) | 1511 (72.2) | 19 (27.5) | |
| Nursing home | 129 (6.0) | 127 (6.1) | 2 (2.9) | |
| Workplace | 80 (3.7) | 68 (3.3) | 12 (17.4) | |
| School | 10 (0.5) | 7 (0.3) | 3 (4.3) | |
| Street | 184 (8.5) | 163 (7.8) | 21 (30.4) | |
| Public building | 221 (10.2) | 209 (10.0) | 12 (17.4) | |
| Sport | 7 (0.3) | 7 (0.3) | 0 (0) | |
| Prehospital resuscitation | ||||
| Witness by bystander, n (%) | 934 (43.2) | 881 (42.1) | 53 (76.8) | −0.756 |
| Witness by EMS, n (%) | 70 (3.2) | 67 (3.2) | 3 (4.3) | −0.060 |
| Witness by bystander or EMS, n (%) | 940 (43.5) | 887 (42.4) | 53 (76.8) | −0.749 |
| Bystander CPR, n (%) | 1210 (56.0) | 1162 (55.5) | 48 (69.6) | −0.293 |
| Initial shockable rhythms, n (%) | 595 (28.0) | 574 (27.9) | 21 (30.4) | −0.066 |
| Defibrillation at any time by EMS, n (%) | 510 (23.6) | 460 (22.0) | 50 (72.5) | −1.172 |
| LMA insertion, n (%) | 1548 (71.6) | 1499 (71.7) | 49 (71.0) | 0.014 |
| Tracheal intubation, n (%) | 217 (10.0) | 210 (10.0) | 7 (10.1) | −0.004 |
| Mechanical CPR, n (%) | 425 (19.7) | 417 (19.9) | 8 (11.6) | 0.230 |
| Epinephrine administration, n (%) | 770 (35.6) | 745 (35.6) | 25 (36.2) | −0.013 |
| Amiodarone administration, n (%) | 66 (3.1) | 53 (2.5) | 13 (18.8) | −0.547 |
| Time to EMS arrival, min | 4.0 (3.0 to 7.0) | 5.0 (3.0 to 7.0) | 3.0 (2.0 to 5.0) | 0.499 |
| Duration of prehospital CPR by EMS, min | 20.0 (15.0 to 24.0) | 20.0 (15.0 to 24.0) | 17.0 (11.8 to 21.0) | 0.462 |
| ED resuscitation | ||||
| Initial rhythm at ED arrival, n (%) | 1.493 | |||
| Ventricular fibrillation | 101 (4.7) | 77 (3.7) | 24 (34.8) | |
| Pulseless ventricular tachycardia | 16 (0.7) | 14 (0.7) | 2 (2.9) | |
| Pulseless electrical activity | 849 (39.3) | 810 (38.7) | 39 (56.5) | |
| Asystole | 1195 (55.3) | 1191 (56.9) | 4 (5.8) | |
| Blood gas pH value | 6.91 (6.88 to 7.03) | 6.90 (6.87 to 7.02) | 7.03 (6.98 to 7.16) | −0.890 |
| Extracorporeal CPR, n (%) | 146 (6.8) | 128 (6.1) | 18 (26.1) | −0.565 |
| Duration of CPR in ED, min | 28.0 (12.0 to 33.0) | 28.0 (13.0 to 33.0) | 10.0 (6.0 to 21.0) | 0.817 |
| Post‐ROSC interventions, n (%) | ||||
| Targeted temperature management | 290 (13.4) | 254 (12.1) | 36 (52.2) | 0.949 |
| Coronary angiography | 217 (10.0) | 163 (7.8) | 54 (78.3) | 2.026 |
| Percutaneous coronary intervention | 155 (7.2) | 120 (5.7) | 35 (50.7) | 1.154 |
| Clinical scoring systems | ||||
| UB‐ROSC score | −18.0 (−24.0 to −7.0) | −19.0 (−24.0 to −9.8) | 6.0 (−7.0 to 10.0) | −1.482 |
| P‐ROSC score | 38.0 (20.0 to 54.0) | 38.0 (19.0 to 52.0) | 69.0 (49.0 to 89.0) | −1.138 |
| SWAP score | 3.0 (2.0 to 4.0) | 3.0 (2.0 to 4.0) | 1.0 (0 to 2.0) | 1.390 |
| SCARS model, % | 1.35 (0.76 to 2.97) | 1.35 (0.76 to 2.97) | 2.97 (2.96 to 19.04) | −0.835 |
| Outcomes, n (%) | ||||
| ROSC | 1440 (66.6) | 1371 (65.5) | 69 (100) | −1.026 |
| Survival to hospital discharge | 156 (7.2) | 87 (4.2) | 69 (100) | −6.789 |
Data are presented as median (interquartile range) or count (proportion). CPR indicates cardiopulmonary resuscitation; ED, emergency department; EMS, emergency medical services; LMA, laryngeal mask airway; P‐ROSC, Prehospital Return of Spontaneous Circulation; ROSC, return of spontaneous circulation; SCARS, Swedish Cardiac Arrest Risk Score; SWAP, Shockable Rhythm–Witness–Age–pH; and UB‐ROSC, Utstein‐Based Return of Spontaneous Circulation.
We first validated these 4 scoring systems in predicting their respective intended outcomes, demonstrating that the performance ranged from poor to excellent (Table 2). The AUCs for UB‐ROSC score and P‐ROSC score were 0.60 (95% CI, 0.58–0.62) and 0.62 (95% CI, 0.59–0.64), respectively, in predicting sustained ROSC; the AUC for the SCARS model was 0.70 (95% CI, 0.68–0.72) in predicting survival to hospital discharge; and the AUC for the SWAP score was 0.82 (95% CI, 0.81–0.84) in predicting favorable neurological recovery at hospital discharge. Compared with the AUC values reported in the original studies, only the AUC of the validated SWAP score remained similar. In contrast, the AUC values of the UB‐ROSC and P‐ROSC scores and the SCARS model dropped substantially.
Table 2.
Validation of the Clinical Scoring Systems for the Predictive Outcomes
| Clinical scoring systems | Intended outcome, AUC reported in the validation cohort of the original study | Predictive outcomes | AUC | Sensitivity | Specificity | PPV | NPV |
|---|---|---|---|---|---|---|---|
| Statistical modeling | |||||||
| UB‐ROSC score: logistic regression model | Sustained ROSC until arrival at the ED, internal cross‐validation: 0.82 (0.80–0.84); external validation: 0.77 (0.74–0.80)* | Favorable neurological recovery | 0.85 (0.83–0.87) | 0.97 (0.90–1.00) | 0.32 (0.30–0.34) | 0.04 (0.04–0.05) | 1.00 (0.99–1.00) |
| Sustained ROSC | 0.60 (0.58–0.62) | 0.91 (0.90–0.93) | 0.11 (0.10–0.13) | 0.48 (0.47–0.48) | 0.60 (0.54–0.66) | ||
| Survival to discharge | 0.74 (0.72–0.76) | 0.94 (0.93–0.96) | 0.22 (0.20–0.25) | 0.52 (0.51–0.53) | 0.81 (0.77–0.85) | ||
| SWAP score: logistic regression model | Neurological outcome at hospital discharge, 0.877 (NA) | Favorable neurological recovery | 0.82 (0.81–0.84) | 0.99 (0.92–1.00) | 0.28 (0.26–0.30) | 0.04 (0.04–0.04) | 1.00 (0.99–1.00) |
| Sustained ROSC | 0.59 (0.56–0.61) | 1.00 (1.00–1.00) | 0 (0–0) | 0.47 (0.45–0.49) | NA | ||
| Survival to discharge | 0.76 (0.74–0.77) | 0.97 (0.95–0.98) | 0.29 (0.26–0.31) | 0.55 (0.54–0.55) | 0.91 (0.88–0.93) | ||
| Machine learning | |||||||
| P‐ROSC score: random forest–based model | Transient or sustained ROSC at the scene, 0.806 (0.799–0.814) | Favorable neurological recovery | 0.79 (0.78–0.81) | 0.99 (0.92–1.00) | 0.09 (0.08–0.11) | 0.03 (0.03–0.04) | 0.99 (0.97–1.00) |
| Sustained ROSC | 0.62 (0.59–0.64) | 0.93 (0.91–0.94) | 0.11 (0.09–0.13) | 0.48 (0.47–0.49) | 0.63 (0.57–0.69) | ||
| Survival to discharge | 0.72 (0.70–0.74) | 0.96 (0.94–0.97) | 0.15 (0.13–0.18) | 0.50 (0.49–0.51) | 0.80 (0.74–0.84) | ||
| SCARS model: extreme gradient boosting–based model | Survival at 30 days following OHCA, 0.972 (NA) | Favorable neurological recovery | 0.79 (0.77–0.81) | 0.97 (0.90–1.00) | 0.26 (0.24–0.28) | 0.04 (0.04–0.04) | 1.00 (0.99–1.00) |
| Sustained ROSC | 0.53 (0.51–0.55) | 1.00 (0.99–1.00) | 0 (0–0) | 0.47 (0.47–0.47) | NA | ||
| Survival to discharge | 0.70 (0.68–0.72) | 0.94 (0.92–0.95) | 0.20 (0.18–0.23) | 0.51 (0.50–0.52) | 0.78 (0.73–0.82) | ||
The calculation of sensitivity, specificity, PPV, and NPV was based on maintaining the lower boundary of the 95% CI above 0.90. AUC indicates area under the receiver operating characteristic curve; ED, emergency department; NA, not available; NPV, negative predictive value; OHCA, out‐of‐hospital cardiac arrest; PPV, positive predictive value; P‐ROSC, Prehospital Return of Spontaneous Circulation; ROSC, return of spontaneous circulation; SCARS, Swedish Cardiac Arrest Risk Score; SWAP, Shockable Rhythm–Witness–Age–pH; and UB‐ROSC, Utstein‐Based Return of Spontaneous Circulation.
In the original study, the researchers tested the performance of UB‐ROSC with a cohort of 10‐fold cross‐validation and a cohort of temporally split external validation, respectively.
As shown in Figure 3 and Table 3, when these 4 scoring systems were used to predict neurological outcomes, the performance ranged from fair to excellent. The AUC for UB‐ROSC score was 0.85 (0.83–0.87), SWAP score 0.82 (0.81–0.84), P‐ROSC score 0.79 (0.78–0.81), and SCARS model 0.79 (0.77–0.81) in predicting neurological outcomes. Furthermore, UB‐ROSC score significantly outperformed P‐ROSC score (P<0.001) and the SCARS model (P=0.007). As shown in Table 2, when the sensitivity was fixed at ≈0.9, the specificity ranged from 0.09 (P‐ROSC score) to 0.32 (UB‐ROSC score). When these 4 scoring systems were applied to sustained ROSC or survival to hospital discharge, the performance was poor and fair, respectively. Overall, among all the pairwise comparisons, statistical models outperformed ML‐based models in most of the comparisons with significant differences.
Figure 3. Comparison of ROC curves for each outcome among the 4 clinical scoring systems.
Comparisons among the 4 clinical scoring systems for the outcome of (A) favorable neurological outcome at hospital discharge; (B) sustained return of spontaneous circulation; and (C) survival at hospital discharge. AUC indicates area under the receiver operating characteristic curve; P‐ROSC, Prehospital Return of Spontaneous Circulation; ROC, receiver operating characteristic; SCARS, Swedish Cardiac Arrest Risk Score; SWAP, Shockable Rhythm–Witness–Age–pH; and UB‐ROSC, Utstein‐Based Return of Spontaneous Circulation.
Table 3.
Comparison of Predictive Performance Between Different Clinical Scoring Systems on Primary and Secondary Outcomes
| Clinical scoring system | Performance | Statistical models | Machine learning‐based models | Overall comparison | ||
|---|---|---|---|---|---|---|
| UB‐ROSC score | SWAP score | P‐ROSC score | SCARS model | |||
| Favorable neurological recovery | AUC | 0.85 (0.83 to 0.87) | 0.82 (0.81 to 0.84) | 0.79 (0.78 to 0.81) | 0.79 (0.77 to 0.81) | P value <0.001 |
| UB‐ROSC score | 0.85 (0.83 to 0.87) | |||||
| SWAP score | 0.82 (0.81 to 0.84) | 0.03 (−0.01 to 0.07), P=0.17 | ||||
| P‐ROSC score | 0.79 (0.78 to 0.81) | 0.06 (0.03 to 0.09), P<0.001* | 0.03 (−0.01 to 0.07), P: 0.14 | |||
| SCARS model | 0.79 (0.77 to 0.81) | 0.06 (0.02 to 0.10), P=0.007* | 0.03 (−0.02 to 0.08), P=0.22 | 0.002 (−0.06 to 0.06), P=0.94 | ||
| Sustained ROSC | 0.60 (0.58 to 0.62) | 0.59 (0.56 to 0.61) | 0.62 (0.59 to 0.64) | 0.53 (0.51 to 0.55) | P value <0.001 | |
| UB‐ROSC score | 0.60 (0.58 to 0.62) | |||||
| SWAP score | 0.59 (0.56 to 0.61) | 0.01 (−0.02 to 0.02), P=0.14 | ||||
| P‐ROSC score | 0.62 (0.59 to 0.64) | −0.02 (−0.03 to 0.01), P=0.14 | −0.03 (−0.05 to −0.01), P‐0.003 | |||
| SCARS model | 0.53 (0.51 to 0.55) | 0.07 (0.05 to 0.09), P<0.001* | 0.06 (0.04 to 0.08), P<0.001* | 0.09 (0.06 to 0.11), P<0.001 | ||
| Survival to discharge | 0.74 (0.72 to 0.76) | 0.76 (0.74 to 0.77) | 0.72 (0.70 to 0.74) | 0.70 (0.68 to 0.72) | P value <0.001 | |
| UB‐ROSC score | 0.74 (0.72 to 0.76) | |||||
| SWAP score | 0.76 (0.74 to 0.77) | −0.01 (−0.04 to 0.02), P=0.38 | ||||
| P‐ROSC score | 0.72 (0.70 to 0.74) | 0.02 (−0.007 to 0.05), P=0.13 | 0.04 (0.006 to 0.07), P=0.02 | |||
| SCARS model | 0.70 (0.68 to 0.72) | 0.04 (0.006 to 0.08), P=0.02 | 0.05 (0.02 to 0.09), P=0.004* | 0.02 (−0.03 to 0.07), P=0.44 | ||
AUC is presented as the effect estimate (95% CI). The differences in AUC (95% CI) and P value in each cell represents the comparison between the AUC of the clinical scoring systems of the column and the row. The overall comparison indicates the comparison among the 4 clinical scoring systems for each outcome. The significance levels of the overall and pairwise comparisons were determined at a 2‐tailed P value <0.05 and < 0.008, respectively. Italic type indicates the machine learning‐based model significantly outperforms statistical models. AUC indicates area under the receiver operating characteristic curve; P‐ROSC, Prehospital Return of Spontaneous Circulation; SCARS, Swedish Cardiac Arrest Risk Score; SWAP, Shockable Rhythm–Witness–Age–pH; and UB‐ROSC, Utstein‐Based Return of Spontaneous Circulation.
The statistical models significantly outperforms machine learning‐based model.
The gray shades indicate no comparions made between the two scores.
Table S1 demonstrates that patients resuscitated after 2020 had lower rates of tracheal intubation, experienced longer times to EMS arrival and prehospital CPR durations, exhibited lower blood gas pH values upon ED arrival, and had lower rates of favorable neurological outcomes and survival. Nevertheless, Table 4 shows that the predictive performance did not differ significantly between patients resuscitated before and after 2020.
Table 4.
Predictive Performance of Different Clinical Scoring Systems in Patients Resuscitated Before and After 2020
| Clinical scoring system | Pre‐2020 resuscitation, AUC (95% CI) | Post‐2020 resuscitation, AUC (95% CI) | P value |
|---|---|---|---|
| Favorable neurological recovery | |||
| UB‐ROSC score | 0.88 (0.86–0.90) | 0.81 (0.79–0.83) | 0.18 |
| P‐ROSC score | 0.82 (0.80–0.85) | 0.76 (0.74–0.78) | 0.30 |
| SWAP score | 0.83 (0.81–0.86) | 0.79 (0.77–0.81) | 0.38 |
| SCARS model | 0.81 (0.78–0.83) | 0.77 (0.74–0.79) | 0.42 |
| Sustained ROSC | |||
| UB‐ROSC score | 0.59 (0.55–0.62) | 0.63 (0.60–0.66) | 0.09 |
| P‐ROSC score | 0.58 (0.54–0.61) | 0.63 (0.60–0.66) | 0.03 |
| SWAP score | 0.61 (0.58–0.64) | 0.61 (0.59–0.64) | 0.87 |
| SCARS model | 0.58 (0.55–0.61) | 0.57 (0.54–0.60) | 0.61 |
| Survival to discharge | |||
| UB‐ROSC score | 0.76 (0.73–0.79) | 0.73 (0.70–0.75) | 0.46 |
| P‐ROSC score | 0.72 (0.69–0.75) | 0.73 (0.70–0.75) | 0.91 |
| SWAP score | 0.77 (0.74–0.80) | 0.73 (0.71–0.76) | 0.34 |
| SCARS model | 0.72 (0.69–0.75) | 0.68 (0.65–0.71) | 0.36 |
AUC indicates area under the receiver operating characteristic curve; P‐ROSC, Prehospital Return of Spontaneous Circulation; SCARS, Swedish Cardiac Arrest Risk Score; SWAP, Shockable Rhythm–Witness–Age–pH; and UB‐ROSC, Utstein‐Based Return of Spontaneous Circulation.
DISCUSSION
Main Findings
First, in the validation of the intended outcomes for the 4 clinical scoring systems, only the SWAP score demonstrated excellent predictive accuracy for neurological outcomes, mirroring the performance noted in its original study. Second, when focusing on neurological outcomes, the UB‐ROSC score significantly outperformed both the P‐ROSC score and the SCARS model in predictive capability. Finally, the performance of these scoring systems remained largely unchanged when applied to patients resuscitated after 2020.
External Validation of the Clinical Scoring Systems for Their Intended Outcomes
In validating the intended outcomes, the SWAP score and the SCARS model displayed excellent and fair accuracy, respectively, in predicting neurological outcomes (AUC, 0.82) and survival at hospital discharge (AUC, 0.70) (Table 2). The performance of the SWAP score in predicting neurological outcomes was consistent with that in its original study, 7 which reported an AUC of 0.877. Conversely, the SCARS model's performance in predicting survival substantially decreased from its initially reported AUC of 0.972 9 to 0.70 in our evaluation. In terms of predicting sustained ROSC, both the UB‐ROSC (AUC, 0.60) and the P‐ROSC (AUC, 0.62) scores demonstrated reduced performance compared with those in their original studies (AUC: UB‐ROSC 6 : internal cross‐validation: 0.82, and external validation: 0.77; P‐ROSC 8 : 0.806). This observation suggests that predictive performance may be influenced by the specific outcome being predicted more than the specific model being assessed.
The UB‐ROSC and P‐ROSC scores were specifically developed to estimate the likelihood of ROSC upon ED arrival and during prehospital resuscitation, respectively. Variability in prehospital CPR tactics by EMS, including the “scoop‐and‐run” versus “stay‐and‐play” approaches or the implementation of termination‐of‐resuscitation policies, could probably affect the incidence of prehospital ROSC due to disparities in the level of care provided. For example, the rate of medication administration or the use of advanced airway management techniques might vary considerably across different EMS systems, leading to significant differences in the predictive accuracy of these scoring systems in predicting prehospital ROSC. The original study by Liu et al 8 reported considerable performance variability (AUC, 0.671–0.859) of P‐ROSC score across the study communities, indicating a possible association between specific EMS practices and a model's predictive capacity. In contrast, the prediction of survival and neurological outcomes at hospital discharge may be more consistent, as by this stage patients are expected to have received a comparable standard of care across all stages of the chain of survival. 13 , 25 , 26 This consistency likely explains the better validated performance of the SWAP score and SCARS model in our data set.
Comparison of the Clinical Scoring Systems on the Basis of the Neurological Outcomes
The results from the previous stage suggested that neurological outcomes may provide more consistent prediction results across different populations compared with ROSC or survival. Therefore, in this phase, all included scoring systems were compared on the basis of their predictions of neurological recovery, even though the UB‐ROSC and P‐ROSC scores were not originally developed for this purpose. This approach allowed us to examine how predictive accuracy is influenced by the different architectures of the scoring systems on an even ground when using similar pre‐ROSC variables. Our findings indicate that the UB‐ROSC and SWAP scores performed excellently, while the P‐ROSC and SCARS models demonstrated fair success in predicting neurological outcomes (Table 2). Additionally, the UB‐ROSC score significantly outperformed both the P‐ROSC and SCARS models (Table 3).
Nevertheless, although there were statistically significant differences in AUC among the included scoring systems, there may be few clinical differences. The comparison of predictive performance in AUC is less relevant to clinical care. In a given clinical context, the AUC of the scoring systems can be transformed into sensitivity or specificity, which may be more familiar to clinicians. All these scoring systems use only pre‐ROSC factors to predict outcomes, making them suitable for front‐line health care providers to stratify patients into different risk categories, thereby facilitating decision‐making processes, such as transfer from the scene to the hospital, implementation of extracorporeal CPR, or termination of resuscitation. However, current guidelines 13 , 14 do not suggest any decision threshold for these therapeutic decisions. A recent meta‐analysis by Smyth et al 4 revealed pooled sensitivity and specificity of 0.62 and 0.88, respectively, for 19 studies on universal termination‐of‐resuscitation rules. Although the 4 included scoring systems were not specifically designed for termination of resuscitation, we chose a sensitivity of 0.9 as the threshold (similar to the specificity of 0.88 for a poor outcome) to compare the performance of the included scoring systems. As shown in Table 2, when sensitivity was fixed, the specificity ranged between 0.09 (P‐ROSC score) and 0.32 (UB‐ROSC score), suggesting that the UB‐ROSC score might be a preferred scoring system in this context.
The higher predictive performance of statistical models (UB‐ROSC) compared with ML‐based models (P‐ROSC and SCARS model) might be attributable to the fact that both types of models were developed using structured data. The inherent advantages of ML models in analyzing high‐dimensional data and nonlinear relationships were not fully leveraged due to the reliance on structured data. Incorporating unstructured data, such as intra‐arrest changes in end‐tidal CO2 levels 27 and heart rhythms., 28 could potentially improve the performance and broaden the applicability of ML algorithms beyond that of statistical models.
To our knowledge, this study represents the first attempt to validate these scoring systems with different model architectures in an independent data set. External validations, particularly for ML‐based models, are scarce 29 , 30 and often require specialized computer science expertise for implementation and testing. Fortunately, the researchers behind the P‐ROSC score and the SCARS model have made their algorithms accessible and user friendly, facilitating public testing and comparison with other prediction models.
The current study results suggest that despite different model architectures, the predictive performance of the included scoring systems was similar. Therefore, health care providers could determine which scoring system to use on the basis of the pre‐ROSC variables they have available. For example, the calculation of the UB‐ROSC score requires the classification of pathogenesis, and the calculation of the SWAP score necessitates the examination of blood pH. If the pathogenesis of OHCA or the blood pH is not immediately available to the clinicians, they may adopt the P‐ROSC score, which requires the least information for assessing OHCA outcomes. The similar predictive performance among these scoring systems demonstrated in our study allows clinicians to select the appropriate method for prognostication on the basis of the information they have at hand.
Influence of the COVID‐19 Pandemic
Research indicates that OHCA outcomes deteriorated significantly during the COVID‐19 pandemic compared with those preceding it, 31 potentially as a consequence of the modified prehospital practices, 32 , 33 as shown in Table S1. Recently, Caputo et al 34 used OHCA data from the same geographic area as focused on in the initial study of UB‐ROSC score by Baldi et al, 6 but covering a later time frame (2019–2021) to reevaluate the UB‐ROSC score. They reported a lower AUC of 0.71, compared with 0.82 in the initial study. 6 This reduction in AUC raised concerns regarding the applicability of models developed before the COVID‐19 pandemic to the postpandemic period. Table 4 suggests that, despite these shifts, the performance of the scoring systems remained largely unchanged. This consistency suggests that these 4 models effectively captured the essential variables for predicting OHCA outcomes. Thus, even though COVID‐19 was not considered during the development of these models, the stability and consistency of these models may ensure their applicability in the postpandemic era.
Study Limitations
First, only 4 clinical scoring systems were included for external validation. Some scoring systems were not considered because their inclusion could have further reduced patient numbers due to missing values. Second, some differences were observed in the eligibility criteria among the 4 scoring systems evaluated. Specifically, the SWAP score uses the most stringent criteria, excluding OHCA cases with circumstantial causes and requiring the measurement of blood gas pH values. To facilitate comparison, we adapted our cohort to align with the criteria used by the SWAP score, which may inadvertently bias the results in favor of the SWAP score's performance relative to the other scoring systems. Third, there is a possibility that some of our patients were included in the data set used to develop the P‐ROSC score, as Taiwan was among the communities analyzed in its development. Consequently, further external validation using independent data sets from diverse regions is warranted.
CONCLUSIONS
In this external validation, both statistical and ML‐based models demonstrated excellent and fair performance, respectively, in predicting neurological outcomes despite different model architectures. The predictive performance of these scoring systems remained consistent, even when applied to patients resuscitated after the outbreak of COVID‐19. Based on the available information, clinicians may use appropriate scoring systems for risk stratification and decision making in therapeutic strategies.
Sources of Funding
Dr Wang received a grant (113‐R0006) from the National Taiwan University Hospital and a grant (113‐2326‐B‐002‐009‐MY3) from the National Science and Technology Council, Taiwan. Both National Taiwan University Hospital and National Science and Technology Council had no involvement in designing the study, collecting, analyzing or interpreting the data, writing the manuscript, or deciding whether to submit the manuscript for publication. This project was also supported by grants from the National Health Research Institutes (NHRI‐EX113‐11332PI), the National Science and Technology Council (NSTC 112‐2314‐B‐002‐264).
Disclosures
None.
Supporting information
Table S1
Acknowledgments
We thank the staff of the 3rd Core Lab, Department of Medical Research, National Taiwan University Hospital for technical support.
This manuscript was sent to Shaan Khurshid, MD, MPH, Assistant Editor, for review by expert referees, editorial decision, and final disposition.
Supplemental Material is available at https://www.ahajournals.org/doi/suppl/10.1161/JAHA.124.037088
For Sources of Funding and Disclosures, see page 10.
Contributor Information
Yao‐De Fang, Email: darinmonkey@gmail.com.
Chu‐Lin Tsai, Email: chulintsai@ntu.edu.tw.
References
- 1. Berdowski J, Berg RA, Tijssen JG, Koster RW. Global incidences of out‐of‐hospital cardiac arrest and survival rates: systematic review of 67 prospective studies. Resuscitation. 2010;81:1479–1487. doi: 10.1016/j.resuscitation.2010.08.006 [DOI] [PubMed] [Google Scholar]
- 2. Ong ME, Shin SD, De Souza NN, Tanaka H, Nishiuchi T, Song KJ, Ko PC, Leong BS, Khunkhlai N, Naroo GY, et al. Outcomes for out‐of‐hospital cardiac arrests across 7 countries in Asia: the pan Asian resuscitation outcomes study (PAROS). Resuscitation. 2015;96:100–108. doi: 10.1016/j.resuscitation.2015.07.026 [DOI] [PubMed] [Google Scholar]
- 3. Kiss B, Nagy R, Kói T, Harnos A, Édes IF, Ábrahám P, Mészáros H, Hegyi P, Zima E. Prediction performance of scoring systems after out‐of‐hospital cardiac arrest: a systematic review and meta‐analysis. PLoS One. 2024;19:e0293704. doi: 10.1371/journal.pone.0293704 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Smyth MA, Gunson I, Coppola A, Johnson S, Greif R, Lauridsen KG, Taylor‐Philips S, Perkins GD. Termination of resuscitation rules and survival among patients with out‐of‐hospital cardiac arrest: a systematic review and meta‐analysis. JAMA Netw Open. 2024;7:e2420040. doi: 10.1001/jamanetworkopen.2024.20040 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Cheng P, Yang P, Zhang H, Wang H. Prediction models for return of spontaneous circulation in patients with cardiac arrest: a systematic review and critical appraisal. Emerg Med Int. 2023;2023:6780941. doi: 10.1155/2023/6780941 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Baldi E, Caputo ML, Savastano S, Burkart R, Klersy C, Benvenuti C, Sgromo V, Palo A, Cianella R, Cacciatore E. An Utstein‐based model score to predict survival to hospital admission: the UB‐ROSC score. Int J Cardiol. 2020;308:84–89. doi: 10.1016/j.ijcard.2020.01.032 [DOI] [PubMed] [Google Scholar]
- 7. Shih HM, Chen YC, Chen CY, Huang FW, Chang SS, Yu SH, Wu SY, Chen WK. Derivation and validation of the SWAP score for very early prediction of neurologic outcome in patients with out‐of‐hospital cardiac arrest. Ann Emerg Med. 2019;73:578–588. doi: 10.1016/j.annemergmed.2019.01.017 [DOI] [PubMed] [Google Scholar]
- 8. Liu N, Liu M, Chen X, Ning Y, Lee JW, Siddiqui FJ, Saffari SE, Ho AFW, Do Shin S, Ma MH‐M. Development and validation of an interpretable prehospital return of spontaneous circulation (P‐ROSC) score for patients with out‐of‐hospital cardiac arrest using machine learning: a retrospective study. EClinicalMedicine. 2022;48:101422. doi: 10.1016/j.eclinm.2022.101422 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Hessulf F, Bhatt DL, Engdahl J, Lundgren P, Omerovic E, Rawshani A, Helleryd E, Dworeck C, Friberg H, Redfors B, et al. Predicting survival and neurological outcome in out‐of‐hospital cardiac arrest using machine learning: the SCARS model. EBioMedicine. 2023;89:104464. doi: 10.1016/j.ebiom.2023.104464 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med. 2015;162:55–63. doi: 10.7326/m14-0697 [DOI] [PubMed] [Google Scholar]
- 11. Sun JT, Chiang WC, Hsieh MJ, Huang EP, Yang WS, Chien YC, Wang YC, Lee BC, Sim SS, Tsai KC, et al. The effect of the number and level of emergency medical technicians on patient outcomes following out of hospital cardiac arrest in Taipei. Resuscitation. 2018;122:48–53. doi: 10.1016/j.resuscitation.2017.11.048 [DOI] [PubMed] [Google Scholar]
- 12. Chiang W‐C, Ko PC‐I, Wang H‐C, Yang C‐W, Shih F‐Y, Hsiung K‐H, Ma MH‐M. EMS in Taiwan: past, present, and future. Resuscitation. 2009;80:9–13. doi: 10.1016/j.resuscitation.2008.10.010 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Panchal AR, Bartos JA, Cabañas JG, Donnino MW, Drennan IR, Hirsch KG, Kudenchuk PJ, Kurz MC, Lavonas EJ, Morley PT, et al. Part 3: adult basic and advanced life support: 2020 American Heart Association guidelines for cardiopulmonary resuscitation and emergency cardiovascular care. Circulation. 2020;142:S366–s468. doi: 10.1161/cir.0000000000000916 [DOI] [PubMed] [Google Scholar]
- 14. Soar J, Böttiger BW, Carli P, Couper K, Deakin CD, Djärv T, Lott C, Olasveengen T, Paal P, Pellis T, et al. European resuscitation council guidelines 2021: adult advanced life support. Resuscitation. 2021;161:115–151. doi: 10.1016/j.resuscitation.2021.02.010 [DOI] [PubMed] [Google Scholar]
- 15. Organization WH . WHO director‐General's opening remarks at the media briefing on COVID‐19. 2022;30. [Google Scholar]
- 16. Tran K, Cimon K, Severn M, Pessoa‐Silva CL, Conly J. Aerosol generating procedures and risk of transmission of acute respiratory infections to healthcare workers: a systematic review. PLoS One. 2012;7:e35797.doi: 10.1371/journal.pone.0035797 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Yang WS, Hou SW, Lee BC, Chiang WC, Chien YC, Chen SY, Ma MH. Taipei azalea ‐ Supraglottic airways (SGA) preassembled with high‐efficiency particulate air (HEPA) filters to simplify prehospital airway management for patients with out‐of‐hospital cardiac arrests (OHCA) during coronavirus disease 2019 (COVID‐19) pandemic. Resuscitation. 2020;151:3–5. doi: 10.1016/j.resuscitation.2020.03.021 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Soar J, Monsieurs KG, Ballance JH, Barelli A, Biarent D, Greif R, Handley AJ, Lockey AS, Richmond S, Ringsted C, et al. European resuscitation council guidelines for resuscitation 2010 section 9. Principles of education in resuscitation. Resuscitation. 2010;81:1434–1444. doi: 10.1016/j.resuscitation.2010.08.014 [DOI] [PubMed] [Google Scholar]
- 19. Bhanji F, Mancini ME, Sinz E, Rodgers DL, McNeil MA, Hoadley TA, Meeks RA, Hamilton MF, Meaney PA, Hunt EA, et al. Part 16: education, implementation, and teams: 2010 American Heart Association guidelines for cardiopulmonary resuscitation and emergency cardiovascular care. Circulation. 2010;122:S920–S933. doi: 10.1161/circulationaha.110.971135 [DOI] [PubMed] [Google Scholar]
- 20. Perkins GD, Jacobs IG, Nadkarni VM, Berg RA, Bhanji F, Biarent D, Bossaert LL, Brett SJ, Chamberlain D, de Caen AR, et al. Cardiac arrest and cardiopulmonary resuscitation outcome reports: update of the Utstein resuscitation registry templates for out‐of‐hospital cardiac arrest: a statement for healthcare professionals from a task force of the international liaison committee on resuscitation (American Heart Association, European resuscitation council, Australian and New Zealand council on resuscitation, Heart and Stroke Foundation of Canada, InterAmerican Heart Foundation, resuscitation Council of Southern Africa, resuscitation Council of Asia); and the American Heart Association emergency cardiovascular care committee and the council on cardiopulmonary, critical care, perioperative and resuscitation. Circulation. 2015;132:1286–1300. doi: 10.1161/cir.0000000000000144 [DOI] [PubMed] [Google Scholar]
- 21. Perkins GD, Jacobs IG, Nadkarni VM, Berg RA, Bhanji F, Biarent D, Bossaert LL, Brett SJ, Chamberlain D, de Caen AR, et al. Cardiac arrest and cardiopulmonary resuscitation outcome reports: update of the Utstein resuscitation registry templates for out‐of‐hospital cardiac arrest: a statement for healthcare professionals from a task force of the international liaison committee on resuscitation (American Heart Association, European resuscitation council, Australian and New Zealand council on resuscitation, Heart and Stroke Foundation of Canada, InterAmerican Heart Foundation, resuscitation Council of Southern Africa, resuscitation Council of Asia); and the American Heart Association emergency cardiovascular care committee and the council on cardiopulmonary, critical care, perioperative and resuscitation. Resuscitation. 2015;96:328–340. doi: 10.1016/j.resuscitation.2014.11.002 [DOI] [PubMed] [Google Scholar]
- 22. Yang D, Dalton JE. A unified approach to measuring the effect size between two groups using SAS. Paper/Poster Presented. 2012.
- 23. DeLong ER, DeLong DM, Clarke‐Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44:837–845. doi: 10.2307/2531595 [DOI] [PubMed] [Google Scholar]
- 24. Hosmer DW Jr, Lemeshow S, Sturdivant RX. Applied Logistic Regression. John Wiley & Sons; 2013. [Google Scholar]
- 25. Ling DA, Huang CH, Chen WJ, Chuang PY, Chang WT, Sung CW, Chen WT, Ong HN, Tsai MS. Impact of protocolized postarrest care with targeted temperature management on the outcomes of cardiac arrest survivors without temperature management. Ann Med. 2022;54:63–70. doi: 10.1080/07853890.2021.2016941 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Nolan JP, Sandroni C, Böttiger BW, Cariou A, Cronberg T, Friberg H, Genbrugge C, Haywood K, Lilja G, Moulaert VRM, et al. European resuscitation council and European Society of Intensive Care Medicine Guidelines 2021: post‐resuscitation care. Resuscitation. 2021;161:220–269. doi: 10.1016/j.resuscitation.2021.02.012 [DOI] [PubMed] [Google Scholar]
- 27. Wang CH, Lu TC, Tay J, Wu CY, Wu MC, Chong KM, Chou EH, Tsai CL, Huang CH, Ma MH, et al. Association between trajectories of end‐tidal carbon dioxide and return of spontaneous circulation among emergency department patients with out‐of‐hospital cardiac arrest. Resuscitation. 2022;177:28–37. doi: 10.1016/j.resuscitation.2022.06.013 [DOI] [PubMed] [Google Scholar]
- 28. Wang CH, Lu TC, Tay J, Wu CY, Wu MC, Su PI, Huang CY, Tsai CL, Huang CH, Chen WJ. Prognostic impact of heart rhythm shockability trajectory in out‐of‐hospital cardiac arrest: a multicenter retrospective study. Circ Cardiovasc Qual Outcomes. 2024;17(7):e010649. doi: 10.1161/circoutcomes.123.010649 [DOI] [PubMed] [Google Scholar]
- 29. Ioannidis JP. More than a billion people taking statins?: potential implications of the new cardiovascular guidelines. JAMA. 2014;311:463–464. doi: 10.1001/jama.2013.284657 [DOI] [PubMed] [Google Scholar]
- 30. Ioannidis JP, Tzoulaki I. What makes a good predictor?: the evidence applied to coronary artery calcium score. JAMA. 2010;303:1646–1647. doi: 10.1001/jama.2010.503 [DOI] [PubMed] [Google Scholar]
- 31. Baldi E, Klersy C, Chan P, Elmer J, Ball J, Counts CR, Rosell Ortiz F, Fothergill R, Auricchio A, Paoli A, et al. The impact of COVID‐19 pandemic on out‐of‐hospital cardiac arrest: an individual patient data meta‐analysis. Resuscitation. 2024;194:110043. doi: 10.1016/j.resuscitation.2023.110043 [DOI] [PubMed] [Google Scholar]
- 32. Tanaka C, Tagami T, Kaneko J, Kitamura N, Yasunaga H, Aso S, Takeda M, Kuno M. Impact of the COVID‐19 pandemic on prehospital and in‐hospital treatment and outcomes of patients after out‐of‐hospital cardiac arrest: a Japanese multicenter cohort study. BMC Emerg Med. 2024;24:12. doi: 10.1186/s12873-024-00929-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. Armour R, Ghamarian E, Helmer J, Buick JE, Thorpe K, Austin M, Bacon J, Boutet M, Cournoyer A, Dionne R, et al. Impact of the COVID‐19 pandemic on Canadian emergency medical system management of out‐of‐hospital cardiac arrest: a retrospective cohort study. Resuscitation. 2024;194:110054. doi: 10.1016/j.resuscitation.2023.110054 [DOI] [PubMed] [Google Scholar]
- 34. Caputo ML, Baldi E, Burkart R, Wilmes A, Cresta R, Benvenuti C, Oezkartal T, Cianella R, Primi R, Currao A, et al. Validation of Utstein‐based score to predict return of spontaneous circulation (UB‐ROSC) in patients with out‐of‐hospital cardiac arrest. Resuscitation. 2024;197:110113. doi: 10.1016/j.resuscitation.2024.110113 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Table S1