Simulation-based assessment of trainee’s performance in post-cardiac arrest resuscitation

Abstract

Objectives

To assess trainees’ performance in managing a patient with post-cardiac arrest complicated by status epilepticus.

Methods

In this prospective, observational, single-center simulation-based study, trainees ranging from sub interns to critical care fellows evaluated and managed a post cardiac arrest patient, complicated by status epilepticus. Critical action items were developed by a modified Delphi approach based on American Heart Association guidelines and the Neurocritical Care Society’s Emergency Neurological Life Support protocols. The primary outcome measure was the critical action item sum score. We sought validity evidence to support our findings by including attending neurocritical care physicians and comparing performance across four levels of training.

Results

Forty-nine participants completed the simulation. The mean sum of critical actions completed by trainees was 10/21 (49%). Eleven (22%) trainees verbalized a differential diagnosis for the arrest. Thirty-two (65%) reviewed the electrocardiogram, recognized it as abnormal, and consulted cardiology. Forty trainees (81%) independently decided to start temperature management, but only 20 (41%) insisted on it when asked to reconsider. There was an effect of level of training on critical action checklist sum scores (novice mean score [standard deviation (SD)] = 4.8(1.8) vs. intermediate mean score (SD) = 10.4(2.1) vs. advanced mean score (D) = 11.6(3.0) vs. expert mean score (SD) = 14.7(2.2))

Conclusions

High-fidelity manikin-based simulation holds promise as an assessment tool in the performance of post-cardiac arrest care.

Introduction

Up to 22% of patients survive to hospital admission following out-of-hospital cardiac arrest (OHCA).¹ The majority of deaths following admission occur due to withdrawal of life sustaining therapies from perceived poor neurological prognosis.² Post cardiac arrest care is a critical component of the chain of survival that requires a comprehensive, structured approach to diagnose the underlying etiology of arrest, prevent recurrent arrest, and mitigate hypoxic-ischemic injury to the brain.³ Guidelines recommend diagnosis and treatment of the underlying cause, haemodynamic support, appropriate mechanical ventilation, temperature management, diagnosis and treatment of seizures, and surveillance for and treatment of infection.³

It is unknown how often guideline-concordant post-cardiac arrest care is actually delivered, but there is reason for concern.⁴ Much research in quality of post-arrest care delivery has focused on temperature management, where deviations from guideline-recommended care are common.5., 6., 7., 8. Large, teaching hospitals, with higher volumes of patients who suffered OHCA provide temperature management to eligible patients more often.8., 9., 10. Less is known about the performance of other guideline-recommended care measures.

In large, academic hospitals, trainees take part in post-cardiac arrest care as part of a large, multidisciplinary team. It is possible that the diffusion of responsibility impairs trainees’ education and contributes to suboptimal delivery of post-cardiac arrest care in smaller, community hospitals where recent graduates may have less resources. For decades, simulation-based education has proven to be a highly reliable and valid assessment of trainees’ performance during resuscitative efforts.¹¹ Trainees that pass simulation-based assessments of advanced cardiac life support deliver more guideline-concordant care during actual advanced cardiac life support events.¹² Data regarding trainees’ performance in post-cardiac arrest care, which contributes substantially to patient outcomes, has previously not been studied.

In this study, we aimed to assess the performance of trainees with different training levels in managing post-cardiac arrest care using a high-fidelity manikin-based simulation. We also sought to obtain evidence regarding the validity of our findings.

Methods

Setting and study design

This prospective, observational, single-center high-fidelity simulation-based study was performed between February 2018 and 2021 at the Critical Care and Trauma Simulation Center at Shock Trauma Center, University of Maryland Medical Center, a tertiary care hospital in Baltimore, Maryland. Two neuro-intensivists conducted the simulation. Participants included fourth year medical students (sub-interns), interns, residents, and critical care fellows with backgrounds in internal medicine, emergency medicine, neurology, and surgery rotating in the Neurocritical Care Unit. Attending neuro-intensivists also participated in the study to help establish validity of the assessment as described below but were not included in the primary analysis. Participation was voluntary with formative feedback as debriefing at the end of the session.¹³ Prior to the simulation session, we asked participants to complete a survey on demographics, level of training, prior experience across a variety of neurological emergencies, and perceived skill level for neurological emergencies. We oriented the participants to the manikin and the equipment available to them via a walkthrough of the manikin’s functionality, prior to the simulation session, for a duration of approximately 5–10 min. We did not provide information regarding the content of the simulation before the case started.

Clinical simulation case and trainee assessment development

We developed the case and critical actions checklist using a modified Delphi method, which represents a group consensus strategy that systematically uses literature review, opinion of stakeholders and the judgment of experts within a field to reach agreement. Briefly, a board-certified medical intensivist with additional certification in neurocritical care initially developed the case and critical actions checklist. A board-certified neurologist with additional certification in neurocritical care reviewed and revised the case and critical actions checklist. Both case writers received training in simulation case development through the Center for Medical Simulation (Boston, MA.). We modeled the critical action checklist after the relevant Neurocritical Care Society’s Emergency Neurological Life Support (ENLS) protocols cross-referenced with corresponding guidelines from the American Heart Association, the Epilepsy Foundation/American Epilepsy Society, and the Neurocritical Care Society.14., 15., 16., 17., 18. A board-certified neurologist with additional board-certification in epilepsy and clinical neurophysiology and another board-certified neurologist with additional certification in neurocritical care at a separate institution reviewed the case and critical actions checklist. The final case and critical actions checklist represented a consensus that suggests content validity.¹⁹

Simulator and simulation environment

We conducted the session in a simulated emergency resuscitation bay that was equipped with a programmed manikin, airway equipment (nasal cannula, non-rebreather mask, bag valve mask, laryngoscope, endotracheal tubes, Bougie, and ventilator), medications (intravenous fluids, rapid sequence medications, lorazepam, midazolam, diazepam, propofol, ketamine, fosphenytoin, levetiracetam, valproic acid, lacosamide, phenobarbital, thiamine, and dextrose), arctic sun temperature management device, and electroencephalogram (EEG) machine. We displayed the patient’s demographic information, presenting complaint, emergency medical services (EMS) report, laboratory studies (complete blood count, chemistry panel, troponin), 12 lead electrocardiography (ECG), images (chest radiography, CT head without contrast) and continuous EEG (cEEG) on a high-resolution display within the resuscitation bay when requested by the participant. We embedded a simulated nurse, who acted on instructions from the simulation operator via an earpiece, to assist the participants.

We used the high-fidelity manikin, SimMan 3G (Laerdal; Wappinger Falls, NY), which can (in contrast to a low-fidelity manikin) display a range of physiological and neurological responses such as speech via an internal speaker system, pupillary responses, tonic–clonic seizures, fasciculations, and abnormal respirations. It is unable to display certain aspects of the neurological exam such as movement of the eyes, motor strength, and reflexes. We explained these limitations to the participants during their orientation. The participants were able to obtain this information from the embedded simulated nurse. During the scenario, the facilitator altered the parameters of the displayed vital signs including heart rate, blood pressure, oxygen saturation, temperature, and respirations. Prior to the scenario, the manikin was intubated and mechanically ventilated. We recorded the case on video for review by the raters.

Simulated clinical scenario and response process

The participants performed individually. We informed them that they were the primary provider for a young man who appeared to be in his third decade of life that was found unresponsive in cardiac arrest by EMS. The initial rhythm was ventricular fibrillation, the patient was defibrillated, and regained spontaneous circulation within 15 min. Immediately following the arrest, the patient had a witnessed seizure that ceased following two doses of lorazepam. In the field, the patient received rapid sequence intubation with etomidate and rocuronium for airway protection. No further patient information was available. We expected participants to manage the patient by evaluating for a cause of the arrest through laboratory studies, ECG, and neuroimaging, obtain appropriate consultations, and start temperature management (including management of shivering).

The next phase of the case occurred the following morning in the intensive care unit. In this phase, we expected participants to recognize and treat non-convulsive status epilepticus. The scenario ended when the participant adequately treated non-convulsive status epilepticus, as demonstrated by burst suppression on the EEG (Appendix 1).

Per Downing, we defined the response process as evidence of data integrity such that all sources of error associated with the test administration are controlled or eliminated to the maximum extent possible.²⁰ We achieved this through participant pre-briefing to ensure understanding and expectations, simulation operator training, the complementary use of critical action checklists and global rating scales, as well as rater training and calibration.¹⁹

Outcomes

The primary outcome measure was the sum score of 21 critical action items from the critical actions’ checklist (Appendix 2). Two independent observers, one certified in neurocritical care and the other certified in emergency medicine, reviewed the videotaped recordings of each participant’s simulation, coded completion of the critical action items, and rated the participants on a global rating scale from 1 to 5 representing novice through expert. One observer was aware of the participants’ level of training while the other observer was blinded.

Validity evidence

As noted above, we have previously reported on two sources of validity evidence, including the domains of content evidence and response process.¹⁹ We sought to provide further validity evidence to support our findings in the domains of internal structure and relationship to other variables.²⁰ Internal structure relates to the scoring properties such as reliability and reproducibility. Interrater reliability was calculated for critical action item sum scores and global rating scales. We evaluated relationships to other variables by evaluating checklist performance compared to level of training, training background, self-rated experience, and performance on a multiple-choice pretest that assessed fund of knowledge in neurocritical care.

For the comparison to level of training, participants were divided into four levels ranging from novice to expert. Novice participants included medical students and neurosurgery interns. Intermediate participants included postgraduate year (PGY)-2 neurology residents. Advanced participants included PGY-3 and PGY-4 neurology residents, emergency medicine-trained and medicine-trained critical care fellows. Expert participants included neurocritical care fellows and attending physicians in neurocritical care. We considered these participants to be experts based on the inclusion of checklist action items requiring recognition and treatment of refractory non-convulsive status epilepticus on cEEG.

We evaluated performance of participants with critical care training vs. non-critical care training on critical action items relating to general critical care (ventilator management, cardiology consultation, ECG and laboratory review, developing a differential diagnosis for the arrest, infectious work-up and appropriate identification of pneumonia with initiation of antibiotics, admission to the intensive care unit). We hypothesized that participants with critical care training would perform these critical actions more often than those without critical care training. We also evaluated performance of neurology-trained participants vs. non-neurology-trained participants on items relating to neurological care (neurological exam, head CT review, initiation of and insistence on temperature management, initiation and assessment of cEEG, prophylactic anti-shivering plan, recognition of shivering, ordering of toxicology screen, appropriate treatment of refractory status epilepticus). We hypothesized that participants with neurology training would perform these critical actions more often than those without neurology training.

Standard protocol approvals

The study was reviewed and approved by the University of Maryland, Baltimore Institutional Review Board. The students conducted the project in accordance with the tenets espoused in the Declaration of Helsinki.²¹

Statistical analysis

We reported descriptive summaries as means (standard deviations [SD]) and medians (interquartile ranges [IQR]) for continuous variables and counts and percentages for categorical variables. We tested data for normality using the Shapiro-Wilk test. If we rejected the null hypothesis, we assessed skewness and kurtosis to judge symmetry. If skewness and kurtosis were less than an absolute value of two, we judged the distribution as symmetric and used the Wilcoxon-Mann-Whitney test to assess differences; if we judged the distribution to be asymmetric, we used the median test.²² We used Pearson's product-moment correlation quantify relationships between continuous variables and Kendall's tau to assess association between ordinal and continuous data.²³

Psychometricians widely document a paradox of low kappa values in the presence of high agreement. To avoid this issue, we used Gwet's AC and the intra-class correlation (ICC) to gauge inter-rater reliability.24., 25. We judged Gwet’s AC values of greater than 0.80 as the minimal standard for inter-rater reliability interpreting this value (and greater) as almost perfect agreement; further, we interpreted an ICC value of greater than 0.75 as excellent.26., 27. Investigators used one-way analysis of variance (ANOVA) to examine group differences for checklist sum scores and the global rating scale. Assumptions of normality and homoscedasticity were checked. We judged residuals to be not normally distributed for the GRS and those were rank transformed.²⁸ We used Tukey-Kramer methodology to control the comparison wise error rate for post hoc pairwise comparisons.²⁹

The reporting format is in accordance with the guidelines established by the “Strengthening the Reporting of Observational Studies in Epidemiology- STORB” study as well as the extended guidelines for health care simulation research.30., 31. We used R v4.0.3 for all analyses.

Results

Forty-nine participants completed the simulation (Table 1). Three attending physicians were included in analyses conducted to build validity evidence.

Table 1.

Characteristics of Trainees.

Characteristics	N (%)a
Age (mean (Standard Deviation))	30 (2.2)
Female	21 (43)
Level of Training
Neurology sub-intern	5 (10)
Neurosurgery intern	3 (6)
PGY-2 neurology resident	15 (31)
PGY-3 neurology resident	5 (10)
PGY-4 neurology resident	3 (6)
Internal Medicine critical care fellow	3 (6)
Emergency medicine critical care fellow	5 (10)
Surgical critical care fellow	1 (2)
Neurology critical care fellow	6 (12)
Neurocritical care attending physician	3 (6)
Primary work location
Medical intensive care unit	8 (16)
Surgical intensive care unit	1 (2)
Neurocritical care unit	12 (24)
Neurology floor	23 (47)
Self-rated experience with resuscitation (median (IQR))	3(3)
Self- rated competence with resuscitation (median (IQR))	3(3)

^aData presented as Number (%) except when otherwise indicated.

Primary outcome

The mean (SD) sum of critical actions completed by trainees across all levels was 10.7(4.0)/21 (49%). Performance of individual critical action items is described in Table 2.

Table 2.

Trainees’ Performance of Critical Action Items.

Assessment Critical Actions	N (%)a
Perform a thorough neurological exam	37 (76)
Review Labs- CBC, BMP, Troponin	47 (96)
Review and recognize abnormal ECG	32 (65)
Review and interpret CT head as normal	34 (69)
Verbalize Differential Diagnosis for Arrest	11 (22)
Management Critical Actions
Wean FiO2 and consider reducing tidal volume	8 (16)
Consult Cardiology	32 (65)
Start temperature management	40 (81)
Order continuous EEG	45 (92)
Insist on temperature management regardless of cardiology consult recommendations	20 (41)
Induce Hypothermia	4 (8)
Anti-Shivering plan	7 (14)
Recognize Shivering	22 (45)
Recognize low water temperature, and pursue infectious work up	14 (29)
Review CXR and start antibiotics	9 (18)
Order urine toxicological screen	23 (47)
Admit to Intensive Care Unit	19 (39)
Recognize NCSE on cEEG	22 (45)
Administer appropriate benzodiazepine dose	42 (86)
Administer appropriate 2nd line AED dose	42 (86)
Administer and up titrate 3rd line AED dose	27 (55)

Abbreviations: CBC: Complete blood count, BMP: Basic metabolic panel, ECG: Electrocardiography, CXR: Chest radiograph, NCSE: Non-convulsive status epilepticus, cEEG: Continuous electroencephalography, AED: Antiepileptic drugs.

^aData presented as Number (%) except when otherwise indicated.

Validity evidence

Internal structure

Interrater reliability was excellent. The Gwet’s AC for sum score of critical actions and global rating scale score was 0.96 (95% Confidence Interval [CI] 0.93–0.97) and 0.89 (95% CI 0.83–0.96), respectively. The ICC for sum score of critical actions and global rating score was 0.92 (95% CI 0.87–0.96) and 0.82 (95% CI 0.70–0.89).

Relationship to other variables

There was an effect of level of training on critical action checklist sum scores (novice mean score (SD) = 4.8 (1.8) vs. intermediate mean score (SD) = 10.4 (2.1) vs. advanced mean score (SD) = 11.6 (3.0) vs. expert mean score (SD) = 14.7 (2.2)) (Fig. 1). There was a similar effect of level of training on global rating scale scores (novice mean score (SD)] = 1.1 (0.4)) vs. intermediate mean score (SD) = 2.1 (0.7) vs. advanced mean score (SD) = 2.8 (1.0) vs. expert mean score (SD) = 4.1 (0.7)) (Fig. 2).

Fig. 1.

The effect of the level of training on the sum score of critical action items. There was a significant effect of level of training on performance of critical action items (novice mean score [standard deviation (SD)] = 4.8 (1.8) vs. intermediate mean score (SD) = 10.4 (2.1) vs. advanced mean score (SD) = 11.6 (3.0) vs. expert mean score (SD) = 14.7 (2.2)).

Fig. 2.

The effect of level of training on global rating scale. There was a significant effect of level of training on global rating scale scores (novice mean score (SD)] = 1.1 (0.4)) vs. intermediate mean score (SD) = 2.1 (0.7) vs. advanced mean score (SD) = 2.8 (1.0) vs. expert mean score (SD) = 4.1 (0.7)).

Trainees with critical care training performed better overall than those without critical care training (mean score (SD) = 13.9 (2.5) vs 9.1 (2.5), p < 0.001)). Participants with critical care training also performed better on nine designated critical care-specific critical action items compared to non-critical care trained participants (mean score (SD) = 5.5 (1.4) vs 3.0 (1.8), p < 0.001) (Fig. 3).

Fig. 3.

The effect of critical care training on the sum score of critical care critical action items. Trainees with critical care training performed better on critical care action items, including ventilator management, cardiology consultation, ECG and laboratory review, developing a differential diagnosis for the arrest, infectious work-up and appropriate identification of pneumonia with initiation of antibiotics, admission to the intensive care. (Trainees with critical care training mean score [standard deviation (SD)] = 5.5 (1.4) vs trainees without critical care training mean scores [standard deviation (SD) = 3.0 (1.8), p < 0.001].)

Trainees with neurology training performed slightly better in 12 neurology-specific critical actions compared to those without neurology training, but this finding did not meet statistical significance (median score ([IQR]) = 6.5 (4.3–7.8) vs. 7 (3–10), p =.12) (Fig. 4).

Fig. 4.

The effect of neurology training on the sum score of neurology critical action items. Trainees with neurology training performed slightly more neurology-specific critical action items (neurological exam, head CT review, initiation of and insistence on temperature management, initiation and assessment of cEEG, prophylactic anti-shivering plan, recognition of shivering, ordering of toxicology screen, appropriate treatment of refractory status epilepticus), but this finding did not meet statistical significance (neurology trained trainees median score (Interquartile Range [IQR]) = 7 (3–10) vs. trainees without neurology training median score (IQR) = 6.5 (4.3–7.8), p =.12).

Critical action checklist sum score was moderately correlated with both self-rated experience (Pearson correlation coefficient = 0.59, p < .01) and self-rated proficiency (Pearson correlation coefficient = 0.64, p < .01) in post-cardiac arrest care.

Discussion

We utilized high fidelity simulation to assess the performance of trainees and attending physicians in the evaluation and management of a post-cardiac arrest patient whose course was complicated by status epilepticus. Our simulated case revealed performance gaps in three main areas of post-cardiac arrest care which included evaluation of the etiology of OHCA, ventilator management, and temperature management implementation. Delays in these critical components of post-cardiac arrest care have been associated with increased in-hospital mortality.32., 33., 34., 35., 36. We presented several forms of validity evidence to support the legitimacy of our findings in the domains of internal structure and response to other variables, including excellent interrater reliability, effects of training background on background-specific tasks, and an effect of level of training on both critical action checklist sum scores and global rating scale scores.

While a rich body of literature utilizes simulation-based assessment in the initial links in the chain of survival after OHCA, we are not aware of other studies evaluating the post-cardiac arrest care phase. It is alarming that most participants did not verbalize a differential diagnosis nor did they evaluate potential causes for OHCA in a young patient. Obtaining the ECG is a critical step that is usually performed “automatically” in the academic emergency setting. During debriefing, many trainees noted that the ECG is simply obtained and given to them for review without request. We hypothesize that the automated care that occurs in highly-resourced academic centers may lead to blind spots in trainees’ care plans when left to their own devices. While trainees recognized the importance of reviewing ECGs (even if they forgot to obtain them), many reported that they were unaware of potential negative effects of hyperoxia. Similarly, we revealed performance gaps in implementation of temperature management. When trainees insisted on temperature management, they failed to implement several components of high-quality temperature management.³⁷ The Neurocritical Care Society has published guidelines on temperature management implementation, but core features such as prevention and treatment of shivering are largely missing from traditional didactic teaching sessions and bedside teaching.¹⁸

Of note our study took place prior to the publication of the Targeted Temperature Management 2 trial, which failed to find a benefit of mild hypothermia over aggressive fever prevention.³⁸ Others have hypothesized that results from the Targeted Temperature Management 1 which found no difference between hypothermia to 33 degrees Celsius to either hypothermia to 36 degrees Celsius may have been interpreted as showing lack of benefit for any temperature management (including fever control) in post-cardiac arrest care management.³⁹ We hypothesize that our trainees were easily dissuaded from utilizing temperature management due to uncertainty regarding its effectiveness. The Targeted Temperature Management 2 trial results may further laissez-fair attitudes regarding temperature management, despite guideline updates suggesting active temperature management to prevent temperatures > 37.7 °C.⁴⁰

Prior studies have demonstrated that simulation-based training can lead to durable improvements in cardiac arrest management including improved adherence to ACLS protocols in real patients.12., 41. Our study is novel in its evaluation of the post-cardiac arrest care phase. We are not aware of any other studies evaluating this key link in the chain of survival. We uncovered multiple performance gaps. We recommend that clinical educators expand ACLS sim-based curricula to include post-cardiac arrest care, with special attention to addressing reversible causes of arrest, ventilator management, and temperature management implementation.

Rigorous validation of educational assessments is critically important. We previously presented validity evidence to support the legitimacy of our findings according to Messicks’ framework, including content evidence and response process. To that evidence we add the domains of internal structure, supported by excellent interrater reliability, and relationship to other variables, supported by effects of training background on background-specific tasks, and an effect of level of training on both critical action checklist sum scores and global rating scale scores. We were not able to offer validity evidence regarding consequences as there were no direct consequences to our participants. We posit that our findings show strong support towards the validity of our assessment and results. Future studies of post-cardiac arrest care should use a similar approach to support their validity hypothesis, specifically in the area of content evidence where new advances are likely to influence determination of critical action checklist items.

Our study has several limitations. First, its single center design limits generalizability. The performance of trainees in this study may not reflect those at other institutions. However, participating trainees did have diverse training backgrounds and involvement in post-cardiac arrest care. Second, we were not able to perform assessments on real patients. We did, however, provide several pieces of validity evidence per Messick’s framework to support the findings of our simulation-based metrics. Finally, although we debriefed all trainees, this part of the simulation case was not captured in the data collection. Future studies should include debriefing for qualitative assessment to help understand why trainees fail to perform critical actions.

Conclusion

High-fidelity manikin-based simulation holds promise as an assessment tool in the performance of post-cardiac arrest care. Areas requiring further educational initiatives to improve performance include diagnostic work-up of OHCA, ventilator and temperature management implementation.

Funding

This study was funded by the Faculty Innovation in Education Award from the American Board of Psychiatry and Neurology (Dr. Morris).

Location of work

University of Maryland Medical Center / R Adams Cowley Shock Trauma Center.

Abstract presentation

American Heart Association Scientific Assembly, Resuscitation Symposium, November 2021, in Boston.

CRediT authorship contribution statement

Afrah A. Ali: Data curation, Writing – original draft, Visualization. Wan-Tsu W. Chang: Methodology, Investigation, Writing – review & editing. Ali Tabatabai: Methodology, Writing – review & editing. Melissa B. Pergakis: Investigation, Writing – review & editing. Camilo A. Gutierrez: Methodology, Writing – review & editing. Benjamin Neustein: Investigation, Resources, Writing – review & editing. Gregory E. Gilbert: Validation, Formal analysis, Visualization. Jamie E. Podell: Writing – review & editing. Gunjan Parikh: Writing – review & editing. Neeraj Badjatia: Writing – review & editing. Melissa Motta: Writing – review & editing. David P. Lerner: Methodology, Writing – review & editing. Nicholas A. Morris: Conceptualization, Methodology, Data curation, Resources, Writing – original draft, Visualization, Supervision, Project administration, Funding acquisition.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following are the Supplementary data to this article: