The Simulation-based Assessment of Pediatric Rapid Response Teams

Abstract

Objective

The objectives of the study were to create scenarios of simulated decompensating pediatric patients to train pediatric rapid response teams (RRT) and to determine whether the scenario scores provide a valid assessment of RRT performance with the hypothesis that intensivists-in-training-led RRTs would be better prepared to manage the scenarios than nurse practitioner-led teams.

Study design

A set of 10 simulated scenarios was designed for training and assessment of pediatric RRTs. Pediatric RRTs, comprised of a Pediatric Intensive Care Unit (PICU) nurse (RN) and respiratory therapist (RT), led by a PICU intensivist-in-training or a Pediatric Nurse Practitioner (PNP), managed 7 simulated acutely decompensating patients. Two raters evaluated the scenario performances and psychometric analyses of the scenarios were performed.

Results

The teams readily managed scenarios such as supraventricular tachycardia and opioid overdose, but had difficulty with more complicated scenarios such as aortic coarctation or head injury. The management of any particular scenario was reasonably predictive of overall team performance. The teams led by the PICU intensivists-in-training outperformed the teams led by the PNPs.

Conclusions

Simulation provides a method for RRTs to develop decision-making skills in managing decompensating pediatric patients. The multiple scenario assessment provided a moderately reliable team score. The higher scores achieved by PICU intensivist-in-training-led teams provides some evidence to support the validity of the assessment.

Pediatric Rapid Response Teams (RRT) assist bedside caregivers outside of the intensive care unit (ICU) when a hospitalized child is unstable. The widespread adoption of RRTs has been based on evidence that these teams improve clinical outcomes, decrease the cost of care, and attenuate the stress families and staff experience when a hospitalized child deteriorates.^1,2,3,4 The Joint Commission has provided a National Patient Safety Goal for hospitals to improve recognition and response to changes in patient conditions.^5,6,7

The composition of RRTs is variable. In academic medical centers, concerns about the effect of sleep deprivation have led the Accreditation Council for Graduate Medical Education (ACGME) to mandate work hour restrictions and enhanced supervisory requirements for physician-trainees.^8,9 In children’s hospitals, nurse practitioners (NPs) and hospitalists increasingly provide clinical care, including leading RRTs, in an interchangeable fashion with intensivists-in-training.^10,11 In community settings, the trend toward round-the-clock in-house coverage of ICUs has increased the utilization in critical care settings of nurses (RN), NPs, hospitalists, and physician assistants (PAs) any of which may oversee a hospital’s RRT.^12,13,14,15

There is evidence of both improvement and lack thereof in patient outcomes in studies of RRTs.^16,17 Numerous factors may contribute to negative results, particularly in pediatrics, including the training and composition of RRTs and the diverse clinical conditions that pediatric RRTs must recognize and manage. In addition, the impact of a RRT may be attenuated at an institution where there is a strong preexisting patient care infrastructure.¹⁸ To standardize the preparation of RRTs at pediatric tertiary care centers, a simulation-based training method could be used to provide RRT members with experience in managing a range of conditions that are frequently encountered during a rapid response call. Such an approach could also be used to assess the performance of RRTs.

In this study, we designed simulation scenarios to assess the performance of pediatric RRTs in diagnosing and managing a range of acute conditions that lead to RRT calls. The purposes of the study were to develop a set of scenarios that could be used to educate rapid response teams and to gather initial evidence to support the validity of the simulation-based assessment score based on the hypothesis that intensivists-in-training who led RRTs would be better prepared to manage the scenarios than nurse practitioner-led teams.

Methods

In developing the scenarios, we reviewed critical events that RRTs had encountered and managed at St. Louis Children’s Hospital. These included respiratory conditions such as asthma exacerbation, respiratory failure, and severe airway obstruction; neurologic conditions such as seizures or a decreased level of consciousness; and cardiac conditions such as arrhythmias, congenital heart disease, and cardiac failure. The scenarios created for the study are similar to premorbid conditions that have led to clinical deterioration prompting a RRT call at other tertiary care pediatric hospitals.⁵ The 10 simulation scenarios were designed to reflect a range of ages of children who had cardiovascular, respiratory, and neurologic disease that had either not improved or had worsened during hospitalization (Table I). The teams encountered simulations that followed predictable patterns and could be managed by following recognized treatment algorithms (e.g., child with status epilepticus) as well as those that required reassessment and reevaluation of an atypical presentation that did not adhere to typical heuristics. The 10 assessments included an introductory asthma scenario and the remaining scenarios were divided into 3 sets of three. Each set of three included one scenario incorporating an admitting diagnosis based on incomplete data that required further inquiry by the RRT to obtain the correct diagnosis and provide appropriate treatment. The atypical presentation scenarios were: (1) a child with increasing dyspnea who was admitted with pneumonia but actually had congestive heart failure, (2) an patient with asthma who had continuing bronchospasm and dyspnea and in fact had aspirated a foreign body, and (3) a 6-week-old infant with poor feeding and lethargy who had been admitted with pyloric stenosis but really had coarctation of the aorta.

Table 1.

Pediatric Rapid Response Team [RRT] Training Scenarios

Scenario
1	7-year-old with asthma attack	Asthma attack: Introductory scenario. Team expected to identify and appropriately treat bronchospasm
2	4-year-old, 15 kg ♂ is admitted following tonsillectomy and adenoidectomy earlier in the day. HR 140, BP 90/60, RR 10, O2 93%	Narcotic Overdose: Team expected to review hospital management. Somnolent and hypoxic. Pupils pinpoint. (Oxycodone 5 mg po two hours ago)
3	2-month-old, 3.6 kg ♂ with irritability presents to emergency department. Antibiotics were initiated after cultures were obtained and child was admitted to the floor. On routine vitals, the RN notes that the VS: HR is 260, T 37, BP 60/30, RR 50 crying, O2 97%.	Supraventricular Tachycardia: Team expected to recognize arrhythmia. EKG shows narrow complex, regular tachycardia consistent with SVT. Infant is poorly responsive with decreased perfusion. Therapy is synchronized cardioversion at 0.5–1J/Kg
4	10-year-old ♀ diagnosed with pneumonia is admitted. He continues to have increased work of breathing, more severe tachypnea and continued wheezing. T. 37.5, HR 120, BP 90/60, RR 28, O2 sat 90%	Cardiomyopathy (Failed Heuristic): Team to establish clinical findings of gallop rhythm, hepatomegaly, chest x-ray indicates cardiomegaly. Team corrects diagnosis.
5	6-year-old ♀ with pneumonia was admitted 2 days prior, has been treated with IV antibiotics and supplemental oxygen. T. 37.5, HR 138, BP 90/60, RR 42, 02 sat 80%	Respiratory Failure: Team expected to note increasing oxygen requirement, desaturation and worsening respiratory distress. Prepare for intubation, support ventilation with bag and mask ventilation,
6	8-year-old ♀ with progressive neurologic disorder, seizure disorder, admitted 2 days with dehydration. A seizure that has been ongoing for more than 20 minutes. There is an advanced directive on the chart. T. 37.5 HR 140 BP 90/60 RR 40 O2 sat 80%	Status Epilepticus Management: Team expected to determine nature of advanced directive, treat seizure and support ventilation with bag and mask ventilation
7	3-year-old ♂ diagnosed with exacerbation of asthma, increasing tachypnea and continued wheezing. T. 37.5 HR 140 BP 90/60 RR 40 O2 sat 88%	Bronchial Foreign Body (Failed Heuristic): Team should review maximal asthma management. Review of history and determine undiagnosed cause of tachypnea and wheezing. Evaluation of chest x-ray shows unilateral hyperinflation. Team corrects diagnosis.
8	9-month-old ♂ was in a motor vehicle collision 3 days prior has been on the surgical ward with pulmonary contusion, concussion, femur fracture and liver laceration. T 38, HR 70, BP 120/60, RR 5, O2 sat 93%	Increased Intracranial pressure: Team expected to note poor respiratory effort, somnolence and support ventilation with bag and mask ventilation, prepare to intubate, immediate imaging, need for hyperventilation and mannitol and/or 3% NS.
9	8-year-old boy was admitted with several days of fever and cough. Chest X-ray in the ER shows RLL infiltrate. He was found to be less responsive. T 38.5, HR 150, BP 70/40, RR 40, O2 sat 94%	Septic Shock: Team to provide fluid resuscitation. During resuscitation, continued deterioration in respiratory effort and child needs to be intubated and ventilated. Proper RSI must be undertaken since child was eating throughout the day.
10	24 day old ♀ infant with pyloric stenosis was admitted to the surgical ward with vomiting and poor feeding for rehydration and correction of electrolyte imbalance. T 35, HR 178, BP 88/54, 02 sat 95%	Coarctation of Aorta (Failed Heuristic) Infant condition is not characteristic of pyloric stenosis. Electrolytes are normal, condition is undiagnosed aortic coarctation, child is noted to have worsening perfusion and tachycardia. Team corrects diagnosis.

The scenarios were created by two of the authors and reviewed by intensivists and critical care nurses who all had served as members of RRTs. The scenarios were pilot tested by current pediatric RRT members. Based on the pilot team performances, and the feedback they provided, the scenarios were revised and scoring rubrics developed. The scoring included a list of key actions that RRTs would be expected to perform to diagnose and manage the scenario. These key actions were used to guide the global ratings that the raters provided on a 1–9 scale for each encounter. Ratings of 1–3 were considered unsatisfactory and indicated that the care was not adequate. Ratings of 4–6 indicated that the care was adequate, but not optimal. Ratings of 7–9 indicated that the care provided was effective and efficient.

Following piloting of the scenarios, volunteers were solicited from PICU nurses, RTs, PICU intensivists-in-training, and PNPs at St. Louis Children’s Hospital. The teams reflected the composition of the RRTs at the hospital, and included a PICU nurse and a respiratory therapist led by a PICU intensivist-in-training or a PNP. Participants provided consent, demographic data, and indicated their simulation experience and level of educational attainment. Six of the teams were led by a NP and eleven teams were led by a PICU intensivist-in-training (1 Post graduate year 3 [PGY], 1 PGY4, 9 PGY5). The NPs had been RNs for an average of 15 years (range 9–23), which included a considerable amount of PICU experience, and had been NPs for an average of 4.8 years (range 2–10). The PICU intensivists-in-training included a 3^rd year pediatric resident (PGY3) who was functioning like a fellow in the PICU. Twelve of the teams were led by females; 5 were led by males. The average age of the leader was 34.5 years (min=31, max=45, SD=3.2). All teams completed all their assigned scenarios in this IRB-approved study.

The assessment was conducted in a two-hour session in the Saigh Pediatric Simulation Center at St. Louis Children’s Hospital, the pediatric tertiary care center of Washington University. METI PediaSIM HPS^® and Laerdal SimNewB^® mannequins were used and the team performances were video-captured for later scoring. Seventeen rapid response teams were evaluated across 7 simulation scenarios, yielding 119 total encounters. All RRTs (n=17) participated in the introductory asthma exacerbation scenario that was not included in the scoring as it was designed to familiarize the teams with the process of simulation. The teams then completed two of three randomly assigned sets of scenarios (6 of the 9 remaining), yielding 102 study encounters (Table II). The teams were debriefed after the initial scenario and following the final 7^th scenario. The scenarios were scored by one of the authors for completion of key actions (checklist items) as well as for the global team performance. The global performance was based on a determination on the overall quality of care which included the speed and accuracy of the diagnosis and allowed the raters to consider actions and decisions that were appropriate as well as those actions that were not indicated based on the child’s condition. Because the same rater provided scoring for both the analytic (checklist) and holistic (global) measures, the analysis of team scores was conducted using the global score.

Table 2.

Mean performance by group (Intensivist-in-Training or Nurse Practitioner)

Scenario	Intensivist-in-Training		Nurse Practitioner
	N	(Mean±SD)	N	(Mean±SD)
Narcotic Overdose	8	7.88 ±0.64	3	8.00±0.00
SVT	8	8.38 ±0.74	3	8.00±1.00
Cardiomyopathy	8	7.88±1.36	3	6.67±0.58
Respiratory Failure	7	7.71±1.11	6	7.83±0.75
Status Epilepticus	7	6.71±1.11	6	6.67±0.82
Bronchial Foreign Body	7	6.71±2.14	6	6.50±1.38
Head Injury Infant	7	7.43±0.79	3	5.00±1.73
Septic Shock	7	7.43±0.53	3	7.00±1.00
Coarctation of Aorta	7	6.57±1.40	3	4.33±1.53
Overall Team Performance	66	7.44±1.25	36	6.75±1.46

Statistical Analyses

Several psychometric analyses were undertaken to explore the performance of the 17 RRTs. Variance components were calculated to estimate the reliability of the RRT performance scores for the multi-scenario assessment. Descriptive statistics (mean, SD) summarized performance at the scenario level. A summary score was calculated by averaging each team’s performances across the 6 study scenarios. Correlations between the scenario ratings and the total score (average of scenario ratings) for each team were calculated which provides a measure of scenario discrimination. To investigate if performance varied as a function of the characteristics of the leader (PNP, PICU intensivist-in-training) a 1-way repeated measures analysis of variance (RM-ANOVA) was conducted. The between-subject factor was leader (PNP, PICU intensivist-in-training). The within-subject factor (repeated measure) was scenario. This analysis excluded the initial orientation asthma scenario.

Results

The teams readily recognized and managed scenarios such as supraventricular tachycardia and opioid overdose, but had more difficulty with scenarios such as the infant head injury and aortic coarctation. Table III provides a summary of global team performance as well as discrimination statistics. The most difficult scenario was aortic coarctation (M=5.9±1.7) and the easiest was SVT (M=8.3±0.8) (Table III). The reliability of the global team performance ratings (for a 6-scenario assessment) was 0.57. Scenario discrimination statistics, the correlation between a team’s mean scenario score and their overall score on the allotted scenarios, were all positive indicating that the team performance in one scenario was reasonably predictive of overall performance.

Table 3.

Global Performance (Out of 9) & Discrimination by Scenario (all teams)

Scenario	N	Mean±SD	Minimum	Maximum	Discrimination (D) of scores
Narcotic Overdose	11	7.9±0.5	7.0	9.0	0.60
SVT	11	8.3±0.8	7.0	9.0	0.41
Cardiomyopathy	11	7.6±1.3	5.0	9.0	0.73
Respiratory Failure	13	7.8±1	6.0	9.0	0.50
Status Epilepticus	13	6.7±1	5.0	8.0	0.62
Bronchial Foreign Body	13	6.6±1.8	4.0	9.0	0.63
Head Injury Infant	10	6.7±1.6	4.0	8.0	0.87
Septic Shock	10	7.3±0.7	6.0	8.0	0.70
Coarctation of Aorta	10	5.9±1.7	3.0	8.0	0.59

The 1-way (leader type) RM-ANOVA yielded a significant interaction (F=2.2, p<0.05), indicating that performance differences between the groups (PICU intensivists-in-training, PNPs) were not consistent across scenarios. There was also a significant main effect attributable to the type of scenario (F=5.0, p<=0.01), indicating that the scenarios were not of equivalent difficulty. Finally, there was significant effect attributable to leader type (F=5.3, p<0.05). Averaged over scenarios, the teams led by PICU intensivists-in-training outperformed the PNP-led teams (M=7.4 vs M=6.8, ES=0.51). Table II shows the mean performance, by group (PICU intensivists-in-training, NPs), across scenarios. Although the PICU intensivist-in-training-led teams outperformed the NP-led RRTs, the differences were much greater for scenarios head injury with increased intracranial pressure (M=7.4 vs M=5.0) and aortic coarctation (M=6.6 vs M=4.3).

Discussion

Rapid response teams serve as a link between general inpatient units and the PICU, bringing additional expertise and acute care skills to deteriorating patients. RRTs often receive limited additional training and are expected to draw on their own experiences in managing acutely ill infants and children. The variance in overall performance on the multiple scenarios that we found among RRTs, as well as the range of performances on individual scenarios, may explain in part the observation that rapid response teams have improved patient outcomes in some centers ^1–4 but have had no impact on patient outcomes in others.^16,17 In a recent meta-analysis, rapid response systems were associated with a reduction in hospital mortality and cardiac arrest; the presence of a physician on the team was not associated with a reduction in mortality.¹⁹

This study was designed to provide experiences in managing a range of simulated scenarios that RRTs might encounter on the ward. The scenarios varied in difficulty and included more challenging conditions that required teams to review the history, conduct additional investigations, and recognize an incorrect admitting diagnosis (cardiomyopathy, bronchial foreign body and aortic coarctation). The content validity of the simulation scenario scores is supported by the inclusion of conditions that RRTs typically encounter. We hypothesized that if the scenarios represented valid rapid response conditions then the teams led by PICU intensivists-in-training, who had more training in diagnosis and treatment of medical and surgical conditions, would obtain higher overall scores. This finding provides some evidence to support the construct validity of the assessment scores. The teams led by intensivists-in-training were, on average, better able to manage scenarios that required more advanced diagnostic skills or pharmacologic interventions.

The performance differences between nurse practitioner and intensivist-in-training-led teams were most evident in scenarios such as head injury, aortic coarctation, and cardiomyopathy. In addition to being more clinically complicated, the coarctation and the cardiomyopathy scenarios were designed to defy typical heuristics and required additional interrogation to attain the correct diagnosis. The pediatric NPs also had limited experience in the pediatric cardiac ICU (CICU) whereas the PICU intensivists-in-training receive training experiences in the CICU. Because of resident and fellow duty hour restrictions, nurse practitioners have assumed an ever-increasing role in the care of hospitalized children and frequently function in roles that are interchangeable with resident and fellows including serving as RRT leaders²⁰. Although some performance differences were found, the PNPs effectively directed RRTs to manage many of the simulated pediatric emergencies. Regardless of the team leader, the teams effectively recognized and managed many of the more common conditions that lead to rapid responses such as asthma, respiratory failure, and opioid overdose. The teams all had difficulty managing at least one of the scenarios. This suggests that simulation based training, especially if the scenarios are based on complex patient presentations, could benefit all teams.

The positive scenario discriminations (the correlation between individual scenario scores and total scenario scores) indicate that team performance on any particular scenario correlated with the team’s overall performance and that the scenarios assessed a common domain of pediatric practice, namely acute care management of hospitalized children. However, the range of discriminations observed among the individual scenarios (0.41 for the supraventricular tachycardia scenario to 0.87 for the infant head injury scenario) indicated that no single scenario could reliably predict a team’s overall performance (Table III). Like our previous studies in simulating acute care scenarios in pediatric conditions as well as in trauma settings, we found that the ability of a participant or a team to recognize and manage any individual scenario did not necessarily generalize to their management of other pediatric conditions.^21,22 In the pediatric setting, the span of patient ages and the variety of conditions encountered supports an educational approach that includes a broad range of scenarios to effectively train RRTs as well as to provide a reliable assessment of team performance. The overall reliability of the assessment (r=0.57) did not approach a level that would be useful to make high-stakes judgments concerning a team’s ability. However, the scenarios were useful in providing experiences for the RRTs and yield performance measures that can help educators provide meaningful feedback to team members regarding their strengths and weaknesses. Although valuable as an assessment of learning, teams would have to manage many more scenarios of similar construct if the scores are to be used for summative purposes.²⁰ A further limitation of this study is that the number of participants limits our ability to draw conclusions about the impact of the makeup of the team in contributing to the team performance. This variance was consistent throughout all teams and raises an interesting avenue for further investigation.

Simulation provides a method for RRT members to develop skills in appropriate diagnoses and treatments over a range of critical conditions in children of various ages. In this study, simulation exposed teams to clinical scenarios that required them to process patient data in real time and set management priorities. This type of critical thinking and decision-making is essential in crisis settings yet is difficult to assess either in traditional examinations or by observing clinical performance. The goal of the training was to improve the ability of pediatric RRTs to recognize the premonitory signs and symptoms that herald impending cardiopulmonary arrest and to develop skills to manage cardiovascular, neurologic, and respiratory deterioration more effectively. Incorporating code blue simulations into code team training has been associated with improvements in code team performance and survival to discharge following pediatric cardiac arrest.²³

We developed an inventory of scenarios to educate and evaluate pediatric rapid response teams. By providing RRTs with experience in managing several acute conditions, simulation-based education can be used to prepare teams as well as to identify strengths and deficiencies in their ability to manage acute events. These experiences when combined with feedback can be used to improve their ability to manage these critical events. Although a variety of factors potentially influence morbidity and mortality, improving team performance in managing critical events may be an essential factor in improving hospitalized children’s outcomes.²⁴ Such training has applicability to pediatric residents, intensivists-in-training, nurse practitioners and all who care for children.

Abbreviations used in the manuscript

ACGME

Accreditation Council for Graduate Medical Education

CRM

Crisis Resource Management

CICU

Cardiac Intensive Care Unit

ICU

Intensive Care Unit

NP

Nurse Practitioner

PA

Physician Assistant

PGY

Post graduate year

PICU

Pediatric Intensive Care Unit

PNP

Pediatric Nurse Practitioner

RN

Nurse

RT

Respiratory Therapist

RRT

Rapid Response Team

Appendix