Rapid cycle deliberate practice improves and sustains paediatric resident PALS performance

Abstract

Objectives

Paediatric cardiopulmonary arrest resuscitation is a critically important skill but infrequently used in clinical practice. Therefore, resuscitation knowledge relies heavily on formal training which is vulnerable to rapid knowledge decay. We evaluate knowledge and skill retention post-training using rapid cycle deliberate practice (RCDP).

Design

Pilot, non-blinded, single-arm study.

Setting

Pediatric Simulation Center at Children’s of Alabama.

Participants

42 paediatric residents at a large, tertiary care, academic children’s hospital were enrolled in this simulation-based resuscitation study.

Interventions

Each participant led a 7 min preintervention arrest scenario as a baseline test. After testing, participants were trained individually in the paediatric advanced life support (PALS) skills necessary for resuscitation of a patient in pulseless electrical activity and ventricular fibrillation using RCDP—a simulation method using frequent expert feedback and repeated opportunities for the learner to incorporate new learning. Immediately post-training, participants were retested as leaders of a different paediatric arrest scenario. 3 months post-training participants returned to complete a final simulation scenario.

Main outcome measures

To evaluate knowledge and skill retention following PALS training.

Results

Preintervention data demonstrated poor baseline resident performance with an average PALS score of 52%. Performance improved to 94% immediately post-training and this improvement largely persisted at 3 months, with an average performance of 81%. In addition to improvements in performance, individual skills improved including communication, recognition of rhythms, early chest compressions and rapid administration of epinephrine or defibrillation.

Conclusions

RCDP training was associated with significant improvements in resident performance during simulated paediatric resuscitation and high retention of those improvements.

Introduction

Thousands of children in the USA suffer from in-hospital cardiopulmonary arrest (CPA) each year. Less than 50% will survive to hospital discharge, and many of the survivors will suffer new, permanent neurological injury.^{1 2} In an effort to improve survival and long-term outcomes following paediatric CPA, the American Heart Association (AHA) established a collection of best-practice guidelines for paediatric resuscitation, termed paediatric advanced life support (PALS).³ Adherence to PALS guidelines during a paediatric resuscitation results in shorter delays in providing critical treatments like chest compressions, defibrillation and epinephrine, with resultant improvements in survival and decreased long-term morbidity.^4–6

In addition to establishing PALS guidelines, the AHA created a PALS curriculum that is widely used in children’s hospitals throughout the USA. Like AHA training for basic life support (BLS) and adult advanced cardiac life support (ACLS), paediatric providers typically undergo PALS training every 2 years. Unfortunately, even with this regular PALS training, care provided during paediatric in-hospital arrests often does not adhere to the guidelines.^{7 8} Recognising similar performance gaps following BLS training, the AHA has recently launched the ‘Resuscitation Quality Improvement’ programme. This programme focuses on improving the quality of cardiopulmonary resuscitation (CPR) by emphasising the need for more frequent training for providers.⁹ While PALS and ACLS trainings are currently provided every other year, evidence is accumulating that suggests learner retention is as low as 22% just 3 months after ACLS training.^10–12

Recognising the need for more effective teaching methodology, McGaghie et al developed simulation-based mastery learning programmes for ACLS in the early 2000s.^13–15 This group demonstrated that simulation practice provided high knowledge and skill retention as much as 14 months later.¹⁴ Even more importantly, they demonstrated an improvement in patient care during ACLS codes following this training.¹⁵ While effective, this methodology has not been widely adopted, possibly due to the high time costs. In 2014, Hunt et al ¹⁶ reported on a modified, competency-based method of training paediatric residents in PALS called rapid cycle deliberate practice (RCDP). RCDP has an advantage over traditional curricula by providing the learners multiple opportunities to “do it right” along with efficient, time-sensitive feedback in a psychologically safe environment¹⁶ (figure 1). We believe that the greatest advantage RCDP offers is the possibility of improved knowledge and skill retention when compared with traditional training methodologies. Because of frequent repetition in a short period of time, this methodology has time–cost advantages over traditional mastery learning. Our hypothesis is that paediatric residents who undergo a single RCDP training would demonstrate knowledge retention of at least 75% 3 months after training.

Figure 1.

Shows the learning cycle for rapid cycle deliberate practice. Learners begin the simulation by caring for a simulated arrest. They are coached whenever they make important errors, the scenatio is restarted and the learner has the opportunity to correct the error and continue through the scenario.

Materials and methods

All residents in general paediatrics, medicine-paediatrics and pediatrics-neurology at our institution were invited to participate in the study. All participants held current PALS certification. Individual experience in emergency medicine, neonatology and paediatric critical care were recorded at each assessment point. RCDP training was mandatory for all residents, but only residents who chose to participate in the study completed pre, post and 3-month recorded scenarios. Residents participated individually in each session.

Prior to starting the study, we performed two pilot sessions with first year paediatric critical care fellows who were the most similar learners to the residents who would participate in the study. The pilot studies were used to improve standardisation both within the scenarios and in the recording procedures. Data gathered during the pilot sessions were not included in the analysis and were used only for quality control purposes.

Four simulation staff members were trained as embedded participants (EPs) for this study. An embedded participant is a person who knows the background of the scenario and performs specific duties during the simulation.¹⁷ During each session, two nurses and two additional staff performed roles as assigned by the participant. The EPs completed only the tasks explicitly requested by the participants and did not offer suggestions. This process standardised the study experience for the learners. A crash cart was readily available and the EP nurse was trained to wait 10 s after medications/fluids were requested before removing a prefilled syringe from the code cart. All cases were programmed to ensure consistency in mannequin data and were performed using the same high-fidelity infant simulator (Laerdal SimNewB). All simulations were performed at the Pediatric Simulation Center at Children’s of Alabama.

During the initial phase of the study, each learner participated as the leader of a simulated paediatric arrest preintervention scenario that consisted of 5 min of pulseless electrical activity (PEA) followed by 2 min of ventricular fibrillation (VF). The preintervention scenario was conducted without pauses or coaching of the learner to establish a learner baseline. After the preintervention scenario, participants underwent RCDP training, lasting between 20 and 30 min. In the RCDP training, the participants were again presented with an infant in PEA who would later transition to VF with one investigator present in the room as a ‘coach’. Utilising a predetermined list of ‘hard’ and ‘soft’ stops (online supplementary appendix A), the scenario was paused at points where the participant made a significant error. Stops were determined by three experts in paediatric critical care and simulation and largely followed the stops used by Hunt et al.¹⁵ When the scenario was paused, the participant was coached on the correct next step and provided a brief explanation. The scenario was then restarted from the point where it was paused (for soft stops) or from the beginning of the scenario (for hard stops). Soft stops were also called as needed to offer praise and positively reinforce new behaviours. Once the participant successfully led the resuscitation team through predefined endpoints in both VF and PEA scenarios without committing any hard stops, the RCDP session was halted. The participant then led an uninterrupted postintervention session consisting of caring for an infant in VF for 4 min followed by PEA for 3 min. Following the postintervention session, a short debriefing was provided for each participant. This debriefing was scripted and designed to elicit learner feedback about the process. Specifically, no evaluation of the learner’s performance was provided during the debriefing.

Approximately 3 months after the initial session, residents returned for follow-up sessions in which they individually led a single uninterrupted paediatric arrest scenario. The scenario was a simulated infant arrest with the child in PEA for 5 min who then converted to VF for an additional 2 min. All participants were debriefed following the scenario with a short review of PALS algorithms following the scenario and given an assessment of their performance. All follow-up sessions were limited to 30 min (figure 2).

Figure 2.

Features the design of the study regarding its progression. PEA, pulseless electrical activity; RCDP, rapid cycle deliberate practice; VF, ventricular fibrillation.

All testing and training sessions were videotaped using two separate GoPro cameras (GoPro) mounted to show both the monitor and all actions performed in the room. Times were recorded by two researchers. The initial times were recorded during the simulation test scenarios by direct observation of one researcher. A second researcher independently recorded times based on the video record.

Each scenario lasted 7 min. If participants failed to perform an action in the 7 min, it was recorded as not completed. Resident performance during preintervention, postintervention and follow-up sessions was evaluated using a time-to-event checklist. After each session, the participant completed an anonymous demographics sheet (online supplementary appendix B).

Following data collection, an overall PALS score was calculated for each participant’s performance in each simulation scenario (online supplementary appendix C). The PALS score used in this study was based on the tool developed by Donoghue et al.¹⁸ Because our simulation scenarios involved both PEA and VF, we combined Donoghue’s scoring systems for asystole and dysrhythmia scenarios. We omitted scoring for intravenous/intraosseous access as all scenarios began with the manikin already having intravenous access. We also omitted scoring for recheck of pulse within 30 s of return of spontaneous circulation (ROSC) as PALS guidelines now recommend continuing compressions for 2 min after ROSC. In scoring the residents’ performance during VF, we omitted scoring for perfusion assessment as the authors felt this would be incorrect in a patient without a pulse. This differs from the scenario used by Donoghue, in which the patient begins with a narrow complex tachycardia with a pulse.

Knowledge retention was calculated by comparing the difference between 3-month follow-up and pretest performance with the difference between the pretest and post-test performances to quantify the amount of new knowledge retained.

$${\displaystyle \begin{array}{l}\mathrm{R}\mathrm{e}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}=\frac{3‐\mathrm{m}\mathrm{o}\mathrm{n}\mathrm{t}\mathrm{h}\mathrm{v}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{e}-\mathrm{P}\mathrm{r}\mathrm{e}‐\mathrm{T}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{v}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{e}}{\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{t}‐\mathrm{T}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{v}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{e}-\mathrm{P}\mathrm{r}\mathrm{e}‐\mathrm{T}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{v}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{e}}\times 100\end{array}}$$

SAS software, V.8 was used for analysis. Least squares mean values were calculated using generalised estimating equations to account for repeated measures over the three time points. Normally distributed continuous data over the three time points were compared using an unpaired t-test. Categorical data were analysed using a χ² test. All tests were two-tailed.

Results

Forty-six residents (postgraduate year (PGY)-1 to PGY-5) participated in the initial training from February 2016 to June 2016, and 42 returned for 3-month follow-up testing. There were no statistically significant differences in gender, participation in simulated/mock codes or PALS training when comparing residents between the pre-RCDP session and 3-month follow-up. PGY-1 residents had completed an average of 1 month of neonatology and 1.2 months of emergency medicine prior to training. PGY-2 residents had completed an average of 1.5 months of neonatology, 1 month of paediatric critical care and 2.6 months of emergency medicine prior to training. Average length of time since PALS training was 15.5 months in at the initial training and 18.2 months at the time of the 3-month follow-up.

All resuscitation skills showed statistically significant improvement from the pretraining values at 3-month follow-up and all skills except for announcement of the leader demonstrated a retention of at least 75% (table 1).

Table 1.

Results are listed as mean±SD

Variable	Pretraining	Post-training	3 Months	P value	Retention
PALS score	52%±0.1%	94%±0.1%	81%±0.1%*	<0.001	79%
Time to epinephrine	171±12 s	90±4 s	101±7 s	<0.001	87%
Time to pulse check	30±6 s	21±8 s	11±1 s	0.001	N/A
Time to compressions	38±6 s	15±1 s	19±1 s*	0.001	83%
Backboard placement	29%	100%	88%*	<0.001	83%
Rhythm announced	24%	100%	83%*	<0.001	78%
Leader role declared	12%	98%	62%*	<0.001	58%

P values shown are calculated for pretraining compared with 3-month values

*Denotes 3-month values that are statistically different from post-training values (p<0.05).

NA, not assessed; PALS, paediatric advanced life support.

Scores for each variable were evaluated at all three time points to determine if training year affected performance. Pretraining performance showed statistically significant differences in the performance by training year for time to epinephrine (figure 3), frequency of backboard use and correct rhythm announcement, but did not show a difference by class in overall PALS score (figure 4).

Figure 3.

Chart showing the times for administration of epinephrine for each group of learners before and after the study intervention as well as 3 months postintervention. PGY, postgraduate year.

Figure 4.

Chart demonstrating the improvement of the PALS score mean and the narrowing of variability between the learners. PALS, paediatric advanced life support; PGY, postgraduate year.

Discussion

This study shows poor baseline PALS performance among paediatric residents but significant improvement following RCDP training and very good retention at 3 months. This is critical as evidence is growing that paediatric residents consistently fail to adhere to PALS algorithms both in simulated and actual CPAs; we still have an incomplete understanding of how to improve this performance.^{5 8} A number of important factors likely contribute to this wide-reaching deficiency. First is the infrequency of total paediatric CPAs in the inpatient setting. Knudson et al reported the national incidence of inpatient CPA at just 0.77 per 1000 admissions.¹ Resident participation in resuscitations has likely further diminished over the past 10 years as 24 hours in-house coverage by paediatric critical care physicians has become increasingly standard.¹⁹ Some institutions have even established ‘code blue teams’ (specific groups of highly trained nurses and physicians who respond to all in hospital CPAs).²⁰ With some of these institutions restricting residents from joining the code blue team, resident exposure to paediatric CPA may nearly be eliminated. Infrequent clinical exposure, combined with relatively long periods between formal PALS training likely leads to lower mastery and higher levels of skill and knowledge degradation. Other barriers, like decreased performance during high stress, magnify the effects of infrequent practice.²¹

Baseline data obtained prior to RCDP training adds to the current literature supporting the inadequacy of current training methodologies. Despite regular simulation and mock code exposure and AHA PALS training every 2 years, our residents had an average overall PALS score pre-RCDP of 51.7%. This level of performance is consistent with resident performance in the studies by Donoghue et al ¹⁸ and Levy et al.²² Levy et al further evaluated resident performance pre-AHA and post-AHA PALS training. Following standard PALS training, the residents demonstrated an improvement of 10% of the PALS score.²² By contrast, absolute scores achieved by residents in our study improved by 42% immediately after training. Furthermore, our residents retained an impressive 33% improvement in absolute score at 3 months, giving a 3-month retention of 79%.

While overall resident performance showed deficiencies, especially concerning was the lack of statistically significant improvement in overall performance between firstyear, second-year and third-year residents. We interpret this data to mean that our current residency training is insufficient for residents to improve skills and achieve competency in paediatric resuscitation. This is not unique to our institution. Both Hunt et al ⁸ and Levy et al ²² found a similar lack of longitudinal improvement. To date, we are aware of only one study by Donoghue et al ¹⁸ that showed higher levels of resuscitation performance in more experienced residents.

Seeking to improve PALS training methodology, Hunt et al ¹⁶ adapted the educational concepts of deliberate practice and the mastery learning utilised by Wayne et al ^13–15 to create a novel form of simulation training that focused on rapidly repeated, deliberate practice interspersed with directed feedback. Following implementation of a RCDP curriculum, Hunt’s group demonstrated a significant improvement in resident performance during mock codes, especially with respect to early and continuous compressions and early shock for pulseless ventricular tachycardia.¹⁶ Our study supports Hunt’s results showing RCDP to be an effective methodology for teaching paediatric residents proper care of a patient in CPA. We demonstrate improvement both immediately after training and 3 months later in the performance of all measured resuscitation skills. Like Hunt, we saw significant improvements in time to chest compressions, but we also showed improvements in the use of backboard, time to epinephrine and use of effective communication skills. The results of our study are encouraging in that residents not only gained knowledge and skill immediately following training but also retained a high percentage 3 months after training.

The primary goal of our study was to evaluate learner retention 3 months after RCDP training. Knowledge retention has been increasingly studied in both medical and non-medical education and the data on resuscitation education shows significant opportunity for improvement.^{10–12 23} For example, Bell et al ²³ evaluated cognitive knowledge retention among internal medicine and family medicine residents following an online American Diabetes Association tutorial. At just 1 week post-training, new knowledge retention was less than 50%. Bhatnagar et al ²⁴ showed near complete knowledge decay 6 months after CPR training using typical ACLS methodologies including lectures and hands-on training. Given the infrequent but high-stakes nature of paediatric resuscitation, knowledge retention should be of pre-eminent importance when evaluating training methodologies.

This study demonstrated retention of nearly 80% for overall PALS performance 3 months after a single RCDP training session. This is a substantial advantage when compared with previously published data on retention following standard AHA ACLS training^10–12 and PALS procedural/psychomotor training^{25 26} Smith et al, for example, demonstrated a performance retention of just 22% 3 months after ACLS training.¹⁰ The higher retention seen in our study is likely the product of increased time in deliberate practice, improved retention following completion of multiple tests (testing effect) and improved memory after directed feedback.²⁷ Retention was significant for all variables, but noticeably higher decay was seen in leader role declaration (98% of residents performed this step post-training compared witho 62% at 3 months). This was also the skill that was most deficient prior to training (12%), and residents report practising this skill makes them the most uncomfortable. We hypothesise that less experienced physicians are less comfortable taking the leadership role and especially declaring this verbally.

Our hope is that this data encourages more widespread use of RCDP methodology and lays the groundwork for an evidence-based approach to PALS retraining intervals. Our data suggests that if trainees are provided RCDP training, then the training interval could be at least 3 months. However, this current study is limited in the use of only one time interval between training and retesting. Future studies are planned to evaluate knowledge retention at longer time intervals. These results will help us to more accurately determine PALS retraining intervals. Additional studies on the amount of retraining required would also be helpful; education literature has shown that retraining, even after significant knowledge decay, occurs much faster and with less practice than does training a learner anew.²⁸

This study has several limitations. Because of the controlled environment used for evaluations and training, participants were informed in advance of both the training and follow-up testing. This likely influences performance, but the notification process was similar in pretesting and follow-up testing. This study was conducted at a single institution and the training was performed by two study investigators. Expansion to additional institutions and inclusion of more trainers would be helpful in evaluating the ability to translate the effect in other environments. Finally, the study design does not include a control learner group. While we compare learning retention with that of historical studies, a true concurrent control group using traditional simulation with debriefing would have been more robust. We chose a single-arm study design as our interest was specifically in the skill and knowledge retention of RCDP training and a single arm allowed us a larger study population.

Conclusion

In conclusion, this study showed that RCDP training was associated with significant improvements in resident performance during simulated paediatric resuscitation. Furthermore, the improvement was highly retained 3 months later with no additional training. Additional studies are needed to determine whether this improvement translates to improved clinical care.