Development and Evaluation of a Rural Longitudinal Neonatal Resuscitation Program Telesimulation Program (MOOSE: Maine Ongoing Outreach Simulation Education)

Abstract

Objectives:

Neonatal resuscitation is a high acuity, low-occurrence (HALO) event and many rural pediatricians report feeling underprepared for these events. We piloted a longitudinal telesimulation (TS) program with a rural hospital’s interprofessional delivery room teams aimed at improving adherence to Neonatal Resuscitation Program (NRP^®) guidelines and teamwork.

Study Design:

A TS study was conducted monthly in one rural hospital over a ten month period from November 2020 to August 2021. TS sessions were remotely viewed and debriefed by a neonatologist and a simulation educator. Sessions were video recorded and assessed using a scoring tool with validity evidence for NRP^® adherence. Teamwork was assessed using both TeamSTEPPS 2.0 Team Performance Observation Tool and Mayo High-Performance Teamwork Scale.

Results:

We conducted ten TS sessions in one rural hospital. There were 24 total participants, who rotated through monthly sessions, ensuring interdisciplinary team composition was reflective of realistic staffing. NRP^® adherence rate for full code scenarios improved from a baseline of 39% to 95%. Compared with baseline data for efficiency, multiple NRP^® skills improved (e.g. cardiac lead placement occurred 12x faster). Teamwork scores showed improvement in all domains.

Conclusions:

Our results demonstrate that a TS program aimed at improving NRP^® and team performance is possible to implement in a rural setting. Our pilot study showed a trend towards improved NRP^® adherence, increased skill efficiency, and higher quality teamwork and communication in one rural hospital. Additional research is needed to analyze program efficacy on a larger scale and to understand the impact of training on patient outcomes.

Introduction

Neonatal resuscitation is a high acuity, low occurrence (HALO) event. Optimal newborn outcomes depend on skillful and timely implementation of the steps of the Neonatal Resuscitation Program (NRP^®). Successful implementation of NRP^® guidelines requires at least three fundamental components: (1) clinical skills, (2) teamwork, and (3) maintenance of these skills. Nationwide, more than 400 maternity services closed between 2006 and 2020¹, and in Maine, 17% of delivery hospitals have closed, leading to “Maternity Deserts”. Additionally, physicians’ availability to attend deliveries is decreasing. One study demonstrated that in a national cohort of pediatricians and family medicine physicians only 25% attended a delivery in the last year and of those 44% felt inadequately prepared during at least one delivery². Additional training is needed to better prepare rural delivery room clinicians.

Many centers have effectively used in situ simulation to improve interprofessional (IP) team performance in resuscitation³. This program was born from a needs assessment of hospitals in our system who requested increased access to simulation training for NRP with neonatology support. However, in situ simulation is costly and often unavailable in rural settings without nearby simulation centers, particularly at the frequency needed to prevent skill decay over time^4,5. Telesimulation (TS) is a process using telecommunication and simulation to educate, train or assess learners from a remote location^6,7 and may provide innovative learning opportunities. The use of TS is an increasingly emerging field for delivering medical education^8–10. TS circumvents geographical, scheduling, and resource allocation challenges including limited specialists and simulation experts in rural areas. Neonatologists have called for wider use of TS to sustain resuscitation skills¹¹, but assessment of longitudinal TS programs is lacking. We aimed to pilot a monthly TS training program at one rural community hospital assessing NRP^® adherence, teamwork, and communication over time.

Methods

Study design:

We implemented a monthly TS program at one rural hospital. We assessed NRP^® adherence, teamwork, and communication skills. We collected data on attendance and frequency of sessions including cancellation rate. The study was reviewed as exempt research.

Setting:

The TS sessions occurred in a delivery room within a rural hospital in Maine with approximately 200 annual births. Simulation and neonatology experts from the quaternary hospital participated remotely. The community hospital and quaternary hospital are part of a healthcare network of nine hospitals that cover much of southern Maine and eastern New Hampshire.

Participants:

Rural delivery clinicians with NRP^® training were eligible to participate including pediatric physicians (on call, off-site to respond to deliveries), hospital-based respiratory therapists, and labor and delivery nurses. These teams were composed of different individuals at each session to replicate the variability in daily staffing, one provider, a respiratory therapist and two nurses.

Prior to participation, clinicians were provided information on the study. A full day of skills training and in situ neonatal resuscitation simulation at the rural hospital was led by the neonatologists and simulation team, conducted 14 months prior to the start of the TS sessions. This 14-month delay between the in situ and starting TS was due to equipment procurement delays. Demographic data collected was limited to clinical role. Participants were not compensated.

Equipment:

The rural hospital was supplied with a high-fidelity mannequin, Sim NewB and audiovisual (AV) system, SimCapture (Laerdal Medical - Wappingers Falls, NY), including all software and hardware required to operate the mannequin. SimCapture was outfitted in the nursery including two cameras permanently mounted with a fixed internet connection. The cameras were mounted above the warmer and in the corner of the room to allow direct views above the mannequin and of the entire room for team positioning (Figures 1). The AV system provided live streaming, allowing experts to observe remotely (with audiovisual observation) in real-time and recordings. The AV system allowed for annotation and timestamping for task completion (e.g. placement of ECG leads). Zoom was utilized for debriefings.

Figure 1:

Telesimulation Set-up

The schema on the left presents the setup of the TS equipment in the newborn nursery at the rural hospital. Camera A is positioned over the mannequin to assess the technique of the interventions. Camera B is positioned in the corner of the room to assess team dynamics and communication. The image on the right presents the view seen by the remote team of neonatology and simulation specialists, located ~75 miles away.

Set up:

The rural hospital identified simulation lead team members, including one nurse, one respiratory therapist, and one physician. These IP leads were responsible for recruitment and scheduling of respective clinical teams. The nurse and respiratory leads were trained to run the mannequin and AV equipment and were present at all sessions. They received training in simulation prebriefing and debriefing processes by the simulation center’s nurse educator (MC).

Telesimulation Scenarios:

TS sessions were scheduled during regularly scheduled staff meetings at the rural hospital to maximize participation. The first time a pediatric physician participant attended a TS, an airway scenario (airway case) was performed. Subsequently, each pediatrician participated in two identical full code scenarios (code case) these sessions were scored using an adapted version of the scoring tool for adherence to NRP^{® 12,13}. This resulted in a total of six full code scored sessions.

Debriefing Session:

Upon conclusion of the TS case, participants virtually joined the IP team of neonatologists, neonatal nurses, and simulation educators via Zoom for a facilitated debriefing. The “debriefing with good judgment” format was used, which included the phases of reactions, analysis, and summary^14–19 (Appendix A). Neonatologists led the discussion on resuscitation skills, while the simulation educators led the teamwork, communication, and safety discussion.

NRP^® Adherence Scoring Tool:

The scoring tool was adapted from a previously published tool with validity evidence¹² to assess NRP^® performance. Adaptations reflect updates from the eighth edition of NRP^®, including the addition of ECG monitoring, lack of immediate suctioning for non-vigorous meconium infants, and the removal of naloxone use ²⁰. Assessment of eight domains occurred, including group function, preparation and initial check, heart rate evaluation, oxygen, bag mask/positive pressure ventilation, intubation, cardiac compressions, and drugs/volume (Appendix C). We evaluated efficiency by annotating recorded TS sessions and marking for time to complete key NRP^® tasks within AV software. Recorded tasks were marked from video time of birth to the point of completion of the task. Baseline scores were generated using the average of the two in situ full code scenarios.

NRP^®Adherence Scoring:

Using this adapted scoring tool, NRP^® adherence was assessed by two board-certified, simulation instructor-trained neonatologists (AZ and MM)^12,13. Scores were generated by each reviewer utilizing an anchoring tool (Appendix B) that was developed by our team and reviewed and endorsed by an expert panel, Neonatal Improving Pediatric Acute Care Through Simulation (NeoImPACTS)²¹ to promote consistency and inter-rater reliability. Differences were reconciled through discussion and re-review of the recordings. Scores were calculated from the sum of the scores across the eight subcategories based on the appropriateness and execution of the skill (Appendix C). Overall scores were generated by combining subscores.

Team Performance Scoring Tool:

Team performance was evaluated using two assessment tools: Team Strategies and Tools to Enhance Performance and Patient Safety (Team STEPPS) 2.0 Observation tool and the Mayo High-Performance Teamwork Scale (MHPTS). Each tool emphasizes different aspects of teamwork. Team STEPPS assesses five domains: team structure, communication, leadership, situation monitoring, and mutual support. The MHPTS rates teams in sixteen essential skills of teamwork, including leadership and communication skills, and produces a summary rating. This tool has been shown to provide reliability and consistency in rating key behaviors during crisis resource management training²².

Team Performance Scoring:

Videos were analyzed independently by two experienced simulation educators (JH and MC). Team STEPPS domain scores were assigned using a Likert scale from 1 (very poor) to 5 (excellent). Sub-scores were summed to generate a total score. In the MHPTS, because items 9–16 rely on branching logic (only used if certain behaviors occur, i.e., disagreement amongst team members), we have only included items 1–8 in the summary rating for a maximum possible summary rating of 24. This ensures a consistent denominator between events and is a method used by previous similar studies^23,24. Total raw scores for each event are included in Appendix D. Differences were reconciled through discussion and re-review of the recordings. The study team created a Team STEPPS anchoring tool (Appendix E) with specific examples for clarity as they scored, promoting consistency and inter-rater reliability. Teamwork scores were converted to a percentage to total possible points.

Analysis:

Analysis was restricted to descriptive trends due to the inclusion of only one site for this pilot study. We avoided inferential statistics because of the non-generalizability of a pilot program at a single site. Descriptive statistics and visualization were conducted in Rv.4.2.1.

Results

Simulation Sessions:

There were ten simulation sessions: four airway cases and six code cases (Figure 2). One session was rescheduled due to clinician availability. SimCapture software updates required additional preplanning and troubleshooting prior to the following session to address connectivity issues; all sessions were able to be recorded. No sessions were canceled due to technology difficulties.

Figure 2:

Timeline of Simulation Sessions

Study Participants:

The ten TS sessions included a total of 24 individual delivery team members represented by four physicians, eleven nurses, and nine respiratory therapists. Sixteen of these clinicians participated in the six code case sessions included in analysis, eight of whom participated in more than one session.

NRP^® Adherence:

The average baseline NRP^® adherence rate for the full code scenario in situ training was 39%. Through the six subsequent full code TS sessions, there was improvement from baseline to a maximum of 95% adherence (Figure 3). The fifth session was notable for a relative decrease in NRP^® adherence compared to the prior two sessions. During this session’s debriefing, a participant self-identified as inexperienced in both simulation and real-life neonatal resuscitation. In addition to the overall NRP^® improvement, all subcategories improved over time as noted in Figure 3.

Figure 3:

NRP^® Adherence over the Training Period

Dots represent the cumulative NRP adherence score of the team for a given simulation.

Task Efficiency:

Compared with baseline data for efficiency, multiple NRP^® skills also improved over TS sessions; ECG lead placement occurred 12x faster (0:31 sec vs. 6:21 min), a definitive airway was placed 1.9x faster (6:06 vs. 11:25 min), and the first epinephrine dose was given 2.2x faster (7:54 vs 17:06 min).

Teamwork/Communication:

Changes in overall TeamSTEPPS and TeamSTEPPS subcategories can be seen in Figures 5 and 6. Overall TeamSTEPPS ratings improved over time (baseline 69%, 63%, 73%, 78%, 88%, 67%, 83%) (Figure 6). Overall MHPTS ratings improved over time as well, culminating in a perfect performance rating (79%, 81%, 80%, 91%, 97%, 85%, 100%). Similar to NRP^® adherence scores, teamwork scores for the fifth session had a relative decrease compared to the prior sessions (Figure 5).

Figure 5:

TeamSTEPPS and Mayo Team Performance Scores over the Training Period

Figure 6:

TeamSTEPPS Subcategory Scores over the Training Period

Discussion

The results of our pilot study demonstrate that a longitudinal NRP^® TS program is feasible in a rural hospital. This program led to a trend in improved adherence to neonatal resuscitation skills and IP team performance across all measured categories in a rural hospital delivery room. This study is unique as it presents longitudinal training opportunities, examining both skill acquisition and team performance and incorporates realistic variability in team composition. Despite this variability, results demonstrated an improved trend in resuscitation skills and teamwork. This suggests that TS may offer a practical means to train for neonatal resuscitation at rural hospitals.

Simulation provides opportunities for hands-on deliberate practice of HALO events and is critical in settings where opportunities for resuscitation are rare. Prior studies demonstrated that high-fidelity in situ simulation effectively improves team performance in multiple types of resuscitation^25–28, including neonatal resuscitation^29–32. Barbato et al used simulation to train providers at 25 community hospitals to manage preterm infant thermoregulation effectively. They demonstrated improvement in performance both immediately and a year later³³. While their study shows skill retention in the focused area of thermoregulation, other studies looking at more comprehensive resuscitation performance indicate skills decay over time with performance rates diminishing as soon as six months^25,34,35. Our program specifically sought to use TS on a regular basis to support skill retention. Haynes et al also used “low dose, high frequency” in situ simulation to combat this decay and found it was an effective solution to maintain neonatal ventilation skills, but not overall competence scores, in multidisciplinary teams in a large teaching hospital with up to twice monthly trainings³¹. In rural environments, twice per month in situ training is impractical, as it is far too resource intensive in terms of time and travel. Using TS can offset some of the resource requirements inherent in high fidelity simulation (e.g. avoid travel to the rural hospital), enabling a more practical training solution. However, it is important to note that telesimulaion is still costly, the program as described required approximately $40,000 of investment in equipment to implement.

Our pilot study assessed team performance and still showed a steady upward trajectory in NRP adherence scores over time, despite variable individual attendance. Consistent emphasis on communication and empowering all team members, regardless of role, to speak up during resuscitation to suggest improvements or correct errors may have contributed to this by emphasizing the benefits of “collective intelligence” over individual knowledge^36–38. Improvements in teamwork may be attributable to an intentional debriefing structure where the five key principles of TeamSTEPPS (leadership, team structure, situation monitoring, communication, and feedback) were emphasized. Expansion of this model throughout additional hospitals in our health system will allow us to understand the impact of team composition and individual versus team performance.

Session five reflects a deviation in the upward trend of performance over time. In session five, scores dramatically fell across both skills and teamwork categories. One notable difference was the lack of leadership during critical times of the simulated scenario. The initial clinical team did not perform the correct beginning steps of NRP^® and prioritized cardiac compressions over establishing effective positive pressure ventilation, leading to lower scores for NRP^® adherence. During the reflective debrief one participant identified that they relied upon their background as an emergency room nurse and Pediatric Advanced Life Support (PALS) and Advanced Cardiac Life Support (ACLS) protocols (both of which focus on the importance of circulation and cardiac compressions and are a significant divergence from NRP^®’s focus on the initial steps of resuscitation and establishment of effective ventilation). Without strong leadership from another team member to correct course, the resuscitation skills scores were much lower. Effective leadership is critical to team performance and can affect outcomes^39,40. Both team performance tools used in our pilot assess the team leadership, which could account for the concomitant decline in those scores as well. While it is impossible to know for certain the role if any, these factors played in the aberrant performance, it suggests that both individual and team dynamics can play a role in successful resuscitation and merits further study.

There are several strengths of our study. Our study is unique because it measures multiple elements of resuscitation skill performance over time. Most available published reports of TS are single session interventions, such as that of Mileder et. al., who described success using TS to improve NRP^® knowledge in a one-time low-fidelity session⁴¹. Another novel aspect of our pilot study was team performance assessment over time during TS training with standardized assessment tools. TeamSTEPPS and MHPTS are both well-established team performance scoring tools. While TeamSTEPPS lacks validity evidence for use as a stand-alone tool (outside the TeamSTEPPS program), we opted to use it in addition to MHPTS because the two tools take different approaches to teamwork evaluation, and this was a pilot study. The inclusion of both tools, as well as the use of two experienced evaluators strengthens our study as the results mirror each other, and both show improvement over time. Participants were scheduled for TS sessions according to professional role, (pediatrician, nurse, respiratory therapist) rather than deliberately chosen individuals per session to ensure coverage or consistency. The successful implementation of this TS pilot program, despite multiple challenges common to rural health systems including decreased staffing, census fluctuations, and staff turnover, supports the feasibility of the model.

There are also significant limitations to this study. This study piloted a longitudinal TS program in one hospital prior to planned expansion across our rural health system. As such using inferential statistics to make claims about the generalizability of similar performance improvement to other rural hospitals is not appropriate. During the COVID-19 pandemic, this hospital experienced a significant delay in TS equipment procurement, leading to a 14 month delay to starting telesimulation after the on site visit that was used to calculate baseline scores. Other factors may have influenced improvements from baseline to the first TS, however we still noted improvements from between TS visits. Post-pandemic, we also saw higher-than-normal rate of staffing changes, which may have influenced team performance. Technological difficulties are challenging with any simulation training, perhaps more with TS training. We experienced technological issues including firewall challenges and connectivity of TS technology but were generally able to overcome them using local expertise. This expertise may not be present at all small rural community hospitals, especially those not affiliated with a larger simulation center. The pilot’s success depended on local teams’ buy-in, including support from leadership to allow team members to attend trainings and individual motivation to ensure consistent attendance. While this study suggests a trend toward improvement in neonatal resuscitation skills and team performance, the gold standard of outcome measurement is real-life perinatal morbidity and mortality measures. In rural hospitals with low delivery volumes, neonatal resuscitation is a HALO event, and adverse outcomes are multifactorial. A large multi-centered study will be needed to analyze the effect of TS training on real patient outcomes.

Future directions include expanding this model across our health system to further test the significance of findings and future multicenter collaboration to analyze the implementation of a similar model in different health systems and geographic areas. In our expansion program, we will use linear mixed effects models with hospital as a random effect to test the generalizability of improvement in NRP^® adherence over time. We will also perform a secondary analysis with unique participant identifiers as a fixed effect to understand the impact of individuals on team performance. We are also seeking to better understand contextual factors including staffing models, team size, and training frequency for the successful implementation of TS training to improve generalizability.

Conclusions

Our pilot study suggests that longitudinal TS training in native clinical environments with realistic team composition is feasible to maintain readiness for neonatal resuscitation in a rural hospital. Our program utilized assessment tools to measure resuscitation and team performance over time and showed trends toward improvement in both. Through ongoing partnership with ImPACTS collaborative, we are utilizing their successful hub-and-spokes model in a NeoImPACTS arm to replicate this resuscitation training program, mirroring the initial model developed for ImPACTS ED by offering resources to implement the program and develop nationwide data repositories.

Figure 4:

NRP^® Adherence Subcategory Scores over the Training Period

Key points:

• Optimal newborn outcomes depend on skillful implementation of NRP.

• Telesimulation can deliver medical education that circumvents challenges in rural areas.

• A longitudinal NRP TS program is possible to implement in a rural setting.

• A rural NRP telesimulation program may improve interprofessional resuscitation performance.

• A rural NRP telesimulation program may improve interprofessional resuscitation teamwork.

Abbreviations:

NRP^®

Neonatal Resuscitation Program^®

TS

telesimulation

HALO

high acuity, low occurrence

IP

interprofessional

AV

audiovisual

TeamSTEPPS

Team Strategies and Tools to Enhance Performance and Patient Safety 2.0 Observation tool

MHPTS

the Mayo High-Performance Teamwork Scale

NeoImPACTS

Neonatal Improving Pediatric Acute Care Through Simulation