Just-in-Time, Just-in-Place Virtual Training in the Pediatric Emergency Department: A Novel Approach to Impact the Perfusion Exam

Abstract

Background

Early identification of shock is vital in decreasing morbidity and mortality in the pediatric population. Although residents are taught the perfusion portion of the rapid cardiopulmonary assessment at our institution, they perform it at the bedside with 8.4% completing 1 part of the assessment and 9.7% verbalizing their findings. Newer technologies, including virtual reality (VR), offer immersive training to close this clinical gap.

Objective

To assess senior pediatric residents’ performance of a perfusion exam and verbalization of their perfusion assessment following VR-based Just-in-Time/Just-in-Place (JITP) training compared to video-based JITP training. We hypothesized that JITP media training was feasible, and VR JITP was more effective than video-based training.

Methods

Residents were randomized to VR or video-based training during shifts in the emergency department. Clinical performance was assessed by review of a video-recorded patient encounter using a standardized assessment tool and by an in-person, two question shock assessment. Residents completed a survey assessing attitudes toward their intervention at the time of training.

Results

Eighty-five senior pediatric residents were enrolled; 84 completed training. Sixty-four (76%) residents had a patient encounter available for video review (VR 33; Video 31). Fourteen residents in the VR group (42.4%, 95% CI 25.5% to 60.8%) and 13 residents in the video group (41.9%, 95% CI 24.6% to 60.9%) completed a perfusion exam AND verbalized an assessment during their next clinical encounter (X² p-value 1.00). Fifty-one of 64 residents (79.7%) completed the two-step shock assessment; 50 (98%) agreed with supervising physician’s assessment. VR was rated more effective than reading, low-fidelity manikin, standardized patient encounters, traditional didactic teaching, and online learning. Video was rated more effective than online learning, traditional didactic teaching, and reading.

Conclusion

Novel video and VR JITP perfusion exam and assessment trainings are impactful and well-received by senior pediatric residents.

Background

Timely and accurate rapid cardiopulmonary assessments (RCPA) is a crucial initial step in recognizing and managing patients presenting in shock, which is often under-recognized in pediatrics.^1,2 Though the Pediatric Advanced Life Support (PALS) course provides trainees with a structured and standardized approach to the assessment of shock, it has not translated to sustained optimal performance of these skills. At our institution, Sobolewski et al found that senior pediatric residents do not promptly nor thoroughly perform an initial RCPA on actual patients in the resuscitation area despite it being an expectation.³ In this historical cohort, 6 (8.4%) of 71 senior residents assessed via video review completed any component of the perfusion exam and 7 (9.8%) of 71 verbalized their assessment of perfusion.³ Experience and repetition of clinical assessments have been shown to improve exam skills; but pediatric residents have limited clinical opportunities to apply them in patients at risk for or in shock.^4,5

Educators have attempted to address this experience gap through high-fidelity simulation, which leads to increased skill retention compared with didactic training and lower-fidelity simulation.^6,7 Many key clinical findings are unable to be accurately depicted on computerized manikins.⁸ Virtual reality (VR) offers an avenue to further enhance the level of realism and immersion through virtual environments and content. Specifically, VR allows the incorporation of key patient exam findings (eg, capillary refill, skin color, mental status) that were previously unable to be effectively simulated.⁹ However, there is a paucity of literature examining the use and impact of VR simulation-based training for identifying perfusion exam findings in critically ill simulated patients.

Just-in-time (JIT) is defined as a

specific task/intervention, which can immediately address the knowledge or skills deficit, be brief, focused and drive best practice, ideally through automated feedback

as close to the event or procedure as possible.¹⁰ In the emergency department, certain events are difficult to predict, and their acuity precludes a true JIT intervention. However, educators can complete just-in-place (JIP) training which refers to completing the intervention in the location in which the event or procedure is occurring.¹⁰ Although just-in-time/just-in-place (JITP) training has been used for procedural training, there is no literature surrounding its use to improve exam skills during high-acuity events.

Since Sobolewski’s study, there has been minimal additional training offered to residents to improve their completion of the RCPA at our institution. We believed that introducing such training during their shifts (JIT) in the ED (JIP) would heighten awareness and allow immediate application, two facets important for adult learning. This study aimed to assess senior pediatric resident performance of a perfusion exam and verbalization of their assessment of examination findings following VR-based JITP training compared to those exposed to video-based JITP training. We hypothesized that implementing JITP video and VR training would improve completion of a complete perfusion assessment compared to our historical cohort and that JITP VR-based training leads to increased completion of a complete perfusion assessment compared to JITP video-based training.

Methods

Study Design

This study was approved by the Cincinnati Children's Hospital Medical Center Institutional Review Board. It was a prospective, randomized study of pediatric residents at a large academic center. This study adheres to the STROBE guidelines for observational studies and guidelines outlined in the Declaration of Helsinki.

Setting

This study was conducted in a quaternary referral center with an annual Pediatric Emergency Department (PED) volume of 62,000 patients, with approximately 4500 (7%) patients cared for in the Shock Trauma Suites (STS). Only senior residents (PGY-2 and above) respond as members of the interprofessional team for patients treated in the STS. Senior residents are expected to perform the initial RCPA and verbalize their findings to the physician team leader (attending or third-year fellow) and the nurse team leader. At our institution, residents complete a mandatory online emergency department rotation orientation explicitly states that residents will perform the RCPA. In their orientation video, there is a portion that shows how they are expected to perform the RCPA in the resuscitation bay. Residents in the historical and current cohort receive traditional simulations, pre-intern and pre-3rd year PALS training, and direct teaching through patient encounters. There was no significant difference in training or orientation between the historical and current study cohort.

Study Participants

Eligible study participants were senior (PGY-2 and above) categorical and combined pediatric training program residents during their emergency medicine rotation. First-year residents were excluded as they do not provide care to critically ill patients in the resuscitation area. We attempted to recruit all eligible residents over a 12-month period.

Eligible residents had informed consent obtained by a Clinical Research Coordinator (CRC). They were enrolled by email or approached during one of their initial shifts and consented by a Clinical Research Coordinator (CRC). After enrollment, residents were randomized to receive either VR-based or video-based training.

Interventions

We developed two educational training experiences: (1) a VR-based simulation (Supplementary Videos 1 and 2) a de novo educational video (Supplementary Video 2) detailing the idealized performance of the perfusion exam. A control group was not possible as our IRB would not allow a prospective study where residents did not receive training. Residents individually completed the randomized training intervention with a member of the study team.

VR-Based Training Experience

The VR-based application was created by members of the study team over 1 year. It was constructed using the Unity™ development platform (Unity Technologies, San Francisco, CA) and experienced through an Oculus Rift™ (Meta Platforms Inc. Menlo Park, CA) headset and hand-held controllers. A virtual toddler avatar able to convey key physical exam findings, including mental status (alert, responsive to verbal stimuli, responsive to pain, unresponsive – commonly referred to as AVPU), capillary refill (<3 seconds or >3 seconds), palpable pulses (corresponding to heart rate with variable pulse quality spanning normal, weak, or absent), skin appearance (normal, mottled, gray, cyanotic), and skin temperature (warm, cold). IBM Watson™ (Armonk, NY) driven voice command using responses simulated the mental status examination, and the virtual patient would respond to basic questions if clinically appropriate. Capillary refill was assessed via a tactile trigger on the palm of the hand, sole of the foot, and chest overlying the sternum. Pulses were assessed via the Oculus controller haptic vibration function, with the ability for the pulse quality to vary in both rate and intensity. Finally, touching the hands of the virtual patient would trigger the patient’s hands to turn blue or red, corresponding to a cool or warm skin temperature, respectively.

The VR simulation consisted of a 15-minute experience with three preset phases: 1) a 5-minute, autonomous tutorial of how to interact with the VR avatar with normal exam findings, 2) a 5-minute simulation of compensated shock, and 3) a 5-minute simulation of decompensated (hypotensive) shock. Resident performed the 4 aspects of the perfusion exam and verbalized their assessment.

Video-Based Training Experience

The video training experience was recorded with a volunteer well-child acting as the patient and covered the same perfusion assessment content as the VR-based intervention. The video instrument included didactic instruction on assessing the perfusion exam, specifically mental status (using AVPU), central and peripheral pulses, capillary refill, and skin temperature and color. The learning objectives mirrored those of the VR simulation. The video footage was edited and produced using Final Cut Pro X (Apple, Cupertino, CA) and hosted and password protected on Vimeo.com (New York, NY). The video was 7 minutes in length. The development of the video was over 1 month.

Both interventions concluded with a structured, post-training debrief to answer subsequent questions the senior residents had on the perfusion assessment.

Outcomes

The primary outcome was performance AND verbalization of the perfusion exam at the bedside during the care of critically ill patients in the STS. If the resident completed 3 of 4 portions of the perfusion exam and verbalized an assessment (normal, shock – compensated or decompensated), it was considered complete. In the previous study (Sobolewski et al) performed in the same setting, residents were credited with complete performance of components of the RCPA if they 1) performed at least one element of the exam and 2) verbalized an assessment for each component of airway, breathing, and circulation; a definition that was less strict than this study.

Secondary outcomes were (a) participant reaction to and attitudes towards their allocated training and (b) comparison of resident’s perfusion assessment with the physician team leader’s perfusion assessment in actual cases.

Training

The residents were randomly assigned using an online block randomization tool in blocks of eight learners to receive either the VR-based or video-based intervention, stored in a secure web-based application (Research Electronic Data Capture [REDCap]). The intervention was completed in the PED and during the residents’ shift proctored by a trained study investigator (AS, GG, MZ).

Assessment of Outcomes

Following completion of the educational intervention, for each trained resident, the video of their first real patient cared for in the STS was reviewed. Video of STS care is recorded 24 hours a day on a secure server via B-Line’s LiveCapture platform (B-Line Medical, Washington, DC). Two study investigators, both with extensive experience in video-based review, were trained in data extraction using a standardized assessment tool (Supplemental Figure 1). The primary video reviewer (BS), blinded to intervention group, completed data extraction on every enrolled resident who had a video available. Every 10th video was independently co-reviewed by GG (also blinded to the intervention group assignment) and compared with the primary reviewer to prevent drift and provide quality assurance. Any discrepancies were discussed and adjudicated, and 100% consistency was required before BS moved on to the subsequent group of ten videos.

Resident reaction to and attitudes toward their allocated training experience were assessed immediately after the JITP training. One survey assessed attitudes towards learners’ allocated educational intervention and JITP training.¹¹ The VR group completed a second survey to assess realism and immersion of the VR environment and included the Simulator Sickness Questionnaire (SSQ) to determine VR-related side-effects (Supplemental Figure 2).¹²

Resident agreement with the physician team leader’s assessment of perfusion was assessed using a brief 2-question tool. If the resident had not verbalized an assessment at the 5-minute mark, a clinical research coordinator (CRC) prompted the nurse team leader to ask, first the resident, then the physician team leader if they thought the patient was in shock (to prevent bias in the resident’s response). The CRC recorded answers on an electronic tool using a digital tablet with input uploaded into REDCap.

Data Analysis

Demographic data was assessed via descriptive statistics. The primary outcome was assessed using chi-square statistics to determine if there was a difference between the VR and the video group on completion of a full perfusion exam and assessment. Distributive frequencies were generated for the VR and video groups with 95% confidence intervals, with associated p-value to assess for differences between the two groups. Secondary outcomes were categorized by intervention and characterized with descriptive statistics.

Results

Enrollment

Eighty-five residents were enrolled and randomized, with 84 completing training (Figure 1). One withdrew from the VR group due to side effects, resulting in 41 in the VR group and 43 in the video group. Allocated groups were balanced for nearly all demographic categories (Table 1). Most were categorical pediatric residents (71%); the majority were in their second year of training (55%).

Figure 1.

Enrollment. Bold represents number of residents.

Table 1.

Demographic Information

Demographic
	VR (n)	Video (n)
Total Trained	41	43
Residency Year
2nd	22	24
3rd	15	13
4th	4	6
Program
Pediatrics	31	29
Medicine/Pediatric	5	8
Pediatric/Physical Medicine and Rehab	1	1
Pediatric/Child and Adult Psychiatry	2	1
Pediatric/Genetics	0	1
Pediatric/Neurology	2	3

Abbreviation: n, number of participants.

In total, 64 (76%) residents had a clinical STS video available for review (VR 33; Video 31). Residents without an available video were due to (1) a resident not having an STS encounter following training, (2) a need to alter research guidelines due to SARS-CoV-2, or (3) videos being deleted by institution protocol prior to review.

Primary Outcome

There was no difference in the primary outcome between groups. For the primary outcome, 14 of 33 residents in the VR group (42.4%, 95% CI 25.5% to 60.8%) and 13 of 31 residents in the video group (41.9%, 95% CI 24.6% to 60.9%) completed 3 of 4 physical exam components AND verbalized an assessment during their next clinical STS encounter (difference 0.0049, standard error of difference 95% CI −0.237 to 0.247, X² p-value 1.00). Twenty-two of 33 residents in the VR group (66.7%, 95% CI 48.2% to 82.0%) and 20 of 31 residents in the video group (64.5%, 95% CI 45.4% to 80.8%) completed 3 of 4 physical exam components WITHOUT a verbalized clinical assessment during their next clinical STS encounter after JITP training. Sixteen of 33 residents in the VR group (48.5%, 95% CI 30.8% to 66.5%) and 15 of 31 in the video group (48.4%, 95% CI 50.2% to 66.9%) verbalized a perfusion assessment during this encounter. Of the 4 individual physical exam components, pulse assessments were most often completed while skin assessments were least often completed (Table 2).

Table 2.

Frequency of Individual Exam Components Completed by Group

	VR n (%, 95% CI)	Video n (%, 95% CI)	p-values
			Fisher’s Exact Test
Pulses
Exam
Yes	30 (90.9, 75.7 to 98.1)	29 (93.6, 78.6 to 99.2)	1.00
Reported
Yes	21 (63.6, 45.1 to 79.6)	19 (61.3, 42.2 to 78.2)	1.00
Capillary Refill
Exam
Yes	29 (87.9, 71.8 to 96.6)	24 (77.4, 58.9 to 90.4)	0.33
Reported
Yes	18 (54.6, 36.4 to 71.9)	16 (51.6, 33.1 to 69.9)	1.00
Mental Status
Exam
Yes	22 (66.7, 48.2to 82.0)	20 (64.5, 45.4 to 80.8)	1.00
Reported
Yes	11 (33.3, 18.0 to 51.8)	9 (29.0, 14.2 to 48.0)	0.79
Skin Assessment
Exam
Yes	6 (18.2, 7.0 to 35.5)	7 (22.6, 9.6 to 41.1)	0.76
Reported
Yes	3 (9.1, 1.9 to 24.3)	6 (19.4, 7.5 to 37.5)	0.3

Abbreviations: n, number of participants; CI, confidence intervals.

Secondary Outcomes

Eighty-four residents completed the post-training survey (41 VR, 43 Video). Residents agreed or strongly agreed that both VR and video interventions reinforced key pieces of the assessment of patients with poor perfusion or shock (VR 41, 100%; Video 41, 95.3%) and would impact how they assess patient perfusion (VR 41, 100%; Video 41, 95.3%). VR and video were both felt to depict the spectrum of the perfusion exam, including decompensated (hypotensive) shock (VR 40, 97.5%; Video 34, 79.1%).

On a 3-point scale from less effective = 1 to more effective = 3, residents felt VR to be more effective than standardized patient encounters (mode 27, 64.1%), low fidelity manikin training (mode 36, 87.8%), traditional didactic teaching (mode 39, 95.1%), online learning (mode 39, 95.1%), and reading (mode 40, 97.6%) (Figure 2a). Residents felt video was more effective than online learning (mode 30, 71.1%), traditional didactic teaching (mode 30, 71.1%), and reading (mode 39, 90.1%) (Figure 2b).

Figure 2.

(a) Virtual reality educational interventions compared to other educational modalities. (b) Video educational interventions compared to other educational modalities.

Participants universally valued JITP training. Amongst all residents, 85% agreed or strongly agreed that they enjoyed JITP format, 88% felt JITP helped to better teach key points on the topic, and 73% felt JITP helped residents experience what they did not understand about the topic’s key points. Whereas, 78% disagreed or strongly disagreed that JITP was an inconvenient method of learning, and when asked if they preferred other formats over JIPT, 49% disagreed, 44% were neutral, and 5% agreed.

No VR-trained resident experienced severe side effects (Figure 3). One resident withdrew early due to moderate nausea, difficulty concentrating, and dizziness with eyes open. The most common side effects were general discomfort (33% slight, 5% moderate), eye strain (33% slight, 5% moderate), and difficulty focusing vision (19% slight, 9% moderate). All residents, including the resident who withdrew, were able to return to and perform clinical duties following either training method.

Figure 3.

Proportion of residents who experienced none, at least one slight, at least one moderate, or any severe side effect.

A 2-step assessment was completed for 51 of the 64 residents (79.7%) with video-assessed performance (VR 26; Video 25). Prompting by the CRC at 5-minutes was required in all but one encounter. Three (6%) patients were identified as in shock by the residents. There was 98% (50 of 51) agreement between resident and attending physicians; in the 1 disagreement the resident felt the patient was in shock and the physician team leader did not.

Discussion

There was not a significant difference between the JITP VR and JITP video intervention groups in completion of the perfusion exam (p-value 0.93) and verbalizing assessment (p-value 0.81). Specifically, 42.4% of residents in the VR group and 41.9% in the video group did both a full exam and verbalized their assessment to the team leader. Both interventions improved outcomes compared to 8.4% completing any component of the perfusion exam and 9.8% verbalizing their assessment of perfusion in the historical cohort. Of those who completed the CRC-prompted questionnaire in the resuscitation bay, 98% agreed with their supervising physician’s assessment of perfusion. In comparison to other methods, VR was felt to be more effective in teaching the perfusion exam and assessment than standardized patient encounters and low fidelity manikin training. Both video and VR interventions were felt to be better than traditional didactic teaching, online learning, and reading. Additionally, residents felt that JITP training while on a clinical shift was convenient and an enjoyable way to learn the perfusion exam and assessment.

Training for low frequency, high acuity events has always been challenging, especially in the pediatric setting. Changing patient volumes have limited trainee exposure to patients in severe distress.¹¹ Although there was an improvement, it is still concerning that only <50% of residents verbalized their assessment. This further emphasizes the need for focused deliberate practice. While there was no statistical difference in resident performance between the VR and video-based groups, VR offers a more immersive environment for that deliberate practice. In medical education, VR has primarily been used by the surgical subspecialties and has yielded mixed results.^6–9 In pediatrics, a 2016 study by Zaveri et al noted that pediatric resident preparation for procedural sedation by VR was more enjoyable and equally efficacious compared with online modules.⁹ Zackoff et al used a VR-based simulation to teach medical students how to recognize respiratory distress.¹³ As the technology improves, virtual avatars will offer more realistic and immersive experiences including hard to simulate findings such as pulse quality and capillary refill. Residents completing this study almost unanimously felt that VR accurately depicted the perfusion exam, consistent with other studies.^13,14 Our results support the use of video and VR in medical education as tools. This study builds the growing evidence that VR is a feasible educational tool and adds that it can be used as a just-in-time, just-in-place tool.

Side effects were common, though minor, in most VR-trained residents. No resident experienced severe side effects and only 1 (2%) resident withdrew due to moderate side effects. Previous studies in VR have shown that may cause transient effects beyond the SSQ.¹⁵ All residents returned to work and although not formally assessed, no anecdotal feedback regarding detrimental effects were reported. Overall, we feel the potential benefits of JITP VR-based training outweigh the potential risks.

Limitations

First, a third control group of residents who received no JITP training on perfusion would have been ideal, but our institution would not have approved a prospective study where training was withheld from a cohort of residents.

Second, the video training session was recorded using a healthy volunteer, thus limiting our ability to show some abnormal findings; namely abnormal color, pulse quality and capillary refill. Since each resident was trained with only one method, we could not assess whether VR-based or video-based training was preferred.

Third, a majority experienced at least 1 side-effect, although most were identified as mild. While only 1 resident reported symptoms strong enough to withdrawal and all residents were able to return to clinical activities; delayed or persistent effects were not evaluated. The potential for or existence of delayed side effects from VR-based training is relatively unknown in the literature. Given the overall positive feedback, we do not feel this was a hinderance to the resident’s education or engagement.

Finally, we were limited to a single center with the dedicated resources and infrastructure to build the VR program and complete in situ simulation and JITP training in the PED with the help of CRCs. A cost estimate for this type of educational modality including Oculus Rift headset, controllers, and a computer capable of running the program is $2000, although our equipment has been used for multiple, unique projects. However, our results show that any JITP training is valuable to and creating a less resource-intensive video can provide benefit to trainees.

Next Steps

Our next steps include a direct comparison with computerized manikin simulation, to further assess the benefits and value of VR in clinical training, and assessment of longitudinal knowledge and skill retention at the clinical bedside. Furthermore, gaining an understanding of why certain exam findings were completed more frequently compared to others may help focus interventions. With technology continually advancing, further evaluating VR’s use in simulating difficult to simulate environments or situations to improve experiential learning is needed.

Conclusions

Our novel JITP training methods for the perfusion exam and assessment were feasible in our clinical environment and well-received by senior pediatric residents. Both VR- and video-based JITP training improved examination and assessment of patient perfusion at the bedside compared to a historical cohort although one modality was not better than the other.

Funding Statement

There is no funding to report.

Author Contributions

All authors made a significant contribution to the work reported, whether that is in the conception, study design, execution, acquisition of data, analysis and interpretation, or in all these areas; took part in drafting, revising or critically reviewing the article; gave final approval of the version to be published; have agreed on the journal to which the article has been submitted; and agree to be accountable for all aspects of the work.

Disclosure

The authors report no conflicts of interest in this work.