Sequential Behavioral Analysis: A Novel Approach to Help Understand Clinical Decision-Making Patterns in Extended Reality Simulated Scenarios

Summary statement

Extended Reality (XR)-based simulation training offers unique features that facilitate collection of dynamic behavioral data and increased immersion/realism while providing opportunities for training healthcare professionals on critical events that are difficult to recreate in real life. Sequential analysis can be used to summarize learning behaviors by discovering hidden learning patterns in terms of common learning or clinical decision-making sequences. This project describes the use of sequential analysis to examine differential patterns of clinical decision-making behaviors in observed XR scenarios, allowing for new insights when using XR as a method to train for critical events and to trace clinical decision making.

Introduction

Despite the fact that decision-making for most clinical situations is well prescribed, there is typically some variation in the exact order of steps performed during these situations, either due to clinician experience or training background.¹ Importantly, any delay in correct diagnosis and treatment may result in increased morbidity and mortality.² This is particularly important for clinical situations that are relatively rare and thus difficult to master in routine clinical practice. To this end, simulation-based instruction has become an important part of critical clinical management training.^3–5

The use of Extended Reality (XR) simulation to train healthcare professionals has begun to emerge as a viable, innovative and scalable tool in healthcare education.^6,7 When compared to current manikin–based simulation training, XR-based training offers several unique features that provide greater accessibility, automatic collection of behavioral data, improved resource utilization, and increased immersion/realism while still providing opportunities for training healthcare workers on the critical ‘high acuity, low frequency’ events that are difficult to recreate in real life. Simulation-based training with this XR innovative modality allows learners to make decisions, as well as make mistakes, without risk to the patient.⁶

Sequential analysis is a technique which helps identify sequential patterns present in a large amount of data. Use of sequential analysis has been suggested as a means to study learners’ sequences of behaviors and transitions between them.^8,9 In essence, sequential analysis helps detect whether the occurrence of one event is linked to the subsequent occurrence of another event in real time.^10,11 Despite the fact that sequential studies provide correlational data only, the identification of close temporal relationships between two events can be used to identify educational targets for improvement and potentially be linked to learning outcomes. While previous work has demonstrated the prominence of sequential analysis at identifying sequential behavior patterns in educational research in general, very few studies have applied these advanced methods in the context of healthcare simulation.^12–15

To date, most research in clinical simulated environments has concentrated on technical skills and non-technical practices assessed by subjective human raters. Although these assessment approaches have been critical in advancing the way in which providers assess skills in simulation training, limitations of these techniques are that they are labor-intensive, costly, prone to personal judgement and focus primarily on global rating scales and outcomes rather than processes. With support using actual data, this paper will discuss how sequential analysis can be used to identify patterns of decision-making among trainees and practicing healthcare providers in simulated XR scenarios.

Conceptual Underpinnings

Extended Reality (XR) is a concept that encompasses all immersive technologies, including virtual reality (VR), augmented reality (AR), and mixed reality (MR). All are designed to extend perceptions of reality by fusing or overlaying real-world components with computer-generated features and by providing a fully immersive virtual experience that users can interact with. Whereas learner assessment in conventional simulation typically evaluates global actions and performance, the virtual environment allows for the collection of discrete, nuanced, and dynamic data of processes that unfold over time and which were previously either difficult or impossible to observe during conventional simulation training.^16,17 The collection of behavioral log data (computer-generated timestamped records of human activity) can potentially allow for the examination of meaningful associations, observation of trends, and provision of learner-specific feedback to each participant.^6,18 Importantly, whereas conventional simulation provides a more varied experience between learners, XR can offer a more consistent learning experience by providing stimuli that are standardized and which respond to learners’ responses in a more reliable manner.^19–21

Sequential analysis can be used to summarize learning behaviors by discovering hidden learning patterns in terms of common learning or clinical decision-making sequences. Sequence analysis can be used to investigate research questions: such as: does behavior A lead to or hinder behavior B, C, or D? Which of the following actions (A, B, or C) makes behavior D more likely to occur? Which of the behaviors A, B, or C can delay behavior D from occurring? Or, does behavior A trigger more of the same subsequent behavior?²² For example, Kolbe et al. in 2014 used sequence analysis to examine behavior patterns of high- and low-performing anesthesia teams.²³ The results of sequence analysis revealed that higher-performing teams were shown to have patterns in which monitoring behavior was followed by speaking up, providing assistance, and giving instruction, as well as patterns in which talking to the room was followed by improved talking to the room and fewer instructions.

A number of educational studies have identified the relationships between behavioral patterns and team or individual performances, e.g., the quality of discourse, essay quality, and performance in the construction of knowledge. For example, to examine students’ learning behaviors in a computer-based learning environment, Kinnebrew et al. studied students’ productive and unproductive learning behaviors and compared the behavior sequences between high and low performers.¹³ These authors found that high performers often conducted reading in a monitoring context, while low performers perform irrelevant and short reads. Another study used sequential pattern mining (a set of statistical techniques for discovering frequently occurring series of events or subsequent actions as patterns in a sequential dataset) to analyze college students’ chat in a simulation-based collaborative-inquiry-learning environment.¹⁴ Results identified significant differences between successful and less successful groups regarding their collaborative inquiry learning process: successful groups tended to ensure a shared understanding before moving on to the next stage, while less successful groups did not.

One advantage of sequential pattern mining is that it can identify patterns where two elements are steps away. The input is a group of sequences, while the output is a number of sequences associated with their occurrence frequency. For instance, given sequences: A-B-C-D, B-B-C, B-D-D-C, and B-C-E-D-F sequential pattern mining is able to identify a pattern B-C with an occurrence frequency of 4, a pattern B-D with an occurrence frequency of 3, and patterns B-C-D and B-D whose occurrence frequency are both 2. Depending on the experimental questions being asked and the context, it is, however, necessary for the researchers to determine apriori which frequencies are considered most common. An alternative approach would be to look at the top k ranked identified sequences.

As a means to demonstrate the potential value of sequential analysis on differential decision-making behaviors, we describe this process in the context of data obtained from XR-simulated scenarios. Critically, the sequential patterns resulted from this analysis cannot be generalized to similar real-world clinical situations without extensive additional analysis of the fidelity of both the simulation scenario and the extended reality (XR) environment. This approach enables exploration of patterns of learner success and decision-making trajectories within the simulated environment.

Exemplar

In an attempt to examine the potential utility of sequential analysis on clinical decision-making behaviors observed in simulated XR scenarios, we utilized data collected from a variety of healthcare providers and trainees for an Institutional Review Board approved study of XR-simulated pediatric airway emergencies.²⁴ Importantly, this exemplar describes an application of the sequential analysis in the context of learning scenario evaluation and not as a way to determine optimal decision-making patterns for the real-world clinical environment. The XR scenarios used for this project had predetermined end-points that were coded into the program.

Learning environment and task

All participants reviewed a series of interactive instructional videos describing pediatric airway anatomy and the recognition and management of critical airway events in children. Participants were then randomized to manage either a foreign body aspiration or anaphylaxis clinical scenario using the XR technology. The 3D visualization using HoloLens® technology (Microsoft Corp., Redmond, WA) allowed the user to immerse themselves in the virtual pediatric medical scenarios with the opportunity to make decisions. For a more in-depth explanation of the development of the extended reality (XR) technology, please refer to previously published work using the same scenarios and XR software.²⁴ Two rare but life threatening critical pediatric airway emergency clinical scenarios were selected for the project: foreign body aspiration (FB) and anaphylaxis (AP).

Multiple interventions and management options were available to the participants during the scenario as seen in Tables 1 and 2. The software logged each participants’ decision-making selection, and considered their actions correct or incorrect by the inherent coding rules of the software. Real-time visual and sound feedback through the XR program indicated whether an action was correct or incorrect. There were multiple paths the participant could take to reach the successful outcome, but only one possibly successful outcome was attainable.

Table 1.

Behavior categories in scenario FB.

Main behavior category	Action
Definitive treatment	Inserted Magill Forceps (Closed)
Diagnostic	Applied Oximeter
	Performed Cardiac Exam
	Performed Lung Exam
	Placed Laryngoscope
Inappropriate	User Neglected Patient
	Applied Lubricant
	Inserted Magill Forceps (Open)
Outcome	Removed Foreign Object
	Patient Expired
Rescue	Administered Albuterol
	Administered Epinephrine IM
	Administered Oxygen (Blow By)
	Administered Oxygen (Canula)
	Administered Oxygen (Non-Rebreather)
	Performed Heimlich Maneuver
	Performed Needle Cricothyrotomy
	Administered Epinephrine Nebulized
Secondary treatment	Administered Anesthesia
	Administered Dexamethasone IM
	Administered Dexamethasone IV
	Administered Dexamethasone PO
	Administered Ranitidine IV
	Administered Ranitidine PO
	Used Suction Tube

Table 2.

Behavior categories in scenario AP.

Main behavior category	Action
Diagnostic	Applied Oximeter
	Performed Lung Exam
	Placed Laryngoscope
Inappropriate	Performed Heimlich Maneuver
	User Neglected Patient
Outcome	Patient Expired
	Administered Epinephrine IM
Rescue	Administered Albuterol
	Administered Epinephrine IV
	Administered Epinephrine Nebulized
	Administered Oxygen (Blow By)
	Administered Oxygen (Canula)
	Administered Oxygen (Non-Rebreather)
	Performed Cricothyrotomy
	Performed Intubation
	Administered Epinephrine IM
Secondary Treatment	Administered Anesthesia
	Administered Dexamethasone IM
	Administered Dexamethasone IV
	Administered Dexamethasone PO
	Administered Diphenhydramine PO
	Administered Diphenhydramine IV
	Administered Ranitidine IV
	Administered Ranitidine PO
	Performed Needle IV

IV = Intra-venous; IM = Intra-muscular; PO = Oral

Measurements

Clinical management behaviors.

For analysis of the potential actions available in both scenarios, authors EP and LR categorized each action into one of six broad clinical management behavior categories: Diagnostic, Rescue, Secondary treatment, Definitive Treatment, Outcome, and Inappropriate (Tables 1 and 2). These behaviors were inherent to the coding of the XR program. To identify meaningful behavior patterns in managing the XR simulated scenarios, we used “outcome” to mark the end of a valid sequence. In other words, the subsequent actions that occurred after the “outcome” are excluded from the sequence analysis. Successful outcomes were defined by the coded actions within the programming. The second step was to conduct analysis and perform visualization. To identify the most common patterns among participants’ behaviors, we applied PrefixSpan, a widely used sequential pattern mining algorithm outperforming many of its counterparts.^25,26 In our case, the goal was to find common behavior sequences in managing pediatric airway XR simulated scenarios. To summarize, a behavior sequence is a number of consecutive main behavior categories extracted from an attempt. The length of a behavior sequence is equal or less than the total number of behavior categories in that attempt. Sequences can only be extracted from valid attempts. Hence, the total number of behavior sequences is equal or less than the total number of attempts. Sequential analysis was conducted using Python (Python Software Foundation, Wilmington, DE, USA). We collected 81 valid attempts from 41 participants in both FB and AP scenarios. Of 42 valid attempts in the FB scenario, 25 were successful and 17 were unsuccessful. Of 39 valid attempts in the AP scenario, 34 were successful and only 5 were unsuccessful.

In terms of the behavior sequences, we identified the common behavior patterns for each scenario. Figures 1 and 2 show the top 8 most common sequential patterns for each scenario. Since more than one sequence has the same frequency, there are 9 sequences in some groups. Each node represents one behavior category, while each line is a sequential pattern. The number above the line is the occurrence frequency of that pattern in its group. The higher the frequency, the wider the line. We observed some distinct behavior patterns between more successful and less successful participants in each scenario.

Figure 1.

Sequential patterns for scenario FB.

Figure 2.

Sequential patterns for scenario AP.

In the successful FB scenario attempts, the sequential pattern Rescue-Inappropriate-Outcome occurs in each attempt. Several other patterns like Diagnostic-Diagnostic-Outcome, Definitive Treatment-Outcome and Rescue-Definitive Treatment-Outcome have a rather high frequency as well. In the unsuccessful FB group, the most common pattern is Diagnostic-Outcome, followed by Rescue-Inappropriate-Outcome and Diagnostic-Inappropriate-Outcome. Meanwhile, Definitive Treatment, which shows in more than half of identified patterns in the successful participant group, does not appear in common patterns of the less successful participant group at all.

In the successful scenario AP group, the leading patterns are Rescue-Outcome and Rescue-Rescue-Outcome, whose frequencies are both 29. Other patterns have a much lower frequency. The most common pattern for the unsuccessful scenario AP group, is Rescue-Rescue-Outcome. Since the less successful participant group has a small sample size, the frequency of patterns does not vary significantly. While Secondary Treatment appears in several common patterns in the successful group, it does not show up in the unsuccessful participant group. The case for Inappropriate is opposite: it shows up in the unsuccessful participant group and is entirely absent from the successful group.

Discussion

In this article, we showed the potential value of sequential behavior analysis on differential decision-making behaviors in the context of data obtained from XR-simulated clinical scenarios. Advanced learning analytics afforded the possibility to trace clinical decision making and document the participants’ thought processes in real-time. The technology also allowed for steps to be recorded in the order they were performed, in an exact fashion, allowing for detailed analysis following data collection. Recording of the exact sequence of steps could provide material for robust feedback and debriefing with trainees, as well as a way for instructors to assess where a particular learner may need to improve their focus. Aggregating across trainees, these analyses can also be conducted for the full sample to reveal common challenges faced by most learners, or by identified subsets such as high versus low performers, level of training, specialty, or practicing clinicians in critical access hospitals vs. academic medical centers etc.

Typically, assessment in clinical simulated environments concentrates on technical skills and non-technical practices assessed by subjective human raters. The sequential analysis based on the behavioral log data presented here demonstrates a promising approach in addressing this important measurement problem. By examining the differential patterns of behaviors in managing pediatric airway XR simulated scenarios, we explore how sequential behavior analysis may be beneficial in XR-based systems. In essence, sequential analysis helps identify common clinical decision-making patterns while participants manage critical simulated clinical emergencies. Such a data-driven method focuses on behaviors displayed during the simulation allowing for feedback on these high-level skills. Understanding markers that may predispose individual learners or teams to errors or delays in therapeutic interventions may provide a significant insight into a more holistic assessment of individual and team performance and provide unique opportunities for feedback, practice, and/or remediation. With the addition of behavioral log data and sequential analysis, critical feedback can be provided by instructors during simulation debriefing sessions to allow for more targeted intervention and more rapid development of these complex skills. Again, the sequences of actions that are found to be most appropriate and deemed successful in the simulated learning scenario might not be the same as the actions that would improve health outcomes during actual critical care management. Any parallels drawn between simulated and real-world clinical processes would require a more in-depth examination of the real-world case and the simulation modality equivalence.

This project has limitations and implications for future work that need to be acknowledged and addressed. In terms of the simulated clinical scenario, the software program had pre-programmed endpoints that determined whether an outcome was successful or unsuccessful. Limitations to consider when using sequential analysis include recognition that identified sequential patterns overlook the distance between two actions, such that for a given pattern, the actions do not necessarily follow closely one another. It is also important to remember that one’s decision-making pattern in a controlled environment may not translate into real-world clinical skills.

While application of sequential behavioral analysis in medical education is not a new phenomenon, its use as a means to analyze clinical decision-making in an XR simulated setting is quite novel. It will be important to provide evidence for this type of analysis so that in the future it may be applied to conventional simulation or even to real-life situations and potentially increase patient safety. This method of teaching and assessing behaviors may be able to impact future curriculum development based on analysis of sequential behaviors and learner decision-making patterns. It is our hope that this article lays a foundation for using XR technologies and sequential analysis for healthcare workers and trainees as a method to help identify each learner’s unique needs in order to deliver focused, individualized feedback.