Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2018 Dec 11.
Published in final edited form as: IEEE Int Interdiscip Conf Cogn Methods Situat Aware Decis Support. 2018 Aug 2;2018:77–82. doi: 10.1109/COGSIMA.2018.8423995

Development of an Interactive Dashboard to Analyze Cognitive Workload of Surgical Teams During Complex Procedural Care

Roger D Dias 1, Heather M Conboy 2, Jennifer M Gabany 3, Lori A Clarke 4, Leon J Osterweil 5, George S Avrunin 6, David Arney 7, Julian M Goldman 8, Giuseppe Riccardi 9, Steven J Yule 10, Marco A Zenati 11
PMCID: PMC6289194  NIHMSID: NIHMS973713  PMID: 30547096

Abstract

In the surgical setting, team members constantly deal with a high-demand operative environment that requires simultaneously processing a large amount of information. In certain situations, high demands imposed by surgical tasks and other sources may exceed team member’s cognitive capacity, leading to cognitive overload which may place patient safety at risk. In the present study, we describe a novel approach to integrate an objective measure of team member’s cognitive load with procedural, behavioral and contextual data from real-life cardiac surgeries. We used heart rate variability analysis, capturing data simultaneously from multiple team members (surgeon, anesthesiologist and perfusionist) in a real-time and unobtrusive manner. Using audio-video recordings, behavioral coding and a hierarchical surgical process model, we integrated multiple data sources to create an interactive surgical dashboard, enabling the analysis of the cognitive load imposed by specific steps, substeps and/or tasks. The described approach enables us to detect cognitive load fluctuations over time, under specific conditions (e.g. emergencies, teaching) and in situations that are prone to errors. This in-depth understanding of the relationship between cognitive load, task demands and error occurrence is essential for the development of cognitive support systems to recognize and mitigate errors during complex surgical care in the operating room.

Keywords: cognitive load, cardiac surgery, heart rate variability, process model

I. Introduction

Recent estimates rank medical errors as the third leading cause of death in the U.S. [1–2]. Fifty to 65% of the adverse events and errors experienced by hospitalized patients are attributed to surgical care and more than half of these events are preventable. Understanding and managing conditions leading to errors is critical to reduce preventable patient harm in the operating room (OR) [3].

Complex procedural care (e.g. surgery, interventional radiology and interventional cardiology) is a high-consequence team-based sociotechnical system with critical requirements for communication and coordination. Contemporary sociotechnical systems research has moved away from the individual as the unit of cognitive analysis, and a new focus on the activity system (a group of human actors, their tools and environment) has been proposed; this framework has been referred to as “distributed cognition” [4–6]. In fact, great effort has been devoted in the past years to assess intraoperative performance of surgical teams and to pinpoint the many factors that may either improve, or impair the quality of surgical care [7]. In the surgical setting, team members constantly deal with a high-demand operative environment that requires simultaneously processing a large amount of information. In certain situations, high demands imposed by surgical tasks and other sources, such as distractions, interruptions and concurrent activities (e.g. teaching, answering pagers and telephones), may exceed team member’s cognitive capacity, leading to a potentially risky state known as cognitive overload. A growing body of evidence has found that this state impairs human performance during surgical tasks, especially those involving complex procedural care [8, 9].

Despite the wide variety of methods that have been used to assess cognitive load in various fields (e.g. medicine, education, aviation and sports), most of them involve subjective and self-report instruments administered post-hoc. Consequently, tools like NASA-TLX are not adequate to support measures of intraoperative cognitive load fluctuations in real-time [10]. In addition, even when real-time methods are used (e.g. electrocardiography, electroencephalography), most studies to date have investigated only the average cognitive load measure during the entire procedure, not providing a path to understanding the relationship between certain cognitive load levels and specific tasks (e.g. tasks lasting few minutes), unexpected events (e.g. emergencies, errors) or contexts (e.g. teaching, interruptions). In a recent systematic review of literature carried out by our group, we found that heart rate variability (HRV) analysis is an objective and unobtrusive method capable of capturing reliable metrics of cognitive load in surgery [10]. In addition, preliminary studies of surgeons performing real-life cardiac surgeries have shown that HRV is strongly correlated with surgeons’ SURG-TLX (Surgical Task Load Index) score which is a self-report questionnaire based on the widely used NASA-TLX (NASA Task Load Index) [11]. Both sympathetic and parasympathetic branches of the autonomic nervous system are involved in the regulation of heart rate and interbeat intervals. Recent research focusing on the neurovisceral integration theory have demonstrated that executive, emotional and cognitive control functions are regulated by brain systems also involved in the regulation of the autonomic system, and some cognitive processes, such as working memory utilization can be assessed by measuring the heart rate variability [12]. Among several parameters generated by HRV analysis, the low frequency/high frequency ratio (LF/HF ratio) appears to present the strongest correlation with cognitive load [13].

In the present study, we describe a novel approach to integrate a real-time and objective measure of cognitive load (heart rate variability) with procedural, behavioral and contextual data from real-life cardiac surgeries. This approach is innovative because it uses a multimodal data source to create an interactive dashboard, allowing a comprehensive analysis of multiple team members while simultaneously performing high complex surgical procedures.

II. Methods

We have investigated the cardiac surgery team performing two different complex procedures: coronary artery bypass grafting (CABG) and aortic valve replacement (AVR). Regulatory approvals were obtained by the local Institutional Review Board (IRB) of record (IRB#3047), including additional protections for employees as vulnerable subjects. Both patients and staff signed an IRB approved informed consent form and separate authorization for the release of information due to audio/video recording of the procedures. Patients were approached during the preoperative testing phase for informed consent to participate in a research study. Staff received protocol training prior to implementation during a regularly scheduled staff education in-service. Following the training, the majority of staff was consented at that time; other staff was consented in the operating room prior to participating in research activities.

A. Capturing Procedural, Contextual, Behavioral and Physiological Data in the OR

1) Surgical Process Modeling

Based on previous research developed by our group, [14] a precisely-defined surgical team process model was used, providing a detailed hierarchical view of how the cardiac surgical team members perform tasks, and how they coordinate with each other as well as with a suite of medical devices such as a cardiopulmonary bypass pump (CPB), a lung ventilator, and patient monitors. The hierarchical nature of the model supports providing a high-level view of team coordination as well as decompositions that support arbitrarily detailed specification of how process performers carry out their steps. These models were elicited, and iteratively improved, by interviewing surgical domain experts, by observing process performers, by consulting medical literature, and by codifying best practices. In general, the cardiac surgery process model uses a 4-level hierarchical structure involving four sub(teams): surgeon, anesthesiologist, perfusionist and nurse. The cardiac procedures were segmented in 3 stages, 13–14 steps, 160–180 sub-steps, and approximately 200 tasks, depending on the procedure type (CABG or AVR).

2) Patient Reports

In order to capture contextual information that can impact team members’ cognitive load, we gathered data from three different reports in the patient’s electronic health record (EHR): surgical, perfusion and anesthesia. Contextual variables were divided in three different components: variables related to the patient (age, gender, and pre-operative risk - infection, morbidity, and mortality); variables related to the surgical team (experience, presence of trainees); variables related to the procedure (step, type, execution time, degree of concurrency, depth of exception handling, bypass duration).

3) Audio and Video Recording

Two GoPro cameras (HERO4) were placed in the OR at the room corner and surgical light head, and configured to record an ultra-wide (entire team) and a narrow (surgical field) field of view at 30 frames per second and 960 pixels of video resolution. Three stereo digital voice recorders (Sony ICD-PX440) captured audio at 44.1 KHz/192 Kbps from three team members (surgeon, anesthesiologist and perfusionist) via a lapel microphone. Video and audio files were recorded in .MP4 and .MP3 formats respectively and synchronized during post-production.

4) Heart Rate Variability Analysis (HRV)

Our systematic approach to measure cognitive load involves the use of a heart rate sensor (Polar H7 chest strap) to capture beat-to-beat intervals from three team members simultaneously in real-time and unobtrusively. Each of the heart rate sensors transmits the interbeat intervals (in milliseconds) via Bluetooth connection to a smartphone (iPhone 6) application (Elite HRV, version 3.14.6). After the recording was completed, raw data from each team member was exported in .csv format. The .csv files were used in a HRV analysis software (Kubios HRV Standard, version 3.0.2) for calculation of time-domain and frequency-domain parameters by using time series analysis (Fast Fourier Transform). In the present study, we used an ultra-short-term analysis of HRV, [15] calculating the LF/HF ratio for each 1 minute of the procedure. In Kubios software, a very low artefact correction algorithm (threshold = 0.45 seconds) and a smoothness priors detrending method (lambda = 500) were used.

B. Synchronizing Multiple Data Sources

Audio and video were synchronized in a multimedia editing software (Movavi video suite 16) by aligning audio waveforms, and the actual time was displayed in the video to allow synchronization with the surgical reports. Using a spreadsheet (Excel 2016), the entire procedure was coded for each 1-second length, from the moment the patient arrives until the patient leaves the OR (Fig. 1). For instance, a procedure lasting 4 hours generates 14,400 rows. A physician with expertise in human factors and behavioral coding observed the videos and annotated all components of the process model into the spreadsheet. Beside the 4-level hierarchical process model, the following intraoperative aspects were also coded: events and context. Intraoperative events were defined as any event that indicates the start or end of a task (e.g. electrocautery is turned on, aorta was cross clamped), or any non-nominal events that are not captured in the process model (e.g. errors, adverse event, emergency, critical communication). Intraoperative context was annotated to capture relevant situations that occur during the procedure (e.g. surgeon is teaching fellow, fellow is performing task and surgeon observing, another person enters the OR to talk to the surgeon). In addition, HRV data (LF/HF ratio) from each team member (surgeon, anesthesiologist and perfusionist) were integrated into the spreadsheet. Fig. 2 illustrates our multimodal approach to capture, integrate, synchronize and display data for analysis.

Fig. 1.

Fig. 1

Open in a new tab

Procedural, contextual, behavioral and physiological data synchronization.

Fig. 2.

Fig. 2

Open in a new tab

Novel approach to embed multimodal data into the surgical process model.

III. Interactive Analytic Dashboard

The database created in the Excel spreadsheet (format .xlsx) was connect to a data analytics software (Tableau Desktop, version 10.4). A surgical analytics dashboard integrating 4-level hierarchical process model (stage, step, substep, task), events and contextual information with a 1-second timestamp was created (Fig. 3).

Fig. 3.

Fig. 3

Open in a new tab

Interactive dashboard displaying cognitive load measures of three team members: surgeon (top), anesthesiologist (middle), perfusionist (bottom), 4-level hierarchical process model, events and contextual information during an AVR procedure.

A major innovation of this dashboard is that it is dynamic and interactive (see http://www.goo.gl/jqGM2U), allowing the analysis of individual procedures or summary measures (mean and standard deviation) and confidence intervals across many procedures. Cognitive load metrics (LF/HF ratio) can also be compared between the surgeon, anesthesiologist and perfusionist during the entire surgery or in specific steps/tasks. In addition, context variables related to patient characteristics, surgical team composition and procedure are displayed in the dashboard, allowing an in-depth analysis of the many factors impacting on team members’ cognitive load.

In Fig. 4, we captured an example of interruption (perfusionist informs lab result to surgeon - ACT test) during a high demanding substep (aortic cannulation), leading the surgeon to switch from the primary task (purse-string preparation) to a secondary task (discussing the reliability of the lab test system), and consequently increasing the cognitive load (LF/HF ratio reached 130).

Fig. 4.

Fig. 4

Open in a new tab

Inter-ruption during a demanding task that increases surgeon’s cognitive load, likely due to task switching.

IV. Discussion

In the present study, we have described an innovative approach to integrate and analyze distinct types of concurrent information generated by multiple data sources during complex procedural surgical care. A detailed process model was developed to hierarchically decompose CABG and AVR procedures into a 4-level process structure. By using heart rate variability analysis, we were able to objectively assess the cognitive load of three team members, simultaneously, in near real-time. The behavioral coding generated by audio-video analysis from real-life procedures has allowed us to capture unique intraoperative events and contextual information, providing an in-depth understanding of the myriad of extrinsic factors contributing to cognitive load that likely add to the intrinsic demand imposed by procedural tasks. By also incorporating contextual variables defining patient characteristics, surgical team composition and procedural factors this model provides us with additional information to investigate the impact of factors such as severity of illness, level of provider’s experience and teaching on team performance.

A. Surgical Data Science

Surgical data science is an emerging discipline with the aim of improving safety, quality and effectiveness of surgical care by means of data acquisition, modeling and analysis. Core to this field is the possibility of overcoming some limitations imposed by traditional methodologies (e.g. post event analysis, registry data), through the capture of multisource intraoperative data in real-time and using advanced analytic methods, such as machine learning for predictive analysis [16]. In the last few years, an increasing number of studies have intended to provide an integrative framework that encompasses data capture, synchronization, analysis and application of surgical data for quality improvement. Interestingly, some initiatives in surgical data science have attempted to assess and provide remediation not only for technical aspects of surgical performance, but also for non-technical skills, such as situational awareness, teamwork and communication [17]. Accordantly, our methodological approach to integrate multisource data allows an in-depth analysis of cognitive load imposed by highly complex procedures. Understanding how cognitive load is distributed throughout a complex team, and how it fluctuates over time during the course of surgery, will allowed us to investigate in an individual and team levels the myriad of factors contributing to overload, and consequently adverse events and errors in the surgical setting.

B. Future Directions

The approach described in this paper can be used to develop cognitive aid systems to support teams during complex procedural care. As an example, intelligent interruption management systems could use real-time and predicted metrics of cognitive load to guide the surgical team regarding the appropriateness of an interruption based on the cognitive load imposed by specific tasks and context. This type of system would enhance team situational awareness, since it allows each team member to be aware of other’s cognitive state in real-time and to choose the most appropriate moment to interrupt procedural flow. As alarm management, clinical decision support, and closed-loop control algorithms become more widely used, these algorithms will need to interact with clinical teams. Integrating cognitive load metrics with other contextual information will allow algorithms to interact with humans in a less intrusive manner. Other real-time measures of cognitive load can also be used, such as galvanic skin conductance, pupil dilation, electroencephalography and functional near-infrared spectroscopy [10]. An advantage of these methods compared to HRV is the low latency. While HRV analysis generate cognitive load metrics for each 5-minute (short-term analysis) or 1-minute (ultra-short-term analysis) timestamp, galvanic skin conductance and pupil dilation, for example, can detect cognitive changes within few seconds [18, 19]. Future studies can attempt to integrate multiple physiological signals into the surgical process model.

V. Conclusion

The unique approach discussed in this paper not only allows us to track the cognitive load of multiple team members during each surgical stage, step, substep and task, but also enables us to detect cognitive load fluctuations either over time, under specific conditions and in situations that are prone to errors, distractions and interruptions. This in-depth understanding can support the development of cognitive-aid systems to prevent and mitigate errors during highly complex care.

Acknowledgments

The authors wish to acknowledge the contribution, dedication and commitment to excellence of the cardiac surgery team and operating room staff at VA Boston Healthcare System.

This material is based upon work supported by the National Heart, Lung, and Blood Institute (NHLBI) of the National Institutes of Health under award 1R01HL126896-01A1. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies.

Contributor Information

Roger D. Dias, STRATUS Simulation Center, BWH, Harvard Medical School Boston, MA, USA

Heather M. Conboy, College of Inform. and CS University of Massachusetts, Amherst, USA

Jennifer M. Gabany, Division of Cardiac Surgery, VAHCS, Harvard Medical School, Boston, MA, USA

Lori A. Clarke, College of Inform. and CS, University of Massachusetts, Amherst, USA

Leon J. Osterweil, College of Inform. and CS, University of Massachusetts, Amherst, USA

George S. Avrunin, College of Inform. and CS University of Massachusetts, Amherst, USA

David Arney, Department of Anesthesia, MGH, Harvard Medical School Boston, USA.

Julian M. Goldman, Department of Anesthesia, MGH, Harvard Medical School, Boston, USA

Giuseppe Riccardi, Depart. of Inform. Eng. and CS, University of Trento, Trento, Italy.

Steven J. Yule, STRATUS Simulation Center, BWH, Harvard Medical School Boston, MA, USA

Marco A. Zenati, Division of Cardiac Surgery, VAHCS, Harvard Medical School, Boston, MA, USA

References

  • 1.Stefl ME. To Err is Human: Building a Safer Health System in 1999. Front Health Serv Manage. 2001;18(1):1–2. [PubMed] [Google Scholar]
  • 2.Makary MA, Daniel M. Medical error-the third leading cause of death in the US. BMJ. 2016;353:i2139. doi: 10.1136/bmj.i2139. [DOI] [PubMed] [Google Scholar]
  • 3.Gawande AA, Zinner MJ, Studdert DM, Brennan TA. Analysis of errors reported by surgeons at three teaching hospitals. Surgery. 2003;133(6):614–621. doi: 10.1067/msy.2003.169. [DOI] [PubMed] [Google Scholar]
  • 4.Klampfer B, Flin R, Helmreich R, Häusler R, Sexton B, Fletcher G, Field P, Staender S, Lauche K, Dieckmann P, et al. Enhancing Performance in High Risk Environments: Recommendations for the use of Behavioural Markers. GIHRE. 2001 [Google Scholar]
  • 5.Hazlehurst B, McMullen CK, Gorman PN. Distributed cognition in the heart room: how situation awareness arises from coordinated communications during cardiac surgery. J Biomed Inform. 2007;40(5):539–551. doi: 10.1016/j.jbi.2007.02.001. [DOI] [PubMed] [Google Scholar]
  • 6.Dekker SW, Leveson NG. The systems approach to medicine: controversy and mis-conceptions. BMJ Qual Saf. 2015;24(1):7–9. doi: 10.1136/bmjqs-2014-003106. [DOI] [PubMed] [Google Scholar]
  • 7.Flin R, Youngson G, Yule S. Enhancing Surgical Performance. Boca Raton, FL, US: CRC Press; 2016. [Google Scholar]
  • 8.Yurko YY, Scerbo MW, Prabhu AS, Acker CE, Stefanidis D. Higher mental workload is associated with poorer laparoscopic performance as measured by the NASA-TLX tool. Simul Healthc. 2010;5(5):267–271. doi: 10.1097/SIH.0b013e3181e3f329. [DOI] [PubMed] [Google Scholar]
  • 9.Ruiz-Rabelo JF, Navarro-Rodriguez E, Di-Stasi LL, Diaz-Jimenez N, Cabrera-Bermon J, Diaz-Iglesias C, Gomez-Alvarez M, Briceño-Delgado J. Validation of the NASA-TLX Score in Ongoing Assessment of Mental Workload During a Laparoscopic Learning Curve in Bariatric Surgery. Obes Surg. 2015;25(12):2451–2456. doi: 10.1007/s11695-015-1922-1. [DOI] [PubMed] [Google Scholar]
  • 10.Dias RD, Ngo-Howard MC, Boskovski MT, Zenati MA, Yule SJ. Systematic review of measurement tools to assess surgeons’ intraoperative cognitive workload. Br J Surg. 2018 doi: 10.1002/bjs.10795. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Dias R, Zenati M, Conboy H, Gabany J, Arney D, Goldman J, Osterweil L, Avrunin G, Clarke L, Yule S. Embedding Real-Time Measure of Surgeons’ Cognitive Load into Cardiac Surgery Process Modeling. Academic Surgical Congress: 2018; Jacksonville, FL, USA. 2018. [Google Scholar]
  • 12.Thayer JF, Hansen AL, Saus-Rose E, Johnsen BH. Heart rate variability, prefrontal neural function, and cognitive performance: the neurovisceral integration perspective on self-regulation, adaptation, and health. Ann Behav Med. 2009;37(2):141–153. doi: 10.1007/s12160-009-9101-z. [DOI] [PubMed] [Google Scholar]
  • 13.Böhm B, Rötting N, Schwenk W, Grebe S, Mansmann U. A prospective randomized trial on heart rate variability of the surgical team during laparoscopic and conven-tional sigmoid resection. Arch Surg. 2001;136(3):305–310. doi: 10.1001/archsurg.136.3.305. [DOI] [PubMed] [Google Scholar]
  • 14.Avrunin GS, Clarke LA, Osterweil LJ, Christov SC, Chen B, Henneman EA, Hen-neman PL, Cassells L, Mertens W. Experience modeling and analyzing medical processes: UMass/baystate medical safety project overview. Proceedings of the 1st ACM In-ternational Health Informatics Symposium; Arlington, Virginia, USA. ACM; 2010. pp. 316–325. [Google Scholar]
  • 15.Baek HJ, Cho CH, Cho J, Woo JM. Reliability of ultra-short-term analysis as a surrogate of standard 5-min analysis of heart rate variability. Telemed J E Health. 2015;21(5):404–414. doi: 10.1089/tmj.2014.0104. [DOI] [PubMed] [Google Scholar]
  • 16.Maier-Hein L, Vedula SS, Speidel S, Navab N, Kikinis R, Park A, Eisenmann M, Feussner H, Forestier G, Giannarou S, et al. Surgical data science for next-generation interventions. Nature Biomedical Engineering. 2017;1(9):691–696. doi: 10.1038/s41551-017-0132-7. [DOI] [PubMed] [Google Scholar]
  • 17.Goldenberg MG, Jung J, Grantcharov TP. Using Data to Enhance Performance and Improve Quality and Safety in Surgery. JAMA Surg. 2017 doi: 10.1001/jamasurg.2017.2888. [DOI] [PubMed] [Google Scholar]
  • 18.Ghosh A, Danieli M. Riccardi G: Annotation and prediction of stress and workload from physiological and inertial signals. Proc. EMBC, IEEE Conf. Engineering in Biology and Medicine Society; Milan. 2015. [DOI] [PubMed] [Google Scholar]
  • 19.Bailey BP, Iqbal ST. Understanding changes in mental workload during execution of goal-directed tasks and its application for interruption management. ACM Trans Comput Hum Interact. 2008;14:4. Article 21. [Google Scholar]

RESOURCES