Systematic review of measurement tools to assess surgeons' intraoperative cognitive workload

Abstract

Background

Surgeons in the operating theatre deal constantly with high-demand tasks that require simultaneous processing of a large amount of information. In certain situations, high cognitive load occurs, which may impact negatively on a surgeon's performance. This systematic review aims to provide a comprehensive understanding of the different methods used to assess surgeons' cognitive load, and a critique of the reliability and validity of current assessment metrics.

Methods

A search strategy encompassing MEDLINE, Embase, Web of Science, PsycINFO, ACM Digital Library, IEEE Xplore, PROSPERO and the Cochrane database was developed to identify peer-reviewed articles published from inception to November 2016. Quality was assessed by using the Medical Education Research Study Quality Instrument (MERSQI). A summary table was created to describe study design, setting, specialty, participants, cognitive load measures and MERSQI score.

Results

Of 391 articles retrieved, 84 met the inclusion criteria, totalling 2053 unique participants. Most studies were carried out in a simulated setting (59 studies, 70 per cent). Sixty studies (71 per cent) used self-reporting methods, of which the NASA Task Load Index (NASA-TLX) was the most commonly applied tool (44 studies, 52 per cent). Heart rate variability analysis was the most used real-time method (11 studies, 13 per cent).

Conclusion

Self-report instruments are valuable when the aim is to assess the overall cognitive load in different surgical procedures and assess learning curves within competence-based surgical education. When the aim is to assess cognitive load related to specific operative stages, real-time tools should be used, as they allow capture of cognitive load fluctuation. A combination of both subjective and objective methods might provide optimal measurement of surgeons' cognition.

Complex field

Introduction

In the past two decades, great effort has been devoted to assess surgeons' intraoperative performance and pinpoint the many factors that may either improve, or impair surgical care^1–4. In certain situations, high demands imposed by surgical tasks and other factors, such as teaching and flow disruption, may exceed surgeons' cognitive capacity, leading to a potentially risky cognitive overload. Many studies have demonstrated a direct relationship between surgical performance metrics and patient outcomes^5–7. Although there are several methods for assessing cognitive performance, the majority of existing tools are administered post hoc and do not allow recognition or correction of a surgeon's performance in real-time⁸. One barrier to implementing these tools to support surgical performance is a lack of understanding of the evidence supporting their use.

Regardless of how competent and expert surgeons may be, they are still subject to the common cognitive limitations, frailties and fallibilities that characterize the human brain. The sensory memory system can receive and process a vast amount of visual and auditory information. Modulated by attention and situational awareness, incoming information is allocated into working memory, which organizes it to be stored efficiently in long-term memory, and retrieved when required. Although the human brain theoretically has unlimited capacity, working memory is able to process only a limited amount of information simultaneously^9–11. Cognitive load is a contemporary scientific term that encapsulates a wide variety of nomenclatures used to describe the phenomenon of working memory use: cognitive workload, mental strain and mental effort. Cognitive load theory assumes that as surgeons engage in intraoperative tasks a certain level of load commensurate with required attention is imposed on working memory. A hallmark of surgical expertise is the ability to balance operative task demands against the available cognitive resources¹². However, surgeons occasionally experience a state of cognitive overload; previous research has shown that this impairs performance during specific tasks, especially those involving complex and/or non-routine operative situations such as emergencies and unexpected events^13–15.

Measurement of cognitive load is also important for surgical education. More experienced surgeons and senior residents can process larger amounts of information concurrently than novices. Even when residents reach the plateau of the learning curve, operating similarly to established surgeons in terms of motion economy, task length, error rate and non-technical skills, they require a higher cognitive load to execute the same tasks^13,16–17. Continued practice may enhance automaticity, and by using fewer cognitive resources, surgeons can dedicate sufficient working memory to deal with high-demand situations that may occur during surgery. Measuring surgeons' cognitive load in addition to the current performance metrics may be a useful strategy towards supporting and enhancing surgeons' cognitive capability.

Despite the large amount of research measuring cognitive load in healthcare and other high-risk industries^10,18, no systematic review has been performed to explore the existing cognitive load measurement methods in surgery, and evaluate the quality of evidence. Before systems can be developed to support cognitive performance in the operating theatre, an understanding of the range of available cognitive load assessment tools and evidence for their reliability and validity in surgery are needed.

Methods

The PRISMA guidelines^19,20 were used to design the present study and report the review findings.

Search strategy and data source

A literature search was conducted in December 2016, using MEDLINE (PubMed), Embase (embase.com), Web of Science (Core Collection), PsycINFO (EBSCO), ACM Digital Library (Guide to Computing Literature), IEEE Xplore (Digital Library), International Prospective Register of Systematic Reviews (PROSPERO) and Cochrane Database of Systematic Reviews (EBSCO) databases. All studies published from inception to November 2016 were considered and no restriction was imposed regarding language or study design. The Medical Subject Headings (MeSH) terms and text words from the MEDLINE search strategy (Table S1, supporting information) were adapted to other databases according to the specific syntax required. In addition, a hand search of references cited in the studies and reviews was conducted to ensure literature saturation.

Selection process and data extraction

Only original articles published in peer-reviewed journals were considered. Studies were included that used at least one cognitive load measurement involving senior surgeons, surgical trainees or medical students performing surgical procedures. Studies that did not assess cognitive load related to the intraoperative phase and studies that measured physiological data without intent to infer relationship with cognitive load were excluded. Two authors screened title and abstract for all search results independently, and identified relevant articles based on the eligibility criteria. For these articles, the full text was read by both investigators independently, who then decided whether the study met the inclusion criteria for the systematic review. The reason for excluding articles after full-text reading was registered. Using standard forms created in the REDCap web-based platform²¹, two authors extracted specific data from each study included in the systematic review independently. In case of disagreement, a third author was consulted.

Assessment of data quality and synthesis

A qualitative narrative synthesis was performed, structured around different methods used to assess cognitive load in surgeons. The methodological quality of all included studies was assessed independently by two authors using the standard Medical Education Research Study Quality Instrument (MERSQI)²². This is a ten-item instrument that reflects six domains of research quality: study design, sampling, data type, validity of assessments, data analysis and outcomes. Each domain scores out of 3 and the maximum total MERSQI score is 18. The mean MERSQI score of both independent assessors for each article that met inclusion criteria was reported.

Results

A total of 551 articles were captured in the broad search strategy, involving eight different databases, and five additional articles were identified through hand search. From these, 391 met eligibility criteria during screening and 84 met inclusion criteria after full-text review (Fig. 1). Pooling of data in a meta-analysis was not carried out owing to the heterogeneity regarding assessment methods and outcome measures.

Fig. 1.

PRISMA flow diagram for the review

Study design and setting

Fifty studies (60 per cent) were prospective cohorts, 18 (21 per cent) cross-sectional and 16 (19 per cent) RCTs. Most studies were carried out in simulated settings (59 studies, 70 per cent), 24 (29 per cent) occurred in clinical settings and only one study (1 per cent)²³ assessed surgeons' cognitive load in both settings.

Cognitive load assessment tools

The cognitive load measurement methods used in the reviewed studies are shown in Table 1. Most studies (58, 69 per cent) used only one tool to measure surgeons' cognitive load, and the remaining 26 (31 per cent) applied two or more methods.

Table 1.

Different methods used to assess surgeons' cognitive load

	No. of studies
Self-reported (post hoc)
NASA Task Load Index (NASA-TLX)	44
Surgery Task Load Index (SURG-TLX)	7
Subjective Mental Effort Questionnaire (SMEQ)	2
Multiple Resource Questionnaire (MRQ)	3
Pass scale	2
Borg scale	1
Subjective Workload Assessment Technique (SWAT)	1
Human Factors Evaluation Questionnaire for Computer Assisted Surgery Systems (HFEQ-CASS)	1
Other non-validated questionnaires	3
Real-time
Physiological parameters
Heart rate variability	11
Eye-tracking (blink rate)	5
Eye-tracking (gaze/fixation)	5
Eye-tracking (pupil dilation)	2
Electroencephalography	4
Functional near-infrared spectroscopy	3
Skin conductance response	3
Electromyography (masseter tone)	2
Heat flux (facial temperature)	1
Secondary task analysis
Reaction time	8
Visual detection rate	4
Task precision	1
Written task	1

Self-report tools

Sixty studies (71 per cent) used subjective (self-report) measurements of surgeons' cognitive load. The most commonly used self-report instrument was the NASA Task Load Index (NASA-TLX), used in 44 (52 per cent) of studies. This is a multidimensional assessment tool that has been used in a wide variety of domains, such as healthcare, aviation and other high-risk industries. Self-perceived cognitive load is rated from 0 to 100 points (with 5-point steps) according to six different subscales: mental demand, physical demand, temporal demand, performance, effort and frustration. The overall workload is calculated by weighting, adding or averaging each domain rating. The NASA-TLX instrument can be administered verbally, using a paper and pencil version, or by a computer-based application. Despite more than 550 studies using NASA-TLX reported in the past 20 years, few have proposed the workload redline when using this tool – a point on the scale that indicates when the workload is considered so high that it may affect human performance²⁴. A modified version of NASA-TLX was developed and validated specifically to capture the surgical context, the Surgery Task Load Index (SURG-TLX)^25–31. In two of the reviewed studies^13,32, the NASA-TLX questionnaire was used to capture surgeons' cognitive load over repeated training sessions, and correlated with technical performance and errors. In several other studies, self-report tools were used to compare the cognitive load imposed by different surgical procedures (open, minimally invasive and robotic surgery)^33–35, as well as different training strategies (for example simulation, cadaveric models, lectures, video-based and virtual reality)^36–38.

Real-time tools

Thirty-eight studies (45 per cent) used objective and real-time measurements of surgeons' cognitive load. The most commonly used real-time measure was heart rate variability (HRV) analysis, used in 11 studies (13 per cent). These tools are often applied in basic and clinical research studies and can be used in an unobtrusive manner using an inexpensive, wearable device, such as a Bluetooth® (Kirkland, Washington, USA) chest strap sensor or smart watch. Recent models encompassing neurovisceral integration and polyvagal theory have proposed a unifying framework, suggesting that a common reciprocal inhibitory cortical–subcortical neural circuit serves as the structural link between emotional regulation, cognitive regulation and physiological processes, and that this circuit can be indexed with HRV^39,40. In fact, neural network studies in humans have reported increased activity in the prefrontal cortex, linking vagally mediated HRV to a set of neural structures implicated in working memory tasks^41,42. HRV metrics are based on the analysis of interbeat intervals (R-R intervals) and attempts to quantify the sinoatrial rhythm variability. These variability measures are divided in two broad categories: time domain and frequency domain parameters, and, in order to extract these parameters from R-R intervals, complex time series statistical methods (such as spectral analysis) are used. Two HRV parameters, low-frequency (LF) band and high-frequency (HF) band, are mostly used to reflect the balance between the sympathetic and parasympathetic autonomic nervous system. In situations imposing a high cognitive demand, there is a sympathetic predominance, increasing the LF/HF ratio. LF/HF ratio has been used as an objective and real-time measure of cognitive load^43–45.

Participants

The number of participants included in the reviewed studies varied from a minimum of one to a maximum of 192, totalling 2053 unique participants. Twenty-three studies (27 per cent) assessed the cognitive load of medical students, 18 (21 per cent) assessed senior surgeons and 15 (18 per cent) assessed surgical trainees. Twenty-eight studies (33 per cent) included a combination of these grades. Only five studies (6 per cent) assessed the cognitive load of multiple operating theatre team members besides the surgeon. Team members included registered nurses (3 studies)^30,46–47, anaesthetists (3 studies)^26,30,47, registered nurse anaesthetists (2 studies)^30,46, theatre nurses (2 studies)^26,46, scrub nurses (2 studies)^30,48, physician assistants (1 study)⁴⁸, surgical technicians (2 studies)^30,46, circulating nurses (1 study)³⁰ and perfusionists (1 study)⁴⁶.

Surgical specialties

In total, 14 surgical specialties/subspecialties were included (Fig. 2). Seventy-six studies included only one specialty, two studies included two specialties, and six studies included three or more specialties (Fig. 3).

Fig. 2.

Distribution of the studies according to surgical specialties

Fig. 3.

Distribution of the studies according to surgical procedures. MIS, minimally invasive surgery

Study quality

The MERSQI score varied from a minimum of 6·5 to a maximum of 16·4 points, with a mean(s.d.) score of 11·1(2·2) points. In general, studies presenting a higher MERSQI score were those that used an RCT design. The main characteristics of all included studies, as well as the MERSQI score for each are shown in Table S2 (supporting information)^{15–17,23,25–38,44–109}.

Relationship with surgical outcomes

Four studies^45,89,94,98 measured actual patient outcomes, but only one⁸⁹ of these investigated the relationship between surgeons' cognitive load and patient outcomes. In that study⁸⁹, involving laparoscopic bariatric surgery, the authors reported a significant correlation between NASA-TLX score, duration of the operation and blood loss in the first 48 h after surgery, but no correlation between time to drain removal and duration of hospital stay.

Discussion

This systematic review has attempted to capture all relevant research to date that involved assessment of surgeons' cognitive load. The review shows that several methods have been developed in different subspecialties, surgical procedures, techniques, task complexities and training curricula. Tools could be categorized according to self-assessment or real-time, depending on the nature of implementation. Additionally, a subset of studies investigated the relationship between cognitive load and other factors intrinsically related to surgical care, as well as the impact of cognitive load on surgical performance.

Self-report tools commonly involve questionnaires administered after task completion to gather participants' recall of their cognitive effort during surgery. These subjective measures are easy to administer and can be used to track change over time, and evaluate interventions. Despite the wide use of psychometrically robust tools, there are limitations regarding concurrent validity of self-report tools¹¹⁰. In addition, these metrics reflect attentional and perceptual differences among individuals, memory¹¹¹, emotions and cognitive bias¹¹², so may not be the best for capturing fluctuations in surgeons' intraoperative cognitive load. For example, it is not known whether NASA-TLX scores reflect the actual or inferred cognitive load in each phase of a particular operation, and it is not clear how long a cognitive overload state must be present in order to cause impact¹⁷. Operations vary in both intensity and duration of cognitive load⁴³. Interrupting an operation to administer a mental load questionnaire may interfere with patient care and the primary surgical task, so is not feasible. Self-report assessment tools must be administered after the procedure, which limits their sensitivity.

Real-time assessment methods applied to surgery included HRV analysis, eye-tracking, electroencephalography (EEG) and skin conductance, together classified as physiological metrics. HRV analysis was the physiological metric used most in the reviewed literature. This is in contrast to studies of absolute heart rate parameters (minimum, maximum and average heart rate), which are insensitive to fluctuations in cognitive load, and are influenced by other factors such as physical effort and psychological stress. Despite the conceptual overlap between stress and cognitive load, research from medicine and other fields provides evidence for HRV analysis as a metric that can assess cognitive workload. HRV analysis compares favourably with the standard questionnaire used to assess mental workload (NASA-TLX), and has been shown to be accurate in detecting isolated increasing mental demand in laboratory studies^{41,43,85,113–114}.

Skin conductance response reflects sympathetic activity and was used in three studies^14,53,93, revealing correlation with perceived mental stress, blink rate, electro-oculogram, EEG frontalis activity and intraoperative performance. Limitations of this method are that psychological stress also activates skin conductance response, so can be a confounder⁸⁴. Eye-tracking devices have been used to assess surgeons' cognitive states by measuring eye movement and pupil dilation, providing information regarding gaze patterns, fixations, blink frequency and sympathetic activity²³. This has been associated with the NASA-TLX score, surgeon experience, surgical modality and task complexity^{53,62,72,93,107,109}. EEG has also been correlated with NASA-TLX scores^64–65,115, as well as expertise, task complexity and poor surgical performance. Similarly, functional near-infrared spectroscopy (fNIRS) has been implemented to capture activation patterns in specific brain areas during surgical tasks in three studies^52,79–80, with resulting correlations to surgical expertise and technical performance. A limitation related to eye-tracking, EEG and fNIRS tools is that a wide variation of analytical methods is applied, making it difficult to replicate or integrate findings across studies. In addition, these are obtrusive and may interfere with a surgeon's primary tasks in the operating room^97,100. Finally, a series of secondary task studies assessed surgeons' cognitive load associated with training modality⁵⁰, surgical technique⁹¹, task complexity⁴⁹, expertise and dexterity metrics¹⁵. However, the secondary task can also be a distraction from the primary task, making it unreasonable to assess surgeons' cognitive load in the operating theatre^66,91. This technique should be reserved for research in simulated settings.

Several limitations should be considered when interpreting the results of this systematic review. First, there is a conceptual overlap in the literature regarding the terms stress, mental strain, cognitive workload and mental demand, as reflected in the measurement metrics that have been developed in the past decade. The same measurement tools have been used to assess different cognitive states, and the present review focused on cognitive workload metrics. Second, alongside the low methodological quality of included studies, there was wide variability of application and analysis of real-time tools, which compromises the generalizability and reliability of these cognitive load metrics. Third, the methodological quality assessed by MERSQI score was low for most studies, indicating that several aspects of research in this field can be improved. Although many studies used extensively validated tools (face, content and construct validity), most of them were done in simulated settings. Few studies assessed predictive validity using patient outcomes, or RCTs to isolate the cognitive demand derived by surgical tasks alone. Future studies should include larger sample sizes, standard measurement methods and patient-centred outcomes.

In the current era of increased technology and constrained work hours for surgical residents¹¹⁶, cognitive load assessment may be integrated into the competence-based medical education framework, as it is correlated to surgical expertise and can capture the impact of new technologies and training modalities on cognitive load. Future innovations can be predicted by understanding the current state of the science. For example, objective measures of surgeons' cognitive load could be integrated into the current concept of the theatre Black Box¹¹⁷, to enhance performance and improve safety and surgical care. Despite the wide variety of available methods to assess cognitive load, there is no reference value or normal range that can be used as a threshold to determine when a state of cognitive overload has been reached. Most studies compared cognitive load in two or more situations (such as different surgical procedures, techniques or experience level) and, therefore, are able to demonstrate only that surgeons display a higher or lower cognitive load when comparing two or more operative conditions. To advance, this relatively new research field must build on the already validated metrics to test criterion validity. Specifically, as there are now several validated methods using objective and real-time metrics, future research should investigate the cognitive load range that is associated with surgeons' performance impairment. This can be characterized as a danger zone¹¹⁸ or redline for overload, and, if associated with risk of patient harm, may be used as a reference for the development of mitigation strategies towards supporting the cognitive performance of surgeons.

Supplementary Material

bjs10795-sup-0001-Tables

Table S1 Search strategy used in MEDLINE database

Table S2 Included articles

Snapshot quiz 18/4