Feasibility of EEG to monitor cognitive performance during venous cannulation: EEG Distracted Intravenous Access (E-DIVA)

Abstract

Background

The feasibility study aims to evaluate the use of EEG in measuring workload during a simulated intravenous cannulation task. Cognitive workload is strongly linked to performance, but current methods to assess workload are unreliable. The paper presents the use of EEG to compare the cognitive workload between an expert and novice group completing a simple clinical task.

Methods

2 groups of volunteers (10 final year medical students and 10 emergency medicine consultants) were invited to take part in the study. Each participant was asked to perform 3 components of the simulation protocol: intravenous cannulation, a simple arithmetic test and finally these tasks combined. Error rate, speed of task completion and an EEG-based measure of cognitive workload were recorded for each element.

Results

EEG cognitive workload during the combined cannulation and arithmetic task is significantly greater in novice participants when compared with expert operators performing the same task combination. EEG workload mean measured for novice and experts was 0.62 and 0.54, respectively (p=0.001, 95% CI 0.09 to 0.30). There was no significant difference between novice and expert EEG workload when the tasks were performed individually.

Conclusions

EEG provides the opportunity to monitor and analyse the impact of cognitive load on clinical performance. Despite the significant challenges in set up and protocol design, there is a potential to develop educational interventions to optimise clinician's awareness of cognitive load. In addition, it may enable the use of metrics to monitor the impact of different interventions and select those that optimise clinical performance.

Introduction

Understanding clinical performance requires a multimodal approach due to the complex nature of the underlying cognitive processes. Expertise is the synergy of knowledge, skills and experience but performance is dependent on the mental processes that enable their application. Simple clinical tasks such as intravenous cannulation may challenge a novice but put little demand on the expert. Education interventions rely on metrics to monitor the impact on an individual's acquisition of skills.¹ Capturing and measuring performance remains the mainstay of clinical skills assessment but increasing the focus has moved to identification of underlying mechanisms and investigating how expertise develops.²

Cognitive load theory (CLT) has been proposed to explain of how an individual's ability to learn and perform is impaired when cognitive load exceeds capacity.³ The effort required to use ‘working memory’ during the performance of a task has been described as ‘cognitive load’ and is the proportion of cognitive capacity in use at any time.⁴ Cognitive load is dependent on the task performed and the individual.⁵ CLT states that there are three elements that contribute to cognitive load:

• Intrinsic—the demand of executing the current task;

• Extraneous—the demand to process information that are unrelated to the current task;

• Germane—the demand to resolve the task into genuine learning.

The implication is that when a learner is exposed to events that exceed working memory capacity then their ability to perform or learn is degraded. When designing educational interventions, it has been suggested that training efficiency may be optimised by controlling cognitive load.⁶

A number of studies have started to link cognitive workload within the medical setting to emotional state and work performance.^{7 8} Traditionally, pupillary dilation, heart rate, eye blinking, response times or cortisol levels have been used as physiological proxies for cognitive load. Workload has also been examined by the use of self-reported questionnaires completed following the task.⁹ Developing strategies to reduce unnecessary cognitive load and support development of enhanced clinical performance requires reliable metrics.

The advent of EEG has enabled development of novel approaches to objectively measure cognitive load.^{10 11} EEG workload levels have been shown to correlate with objective and subjective workload ratings in a variety of tasks.¹² This study reports the novel use of EEG within the medical simulation setting. EEG may hold the potential for a more objective, real-time measurement of cognitive load—offering the ability to quantify the effects of education interventions.¹³ Signatures in the EEG signal can be used to identify sensory and cognitive processes¹⁴ and EEG enables measurement of variations in task load depending on the nature of the task being undertaken. These data may be recorded and analysed in real time with signal processing and statistical techniques to create a number of performance parameters. Real-time EEG analysis has been applied successfully in a number of settings including the training of archers and pilots.^{15 16}

‘Safe and effective’ peripheral vein cannulation is one of the core technical skills acquired by medical students. This is often cited as a challenging task and considerable educational resources are invested to facilitate acquisition of this skill. Emergency medicine (EM) consultants may be considered experts at this skill and have acquired a high degree of automaticity: one of the features of ‘expertise’ is the ability to perform a task without significant cognitive demand.¹⁷

Aim

This pilot study aims to evaluate the feasibility of using EEG to measure workload during a simulated cannulation task.

Objectives

1. Can EEG be used to assess cognitive load during a simple clinical task?

2. Compare EEG findings against previously reported Distracted Intravenous Access (DIVA) study.

Methods

DIVA protocol

The study uses the DIVA protocol previously published by Smith et al¹⁸ (table 1). Paced auditory serial addition test (PASAT) is an established cognitive distraction test which has been shown to directly measure attention.^{19 20} Participants were asked to listen to a continuous series of single-digit numbers and state the sum of the last two numbers from a recording at 4 s intervals. The intravenous cannulation was performed on a manikin and assessed using the checklist focused on performing the correct stages in the correct order (figure 1). During this study, data were collected on errors on cannulation (compared with their checklist), speed of completion and number of arithmetic errors during PASAT.

Table 1.

Scoring of the distracted intravenous access (DIVA) test

Score component	How score component is derived	Range of possible scores (%)
Cannulation	Percentage of checklist items completed correctly (out of a possible 16)	0–100
Cognitive distraction	Percentage of possible answers given correctly in PASAT (total possible varies according to time taken), as a percentage score	0–100
Speed score	=((maximum: time allowed−actual time taken)/(maximum: time allowed−minimum time possible))×100	0–100
Overall score	Mean of cannulation, cognitive distraction and speed scores	0–100

DIVA, Distracted Intravenous Access; PASAT, paced auditory serial addition test.

Figure 1.

Candidate completing cannulation task with EEG monitor.

The original DIVA study demonstrated a difference in the performance of novices (fourth-year medical students), intermediates (1 year postgraduate) and experts (anaesthetic and EM consultants and novices) when performing the combined simulated peripheral vein cannulation and simultaneously a PASAT. We sought to repeat the DIVA study while measuring surface EEG of participants in an attempt to make an objective measurement of cognitive load during the task.

Participants

We recruited 10 final year medical students (novices) and 10 EM consultants (experts) to the study. Recruitment was on an opportunistic basis and candidates were tested by DJL, SAJ or AL following consent. Ethical approval was granted by the Edinburgh University Ethics Committee.

EEG

The study used the B-Alert X10 Bluetooth wireless system (Biopac). The system enabled wireless recorded of EEG waveforms to a study laptop running Acqknowledge V.4 software (Biopac) for data analysis. Workload during each task was recorded and averaged over the duration of each task. Each participant went through a calibration to ensure the impedance of nine-channel EEG, which would ensure real-time monitoring of engagement and cognitive workload. The EEG workload index ranges from 0 to 1.0, which rises with increasing demand.

EEG Distracted Intravenous Access

Before starting the study, each candidate was familiarised with both PASAT and the checklist for cannulation and given the opportunity to practice each, following set up and calibration of the EEG system. A trial information sheet detailing the protocol was also provided. All episodes were recorded in a clinical skills laboratory, with standardised equipment.

The protocol involved three stages as detailed in the original DIVA study and utilised the same checklist for the cannulation task.¹⁷ Participant's EEG workload was measured during each of the three stages:

1. Insertion of intravenous cannula;

2. During a PASAT;

3. Insertion of intravenous cannula while completing PASAT.

Completion of the tasks was timed and arithmetic errors were recorded when PASAT was administered. Data were collected on time to complete task, errors in cannulation, error in PASAT and EEG workload. EEG data were extracted from the AcqKnowledge software. Analysis was performed using Graph Pad Prism (GraphPad Software, California) with unpaired Student's t-test to detect statistical significance.

Results

DIVA protocol

Table 2 presents the results recorded using the standard DIVA protocol. PASAT scores were similar between novice and expert groups but experts completed the cannulation task significantly more rapidly and accurately (63–155 s (106.5 s) vs 128–300 s (198.9 s), p=0.0015).

Table 2.

Mean (95% CIs) for each component of the test for each group

Group	Cannulation score	Cognitive distraction score	Speed score
Novice	86.28 (82.77 to 89.78)	78.90 (71.75 to 86.05)	45.41 (20.50 to 70.32)
Expert	96.25 (93.12 to 99.38)	87.2 (79.00 to 95.40)	86.18 (79/57 to 92.79)

Figure 2 displays the scores for each of the elements of the DIVA protocol and the comparison between the expert and novice group. The two groups differed in all components of the test, and the difference was statistically significant (at the 5% level) in all components apart from the cognitive distraction. The results here are consistent with those found in the original DIVA study.¹⁸

Figure 2.

Means and 95% CIs for the DIVA scores. DIVA, Distracted Intravenous Access.

EEG results

Figure 3 presents mean cognitive workload as measured by EEG during each of the elements of the study protocol. The mean workload average value during each stage was analysed for all groups. There was no significant difference between the workload average of the novices versus experts during either the cannulation (p=0.41, 0.45 to 0.71 (0.57) vs 0.27 to 0.67 (0.52); 95% CI −0.07 to 0.16) or PASAT (p=0.19, 0.39 to 0.66 (0.54) vs 0.21 to 0.63 (0.44); 95% CI −0.05 to 0.24) tasks when performed separately. During the combined DIVA protocol, we found a significant result in workload average between the novices and experts (p=0.001, 0.49–0.73 (0.62) vs 0.32–0.72 (0.43); 95% CI=0.09 to 0.30).

Figure 3.

Means and 95% CIs for the EEG scores. PASAT, paced auditory serial addition test.

Discussion

With regard to our first objective, we found that collecting data on cognitive workload using EEG was achievable using the methods and equipment detailed above. The study demonstrates that collecting EEG data on clinician's cognitive load within a simulation environment is feasible, although there are challenges in terms of the time required to set up and calibrate systems ∼30 min for each participant. Each participant required creation of a baseline individual EEG profile, which enables comparison of data between individuals. In addition, placing the sensor strip and establishing reliable scalp contact and therefore low impedance required practice; there was a learning curve to successful application. The computer interface was relatively intuitive but creation and interpretation of data files and initial set up for the study required manufacturer support.

The addition of a secondary task (PASAT) increases the cognitive workload of the participants. This extra task causes an additional cognitive demand which impacts on task performance. For the experienced clinician, the increased workload did not degrade the completion of the primary task (cannulation) to the same extent as for the novice. It would be expected that placing a cannula for an expert is a relatively automatic task. This may reflect the extent to which the extraneous load is not increased to the same extent as novices. It is unsurprising that EEG workload for the PASAT task was not significantly different between groups, and it was only when PASAT was added to a clinical task—in which the expert group revealed a difference in cognitive demand. It would be expected that each of the study group's abilities to complete an arithmetic task would be similar, irrespective of age or clinical experience.

Our second objective of this study was to compare our findings to the previously reported findings of the DIVA study. We chose an existing protocol (DIVA) as our initial foray into measuring cognitive load to provide a comparison with an established validated objective score. While the addition of a subjective score was considered, no validated score existed. In any case, from the trial, it was apparent that the combined task was sufficiently challenging to all participants.

The finding of a significant difference between novice and expert during the combined task reflects the results from the DIVA study. The lack of significant difference when completing the cannulation task between the two groups was unexpected. There are a number of possible explanations for this. First, completion of the task within study conditions, that is, with the addition of an EEG monitor and observers increased the workload for both groups irrespective of task familiarity for the experts: these additional demands resulted in a task of equivalent difficulty for both groups. Second, medical students are more experienced in completing simulated intravenous cannulation tasks than experts, fulfilling criteria on a checklist and being observed than experts: these experiences reduced the competence differential. Third, the expert's usual degree of automaticity was reduced as they were completing the task following a defined and unfamiliar checklist. Finally, sample size may not have been sufficient to detect a significant difference. The addition of the arithmetic task may have provided sufficient cognitive load to expose the spare capacity that experts possessed to complete the protocol as their focus reverted to completion of the arithmetic element. Further testing during different protocols or for longer tasks may clarify this.

An alternate approach considered was simulated scenario for the two groups of increasing complexity and asking the participant to maintain a leadership role rather than combining a manual task with cognitive task. EEG workload may therefore be useful tool to delineate between workload during clinical tasks and provide insight into the development of competency, expertise and leadership beyond observation and checklists. This opens up the possibility of a number of avenues of research and application for which further work is required.

EEG workload analysis may provide valuable insight into team performance and the impact of leadership on cognitive performance. Combining EEG analysis with video review offers the opportunity for the development of enhanced metrics to monitor the impact of new clinical information, defined tasks or the deteriorating patient on the individual and the team.

The study has a number of limitations. Small numbers of volunteers were enrolled and we did not investigate the impact of wearing the EEG technology on task completion. Further testing with larger numbers would strengthen the conclusions of the study. The study does, however, use an established protocol and within a similar type of setting. A standardised mark sheet for arithmetic and cannulation task was used, and was scored by a single individual. Some participants appeared to selectively ignore the PASAT element to ensure rapid completion of the cannulation task rather than attempt to multitask, perhaps recognising that they had exceeded capacity. We elected not to utilise a subjective workload questionnaire to correlate to EEG but, with hindsight, this may be useful addition. Participants' awareness of headset and being part of study may have influenced the EEG workload activity recorded.

Future applications

There are a number of potential applications within both the clinical and education domain. The term ‘neuroergonomics’ has been coined to describe the development and design of work environments that are optimised according to enhanced understanding of cognition and human behaviour.¹² Providing a cognitive aid may enable clinicians to respond to overload and potential task completion degradation to enhance patient safety. EEG offers the opportunity to examine cognitive load within the clinical domain while other strategies such as fMRI are anchored to the laboratory. Further iterations of the technology and validation in other studies would help determine the utility of EEG application within the clinical environment. Real-time measurement of cognitive load may allow development of feedback mechanisms, for example, haptic warning of approaching overload. Simulation-based education has consistently been challenged to prove efficacy and, while limited translational outcomes have been established, metrics on the individual's cognitive improvement remains elusive.²¹ Measuring cognitive load may provide a tool to assess the effectiveness of different teaching modalities, for example, the impact of repeated exposure to challenging scenarios in increasing the threshold for cognitive overload.²²

Conclusion

EEG may provide a greater understanding of the cognitive load during task completion. Its application may not just be limited to completion of technical tasks but applied to clinical situations in which decision-making is challenged by extraneous distraction. Design and evaluation of educational interventions may be aided by identifying which strategy has the greatest impact on reduction of cognitive load or raise clinician's threshold for cognitive overload. Creation of metrics using EEG to assess efficacy of teaching interventions and provide clinicians with real-time feedback on cognitive load is feasible. We hope to extend the use of EEG into simulation and ultimately clinical settings. The use of EEG to measure cognitive load and its relationship to clinical performance is an interesting area that deserves further exploration. Optimising education interventions to individual cognitive capacity could lead to enhanced clinical performance and ultimately improved patient outcomes.