Exploring cognitive flexibility with a noninvasive BCI using simultaneous steady-state visual evoked potentials and sensorimotor rhythms

Abstract

EEG-based brain-computer interface (BCI) technology creates non-biological pathways for conveying a user’s mental intent solely through noninvasively measured neural signals. While optimizing the performance of a single task has long been the focus of BCI research, in order to translate this technology into everyday life, realistic situations, in which multiple tasks are performed simultaneously, must be investigated. In this work, we explore the concept of cognitive flexibility, or multitasking, within the BCI framework using a two dimensional cursor control task suing sensorimotor rhythms (SMR) and a four target visual attention task using steady-state visual evoked potentials (SSVEPs), both individually and simultaneously. We found no significant difference between the accuracy of the tasks when executing them alone (SMR – 57.9 ± 15.4%, SSVEP – 59.0 ± 14.2%) and simultaneously (SMR – 54.9 ± 17.2%, SSVEP – 57.5 ± 15.4%). These modest decreases in performance were supported by similar, non-significant changes in the electrophysiology of the SSVEP and SMR signals. In this sense, we report that multiple BCI tasks can be performed simultaneously without a significant deterioration in performance; this finding will help drive these systems toward realistic daily use in which a user’s cognition will need to be involved in multiple tasks at once.

I. Introduction

On a daily basis, the substantial amount of stimuli naturally occurring in the world provides a large influx of sensory information to the brain that we are adept at processing simultaneously. The ability to interpret and respond to these stimuli at the same time lends itself to the concept of cognitive flexibility, or multitasking, often referred to as the ability to spontaneously adapt mental processes and respond to dynamic and changing situations [1]. In most day-to-day cases where multiple tasks are performed at the same time, as in driving a car and listening to music, cognitive flexibility occurs naturally without requiring much overt thought; however, for people with physical dysfunctions, this ability can become severely diminished. Neuromuscular impairments that include spinal cord injury and amyotrophic lateral sclerosis can render useless a large portion of the natural motor circuitry that allows human beings to naturally perform multiple tasks at once. With cognitive function still largely or completely intact in these patients, direct brain-based communication through brain-computer interface (BCI) technology is a viable option for replacing the ability to perform these tasks. One of the major pitfalls with BCI, however, is that it is generally thought that a user must be highly focused on the single-task paradigm being performed. Under this assumption, the rapidly adapting cognitive processes that allow for multitasking are neither tested nor utilized to make these systems more effective for realistic use.

EEG-based BCI provides a unique opportunity to study cognitive flexibility from a purely brain-centered perspective since this technology is particularly useful at conveying information solely through neural signals [2], [3] without using alternative, confounding modes of communication such as physical action. One popular BCI modality is that which uses sensorimotor rhythms (SMRs), relying on users to perform motor imagination tasks to produce unique spectral patterns within the scalp EEG recorded over the motor cortical regions [4]. When detected, these patterns can be translated into actions of an end effector to convey the user’s motor related mental intent. Steady-state visual evoked potentials (SSVEPs) are an alternative signal that is commonly used in BCI, founded on visually attending to a single stimulus (usually one of many) flickering at a specific frequency [5]. The frequency being attended to manifests within the EEG, with an increased amplitude being revealed in the power spectrum of electrodes covering visual brain regions. SMR- and SSVEP-based BCI have been widely studied within the field of noninvasive neural communication, inspiring innovations such as novel stimulus paradigms [6], [7], advanced signal processing algorithms [8], [9] recreational applications [10]–[12], and more. These BCIs have further demonstrated more than capable clinical use through the control of wheelchairs [13]–[15], stroke rehabilitation [16], [17], and various robotic prosthetics/orthoses [18]–[20].

Despite these promising advances, the translation of such physical devices into daily use will require users to be able to perform at least one task involving cognitive EEG-based communication while simultaneously performing at least a second cognitive task (or a separate physical task). One example of this would be using SMR BCI to control the 2D navigation of a wheelchair while also having a coherent conversation with a friend or colleague, or mentally preparing for the next task once the destination has been reached. In the current work, we sought to test this concept solely through a noninvasive BCI to demonstrate the feasibility of cognitive flexibility as well as to characterize this ability in both the spatial and temporal domains. While the aforementioned example would be a highly practical situation for eventual BCI users, it is difficult to quantify coherent speech, and rather in the current study we sought to focus on two demanding BCI tasks that can be rigorously examined; a motor imagery (MI) cursor control task utilizing SMRs and a visual attention task with SSVEPs.

Several previous studies have integrated various noninvasive signals into a single “hybrid” BCI, often utilizing SMRs and either SSVEPs [15], [21]–[23] or an evoked potential termed the P300 signal [24], [25] to simultaneously control various virtual and real objects in 2D. The primary objective of these previous studies has been to expand the dimensionality of noninvasive BCI by using multiple signal types to control different aspects of the same task. In this sense, MI may be used for horizontal control of a device while SSVEPs [22] or the P300 [24] may be used for vertical control, or MI for 2D directional navigation of a device and SSVEPs [15] or the P300 [25] to control the speed. Regardless, these two input signals ultimately serve complementary purposes of accomplishing a single, unified goal, and thus do not necessarily require dichotomous cognitive attention. Similar hybrid BCI studies have utilized multiple neural/biological signals for multiple independent tasks, most commonly with one signal type for controlling the BCI and another acting as a switch, indicating when the BCI should and should not make decisions [26], [27]. In these cases, the two tasks are in fact independent, but not performed simultaneously, allowing users to dynamically switch his/her cognitive load to focus on one task at a time. Thus, these works do not examine how the performance of these tasks are affected when two independent tasks are performed at the same time.

Nevertheless, there have been a few BCI studies that incorporate a secondary non-BCI task together with a primary BCI task, involving either SMR or SSVEP signals, that do not aspire to a unified goal and still occur at the same time [28], [29]. These studies are structured in ways to determine how much attention and awareness is required to specifically complete the BCI task by incorporating various distractors either in the form of real-world noise situations [28] or tasks that affect dissimilar sensory channels (auditory vs. visual) from the BCI task [29]. While it is generally accepted that the brain has a finite cognitive capacity [30], these previous works merely indicate that there is still room left in the “cognitive tank” after accounting for the attention processes devoted to a single BCI task. Therefore, in the current work, we propose an alternative combination of BCI tasks, implementing a 2D cursor control and 4-target SSVEP task as two different modalities for accomplishing two independent and concurrent goals. Furthermore, since both tasks heavily occupy the visual channel, we are able to examine how the cognitive workload devoted to a single sensory input pathway must accommodate in order to successfully complete two separate tasks at once.

II. Methods

Nine healthy human subjects (6 right-handed, 8 male, average age 23.8±4.4 yrs.) with previous MI BCI experience (naïve to SSVEP BCI) participated in this study, providing written consent to a protocol approved by the University of Minnesota Institutional Review Board. Experienced users were specifically chosen to alleviate the effects of learning; with previous training, these subjects should be able to perform the learned tasks nearly automatically [31].

A. Experimental Setup

The cursor task was presented on a computer monitor measuring 53 cm × 34 cm located approximately 75 cm in front of the subject. Four 8 × 8 (6.3 cm × 6.3 cm) NeoPixel NeoMatrix LED panels (Adafruit Industries, New York City, New York, USA) were placed at each of the four corners of the monitor and programmed to flicker at different frequencies and colors: top left - 7 Hz (blue), bottom left - 8.5 Hz (white), top right - 10.5 Hz (green), and bottom right - 12.5 Hz (red) (See Fig. 1a). The LEDs were controlled via an Arduino Uno Rev3 microcontroller that communicated with BCI2000 through a custom Processing script that synchronized the LEDs with the trial structure.

Fig. 1.

(a) General trial structure and diagram of experimental setup (far left). Each trials begins with 3 seconds of rest and is followed by a 2 second preparation period and a 6 second go period. (b) (left) A weighted combination of the C3 and C4 electrodes are used for cursor control and the Oz and Iz electrodes are used for SSVEP identification. (right, top row) In the SMR only task, subjects control the 2-dimensional movement of a virtual computer cursor to hit one of four yellow targets. Subjects have up to 6 seconds to hit a target before the trial times out. (right, bottom row) In the SSVEP only task, subjects attend to one of four LED panels (flickering at different frequencies) indicated by the target picture in the center of the screen. For each target, subjects are given 0.5 seconds to avert their gaze towards the correct LED panel with the following 1 second of data used for classification. If the correct target is identified in the EEG, a new target is presented. Subjects are given up to three attempts for a single target, before it changes to a new one.

During the SSVEP alone and Multitask runs, eye tracking was performed to record subjects’ point of gaze (POG) throughout the tasks. Eye tracking was performed using a Gazepoint GP3 eye tracker [32] at sampling rate of 60 Hz. The eye tracker was located beneath the computer screen, approximately 65 – 75 cm from the subject and was calibrated before experimental blocks containing the SSVEP task using a five-point gaze procedure and monitored by the operator throughout the session. Subjects were instructed to remain as still as possible to maintain calibration during these blocks.

B. Experimental Design

Subjects participated in three experimental sessions composed of three conditions; SMR only, SSVEP only, and Multitask. Subjects performed a 2-dimensional cursor control task during the SMR only block, a visual attention task during the SSVEP only block, and both of these tasks simultaneously during the Multitask block. All three conditions contained eight runs and were performed in block-randomized order in each session.

C. Cursor Control Task

Each SMR run contained 21 trials, the first of which in each dimension was used to calibrate the buffer for the corresponding dimension. Feedback (cursor movement) was not provided until each buffer had been initiated. Trials were structured as follows: 3 seconds of rest were followed by a 2 second preparation period, in which a target cue appeared, and then concluded with a “go” period lasting up to 6 seconds. During this “go” period, subjects were instructed to move the cursor towards the indicated target using specific MI commands; left- and right-hand MI to move the cursor to the left and right respectively, MI of both hands to move the cursor up, and rest to move the cursor down. If the correct or an incorrect target was hit before the allotted six seconds, a trial would end, and be labeled as a hit or miss, respectively. If no target was reached within 6 seconds, the trial would be labeled an abort.

D. Attention Task

The same general structure was used for the SSVEP runs except that rather than a cursor task being performed during the “go” period, up to four SSVEP trials took place instead. For each SSVEP trial, a target (shape and color of the LED panel, see Fig. 1a, b) was presented in the middle of the screen. The subject was instructed to attend to the corresponding SSVEP whenever a target was presented; 0.5 seconds were provided for eye movement, followed by 1.0 second of attended gaze used for classification. All four LED panels flashed for the entire duration of the “go” period. Thus each decision took 1.5 seconds, using the most recent 1.0 second of frequency-tagged EEG. If the correct dominant frequency was identified in the EEG, the trial was considered a hit, and a new target was presented. If the dominant frequency was identified as one of the other three frequencies, the trial was considered a miss, and if identified as a nuisance frequency, defined as those halfway between the target frequencies and ± 1 Hz of the highest and lowest frequency, respectively, the trial was considered an abort. Subjects were given up to three chances to hit a particular target, after which the target would change, as was done after a hit trial. The target order was automatically determined before each run in a pseudo-random fashion, ensuring that that no two consecutive targets were the same.

E. Multitask

In the Multitask runs, subjects were instructed to perform the cursor control and attention tasks at the same time. SSVEP trials were only classified if the entire 1.0 second of classifiable EEG (a full 1.5 second trial) was acquired. Therefore, if the cursor trial ended in the middle of an SSVEP trial, this SSVEP trial would not be classified and would not count towards a hit, miss, or abort.

F. Online Signal Processing

64 channel EEG was acquired using an ActiveTwo (BioSemi, Amsterdam, The Netherlands) amplifier, sampled at 1024 Hz. EEG was acquired using the BCI2000 software [33] and was then sent via the FieldTrip buffer [34] to MATLAB (The MathWorks, Natick, Massachusetts, USA) and processed using custom scripts. Processing included downsampling to 128 Hz and filtering between 6 Hz and 30 Hz using a fourth order Butterworth filter, before being referenced to the common average.

G. Sensorimotor Rhythm Analysis

For cursor control, the 11 Hz alpha power was extracted from the C3 and C4 electrodes located over the bilateral sensorimotor areas using a Morlet Wavelet approach previously described in [35]. A weighted sum of the 11 Hz power in these electrodes, similar to [36] and [11] was used to compute the instantaneous cursor control signal for the two dimensions. Weights of −1 and 1 were applied to the C3 and C4 power, respectively, for the horizontal movement, and a weight of −1 was applied to the power in both electrodes for vertical movement. These weights were meant to capture the ipsilateral increase and contralateral decrease in 11 Hz power generated in response to hand MI tasks for intuitive control of the cursor trajectory [37]. For example, during right-hand MI, a positively weighted increase in C4 power added to a negatively weighted decrease in C3 power generates a net positive (right) control signal. Similarly, during left-hand MI, a negatively weighted increase in C3 power added to a positively weighted decreased in C4 power generates a net negative (left) control signal. A similar effect is observed for bilateral decreases and increases in the C3 and C4 power for both hands MI and rest, respectively. These instantaneous control signals were normalized to zero mean and unity gain using those values from the previous five trials stored in buffers for each dimension. This normalization procedure acts as a moving average filter to alleviate large perturbations in the control signal due to nonstationary signal transients and to remove bias towards any one target [38]. This process was performed every 100 ms to update the cursor position (stimulus), using the most recent 250 ms of data.

H. Steady-State Visual Evoked Potential Analysis

For SSVEP analysis, canonical correlation analysis (CCA) [39] was utilized to classify the EEG from the Oz and Iz electrodes covering visual cortical areas. Oz and Iz were selected for online classification based on previous offline investigations. As previously mentioned, the four LED panels were programmed to flicker at 7 Hz, 8.5 Hz, 10.5 Hz, and 12.5 Hz. Reference signals for classification were constructed at these four frequencies, as well as at the “nuisance” frequencies of 6 Hz, 7.75 Hz, 9.5 Hz, 11.5 Hz, and 13.5 Hz. Therefore, a total of nine reference signals were created and the EEG could be classified as the target LED frequency (hit), one of the other three LED frequencies (miss), or one of the five nuisance frequencies (abort) (see II.D for additional details). This design is analogous to the cursor control task where hits, misses, and aborts were also possible.

I. Eye Tracker Data Analysis

Eye tracker data was streamed to MATLAB from the Gazepoint software for real-time visualization and recording via custom scripts. Trigger events were sent to the eye tracker scripts to record times of interest throughout a run, i.e. run start and stop, trial start and stop, etc. While various measurements were recorded, we specifically chose to analyze the user’s POG throughout the SSVEP and Multitask blocks. We computed the relative dwell time users spent gazing (as measured by the POG) at different items on the screen during different phases of the trials by partitioning the field of view and POG measurements into the physical sections covered by the SMR targets, SSVEP targets, and uncovered areas. The time spent in each of these regions during the “go” periods was divided by the total “go” period duration to calculate the relative dwell time (in % run duration).

J. Statistical Analysis

Statistical analysis was performed using custom scripts in R [40]. Unless otherwise stated, the following statistical tests were used for reporting results. For block level and trial level statistical analyses, a repeated measures one-way ANOVA was used. When appropriate, a Bonferroni correction, or Tukey’s HSD post hoc test was used to correct for multiple comparisons. These instances are noted throughout the results section.

III. Results

In the offline analysis we intended to elucidate evidence of cognitive flexibility on multiple time scales. Firstly, we examined behavioral and electrophysiological correlates of each task at the block level by comparing these metrics in those blocks in which subjects performed only one task (SSVEP or SMR only) and the Multitask blocks. Secondly, we performed additional trial level analysis within the Multitask block to specifically illustrate multitasking on a finer time scale. By examining both the block level and trial level analysis, we can characterize the spatial and temporal behavior of cognitive flexibility within the BCI framework.

A. Block-Level Analysis

The primary metric we will refer to throughout this section is the percent valid correct (PVC), calculated as the number of hits divided by the sum of the number of hits and misses (valid trials). As can be seen in Fig. 2a, the overall online block level PVC for SMR only and SSVEP only tasks reached 57.9 ± 15.4% and 59.0 ± 14.2% respectively. During the Multitasking block, the performance of these tasks saw a modest decrease to 54.9 ± 17.2% for the SMR task and 57.5 ± 15.4% for the SSVEP task. Nevertheless, these values are well above the chance level of 25% for both tasks and conditions. This drop in performance was expected, however, the reduction in accuracy did not prove to be statistically significant for either task (SSVEP F = 0.33, p = 0.58; SMR F = 1.60, p = 0.24). Additionally, we constructed confusion matrices for both tasks during the respective SMR/SSVEP only and Multitask condition (Fig. 3). These confusion matrices provide details regarding the occurrence of false positives for each target type; there were no notable differences between the SMR and SSVEP only and Multitask matrices. However, specifically for the SSVEP task, the accuracy for the 12.5 Hz target was markedly lower than for the other three targets during both the SSVEP only and Multitask conditions.

Fig. 2.

(a) Block-level accuracy for the SSVEP task (left) and SMR task (right). (b) Block-level eye tracking data during the SSVEP task; when subjects are fixating at the correct SSVEP target (left) and incorrect SSVEP target (right). In both (a) and (b) the SSVEP/SMR only values are shown in gray and the Multitask values in black. (c) Block-level information transfer rate (ITR) compared across SMR only, SSVEP only, Multitask conditions (left). ITR for the SMR task (middle) and SSVEP task (right) in both the SSVEP/SMR only (solid) and Multitask (striped) blocks. Bars represent mean ± SEM.

Fig. 3.

Confusion matrices for the SMR task during the SMR only (a) and Multitask (b) blocks, and for the SSVEP task during the SSVEP only (c) and Multitask (d) blocks. The true target is indicated by the row and the predicted target by the column. Each row adds to 100% - off diagonal elements within a row indicate false positives.

The information transfer rate (ITR) is another common metric that measures the information content of these BCI decisions, measured in bits per minute, and depends on both the accuracy of the task as well as how fast the task can be performed [41]. Fig. 2c displays the ITR for the SMR and SSVEP tasks during the various blocks (middle and right plots) as well as the total ITR of three block types (left plot). The SMR ITR dropped from 11.0 ± 8.0 bits/min during the SMR only block to 10.8 ± 9.1 bits/min during the Multitask block. We observed a similar decrease in ITR for the SSVEP task with values of 27.2±20.8 bits/min and 26.6±19.2 bits/min in the SSVEP only and Multitask blocks respectively. In neither instance did we observe a statistically significant reduction (SMR F = 0.02, p = 0.91; SSVEP F = 0.05, p = 0.83). The total ITR of the Multitask block was 37.4±18.5 bits/min; a statistically significant effect of block type (F = 29.6, p = 4.82 * 10⁻¹⁰) was found and post hoc testing revealed that the Multitask ITR was significantly higher than the SMR ITR (Tukey’s HSD, p = 5.8*10⁻³). These performance metrics are further supported by various electrophysiological observations.

When examining the POG data (Fig. 2b), we found that subjects spent 64.0 ± 3.7% of the task period looking at the correct SSVEP target during the SSVEP only block, whereas during the Multitask block this value fell to 57.7±4.4%. The two values both reside within a reasonable range given that during the SSVEP task, in both the SSVEP only and Multitask blocks, ⅓ of the online time (0.5 second) is allotted for eye movement towards the intended target and ⅔ of the time (1.0 second) is intended to be spent gazing at the indicated target. Further, it is reasonable to expect that behavior would decrease during the Multitask block given the increased traffic occupying the visual channel. Similarly, on average, subjects spent only 15.0 ± 0.3 % and 14.8 ± 0.3 % of the task duration looking at the incorrect targets during the SSVEP only and Multitask blocks, respectively. The remainder of time unaccounted for was largely spent either looking at the center of the screen, or at one of the SMR targets in the Multitask case. Nevertheless, there was again no significant difference for either gaze metric (Correct SSVEP F = 1.89, p = 0.21; Incorrect SSVEP F = 1.83, p = 0.21) between the SSVEP only and Multitask blocks.

This behavioral observation also manifests itself in the EEG of electrodes covering the visual cortical areas. Fig. 4a and 4b display the power spectra, averaged across electrodes Iz and Oz, for the different SSVEP targets in both the SSVEP only and Multitask blocks. In both cases, distinct peaks can be seen at the intended frequencies of 7 Hz (blue), 8.5 Hz (black), 10.5 Hz (green), and 12.5 Hz (red) during trials in which those targets were presented. While the peaks appear broad, this can largely be attributed to windowing effects of the 1 second time period used for spectral power estimation in each trial. Nevertheless, the spectra from these two conditions appear nearly identical (See Fig. 4c–f). It should, however, be mentioned that the amplitude of the 12.5 Hz SSVEP (first harmonic) was notably lower than the other three SSVEPs in both the SSVEP only and Multitask conditions, helping to explain the corresponding decrease in classification accuracy identified in the confusion matrices in Fig. 3c–d. Statistical testing using a Bonferroni correction for multiple comparisons (n = 4), revealed no significant difference (7 Hz F = 5.82, p = 0.17; 8.5 Hz F = 2.09, p = 0.75; 10.5 Hz F = 1.62, p = 0.95; 12.5 Hz F = 0.08, p = 1.0) in the amplitudes of the first harmonics of the SSVEPs between these conditions.

Fig. 4.

Average power spectra for the Oz and Iz electrodes during the different SSVEP target trials (blue – 7 Hz, black – 8.5 Hz, green – 10.5 Hz, red – 12.5 Hz) in the SSVEP only (a) and Multitask (b) blocks. (c–f) Statistical comparison of SSVEP amplitude in the four SSVEP frequencies during the SSVEP only (gray) blocks and Multitask (black) block. Bars represent mean ± SEM.

A similar phenomenon can be observed in the electrophysiology underlying the cursor control task. Task-related modulations are measured by regressing the EEG alpha power (9.5 –12.5 Hz) during cursor control against the target locations and obtaining R values across the EEG montage. This analysis was performed for horizontal control (Fig. 5a - left vs. right targets) and vertical control (Fig. 5b - up vs. down targets) separately, as is commonly reported in literature [42], [43]. In the R topographies for horizontal and vertical control, in both the SMR only and Multitasking conditions, there are very obvious focal regions of modulation across electrodes covering the bilateral motor cortical regions. Quantitative comparisons in the C3 and C4 electrodes were performed for both control dimensions since these electrodes were used for the online cursor control. For horizontal control, depicted in Fig. 5c, the R values in C3 and C4 were found to be −0.32 ± 0.24 and −0.25± 0.35 for the SMR only condition, and −0.27 ± 0.24 and 0.29 ± 0.37 for the Multitask condition. There was no significant effect of condition for either electrode (C3 F = 0.28, p = 1.00; C4 F = 0.12, p = 1.00, Bonferroni corrected, n = 2). As noted in Fig. 5d, the R values for vertical control were larger than for horizontal control, reaching −0.43 ± 0.19 and −0.50 ± 0.28 for C3 and C4 in the SMR only condition, and −0.21 ± 0.35 and −0.35 ± 0.40 for the same electrodes in the Multitask condition. Similar to horizontal control, no significant effect of condition was found for either electrode (C3 F = 3.74, p = 0.17; C4 F = 1.22, p = 0.60, Bonferroni corrected, n = 2).

Fig. 5.

R topographies for left vs. right cursor targets (a) and up vs. down cursor targets (b). Statistical comparison of R values at the C3 and C4 electrodes for left vs. right cursor targets (c) and up vs. down targets (d). Bars represent mean ± SEM. SMR only values are shown in gray and Multitask values in black.

B. Trial-Level Analysis

Although subjects were instructed to perform both tasks at the same time during the Multitask runs, the similarity of behavioral and electrophysiological results during the Multitask block compared to the SSVEP/SMR only blocks (Fig. 2–4) may not be due to truly performing both tasks simultaneously. For example, in one 6 second trial a subject may focus solely on obtaining a “hit” in the SMR cursor task while sacrificing performance in the SSVEP trials. In the next trial, the subject may switch his/her cognitive focus and perform the opposite, focusing on “hits” in the SSVEP task while settling for a “miss” in the SMR task. In this sense, subjects could achieve well above chance performance of each modality; however, it would not be due to successful simultaneous task execution. Therefore, in order to investigate cognitive flexibility more in depth and at a finer time scale, we examined the performance of each task on a trial-by-trial basis within the Multitask block. Because the SMR task determined overall trial duration, we specifically examined the SSVEP accuracy during various SMR result trials (hits, misses, and aborts). While there was a statistically significant effect of SMR trial type on SSVEP accuracy (F = 10.5, p = 1.0*10⁻⁴); Tukey’s HSD post hoc test did not reveal any pairwise significant differences (Fig. 6b), indicating that subjects performed the SSVEP consistently regardless of their performance on the SMR task.

Fig. 6.

(a) Conceptual diagram of congruent and incongruent targets. White arrows indicate those SSVEP targets that are incongruent (across the workspace) to the bottom SMR target and black around indicate those that are congruent (neighbors within the workspace). (b) SSVEP accuracy during SMR hit, miss, and abort trials. Values for SMR hit trials are shown in light gray, SMR miss trials dark gray, and SMR abort trials black. (c) Congruency analysis for the SSVEP targets during SMR hit (left), miss (middle), and abort (right) trials. Bars represent mean ± SEM. Accuracy values for congruent targets are shown in solid bars and those for incongruent targets are shown in striped bars.

To expand on this analysis, we looked into the spatial dependency of cognitive flexibility by further examining SSVEP performance when the SSVEP targets were congruent (close) vs. incongruent (far) from the SMR targets (Fig. 6a). SSVEP targets were specifically placed at the corners of the workspace to be congruent and incongruent with two SMR targets each, simplifying this analysis to two categories. The congruency analysis was again broken down for SMR hit, miss, and abort trials. Interestingly, as can be seen in Fig. 6c we found very similar trends across all three cases with congruent targets resulting in SSVEP accuracies of 60–65% and incongruent targets resulting in reduced values of 35–45%. In all three types of SMR trial results, there proved to be a statistically significant effect of target congruency (SMR hits F = 23.1, p = 1.4 * 10⁻³, SMR miss F = 13.4, p = 6.4 * 10⁻³, SMR aborts F = 55.0, p = 7.5 * 10⁻⁵) on SSVEP accuracy. While the SSVEP accuracy during incongruent targets in the SMR hit trials was lower than that in the SMR miss and SMR abort trials, it was still noticeably above theoretical chance level (25%).

In addition, we performed a similar analysis on the SSVEP and SMR EEG data to examine corresponding effects of target congruency on the electrophysiology. The amplitudes of all four SSVEP targets and all SMR trial types are displayed in Fig. 7a–d. There was a significant main effect of SMR result type only on the amplitude of the 8.5 Hz SSVEP (F = 3.64, p = 0.031; Fig. 7b), however, no pairwise differences reached significance in post hoc testing. When testing for the main effect of target congruency on the SSVEP amplitude (Fig. 7e–h), we observed a significant reduction in the 10.5 Hz signal in SMR abort trials (F = 5.72, p = 4.3 * 10⁻²; Fig. 7g) and in the 12.5 Hz signal in SMR miss trials (F = 5.15, p = 5.2 * 10⁻²; Fig. 7h). While additional instances displayed a near significant reduction in SSVEP amplitude for incongruent targets, an increase (not significant) in amplitude was observed in only two instances; the 7 Hz target in SMR abort trials and the 8.5 Hz target in SMR hit trials.

Fig. 7.

SSVEP amplitude during SMR hit, miss, and abort trials for the 7 Hz (a), 8.5 Hz (b), 10.5 Hz (c), 12.5 Hz (d) SSVEP targets. Values for SMR hit trials are shown in light gray, SMR miss trials in dark gray, and SMR abort trials in black. Congruency analysis for the SSVEP targets during SMR hit (left), miss (middle), and abort (right) trials for the 7 Hz (a), 8.5 Hz (b), 10.5 Hz (c), 12.5 Hz (d) SSVEP targets. Bars represent mean ± SEM. Amplitude values for congruent targets are shown in solid bars and those for incongruent targets are shown in striped bars.

Congruency analysis for the C3 and C4 R values across the different SMR trial types are displayed in Fig. 8–9. A significant main effect of SMR result type was found for the C3 electrode for both and left vs. right targets (F = 3.21, p = 4.6 * 10⁻²; Fig. 8c–b) and up vs. down targets (F = 5.83, p = 4.5*10⁻³; Fig. 9c–b), however, no significant pairwise differences were observed for either electrode or control dimension. Additionally, we observed a significant main effect of target congruency in electrode C4 during left vs. right SMR miss trials (F = 11.3, p = 9.9 * 10⁻³; Fig. 8d) and in electrode C3 during up vs. down SMR abort trials (F = 8.12, p = 2.9 *10⁻²; Fig. 9d). Unlike the SSVEP amplitudes, there was no consistent trend of reduced electrophysiological measures during incongruent trials compared to congruent trials.

Fig. 8.

(a) R topographies during SMR hit (left), miss (middle), and abort (right) trials for left vs. right targets. (b) Statistical comparison of R values at the C3 and C4 electrodes for left vs. right targets during SMR hit (light gray bar), miss (dark gray bar), and abort (black bar) trials. (c) R topographies during congruent and incongruent trials for SMR hit (left), miss (middle), and abort (right) trials. (d) Statistical comparison of R values at the C3 and C4 electrodes during congruent and incongruent trials. Bars represent mean ± SEM. R values for congruent targets are shown in solid bars and those for incongruent targets are shown in striped bars.

Fig. 9.

(a) R topographies during SMR hit (left), miss (middle), and abort (right) trials for up vs. down targets. (b) Statistical comparison of R values at the C3 and C4 electrodes for up vs. down targets during SMR hit (light gray bar), miss (dark gray bar), and abort (black bar) trials. (c) R topographies during congruent and incongruent trials for SMR hit (left), miss (middle), and abort (right) trials. (d) Statistical comparison of R values at the C3 and C4 electrodes during congruent and incongruent trials. Bars represent mean ± SEM. R values for congruent targets are shown in solid bars and those for incongruent targets are shown in striped bars.

IV. Discussion

In the current study, we demonstrate the ability of healthy human subjects to perform two independent cognitive tasks successfully and simultaneously, and investigate the physiological correlates thereof on various time scales. By examining cognitive flexibility at the block level, it can be seen that users perform two independent tasks simultaneously and competently on the temporal scale of 3–4 minutes (approx. length of a single run). While a shorter time scale (trial-by-trial, seconds) of cognitive flexibility will be discussed in subsequent paragraphs, we feel that the longer time scale is most relevant for realistically integrating BCI technology into daily life. As most complex tasks that are accomplished on a daily basis take time to complete, i.e. spatial movement, oral conversations, etc., having temporal flexibility to perform these tasks is necessary in order to accomplish multi-stage tasks or even plan out future movements/actions. Many of these concurrent tasks (physical or mental) are performed automatically by healthy people without the conscious need to split his/her attention; however, if physical limitations diminish such an ability, it would be paramount that users be able to actively perform multiple cognitive tasks at once. In the current study, we have shown that both behavioral and electrophysiological measures of BCI control display only a minor drop when doing two cognitive tasks at one time, compared to just one. In this sense, we provide support for the concept of multitasking from both a BCI and neuroscience perspective.

Overall, the accuracies achieved in the current study during the SSVEP/SMR only blocks, as well as during the Multitask blocks, are on par with relevant studies previously reported in literature. While state-of-the-art SSVEP BCIs using subject-specific training data can achieve PVCs of >90% with 40 targets [44], similar plug-and-play setups to what was used in the current study (four target frequencies, 1 second trials, CCA classification) commonly report accuracies in the 50–60% range [5], [45]. The group level results presented here fall well within that range with accuracies of 59.0 ± 14.2% for SSVEP alone, and 57.5 ± 15.4% for Multitask. Despite the high variability in trial structure, montage selection, and various other factors, the reported 2D SMR performance of 57.9 ± 15.4% for SMR alone and 54.9 ± 17.2% for Multitask also resembles that achieved in other EEG [18], and even ECoG [46] studies by subjects with similar experience levels to those who participated in the current study. Nevertheless, while more advanced systems have been reported in literature, it should be reiterated that the overall goal of this work was not to maximize performance in either BCI task, but rather to demonstrate the concept of cognitive flexibility through a BCI framework and investigate the extent into which future BCI systems may be better integrated into natural daily life.

Similar support for cognitive flexibility across a longer time scale is found when examining the electrophysiology across these different block conditions. Specifically, our eye tracking data indicates that subjects physically perform the task very similarly in both the SSVEP only and Multitask blocks, averting their gaze towards the intended targets for nearly the same amount of time during each condition. However, it has been shown that the detection of SSVEPs, and therefore the performance of these systems, largely depends on attending to the flicker in addition to gaze fixation, rather than simply fixation alone [47], [48]. In this sense, the SSVEP amplitudes in the two conditions are more indicative of the user’s cognitive effort devoted to the task. Interestingly, when examining the SSVEP peak amplitudes in the two conditions, we see that the amplitudes for all four frequencies used in this study are not significantly different between the SSVEP only and Multitask conditions, The combination of these behavioral and physiological results strongly point to the idea that a similar cognitive load is devoted to the visual SSVEP task when doing it alone and when simultaneously also performing the SMR task.

It has also been suggested that event-related (de)synchronization (ERD/S) is a product of specific cognitive processes related to selective attention of the motor system [37]. ERD/S is the underlying phenomenon governing the modulatory activity of the central mu rhythm and is captured via the regression analysis producing the R values displayed in Fig. 5a–b. While a somewhat large cluster of electrodes is typically implicated in successful SMR BCI control, we chose to specifically examine the R values at C3 and C4, partly because these electrodes were used for the online cursor control in this study and partly because these are generally known to overly the hand cortical regions and capture various hand MI tasks. Furthermore, R values highlight the directionality of mu modulation for the different control directions and can elucidate MI strategies for the different cursor directions. For horizontal control in both conditions, we observed negative correlations at left hemisphere electrodes and positive correlations at right hemisphere electrodes, supporting the weights used for online control. Similarly, for vertical control, negative correlations were observed bilaterally for both conditions. Collectively, these patterns indicate that the proper MI tasks were being performed by the subjects and that proper electrodes were selected for online control.

Contrary to the longer time scale, examining the Multitask blocks on a trial-by-trial basis provides insight into the cognitive characteristics of concurrent task execution. In total, we found that during the Multitask block, subjects were able to perform both the SSVEP and SMR task correctly in 18.6% of the trials, and at least one of the two tasks correctly in almost 56.7% of the trials. In examining these trials, we specifically looked at SSVEP performance during trials that are categorized by the SMR result. If users were to only perform the SSVEP task correctly, or significantly more often, during SMR miss trials, we would conclude that they are focusing most or all of their cognitive workload on one task at a time (the SSVEP task). This would similarly be observed as very low SSVEP accuracy during SMR hit trials; in this case the focus would be on the SMR task. The strong balance in SSVEP accuracy during SMR hit and SMR miss trials suggests that users are able to effectively split their cognitive workload and perform both tasks at once.

While we can look at the ability to multitask across multiple time scales, the spatial aspect of cognitive flexibility reveals itself much more clearly in the shorter, trial-by-trial case. In order to quantify this, we broke down the overall SSVEP accuracy into that for SSVEP targets next to the SMR target (congruent targets) and SSVEP targets on the opposite side of the workspace from the SMR target (incongruent targets). In all SMR result cases (hit, miss, abort) there was a clear and statistically significant difference in the SSVEP accuracy when SSVEP targets were congruent vs. incongruent to the SMR targets. These strong differences indicate that there is a spatial dependency on cognitive flexibility and that these two tasks, both of which heavily occupy the visual channel (SSVEP for attention and SMR for feedback/input), are more successful when the tasks occur in a similar region of the visual field. Based on these results, it appears that cognitive flexibility was precisely exploited during trials containing spatially congruent SSVEP and SMR targets, as evidenced by the ability of users to maintain strong SSVEP performance during successful SMR trials (Fig. 6c left). While the results of above chance performance for incongruent targets may still support the concept of flexible spatial processes of attention [49], when compared to congruent targets these findings fall in line with previous human studies that found degraded behavioral performance when attention is split across various visual fields [50].

Similar to SSVEP accuracy, we detected a drop in SSVEP amplitudes during incongruent target trials compared to congruent target trials (Fig. 7 e–h). While significant reductions during incongruent trials were found in only two of the 16 cases (with two more nearly significant), there was a highly consistent trend of reduced amplitude across all SMR result types and target frequencies that can be attributed to the decreased performance. For the modulatory R values computed from the SMR EEG data, however, less consistency was seen during congruent and incongruent target trials. As expected, for both left vs. right and up vs. down targets, we observed the strongest modulation during SMR hit trials, followed by SMR abort trials and then SMR miss trials (Fig. 8a, 9a). These trends follow the proportional relationship between increased modulation and increased performance, and are generally reflected in the sensorimotor C3 and C4 electrodes highlighted in the current analysis (Fig. 8b, 9b). For the horizontal targets, we noticed countermodulation of the left and right hemispheres for both congruent and incongruent target trials (Fig. 8c–d), but also recognized a largely global negative modulation during incongruent trials (Fig. 8c). A similar observation was made for the up vs. down targets (Fig. 9c–d); where focal regions of modulation were displayed during congruent trials, widespread modulation was found across the scalp, notably in frontal regions, during incongruent trials. While no concrete conclusions can be drawn from these qualitative findings, these results indicate the target incongruency and the associated attention mechanisms involved in processing information across visual fields may affect brain regions/networks outside of the sensorimotor areas.

We did not systematically investigate the case where two simultaneous tasks require different sensory input pathways, e.g. an auditory task and visual task. As previously mentioned, a highly practical real world example of this scenario would be performing MI BCI while maintaining a conversation; however, in such a case one would think that the lack of task competition within the same input channel would lead to increased overall performance and a reduced effect of incongruent goals. In fact, this effect was previously observed in [29] where high performance was found in concurrent visual and auditory tasks. Nevertheless, the participants in the current study, who performed two tasks involving visual input, did not report feelings of overwhelming sensory inflow or mental fatigue, when compared to performing each task individually. Furthermore, users who do feel overloaded with sensory input may be aided by various meditation practices. It has been suggested that meditation and mindfulness training are strongly intertwined with cognitive flexibility [51], indicating that combining these practices with BCI use may not only improve overall performance [52], but also ease the transition of BCI technology into everyday life.

V. Conclusion

In the present work, we present a framework for investigating cognitive flexibility through the combination of two distinct BCI tasks. We found that healthy users are able to perform two independent cognitive tasks simultaneously with high accuracy. Over the long time scale (task run, minutes), we found that there was a negligible decrease in performance in both the SSVEP and SMR tasks when performing them concurrently, as compared to when doing them individually. We also reported a very similar effect on the short time scale (task trial, seconds), revealing that users do not necessarily sacrifice performance in one task for the successful completion of the other. Nevertheless, we found that there is a spatial dependency to this ability, whereby success in both tasks is much more likely when both of them occupy a similar region of the user’s visual field.

When particular neurological dysfunctions inflict physical disabilities, subconscious tasks that are the building blocks for downstream actions may become disabled. These patients would ideally focus on replacing these primary tasks and have to perform the downstream action with a significant delay, if at all. The concept of cognitive flexibility demonstrated in the current study, whether inherent or learnt, may allow users in such situations to both perform fundamental cognitive tasks while also preparing or executing downstream or supplementary actions simultaneously, significantly enhancing the utility of BCI technology and the user’s quality of life. In all, revealing such cognitive flexibility with BCI demonstrates the possibility for moving noninvasive BCI into everyday life.