3-dimensional Brain-computer Interface Control through Simultaneous Overt Spatial Attentional and Motor Imagery Tasks

Abstract

Objective:

While noninvasive electroencephalography (EEG) based brain-computer interfacing (BCI) has been successfully demonstrated in two-dimensional (2D) control tasks, little work has been published regarding its extension to practical three-dimensional (3D) control.

Methods:

In this study, we developed a new BCI approach for 3D control by combining a novel form of endogenous visuospatial attentional modulation, defined as overt spatial attention (OSA), and motor imagery (MI).

Results:

OSA modulation was shown to provide comparable control to conventional MI modulation in both oneand two- dimensional tasks. Furthermore, this work provides evidence for the functional independence of traditional MI and OSA, as well as an investigation into the simultaneous use of both. Using this newly proposed BCI paradigm, sixteen participants successfully completed a 3D eight target control task. Nine of these subjects further demonstrated robust 3D control in a twelve target task, significantly outperforming the information transfer rate achieved in the 1D and 2D control task (29.7±1.6 bits/min).

Conclusion:

These results strongly support the hypothesis that noninvasive EEG based BCI can provide robust 3D control through endogenous neural modulation in broader populations with limited training.

I. INTRODUCTION

Brain-computer interfacing (BCI) is a promising method for providing alternative connections between the brain and the outside world in concert with natural connections or as replacements for natural links potentially disrupted by disease or injury [1],[2]. One of the ultimate goals of BCI is to establish and restore natural limb movement in prosthesis, robotic substitutes, and natural limbs. While invasive BCI research has shown promise in demonstrating the control of prosthesis [3]–[6] and restoring function in limbs through electrical stimulation [7], the invasive nature of these approaches poses risks to the patients, often requiring surgeries, the implantation of cortical electrodes, and the management of post-implantation complications and maintenance [8]. Efforts have been made to develop noninvasive BCI techniques that offer alternatives to invasive technology. While there are a few studies which have tried to use noninvasive electrophysiological signals for advanced control, such as the one-dimensional (1D) control of a prosthetic limb [9],[10] or the pseudo three-dimensional (3D) control of a robotic arm [11] through the sequential combination of two-diamensional (2D) and 1D control signals, there still exists a noticeable gap, especially in noninvasive BCI, between the tasks involved in experimentation and those of daily-life.

Electroencephalography (EEG) is particularly suitable for BCI due to its portability, safety and relatively low cost for researchers and end users [12]. Furthermore, the development of dry electrodes [13,14] and the wireless transmission of EEG signals, allows for the creation of more practical BCI applications in daily life, including drowsiness detection [15] and wearable robotics [16]. There are effectively three different kinds of BCI systems based on noninvasive EEG: BCI based on endogenous modulation [2],[17],[18], exogenous stimulation presentation, such as P300 [19] and steady state evoked potential (SSVEP) [20],[21], and a mixture of the different modalities [22]–[24]. BCI based on the endogenous modulation of brain rhythms is particularly suitable for real world continuous control in 3 or lower dimensional space because it does not require stimulus targets, which would have to be displayed in pre-designated locations. Although the motor imagery (MI) based modulation of brain rhythms has successfully demonstrated robust performance in various continuous control applications [2],[11],[18],[25]–[29], the number of independent MI induced signals is fairly restricted, due to EEG’s limited spatial resolution. The most common, reproducible brain patterns used in MI BCI consist of combinations of imagery involving both hands and feet [25],[30]. Unfortunately, as there is high variability in individual aptitude, a large portion of subjects find the immediate control of MI BCI (without training) difficult [31]. Furthermore, tasks become much more challenging when several different motor imaginations, such as hands and feet, are combined for high dimensional control in a non-intuitive fashion. So much so that outside of this work there exists a singular noninvasive continuous 3D control study in which only four subjects demonstrated proficiency [32]. As endogenous modulation provides a control strategy with high agency, and has been previously demonstrated as suitable for continuous control, a feature not-defined in exogenous control strategies, it is essential to explore novel strategies for endogenous modulation, further expanding the number of independent control signals and facilitating the 3D control of BCI in a wider population.

In this study, we introduce a BCI control strategy based on the endogenous modulation of overt spatial attention (OSA). Previous studies have demonstrated abundant evidence that human subjects can covertly deploy their attention to different spatial locations [33]–[36] although the underlying mechanism and the relationship between neuronal modulation and behavioral outcome are still largely unknown [37],[38]. Traditionally, covert spatial attention (CSA) induces a distinct spatial and temporal modulation of alpha rhythms, which are hypothesized to be functionally correlated with the enhancement and or suppression of neuronal activity associated with attended and unattended targets [36],[38],[39]. Previous studies have demonstrated the viability of using CSA to classify the subjects’ spatial locus of attention through offline analysis [40]–[42] or online binary classification [43]. However, no control utilizing continuous feedback, which is critical for its natural application [25]–[29], has been shown. Noting that the direction of gaze and the direction of attention are usually aligned [37], we hypothesized that the modulation of visuospatial attention might be significantly increased if subjects are allowed to shift their gaze voluntarily. Additionally, attention might have to be dynamically deployed for the natural control of a robotic or prosthetic arm in practical, interactive scenarios. Therefore, we propose OSA (Fig. 1A) as a strategy to perform BCI control. In this work, we investigated (hypothesis I) whether subjects are able to gain BCI control via the newly proposed OSA modulation strategy and if said control was comparable to conventional MI modulation (Fig. 1A), and (hypothesis II) whether the concurrent modulation of the newly proposed OSA and established MI allows for 3D control.

Fig. 1.

A) Overall experimental strategy for control via OSA and MI. Subjects overtly attend to either the left or right side according to the task cue, directing the cursor’s movement to either the left or right; subjects imagine the repeated movement of their left or right hand, according to the task cue, directing the movement of the cursor to the left or right during MI modulation. B) Overall study design. Each participant was randomly assigned to one of two groups and completed three screening sessions. In group one, OSA was performed first, followed by MI; while group two performed the reverse. A subset of the subjects whose performance was higher than the pre-determined threshold was invited to complete three sessions of 3D 8 target tasks with further subsection of those subjects participating in two additional sessions of 3D 12 target tasks.

With the research questions in mind, twenty-three healthy subjects were recruited to participate in multiple sessions of experiments. They were randomly assigned into one of two groups for the first three screening sessions in order to compare the performance between the OSA and MI modulation tasks (hypothesis I). After which, a subset of the subjects who exceeded a specified performance threshold was asked to participate in several sessions of 3D control (hypothesis II).

The paper is organized as follows. The experimental design, online signal processing and offline evaluation criterions are described in Section II. The experimental results and neurophysiological analyses are presented in Section III with the discussion and conclusion following in Sections IV and V, respectively.

II. MATERIALS AND METHODS

A. Subjects and Experimental Setup

Twenty-three healthy subjects (9 females; 1 left handed; average age 26.1±8.9; range: 19–55; 5 subjects had BCI experience with the MI task, none had experience using OSA for BCI) participated in the study of online cursor control BCI. Each subject was requested to participate in 3 screening sessions, 3 sessions of 3D 8 target cursor control and 2 sessions of 3D 12 target cursor control. Twenty-one out of 23 subjects completed all 3 screening sessions, with 2 subjects dropping due to scheduling conflicts. Sixteen subjects passed the screening sessions and finished the three sessions of 3D 8 target tasks, while 9 subjects completed the two additional sessions of 3D 12 target tasks. All procedures and protocols were approved by the Institutional Review Boards of the University of Minnesota and Carnegie Mellon University. Informed consent was obtained from all subjects prior to their participation in the experiment.

Sixty-four channels of EEG were acquired at a sampling frequency of 1kHz using a Neuroscan SymAmps RT system (Neuroscan Inc, Charlotte, NC). A bandpass filter encompassing 0.5 to 200 Hz as well as a notch filter at 60 Hz were applied to the raw EEG signals. During all recordings, the vertex was used as the reference while the forehead served as the ground. The impedances of all electrodes were kept below 5kΩ at the beginning of the experiment. Impedance was not checked during the active portion of the experiment in order to avoid interrupting both the experiment and the subject. However, the impedance was checked at the conclusion of each session with 97% of electrodes remaining below 5kΩ.

An eye tracker (Gazepoint GP3) was used to track the eye movement during each session, and the data were recorded and synchronized with BCI2000 key events through a customized MATLAB script.

B. Experimental Design and Protocol

For each session, subjects were randomly assigned to one of two groups (see Fig. 1B). The number and types of trials were consistent across groups, but shuffled to diminish potential confounds introduced through subject exhaustion. Subjects either performed OSA before MI (Group One) or the converse (Group Two).

Each session is composed of a fixed number of runs, with each run consisting of 25 trials. During each trial of the OSA modulation task, the subjects were instructed to focus their attention on the highlighted target bar (corresponding to the directions of left, right, up and down) and to minimize their gaze movements away from the target (try their best to avoid saccades induced by cursor movement, See Fig. 1A). The eye tracker was used to record their gaze points during the whole experiment and to make sure that the subjects followed the instructions properly. The operator would provide a gentle reminder if the subjects did not follow the instructions. They were instructed to move the cursor towards the designated target through their spatial attention. During each trial of the MI modulation task, the subjects were instructed to imagine the repeated movement of either their left or right hand, in order to move the cursor left or right respectively, both hands to move the cursor up and to relax to move the cursor down [11]. Note that the subjects also felt comfortable moving the cursor forward and backward by imagining either both of their hands or relaxing, during the 3D control task.

Each trial started with a black screen for 2 seconds. During these initial seconds the subject was instructed to stay relaxed and still. Following the initial two seconds a highlighted yellow bar appeared at either the top, bottom, left, or right edge of the screen in 2D cases, depending on the task and target, or at the edges of the unit cube in the 3D 8 and 12 target tasks. To allow for subject preparation, the highlighted yellow bar was shown to the subjects for 1.5 seconds prior to allowing control. A pink cursor appeared at the center of the screen at second 3.5 and the subjects were allowed to move the cursor through the modulation of their brain waves. Subjects were given a maximum of 9 seconds in each trial to hit the correct target while avoiding the other targets. Under this paradigm each trial could result in a hit, miss (incorrect target), or abort (no target reached). During a one second period following feedback the cursor remained on-screen, frozen in place. Then a new trial began under the same procedure. In all experimentation, the movement of the cursor was presented in the software BCI2000 [44].

In the first session, group one performed the OSA tasks first, which consisted of two runs of left versus right (LR) control without feedback followed by two runs of LR control with feedback. This procedure was then repeated with the up versus down task (UD). Each run was followed by a short break, about 1–3 minutes, with the length depending on a subject’s willingness to proceed. Following the completion of the OSA tasks in group one, the same series of experiments were performed using MI for control instead of OSA. So that the influence of feedback on these modulations could be determined, control tasks consisting of no feedback (either LR or UD) were always presented to the subject prior to the corresponding task containing feedback. As mentioned previously, group two performed the MI tasks first, followed by the OSA tasks (See Fig. 1B, 1st column). In the second session, group one performed four runs of LR and UD control with feedback using MI first followed by OSA, while group two performed the same tasks utilizing OSA for control first (Fig. 1B, 2nd column). In the third session, both groups performed two runs of LR control, followed by two runs of UD control, and 4 runs of 2D control. As before, group one performed OSA before MI, with group two performing the converse (Fig. 1B, 3rd column). For all experimentation, each subject underwent a maximum of one session per day.

Subjects with Percent Valid Correct (PVC) [11],[45] higher than 70% in any two of the consecutive runs of LR control, as well as UD control and PVC higher than 40% in any two consecutive runs of 2D control [46] were considered eligible for participation in the subsequent 3D control experiments. This criterion was set to exclude subjects who could not gain meaningful BCI control during their limited exposure. For each of the participants, an optimal combination of MI and OSA modulation was customized for subject specific 3D control. If the OSA performance was higher than the MI performance for a particular subject, then OSA modulation was used to control the movement in the frontal plane, while MI modulation was chosen for the control of the 3rd dimension, i.e. depth control (see Fig. 1B, top panel of the right column). On the other hand, if the MI performance was stronger, then MI modulation was selected to control the movement in the 2D horizontal plane while OSA modulation was chosen for control of the 3rd dimension, i.e. the UD control (see Fig. 1B, bottom panel at the right column). Each of the three sessions of 3D control was performed on a different day. During these sessions, 8 runs of 8 target 3D BCI control were performed with the targets located on the edges of the front and back faces of a cube centered on the cursor’s starting position Subjects who demonstrated sufficient 3D control were asked to subsequently participate in two additional sessions of 12 target 3D BCI, in which the targets were placed on all edges of a cubic workspace, centered on the starting position.

C. Online Signal Processing

For online cursor control, the channels which cover the sensorimotor and parietal-occipital regions were selected (22 electrodes in total). Higher alpha band (10–14 Hz) power was extracted using an autoregressive (AR) approach, previously described [11], from the C3 and C4 electrodes located over the bilateral sensorimotor areas for MI control, and the P3, P4, Pz and Poz electrodes located over the parietal-occipital areas for OSA control. Previous studies have verified that during MI modulation the higher alpha rhythm is functionally dissociated from the lower alpha rhythms and has a more focal and movement-specific topography [47],[48]. We adopted a similar idea for OSA modulation here, and used the higher alpha power for online control. A small Laplacian filter [49] was used to filter out common sources of local electrical activity, and was applied to each of the electrodes prior to calculating the alpha power. A weighted sum of the alpha power in either C3 and C4 or P3, P4, Pz and Poz was used for instantaneous cursor control in the MI and OSA tasks respectively. The weights for each electrode were fit to the data collected during the appropriate task without feedback by maximizing the discrimination accuracy and/or adjusting the results based on prior electrophysiological knowledge and experience [11],[45]. The instantaneous control signals for each dimension were stored in an online buffer and were normalized to zero mean and unit variance to control the cursor’s speed, effectively smoothing out cursor movement and correcting for large transient artifacts in the non-stationary EEG signals. Besides the filtering previously mentioned, no artifact removal algorithm was utilized during online signal processing.

D. Evaluation of Behavioral Performance and Electrophysiology

The behavioral performance of online BCI control was evaluated in terms of Percent Valid Correct (PVC) [41] and information transfer rate (ITR) [46]. PVC and ITR are both widely adopted metrics [19]–[24] in various BCI applications for the evaluation of online continuous BCI control. PVC is calculated as the number of hits divided by the sum of the number of hits and misses (valid trials) [11],[45]. ITR measures the information content of the BCI decisions, measured in bits per run, and depends on both the accuracy of the task as well as how fast and how many hits can be performed in each run [50]. The statistical analysis was performed using custom scripts in R by comparing the group average performance among different modulation conditions, e.g. the OSA tasks and MI tasks, 1D control, 2D control and 3D control. When applicable, results were expressed as mean ± SEM (standard error of the mean), unless otherwise stated. Mixed repeated measures ANOVAs, linear mixed effect models (lme, a function included in an R package ‘nlme’) and paired T-tests were employed to evaluate the statistical significance of group performance (across sessions) or group average R-values among different modulation conditions. The level of significance testing was set to p<0.05. When appropriate, a post hoc Tukey’s test was used to correct for multiple comparisons.

In addition to the evaluation of behavioral performance, the examination of neural electrophysiology could help discern characteristics of the modulation due to the OSA and MI tasks. R-values are frequently used to quantify how strongly the means of two distributions, e.g. the band power of left and right hand imagination or attentional modulation, differ relative to their variance [2],[51]. R-values can be calculated at each electrode, according to their definition, with the corresponding R-value topography showing how strongly the band-power of electrodes correlates with the task. In the offline analysis, R-values were calculated based on all trials and a subset of trials in which the correct targets were hit, separately, in the alpha frequency band previously used for online control. As all subjects performed the instructed tasks first without any online neuronal feedback, and then with online feedback, which consisted of the cursor movements generated from decoded EEG signals, this dataset offers a unique opportunity to investigate the role of neuronal feedback on electrophysiology. The R-values were calculated for all subjects and sessions independently, with an R-value topography grand average over subjects and sessions during different conditions being derived and compared.

III. RESULTS

A. BCI Behavioral Performance of 1D, 2D control across Sessions

The behavioral performance of the OSA and MI modulation was compared in the conditions of LR control (Fig. 2A), UD control (Fig. 2B) and 2D control (Fig. 2C), separately (N=21). No statistically significant differences were found between the PVC of the OSA and MI modulation tasks (L/R or U/D) across all three sessions. The group average performance and the standard error of the mean (SEM) across all three sessions of OSA and MI modulation for LR and UD control were 73.5±3.9% and 84.0±4.2% (LR), and 80.0±3.2% and 77.6±4.2% (UD), respectively. A mixed repeated measures ANOVA was used to determine whether the two methods produced different BCI performance over the three sessions. For the LR control task, the main effect of the method is F(1,40) 3.36, p = 0.07, n² = 0.07 (generalized Eta-Squared measure of effect size); the main effect of session is F(2,80) = 0.77, p = 0.47, n² < 0.01; interaction effect of method and session is F(2,80) = 0.21, p =0.81, n² < 0.01. For the UD control task, the main effect of the method is F(1,40) = 0.52, p = 0.47, n² = 0.01; the main effect of session is F(2,80) = 0.22, p = 0.80, n² < 0.01; the interaction effect of method and session is F(2,80) = 0.05, p =0.95, n² < 0.01. In the 2D control task, individual performance was displayed on the left side of the dashed line while the group average was shown on the right. The group averages of the OSA and MI modulation for 2D control were 48.0±3.7% and 55.0±5.5%, respectively, with no significant difference 0.30). Although there were no statistically significant group level differences between the control achieved by OSA and MI, a large portion of individuals did demonstrate different performances, which can be clearly seen in Fig. 2C.

Fig. 2.

BCI behavioral performance of online cursor control via the OSA modulation and the MI modulation in terms of PVC. The performance accuracies of OSA are shown in solid bars and those for MI tasks are shown in striped bars. A) Comparison of accuracy for left vs right 1D control task. B) Comparison of accuracy for up vs down 1D control task. C) Comparison of accuracy for 2D control task. In the bottom panel, the individual performance for each subject is shown on the left side of the dashed line and the group average performance for each task is displayed on the right side of the dashed line. Subject IDs are sorted according to the difference in performance between the two tasks and are color coded; the color bar shows the percentage difference between OSA and MI.

B. BCI Behavioral Performance of 3D Control Across Sessions

Sixteen of the twenty-one subjects qualified to participate in the 3 sessions of 8 target 3D BCI control (Fig. 3A). Of these sixteen subjects, five demonstrated that OSA modulation was suitable for controlling movement in the frontal plane while concurrently using MI for the 3rd dimensional control (Group One), while the remaining eleven subjects demonstrated UD control via OSA and horizontal plane control via MI (Group Two). Both paradigms performed control in an identical workspace, where targets were placed on the frontal and rear planes of a unit cube centered on the cursor’s starting position. The group average performances for each session of 3D Group One (N=5) were 47.5±5.0%, 46.5±4.8% and 58.4±7.1%, and (N=11) 52.5±7.1%, 60.0±5.9% and 59.1±5.6% in 3D Group Two. The average of the two groups in the total of sixteen subjects for the 8 target 3D control task is shown in Fig. 3C. The average performance and SEM for each session were 50.9±5.3%, 55.8±4.8% and 58.9±4.8%, with random chance being 12.5%. A linear mixed effect model was applied to determine the effect of training across sessions. Statistical analysis revealed that there was a significant improvement in accuracy from session one to session three (p = 0.048) after Tukey’s correction for multiple comparison. Nine out of the sixteen subjects (1/5 3D Group One , 8/11 3D Group Two) were able to finish two additional sessions of 3D control in 12 target tasks (Fig. 3B). Their average performance and SEM for the two sessions were 51.5±6.9% and 50.5±7.1%, respectively (Fig. 3D). No significant difference of accuracy was found between the two sessions. Note that with the increase in the number of targets in the 12 target task, the random chance level fell to 8.3%.

Fig. 3.

BCI behavioral performance for online 3D cursor control via the combination of OSA and MI modulation in terms of PVC. A) A scene of the 8 target 3D cursor control task. The highlighted bar indicated the target to hit. B) A scene of the 12 target 3D cursor control task where the highlighted bar indicated the target to hit. C) BCI accuracy of 3D control across the three sessions with the dashed green line indicating random chance (12.5%). D) BCI accuracy of 3D control in the 12 target task across the two sessions with the dashed green line indicating random chance (8.3%).

C. Comparison of ITR among 1D, 2D and 3D Control

The classification accuracy revealed that subjects succeeded in each modalities individually and in combination, although overall classification accuracy decreased with increased task complexity (2D and 3D control). Besides the apparent flexibility the subjects gained through higher degrees of freedom, i.e. 2D and 3D control, the efficiency of 2D and 3D control was examined through the calculation of ITR. The group averaged ITR of the subset of nine subjects who completed all tasks can be seen in Fig. 4A. The highest group average ITR across one and two dimensional control was achieved via MI (1D control 15.1±1.7 (SEM) bits/min 2D control 21.2±3.4 bits/min). The group average ITR and SEM of 3D control for the 8 target and 12 target tasks were 24.1±1.5 bits/min and 29.7±1.6 bits/min, respectively. A linear mixed effect model was applied to evaluate both the effects of the control strategy and the number of dimensions on the ITR. The statistical analysis (after correction for multiple comparisons) in Fig. 4B showed that there was a significantly higher ITR for the complex 3D control task (8 target and 12 target) compared to both the 1D and 2D tasks. There was no significant difference in ITR between the 3D 8 target task and the 12 target task, although the ITR was on average higher in the 3D 12 target task.

Fig. 4.

Comparison of ITR in a subset of nine subjects for all of the experimental conditions. The group average of ITR for each condition is displayed by the height of the bar with the SEM being overlaid. The p-values for statistical analysis by a linear mixed effect model are shown in the subplot at the bottom. A post hoc Tukey’s test was used to correct for multiple comparisons.

D. Comparison of Electrophysiology with and without Neuronal Feedback

The R-value topography can be used to measure the task-related modulation and is calculated by regressing the EEG alpha power during the control task against the target labels/locations, allowing us to analyze the underlying electrophysiology of the paired control tasks with and without neuronal feedback. The analysis for LR control (first row of Fig. 5A and Fig. 5B) and UD control (second row of Fig. 5A and Fig. 5B) was performed for the OSA modulation (left column of Fig. 5A and Fig. 5B) and the MI modulation (right column of Fig. 5A and Fig. 5B) separately. All of the trials were included in the calculation of the R-values shown in figure 5. The topography of R-values before receiving any feedback is displayed in Fig. 5A and its counterpart after feedback is shown in Fig. 5B. There were obvious focal regions of modulation across electrodes covering the parietal-occipital cortical regions for OSA modulation and bilateral motor cortical regions for MI modulation. An apparent difference of modulation strength was also observed before and after the feedback was given. Quantitative comparisons of the electrodes used for control (OSA: P3, P4, Pz and Poz; MI: C3, C4) were performed for both control dimensions during the corresponding modulation strategy. Paired t-tests were used to evaluate whether there exists a change in modulation at each relevant electrode between the two conditions, before and after the feedback was given. There was a significant difference in the P3 (p = 9.3×10⁻⁴), but not P4 electrodes (p = 0.1) between conditions with and without feedback for LR control via OSA modulation (see Fig. 5C); there were significant differences in the PZ (p = 8.8×10⁻⁶) and POZ (p = 5.5×10⁻⁶) electrodes between the with and without feedback conditions for the UD control task during OSA modulation (see Fig. 5E). In contrast, statistical analysis revealed that during MI there were no significant differences in the C3 (p = 0.60) and C4 (p = 0.18) electrodes between the with and without feedback conditions for LR control (see Fig. 5D) and that there were significant differences in the C3 (p = 0.1×10⁻³) and C4 (0.1×10⁻³) electrodes between the with and without feedback conditions for the UD control task (see Fig. 5F).

Fig. 5.

The R topography map of OSA modulation (the first column of each panel) versus MI modulation (the second column of each panel) without feedback in subfigure (A) and with feedback in subfigure (B). R values were calculated using all trials. The first row of each subfigure displays the R topography of the left versus right control task; the second row of each subfigure shows the R topography for the up versus down control task. C) The statistical comparison of R-values at the P3 and P4 electrodes during left versus right control via OSA modulation, D) at the C3 and C4 electrodes during left versus right control via MI modulation, E) at the Pz and Poz electrodes during up versus down control via OSA, and F) at the C3 and C4 electrodes during the up versus down control via MI modulation.

A subset of the trials in which correct targets were hit was used in the calculation of the R-values in figure 6, and demonstrated similar results to figure 5. Quantitative comparisons in the P3, P4, Pz and Poz electrodes were performed for both control dimensions during OSA modulation, since these electrodes were used for the online cursor control. Similarly, quantitative analysis of the C3 and C4 electrodes was performed for both the LR and UD control conditions during MI modulation. Note that the topography shown in Fig. 6 is similar to the topography in figure 5, with much stronger R-values being found when only hit trials were used, compared to the use of all trials in figure 5. There was a significant difference in the P3 (p = 4.8×10⁻⁴) and P4 electrodes (p = 3.9×10⁻⁴) between conditions with and without feedback for the LR control via OSA modulation (see figure 6C). There was a significant difference in the PZ (p = 1.7×10⁻¹²) and POZ (p = 2.9×10⁻¹³) electrodes between conditions with and without feedback for the UD control via OSA modulation (see figure 6E). Similarly, the statistical analysis revealed that there was a significant difference in the C4 (p = 8.0×10⁻³) electrode but not the C3 electrode between the conditions with and without feedback for the LR control via MI modulation (see figure 6D) and that there was a significant difference in both the C3 (p = 5.3×10⁻⁸) and C4 (1.8×10⁻⁸) electrodes between the conditions with and without feedback for the UD control via MI modulation (see figure 6F).

Fig. 6.

R topography map of the OSA modulation (the first column of each panel) versus the MI modulation (the second column of each panel) without any feedback in subfigure A) and with feedback in subfigure B). Calculation is based on the subset of the trials in which targets were correctly hit. The first row of each subfigure displays the R topography of left versus right control task; the second row of each subfigure shows the R topography of the up versus down control task. C) Statistical comparison of R-value at the P3 and P4 electrodes of OSA modulation during the left versus right control, D) at the C3 and C4 electrodes of MI modulation during the left versus right control, E) at the Pz and Poz electrodes of OSA modulation during the up versus down control, and F) at the C3 and C4 electrodes of MI modulation during the up versus down control.

E. Confusion matrices of different strategies for the 3D control tasks

The confusion matrices for the two groups using different strategies (illustrated in Figure 7A–7B) were calculated separately in order to determine if both strategies were effective in producing 3D control (Figure 7). The group level confusion matrices for group one and group two were displayed in Figure 7C–7D, respectively and the corresponding standard deviation of the confusion matrices is shown in Figure 7E – 7F, respectively, for group one and group two. These group level confusion matrices provide details regarding the occurrence of false positives and negatives for each target type. This analysis was performed in order to rule out the possibility of highly proficient 2D control being the underlying mechanism of the behavioral results. Instead, the results demonstrate that there were no global preferences to individual planes, given a control strategy, and that both strategies were suitable for 3D control. Six representative individual confusion matrices (4/11 from group one, 2/5 from group two) are displayed in Figure 8. While no clear patterns of group level target preference were found between or within control strategies (Figure 7C–7F), individual subjects demonstrated notable preferences (Figure 8). These notable differences in subject specific preference highly influence the global map, leading to non-obvious or easily interpreted patterns. We suspect that these patterns arise from the high variance in our measurements, due to low sample numbers, and the innate variability [31] of BCI (Figure 7E–7F).

Fig. 7.

Confusion matrices for the 3D control tasks. A) The target location and the control strategy for the 5 subjects who preferred to use OSA modulation to move the cursor in the frontal plane (parallel to the screen) and MI modulation to move the cursor in and out of the vertical plane. The workspace was rotated for the sake of visualization and subjects viewed the workspace from the top down perspective. B) The target location and the control strategy for the 11 subjects who preferred to use the MI modulation to move the cursor in the horizontal plane and the OSA modulation to move the cursor along the vertical axis. A similar rotation was applied for visualization with the subjects also viewing the workspace from the top down perspective. C) Group level confusion matrices (colorbar unit: percentage) for the 5 subjects who use the strategy illustrated in A). D) Group level confusion matrices for the 11 subjects who use the strategy displayed in B). The true target is indicated by the row and the predicted target by the column. E) Standard deviation of confusion matrices for the 5 subjects. F) Standard deviation of confusion matrices for the 11 subjects.

Fig. 8.

Individual confusion matrices (color bar unit: percentage) for the 3D control tasks. A)–D) Four examples of individual confusion matrices among the 11 subjects who use the strategies in the figure 7B. E)–F) Two examples of individual confusion matrices among the 5 subjects who use the strategies in the figure 7A.

F. Average cursor trajectory

An example of the average cursor movement trajectories for a particular subject was shown in figure 9. This example corresponds to the subject in figure 8A, one of the better 3D control performers. The average trajectories showed a relatively linear path from the starting point (origin) to the middle point of the targets. Note that the cursor is a sphere with a diameter of 10% of the workspace, thus the ellipsoids which display the estimated distribution of the cursor center’s endpoint may not be in contact with the targets. The average trajectories of this particular example demonstrated that online cursor control was performed in three dimensions continuously, and was not the product of sequential 2D and 1D manipulations.

Fig. 9.

Average cursor movement trajectories for a particular subject. The virtual cubic workspace was rotated for easier visualization of all of the trajectories. The colored bars show the targets in the workspace and the ellipsoids display the estimated distribution of the end points of cursor during all of the hit trials. The movement path was derived by averaging all of the hit trials.

IV. DISSCUSSION

A. Comparison of performance between conventional paradigms and the proposed OSA

In the current study, we demonstrated a new modulation modality, OSA modulation, which produces a similar performance level in terms of the classification accuracy compared to conventional MI modulation. Compared to previous studies using covert (visuo)spatial attention (CSA), our online classification accuracy results were marginally higher than the average offline results of both 69% for 1D and 41% for 2D tasks in a group of 15 subjects [40] and were comparable or higher than other studies [41]–[43]. It is important to note that most of the studies previously reported only include offline analysis, and none demonstrated online 2D control. Although a valid form of analysis, performing classification offline allows the researchers ample opportunity to potentially fit their data in a non-generalizable fashion, and to employ resources or methods which are not accessible during rapid online decoding. In our current iteration of the OSA modality, subjects were allowed to shift their gaze towards targets as well as covertly pay attention to the movement of the cursor, in order to hit the correct target. This is slightly different than the previous CSA studies, where subjects were instructed to focus on the center while covertly attending to a peripheral target. The flexibility of overtly shifting their gaze allows subjects to naturally and comfortably deploy their spatial attention along with the attended direction. Through this flexible shifting, subjects were capable of easily combining OSA and MI modulation. There were shown to be no group level statistical differences between 1D and 2D control using either modality, although some individuals demonstrated superior modulation abilities in a specific modality. As an estimated 15%−30% of the population is incapable of producing the conventional MI based BCI control signal [52], the introduction of OSA modulation potentially provides an effective alternative for BCI control. Moreover, OSA modulation utilizes the parietal-occipital cortex, unlike MI’s use of the sensorimotor cortex, providing the possibility to extend BCI control beyond 2D.

B. 3D control strategy was individualized according to subjects’ performance preference but no difference was found between strategies

In the 3D experiments, a combination of OSA and MI modulation was used for online control. We found that two particular combinations worked best and were accepted by the participants. Subjects with superior performance using MI, compared to OSA, preferred to use MI and OSA for the horizontal plane and vertical axes respectively. While subjects who demonstrated superior performance via OSA, compared to MI modulation, preferred to use OSA to control the cursor in the frontal (parallel to the screen) plane and MI (both hands versus relax) to control the cursor’s depth (in/out of the frontal plane). We utilized identical target positions for both groups, regardless of the control strategy being employed. By using consistent target locations, we were able to further demonstrate the lack of preferred control dimensions within either control strategy in the same experiment. This is further exemplified in the confusion matrices, which reveal that on average both strategies used in this study produced successful 3D control. Although the group level averages were similar, there were noticeable individual differences among the subjects’ control abilities. Some subjects, like the example in figure 8A, demonstrated superior performance for all target locations, while others preferred specific axes. Although both the MI and OSA modulations consume attention resources, the 3D control results revealed that subjects could effectively deploy their attentional resources simultaneously and properly in order to successfully complete the tasks. This simultaneous deployment of attentional resources corroborates our previous study, in which we explored cognitive flexibility through simultaneous SSVEP and MI modulation for BCI control [53].

More recently, wearable BCI systems have become more readily available due to the rapid improvement of dry EEG electrodes [13,14] and wireless, compact EEG systems. These wearable BCI systems have been used in a diverse set of applications, ranging from the evaluation of driver vigilance [54] to robotic exoskeletons [16]. The 3D control paradigm proposed in this study uses a sparse electrode configuration for online control, an advantage that allows for its simple implementation within wearable BCI systems. Not only can previously established algorithms, such as the one presented here, be implemented in wearable systems, given the increased comfort and mobility offered to BCI users by these systems, there is no doubt that they will help attract more BCI users, helping to further the development of BCI technology.

C. New paradigm of 3D control does not require extensive training and improve ITR significantly

Prior to our work, there was only one study of noninvasive 3D BCI control based on endogenous modulation. In it, McFarland et al. proposed to use the imagination of movement of both hands and feet to perform EEG based 3D control [32]. They designed a center out task with eight targets located in the corners of a cubic workspace, where only one target was displayed per task. Each subject was asked to move a cursor, which appeared in the center, to the displayed target at one of the corners within a 15s time limit. If the subject was unable to contact the target by the end of the 15 seconds, the trial was recorded as a miss. Unlike the work presented here, if the cursor moved to other target locations, it did not result in a miss. In their work, they studied four subjects (one naïve BCI subject) who practiced 1–4 sessions of 1D control, 10–12 sessions of 2D control and 21–42 sessions of 3D control. Initially these subjects completed about 20%−60% of the 3D trials, improving to 60%−90% following extensive training. Our current study demonstrated that a group of 16 subjects (11 BCI naïve subjects) could complete three sessions of 3D 8 target tasks with starting and ending group accuracies of 50% and 59%. Unlike the previous work done by McFarland and colleagues, these subjects only performed three sessions of 1D and 2D BCI training before moving on to the 3D control task. Note that the two accuracies are not directly comparable since the first accuracy [32] represented the proportion of completed trials, while the accuracy in the present study was PVC, where contact with incorrect targets results in misses. This difference increases the difficulty of the task in this study and is also closer to daily life application. Furthermore, the subjects in our study underwent substantially less training in the 3D tasks (3–5 sessions versus 21–42), and still demonstrated high levels of accuracy and control, further suggesting that our proposed control strategy is useful in practical situations. The efficiency of the BCI system was investigated through the evaluation of the ITR, which not only depends on the classification accuracy but also correlates with the number of hits and speed in each run. Previous studies [40],[50],[55] have reported the ITR for 1D (two classes) and 2D (four classes) cursor control tasks or up to five classe offline analysis, which produced ITRs of 4.9 bits/min – 14.1 bits/min for 1D tasks [40],[55], ITRs of 11.0 bits/min – 20 bits/min for 2D tasks [50],[55], and ITRs of 0.42 bits/trial – 0.81 bits/trial for the five class individual offline analysis [50]. It is difficult to directly compare ITRs across these studies since some of these works only reported the peak ITR for individuals and the experimental paradigms were different, potentially affecting the evaluation of ITR. In our work, the average ITR for 1D control is 11.1 bits/min (0.68bits/trial), which is comparable to the reported maximum individual ITR across all tasks (two to five class offline analysis) [50]. This verifies that the ITR for 1D control in this study can serve as a strong baseline, and that the improvement of ITR through increased dimensionality is not a trivial result due to weak starting accuracy. Moreover, all of the 1D, 2D and 3D tasks in this study were performed and evaluated in the same paradigm, which allows for a more fair comparison between different control tasks.

D. Online control increase R-value significantly

Event-related (de)synchronization (ERD/S) has been suggested to be a product of specific cognitive processes which might be related to the selective attention of multiple neuronal systems, such as the motor [56] and visual systems [38],[39]. The modulation of mu rhythms was topographically quantified via the regression analysis R-value displayed in Fig. 5. A clear pattern or cluster of active electrodes between the compared tasks usually suggests the discriminability of the designed tasks and the possibility of online control. The R-value topography, without any feedback, demonstrates similar results to the previous studies which employ covert spatial attention [40],[41]. Here, we showed that the OSA modulation task produces similar but much stronger parietal-occipital activity, especially after providing online feedback. The statistical analysis of the R-values of the corresponding control electrodes (P3/P4 LR OSA; Pz/Poz UD OSA; C3/C4 LR/UD MI) revealed that following the presentation of feedback, there was a significant increase in the modulation of activity. This further validates that online feedback could greatly promote the cognitive processes [57] and might be essential for BCI control [58],[59].

CONCLUSION

In this work we propose a new OSA task which utilizes the endogenous modulation of visuospatial attention and demonstrate similar performance to conventional MI BCI control. The OSA task, without feedback, produces activity patterns similar to those presented in previous studies of CSA. These patterns which are focused around parietal-occipital cortices are further strengthened through the presentation of feedback. This stronger cognitive process coincides with the successful modulation of the brain rhythms and the completion of the tasks. A substantial portion of the subjects further demonstrated that the combination of the two strategies (MI and OSA) allows for the control of a virtual cursor in 3D space through the completion of center out tasks with significantly higher accuracy than chance. Furthermore, subjects showed significant performance improvement across the three sessions of 8 target 3D tasks. Note that, nine of these subjects further demonstrated robust control in a twelve target 3D control task, resulting in a group average information transfer rate of 29.7±1.6 bits/min, which significantly outperformed the lower dimensional 1D and 2D control task. The successful completion of a 3D task through the combination of multiple control strategies corroborates our previous findings that subjects have the cognitive flexibility to simultaneously deploy their attention in two different cognitive tasks. By taking advantage of this cognitive flexibility, future work could improve and expand upon the efficiency and dimensionality of BCI control, as well as potentially shed light on the study of cognitive processes, such as attention, through BCI.

Significance:

Through the combination of the two strategies (MI and OSA), a substantial portion of the recruited subjects were capable of robustly controlling a virtual cursor in 3D space. The proposed novel approach could broaden the dimensionality of BCI control and shorten the training time.