Spatial-temporal aspects of continuous EEG-based neurorobotic control

Abstract

Objective:

The goal of this work is to identify the spatio-temporal facets of state-of-the-art electroencephalography (EEG)-based continuous neurorobotics that need to be addressed, prior to deployment in practical applications at home and in the clinic.

Approach:

Nine healthy human subjects participated in five sessions of one-dimensional (1D) horizontal (LR), 1D vertical (UD) and two-dimensional (2D) neural tracking from EEG. Users controlled a robotic arm and virtual cursor to continuously track a Gaussian random motion target using EEG sensorimotor rhythm modulation via motor imagery (MI) commands. Continuous control quality was analyzed in the temporal and spatial domains separately.

Main Results:

Axis-specific errors during 2D tasks were significantly larger than during 1D counterparts. Fatigue rates were larger for control tasks with higher cognitive demand (LR, left- and right-hand MI) compared to those with lower cognitive demand (UD, both hands MI and rest). Additionally robotic arm and virtual cursor control exhibited equal tracking error during all tasks. However, further spatial error analysis of 2D control revealed a significant reduction in tracking quality that was dependent on the visual interference of the physical device. In fact, robotic arm performance was significantly greater than that of virtual cursor control when the users’ sightlines were not obstructed.

Significance:

This work emphasizes the need for practical interfaces to be designed around real-world tasks of increased complexity. Here, the dependence of control quality on cognitive task demand emphasizes the need for decoders that facilitate the translation of 1D task mastery to 2D control. When device footprint was accounted for, the introduction of a physical robotic arm improved control quality, likely due to increased user engagement. In general, this work demonstrates the need to consider both the physical footprint of devices, the complexity of training tasks, and the synergy of control strategies during the development of neurorobotic control.

Introduction

Naturalistic robotic arm control, for both commercial and clinical purposes, has long been a goal of brain computer interface (BCI) research [1]–[8]. While pursuing invasive methods of interfacing with the brain, such as implantable electrode grids [9]–[11] and subdural electrode arrays [12]–[14], has resulted in substantial progress towards thought controlled devices, these techniques require skilled surgeons, are expensive, and are only suitable for a small number of individuals [15]. Unlike their invasive counterparts, neural signals used for noninvasive BCIs are typically hindered by the limited spatial resolution and low signal-to-noise ratio [16]–[18]. In particular, the low temporal resolution of noninvasive neuroimaging modalities that measure hemodynamic signals has driven the popularity of electrical recording methods such as electroencephalography (EEG) for noninvasive BCI. In addition to measuring fundamentally faster processes on the order of milliseconds, EEG offers moderate spatial resolution (centimeter scale) [19], total scalp coverage, and portability at an economical price.

Given the low spatial resolution of neural activity detected via EEG, traditional BCI tasks remain fairly simple [20], with the majority consisting of offline analyses or sparse online feedback during relatively simple behaviors [21]–[23]. While exceptions do exist, namely various tasks controlled via motor imagery (MI) [6], [20], [24]–[29], steady-state visually evoked potentials [30]–[33], overt spatial attentional control [34], or a combination thereof, even these embodiments exist fairly far from any practical, real world scenario in which neurorobotic control can be used.

To address these issues, we recently developed a novel continuous pursuit (CP) task paired with an online electrophysiological source imaging (ESI) decoder to control a virtual cursor and robotic arm in a practical manner useful for daily activities utilizing MI (Figure 1a–c) [35]. By focusing on sensorimotor rhythm modulation via MI, we allowed individuals to have complete continuous control over the movement of a robotic arm and virtual cursor, unlike methods which employ queue presentation-based tasks (for example steady-state visually evoked potential or P300 BCIs). In our previous work, we systematically demonstrated that this approach improved BCI performance compared to traditional paradigms based on center-out tasks and sensor-based decoding from both a user and machine perspective. Importantly, in a small group of users, we showed that there was no significant difference between the quality of control of a virtual cursor and a physical robotic arm during CP, with minimal and equal training times. Nevertheless, while control quality appeared quantitatively equivalent, subjects reported that the presence of the robotic arm often obscured the target, resulting in a subjectively more difficult task. As the integration of BCI-controlled devices into daily life will depend on the “neural ergonomics” of these systems, it is necessary to examine in further detail how the use of a real-life device affects performance. These human-device interactions can then help inform specific design requirements of BCI systems, such as the physical placement of the device relative to the user, the optimal duration of prolonged control, and many others that must be considered for practical use.

Figure 1. Concept Design.

(a-b) Experiments consisted of a variable composition of 24, 60 second trials of continuous tracking. Block randomization across interface (robot or virtual cursor) was conducted between the first and second 12 trials performed during each of the five sessions. (c) Real-time estimates of cortical activity were decoded using noninvasive EEG source imaging relying on left and right hand motor imagery for ipsilateral movement and both handed motor imagery and rest for upward and downward movement, respectively. (d-e) Temporal analysis consisted of fitting linear fatigue rates to trisected trials. (f-g) Spatial analysis consisted of either vertically or horizontally bisecting the active workspace and determining error rates as a function of cursor position, target position, and cursor-target error vector direction.

Inspired by this concept, we aimed to identify the spatial and temporal behavioral tendencies that manifest during sustained sensory-motor rhythm (SMR) modulation (Figure 1). More specifically, by increasing our previously studied population by 50% and analyzing the dependence of performance on space, time, and control interface (robotic arm or virtual cursor) we identified previously unknown confounding factors within discrete control strategies that arose when applied to continuous SMR modulation. Standard metrics consisted of the mean squared error (MSE) and tracking correlation (linear correlation coefficient). In our previous work, we established these metrics as capable of both assessing continuous tracking performance and the dynamic skill acquisition thereof. Temporal analyses were designed to investigate the dynamics of prolonged SMR modulation, namely how proficiency changes as a function of continuous control time, a feature not previously known due to the short, discrete nature of classical SMR BCI tasks. Finally, the spatial analysis was intended to investigate the real-world implications of introducing a physical system to a noninvasive SMR BCI. As the eventual goal of all BCI research is its practical deployment at home and in the clinic, we take advantage of this unique dataset to identify considerations that need to be made in future designs.

Methods

Experimental Protocol

In total, nine subjects (average age 25.6 +/− 5.8, 8 right-handed, 6 female) participated in the current study. All participants provided written informed consent to a protocol approved by the institutional review board of Carnegie Mellon University. Briefly, subjects who had previously excelled in traditional SMR-based one-dimensional (1D) and two-dimensional (2D) discrete trial center-out BCI control participated in five sessions of CP virtual cursor and robotic arm control (Figure 1a). Each session consisted of 24 trials, with the robotic arm and virtual cursor control presented in block-randomized order (12 trials per block). Each block was further broken down into Left-Right (LR), Up-Down (UD) and 2D control tasks. Sessions 1–2 consisted only of LR and UD trials (6 per interface), while sessions 3–5 contained LR (3), UD (3), and 2D (6) trials (Figure 1a). At the beginning of each session, subjects performed one run of both LR and UD MI without feedback in which they simply performed a cued MI task for a set time. These MI without feedback runs were later used to statistically identify features for online control. EEG for all of these runs was acquired using a 128-channels Biosemi system (BioSemi, Amsterdam, The Netherlands).

Continuous Pursuit Task

During each trial, subjects tracked the continuously varying movement of a target for 60 seconds using either a virtual cursor or a robotic arm via 1D or 2D MI (Figure 1b–c) [35]. The target and virtual cursor were presented via BCI2000 [36], BcPy2000 [37], and custom python code on a computer screen measuring 97 cm wide and 56.4 cm tall. For robotic arm control, the virtual cursor position was linearly translated to robotic arm position in a physical space constrained by the edges of a square workspace (56cm × 56cm) in front of the computer screen. At the beginning of each trial, the target and cursor/robotic arm were both located in the center of the workspace and the position of each varied across the 60 seconds according to the applied random forces and user control signals, respectively.

Target motion was defined by a random Gaussian process in either one or two dimensions, as previously reported [35]. In short, a zero-mean and unit-variance force vector was applied to the target in a random direction along the relevant axes of control. For example, during the 2D task, both a horizontal and vertical random force were independently and simultaneously applied to the target. To achieve smooth target motion, a drag and friction force were applied as well. Due to the physical constraints of robotic arm movement and the random target dynamics, boundary repulsion behaviors were implemented to prevent target stagnation in the corners or along the edges of the workspace. Here, if the target entered the outermost 10% of the workspace on either axis, all outward applied force-vectors were scaled and inverted, directing the target’s motion away from the boundaries. As the velocity of the cursor/robotic arm and target were dictated by two different mechanisms, a scaling factor was applied to the user-derived control signal to ensure overlapping ranges of velocities between the cursor/robotic arm and target.

Cursor/Robotic Arm Control Signals

Unlike the force model used to drive target movement, the EEG-derived control signals applied to the cursor directly dictated its axial velocities. In order to move the cursor/arm to the left and right, participants were instructed to imagine opening and closing their left and right hands, respectively. For vertical control, participants were instructed to imagine opening and closing both hands simultaneously to move the cursor/arm upwards and to perform active resting to move the cursor/arm downwards. Subject-specific EEG features representing these motor-related mental states were extracted from two runs of MI based BCI without feedback performed at the beginning of each session. Briefly, the EEG from each run was first projected onto the anatomically and functionally parcellated cortical surface of the Colin27 template brain using a weighted-minimum-norm estimate in the frequency domain. Cortical parcels within the sensorimotor cortex were defined for each participant using the multivariate source prelocalization method[38]. A stepwise linear regression approach with an inclusion and exclusion criteria of p < 0.01 then identified sensorimotor parcels (and their weights [−1 or +1]) that best separated the relevant tasks (left vs. right hand MI and both hand MI vs. rest) within the alpha band (8–13Hz) for each dimension. Online control signals were derived via online ESI in the cortical source space using these features and weights every 100 ms, using the previous 250 ms of data (real-time) [35], [39]. The resulting control signals were then temporally z-scored utilizing the previous 30 seconds of recording history. Finally, these normalized control signals were then applied as axis specific directional velocities for the cursor/robotic arm. Detailed methods regarding the CP task and control signals used are provided in the original publication describing this work [35].

Offline Analysis

Standard Metrics

Tracking quality was assessed using both mean-squared error (MSE, Eq. 1) and the linear correlation (ρ, Eq 2.) between the target and the cursor/arm positions. MSE was computed according to Eq. 1 for the horizontal performance assessment where T_x,i, and C_x,i indicate the i-th target and cursor X-positions, respectively. Correlation was computed according to Eq. 2 where ${\overline{T}}_{\text{x}}$ and ${\overline{C}}_{\text{z}}$ indicate the average of all target and cursor positions across a trial for a given axis. N corresponds to the number of measurements made in a trial. The same equations were applied to the Y-positions for the vertical performance assessment.

$$MS{E}_x=\frac{1}{N}{\sum }_{i=1}^N{\left(T_{x,i}-C_{x,i}\right)}^2$$ Eq. 1

$${\rho }_x=\frac{{\sum }_{i=1}^N\left(T_{x,i}-\overline{T_x}\right)\left(C_{x,i}-\overline{C_x}\right)}{\sqrt{{\sum }_{i=1}^N{\left(T_{x,i}-\overline{T_x}\right)}^2}\sqrt{{\sum }_{i=1}^N{\left(C_{x,i}-\overline{C_x}\right)}^2}}$$ Eq. 2

All positions were normalized with respect to screen size in the range of -½ to ½. Data were analyzed using custom MATLAB scripts, EEGLAB [40], and R. Metrics for each subject were averaged across all trials for each session of a single task.

Temporal Analyses

The within-trial effects of control duration were evaluated by estimating the rate of change in MSE over the course of a trial. To accomplish this, MSE was calculated for three 20 second, nonoverlapping segments (thirds) of each trial and was robustly regressed against its corresponding ordinal position (1–3) for each subject, session, task and interface independently (Figure 1d–e). Instead of averaging all trials for a single MSE value, as is done in the standard analysis, the slope was fit to all trials simultaneously. Given that a decrease in performance over a single trial cannot be observed in short, segmented tasks, we hypothesized and present data here suggesting that the MSE rate-of-change can serve as a single parameter estimate for subject fatigue. We specifically examined the differences in axis-specific fatigue rates for tasks of high and low complexity (e.g. 2D_LR vs 1D_LR control), tasks involving physical or virtual end effectors (robotic arm vs. virtual cursor), and for tasks involving more vs fewer active cognitive elements (LR: two MI tasks vs UD: one MI task).

Spatial Analyses

The within-trial effects of spatial coordinates were investigated by examining the relationship between the target and robotic arm location and the corresponding propensity and magnitude of errors. Unlike standard center-out tasks, the CP task results in unique combinations of target-cursor position pairings. This enables a more in-depth evaluation of how a user’s physical environment affects BCI performance. As previously mentioned, investigating spatial confounds of CP stems from observations and subject reports regarding tracking difficulty when the robotic arm obscures the target. Such a situation can easily arise during CP as the robotic arm moves throughout the user’s visual field and can hide the target behind it. Nevertheless, the random motion of the target and the relatively small sample size in terms of total experimental trials (n = 1080 across the study cohort) led to a non-uniform sampling of error vectors at each workspace location. Therefore, it is difficult to comprehensively map the effect of every target-cursor/robotic arm combination to user performance. We therefore aimed to simplify this analysis to draw more pragmatic conclusions for the implementation of such a BCI paradigm in real life. To do so, we analyzed the MSE in relation to the target and cursor’s general location on the left vs right or top vs bottom halves of the workspace (Figure 1f–g).

As these limited measurements serve as approximations of the true error distributions, to assess the potential impact of a physical device we propose a probabilistic perspective on assessing error types. Since the robotic arm is anchored to the right of the user, there is a high probability that the target will be obscured when the robotic arm reaches to the left half of the workspace compared to the right half. We hypothesize that the occurrence of this phenomenon will alter user behavior and produce spatially biased error during CP BCI.

Statistical Analysis

All statistical analysis was conducted in either R or MATLAB in a two-tiered fashion. Initial investigations were performed using multi-factor ANOVAs with all available factors and data subsets. In cases where there were potential confounding factors not directly described in the measurements, single covariant, multiple factor ANCOVAs were used. Covariant factors and primary variables were scaled using R’s inbuilt rank preserving approach (scale). Individual comparisons of corrected variables within the main effects and interactions of ANOVAs and ANCOVAs were assessed using two-tailed t-tests and were corrected for multiple comparisons using Tukey’s HSD test. During covariant analysis, samples which exceeded two standard deviations of a chi-squared test were removed.

Results

Standard Analysis

Whole-trial performance metrics revealed no statistical differences between virtual cursor and robotic arm control (two-way ANOVA) (Figure 2, Supplementary Table ST6–9). A significant main effect of session number was found for the correlation coefficient (Figure 2 a–b) during 1D control (F(4,155) = 2.596, p = 0.038, η_p² = 0.06), and post hoc analysis using Tukey’s HSD test indicated superior performance in the fourth session compared to the first. These results support the concept that BCI skill can easily transition from a simple virtual interface to a real-life device and can improve across multiple sessions.

Figure 2. Standard Analysis.

(a-b) 1D Session wise, axis specific tracking correlation coefficient. (c-d) 2D Session wise, axis specific tracking correlation coefficient. (e, f) Session averaged axis specific MSE for both 1D and 2D tasks. (g,h) Session averaged axis specific tracking correlation for both 1D and 2D tasks. Statistical Markers: Main effect of ANOVA analysis (#), t-test ($).

Direct comparisons between 1D and 2D control were only possible for the final three sessions of experimentation due to the experimental design (Figure 1a). Axis-specific MSE (F(1,192) = 7.767, p = 0.006, η_p² = 0.039, Figure 2 e–f) and correlation coefficient values (F(1,191) = 4.407, p = 0.037, η_p² = 0.023, Figure 2 g–h) were significantly larger and smaller, respectively, for the 2D control task compared to the 1D tasks. Notably, the axis-specific MSE components of 2D control were larger than the 1D counterparts across all four task combinations (Robotic Arm, Virtual Cursor, Horizontal Control, and Vertical Control), suggesting that the increased complexity in the 2D task, compared to the 1D tasks, led to lower performance. Unlike total MSE, the comparisons performed on axis-specific errors are drawn from identical distributions. Therefore, the reduction in performance between the 1D and 2D tasks observed here is likely due to the 2D task being more difficult; had these numbers been statistically equivalent, improvement in the 1D tasks would likely directly transfer to 2D control. The evidence here suggests that superior performance in the 2D task requires exposure to the 2D environment and cannot simply transfer from 1D mastery alone.

In a secondary analysis of session-averaged tracking correlations and MSE using a simple paired t-test, we found significantly lower tracking correlation values during robotic arm control compared to virtual cursor tracking (paired t-test, p = 0.016) across both axes of control during the 2D task (Figure 2 g–h). This factor was initially obscured by the potentially confounding subgroups analyzed in the multifactor ANOVA and the fact that only three of the five 1D control sessions could be compared to those of 2D control (potential instability across sessions) (Figure 2f–g). We hypothesized, and demonstrate in subsequent spatial analysis, that this reduction in tracking correlation was not related to a reduction in control level, but rather was likely due to subjects moving the robotic arm independently of the target position in order to prevent the interface from obscuring the target.

Spatial Analysis

The introduction of a non-transparent robotic arm into an active workspace can obscure the target position and general environment during active control. We therefore hypothesized that subpopulations of errors existed depending on the relative position of the target and cursor/arm. It should be noted that the base of the robotic arm was always mounted to the right of the user and extended across the visual field from the bottom right direction (Figure 1c). More specifically, we hypothesize that the likelihood of target obfuscation, which is increased when both the target and robotic arm are on the left half of the workspace, can alter behavioral patterns and lead to quantitatively inferior tracking quality. This initial hypothesis was driven by experimental observations and unprompted subject statements, in which subjects self-reported difficulties observing the target during specific robotic arm control situations, particularly when the target was on the left-hand side of the workspace. To address whether the presence of the physical apparatus produced errors that could not be observed using the previously described standard analysis, errors were inspected after bisecting the workspace horizontally and vertically. By doing so, we were able to examine the vector orientation of cursor-target errors based on the target and cursor positions, i.e. being located in the upper/lower or left/right half of the screen in both the 1D and 2D tasks (Figure 1f–g).

When bisecting the workspace, it was noted that the prevalence of target-cursor location pairings was non-uniform and was heavily correlated with the underlying MSE. These non-uniformities were likely due to random sampling and the effects of user-based control and behavioral biases. An initial ANOVA analysis of the MSE values did not account for this and found numerous significant differences across the various combinations (Supplemental Table ST11–14). When MSE values were corrected for their covariant, we observed both increases and decreases in the sub-group differences.

Horizontal ANCOVA

To determine whether the spatial position of the cursor, target, or the direction of the error vector had any effect on the subjects’ horizontal control quality, the workspace was partitioned into a left and right half (Figure 3a). The density of positions (fraction of possible time spent in a current configuration during control) was highly covariant with horizontal MSE for both the 1D (p < 2e-16, ρ = 0.450) and 2D tasks (p < 2e-16, ρ =0.544) (Supplement Figures S6 and S7: Complex Pairing S3–5: R_cursor² = 0.22, R_robot² = 0.38, 1D Pairing S1–5: R_cursor² = 0.12, R_robot² = 0.29). When accounting for this, a significant interaction between the target and cursor position was found for both the 1D (F(1,453) = 288.377, p < 0.001, η_p² =0.389, Figure 3 b) and 2D (F(1,283) = 168.397, p < 0.001, η_p² = 0.373, Figure 3 c) tasks. This result was anticipated, as a larger MSE is expected when the target and cursor are on opposite halves of the screen compared to when they are on the same half. Additionally, a significant interaction between cursor/robot position and session number was identified (F(2,283) = 4.285, p = 0.015, η_p² = 0.029) during 2D CP. Further inspection of 1D behavior showed an interesting prevalence of interactions between the interface type and session (F(4,453) = 5.12, p < 0.001, η_p² = 0.043) and the interface type and target position (F(1,453) = 6.559, p = 0.011, η_p² =0.014).

Figure 3. Spatial Analysis – Horizontal Bisection.

(a) schematic for types of lateral errors present during continuous control, dashed line indicated midline of workspace. C stands for Cursor, while T stands for Target. (b,c) MSE averaged for shared left position between cursor and target, split cursor and target side, and shared right side positions of cursor and target for 1D and 2D control. (d,e) Horizontal MSE averaged according to cursor position for the 1D and 2D tasks. (f,g) Horizontal MSE averaged according to target position for 1D and 2D tasks. Statistical Markers: Significant interaction (##), Signfiicant Post-Hoc HSD Corrected T-Test (p<0.001 ***)

Post hoc analysis revealed that the 1D interface-session interaction only reached significance for the 1^st session, which was the subject’s first exposure to robotic arm control. In this session, robotic arm control exhibited a significantly lower MSE than virtual cursor control (p<0.001) after correction for the density of segmented positions. This is likely due to the novelty of robotic arm control and increased attention or focus. Additional analysis of the interactions between interface type and target position showed, in direct contradiction to our initial hypothesis, that users displayed only marginally worse robotic arm control, compared to virtual cursor control, under conditions favorable for target obstruction (left-sided targets). Even more surprising was the finding that when the user’s view was unobstructed (right-sided targets), robotic arm control errors were significantly lower than virtual cursor errors (after position density correction, p = 0.030) (Figure 3 f). This evidence lends strong credence to a theory of increased user engagement during robotic arm control.

Vertical ANCOVA

To discern if the robotic interface biased performance during vertical cursor control, the workspace was split into two vertical halves (top and bottom, Figure 4a). In this analysis, only the vertical MSE was accounted for in the 1D and 2D tasks. As in the horizontal analysis, the MSE values from both the 1D and 2D tasks revealed significant interactions among cursor and target position (1D: F(1,448) = 283.605, p<0.001, η_p² = 0.388; 2D: F(1,283) = 156.455, p<0.001, η_p² = 0.356, Figure 4b–c). Both the 1D and 2D tasks also demonstrated a significant main effect of session number (1D: F(4,448) = 2.6, p = 0.036, η_p² = 0.023; 2D: F(2,283) = 6.614, p = 0.002, η_p² = 0.045, see supplemental information for session specific results) on performance. For the 2D case, the error during the 5^th session was significantly lower than during the 4^th (p = 0.013) and was moderately lower than during the 3^rd (p = 0.06).

Figure 4. Spatial Analysis – Vertical Bisection.

(a) schematic for types of lateral errors present during continuous control, dashed line indicated midline of workspace. C stands for Cursor, while T stands for Target. (b,c) MSE averaged for shared bottom-half position between cursor and target, split cursor and target side, and shared top half positions of cursor and target for 1D and 2D control. (d,e) Vertical MSE averaged according to cursor position for the 1D and 2D tasks. (f,g) Vertical MSE averaged according to target position for 1D and 2D tasks. (h,i) Breakdown of all possible subgroups of vertical MSE for the 1D and 2D tasks. Statistical Markers: Main Effect (#), Significant interaction (##), Significant Post-Hoc HSD Corrected T-Test (p<0.05 *, p<0.001 ***)

One-dimensional vertical control further showed a significant main effect of cursor position on MSE (F(1,488) = 13.335, p < 0.001, η_p² = 0.029, Figure 4d), while 2D vertical error revealed a significant interaction between cursor position and error direction (F1,283) = 4.712, p = 0.031, η_p² = 0.016, Figure 4h). The associated post hoc analysis determined that during 1D vertical control, there was significantly more error when the cursor was at the top of the screen compared to when it was at the bottom. For the 2D case, there was significantly more upward facing error if the cursor was on the bottom of the screen (p = 0.041, Figure 4i). In general, these results confirm that there was no spatial effect on MSE introduced by the robotic arm during vertical control, strengthening the confidence in the results of the horizontal analysis. Interestingly, these results highlight and identify a difference in the ability to perform specific MI commands, primarily regarding the application of downward commands in a graded fashion. This is evidenced by the increase in upward facing error (when the error vector from cursor to target points upward) when the target was in the lower half of the workspace, and generally larger error when it was in the top half of the workspace. Effectively, CP requires users to apply control with magnitudes finely tuned to their situation. During binary on/off commands, as is the case with downward ‘rest’ motor imagery, control step sizes can easily far exceed the required output, leading to overshoot. These findings are unique to CP BCI control, as classical center-out tasks require only one-directional control, and overshoot is irrelevant. Furthermore, the contradiction of simultaneous binary and graded potential commands (the ability to apply variable control magnitude, such as the two handed MI SMR modulation for upward movement included here), presents unique challenges for the end-user, not present during control via two graded potential commands (Left Handed MI vs. Right Handed MI). While the results presented here are from a relatively small cohort of 9 subjects, they reveal important design aspects that need to be considered during the application of traditional control signals to CP. These results should be verified in a larger cohort of participants, however, they emphasize the necessity for including complex tasks in both the training and evaluation of noninvasive BCIs.

Temporal Analysis

While conducting the experiments, deterioration in performance over the course of each 60 second trial was routinely observed. Upon closer inspection, these trends appeared to be linear with respect to time (Figure 5a–e). This led to the hypothesis that the observed reduction in performance was the result of user fatigue and may depend on task complexity. Prior to making any assertions regarding differences between control strategies and interfaces, it is important to ensure that the observed trends (linear increase in error) were not explicitly due to CP task design. As previously mentioned, each trial began with both the cursor/robotic arm and target located in the middle of the workspace. It is therefore anticipated that even in the presence of no user control, the MSE would increase throughout a single trial. Here, to estimate the increase in error over the course of a trial, we applied robust linear regression to the axis-specific MSE values of trisected trials and used the slope as a measure of the rate of performance decline due to fatigue (See Methods). To ensure that the observed fatigue rates were not produced simply by task design, we acquired 210 control trials (n = 70 for 1D LR, 1D UD, 2D) in which previously established subject-specific classifiers used for online CP control were used to evaluate chance level performance. Here, instead of recording EEG from the scalp of a human performing MI, the EEG electrodes were placed on a table and recorded ambient electromagnetic noise for the duration of the trial. We then investigated the difference between the MSE slopes of the noise and user-controlled trials. As expected, a Wilcoxon-Rank Sum test confirmed that the error rate (slope) under noise control was significantly larger than under user control (μ_Noise = 0.067 +/− 0.042, μ_Control= 0.017 +/− 0.041, p < 0.001).

Figure 5. Temporal Analysis – Fatigue Rate.

(a) Total MSE during trisected continuous pursuit (CP) 2D control for both the robotic arm and virtual cursor. (b-e) Axis specific MSE during trisected CP trials for 2D and 1D control. (f-i) Session averaged, axis specific fatigue rates across conditions for both 1D and 2D tasks (j-k) Session specific, axis specific fatigue rates across conditions for both 1D and 2D tasks. Statistical Markers: PostHoc HSD corrected t-test (p<0.001 ***).

Having ensured that differences in observed effects were introduced through subject control, we applied a similar analysis to the behavioral data alone. An initial ANOVA analysis suggested that no significant differences existed (supplemental table ST 10) between the fatigue rate of the different control interfaces, sessions, axes of control, or task complexities in the final three sessions of control. Interestingly, when these estimates were corrected for the covariant underlying MSE, numerous significant differences arose. This was likely due to the heavy dependence that fatigue has on user skill: users with high skill levels and low MSE experienced significantly lower levels of fatigue than those with low skill levels and high MSE. This trend was apparent in the ANCOVA analysis via a linear correlation of 0.396 and R² values of 0.16 for both simple and complex tasks (p<0.001, Supplemental Figure S3).

When correcting for baseline MSE, the fatigue rate for horizontal control was found to be significantly larger than for vertical control (F(1,187) = 6.298, p = 0.013, η_p² = 0.034, Figure 5 f–i). Additionally, there was a significant interaction between the axis of control, interface, and session number (F(2,187) = 3.617, p = 0.029, η_p² = 0.019, Figure 5 j–k). Post-hoc interaction analysis demonstrated a significant difference in performance for the two control axes in the 3^rd session of robotic arm control (only sessions 3–5 were included, Figure 5a, d). During this session, the fatigue rate for the horizontal case was significantly larger (p = 0.007). The lack of highly significant post-hoc results is somewhat expected based on the small partial eta squared of the interaction. On the other hand, the moderate partial eta squared of the axis-wide main effect (0.03) demonstrates a fairly large difference between the collective groups and was not attributed to any specific subgroup interaction. Furthermore, a Cohen’s d effect size for the axis wide main effect was determined to be 0.502. In combination, this evidence suggests that the real statistical result is the main effect between horizontal and vertical fatigue rates. This difference in fatigue rates, while preliminary, suggests the need for controller intervention during more laborious cognitive tasks. Future attempts at noninvasive continuous neurorobotic control may therefore benefit from characterizing an individual’s fatigue rate in terms of the control signals of interest. Ideally, decoders can then be designed to account for a decrease in magnitude/efficiency during prolonged and continuous use.

Discussion

In this work, we attempted to delineate the behavioral tendencies of continuous SMR-based robotic arm control that can significantly influence the effective and efficient translation of BCIs to daily life. While biases were not apparent during simplified analyses, our in-depth examination of continuous BCI control using both a virtual cursor and robotic arm revealed meaningful temporal and spatial prejudices that should be considered when designing these systems for real life. While there existed relatively few significant differences in standard metrics between the robotic arm and virtual cursor results, we observed a strong difference in control quality between these two interface types in the 2D task while conducting experiments. This difference did not produce any significant results in our ANOVA analysis, largely due to a lack of structure among the session-wise trends. However, in secondary evaluations using simple paired t-tests, we found significantly lower tracking correlation values during robotic arm use compared to virtual cursor use (paired t-test, p = 0.016) across both axes during 2D control. This difference can be explained by the ability of the robotic arm to obscure small movements of the target during continuous control, compared to the cursor, due to its relatively larger size. It is interesting that we found significant differences for these results in the current work where we did not in the original work [35]. This difference is likely because of the increased sample size presented here and the removal of the hidden cursor performance underlying the robotic arm control (before mapping to physical device position) in our statistical models. While appropriately addressed in the original work, as the users had no knowledge of the hidden cursor position or dynamics, it was not included here, primarily because it could not serve as an effective measure for determining behavioral biases.

Other major effects of interface type were observed during the 1D horizontal spatial analysis. Although the effect of interface type alone did not reach significance (p = 0.096), the interaction between target position and interface type did. Interestingly, this result opposed our initial hypothesis that the use of the robotic interface would cause a deterioration in performance. Rather, we observed lower MSE values during robotic arm control in the presence of right-sided target positions.

A plausible explanation is that user engagement was initially higher for robotic arm control compared to virtual cursor control. This idea is supported by lower overall MSE for robotic arm use, compared to virtual cursor use during the 1^st session. In other words, the virtual cursor and robotic arm tasks may have been equivalent in difficulty when the target was located on the right-hand side of the workspace (no robotic arm visual obstruction), but subjects were more engaged with the use of the arm and thus performed better. In the case of left-sided targets, where visual obstruction was caused by the physical device reaching across the users’ visual field, robotic arm performance decreased, but only enough to cancel out the positive effects created by the heightened engagement induced by robotic arm use. It should be noted that this bias in behavior did not occur during the horizontal bisection of the 2D task. This is likely due to the increased ability of the user to avoid visual obstruction of the target with vertical arm movement. In effect, when a user’s visual field was equally clear for both interface types, the use of a robotic arm improved BCI performance. This conclusion is supported by decades of study suggesting the importance of user engagement in task proficiency [41], [42]. It further emphasizes that by minimizing adversarial conditions during continuous neurorobotic BCI control, it is not only possible to translate control to practical environments, but that the inclusion of a physical robotic arm facilitates such a transition.

Surprisingly, there were no significant main effects or direct interactions between error direction and interface of control (Supplemental Table ST4). The evidence here suggests that the deterioration in performance was not due to an abundance of situations where the robotic arm obscured the target, but rather, was due to the relatively higher rate of target obfuscation during left-sided target positions compared to right-sided target positions. While such a spatial segmentation of the workspace is relatively course, it reveals a significant alteration in behavior that manifests as a decrease in performance. Future experiments will need to delineate the spatial scale at which these behavioral differences arise or whether these differences are due to task design. Potential scenarios to accomplish this could be the manipulation of the robotic arm mounting position, the informing of subjects of certain behavioral tendencies, or the alteration of target distribution properties.

When the workspace was bisected vertically, we observed significant interactions between target and cursor position during both the 1D and 2D tasks. We observed increased overall error when the cursor was at the top of the screen during 1D control and increased upward facing error when the cursor was on the bottom of the screen during 2D control. Taken together, these phenomena suggest an issue in graded downward control. Effectively, when the cursor was located on the bottom half of the screen, the user would tend to move the cursor downward and overshoot the target. In contrast, when the cursor was located on the top half of the screen, the user would tend to make larger control mistakes, but not preferentially in any specific direction. This suggests that when an individual attempted a downward command, the resulting control signals were larger than the expected output compared to the upward commands. In the case of cursor positions at the top of the screen, this effect manifested as a yo-yoing around the target. When the cursor was on the bottom of the screen, most of the errors pushed the cursor below the target. This observation, while supported by the individual statistical comparisons, will need to be validated in future studies consisting of horizontal on/off control, as well as vertical adversarial (on vs. on) control.

When investigating the dependence of fatigue rate on task complexity and axis of control, we found that horizontal tasks induced significantly more fatigue than vertical tasks, independent of task complexity (2D v 1D). While this was not surprising, as horizontal control is composed of two active cognitive processes (left- and right-hand MI) compared to one active (both hands MI) and one passive (rest) cognitive process during vertical control, the lack of dependence on task complexity was unexpected. In the standard metrics, axial errors were larger during the 2D task, suggesting an increased cognitive load and increased mistakes. The lack of increased fatigue during complex tasks (2D) compared to simple tasks (1D) is largely due to the weak relationship between MSE and fatigue rate. Accounting for this relationship only seems to correct for the extreme cases in which subjects show poor control and is split evenly between complex and simple tasks. This further suggests that these differential errors were not increasing cognitive load but are rather a function of one’s ability to control the cursor/robotic arm movement in each axis independently and simultaneously. Given this evidence, we hypothesize that the subjects were mentally switching between control axes instead of making angular movements as linear combinations of the commands (a task which would be conceivably much more fatiguing). Unlike the behavioral decisions exhibited by the subjects, the target continuously experienced forces that were linear combinations of axial-level motions, leading to movements that were not necessarily perpendicular to the current target position. This mismatch led to, on average, more error in the axial elements of 2D control. Much in line with the results from the standard analysis, task switching only occurred during 2D control and not during 1D control. This evidence further suggests that 2D training is necessary above and beyond 1D task mastery.

The increased temporal fidelity analysis included in the current work additionally suggests that the strategy of cognitively switching between control tasks/axes did not actually increase the mental complexity or resource requirement of the 2D task compared to the 1D task. This is highly relevant for future task and decoder development and design. Future approaches should attempt to either facilitate a subject’s ability to change control axes or potentially develop classifiers that emphasize natural axial level control by including truly independent control signals. Approaches which have previously emphasized non-overlapping control signals, such as using tongue and foot imagery for vertical commands [43], [44], often do so at the sacrifice of intuitiveness. This can lead to a lack of axial-level control compared to intuitive commands. Future research with an emphasis on synergistic effects and the relative position within the active workspace may mitigate these problems while preserving initial axial level control [34], [45], [46].

Finally, given that the “fatigue rate” was higher during noise control, compared to subject control, we extend the concept of quantifying the rate-of-change in error to encompass an ability to resist a disconnected end-state. In the noise condition, the target and cursor positions rapidly and perpetually decorrelate throughout a trial. Comparatively, during active control, subjects were capable of reducing overall error as well as the rate at which the maximal error occurred. Here, through control, mental effort must be exerted to resist the system’s tendency towards disorder (higher error, decorrelated target and cursor positions). Therefore, differences in the ability to fend off disorder in tasks of similar complexity can be linked to the rate of expenditure of cognitive resources, and thus gives us a proxy for subject fatigue. A similar metric can be employed in practical BCI design by incorporating known or estimated tolerances of control before automatically shutting down, or by modulating robotic arm velocity through confidence in control signal quality.

Overall, these results emphasize the need for more complex environments for the training and evaluation of BCI tasks. The evaluation of classifiers in the context of discrete trial center-out tasks ignores the biases which exist during complex tasks, especially those that pertain to fatigue, spatial position with adversarial control conditions, and the interactions between control signals. Environments similar to the one presented in our original work are necessary for the evaluation of practical control which can translate from the benchtop to aiding and assisting individuals suffering from various neurological dysfunctions.

Conclusion

The results presented in this work highlight the effects of increased task complexity and continuous control on subject performance during both virtual cursor and robotic arm control. Overall, we observed similar fatigue rates but higher axial level errors in complex (2D) tasks compared to simpler (1D) tasks. These rates were also found to be higher during more cognitively active tasks (LR > UD), emphasizing the need to evaluate control strategies in complex environments, as 1D mastery does not directly extend 2D deployment. The difference between axial level control quality between simple and complex tasks, despite their similar fatigue rates, suggests that controllers with synergistic or at least functionally independent cognitive tasks may better extend to higher dimensional control and prevent the isolation of axis-specific commands. Finally, we found lower overall tracking correlations during robotic arm control, but also lower MSE for the robotic arm during unobstructed scenarios. These results suggest that when the size and position of a robotic arm are taken into consideration during EEG BCI design, the inclusion of said robotic arm may facilitate control and lead to more rapid and proficient skill acquisition. In all, the analysis presented here emphasizes the necessity of complex, realistic testing environments in order to develop control interfaces that can extend to practical, helpful neurorobotics for those suffering from neurological dysfunctions.

List of Abbreviations:

CP

Continuous Pursuit

1D

One Dimensional

2D

Two Dimensional

LR

Left/Right

UD

Up/Down

EEG

Electroencephelpography

BCI

Brain Computer Interface

MI

Motor Imagery

SMR

Sensory Motor

MSE

Mean Squared Error