Exploration-based feedback for BCI training: a case study with an adolescent with paraplegia

Abstract

Purpose

Brain–computer interface (BCI) inefficiency limits clinical utilization of BCIs, as many users struggle to produce consistent machine-recognizable electroencephalography (EEG) patterns for reliable control. While training can improve BCI performance for most users, the required duration and intensity may hinder BCI accessibility. Prior BCI and motor learning research suggests that feedback enabling efficient exploration of different task strategies may enhance training.

Methods

An eight-session BCI training case study was completed with an adolescent participant with paraplegia. To support training, a novel feedback system that visualized EEG signal pattern states identified via K-means clustering within the EEG covariance space. Unlike common classifier feedback, this interface presented the EEG signal patterns produced throughout each trial, allowing the participant to explore strategies that yielded task-specific pattern states.

Results

The participant was initially a low-performing user and showed little progress across the first five sessions. After transitioning to a simplified feedback mode emphasizing deviation from resting state patterns in the sixth session, the participant displayed significant improvement in task-related physiological signal discriminability. Post-training analysis, however, revealed that this improvement was partially attributable to electromyography (EMG) activity from cranial muscles.

Conclusion

Although the observed gains were not solely attributable to neuro-cortical signal modulation, the case study highlights the potential of simplified feedback to support task exploration in low-performing users and presents potential implications for hybrid EEG-EMG BCIs for relevant clinical populations.

Background

Electroencephalography (EEG)-based brain–computer interfaces (BCIs) hold promise as assistive technologies for individuals with severe motor disabilities [1, 2]. However, clinical adoption has been limited because many users fail to produce machine-discriminable EEG signals [3–5]. It is estimated that up to 50% of individuals experience “BCI inefficiency,” where the computer decoder cannot reliably distinguish between the user’s mental control tasks [3, 4, 6].

Many studies have found that users can often improve their ability to induce targeted neural activity and produce more easily discriminable signals with training [7–14]. Simultaneously, however, there have been several cases where limited progress has been achieved even after extended training [15–17]. Because BCI training can be mentally exhausting and poor performance and high effort often lead to assistive device abandonment [18], there has been considerable motivation to develop more effective user-training methods [5, 9, 19, 20].

Early BCIs relied on operant conditioning, requiring users to discover mental strategies that elicited target neural responses through an exploratory trail-and-error process [21–23]. Machine learning-based approaches later enabled significantly faster calibration by automatically identifying user-specific EEG patterns [24, 25], and classifier-based feedback became the dominant user training paradigm. However, for low-performing users, classifier feedback often appears random because the models are conditioned on poorly discriminable data [26, 27], thereby limiting the instructional value of the feedback [28, 29].

Recent work has revisited exploratory, operant-style feedback methods that guide users toward distinct and stable EEG patterns rather than solely providing task recognition-based feedback. For example, training feedback provided as real-time cortical maps [30] and low-dimensional visualization of task distributions [17] have been shown to improve performance in initially low-performing users. Moreover, [31] found participants achieved significant performance improvements when receiving feedback from classifiers initialized using high-performance user data.

A shared goal among these approaches is to support efficient exploration of mental strategies during early learning. Contrary to feedback from poorly-conditioned classifiers which can incorrectly connote a degree of “correctness” for each trial, these approaches convey when users produce similar patterns for multiple tasks (i.e., similar heatmaps, data point projections, or stable classifier outputs). Consequently, for initially low performance users, these feedback methods provide trial-wise feedback that can indicate when a user has achieved EEG signal patterns that differ from patterns that they previously generated. This type of exploratory learning supported by action-perception feedback loops is believed to play an important role in skill learning [32, 33].

With the aim of enabling efficient exploration of mental task strategies during early stages of learning, we developed a novel training protocol and feedback system. This approach, summarized in Fig. 1, builds on unsupervised EEG clustering techniques [34, 35] that define a discrete state space of K+1 EEG pattern states: K states derived from prior data and an additional state representing statistically dissimilar patterns. As the segmentation is derived using unsupervised methods, the model does not attempt to identify associations between the states and any single task. Therefore, states in this model have the flexibility to be associated with multiple tasks or to be task-specific. Similarly, some tasks may commonly yield multiple pattern states [34–36].

Fig. 1.

Overview the EEG pattern state analysis used as the basis of the training feedback deployed within this study. a Apply K-means clustering methods to define set of discrete pattern states . b Segment trials into shorter overlapping temporal epochs . c For each trial, model temporal epochs as a sequence of K+1 pattern states. The state ( within the figure) represents the set of all EEG patterns that are statistically dissimilar to the patterns represented by states

This model flexibility provides agency for the user to explore task execution approaches and form the associations between their intrinsic senses related to task execution and the EEG pattern state outcome. More concretely, during training, short EEG epochs are classified into one of the K+1 pattern states, and users receive feedback indicating the current state. Unlike classifier feedback, this approach does not signal task correctness but instead encourages users to discover strategies that yield stable, task-consistent, and mutually distinct EEG patterns. Unlike the exploratory feedback approaches discussed above, this approach removes the requirement for users to generate an a priori defined signal response via preselected tasks. Such constraints may be counterproductive for some clinical populations (e.g., pediatric), where the standard neural patterns observed in neurotypical adults, such as lateralized event-related (de-)synchronizations, may be absent or altered due to developmental or clinical factors [37, 38].

As an initial evaluation of the system, we recruited an adolescent with paraplegia who initially exhibited low BCI performance (i.e., someone likely to experience BCI inefficiency) to complete the training protocol. Herein, we describe the participant’s outcomes and experience across training sessions.

Methods

Participant

The participant was a 17-year-old right-handed male who sustained a T6 level traumatic spinal cord injury, classified as ASIA grade B (incomplete; sensory but not motor function preserved below the lesion) [39], 19 weeks prior to the beginning of the study. He had no prior neurodevelopmental diagnoses, used a manual wheelchair, and regularly attended physiotherapy and occupational therapy sessions during the study. His only previous BCI experience was a recreational activity not involving mental-task control.

EEG equipment and data

EEG was recorded using a R-Net electrode system and LiveAmp amplifier (Brain Products GmbH, Germany). Data were analyzed from the following 18 electrode positions placed according to the international 10–10 system: Fz, FC1, FC2, T7, C3, Cz, C4, T8, CP5, CP1, CP2, CP6, P3, Pz, P4, O1, Oz, and O2. Ground and reference electrodes were placed at FPz and FCz, respectively. Data were sampled at 500 Hz.

Sessional overview

The participant completed 8 sessions (1 strictly for data collection, 7 for user training) over 18 days. Each session took place on a unique day and was approximately 2 h in length, including EEG setup time. Sessions were scheduled at times selected by the participant to adhere to his availability.

Prior to beginning the training, the researchers explained the goal of the sessions (i.e., to identify and practice mental task approaches that produce discriminable EEG signals) to the participant. Additionally, the participant was instructed not to move or intentionally activate any muscles while performing the mental tasks. During EEG setup, he was shown his EEG recordings and asked to perform ocular movements and clenching of his jaw to visualize how the EEG is impacted by the electromyography (EMG) signals produced by these movements. Trials where movements were observed were excluded from analysis. After an observed movement, the researchers paused the session and reminded the participant not to move.

Session 1

The purpose of session 1 was to collect EEG data for generating the initial EEG pattern state space and for selecting the mental control tasks to be used throughout the training. No feedback was provided during this session.

The participant completed 192 mental imagery task trials across four 48-trial blocks. As depicted in Fig. 2, each trial began with a task cue (0 s) indicating the task to perform for that trial, followed by a 7 s task execution period (1–8 s) and a 4 s (8–12 s) break before the next trial.

Fig. 2.

Timing scheme for mental imagery trials

For each 48-trial block in session 1, the participant was instructed to perform a different set of 4 mental tasks. Table 1 describes each task and indicates during which block each was performed. Note that every block included mental rest (i.e., no mental imagery) as one of the tasks. Each block consisted of 12 trials for each of the 4 tasks. At the conclusion of each block, the participant was asked to describe his mental imagery approach to each task.

Table 1.

Session 1 mental imagery tasks

Task	Description	Block(s)
Mental rest	Relax and do not engage in any mental imagery	1–4
Left-hand motor imagery	Imagine first-person movement of the left arm or hand	1
Categorical word generation	Given a category (e.g., animals), generate as many words as possible belonging to it without vocalizing	1
Family member facial imagery	Imagine the face of a family member without closing the eyes	1
Foot motor imagery	Imagine first-person movement of the feet or legs	2
Auditory imagery	Imagine a song or tune without vocalizing or humming	2
Complex mental subtraction	Given a starting number, successively subtract a two-digit number (e.g., 72-13=59, 59-13=46)	2
Right-hand motor imagery	Imagine first-person movement of the right arm or hand	3
Figure rotation	Visualize a cube rotating in three-dimensional space	3
Celebrity facial imagery	Imagine the face of a celebrity without closing the eyes	3
Tongue motor imagery	Imagine first-person movement of the tongue	4
Spatial navigation	Visualize first-person movement through a familiar location (e.g., entering the front door of home)	4
Simple mental subtraction	Given a starting number, successively subtract a one-digit number (e.g., 72-3=69, 69-3=66)	4

Before session 2, task discriminability metrics [26] were computed for different combinations of two active mental imagery tasks and mental rest (MR). All data were normalized using the covariance rebiasing technique (see “Covariance re-centering” section) to limit the impact of inter-block signal nonstationarities on the metrics. These results were reviewed with the participant at the beginning of session 2. The participant was then asked to consider both the discriminability results and his task preferences to select two mental tasks to use for the training sessions. The participant initially selected right hand motor imagery (RMI) and spatial navigation (SN). After session 3, however, he reported difficulty performing the SN task consistently and replaced it with the categorical word generation (CWG) task.

Sessions 2–5

Sessions 2–5 consisted of ten EEG recording blocks as outlined in Fig. 3. Each session began with a 60 s baseline measurement where the participant was instructed to relax with his eyes open.

Fig. 3.

Overview of sessions 2–8

Structured practice blocks

Following the baseline recording, the participant completed two structured practice and four free practice blocks. During the structured practice blocks the participant was cued to begin a mental imagery task at a specific time and then received feedback from the computer during the trial. These blocks consisted of 36 trials (12 trials for each of the MR and two active mental imagery tasks) following the timing structure shown in Fig. 2. Task sequences within each block were randomized subject to the constraint that every consecutive triplet (trials 1–3, 4–6, etc.) included one instance of each task.

Throughout these trials, the participant received feedback that displayed the temporal evolution of his observed EEG pattern states (Fig. 1c; see “Online signal processing” section for signal processing and pattern state details). The pattern state information was conveyed through both a visual interface, depicted in Fig. 4, and vibrotactile feedback. The visual interface consisted of K+1 light bulbs, each one representing one the K+1 pattern states, whose brightness increased according to the number of within-trial temporal epochs in which that state was observed [34–36]. At the start of each trial, all bulbs were off. Within trials, each trial epoch was assigned to a pattern state and the corresponding light bulb would increase in brightness. A light bulb would reach maximum brightness only if every epoch within a trial was assigned to the cognate state. The brightness level of each light would not decrease until all lights were reset at the beginning of the next trial. Pattern state estimates were obtained using 2 s analysis windows, and updated every 0.5 s.

Fig. 4.

Multi-pattern state-based visual feedback interface. Current task cue is given in the box at the top of the screen. Each light bulb below the cue represents one of the K+1 EEG pattern states (K=4 in this example). a At the beginning of a trial, all light bulbs are off. Dashed red ellipses indicate location of all light bulbs. b As the trial progresses, the brightness of each light bulb increases proportionally with the number of within-trial epochs assigned to the corresponding pattern state. In this example, the approximately equal brightness of the two light bulbs on the right indicates equal frequency of those states. No within-trial epochs were assigned to the other 3 states and hence those light bulbs remained off

Simultaneously, the participant also received tactile feedback from K+1 ≤ 5 vibration motors secured to the tips of the fingers of both hands. Like the light bulbs, each motor-finger pairing represented a different pattern state. During a trial, the motor representing the pattern state of the most recent epoch would vibrate; all other motors would remain off. The vibration frequency of the motors was not within the range of sensorimotor activity (8–30 Hz) [40, 41] and therefore not expected to induce EEG activity that would confound pattern state classification [42].

Before the first structured practice block, the participant was instructed on how to interpret the feedback and viewed a video illustrating the visual feedback. He was then asked to verify his understanding of the feedback.

At the conclusion of each structured practice block, summative performance metrics—run-wise classification accuracy and classDistinct [26]—were computed and reviewed with the participant. These summative metrics complemented the pattern state-based feedback by indicating the extent to which EEG signals from different tasks could be distinguished. Following suggested guidelines for effective BCI feedback [20, 28], this review contextualized the trial-wise feedback within the broader training goal of improving task classification accuracy and highlighted the gap between current and target performance.

Free practice blocks

The 4-minute free practice blocks were implemented to follow recommendations from reviews and critical analyses of BCI training schemes [5, 20, 28]. Specifically, these blocks were self-paced (i.e., the participant dictated the start and duration of practice periods) and the participant selected the subset of tasks to practice. Additionally, the participant could select between two feedback modes: (1) synchronous mode where trials were cued and feedback presented as in the structured practice blocks or (2) asynchronous mode where no cues were presented and the visual (light bulb) and tactile (motor) feedback continuously indicated the current EEG pattern state (updated every 0.5 s). The asynchronous mode provided opportunity for the participant to attempt different task strategies and observe the associated pattern state outcomes without rigid timing constraints. The participant was allowed to alter the practice task subset or feedback mode at any time throughout the block. For example, during a single free practice block the participant could practice right hand motor imagery in asynchronous mode for 1.5 min, take a 30 s break, and finally practice all three tasks in synchronous mode for 2 min.

At the beginning of each of the structured and free practice blocks, the participant completed 5 trials of mental rest without receiving any feedback. These trials were used to calibrate the rebias transformation as described within the “Covariance re-centering” section.

No feedback and BrainDriver blocks

Following the structured and free practice blocks, the participant completed a synchronous block of trials where he received no feedback. These blocks consisted of the same number of trials (36) and timing structure as the structured practice blocks. These blocks were not intended to provide instructive training but rather to assess whether the participant’s performance was dependent upon the feedback.

Finally, the sessions concluded with a second baseline measurement and the participant playing the BCI Cybathlon competition BrainDriver game.1 Within the proposed training protocol, the game provided both a motivation goal for the participant and, similarly to the no feedback blocks, an opportunity to assess performance in the absence of the primary training feedback. Because we trained the participant with only three tasks, we removed all “headlight” sections from the race tracks so the participant only needed to produce three commands: (i) straight/neutral (MR), (ii) right turn (RMI), and (iii) left turn (SN/CWG). The participant completed 1 race in each of sessions 2 and 3 and 2 races in session 4. Due to scheduling constraints, the participant did not complete any races in session 5.

Sessions 6–8

Sessions 6–8 followed the same block structure as sessions 2–5 (Fig. 3); however, a modification was made to the pattern state-based feedback presented during the structured and free practice blocks. The modification was made following analysis of the performance metrics from sessions 2–5. The participant was experiencing BCI inefficiency (low task discriminability, chance-level classification accuracy) and had demonstrated no significant performance improvement through sessions 2–5. Analysis of his performance self-assessments (“User experience questionnaires” section) indicated that the multi-pattern state feedback was challenging to interpret because no clear task-pattern state associations emerged. Indeed, reviewing measures of task discriminability and the ratio of inter-task to intra-task variance, it appeared likely that the participant was not consistently producing any consciously controlled task-related EEG modulations.

Therefore, the pattern state model was changed to comprise of only one state (i.e., K=1) that represented the resting state EEG activity that the participant produced during the preceding sessions. Correspondingly, feedback was simplified to two light bulbs and two motors: one light bulb-motor pair activated when EEG patterns resembled those from prior sessions, and the other when they differed significantly (signal processing details in the “Single pattern state deviation” section).

To improve the interpretability of the feedback, visual feedback was made continuous by scaling bulb brightness according to EEG similarity with the resting-state template (Fig. 5; brightness level calculation details in the “Single pattern state deviation” section). Unlike the multi-state version, brightness was updated using only the most recent epoch. Vibrotactile feedback remained binary (ON/OFF).

Fig. 5.

Single-pattern state-based visual feedback interface. The brightness of the left light bulb represents how similar EEG trial epochs are to mental rest trial epochs from previous sessions. The brightness of the right and left light bulbs increase and decrease, respectively, as the trial epoch becomes more distinct from the initialization data distribution. The brightness level of both light bulbs always summed to 100%. The numbers below each light bulb indicate the brightness level of each light

All other procedures matched those of sessions 2–5. The participant completed 3, 3, and 2 BrainDriver races in sessions 6, 7, and 8, respectively.

Online signal processing

Figure 6 presents a summary of the online signal processing framework used throughout sessions 2–8.

Fig. 6.

Online signal processing framework used for the pattern state-based feedback during the structured and free practice blocks (grey boxes) and the BrainDriver (orange) block. Blue boxes represent preprocessing steps that were common to all signal processing contexts. FgMDM = Minimum distance to the mean with geodesic filtering classifier; classifier details provided within the “BrainDriver classifier” section

Filtering, epoching, and covariance estimataion

All EEG signals were subject to a fourth order Butterworth bandpass filter with a passband of 8–30 Hz. All online processing during the structured and free practice blocks was performed on 2 s epochs of EEG data, where was the number of electrode channels and was the number of samples per epoch. Successive epochs were spaced 0.5 s apart, corresponding to a 75% overlap, yielding 11 epochs per 7 s trial. Online processing for the BrainDriver classifier was performed on 4 s epochs () spaced 0.5 s apart.

Spatial covariance matrices were estimated for each epoch with regularization by Ledoit-Wolf shrinkage [43] to ensure symmetric positive definiteness.

Covariance re-centering

To minimize intra- and inter-session EEG nonstationarities, all covariance matrices were aligned to a common Riemannian reference using the adaptive rebiasing transformation [11, 44]. Analogous to mean-centering in a Euclidean space, the rebiasing transformation aligns covariance matrices to a common reference on the Riemannian manifold to which symmetric positive definite matrices belong [11, 44]:

1

where is the Karcher mean of a set of covariance matrices  [45, 46] and is the re-centered covariance matrix.

Let represent the rebias transformation matrix for session s during block b. For each session s, the initial rebias matrix was estimated from the 60 s baseline recording. Data were segmented into non-overlapping 2 s epochs, covariance matrices were computed, and anomalous epochs were removed using a Riemannian potato detector (z-score threshold = 2.5) [47]. The Karcher mean of the remaining baseline covariances defined .

During the feedback blocks, the rebias matrix was adaptively updated using geodesic interpolation between the previous estimate and a new reference computed from mental-rest (MR) trials:

2

where α parameterized the relative weight of . The updated was applied at the beginning of each structured and free-practice block, i.e., , where was the mean covariance from the 5 MR calibration trials preceding the block. Additionally, within structured blocks was updated after every third trial using α =0.1 and equal to the mean of the most recent MR trial’s epochs. Unlike previous implementations of the adaptive rebias algorithm [11, 44], only MR data were used for updates, as MR was expected to best capture session drift independent of task activity.

A similar procedure was applied for the BrainDriver portion of the sessions. The rebias transformation matrix was reinitialized using the baseline recording immediately preceding the BrainDriver block. Similarly, within race updates were applied following each MR segment using the same interpolation rule.

For pattern state model and classifier initialization, all task epoch covariance matrices from session s and block b were re-centered according to (1) using the mean of all mental rest covariance matrices from the same block.

Artifact rejection

Each re-centered epoch covariance matrix was subjected to a Riemannian potato anomaly detector [47] with z-score threshold set to 3.5. The model for each session was initialized using re-centered epoch covariance matrices from previous sessions. The selected threshold exceeded the typical single session values of 2.5-−3.0 [47, 48] because inter-session EEG nonstationarities resulted in baseline differences between the current and previous session. If a trial epoch was deemed to be an outlier, it was not used to update the feedback nor was it included in any rebias transformation updates.

Multi-pattern state classification

Using participant data from prior sessions, the EEG covariance space was segmented into a discrete state space, with each state representing a different EEG pattern template. As in [35], K-means clustering was applied to identify K states representing EEG patterns that had been observed in previously recorded data and a state was defined as the space where EEG patterns were significantly different from the K template patterns (Fig. 1c).

This model was achieved by fitting a Riemannian potato model [47] to represent each of the K pattern states. Each potato model was initialized using a set of N trial epoch covariance matrices assigned to state k after subjecting all epoch covariance matrices to K-means clustering. After removing anomalous covariance matrices from this set during the initialization process [47, 48], the model was characterized by its centroid and the log-transformed mean and standard deviation of the Riemannian distance between its centroid and each of the remaining non-anomalous covariance matrices from its initialization set:

3

4

where denotes the Riemannian distance [45, 46]. With these parameters, a z-score metric could be computed for each to evaluate the degree to which a covariance matrix deviated from the distribution of matrices used to initialize :

5

Using the K potato models and their associated z-score metric (5), any covariance matrix could be assigned to one of K+1 states. A covariance matrix was assigned to the state s with the closest centroid subject to z-score constraint:

6

where τ is a z-score threshold that defines the maximum level of statistical deviation between and ’s initialization set for to be considered a member of state k. For sessions 2–5, where the multi-pattern state feedback was used, τ = 2.5 [35]. The state assignments for the initialization of each were determined using the computational pipeline presented in [34]. The full initialization process for each given a set of mental imagery trials , their associated true task and block (i.e., a set of unique identifiers indicating during which block and session the cognate trial was performed) labels, and a number of states K is presented within Algorithm 1. First, trials were epoched into shorter, overlapping segments: 2 s windows with 75% overlap. Covariance matrices were computed for each epoch and any anomalous covariance matrices were rejected and removed for the set of initialization data. The rebias transformation (1) was applied to all covariance matrices using the mean of all MR covariance epochs within a block to center all covariance matrices from different sessions and blocks to a common reference point. The entire set of re-centered covariance matrices, , were then mapped to a Euclidean space tangent to the manifold of covariance matrices [45, 46]. The tangent space vectors were mean-centered and scaled by their standard deviation along each dimension. Finally, the vectors were subject to K-means clustering to identify the initialization state assignments.

For session 2, the models were initialized using the data from session 1. Because session 1 consisted of a larger variety of tasks than used in session 2, greater weight was given to the RMI and SN task data (i.e., the selected tasks for session 2) during the K-means portion of the initialization by appending a duplicate set of the covariance matrices from trial epochs recorded during those tasks to the initialization data. In sessions 3–5, the models were initialized using data from sessions 2–4. Following the recommendations from [35], K was set to the number of tasks, K=3, for sessions 2–3. In response to the participant struggling to identify any strong task-pattern state associations in sessions 2 and 3, an additional state (i.e., K=4) was added in session 4 in an attempt to generate a segmentation that could better reflect more minute inter-task differences [34, 36].

Single pattern state deviation

In sessions 6–8, the pattern state-based feedback was changed to consist of only K+1=2 states (i.e., one state to represent patterns similar to those produced in prior sessions and another to represent different, emergent EEG patterns). Here, a single was initialized using Algorithm 1 with only MR trials included within the initialization data. Additionally, τ in (6) was reduced to 2.0.

The feedback light bulb brightness levels during these sessions was computed using:

7

where is a epoch covariance matrix and is the z-score for  (5). β reflected the similarity of the current to the reference distribution represented by and was used to set the brightness level of the left light bulb in Fig. 5; correspondingly, the brightness of the right bulb was set to 1-β . β was maximized when and decreased linearly to 0.5 when (i.e., the threshold value where would be classified to be in state K+1).

BrainDriver classifier

The minimum Riemannian distance to the mean with geodesic filtering (FgMDM) [49] method was used to classify tasks during the BrainDriver game. As depicted in Fig. 6, the classifications were made using covariance matrices as inputs; therefore, all preprocessing steps (blue blocks in Fig. 6) were identical to other feedback blocks. The classifier was initialized using trials from the structured practice and no feedback blocks from the current session.

User experience questionnaires

To assess the participant’s interpretation of the feedback, he completed a performance self-assessment following the structured and free practice blocks. The questionnaire asked him to self-assess within the following categories: (i) discriminability of EEG signals for each mental task, (ii) consistency of EEG signals for each mental task, and (iii) predicted classification accuracy if the session’s trials had been subjected to a classifier.

To assess the participant’s motivation he completed the BCI variant of the Questionnaire on Current Motivation (QCM-BCI) [50, 51] at the beginning of each session. Finally, to assess the participant’s experience performing the training and using the feedback interfaces, he completed the NASA Task Load Index (NASA-TLX) [52] and a Subjective Experience questionnaire which asked him to rate how difficult, tiring, and enjoyable the session was on a five-point Likert scale at the end of each session. At the end of the final session, the participant was also invited to provide feedback about his experience throughout the entire study.

Post-training analysis

All post-training analyses were conducted on full 7 s trials from the structured practice, no feedback, and synchronous portions of free practice blocks. Trials from consecutive free practice blocks (i.e., blocks 3–4 and 5–6) were grouped together for block-wise analyses.

Preprocessing and artifact rejection

Unless otherwise specified, all trials were zero-phase bandpass filtered between 8 and 30 Hz using a fourth order Butterworth filter prior to analysis. Then all trials were subjected to the four-step artifact rejection procedure outlined below. Note that the second structured practice block of session 2 was excluded from analysis due to the plurality of trials rejected.

First, trials where the researchers observed participant movements during the sessions were excluded. Second, any bandpass filtered trials containing absolute voltage amplitudes exceeding 70 µV were removed. Third, trials with excessive EMG artifacts were removed according to [53]. All raw trials were filtered using a 1–120 Hz bandpass and 60 Hz notch filter. Then, a first-degree polynomial was fit to the log-log power spectral density (PSD) vs. frequency plot between 7 and 45 Hz. Any trial where the slope of the estimated polynomial exceeded a threshold were excluded. The value for was derived as the slope value representing a 3 decibel power increase between 7 Hz (where EMG contamination is expected to be less prominent) and 45 Hz (where EMG would be expected to dominate EEG). The remaining trials in each block were assessed with a Riemannian potato anomaly detector (z-score threshold = 2.5) to remove anomalous trials [47, 48].

After excluding rejected trials, covariance matrices for each trial were computed and re-centered according to (1) using the mean of all trial covariance matrices within the same block.

Longitudinal task discriminability

To evaluate the participant’s performance throughout the sessions, a z-score-based variant of the classDistinct metric [26] was computed for each block. First, the classDistinct metric for a block was computed using the true task labels. Then, a null distribution was estimated by re-computing the metric using 1000 random permutations of the labels. A z-score was computed for each block by subtracting the mean of the null distribution from the true observed classDistinct and scaling by the standard deviation of the null distribution. This modification facilitated comparison between blocks with differing numbers of trials (i.e., structured vs. free practice blocks) and reduced the metric’s sensitivity to outlier trials. Only free practice blocks where the participant completed synchronous practice of all three tasks were included in this analysis.

Longitudinal performance changes were also assessed by computing 10-times repeated three-fold cross-validation accuracy with a FgMDM classifier for each session. A null distribution of chance level accuracies was derived by repeating the 10-times repeated three-fold cross-validation process with 1000 random permutations of the task labels. Significant (p<0.05) sessional accuracies were identified by comparing against the critical values of this null distribution. Note that during the cross-validation accuracy analysis, the covariance re-centering transformations were computed using data from the training fold only and applied to the data within the held out fold.

Comparison of post-training task discriminability and online feedback

To evaluate the relationship between the feedback received by the participant and the post-training task discriminability metrics discussed above, numerical metrics were used to provide summative feedback scores for each structured and free practice block. The taskDistinct metric [34] was used to score the multi-state feedback sessions. Briefly, this metric computes the difference in the pattern state probability vectors for each task to estimate the discriminability between tasks. The pattern state probability vectors for each task were estimated using the observed counts of each state during the online trial epochs.

For the single-state sessions, the feedback score for each block was computed as:

8

where is the mean z-score value achieved for task for the trial epochs during the block. The score reflects how well the participant achieved the stated objectives, specifically: (i) to minimize the z-score (maximize brightness of left light) for MR tasks and (ii) maximize the z-score (maximize the brightness of the right light) for the RMI and CWG tasks. The score measures the deviation of the mean z-scores from the value 2 because both light bulbs would have equal intensity at this level.

To augment the analysis of the observed feedback results, simulated feedback was generated for each session using the complement of the feedback modality that was deployed (i.e., single-state feedback scores were simulated for sessions 2–5 and multi-state feedback scores simulated for sessions 6–8). State models to simulate these results were initialized using the same data that were used to initialize the real sessions (i.e., only data from previous sessions was used to initialize the state models).

Neurophysiological analysis

Similar to the analyses performed within prior longitudinal BCI-user training studies [10, 11], we investigated the discriminability of different tasks at individual electrode channels. Channel-level discriminability was estimated using a robust variant of the Fisher criterion score comparing the PSDs between the two active tasks (RMI and CWG) versus MR. First, the mean of the PSD (computed using Welch’s method [54]) between 8 and 30 Hz for each channel c and trial i from task ℓ was calculated. Then, inter-task discriminability was computed as:

9

where ℓ denotes one of the active mental tasks, RMI or CWG, MR denotes the mental rest task, is the median of across all trials, is the median absolute deviation of :

10

where is the number of trials for task ℓ .

Mean signal power across temporal epochs of trials was also investigated for evidence of expected EEG responses during the different tasks. Welch’s method was used to compute signal power within 2 s epochs with 95% overlap for each trial. The power of each epoch was averaged across trials. To facilitate comparisons across sessions, all mean epoch power values were z-score normalized.

Results

Lack of task-discriminable EEG during multi-state feedback sessions

The participant struggled to produce task-discriminable EEG patterns during sessions 2–5. This was reflected in the multi-pattern state feedback received by the participant as clear task-state associations did not emerge during the training sessions. Figure 7 presents a visualization of a representative example of multi-pattern state-based feedback received during the second structured practice block of session 5. The trellis visualizations [36] indicated that state distributions were relatively equal across tasks.

Algorithm 1.

Multi-pattern state classifier initialization

Despite the lack of clear pattern state delineations between tasks, the pattern state activation patterns and trellis patterns in Fig. 7 provide possible evidence of expected EEG responses for the RMI task. Specifically, the activation patterns for states S3 and S5 appeared to reflect lower power in motor regions near electrodes CP1 and CP5. Additionally, the RMI trellis showed that state S3 was common during epochs 1–6 before S4, characterized by higher power in the same electrodes, became the dominant state in epochs 8–11. This transitions from low to high power in CP1 and CP5 may have reflected the typical event-related desynchronization (power decrease) followed by a synchronization (power increase) induced by motor imagery [55].

Improved longitudinal task discriminability during single-state feedback sessions

Figure 8 presents the across-epoch average of the feedback received by the participant in sessions 6–8. Notably, the participant seemed to achieve the objective of lower z-scores and higher brightness in light bulb 1 (i.e., β ) during the MR task during the majority of blocks. Additionally, the block-wise average of the MR z-scores was observed to incrementally decrease from sessions 6–8, suggesting that the participant became more adept at producing consistent signals during the MR tasks during this period. To confirm that this decrease in mean MR z-scores was not an artifact of the updates to between sessions, session feedback was simulated using a single state model for all sessions. This analysis (not shown) did not reveal significant changes to the z-scores in Fig. 8.

Fig. 7.

EEG pattern state trellis visualization [36] of typical multi-pattern state-based feedback received by the participant in sessions 2–5. a Visualization of pattern state frequency for each of the three mental tasks. The vertical axis represents each of the K+1 discrete pattern states and the horizontal axis represents the 2 s trial epochs. The radii and opacity of the circles at each epoch are proportional to the number of trials in which a given state was observed during that epoch. Similarly, the width and opacity of the lines between circles at adjacent epochs are proportional to the number of observed transitions occurring for that state pair during that period in the trials. b Common spatial activation patterns [56] of trial epochs assigned to each of the K+1 states

Similarly, across the three sessions, the participant gradually increased the z-scores and reduced the brightness of light 1 during the CWG task. For the RMI task, the average z-scores were generally higher than MR during sessions 6 and 8. However, in the second structured and free practice blocks of session 7, the participant achieved similar z-scores for the RMI and MR tasks. Notably, when describing his task approaches during these blocks, the participant indicated that he focused primarily on identifying successful strategies for the CWG task and, consequently, he believed his engagement waned during the RMI trials.

Figure 9 presents the permutation test-based classDistinct scores within each training block across all sessions. A Spearman correlation analysis revealed a significant positive correlation between the classDistinct scores and the session number. Notably, the scores increased most significantly following the transition from the multi- to the single-pattern state-based feedback between sessions 5 and 6. A Mann–Whitney U test further supported that the classDistinct scores from sessions 6–8 were significantly different (p<0.001) from those in session 2–5.

Fig. 8.

Average of single-pattern state-based feedback from sessions 6–8 for structured practice (SPn) and free practice (FPn) blocks, n = 1, 2. (top) Mean and standard deviation of the single pattern state model z-score from equation (5) for each task for all epochs within a block. (middle) Mean brightness level from equation (7) for light bulb 1 for each task for all trial epochs within a block. (bottom) Percentage of trial epochs where the z-score exceeded 2.0 and were therefore classified as different from those used to initialize

Figure 9b shows the relationship between the post-training classDistinct values and the online single-state feedback scores (eq. (8)). A Spearman correlation test indicated that the online feedback scores generally reflected the post-training task discriminability outcomes. Similar observations were made when comparing classDistinct and the taskDistinct feedback score for the multi-pattern feedback.

Figure 10a displays the 10-times repeated 3-fold cross-validation results for each session. Scores for sessions 2–5 did not significantly exceed chance-level discrimination between tasks. Similar to the classDistinct results, there appeared to be an increase in scores coinciding with the transition to the single-pattern state feedback in session 6. Analysis of the classifier confusion matrices for sessions 6–8 (e.g., Fig. 10b) indicated greater discrimination between MR and the two active tasks than between the two active tasks. Similar trends in improved class discriminability during the latter sessions was observed in the BrainDriver race completion times (Fig. 11).

Fig. 9.

a Best linear fit (dashed line) to permutation test-based classDistinct z-scores for structured practice (circles), no feedback (diamonds), and free practice (x) training blocks across sessions with multi-pattern state (sessions 2–5; grey background) and single-pattern state (sessions 6–8; yellow background) feedback. b Permutation test-based classDistinct z-scores for structured and free practice blocks vs. single-state online feedback score (Eq. (8)). Dark points indicate feedback scores that were observed while using the single-state feedback in sessions 6–8. Light points indicate simulated single-state scores for sessions where multi-state feedback was used. r values are Spearman correlation coefficient

Fig. 10.

a 10-times repeated 3-fold cross-validation accuracies for multi-pattern state (sessions 2–5; grey background) and single-pattern state (sessions 6–8; yellow background) feedback sessions. Error bars represent standard deviations. Dashed horizontal lines show the upper bound of the 95% confidence interval for the sessional chance accuracy estimated by permutation tests. b Session 8 confusion matrix averaged across all cross-validation repeats and folds. Numbers within each square represent the average proportion of trials decoded by the classifier as belonging to each class. MR = mental rest, RMI = right hand motor imagery, CWG = categorical word generation

Evidence of implicit learning of multimodal physiological control

Figure 12 indicates that the channel-wise PSD Fisher score results match the trend from the task discriminability analysis; PSD Fisher scores were higher in sessions 6–8 than in sessions 2–5. Notably, however, these results suggest that the improved task discriminability cannot be exclusively attributed to neuro-cortical signals. The higher signal power observed during the active mental tasks in lateral regions, particularly channels T7, T8, CP5, and CP6, is characteristic of EMG contamination from the temporalis muscles [53]. This suggests that an inadvertent outcome of the training may have been that the participant learned to activate these muscles to achieve signals that deviated from the resting state baseline. Spurred by this observation, we repeated the cross-validation accuracy analysis using common EMG classification approaches [57]. The results (not shown) of this reanalysis closely matched the trends of our initial EEG classifier results, namely, chance-level accuracies in sessions 2–5 and above chance-level accuracies in sessions 6–8, further supporting that EMG activity played a meaningful role in increasing the task discriminability observed within sessions 6–8.

Fig. 11.

Best linear fit (dashed line) to BrainDriver race completion times for each race attempted throughout the training sessions. r is Spearman correlation coefficient. Horizontal dotted line indicates the race completion time had the participant made no left or right turn commands

Despite the apparent role of EMG in achieving improved BCI results, analyses of temporal signal power within select blocks provided some possible evidence of improving neuro-cortical performance. Figure> 13 shows the median electrode channel power timeseries for electrodes C3, CP5, and T7 for the three blocks with the highest classDistinct scores in each of sessions 5, 7 and 8. In the C3 and CP5 timeseries for session 7—where the topographic maps in Fig. 12 suggested less EMG activity than in sessions 6 and 8—signal power was lower during early portions of the trial for the RMI task compared to the other two tasks, potentially reflecting a left-lateralized event-related desynchronization characteristic of RMI tasks [40, 55]. The same pattern does not appear as distinctly prior to observed improvements in BCI control (e.g., session 5), thereby suggesting that some of the performance increases were due to the emergence of more discriminating cortical activity. Similarly, the increase in T7 and CP5 signal power during latter stages of the trial for the CWG task in session 7 align with previously observed EEG activation patterns for word-based mental tasks [41]. In contrast to session 7, the timeseries for session 8 do not reflect these expected EEG activation patterns. Instead, the higher power in all three channels in the active tasks compared to mental rest is again indicative of task-related EMG activity [53].

Fig. 12.

Channel-wise PSD Fisher scores for right-hand motor imagery (top row) and categorical word generation (bottom row) versus mental rest for each session. Note sessions 2 and 3 are omitted for the bottom row because the categorical word generation task was not introduced until session 4. Scores listed below each topographic map indicate mean Fisher scores across all channels. Positive and negative signs of the fisher scores within the topographic maps indicate where median power levels were higher or lower, respectively, in active tasks versus mental rest. All scores for topographical visualization were normalized to be within [- 1, 1]

Participant experience

Figure 14 depicts the results of the QCM-BCI, NASA-TLX, and Subjective Experience surveys. QCM-BCI results were consistent throughout the sessions and indicated favorable motivation scores for BCI training [50] with high interest in the training, high confidence that he would be successful in learning how to use the BCI, strong belief that the training would be challenging, and moderate-to-low fear of incompetence.

Fig. 13.

Z-score normalized median power for electrodes C3 (top row), CP5 (middle row), and T7 (bottom low) throughout mental imagery trials for the three blocks in sessions 5 (left column), 7 (middle column), and 8 (right column) with the highest classDistinct scores. Shaded region represents median absolute deviation

Notably, the NASA-TLX frustration and performance (lower performance scores indicate better performance) scores peaked within sessions 3–5 when the participant was struggling to make significant improvement. Scores for both of these scales decreased in session 6–8 when the participant began achieving more favorable feedback results with the single-pattern state system. Following the feedback change, there was a slight increase in the mental and temporal demand reported by the participant; however, both of these decreased by session 8. The initial increase may have been a response to having to perform the tasks while adapting to and learning a new feedback interface.

Finally, the Subjective Experience survey results indicated that the participant generally found sessions 6–8 with single-state feedback less difficult and fatiguing than sessions 2–5. The decrease in difficulty and fatigue aligns with the downward trends in mental and temporal demand in NASA-TLX responses for sessions 6–8. Indeed, the participant indicated after session 8 that he felt it was becoming far easier for him to transition from MR to the active tasks, something that he believed he struggled with earlier in the training. While the participant reported enjoying the training throughout the sessions, he did indicate after session 6 that “making progress made it more fun.” At the end of the final session, the participant shared that what he enjoyed most about the training was experimenting with different mental task techniques and finding which approaches yielded the best results.

Fig. 14.

QCM-BCI (left column), NASA-TLX (middle column), and Subjective Experience survey (right column) results for each training session. The participant received multi-state feedback during sessions 2–5 and single-state feedback during sessions 6–8. No feedback was provided during session 1

Discussion

Analysis of both online feedback and offline post-training task discriminability-based measures suggested that the participant achieved increases in the machine-discernibility of physiological signals associated with active mental tasks versus their resting-state baseline over the course of the training sessions, particularly in sessions 6–8. Post-training analysis suggested that a prominent component differentiating mental task trials from resting state were EMG signals originating from low intensity activation of scalp muscles. Consequently, while conclusions regarding the effectiveness of the training protocol and feedback for training purely EEG-based BCI control cannot be drawn from the results of this study, the feedback did appear to support the development of multi-model physiological control signals. Implications of these observations and relevant lessons drawn from this case for EEG- and hybrid-BCI training are discussed below.

Merits of single-state feedback

The increase in online feedback and longitudinal signal discriminability results beginning in session 6 when the single-state feedback was introduced indicated that this feedback provided a platform for the participant to explore and identify task strategies that yielded machine-discriminable signals. Although the offline analysis revealed that the signals within sessions 6–8 were of mixed neuro-cortical and cranial-muscular origin, the participant appeared to ably utilize the feedback to improve during these sessions.

While the observed increases in task discriminability coincided with the change to single-state feedback in session 6, the structure of this case study (i.e., changes were introduced in response to case study particpant difficuly, not within a controlled experiment) prevents exclusive attributation of performance improvements to feedback modality change. Indeed, effects such as accumulated practice and task habituation cannot be controlled for in this case study and it is unclear to what degree such factors contributed to the progression of the task discriminability. Furthermore, changes observed in the user’s experience (fatigue, cognitive demand) captured via the Subjective Experience survey and NASA-TLX cannot be causally attributed to the introduction of the single-state feedback system.

Nonetheless, the observations indicate that single-state feedback may be effective in early stages of training where users are exploring different task approaches and benefit from indication when signals deviate from baseline activity. Indeed, the increased discrimination between the MR and the active tasks (Fig. 10b) supports that the feedback reliably communicated when the user’s neurophysiological activity deviated from or resembled the MR baseline and the user was able to leverage that information to improve the discriminability between MR and the two active tasks.

The identification of task strategies that yield EEG signals that significantly differ from a users’ basal state is a significant challenge for training users with initially low performance [4, 34, 58]. This study’s longitudinal task-discriminability observations and the participant’s favorable subjective experience during sessions 6–8 indicate that the proposed training and feedback approach warrant further exploration. Given the influence of EMG activity in this study, it would be prescient to directly measure cranial muscle activity in future research to determine the extent to which EEG signal changes reflect muscle rather than cortical activity and to provide this information as an additional layer of user feedback.

Challenges with multi-pattern state feedback

Meaningful task discriminability improvements were not observed during sessions 2–5 when the participant received the multi-pattern state-based feedback. Further investigation would be required to more rigorously assess the efficacy of the multi-modal feedback for initially low-performance users. Nevertheless, the observations from this study provide valuable insights for future evaluations of this and similar feedback systems.

As discussed in the brief review of other feedback approaches for BCIs [17, 30, 31], the theoretical value of such feedback approaches is to provide a stable representation that yields similar outcomes when users apply the same mental task approach. This stability provides the foundation for users to learn associations between the observed feedback outcome and their task execution approach. Indeed, a primary design objective for this feedback methodology was to provide this stable representation and indicate to the user when they were producing similar EEG patterns for multiple tasks. However, the combination of minute inter-state differences and the high frequency of intra-trial state transitions (Fig. 7) likely obscured this information from the user.

During sessions 2–5, inter-state boundaries likely failed to reflect signal states that the user could consciously modulate between. Consequently, although most trials consisted of signals with similar patterns, representing the trials as sequences of multiple discrete states likely misleadingly implied the presence of significant inter- and intra-trial signal variations. Similarly to how poorly-conditioned classifiers produce feedback that users can struggle to interpret because associations between action and classifier output are challenging to identify [28, 29], the results of this study suggest that discrete pattern state transitions that represent minute signal changes may be equally challenging for users to interpret.

Furthermore, the high frequency of intra-trial state transitions may have falsely conveyed an ability to consciously modulate EEG signals. To the participant, this may have created the illusion of conscious actions yielding meaningful modulations in the EEG signals, when, in reality, inter-state differences were relatively minute and transitions were prone to occur in response to spurious signal activity. Even in cases where the multi-state feedback could have plausibly been reflecting an expected neurophysiological response (i.e., RMI task in Fig. 7, the shift from S3 to S2 possibly reflecting a event related desynchronization followed by a synchronization), the pattern sequence was likely obscured by these states occurring with similar frequency during the other tasks.

These observations suggest that deployment of multi-state feedback for individuals experiencing BCI inefficiency should be deferred until a user demonstrates the ability to consistently produce consciously-induced signals that deviate from baseline activity (see next section). Note that deferring use of multi-state feedback until consistent deviation from baseline can be achieved would not necessitate that the states within the model be uniquely associated with a single task (as is typical with classifier feedback). Multi-state models would retain the flexibility to identify and communicate patterns that are produced during multiple tasks.

Another related challenge with multi-state feedback exposed during this case were unstable variations in the pattern states between sessions. Because the participant did not produce discriminable EEG patterns during the early sessions, outlier patterns were more likely to be identified during the K-means clustering process. However, these outlier patterns were unlikely to be repeated during subsequent sessions (e.g., S1 in Fig. 7), reducing the number of states available to represent meaningful differences within typical EEG activity. For users experience BCI inefficiency, risk of these outlier states emerging during K-means analysis is heightened due to the lack of robust inter-task signal differences. This further supports deferring multi-state feedback until conscious modulation abilities are demonstrated. Additionally, while the risk of outlier states is less prevalent for higher-performance users, identification of outlier states remains possible. Therefore, methods to discern the physiological origins of identified pattern states and prune such outlier states from the model should be developed in future work.

Combined, the lack of identifiable associations between states and task actions and the high frequency of state transitions likely contributed to the heightened frustration, performance, fatigue, and difficulty ratings reported by the user in the NASA-TLX and Subjective Experience surveys during sessions 3–5. The increase in these factors combined with the high cognitive load required to try and identify potentially non-existent task-state associations with the multi-state feedback model could reduce the likelihood of achieving BCI control [59–62] and increase the likelihood of device abandonment [18, 63]. Ultimately, the combination of these observations suggest that multi-state feedback may not be an optimal feedback modality for initial training for users experiencing BCI inefficiency. Instead, an incremental training and feedback approach that builds towards multi-state feedback would be more appropriate.

Progressive feedback: single- to multi-pattern state

In future implementations of the proposed training protocol for users with initially low BCI performance, a more effective training strategy may be to begin with single-state feedback before transitioning to multi-state feedback. Once the user demonstrates an ability to produce signals that deviate from their baseline activity with single-state feedback, the transition to multi-state feedback could be made to enable identification of strategies that yield different patterns for each mental task. Such an approach could mitigate against the challenges and risks identified with the multi-state feedback discussed in the previous section. Similarly, transitioning to the multi-state feedback once the user demonstrates ability to reliably deviate from baseline would overcome the primary limitation of the single-state feedback: positive reinforcement was provided whenever the participant produced signals that deviated from baseline activity, regardless of whether they aligned with any task-specific pattern template. While this may be appropriate in early stages of learning, it provides limited opportunity for the user to receive feedback on the discriminability between active tasks. In This study, this limitation likely contributed to the lower classifier discrimination observed between the RMI and CWG tasks compared to their individual discrimination versus MR (Fig. 10b). Progressing to multi-state feedback would add degrees of freedom for the feedback to indicate when active tasks produced similar signals.

Spontaneous emergence of multimodal neurophysiological control

As presented in the “Evidence of implicit learning of multimodal physiological control” section, EMG signals from subtle cranial muscle activations appeared to contribute meaningfully to the participant’s ability to produce signals deviating from baseline activity. Prior studies investigating EEG activity before and after neuromuscular paralysis have demonstrated significant presence of EMG in EEG frequencies exceeding 20 Hz in relaxed states due to tonic muscle activation [64–66]. These EMG signals become more prominent during scalp and jaw muscle contractions [53] that are common responses to increased concentration or performance of a challenging task [67, 68]. A likely explanation for the performance changes observed in this study, therefore, are that the participant increased his state of focus and concentration during the active mental imagery tasks in later sessions. Consequently, he inadvertently introduced EMG resulting from subtle muscle activations associated with increased focus and concentration. With the improvement indicated by the online feedback, the participant likely conflated the increased concentration, and its associated muscle activity, with improved mental task performance further reinforcing concentration strategies that induce muscle tension.

While the EMG activity aided the participant in controlling the BCI and appeared to increase his overall experience and enjoyment of the training, there are potential limitations to EMG-based control schemes. McFarland et al. [68] observed that individuals relying on EMG activity for BCI control during early training sessions struggled to maintain control in later sessions, potentially because they were unaware of their initial reliance on EMG. As exemplified by the oscillating levels of EMG present between sessions 6–8 in this study, it is likely that the participant was similarly not aware of the influence of EMG activations and not consciously attempting to modulate EMG activity to achieve control. It is unclear, therefore, whether the EMG activations that the participant successfully utilized within session 8 represent a learned activation that would be consistently repeated in future sessions. Additionally, while EMG alone can be reliably used to produce discriminable binary states, it may prove more challenging for discrimination in cases with more than two classes. Finally, EMG would not a viable control modality for individuals with muscle disorders, such as amyotrophic lateral sclerosis or dyskinetic cerebral palsy; two populations of individuals who could benefit from BCIs [53].

Implications for hybrid BCI approaches

For individuals with the ability to consciously control their cranial muscles and who display low task-discriminability of EEG signals, a hybrid EEG-EMG approach may be a viable strategy to enable reliable activations and keep users engaged during longer-term training. Indeed, previous BCI training studies have found that users with initially low performance (i.e., near-chance level classification accuracy) can struggle to significantly improve their performance with practice [3, 6]. Sustained low performance combined with the mentally taxing experience of BCI training would likely lead to disinterest and abandonment of the technology [18], particularly within pediatric populations [63]. The increased frustration and perceived training difficulty reported by the participant in session 5 of this study may have reflected early indications of waning interest due to the low performance and lack of clear progression.

In such cases, it may be beneficial for the user to utilize cranial EMG activations as an alternative, secondary control modality to revert to when the user cannot reliably produce discriminable EEG activations [69–71]. For example, if utilizing gamified training feedback (e.g., BrainDriver) users could revert to an EMG activation if their initial attempt at EEG-based control is unsuccessful. Such an approach would permit users to continue practicing EEG-based control while still enjoying and actively engaging in the game experience. Similar hybrid approaches combining mental imagery and other modalities like steady-state visually evoked potentials have enhanced BCI accuracies while users develop mental imagery proficiency [72]. Other approaches have combined EEG and EMG from muscles in the extremities [73, 74] to improve reliability. The multi-pattern feedback presented herein could naturally be extended for such a hybrid approach as different states could be dedicated to EEG or EMG activity.

A hybrid EEG and cranial EMG approach would also be appropriate for some clinical BCI-use cases such as switch-based play and therapeutic recreation [75] where it is mostly irrelevant whether the activation is EEG- or EMG-based as long as the user can reliably produce a distinct signal. As users in these cases may utilize BCIs infrequently, they would be less likely to pursue long-term training programs to develop robust EEG-based control. Therefore, it would be imperative to quickly identify techniques that result in recorded signals that deviate from baseline. This study’s results suggest that the single-pattern state feedback could effectively provide such training.

Conclusion

This case study presented a novel mental imagery BCI user training protocol rooted in exploratory, action-perception learning and presented the experience of an adolescent with paraplegia. Initially, the participant struggled to produce task-specific signals that could be discerned above chance levels. While our initial feedback system was designed to try and address this training challenge, results within initial sessions indicated that the feedback may have been overly complex for the user and did not support his training. In response, the feedback was simplified to support the participant identifying task approaches to produce signals that deviated from baseline activity. Online results and post-training analysis indicated that the participant learned a multimodal strategy (EEG and EMG) to produce signals that deviated from baseline and that could be used to achieve above-chance-level BCI control. The observations from the case provide useful implications for further refinement and study of similar training approaches and for the development of hybrid BCIs.

Abbreviations

BCI

Brain–computer interface

CWG

Categorical word generation

EEG

Electroencephalography

EMG

Electromyography

FgMDM

Minimum Riemannian distance to the mean with geodesic filtering

FS

Fisher score

MR

Mental rest

NASA-TLX

National aeronautics and space administration task load index

QCM-BCI

Questionnaire for current motivation, brain–computer interface variant

RMI

Right hand motor imagery

PSD

Power spectral density

SN

Spatial navigation

Author contributions

NI and TC conceptualized the study. NI developed the methodology and software, conducted the data collection sessions, analyzed the data and wrote the initial draft of the manuscript. TC provided supervision and reviewed and edited the manuscript. All authors reviewed and approved the final manuscript.

Funding

Nicolas Ivanov received funding from an NSERC Canada Graduate Doctoral Scholarship, an Ontario Graduate Scholarship, and The Kimel Family Graduate Student Scholarship in Pediatric Rehabilitation Award from the Bloorview Research Institute, the Holland Bloorview Kids Rehabilitation Hospital Foundation and the University of Toronto.

Data availability

Data is not made publicly available because the participant did not provide consent to public sharing of his anonymous data. Reasonable requests for sharing of the data can be made to the corresponding author.

Declarations

Ethics approval and consent to participate

The study was conducted in accordance with the Declaration of Helsinki (2013) and approved by the Research Ethics Boards at the Holland Bloorview Kids Rehabilitation Hospital (protocol number 0697) and the University of Toronto (protocol number 00039652). The participant gave written informed consent to participate in this study.

Consent for publication

The participant gave written informed consent for publication of their data as presented in this article.

Competing interests

The authors declare no competing interests.