Noninvasively recorded high-gamma signals improve synchrony of force feedback in a novel neurorehabilitation brain-machine interface for brain injury

Abstract

Objective.

Brain injury is the leading cause of long-term disability worldwide, often resulting in impaired hand function. Brain-machine interfaces (BMIs) offer a potential way to improve hand function. BMIs often target replacing lost function, but may also be employed in neurorehabilitation (nrBMI) by facilitating neural plasticity and functional recovery. Here, we report a novel nrBMI capable of acquiring high-γ (70–115 Hz) information through a unique post-TBI hemicraniectomy window model, and delivering sensory feedback that is synchronized with, and proportional to, intended grasp force.

Approach.

We developed the nrBMI to use electroencephalogram recorded over a hemicraniectomy (hEEG) in individuals with traumatic brain injury (TBI). The nrBMI empowered users to exert continuous, proportional control of applied force, and provided continuous force feedback. We report the results of an initial testing group of three human participants with TBI, along with a control group of three skull- and motor-intact volunteers.

Main results.

All participants controlled the nrBMI successfully, with high initial success rates (2 of 6 participants) or performance that improved over time (4 of 6 participants). We observed high-γ modulation with force intent in hEEG but not skull-intact EEG. Most significantly, we found that high-γ control significantly improved the timing synchronization between neural modulation onset and nrBMI output/haptic feedback (compared to low-frequency nrBMI control).

Significance.

These proof-of-concept results show that high-γ nrBMIs can be used by individuals with impaired ability to control force (without immediately resorting to invasive signals like ECoG). Of note, the nrBMI includes a parameter to change the fraction of control shared between decoded intent and volitional force, to adjust for recovery progress. The improved synchrony between neural modulations and force control for high-γ signals is potentially important for maximizing the ability of nrBMIs to induce plasticity in neural circuits. Inducing plasticity is critical to functional recovery after brain injury.

Introduction

Brain injury from trauma or stroke is a leading cause of long-term disability worldwide, constituting a major public health concern (Dewan et al. 2018, Virani et al. 2021). For the individual, loss of fine motor control following a brain injury can restrict one’s ability to carry out activities of daily living (ADLs) independently. Many ADLs depend on the ability to precisely regulate grasp force; unfortunately, brain injuries can compromise one’s ability to generate and hold grasp force. To treat this and other deficits arising from brain injury, there is growing interest in approaches that use some form of neuromodulation to induce activity-dependent plasticity, as a supplement to traditional physical therapy (Soekadar et al. 2015, Krucoff et al. 2016). In particular, such approaches could create a pathway for individuals to participate in neurorehabilitation even if they lack any residual movement. Brain-machine interfaces (BMIs) can translate motor intent to action through the direct interpretation of brain signals sampled from motor cortices (Slutzky and Flint 2017, Slutzky 2019). BMIs have largely been designed to replace lost function, for example through control over robotics (Hochberg et al. 2012, Collinger et al. 2013, Balasubramanian et al. 2017) or functional electrical stimulation (FES; Do et al. 2011, Ajiboye et al. 2017). However, BMIs can also be used to help induce plasticity and facilitate neurorehabilitation (Ramos-Murguialday et al. 2013, Bundy et al. 2017, Biasiucci et al. 2018). Here, we introduce a novel neurorehabilitation BMI (nrBMI) that enables users to exert continuous, proportional control of thumb force while receiving haptic feedback that is tightly synchronized to the onset of motor intent. This nrBMI design allows control to be shared between decoded intent and voluntary (active) force, to maximize engagement and promote recovery. The current results provide noninvasive proof of concept in support of a future high-γ driven nrBMI that will be implemented with invasively acquired signals, such as subdural surface potentials (electrocorticography or ECoG).

There are multiple ways that nrBMI-based therapies could help improve function. Feedback from nrBMIs can be provided on attempted motor execution, even without detectable movement or force. This in itself provides a powerful motivator and enabler of learning, since learning and recovery generally benefit from sensory feedback (Vidoni et al. 2010, Abela et al. 2012). Also, nrBMIs potentially can enable targeting of plastic changes to specific brain areas. Perhaps most relevant, previous work has shown that functional gains after stroke are possible using nrBMI control of an exoskeleton (Ramos-Murguialday et al. 2013, Hortal et al. 2015, Bundy et al. 2017, Frolov et al. 2017) or FES (Biasiucci et al. 2018). See also Cervera et al. (2018) and Carvalho et al. (2019) for meta-analyses indicating an overall benefit for nrBMI training after stroke.

In spite of the evidence that nrBMI use can be beneficial, challenges remain before nrBMIs can be widely used in clinical settings. One challenge regards the timing of feedback. Rehabilitation of function through Hebbian-type neural plasticity ultimately depends upon synchrony between pre- and post-synaptic neurons (Caporale and Dan 2008), so it will be crucial to synchronize feedback to motor intent. However, to date nearly all nrBMIs have relied on electroencephalography (EEG) or MEG, using relatively low-frequency (<40 Hz) portions of the spectral domain. This can result in feedback delays on the order of 1 s (King et al. 2014). Such delayed feedback may be one reason why it remains unclear whether low-frequency signals can foster the plasticity required for long-term functional gains (Bai et al. 2020). By contrast, high-γ band (>70 Hz) power has greater temporal resolution than low-frequency signals (Crone et al. 2001, Wang et al. 2016), which should enable greater synchrony, and therefore induce more plasticity. In addition, high-γ has more movement-related information than μ–β band, as shown with intracortical recordings (Stark and Abeles 2007, Zhuang et al. 2010, Flint et al. 2012a, Flint et al. 2012b) as well as ECoG (Ball et al. 2009, Pistohl et al. 2012, Flint et al. 2014). Thus, we hypothesized that employing high-γ for nrBMI control would allow users to exert continuous, proportional control over motor execution, instead of relying on classification of discrete command signals. This could enable (for example) an nrBMI that continuously decodes graded finger joint kinematics or grip force in real time, instead of classifying a binary, “open/close” grasp command. In an nrBMI, proportional control during therapy could be important to successfully inducing neural plasticity, by aligning feedback with expectations, both in timing and intensity. It also might enable recovery of more fractionated movement than is possible with discrete, binary classification methods.

High-γ signals can be obtained with implanted electrodes, at intracortical, subdural, or epidural levels, but high-γ is severely attenuated by the skull in traditional EEG. Here, we present a novel nrBMI that uses high-γ power from hEEG to control haptic feedback to the thumb in an isometric grasp-like force task. The hEEG platform enables us to prototype an nrBMI using high-γ before using implanted electrodes. We and others have previously shown that high-γ motor information can be acquired using EEG recorded in the presence of a hemicraniectomy (hEEG; Voytek et al. 2010, Vaidya et al. 2019, Li et al. 2020b). Hemicraniectomy procedures are typically performed with individuals following severe traumatic brain injury (TBI) to prevent brain herniation. Our research participants included individuals with varying degrees of hand weakness following TBI. The ability to accurately regulate force is a generally underserved area in BMI research (although see Downey et al. 2018, Flint et al. 2020, Rastogi et al. 2020), and the importance of force to functional grasp means that filling this gap is an important step toward clinically useful BMIs.

In this study, individuals with TBI were able to successfully control the nrBMI while receiving haptic feedback that was time-locked to the onset of force intent. Feedback control was shared between active (or attempted) force and decoded intent. We term this nrBMI-shared control. Importantly, we found that the synchronization of sensory feedback to motor intent was significantly tighter when using high-γ based nrBMI-shared control, compared to μ–β based nrBMI-shared control. These results provide a first step towards developing nrBMIs that sample high-bandwidth neural signals with high temporal precision for targeted therapy in a clinical setting.

Methods

Our ultimate goal is to develop an nrBMI that facilitates the recovery of function by promoting neural plasticity and helping users re-acquire motor skills that were lost after brain injury. In this study, we developed, tested, and validated a novel nrBMI at Northwestern University, the University of California, Irvine, the Illinois Institute of Technology, and the Shirley Ryan AbilityLab in Chicago. All experimental protocols were approved by the respective Institutional Review Boards.

Target clinical population

We worked with human participants in two groups: (1) individuals with no brain injury and unimpaired motor function, and (2) individuals who had undergone decompressive hemicraniectomy following severe TBI. The inclusion criteria for group (2) were: minimal hyperalgesia, allodynia or neglect in the hand/arm (at least some residual pressure-sensation function was needed in the thumb); ability to tolerate wearing an EEG cap for a 2-hour session; ability to understand and follow instructions; and ability to attend to a task for at least five minutes at a time. We emphasize that weakness in the hand was not an exclusion criterion; our post-TBI participants included an individual with a score of 0 on the Medical Research Council muscle strength scale (MRC; No 1976), defined as “no visible contraction” of the thumb. Here, we present validation data from 3 participants with TBI (2 female; ages approximately 25–65, designated T1 through T3) and 3 control participants (3 male; ages approximately 20–30, designated N1 through N3).

nrBMI novel neurorehab platform: overview

The brain signals we acquired were EEG or hEEG. We specifically focused on modulations in high-γ power that occur with isometric thumb force (or attempted force). We provided haptic feedback using linear actuators to apply pressure (isometric force normal to the participant’s thumbnail) that was controlled by the intended force. We used frequency decomposition (via short-time Fourier transform) to calculate high-γ power in real time and decoded intended force continuously. The nrBMI translated this intended force into a haptic feedback force, applied it to the thumb, and measured both total force and haptic feedback force continuously. A visual feedback display was available to the participants, showing both task goals and a visual representation of instantaneous total force applied. Intact-skull participants used μ and β power to control the nrBMI. We envision a use-case where an individual with reduced hand function uses the nrBMI to gain assistance in generating force, while receiving sensory feedback synchronized with their intent. Over time, with improvements in hand function, the degree of nrBMI assistance could be reduced, to facilitate strength-building and a path towards independent function.

EEG/hEEG acquisition

We acquired EEG or hEEG signals using a 128-channel actiCAP (BrainVision, Inc) together with a 128-channel Neuroport Multineuron Acquisition Processor (Blackrock Microsystems, Inc.). The actiCAP system embeds EEG electrodes with active electronics in a stretchable fabric cap. To optimize the signal quality in our recordings, we measured electrode impedance in real time at the beginning of each recording session, while applying gel. We targeted an impedance level of <25 kΩ for each electrode in these sessions. After applying gel, we wrapped the cap in kerlix gauze. In hEEG recordings, we placed gauze sponges above the craniectomy (below the kerlix), to maintain electrode-skin contact. A rendering of the EEG cap fitted to a participant with a hemicraniectomy (Structure Sensor Pro, Occipital, Boulder, CO) is shown in figure 1(a). The EEG and hEEG signals were lowpass filtered (500 Hz) and sampled at 2000 Hz. For participants with TBI, we removed 2–3 electrodes from the peripheral channels of the head cap, using them to record EMG at the trapezius and masseter muscles in preparation for an EMG-removal procedure (see ‘Offline analysis’, below).

Figure 1.

Overview of the neurorehabilitation BMI for force. (A) Participant T1 with electrode locations marked. The extent of the craniectomy is shown with a black outline. (B) The haptic feedback device. White arrows indicate sensor locations for F_active (force at palmar side of thumb, see main text) and F_haptic (external force applied to dorsal thumb surface), as well as the linear actuator. (C) Individual frames illustrating the progression of visual feedback during a trial. The target force is shown in the top half of the thumbnail image, while the current applied force (ranging from cyan to purple for low to high force, respectively) is shown in the base of the thumbnail. Target force (high force in this trial) is also shown around the outer border of the screen, except during rest periods, denoted by white (first image). Acquiring the target force level resulted in a yellow outline around the thumbnail. Successfully holding in the target for the hold duration produced a green success symbol. (D) System overview during nrBMI-shared control operation. Intended force was calculated by the nrBMI from brain signals, ultimately generating the haptic output F_BMI. F_haptic (generated by F_BMI) and F_active (generated by voluntary force) combined to generate F_total (main text, equation 1), which controlled visual feedback. During hand control, F_haptic=0 and visual feedback was controlled by F_active only.

Haptic device

In the experimental protocol, participants applied (or attempted) force with the thumb’s palmar surface to match a random force level target (details in ‘Visual feedback during hand control’, below). Accordingly, we designed and constructed the haptic feedback device (figure 1(b)) to perform three main functions: (1) sense force applied by the palmar surface of the thumb (F_active); (2) apply pressure to the dorsal surface of the thumb (i.e., the thumbnail), forming an “assisted” pinch; and (3) sense the external force applied to the thumbnail, called F_haptic. We used two independent load cells (1 degree of freedom FC20–10kg, Forsentek, Shenzen, China) built into custom fittings in the haptic device to sense F_active and F_haptic. We produced haptic force feedback using a linear actuator (pq12, Acutonix, Victoria, Canada; figure 1(b)) mounted alongside two linear rails (Thompson, Radford, VA). This assembly was capable of applying a maximum of approximately 50 N feedback force to the thumb (Barry et al. 2018). The entire housing of the haptic device could be rotated, to maintain a comfortable and consistent wrist and forearm angle (45° internal forearm rotation, 0° wrist flexion). We designed and built custom electronics to amplify and filter the load cell output, power the actuators, and electrically isolate the haptic device’s power and input/output from the participant and data acquisition equipment. This was done for participant safety, and to prevent signal contamination from mains power.

System calibration - force

This section of Methods, and the following four sections are laid out in order of a typical experiment. For all participants, we began experimental sessions by calibrating both force sensors (F_active and F_haptic), as well as the ergonomic limits of the force that could be applied by the BMI (F_BMI) for feedback. The participant’s thumb was fixed in place for the duration of the experiment, by means of an adhesive Velcro sticker. During F_active sensor calibration, the participant was instructed to apply (or attempt) a firm pressure on the load cell (maximum voluntary force, MVF), holding it for approximately 1 s and releasing. Then, F_BMI and F_haptic were calibrated simultaneously. The linear actuator was advanced until the feedback applicator (the part of the device that applied force to participant’s thumb) lightly but continuously contacted the thumbnail. From there, the haptic applicator was advanced in small increments until just before the participant reported discomfort. This established the lower and upper limits of displacement for the haptic feedback device during nrBMI-shared control (described in Software architecture: nrBMI-shared control, below).

Visual feedback during hand control

As in previous work (Flint et al. 2020), we used a behavioral task inspired by random-target pursuit tasks used in studies of reach kinematics. Visual feedback was presented to the participants on a monitor placed just above their hand. During hand control mode, the haptic feedback device did not apply force (i.e., F_BMI=0 and F_haptic = 0 throughout). The participant’s voluntary force level (F_active) controlled the color of a section of a monitor’s screen (figure 1(c)). The screen area being controlled was superimposed on a static image of a human thumb to encourage gaze fixation: the color of the thumbnail changed with the amount of force applied. The color scale ranged from cyan (low force) to purple (high force; figure 1(c)). All aspects of visual feedback were controlled by a custom-designed Python application, including randomization of target level and the internal state machine governing progress through the trial. Each trial consisted of a target force presentation with a timeout of 5 s. If the applied force matched within 10% of the target force level, the participant was cued to hold that level for at least 500 ms (figure 1(c)). A Success trial required matching the force level to the target level throughout the hold period, which increased the participant’s score and was paired with a video-game like reward sound and visual cue for motivation. Exiting the target force level prior to the end of the hold period necessitated re-acquisition of the target and holding for a new 500 ms period (i.e., transitory matches of the target force level were not Successes). Each Success (or Failure, due to timeout) was followed by a 4 s inter-trial interval marked by a white border, during which participants were instructed to relax. For T1 (who was unable to exert voluntary hand force), instead of random force level targets we issued instructions to attempt 3 levels of force: light, medium, and firm thumb pressure. In this case, the visual feedback was not used for hand control, and we instructed the participant to concentrate on force attempts. Hand control recordings were used for building decoders (see next section), and for offline analysis.

Decoder construction

We prepared to conduct nrBMI-shared control experiments by using a fast decoder building procedure to prepare a decoder from hand control recordings on the same day (usually requiring <5 minutes to calculate). For N1-N3, we built decoders from EEG electrodes in rows F to P (10–5 labeling system; Oostenveld and Praamstra 2001) on the contralateral side of the hand used in testing (side to test chosen randomly). For T1-T3, we built the decoders from hEEG electrodes in rows F through P that were also located over the hemicraniectomy. In both participant groups, we manually rejected electrodes with high-amplitude artifacts contaminating the EEG signal. We performed a common-average re-reference on the remaining set. We then used frequency decomposition, similar to Flint et al. (2013), to generate frequency-domain features from the preprocessed signals. Briefly, we performed short-time Fourier transforms, followed by log normalization to calculate the power in frequency bins from 0 to the Nyquist frequency (sampling rate 2 kHz). We subtracted the mean (over the entire recording) from each frequency bin, then averaged across the band(s) of interest. For N1-N3, we used both μ (7–12 Hz) and β (15–35 Hz) bands. For T1-T3, we used a single high-γ (70–115 Hz) band. We used these spectral features, along with force generated during hand control, to train a Wiener cascade decoder, which consisted of a linear Wiener filter (with ridge regression) convolved with a static nonlinearity (Korenberg and Hunter 1986). We built multiple Wiener cascade decoders on the hand-control dataset using 10-fold cross-validation, saving the highest-performing decoder for use in nrBMI-shared control.

Visual feedback during nrBMI-shared control

During nrBMI-shared control, visual task presentation was the same as in hand control: randomly selected force targets were presented as colors in a spectrum (see figure 1(c)). Here, control over visual feedback was shared between the participant’s voluntary force F_active (when able), and nrBMI-controlled force measured by F_haptic (figure 1(d)). Visual feedback in nrBMI-shared control was governed by the equation

$$F_{\mathit{\text{total}}}=\mathit{\text{MVF}}(\epsilon F_{\mathit{\text{active}}}+(1-\epsilon )F_{\mathit{\text{haptic}}})$$ (1)

where color was directly controlled by F_total, MVF (maximum voluntary force) was a scale factor accounting for the maximum force the participant could exert (as measured during calibration, see ‘System calibration - force‘ above), and ε was the degree-of-sharing parameter. At ε = 1, control over the force color was the same as in hand-control mode; at ε = 0, the force color was entirely governed by the nrBMI output (scaled decoded force intent, or F_BMI in figure 1(d), see also next section). The shared-control paradigm allowed the nrBMI to be tuned to the individual’s level of recovery, with higher proportion of control allotted to nrBMI output (ε closer to 0) for those with less ability to generate F_active on their own. Thus, with changing ability (e.g., improvement of strength over time), the sharing parameter could be increased to require a greater degree of hand control, and thus increase the physical challenge to build strength. In these experiments, ε was initialized according to each participant’s hand strength using the MRC scale (0=no visible contraction, 5=normal strength). We set ε to 0.1 for an MRC score of 0; otherwise, we set ε=MRC/10, where MRC was the participant’s muscle strength score. The study was not designed to systematically vary ε, in part due to the limited time available with each participant (see Results).

Software architecture: nrBMI-shared control

The nrBMI was designed to provide haptic feedback that could be controlled by some combination of active force and modulation of brain signals. During nrBMI-shared control, the decoder output (F_intent in figure 1(d)) was passed via UDP interface to a custom-built virtual instrument (LabVIEW, National Instruments). There it was scaled according to participant-specific calibration of ergonomic force limits (see above ‘System Calibration - force‘) to generate the nrBMI output (F_BMI). F_BMI was then appropriate to drive the linear actuator directly. F_active and F_haptic were sensed by the haptic feedback device’s built-in sensors, and those signals drove the visual feedback for the participant (see previous section) via UDP communication between BCI2000 and the custom Python visual-feedback application. Overall, the system updated at a rate of 20 Hz (i.e., a bin size of 50 ms).

Quantification of nrBMI-shared control performance

The success rate for nrBMI-shared control performance was the fraction of force targets that were successfully acquired. We also calculated a time-to-target (TTT) metric on Success trials, as the elapsed time between target presentation and the earliest time that the participant-controlled F_total matched within 10% of the target force. We note that for nrBMI-shared control, participant T1 (who lacked the ability to exert voluntary force) could be evaluated for performance using the same criteria as participants with healthy force-generating abilities. In addition to overall success rate, we also calculated success rate in a sliding window of 50 trials, allowing us to examine learning of the task over time.

Offline analysis: detecting modulation of high-γ in hEEG

To understand how high-γ modulation could drive BMI execution, we analyzed offline the signals recorded during BMI-shared control sessions. First, we cleaned the hEEG of electromyogram (EMG) artifact—a known complication when recording high-frequency EEG signals. We applied a technique called Electromyogram Removal by Adding Sources of EMG (ERASE) (Li et al. 2020a) to clean the hEEG. Briefly, this approach models EMG artifact in the hEEG signals by comparing it to real EMG (for example, collected from masseter or trapezius muscle) using independent components analysis. Independent components (ICs) identified as EMG are then subtracted, so the cleaned hEEG signal can be recovered from the remaining ICs. After cleaning hEEG using ERASE, we calculated the power amplitude as described above (see above, ‘Decoder construction‘) and examined high-γ modulations in each electrode, averaged across trials in a window around force onset.

Validation: analysis of feedback lags for μ–β vs. high-γ control

We analyzed the delays between the onset of the participants’ neural modulations and the onset of force generated by the nrBMI. We explored two relevant types of delays. First, we examined delays inherent to the nrBMI itself, i.e., the machine (computational and mechanical) elements of the brain-machine interface. Second, we quantified the lag associated with the type of neural signal (high-γ or μ–β) being used to control the nrBMI.

To establish a lower bound for inherent nrBMI delays, we tested the nrBMI step response. We performed two step response experiments; in each case using a simplified decoder that passed through a step input (5 V) with minimal processing, such that F_BMI was also a step. In the first experiment, we used a rigid block to transmit force applied by the haptic device (the F_BMI step output) directly to the F_active load cell. This simulated an idealized user-in-the-loop mode of nrBMI operation. In the second step response experiment, a volunteer’s thumb was positioned in the haptic device and the step command from the nrBMI applied force to the thumbnail. The difference between the delays from the two step-response experiments revealed the amount of delay related to thumb compliance.

The second type of delay we examined was the lag in the detection of force intent from the hEEG (or EEG) features during nrBMI-shared control. Quantifying this lag allowed us to test our hypothesis that high-γ control results in greater temporal precision than μ–β control. To do this, we analyzed the data from participants who used high-γ features for nrBMI-shared control, compared to data from those who used μ–β features for nrBMI-shared control. Analysis of hEEG data began with cleaning via ERASE as in the previous section. Then, we found the onset of neural modulation in target electrodes (see next paragraph) and measured the lag between the neural modulation onset time and the time of F_active onset (figure 2).

Figure 2.

Calculation of feature modulation onset time for an electrode used in high-γ nrBMI-shared control. The ERSP, or trial-averaged spectrogram (colormap) was aligned to F_active onset (time 0). The high–γ ERFP (white-on-black line) was the trial-averaged mean spectral power in the 70–115 Hz frequency band. F_B and F_R were calculated from the dark grey and light grey boxes, respectively. Sliding these windows across the time course of the ERFP, we calculated IOM = F_R – F_B at each time point (IOM curve not shown). The peak value of IOM was the onset time for the ERFP. In this example, the calculated onset of ERFP modulation was −0.4s.

To find the neural modulation onset time for an electrode, we first calculated the event-related feature perturbation (ERFP), which is found by averaging the overall event-related spectral perturbation (ERSP; Makeig 1993) across a frequency band of interest (high-γ or μ–β). ERSPs and ERFPs were calculated in a window of −1 to 0.1 s, aligned to Factive onset. From the ERFP, we calculated an index of modulation IOM = F_R – F_B, where F_B and F_R were the mean ERFP values in “background” and “response” segments, respectively. These were consecutive 0.25s sliding windows in the ERFP. We calculated IOM every 0.05 s. We inverted the sign of IOM for μ–β features, which decrease in power with motor onset. The neural modulation onset was taken to occur at the peak value of IOM, defining the lag due to nrBMI detection of force intent. See figure 2.

Results

We recorded hEEG from three individuals who had undergone decompressive hemicraniectomies following TBI (participants T1,T2,T3). We excluded three individuals due to cognitive impairment (including inability to follow simple instructions, attention deficit, and post-traumatic amnesia). We also recorded standard EEG with three control (non-TBI) participants (N1,N2,N3), using μ–β signals for BMI execution.

Modulation of high-γ in hEEG

In designing the nrBMI, we targeted high-γ because of these signals’ demonstrated capacity to carry detailed information about motor intent. We examined high-γ signals from nrBMI-shared control sessions, trial-averaged around F_active onset. Figure 3(a) shows high-γ power increase in two post-TBI participants, occurring just prior to force onset. We emphasize that it is ordinarily not possible to measure high-γ modulations in EEG signals, as exemplified in figure 3(b), which shows an ERSP from a non-TBI participant.

Figure 3.

Example spectral variation relative to F_active onset during nrBMI-shared control. (A) Example ERSPs from T1 and T3, respectively (high-γ nrBMI-shared control). (B) Example ERSP from N3 (μ–β nrBMI-shared control). Top: single-trial force traces (gray) and trial averaged force (black) aligned to force onset (vertical dashed line). Bottom: mean spectrograms across trials.

nrBMI-shared control performance

Using high-γ hEEG (or μ–β EEG) modulations for nrBMI-shared control, all six of our participants were able to match force targets successfully (Table 1). Type of control (high-γ or μ–β) was not a significant source of difference in the mean time-to-target (TTT; t-test, p=0.81).

Table 1.

Summary of nrBMI-shared control performance across all trials. Sessions are equivalent to recording days; Runs are individual recording blocks (approximately five minutes duration). Time to target (TTT) and the increase or decrease in normalized log spectral power are reported as mean ± s.d.

Participant	Sessions; Runs	MRC strength	EEG feat. used	Trials	Success Rate	TTT (s)	Norm. log power increase (decrease)	Days since injury
N1	1; 1	5	μ-β	61	100%	1.8 ± 1.5	(1.9 ± 0.4)	N/A
N2	1; 4	5	μ-β	124	60%	2.3 ± 1.7	(2.2 ± 0.3)	N/A
N3	1; 5	5	μ-β	130	90%	1.7 ± 1.3	(2.3 ± 0.4)	N/A
T1	1; 5	0	high-γ	159	28%	2.3 ± 1.7	2.1 ± 0.25	92
T2	2; 2	5	high-γ	74	99%	2.8 ± 1.0	2.1 ± 0.2	79
T3	3; 11	5	high-γ	256	60%	1.2 ± 1.2	2.3 ± 0.5	52, 54, 57

We note that participant T1, who scored 0 on the MRC strength scale (indicating no detectable active thumb flexion force), successfully used the nrBMI to exert some degree of force on a majority of trials, despite a relatively low overall success rate. We also found evidence of improved success over trials for 4 of 6 participants, including T1 (figure 4). Success rates for N1, N3, T2, and T3 all encountered a ceiling effect at 100% success and could not improve farther. Success rate for N1 and T2 began at approximately 100% and remained largely flat throughout their recording sessions.

Figure 4.

Success rate in a 50-trial sliding window. Solid lines show linear fit. Pearson’s correlation coefficient for each subject is shown in the legend. N1 and T2 had no significant correlation between trial and success rate.

The degree-of sharing parameter (ε in equation 1) was initialized for each participant according to their muscle strength on the MRC scale (see Methods). Due to a limited number of recording sessions in most participants, we were not able to systematically test effects of varying ε; instead, we varied ε empirically to maximize user engagement. There was no significant correlation between ε and success rate in these data (R=−0.2, p=0.4, Pearson’s coefficient).

Temporal precision of haptic feedback in high-γ vs. μ–β control

One goal of providing somatosensory feedback during rehabilitation is to help trigger plasticity in neural circuitry, ideally promoting re-learning or adaptation as a means to restore function. Thus, the timing of feedback delivery should be a key factor in improving functional outcomes. With this in mind, we examined the delays between the onset of the participants’ neural modulations and the onset of haptic feedback delivered by the nrBMI.

We did two step response experiments to quantify feedback delays (see Methods). The results of these step response experiments are shown in figure 5(a). In the first experiment, we used a rigid block to transmit force applied by the haptic device directly to the load cell (figure 5(a), left). In the second step response experiment, a volunteer’s thumb was placed in the haptic device and passively allowed the step output from the nrBMI to apply force to their thumbnail (figure 5(a), right). Across both step response experiments, the time between the step input and F_BMI onset averaged 0.031 ± 0.003 s. Overall, the time between the step input and F_active onset in the rigid-block experiment was 0.104 ± 0.013 s (figure 5(a), inset), implying that approximately 0.07 s delay could be attributed to sources like communication between BCI2000 and LabVIEW, or time for the linear actuator to overcome inertia. When a human thumb was used in the nrBMI, the time between step input and F_active onset was 0.23 ± 0.014 s (figure 5(a), inset). Therefore, approximately 0.13 s of the overall delay can be attributed to the compliance of the thumb.

Figure 5.

Timing of nrBMI haptic feedback. (A) Delays between step input and F_BMI (black) or F_active (green) onset. Inset plots show example trials from both step response tests. In both tests, step inputs (inset; solid grey lines) passed through simplified BMI decoders, generating F_BMI (inset; black dashed line). Black circular markers show the F_BMI delay (also shown in the inset with a black arrow); black horizontal lines show the mean F_BMI delay across trials; black vertical lines show the (very small) F_BMI delay s.d. Green circular markers and horizontal/vertical makers show the delay between step input and F_active onset, in a similar manner. Green arrows (inset) show example single-trial F_active onset delay. (B) similar to (A), but showing delays between neural modulation onset and Factive onset for μ–β (blue) or high-γ (red) nrBMI-shared control.

Next, we tested the hypothesis that high-γ confers an advantage (in feedback timing precision) over μ–β when it comes to driving the nrBMI. Using data from nrBMI-shared control in our participants, we calculated trial-averaged features (high-γ or μ–β) around F_active onset, and identified the neural modulation onset time for each electrode (see Methods, figure 2). Figure 5(b) shows the results, including only electrodes in the C row to focus specifically on the motor cortex signal. Across participants, on average F_active lagged neural modulation onset time by 0.38 ± 0.09 s for high-γ control (figure 5(b), red), while F_active onset lagged μ–β modulation by 0.61 ± 0.06 s (figure 5(b), blue). Thus, high-γ nrBMI users received haptic feedback significantly faster than μ–β nrBMI users (2-tailed t-test; p = 0.008). Neglecting physical delays arising from nrBMI processing and thumb compliance, the nrBMI detected a change in high-γ power approximately 150 ms after the actual start of neural modulation. This was more than twice as fast as when the nrBMI operated in the μ–β range, where approximately 380 ms was required to detect the onset of neural modulation. This finding has implications for neurorehabilitative BMIs generally, as quicker feedback is more likely to trigger plasticity and enhance learning.

Discussion

Functional recovery after neurological trauma relies on plasticity within the nervous system. This plasticity has finite natural limits, as evidenced by the high number of individuals who experience chronic impairments following brain injuries like TBI (Schneider et al. 2021) and stroke (Furie 2020). In cases where biological plasticity is insufficient to promote motor recovery, it may be possible to use nrBMIs to augment this process. EEG-based nrBMIs have shown some success in improving movement function (Ramos-Murguialday et al. 2013, Bundy et al. 2017, Biasiucci et al. 2018). In this study, we demonstrated a nrBMI that included several novel elements. We successfully implemented a closed-loop nrBMI that decoded continuous, proportional grasp force intent from high-γ signals. The nrBMI also used a novel form of nrBMI-shared control between the decoded force intent and residual hand force. Participants learned to control the nrBMI with high success, especially when given the opportunity to practice. More importantly, we found that high-γ activity provided greater temporal precision in controlling the BMI than did low-frequency bands typically used in nrBMIs.

The haptic feedback provided by the nrBMI simulated the somatosensory experience of successful hand force, providing a potential means to facilitate plasticity and learning in the motor-sensory system. Both TBI and non-TBI participants were able to control the BMI, experience force feedback, and use the applied force to match a series of random force targets. In fact, we found no significant differences in the success rates or TTT between the two participant groups. This is encouraging in terms of the potential for brain-injured patients to use the nrBMI. There was evidence that the participants improved their nrBMI-shared control success rate over time, even when significant hand weakness was present (such as T1; see Table 1 and figure 4). The amount of time spent with the participants was often limited (due to factors discussed below), which in some cases might have constrained their ability to achieve higher overall success rates. Moreover, we note that—while not directly measured in this report—the ultimate purpose of an nrBMI is to generate plasticity, rather than to achieve a high success rate.

We believe that the use of shared control, i.e., using the nrBMI to augment the residual grasp force, could be an important design element in nrBMIs. Most existing nrBMIs use the brain signal alone to control some version of haptic feedback or functional electrical stimulation (for a recent review, see Mansour et al. 2022). In contrast, our design enables scaling the assist level to the participant’s level of impairment. This, in turn, encourages the user to produce as much active force as they are able in each trial, which may help retrain new or enhanced connections spurred by the nrBMI. Further, shared control may help maintain user focus and engagement, as it maximizes existing sensory feedback. The shared control approach used here could also be used in nrBMIs providing different types of feedback, such as exoskeletons. The current dataset did not reveal a significant correlation between the degree-of-sharing and nrBMI success rate.

We found a significant advantage to using high-γ for the nrBMI, as the feedback experience for T1-T3 (measured by F_active onset) was more tightly time-locked to their neural modulation onset than μ–β signals used by N1-N3. In fact, high-γ control delivered feedback approximately twice as quickly as μ–β, after accounting for physical delays (software and mechanical) in the nrBMI. The nrBMI described here had substantial inherent delays, possibly due to not being optimally designed for speed. With optimization, feedback could occur within as little as 150 ms given high-γ control, or potentially even sooner if bin sizes were shortened or updated more frequently. It remains to be determined whether there are specific temporal bounds for the induction of plasticity through nrBMI use. Plasticity in cortical neurons with single-unit BMI control has been demonstrated (Ganguly et al. 2011), as has the importance of feedback timing for enhancing functional improvements post-stroke (Mrachacz-Kersting et al. 2016). Intuitively, more precise synchronization between motor intent and sensory feedback implies a higher potential to trigger Hebbian-type neural plasticity, which is one of the primary goals of neurorehabilitation therapy (Soekadar et al. 2015). While the use of a high-γ nrBMI to drive plasticity is a novel approach, high-γ modulation has been linked to cortical plasticity in general (Traub et al. 1998, Headley and Weinberger 2011) and M1 plasticity in particular (Nowak et al. 2017). Perhaps most relevant, the enhancement of high-γ modulations (via stimulation) has been shown to restore plasticity to impaired motor cortex (Guerra et al. 2020). With the potential for coincident activation (timing overlap between motor intent and haptic feedback), it seems likely that high-γ control could enable nrBMIs to generate greater plasticity, and hence more functional improvement, after brain injuries.

The current study has some limitations, chief among which is the relatively small sample size of participants. Data collection in the hospital setting was challenged by the Covid-19 pandemic, which severely limited (and periodically eliminated) access to research participants. We chose to publish these results as proof of concept, or early evidence that hEEG high-γ signals could enable control of a continuous, proportional nrBMI (here, proportional control is implied by the relatively simple Weiner-cascade decoder), and that using high-γ for nrBMI control is beneficial in terms of feedback synchrony.

The nrBMI performance, or success rate (Figure 4) is not a functional outcome; we do not report functional outcomes in this proof-of-concept report. Instead, successful nrBMI control, combined with very short feedback delays, indicates that an individual could use the nrBMI to receive sensory feedback that is tightly locked to their motor intent, even while lacking the ability to generate volitional force. This makes the nrBMI a potentially valuable tool for inducing plasticity in the motor-sensory system. Future studies should quantify participants’ degree of functional improvement, as measured by standard clinical outcomes.

Working with participants who had hemicraniectomy following TBI made it possible to access high-γ neural signals; however, there are some drawbacks that can accompany the post-TBI clinical profile. Although the scalp was unbroken in our participants, sensitivity of the skin around the edges of the hemicraniectomy was common, and care was required when donning/doffing the electrode cap. Some individuals did not qualify for the study due to substantial cognitive impairments (including attention and memory) that prevented them from being able to attempt nrBMI control. Other individuals experienced allodynia in the affected hand, and headaches unrelated to the experimental procedures. In addition, it was important to guard against participant fatigue, as research sessions were completed largely during inpatient hospital stays, while maintaining full daily schedules of therapy. We anticipate that a translation to clinical use will require a version of the nrBMI that can be used by the individual alone, or with minimal help from therapists or caregivers. Ultimately, epidural or subdural recordings are probably a better way to provide high-γ signals for precisely targeting damaged brain areas with nrBMI therapy. Such signals provide a high amount of motor intent information (Flint et al. 2017, 2020) and stability for BMI use (Pels et al. 2019, Larzabal et al. 2021, Silversmith et al. 2021). Using epidural or subdural signals for nrBMI device design could help optimize tradeoffs in signal stability, invasiveness, device longevity, and richness of motor information (Slutzky and Flint 2017). At the moment, devices using these signals are still under investigation, but continue to improve in design and are advancing in clinical trials (Degenhart et al. 2018, Leinders et al. 2020, Moses et al. 2021). The results of this study, especially the increased temporal precision and movement-related information, indicate that high-γ based nrBMIs warrant further investigation for restoring movement to brain-injured patients.

Conclusion

We presented an nrBMI design that has the potential to improve therapy options for individuals with brain injury, especially those who lack residual motor function. An important feature of the nrBMI is the degree-of-sharing parameter, which can allow the clinician (or the individual) to determine the amount of assistance that is desired from decoded brain signal, in contrast with the amount of voluntary active force that must be provided by the hand. This feature can be adjusted to account for progress in recovery. Using high-γ signals for nrBMI control significantly improved the synchrony of haptic feedback, allowing participants T1-T3 to experience sensory feedback that was more closely aligned in time with the motor intent output signal. This improved synchrony is important for maximizing the ability of the nrBMI to induce plasticity in neural circuits, which is key to improved outcomes.