Intracortical Microstimulation Feedback Improves Grasp Force Accuracy in a Human Using a Brain-Computer Interface

Abstract

After a spinal cord injury, a person may grasp objects using a brain-computer interface (BCI) to control a robot arm. However, most BCIs do not restore somatosensory percepts that would enable someone to sense grasp force. Intracortical microstimulation (ICMS) in the somatosensory cortex can evoke tactile sensations and may therefore offer a viable solution to provide grasp force feedback. We investigated whether a bidirectional BCI could improve grasp force control over a BCI using only visual feedback. When evaluating the error of the applied force during a force matching task, we found that ICMS feedback improved overall applied grasp force accuracy.

Clinical Relevance—

This study establishes that intracortical microstimulation may be used to convey the amount of force a brain-computer interface participant is applying to an object.

I. INTRODUCTION

Grasping an object involves closing the fingers around the object, applying a force, and then releasing the fingers from the object. After a cervical spinal cord injury, a person would typically be unable to move their fingers and feel contact forces. To regain movement, brain-computer interfaces (BCIs) can read the person’s neural intention to grasp an object and then command a robotic hand to perform the grasp [1]. To regain the sensation of contact forces, it is possible to generate artificial sensations via intracortical microstimulation (ICMS) of the somatosensory cortex [2].

The use of artificial sensory feedback during grasp has been investigated in both healthy and amputee subjects. These studies used a wide range of human-machine interfaces, spanning non-invasive interfaces, such as skin stimulation [3] to implanted interfaces, such as peripheral nerve stimulation [4]. Non-invasive interfaces have mapped grasp force to the amplitude of a vibration [5] [6], or to the indentation of a plunger on the skin [7]. Implanted peripheral nerve stimulation provides feedback by stimulating afferent axons in the nerves of the residual limb with varying amplitudes and frequencies [4]. Exploiting this approach, researchers restored the ability to modulate the grip force [8] and improved motor coordination in functional grasping tasks [9].

The impact of sensory feedback on BCI grasp control has yet to be explored. We investigated whether providing ICMS feedback on applied grasp force during a BCI force-matching task improved control over a BCI without ICMS feedback. ICMS feedback had no impact on the overall success rate. However, ICMS feedback did enable more accurate grasp force control as evidenced by lower errors between the applied and target force.

II. METHODS

A. Overview

This study was conducted as part of an ongoing intracortical BCI clinical trial (NCT01894802) under an Investigational Device Exemption (IDE) granted by the US Food and Drug Administration. The study participant was a 31-year-old male with C5/C6 ASIA B SCI that resulted in complete paralysis of his hands with some residual function of the proximal arms and wrists. At age 28, 10-years post-SCI, the participant had two 88-electrode-microarrays (4 mm x4 mm, 1.5mm shank length, Blackrock Microsystems, Salt Lake City, UT, USA) implanted in his left motor cortex and two 32-electrode-microarrays (2.4 mm x 4 mm, 1.5mm shank length) implanted in area 1 of left somatosensory cortex. For the experiments described here, the participant performed a virtual grasp force task using an intracortical BCI that independently decoded grasp force and grasp velocity.

B. Neural Recordings

Neural signals were recorded using a Neuroport Neural Signal Processor (Blackrock Microsystems, Inc), band pass filtered between 0.3–7500 Hz and digitized and high-pass filtered above 750 Hz. For each electrode, the spike count was recorded as the number of times the voltage deviated below the threshold of −4.5 root mean square (RMS) of the signal recorded at the beginning of the session. Threshold crossings on each channel were binned at 20 ms, smoothed with a 2 s boxcar filter, and square root transformed.

C. BCI Decoder Training Trials

Three experimental sessions were used to evaluate the bidirectional BCI. At the start of each session, the BCI decoder was calibrated using the MuJoCo physics engine (mujoco.org, Roboti LLC) to generate a virtual reality (VR) environment on a TV where the participant observed a gripper grasping a spherical object (Fig. 1A). For two of the sessions, the participant also received ICMS feedback on the applied force during the VR gripper observation trials. The trial task dynamics are shown in Fig. 1B. A computer-generated voice first cued either the gentle, medium, or firm target with respective force values equal to 4, 8, and 12 arbitrary units (au). The virtual gripper then closed around the object and applied the cued force. In the close-and-grasp state, the VR gripper had 5 s to reach the force target and hold at the target for 2 s. Next, a second grasp target was cued, chosen from the three possible force levels. The gripper then had another 5 s to reach the new force target and hold at the target for 2 s. After completing the second hold, the VR gripper released the object.

Figure 1.

(A) Visualization of the VR gripper in the MuJoCo environment. (B) Grasp force in arbitrary units (au) (black) and grasp velocity (green) observed by the participant during trials used to train the decoder. Shaded time intervals indicate data used for decoder calibration.

The participant actively listened for the cued force, observed the VR gripper grasp the object, and imagined themselves performing the same action. The imagined actions included two control dimensions, grasp velocity ($g_v$​​) and commanded grasp force ($g_f$​​). Each observation session contained 54 trials, where the nine trial types (three initial force levels × three secondary force levels) were randomly sampled. An indirect optimal linear estimator (OLE) decoder was derived using an encoding model that linearly relates spike rate to grasp velocity and grasp force (Eq. 1):

$$r=\mathrm{ }{b}_0+b_v{g}_v+b_f{g}_f$$ (1)

where $r$ is the filtered and square-root transformed spike rate for a given channel, $b$​ is a regression coefficient for each grasp dimension: grasp velocity ($g_v$​​) and grasp force ($g_f$​​). The regression coefficients were calculated following methods described in [1]. A linear encoding model was chosen because it has enabled kinematic-control of grasp [1] as well as having shown to encode grasp force in ~40% of grasp-related M1 neurons in non-human primates [10].

D. BCI Decoder Grasping Trials

After training, the participant used the decoder to perform a virtual grasping task, Fig. 2. During each trial, the decoder’s positive grasp velocity $g_v$ commanded the VR gripper to close, the grasp force $g_f$ commanded the gripper to apply a force, and the gripper released the object during a negative opening velocity. When using the decoder, the same three force levels were used as during the observation session.

Figure 2.

Example trials with and without ICMS feedback. Shaded region represents the force levels that would yield a successful trial. Dotted lines are the decoded force (green) and velocity (brown). Solid lines are the actual force and velocity used during the course of the task. (A) This unsuccessful gentle trial occurred without ICMS or visual feedback. The gripper successfully closes on the object and but applied too much force. (B) This successful gentle trial occurred with ICMS and without visual feedback. (C)) This unsuccessful firm trial occurred without ICMS and with VR gripper feedback. The participant was quickly ramping up his force application, however, the decoded velocity began drifting downward. After the velocity crossed the velocity-mode threshold of −0.3 rad/s, the gripper suddenly opened and lost contact with the object.

Grasp velocity and grasp force were both continuously decoded during the entire trial. To contextually switch between grasp velocity and force, several state transition rules were implemented. First, each trial started with an open grasp posture. Once the gripper contacted the object, the trial switched to force-control mode. The force applied to the object $g_{fa}$ initialized at 0 au., regardless of the current decoded grasp force $g_f$. Using a PI controller, the applied force $g_{fa}$ ramped up quickly to achieve the grasp force commanded by the BCI decoder, $g_f$. The gripper re-entered velocity-control mode and released the object when an opening velocity ($g_v \le$−0.3 grasp-cycle/s) was decoded for more than 0.25 s. One grasp-cycle was either a complete grasp-closing or grasp-opening movement

E. Feedback Conditions

1). VR Gripper

The MuJoCo physics engine (Roboti LLC), was used to simulate a right-handed robotic gripper with a thumb and index finger. A virtual spherical object rested on the stationary thumb. Under BCI control, the index finger flexed towards the object. When contact was made, the fingers realistically bend to provide visual feedback of the application of force.

2). VR Gripper with Force Visualization

In this condition, the VR gripper was situated on the right side of the television screen and a continuous, real-time trace of the decoded force, $g_f,$ was on the left side of the screen. When the gripper contacted the object, there was also a continuous, real-time trace of the applied grasp force $g_{fa}$.

3). No Visual Feedback

The participant viewed a static background image on the television during the No Visual Feedback condition. At the start of each trial, the computer-generated voice continued to cue the upcoming force target. According to the outcome of the trial, either the failure or success tone was played at the end of the trial.

4). Intracortical Microstimulation (ICMS) Feedback

Artificial tactile feedback was generated through intracortical microstimulation (ICMS) on a single electrode in area 1 of the somatosensory cortex. The stimulation created a combined sensation of pressure, tingle, warmth, with some sharpness on the area surrounding the right-hand index-finger proximal interphalangeal joint. During object contact, we supplied ICMS at a frequency of 100Hz. The stimulation amplitude increased linearly with the applied force such that a force of 0.1 au corresponded to a stimulation amplitude of 20 μA and 16 au corresponded to a stimulation amplitude of 90 μA. The stimulation amplitude was updated every 0.02 s. After each stimulation pulse, there was an electrical stimulation artifact that prevented recording neural activity. For this reason, neural recordings were “blanked,” for ~1.5 ms from the beginning of each stimulation pulse.

5). Sham-ICMS Feedback

To understand the effect of ICMS feedback separate from the effect of blanking neural data, we performed sham-ICMS feedback. In this condition, we stimulated a channel that was not implanted in the participant. This triggered the blanking protocol and thus reduced the neural data in the same manner as ICMS.

F. Performance Metrics

Two metrics were used to evaluate ICMS feedback performance: success rate and applied force error.

1). Success Rate

We measured performance for each six feedback conditions (three visual conditions × two ICMS conditions) as the percentage of successfully completed trials (Fig. 3). A successful trial involved closing the VR gripper, applying force to within ± 2 au of the cued force for 1.0 s, and then opening the gripper.

Figure 3.

Participant’s success rate using a bidirectional BCI during a virtual grasp force matching task with different visual and ICMS feedback conditions across three sessions. Each dot or star represents a block of trials. The feedback conditions are: (N) - No Visual Feedback, (G) - VR Gripper Feedback, and (GF) - VR Gripper with Force Visualization Feedback. Horizontal bar indicates median of data.

2). Applied Force Error

The applied force error $e\left(t\right)$ was calculated as:

$$e\left(t\right)= \left|T- g_{fa}\left(t\right)\right|$$ (2)

where $t$ represents the time in seconds since object contact, $T$ represents the cued force target, and $g_{fa}\left(t\right)$ represents the force the decoder applied to the object. We evaluated the applied force error at the last time bin in the Grasp state in order to allow the participant time to interpret the ICMS feedback and make any desired corrections. Trials were only included in the analysis if the gripper had fully closed and was applying force at the end of the Grasp state (91.2% gentle, 95.8% medium, and 91.2% firm target trials).

G. Experiment Design

Over the course of three experimental sessions, we tested the participant’s ability to use ICMS in conjunction with the three visual feedback methods. This created six different feedback conditions: no visual feedback with ICMS and with sham-ICMS, VR gripper with ICMS and sham-ICMS, and VR gripper with force visualization with ICMS and sham-ICMS. Each condition was tested twice within an experimental session (“early” and “late” blocks, Fig. 3). With each block consisting of 18 trials, each session contained 216 trials (6 feedback conditions × 18 trials × 2 blocks). On the first and third sessions, in the early block of each visual condition, the sham-version of the condition was presented before the ICMS-version. Then, in the late block of each visual condition, the ICMS-version of the condition was presented before the sham-version. This sham/ICMS presentation order was reversed for the second experimental session.

III. Results

A. Success Rate

The success rate was measured for each feedback condition both early and late in the session (Fig. 3). A two-way ANOVA was performed, testing for the effect of visual and ICMS feedback on the success rate. There was a significant effect from visual feedback (p < 0.001), but no significant effect from ICMS feedback (p = 0.411). Within a visual feedback condition, there was also no difference between ICMS and sham-ICMS performance. While the median success rate was higher for ICMS feedback than sham-ICMS feedback during the no visual feedback condition, the difference was not significant due to the large variability across days and sets of trials. The participant had the highest performance and lowest variability during the VR gripper with force visualization feedback condition.

B. Applied Force Error

To assess the participant’s ability to use ICMS feedback despite the state transition challenges, we calculated the applied force error for each trial where the gripper remained closed around the object, Fig. 4. A two-way ANOVA was performed that tested for the effects of visual feedback, ICMS feedback and their interaction on the applied force error. We found that both visual feedback and ICMS feedback served to significantly reduce the applied force error (p < 0.001 and p = 0.022, respectively). The interaction between visual feedback and ICMS feedback did not significantly affect the applied force error (p=0.075). Next, t-tests were used to test between ICMS and sham-ICMS conditions separately for each visual feedback condition. We found that there was no significant difference in the applied force error between the ICMS and sham-ICMS for any of the visual feedback conditions.

Figure 4.

Applied force error during different visual and ICMS feedback conditions, (N) no visual feedback, (G) VR gripper feedback, and (GF) VR gripper with force visualization. No significant difference was seen between the sham-ICMS feedback and ICMS feedback conditions.

IV. Discussion

We investigated whether a bidirectional BCI could be used to improve the control of grasp force. We found that the overall success rate of the bidirectional BCI was not improved over the BCI using only visual feedback. However, we saw a modest but significant effect of ICMS feedback in reducing the applied grasp force error.

While each force level generally showed the same trends, such as lower applied force error during the VR gripper with force visualization feedback condition and higher force error during the no visual feedback condition, there were also some differences. The gentle target had the overall highest amount of applied force error (2.59 au ± 2.63 au, mean ± st. dev.). The participant had noted that the gentle target was the most difficult to achieve using the decoder provided because of coordination necessary to generate both a meaningfully-large closing velocity and a gentle grasp force. The analogy he provided was that the process of using the $g_v$ and $g_f$ components was similar to using the clutch and accelerator in a manual car.

Another difference we observed was with the medium target. ICMS feedback had little influence on the applied force error during trials to the medium target. This was unsurprising to us because the participant had noted that he used an open-loop strategy where he had attempted a normal, comfortable grasp force. The overall applied force error to the medium target was 1.95 au ± 1.96au.

The firm target required effort on part of the participant, such that he experienced something akin to fatigue in his hand if there were several successive firm targets. The overall applied force error to the firm target was 1.15 au ± 1.25 au.

We saw that ICMS feedback reduced errors in applied force, and yet there no significant benefit in overall success rate. The difference between these two metrics largely stems from trial and decoder state transition issues, an example of which can be seen in Fig. 2C with the gradual velocity decrease causing a premature state transition. These state-transition challenges included trials where the fingers never fully contacted or did not remain in contact with the object. This occurred on 8.8%, 4.2%, and 8.8% of gentle, medium, and firm trials, respectively. Many of these trials were due to the opening velocity $g_v$ gradually becoming more negative over the course of the trial, causing the participant to unintentionally enter velocity-mode and open the gripper. This problem is what initiated the experimenters to lower the velocity-mode threshold to $-0.3$ grasp-cycles/s, below the below the more intuitive $0$ grasp-cycles/s.

When the gripper prematurely released the object, it was difficult for the participant to successfully re-grasp and hold the object before the allotted time expired. We attempted to mitigate this problem by changing decoder gains, biases, and trial state transitions, but these interventions did not prevent the problem. To better test the efficacy of bidirectional BCIs for grasping, it will be necessary to implement a decoding strategy that can more gracefully transition between movement-states and grasp-states. For this reason, independently decoding grasp force and grasp velocity may not be the optimal strategy moving forward. Our future plans involve developing a new decoding strategy that more fully describes the neural activity related to grasp force control.