Task-Specific Somatosensory Feedback via Cortical Stimulation in Humans

Abstract

Cortical stimulation through electrocorticographic (ECoG) electrodes is a potential method for providing sensory feedback in future prosthetic and rehabilitative applications. Here we evaluate human subjects’ ability to continuously modulate their motor behavior based on feedback from direct surface stimulation of the somatosensory cortex. Subjects wore a dataglove that measured their hand aperture position and received one of three stimuli over the hand sensory cortex based on their current hand position as compared to a target aperture position. Using cortical stimulation feedback, subjects adjusted their hand aperture to move towards the target aperture region. One subject was able to achieve accuracies and R² values well above chance (best performance: R² = 0.93; accuracy = 0.76/1). Performance dropped during the catch trial (same stimulus independent of the position) to below chance levels, suggesting that the subject had been using the varied sensory feedback to modulate their motor behavior. To our knowledge, this study represents one of the first demonstrations of using direct cortical surface stimulation of the human sensory cortex to perform a motor task, and is a first step towards developing closed-loop human sensorimotor brain-computer interfaces.

1 INTRODUCTION

Somatosensory feedback is essential for efficient and precise movement and to manipulate objects effectively [1], [2]. It has also been identified as a consumer design priority for prosthetics [3]. Incorporating such feedback into advanced prosthetics and other rehabilitative devices will improve their functionality and is therefore of interest to the field of brain-computer interfaces (BCIs), involving direct brain control of external devices [4], [5]. Previous research has demonstrated that somatosensory feedback improves BCI performance. Pistohl et al. provided proprioceptive feedback through a robotic manipulandum to increase human performance in a myoelectric-control task [6]. In non-human primates, both integrating external somatosensory feedback through a robotic exoskeleton [7] and providing cortical sensory feedback through intracortical microstimulation (ICMS) [8] improved BCI task performance as compared to task performance without the additional sensory feedback. ICMS sensory feedback in nonhuman primates has also been used to deliver stimulation proportional to the force exerted on a sensorized prosthetic fingertip [9] and to create a closed-loop BCI, decoding motor intention and encoding sensory feedback [8], [10] (for a more complete review of recent BCI and ICMS sensory feedback research see Bensmaia and Miller, 2014 [11]).

Other sensory feedback approaches including peripheral nerve interfaces [12], [13], [14] and targeted muscle reinnervation [15] have shown success in providing sensory feedback to human users. Tan et al. were able to stimulate through peripheral nerve cuff electrodes and elicit tactile perceptions including pressure and tapping [12]. Still, the field lacks an understanding of the interfaces and their stimulation parameters that may be used to convey a wide range of somatosensory feedback. Sensory stimulation will need to elicit percepts that users can employ to form a coherent behavioral response. To use stimulation for sensory feedback, one needs to explore the broad space of parameters, sensory percepts (including biomimetic and abstract), and behavioral responses.

In non-human primates, a recent study extensively investigated the relationship between different ICMS stimulation parameters, the animals’ detection thresholds, and their just noticeable differences [16]. Ideally, similar studies will be conducted in human subjects as they can provide a qualitative assessment of the qualities and intensity of the stimulation percept. Previous research has demonstrated that human subjects can discriminate the intensity of abstract percepts from electrocorticography (ECoG)-based stimulation (i.e., electrical stimulation of the brain surface) with either varied frequency or varied amplitude [17]. Building upon this work, we test how subjects can utilize abstract sensory feedback in a motor task. We hypothesized that human subjects could use cortical sensory stimulation feedback to continuously modulate their motor behavior to find and follow an unknown target hand position. Three stimulation states (no stimulation, a low-intensity stimulation, and a higher-intensity stimulation) corresponded to three hand aperture states (position of the hand relative to a target position). Our results show that one subject was able to successfully adjust his hand position in response to the ECoG stimulation feedback in order to follow the moving target position with above-chance performance. Two other subjects attempted the task but were unable to complete it; their results are presented primarily in the Supplemental Material.

2 METHODS

2.1 Subjects

Subjects were hospitalized for clinical monitoring of epilepsy with implanted electrocorticography (ECoG) grids (see, e.g., Leuthardt, et al. [18]). Clinicians determined their electrode locations strictly by clinical needs without consideration for research. We conducted stimulation studies after subjects were back on their anti-epileptic medications (AEDs). This experiment was approved by the Institutional Review Board of the University of Washington, and all subjects gave written, informed consent.

2.2 Electrode Localization

We localized electrodes using a preoperative MRI scan, a post-operative CT scan, and custom MATLAB processing scripts as described previously [19]. We then identified a pair of adjacent electrodes over the hand sensory cortex to use as the active and ground electrodes for stimulation (Fig. 1).

Fig. 1.

Subject 2’s ECoG grid location. Left hemisphere implanted ECoG grid with the stimulating electrodes highlighted in white. The subject perceived an abstract sensation in the proximal and middle phalanges of his right middle finger following cortical stimulation over these electrodes.

2.3 Stimulation Protocol

RZ5D BioAmp Processor, IZ2H Stimulator, and LZ48 Battery Pack (Tucker-Davis Technologies, Alachua, FL) were used to deliver constant-current stimulation using biphasic square pulses with 200 μs phase widths. The interpulse interval (IPI) was 4600 μs, yielding a pulse frequency of 200 Hz. Each train of pulses lasted for 200 ms and was followed by an inter-train interval (ITI) of 400–800 ms. We determined the perceptual threshold for stimulation by incrementally increasing the current amplitude in steps of 250–500 μA up to no more than 5000 μA. If stimulation across the pair of electrodes did not cause a sensation in the subject’s hand, or the stimulation caused motor activity, then a new electrode pair was chosen and the thresholding procedure was restarted. We defined a low-intensity stimulation waveform (Stim 1) with the above parameters and a current amplitude slightly above the perceptual threshold. We then defined a higher-intensity stimulation waveform (Stim 2) with the same parameters, but with a current amplitude above that of Stim 1, such that the subject, as self-reported, could clearly discriminate the two stimuli at rest. Stimulation current amplitudes and the ITI were slightly modified during the task for two subjects based on their feedback and are detailed in the Results. The catch trial used an amplitude that was between that of Stim 1 and Stim 2.

2.4 Experimental Protocol

Three subjects wore a 22 degree-of-freedom dataglove (Cyberglove II, CyberGlove Systems, San Jose, CA) to measure the position of their hand, which we sampled every 50 ms with a custom MATLAB script. To begin the task, subjects opened and closed their hand (in a palmar grasp [20], as if grasping a cylindrical object) with the dataglove for approximately 20 seconds. We took the first vector of the singular value decomposition of these movements as the primary trajectory of aperture motion. During the task, every new glove sample was projected onto the primary trajectory, and the magnitude of the resulting vector was taken as the non-normalized aperture value. We then calculated the normalized aperture value ranging from 0 to 1 based on the subject’s minimum (closed) and maximum (open) hand positions. We instructed subjects that they would be asked to open and close their hand to find and follow a target aperture path. Cortical electrical stimulation using the waveforms described above provided feedback to the subjects on their current state. The subjects could be in one of three states (Table 1, Cases A–C). We instructed all subjects to open their hand before the trial and stimulation began so that they would start in the state without cortical stimulation (Case A, Table 1).

TABLE 1.

Possible Hand States or Cases

Case	Hand Position	Aperture Value	Stimulation
A	Too open	> target	None (Stim 0)
B	Within target	Within target	Low (Stim 1)
C	Too closed	< target	Higher (Stim 2)
Catch	All positions	All values	Catch stim

The aperture value was calculated from the subject’s current hand position, and stimulation was dependent upon the aperture value.

We describe below the specific methods and results for one subject (Subject 2) who performed above chance and completed a catch trial. Two other subjects participated in the task but were unable to complete the task with a catch trial due to poor performance (see Results and Supplemental Material). To help learn the relationship between the stimulation pattern and his hand position relative to the target position, Subject 2 was trained with concurrent visual feedback for three trials before receiving stimulation feedback solely. His single catch trial delivered the same stimulation regardless of the subject’s state. (Catch, Table 1).

2.5 Target Paths

We set Subject 2’s target path width to span 15% of his own aperture range from opened (aperture value = 1) to closed (aperture value = 0). Two types of paths were created: a training path and several evaluation paths, for use in training and evaluation trials, respectively. The training path was a simple sine wave with a frequency of approximately 0.02 Hz. For each of the evaluation trials the subject used a different evaluation path that we created by summing four sinusoids with randomly selected frequencies that were no greater than 0.02 Hz. Fig. 2 shows example evaluation paths with the subject’s hand aperture values overlaid. Figs. 3 and 4 illustrate the order of Subject 2’s trial path types.

Fig. 2.

Traces of Subject 2’s hand aperture position relative to the aperture target thresholds with corresponding stimulation current amplitudes (a: Trial 9, b: Trial 13). Stimulation pulses were biphasic, but due to the time scale only the 200 ms stimulation trains are visible not the individual pulses (Stim 1 = 2.0 mA, Stim 2 = 2.4 mA, ITI Trial 9 = 800/800 and ITI Trial 13 = 800/400 for Stim 1/Stim 2, Table 3). Subjects’ aperture values could move outside of the 0 to 1 range if they made hand movements that were outside of the range used during the normalization period (Section 2.4). Subjects were instructed to start each run with their hand open in order to begin in the no-stimulation region (Table 1, Case A). Subject 2 sometimes overshot the target boundaries, but responded to error feedback (Case A, no stimulation; or, Case C, higher-intensity stimulation) by changing his direction of motion. a) Subject 2 was able to follow the target pathway and stay in the target boundaries with a high performance of: accuracy = 0.6145, R² = 0.8194. b) Subject 2 had trouble finding the target region at the beginning of the trial, resulting in lower performance values of: accuracy = 0.4023, R² = 0.1001.

Fig. 3.

Subject 2’s accuracy levels as a measure of performance. Accuracy was calculated as (samples inside target range)/(total samples) while chance levels were determined with 1,000 simulated random walks. Mean chance accuracy values with error bars for the standard deviation are displayed. The subject’s accuracy is above chance level for 11 of the 13 non-catch trials. During the setup trial the subject had trouble mapping the cortical stimulation to the necessary motor response, but used the following 3 training trials with concurrent visual and stimulation feedback (shaded, trials 2–4) to explore the state space and learn to use the feedback. His accuracy dropped to below chance levels during the catch trial (same stimulation feedback regardless of the state) suggesting that he was relying on the cortical stimulation to achieve a high performance. Table 3 lists the stimulation amplitudes and ITIs for each trial. *Trials 13 and 14 used a shorter ITI for Stim 2 than the previous trials.

Fig. 4.

Subject 2’s R² values as a measure of performance. Shaded trials 2–4 used concurrent visual and cortical stimulation feedback. Chance values were simulated with the random walks used for the accuracy chance calculations. Mean chance R² values with error bars for the standard deviation are displayed. The R² values follow a trend similar to the accuracy values (Fig. 3), and considered together the accuracy values and the R² values can illustrate the subject’s overall performance. In trials with high accuracies and high R² values, the subject slowly opened and closed his hand and remained relatively close to the target region even when he exited it. In trial 13, with a low R² value and higher accuracy, the subject deviated largely from the target region while searching for it (Fig. 2b), and then eventually found and followed the path increasing his accuracy but not his R² value. Again, Table 3 lists the stimulation amplitudes and ITIs for each trial. *Trials 13 and 14 used a shorter ITI for Stim 2 than the previous trials.

2.6 Data Acquisition and Analysis

We recorded all ECoG data at approximately 1526 Hz with Tucker Davis Technologies’ biosignal acquisition system (RZ5D BioAmp Processor and PZ5 NeuroDigitizer) and analyzed data with MATLAB.

2.7 Accuracy

We defined accuracy as the fraction of time spent inside the target region and calculated it as: (samples inside target range)/(total samples) over the entire trial. To determine chance accuracy, we simulated 1,000 random walks, calculated the accuracy of each, and then took the mean. Each random walk was created by drawing with replacement from that trial’s set of position changes and cumulatively adding these position changes to a starting position. If the random walk started at the first sample position, when the subject’s hand was too open, it may have never entered the target region. Thus, we defined the starting position as the first point that the subject’s hand entered the target region. We then calculated the chance accuracy only over the period beginning with the starting position to ensure that the random walk always started within the target range and thus calculate a more robust measurement of chance. We computed the resultant chance accuracy as: (samples inside target range)/(last sample–first sample that subject enters target range). We measured the subject’s performance from the beginning of the trial rather than when they first entered the target region, because during some trials subjects did not enter the target region quickly; therefore, calculating the accuracy from when they first entered the target region would result in an inflated value.

2.8 Line Fit: R²

As another measure of performance, we calculated the R² values for each trial by fitting a least-squares linear model of the subject’s recorded hand aperture motions to the optimal path - the path through the center of the target region. The R² value was defined as: R² = SS_reg/SS_tot, where ${\mathit{SS}}_{\mathit{reg}}=\mathit{\text{Sum of squares of the regression}}={\displaystyle {\sum }_i{(f_i-\overline{y})}^2}$, ${\mathit{SS}}_{\mathit{tot}}=\mathit{\text{Total sum of squares}}={\displaystyle {\sum }_i{(y_i-\overline{y})}^2}$, and y is the recorded aperture data while f is the linear model.

The interpretation of this R² value is similar to a traditional goodness of fit of a model to experimental data. In this case we are interested in assessing the goodness of fit between our recorded aperture data and the optimal path, as compared to random walk. We used the 1,000 random walks created in the accuracy calculation to determine the chance R² value for each trial by again calculating the R² value of each random walk, and then taking the mean.

3 RESULTS

Three subjects who had grid coverage over their hand sensory area participated in this task. For each subject, cortical stimulation over the chosen electrodes (Fig. 1 for Subject 2) elicited an abstract sensation in their hand. None of the subjects thought that the stimulation felt normal. We asked them to describe the sensation to us without any guiding questions (Table 2). To maintain two differentiable stimuli we had to increase the amplitude of cortical stimulation several times during the experiment for Subjects 2 and 3 (Table 3).

TABLE 2.

Stimulation Percepts

Subject	Location of Percept	Subject’s Description
1	Left ring finger	“vibration,” and “an intention to move” [Note: Neither we nor the subject believe that her finger actually moved.]
2	Right middle finger, proximal and middle phalanges	“felt like a soft line straight across the bottom”
3	Left ring finger, palmar side, middle and distal phalanges	“little pulse,” and “Noticeable… if I was at a ballgame it’d be noticeable like hair tickling your face”

Location and description of subjects’ stimulation percepts. We asked where subjects felt the stimulation and what it felt like; no further instructions or guiding questions were given.

TABLE 3.

Current Amplitude and Inter-train Interval Values for Stim1 and Stim2

Subject	Trials	Stim 1		Stim 2
		Amplitude (mA)	ITI (ms)	Amplitude (mA)	ITI (ms)
1	All (1–3)	2.50	800	3.50	800
2	1	1.75	800	2.00	800
	2–3	2.00	800	2.25	800
	4–12	2.00	800	2.40	800
	13–14	2.00	800	2.40	400
3	1–3	1.75	800	2.25	400
	4–7	2.00	800	2.50	400
	8–11	2.00	800	2.75	400

To maintain discriminable stimuli, the amplitude values had to be increased several times during the experiment for Subjects 2 and 3. The inter-train interval (ITI) was also changed during Subject 2’s trials.

As mentioned above, only Subject 2 consistently performed above chance levels and completed a catch trial. Subject 1’s and Subject 3’s results are detailed in the Supplemental Material and discussed briefly after Subject 2. Their level of engagement was much less than that of Subject 2 and likely contributed to their poor performances [21].

Subject 2 wore the dataglove on his right hand, the hand contralateral to the implanted ECoG grid, and therefore perceived a sensation from the stimulation on the same hand that he had to move. He used these perceived sensations as feedback on his current hand position relative to the target aperture position and responded by adjusting his hand position. Subject 2 learned to respond properly to the cortical stimulation and follow the unknown target aperture path (Fig. 2a).

Subject 2 completed 14 total trials: a set-up trial, 6 training trials, 6 evaluation trials, and a catch trial (Figs. 3 and 4). During the first three training trials he received visual feedback concurrent with the cortical stimulation feedback and used it to explore the state space and understand the stimulation feedback. During the following three training trials he received only cortical stimulation feedback. The first two trials lasted for 45 seconds each, while the remaining trials lasted for two minutes each, as the subject thought that 45 seconds was too short. We started with amplitudes of 1.75 mA and 2.0 mA, respectively for Stim 1 and Stim 2 for Subject 2 based on his own qualitative report that they were discriminable. During the task we increased the stimulation amplitudes twice based on the subject’s feedback to maintain discriminable stimuli (Table 3). In one case, Subject 2 also noticed a beat or rhythm to the stimulation sensation and wondered if we could change the rhythm of one of the waveforms to make them more easily discriminable. We believe he was noticing the ITI and therefore decreased the ITI of Stim 2, the more intense stimulation, from 800 ms to 400 ms while maintaining its increased amplitude over Stim 1 (trials 13 and 14, Table 3). We notified the subject of this change before he began Trial 13.

We defined the fraction of time spent inside the target region as the subject’s accuracy level. In 11 of the 13 non-catch trials Subject 2 performed above chance level in accuracy (Fig. 3), the two lower performances being the first two trials. During the catch trial, which used the same stimulation feedback regardless of the state, the subject’s accuracy dropped to below chance. In quantifying the subject’s path we also considered the R² value as a measure of the goodness of fit of the subject’s hand motions to the ideal path. Similar to the accuracy levels, the subject’s R² values generally increased during the training trials and fell below chance levels during the catch trial (Fig. 4). During trial 13 Subject 2’s R² value dropped to just over chance, as he had trouble finding the target region for the first half of the trial and made large deviations out of the target region during that time (Figs. 2b and 4).

The two other subjects that participated in this task had poor performances as measured by accuracy and R² values. Subject 1’s and Subject 3’s accuracies ranged from 0.1133 to 0.6446 and 0.0945 to 0.6533, respectively (Figs. S1 and S4). Their R² values ranged from 0.0442 to 0.1505 and 0.0006 to 0.7927, respectively (Figs. S2 and S5). For Subject 3 these values include trials with visual feedback, and for both subjects their performance was often near chance for trials without visual feedback. Their complete results are included in the Supplemental Material for full disclosure.

4 DISCUSSION

Using feedback from cortical sensory stimulation alone, one of our subjects (Subject 2) was able to continuously modulate his motor output to follow the aperture target path and perform well above chance. Two other subjects (Subjects 1 and 3) were behaviorally unable to achieve high performances in the task, and we were not able to complete enough trials with them to comment on learning.

4.1 Subject 2

Subject 2’s performance, as measured by accuracy and R² values, had a generally increasing trend during the training trials. The first two training trials, which included concurrent visual and cortical stimulation feedback, may seem to have surprisingly low performances; however, the subject used these trials to explore the aperture space and stimulation states rather than trying to follow the exact path. This subject’s first two evaluation trials have lower performances than the max training performance, likely due to the use of new and unknown target paths for each evaluation trial. During the catch trial, when the subject received the same stimulation in all states, the performance levels dropped to below the chance values suggesting that the subject had been relying on the varied cortical sensory stimulation to complete the task. In the non-catch trials, the subject must have been able to discriminate between the two stimuli in order to find and follow the target path and achieve a high performance. The discriminability of stimulation waveforms was demonstrated in previous work that varied the current amplitude and the pulse frequency [17]. The subject’s performance in the post-catch trials did not jump back up to pre-catch levels, because as Subject 2 expressed, the catch trial confused him and he was attempting new methods of completing the task.

4.2 Subjects 1 and 3

Working with human subjects who have recently undergone neurosurgery presents various challenges, including transiently reduced attention levels and cognitive abilities. Additionally, our stimulation studies require that subjects be back on their anti-epileptic medications before participating, which limits the amount of available study time. We did not have time to test many stimulation patterns or stimulation to aperture state mappings with these subjects, so we cannot say whether the task was simply too challenging, or whether both Subjects 1 and 3 would have struggled with any tasks during that period of time. Additionally, both subjects became fatigued during this task so we were unable to determine whether the subjects would have learned the task with more time.

It is clear that this task requires complex attention, is not completely intuitive, and is sensitive to fatigue. These limitations are lessons for future somatosensory stimulation experiments. More research will also need to be conducted to determine how subjects can best learn to use cortical stimulation feedback. We also see the need for a method to assess task comprehension and attention. Such a validity measure would allow us to only proceed with a task if the subject was attentive enough to warrant participation.

4.3 Performance Measures

Taken together the two measures of performance, accuracy and R², can highlight different behavioral responses. Accuracy levels reflect the percent of time that the subject remains within the target region, but do not capture how large the subject’s deviations are when they leave the target region. In contrast, R² values take the size of the subject’s deviations into account, but do not directly measure whether the subject was inside the target region. One explanation for high R² values with low accuracy levels (e.g., the subject follows the target path, but often remains just outside of the target region) is that the subject has substantial knowledge and expectations of the pattern of the target path, but does not completely understand how to use the stimulation feedback to re-enter the target region. We do not see such a relationship between the accuracy levels and the R² values in Subject 2’s results. In fact, we see the opposite relationship (high accuracy relative to the R² value) in trial 13 (Fig. 2b). In this instance, the subject started losing the ability to discriminate Stim 1 and Stim 2 and thus had trouble finding the target region. For the first half of the trial he opened and closed his hand in large motions throughout the entire aperture range ‘feeling’ for the target region. When he eventually found that region, he was able to follow the path for the remainder of the trial. This increased his accuracy level, but due to the large deviations from the ideal path made at the beginning of the trial, his R² value remained low.

4.4 Adaptation

During our work with these subjects, we noticed that there appeared to be an element of adaptation or habituation to the stimulation signal over time, consistent with previous observations in cutaneous vibrotactile stimulation in humans in which the perceived stimulation intensity on the subjects’ fingertips decreased over time [22]. All three subjects, at various points in the experiment, expressed that the two stimulation sensations were becoming hard to discriminate, thus making the task more difficult. In response, we increased one or both current amplitudes and the difference between the two amplitudes for Subjects 2 and 3 (Table 3). The differences in stimulation amplitude that we used are similar to those reported in previous work which demonstrated that human subjects can qualitatively recognize differences in ECoG stimulation current amplitudes as small as 0.4 mA [17]. However, this prior work used different stimulation parameters (e.g., phase widths, pulse frequency, train duration, and ITI) than we did, so the results cannot be directly compared.

As in some cutaneous vibrotactile studies, one may be able to exploit adaptation to enhance subjects’ discrimination between two similar stimuli [23], [24]. Specific experiments to study the timing and nature of this adaptation and the just noticeable differences of stimuli will need to be considered. Due to our time limitations we chose to have subjects self-report discriminability, but future experiments could use established psychometric tests [25], [26], [27], [28] to quantitatively measure discriminability and assess changes over time and how the degree of discriminability affects task performance.

4.5 Pulsed Feedback

We chose pulsed feedback with an inter-train interval of 400–800 ms (Table 3) for two primary reasons. First, we wanted to minimize the risk of stimulation-induced seizures, and felt that continuous stimulation increased that risk. Secondly, we chose to use a sufficiently long ITI in order to potentially pull out ECoG recording data between stimulations. This time-division multiplexing (TDM) of the stimulation and recording periods could be used in a closed-loop BCI to allow for tactile feedback without obscuring all neural recordings of motor intention with a stimulation artifact [29]. After removing the stimulation artifact from the ECoG recording we believe that our ITI still leaves enough time for meaningful motor decoding based on our previous experiments which generally use 200 ms windows for decoding. We will test whether these ITIs are sufficient for quality motor decoding in future experiments.

4.6 Closed-loop BCI Application

Along with the use of TDM, we developed this task to resemble the tactile feedback that one could receive in a closed-loop BCI. For example, a prosthetic hand could have three states that signaled: Case A) when the subject was not touching an object (no stimulation, Stim 0); Case B) when the subject was grasping the object with enough force (Stim 1); and Case C) when the subject was grasping the object too tightly (Stim 2). Ideally, more positions or grasp states would be encoded, but they could follow this pattern of increasing stimulation intensity as the tactile input became more intense, as in tightening one’s grasp on an object. This paradigm represents a simplified feedback strategy and provides a framework for the development of future approaches.

4.7 Stimulation Percepts

Recently there has been a discussion of whether tactile feedback for BCIs will need to be biomimetic [11]. One argument holds that BCI applications may be able to use sensory substitution and thereby elicit abstract sensations that the user will substitute for normal tactile sensations. Users may map the abstract sensation to a normal sensation and no longer perceive it as abstract [30]. Alternatively, another argument holds that only biomimetic feedback will allow subjects to regain the sort of dexterous movement and tactile sensations that normally occur [31], [32]. The basis for this argument is that naturally occurring tactile and proprioceptive sensations are so varied that the only way to encode all of them in a meaningful way will be to create biomimetic sensations through stimulation so that users don’t have to create a new representation and mapping of abnormal sensations. With just three subjects, this study does not allow us to speculate on whether sensory substitution will suffice, but we did find that Subject 2 was able to use the abstract sensation (Table 2) to achieve performances well above chance with three defined states. Similar tasks using cortical sensory stimulation on human subjects will provide more insight into users’ needs and a better understanding of the number of states that may be encoded with abstract sensations before biomimicry is required.

5 CONCLUSION

To our knowledge, the results presented here represent one of the first demonstrations of using cortical surface (ECoG) stimulation of the human sensory cortex to perform a motor task. We believe ECoG stimulation will offer new avenues for investigating haptic feedback, as ECoG electrodes allow us to directly stimulate the somatosensory cortex of awake human subjects who can describe the sensation and any differences between two stimuli. Our study demonstrates that subjects can react to cortical sensory stimulation, and while it may require increased attention or cognitive abilities, one subject was able to continuously modulate his motor behavior in response to the stimulation feedback. Although this task was not optimized, the subject was able to use the abstract feedback from cortical stimulation and map it to a new task demonstrating a proof of concept for the use of ECoG somatosensory stimulation for motor task feedback. Future experiments will explore the relationship between stimulation parameters and humans’ perceptions of the stimulation to begin to establish a basis for cortical stimulation waveform development. Ideally such research will compile psychophysical data for human responses to ECoG stimulation just as has been done in non-human primates with ICMS stimulation [16]. Our results demonstrate how ECoG stimulation may be used as a tool to further understand tactile and proprioceptive encoding for closed-loop BCIs in humans.

Biographies

Jeffrey G. Ojemann is a neurosurgeon with a practice focusing on the surgical treatment of epilepsy in adults and children. He is Director of Epilepsy Surgery at the Harborview Medical Center and Seattle Children’s Hospital at the University of Washington. He received his MD in 1992 at Washington University in St. Louis, and remained there for residency in neurosurgery (1999) and fellowship in pediatric neurosurgery at St. Louis Children’s Hospital in 2000. He joined the faculty at the University of Washington in 2003 and holds the Richard G. Ellenbogen Chair in Pediatric Neurological Surgery. He has been funded by NIH or NSF continuously since then with present focus on electrical stimulation and current distribution in human and animal models of cortical surface stimulation. He has over 165 peer-reviewed publications, primarily in neuroimaging and human cortical physiology. He is the co-leader of the Cortical co-adaptation with bidirectional brain-computer interface testbed of the Center for Sensorimotor Neural Engineering (an NSF Engineering Research Center).

Jared D. Olson received an MD degree from the University of Chicago in 2008 and completed residency in Physical Medicine and Rehabilitation at the University of Washington (UW) in 2012. He is faculty at the NSF Center for Sensorimotor Neural Engineering and a clinician-scientist at UW supported by the NIH Rehabilitation Medicine Scientist Training Program (RMSTP) and NIH StrokeNet, mentored by Jeffrey Ojemann. Jared researches basic science and translational questions in human brain-computer interface and neural plasticity.

Jeneva A. Cronin received a BS degree in Biomedical Engineering from the University of Virginia in 2011. She started pursuing her doctoral degree in Bioengineering at the University of Washington in 2014 where her research focuses on encoding sensory feedback through cortical stimulation with brain-computer interfaces. Jeneva is an NSF Graduate Student Research Fellow, and is highly involved with the NSF Center for Sensorimotor Neural Engineering.

Jing Wu received the BSc degree in Bioengineering in 2011 from University of California, San Diego (UCSD), where he conducted published research in functional MRI imaging of affective disorders. Since 2013 he has been a PhD student with the Bioengineering department of University of Washington. His research focus includes fundamental neuroscience of motor control, and decoding of hand and arm movement from cortical signals using tools from signal processing, machine learning, and control theory.

Kelly L. Collins received a BS degree in Electrical and Computer Engineering and Biomedical Engineering and a BS in Biology from Carnegie Mellon University in Pittsburgh, Pennsylvania in 2005 and an MD from the University of Michigan in 2011. Since then she has been training as a resident physician in the Department of Neurological Surgery at the University of Washington. Her research interests include brain-computer interfaces, sensorimotor systems, functional neurosurgery, and stimulation-based therapeutics for neurological disease.

Devapratim Sarma received the BS degrees in Bioengineering and Animal Physiology & Neuroscience from the University of California, San Diego in 2009. He was a research associate at the Swartz Center for Computational Neuroscience (2008–2011). Currently, he is pursuing a doctoral degree in the Department of Bioengineering at the University of Washington. As a member of the Center for Sensorimotor Neural Engineering, his current research focuses on optimizing electrocorticographic brain-computer interfaces for cognitive and motor rehabilitation. He is a student member of IEEE.

Rajesh P. N. Rao is the Director of the NSF Center for Sensorimotor Neural Engineering and a Professor of Computer Science and Engineering at the University of Washington. He is the recipient of a Guggenheim fellowship, a Fulbright Scholar award, an NSF CAREER award, an ONR Young Investigator Award, a Sloan Faculty Fellowship, and a Packard Fellowship for Science and Engineering. He is the author of Brain-Computer Interfacing (Cambridge University Press, 2013) and co-editor of Probabilistic Models of the Brain (MIT Press, 2002) and Bayesian Brain (MIT Press, 2007). His research spans the areas of computational neuroscience, artificial intelligence, and brain-computer interfacing.