Remapping cortical modulation for electrocorticographic brain-computer interfaces: a somatotopy-based approach in individuals with upper-limb paralysis

Abstract

Objective

Brain-computer interface (BCI) technology aims to provide individuals with paralysis a means to restore function. Electrocorticography (ECoG) uses disc electrodes placed on either the surface of the dura or the cortex to record field potential activity. ECoG has been proposed as a viable neural recording modality for BCI systems, potentially providing stable, long-term recordings of cortical activity with high spatial and temporal resolution. Previously we have demonstrated that a subject with spinal cord injury (SCI) could control an ECoG-based BCI system with up to three degrees of freedom [Wang et al., 2013]. Here, we expand upon these findings by including brain-control results from two additional subjects with upper-limb paralysis due to amyotrophic lateral sclerosis and brachial plexus injury, and investigate the potential of motor and somatosensory cortical areas to enable BCI control.

Approach

Individuals were implanted with high-density ECoG electrode grids over sensorimotor cortical areas for less than 30 days. Subjects were trained to control a BCI by employing a somatotopic control strategy where high-gamma activity from attempted arm and hand movements drove the velocity of a cursor.

Main results

Participants were capable of generating robust cortical modulation that was differentiable across attempted arm and hand movements of their paralyzed limb. Furthermore, all subjects were capable of voluntarily modulating this activity to control movement of a computer cursor with up to three degrees of freedom using the somatotopic control strategy. Additionally, for those subjects with electrode coverage of somatosensory cortex, we found that somatosensory cortex was capable of supporting ECoG-based BCI control.

Significance

These results demonstrate the feasibility of ECoG-based BCI systems for individuals with paralysis as well as highlight some of the key challenges that must be overcome before such systems are translated to the clinical realm.

1. Introduction

Brain-computer interface (BCI) technology aims to provide functional restoration to individuals with movement disorders such as spinal cord injury or amyotrophic lateral sclerosis. Electrocorticography (ECoG) records brain activity with electrodes placed on the cortical surface, and has emerged as a promising neural recording modality for BCIs. ECoG recordings from individuals undergoing clinical presurgical brain mapping have been used to investigate human motor [Arroyo et al., 1993, Miller et al., 2007a, Crone et al., 1993, Crone et al., 1998b, Crone et al., 1998a], sensory [Chestek et al., 2013, Wahnoun et al., 2015, Sun et al., 2015, Branco et al., 2017], language [Crone et al., 2001, Mainy et al., 2007, Kellis et al., 2010, Wang et al., 2011, Pei et al., 2011], and other cognitive functions [Edwards et al., 2005, Trautner et al., 2006, Lachaux et al., 2005, Tallon-Baudry et al., 2005, Jung et al., 2008, Ray et al., 2008]. In particular, movement-related information, including individual finger movements, hand posture [Wang et al., 2009, Miller et al., 2009b, Kubánek et al., 2009, Degenhart et al., 2011a, Chestek et al., 2013, Collinger et al., 2014, Flint et al., 2017], and arm movement trajectories can be extracted from ECoG activity [Shimoda et al., 2012, Chao et al., 2010, Bundy et al., 2016]. Such work has enabled the study of ECoG-based BCI systems, including real-time one- and two-dimensional computer cursor control by individuals undergoing clinical ECoG monitoring [Leuthardt et al., 2004, Leuthardt et al., 2006, Felton et al., 2007, Schalk et al., 2008]. Additionally, closed-loop control of prosthetic limbs using ECoG has also been demonstrated in both able-bodied subjects [Hotson et al., 2016] and an individual with sensorimotor impairment [Yanagisawa et al., 2012].

While these studies have provided evidence highlighting the potential of ECoG-based BCIs, a number of questions remain about the feasibility of clinical ECoG BCI systems. Key among these is how best to use ECoG to control high degree-of-freedom (DoF) effectors. In particular, it is unclear what the most effective behavioral strategy is to generate the multiple independent control signals necessary for high-DoF control. Previous BCI studies using single/multi-unit activity (SU/MUA) recorded with intracortical microelectrodes have shown that human subjects can control computer cursors or robotic arms simply by attempting to make the intended movement [Hochberg et al., 2006, Collinger et al., 2012, Hochberg et al., 2012]. This direct mapping approach requires the encoding of the intended movement kinematics in the neural activity being recorded, and potentially enables intuitive and natural control.

In contrast, current EEG and ECoG recording technology is not capable of recording cortical activity at the same spatial resolution as penetrating microelectrodes, and as a consequence, these recordings likely contain less information about detailed movement kinematics than SU/MUA. Hence, most EEG and ECoG BCI studies have used an abstract somatotopic remapping approach in which subjects associate attempted movements with desired BCI control signals, e.g., tongue and hand movements for moving the cursor in the vertical and horizontal directions, respectively [Leuthardt et al., 2004]. Such an approach allows any cortical activity capable of being robustly modulated to be re-purposed for continuous BCI control. Apart from several notable demonstrations of prosthetic hand control [Yanagisawa et al., 2012, Hotson et al., 2016], most ECoG BCI studies have relied upon this approach. However, it is currently unclear how effective the somatotopic remapping approach is for high degree-of-freedom control. In particular, it is possible that as the number of independent control signals needed for BCI control is increased, the spatial resolution of ECoG may place a limit on what can be extracted from motor cortex. In this case, it may be necessary to use the activity of multiple cortical areas to achieve successful high-DoF BCI control using ECoG [Branco et al., 2017].

Though primary motor cortex has been the focus of most BCI studies using ECoG and intracortical microelectrodes, there are reasons to believe that somatosensory cortex may also be capable of supporting robust BCI control. Activation of both pre and post-central gyri is often observed in individuals with chronic spinal cord injury during attempted movement [Cramer et al., 2005, Shoham et al., 2001, Hotz-Boendermaker et al., 2008] and in able-bodied individuals during motor imagery in the absence of overt movement [Miller et al., 2010, Christensen et al., 2007, Lacourse et al., 2005, Porro et al., 1996]. Such somatosensory cortical activity may represent efferent copies of motor control signals [Christensen et al., 2007, Crapse and Sommer, 2008, Gritsenko et al., 2007], or reflect engagement of sensory imagery [Hotz-Boendermaker et al., 2008]. Furthermore, our previous work has presented evidence suggesting that somatosensory cortex can support ECoG-based BCI control [Wang et al., 2013]. However, a detailed examination of the ability of individuals with paralysis to voluntarily modulate somatosensory cortex in order to control a BCI device has not been conducted.

Ultimately, the utility of BCI systems for functional restoration is determined by how reliably they can be controlled by individuals with movement deficits such as upper-limb paralysis. The majority of ECoG-based studies have been conducted in able-bodied individuals undergoing clinical ECoG monitoring for epilepsy treatment or in able-bodied non-human primates [Rouse and Moran, 2009, Chao et al., 2010, Williams et al., 2013]. These studies have provided critical theoretical and practical foundations for ECoG-based BCI applications. However, individuals with movement disorders may face unique challenges obtaining BCI control. Chronic paralysis may result in cortical reorganization and neuronal loss due to degenerative processes in the case of amyotrophic lateral sclerosis (ALS), or loss of connection to spinal and/or peripheral neural networks in the case of spinal cord injury (SCI) or brachial plexus injury [Turner et al., 2001, Wrigley et al., 2009, Verstraete et al., 2010]. These anatomical and neurophysiological changes can negatively impact an individual’s ability to modulate motor and somatosensory cortical activities for BCI control [Yanagisawa et al., 2012]. Thus, it is vital to conduct ECoG BCI studies in individuals with arm paralysis to identify important clinical and translational challenges and guide the research and development of ECoG BCI systems.

Previously, we have presented work demonstrating three-dimensional ECoG-based BCI control by an individual with SCI [Wang et al., 2013]. Here, we expand upon this by including results from two additional subjects with upper-limb paralysis: one with ALS and another with brachial plexus injury. High-density ECoG grids were implanted over sensorimotor cortical areas of each participant for one month. We evaluated ECoG signal modulation during different attempted movements and show that subjects can successfully employ a somatotopic remapping strategy in order to control two- and three-dimensional (2D and 3D) computer cursor movements using ECoG signals recorded from the sensorimotor cortex. Furthermore, we find that somatosensory cortex can be activated by attempted arm and hand movements of individuals with upper-limb paralysis, and that this activity can be leveraged for BCI control. These results demonstrate the potential of ECoG-based BCI systems to provide functional restoration for individuals with various movement disorders.

2. Methods

2.1. Subjects and surgical procedures

All study procedures were approved by the Institutional Review Board at the University of Pittsburgh and followed all guidelines for human subject research. Written informed consent was obtained before study initiation.

Subject 1 (S1) was a 30-year-old right-handed male with tetraplegia caused by a complete C4 level spinal cord injury seven years prior to the study; BCI control results for this subject have been published previously [Wang et al., 2013]. Subject 2 (S2) was a 54-year-old right-handed male with amyotrophic lateral sclerosis (ALS) diagnosed 9 years prior to enrollment in the study. Subject 3 (S3) was a 24-year old male with left brachial plexus injury 3 years prior to enrollment in the study. All subjects were capable of neck and shoulder movement but could not initiate voluntary arm or hand movement of the affected limb. Subjects S2 and S3 were ambulatory at the time of the study.

Functional neuroimaging was used to identify pre- and post-central gyrus targets for ECoG grid placement. Specifically, functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) were used to map cortical activity during attempted movement and imagined tactile sensations of participants paralyzed limb. Specifics of these mapping techniques for placing BCI electrodes in participants with paralysis have been published [Collinger et al., 2013, Flesher et al., 2016]. S3 did not undergo fMRI mapping due to a spinal cord stimulator that was not MRI-compatible. However, a low-resolution clinical MRI was available for anatomical localization for Subject S3.

Subjects were implanted with high-density ECoG grids (PMT Corp, Chanhassen, MN USA) consisting of 32 (subject S1), 34 (subject S2), or 64 (subject S3) platinum disc electrodes embedded in a 2cm × 4cm (subjects S1, S2) or 4cm × 4cm (subject S3) silicone sheet. Electrodes were either 2mm or 3mm in diameter and were spaced 4mm apart, with ground and reference electrodes facing the dura. Electrode grids were implanted subdurally over the hand and arm areas of left (subjects S1 and S2) or right (subject S3) sensorimotor cortex through a small craniotomy (approximately 3cm × 3cm for subjects S1 and S2, 5cm × 5cm for subject S3). Following placement of the electrode grid, the dura was reapproximated and the bone flap was replaced and affixed to the skull using titanium straps. Electrode leads were tunneled subcutaneously to the chest and exited the skin below the clavicle. Approximate electrode locations on the cortical surface were determined post-operatively using computed tomography (CT) imaging combined with cross-registration with pre-operative structural MRI scans [Hermes et al., 2010, Miller et al., 2007b] for subjects S1 and S2. When necessary, manual corrections were applied to electrode locations based on intraoperative images. For subject S3, skull radiographs and low-resolution clinical MRI imaging combined with cortical stimulation were used to identify the approximate locations of electrodes on the cortical surface. Figure 1 shows post-operative radiographs of the implanted electrode grids and the approximate location of the electrodes on the cortical surface. Consistent with U.S. FDA 510(K) regulations for these ECoG electrode grids, electrodes were explanted after 28, 31^†, and 28 days for subjects S1, S2, and S3, respectively.

Figure 1. ECoG grid design and location.

(A, C, E) Post-operative head radiographs showing implanted ECoG grids for subjects S1, S2, and S3, respectively. (B, D, F) Location of ECoG electrodes on the cortical surface. Electrode locations were determined using post-operative head radiograph, computed tomography (CT), structural MRI images, and intra-operative images. Panel (F) shows electrodes localized on a standard brain template due to the unavailability of a high-resolution MRI for Subject S3. Head radiographs from subjects S1 and S3 are reproduced from [Ritaccio et al., 2013] and [Hiremath et al., 2017], respectively. Cortical surface localization for Subject S1 is reproduced from [Wang et al., 2013].

2.2. Neural recording and preprocessing

Neural signals were recorded and digitized at 1200Hz using the g.USBamp biosignal amplification system (Guger Technologies, Austria) and processed in 33.3ms (40 sample) blocks, resulting in a 30Hz system update rate. Dura/skull-facing electrodes served as reference and ground electrodes for all recordings. Raw time-domain signals were notch-filtered at 60Hz, 120Hz, and 180Hz to remove power line noise artifacts. Spectral power of the filtered signals was computed using the Burg autoregressive method [Kay and Marple, 1981] over the (0–200 Hz) frequency range (25th order, 10Hz frequency bands) using 300ms (Subject S1) and 100ms (subjects S2 and S3) sliding windows (33.3 ms step size). Instantaneous power estimates for each feature were log-transformed and then converted to pseudo Z-scores by subtracting the mean and dividing by the standard deviation of the log-power during a baseline resting condition [Edwards et al., 2009, Ray et al., 2008]. This baseline resting data consisted of an approximately 60-second period typically collected at the beginning of each experimental session where the subject was instructed to relax and stare at the center of the computer monitor. All signal processing, neural decoding, and experiment control was performed using Craniux, a LabVIEW-based open-source BCI software suite [Degenhart et al., 2011b].

2.3. Movement screening task

A motor screening task was performed prior to the initiation of brain control and was used to identify attempted movements eliciting strong cortical modulation. The movements identified using this task would serve as the basis for the BCI control strategy used for each subject (see Section 2.5). Subjects sat in front of a display and were presented with videos of a first-person view of isolated, planar movements performed by either a virtual avatar (Subject S1) or a researcher (subjects S2 and S3). Stimuli consisted of shoulder ab/adduction, shoulder flexion/extension, shoulder internal/external rotation, elbow flexion/extension, wrist flexion/extension, wrist pronation/supination, whole-hand grasp, and flexion/extension of individual fingers. Subjects were instructed to attempt the movement depicted by the stimulus.

For Subject S1, movements of the virtual avatar were presented in a continuous manner, with joint angle position driven by a 0.5 Hz sinusoid such that each movement phase (e.g. flexion) was 1s in duration. A single movement sequence (stimulus) consisted of 5 repetitions (cycles) of this movement. For subjects S2 and S3, individual movements consisted of an initial hold period (2s), the first movement phase (e.g. flexion, 1s in duration), a second hold period (2s), and the second movement phase (e.g. extension, 1s in duration), with an entire movement sequence consisting of 5 repetitions of this. As opposed to the movement sequence used for subject S1, this movement sequence allowed cortical responses to the individual movement phases to be isolated. For both subjects, individual movement sequences for selected stimuli were presented in a pseudorandom order interleaved with 2s inter-trial interval periods.

2.4. Brain control task

The majority of study sessions consisted of a center-out cursor control task in a virtual environment [Moran and Schwartz, 1999, Taylor et al., 2002, Wang et al., 2007]. During this task, subjects were given control of a “cursor”, rendered as a sphere in a three-dimensional workspace, and instructed to guide this cursor towards spherical targets equidistant from the center of the workspace. The virtual environment utilized a right-handed Cartesian coordinate system where the X-axis pointed to the subjects right, the Y-axis pointed upward, and the Z-axis pointed toward the subject. The center of the cursor was constrained within the workspace.

Subjects performed the cursor control task with both 2D and 3D target configurations, beginning with the 2D task and progressing to the 3D task once satisfactory 2D performance had been achieved. The 2D target configuration consisted of a set of 4 (Subject S2) or 8 (subjects S1, S3) uniformly-distributed targets presented in the X-Y plane, while the 3D target configuration (subjects S1, S2) consisted of 4-target planes at two different Z-axis coordinates. Trials consisted of presentation of one pseudo-randomly selected target in the workspace; the subject was required to acquire this target with the cursor in order to complete the trial; trials were considered successful if the cursor overlapped with the target at any point (i.e. no target hold time was enforced). For Subject S1, maximum trial durations of 5s (2D) and 7s (3D) were enforced while trial durations of 2s/3s were enforced for subjects S2 and S3; trials in which the subject was not able to acquire the target before the end of the trial were considered as failed trials. Task parameters were changed across subjects in order to improve performance; a 4-target 2D task was used for Subject S2 in order to speed transition to the 3D cursor control task, while trial durations were reduced for subjects S2 and S3 in order to encourage ballistic cursor movements and to increase subject engagement.

2.5. Determination of BCI control strategies

In order to generate directionally-modulated cortical activity to serve as the basis for closed-loop control, subjects were instructed to use a somatotopic control strategy in which they associated attempted arm and hand movement with desired cursor movement direction. Prior to the initiation of any brain control trials, time-frequency responses during the movement screening task (see Section 2.3) were examined to identify those movements eliciting robust broad-band gamma modulation. A subset of movements were then chosen which exhibited spatially-distinct patterns of gamma and high-gamma band (40 – 200Hz) activity; these movements were mapped onto the cursor workspace such that combinations of the identified arm and hand movements moved the cursor in specific directions (see below). In some cases where similar attempted movements elicited similar patterns of robust gamma-band modulation, these movements were combined (e.g., simultaneous ring and little finger movement). Once three such movements were identified, they were mapped onto the cursor movement workspace as shown by Figure 2. Movements eliciting distinct activity patterns were selected irrespective of the source (i.e. motor or somatosensory cortex) of the activity.

Figure 2. Illustration of the somatotopic control strategy.

Arrows indicate desired cursor velocity, with the associated attempted movement shown next to the target (e.g. ‘M1’, ‘M2’, ‘M3’). ‘+’: attempted movement, ‘ø’: relaxation. Strategies are shown for both two-dimensional (left) and three-dimensional (middle, right) cursor tasks. Control strategies for the 3D cursor task has been broken into that used for targets in the far plane (away from the subject, negative Z-axis) and that used for targets in the near plane (towards the subject, positive Z-axis).

The first two movements were arranged in a “push–pull” configuration to control movement of the cursor along the X-axis such that attempting movement M1 would move the cursor in the positive X-direction and attempting movement M2 would move the cursor in the negative X-direction. Movement in the positive Y-direction was generated by attempting M1 and M2 simultaneously, while movement in the negative Y-direction movement was generated by relaxing (i.e. no movement). For example, subject S1 achieved 2D cursor control using right hand movements to move the cursor to the right, attempted right elbow movements move the cursor to the left, simultaneous hand and elbow moved the cursor up, and the absence of attempted movements drove the cursor down. For three-dimensional control, movement M3 was directly mapped to the Z-axis such that attempting M3 would move the cursor in the positive Z-direction and rest would move the cursor in the negative Z-direction.

2.6. Decoder calibration

Normalized time-frequency data over the 40–200 Hz gamma/high-gamma frequency range (16 frequency bands per electrode) was used as the basis for real-time brain control. Dura-facing electrodes, as well as those containing high amounts of noise indicative of poor electrical connectivity, were removed from the set of electrodes used for control. In total, 4, 7, and 4 electrodes were excluded from real-time control for subjects S1, S2, and S3, respectively, resulting in feature sets of 448 (28 electrodes × 16 frequency bands), 432 (27 electrodes × 16 frequency bands), and 960 (60 electrodes × 16 frequency bands) neural features. Neural calibration data consisted of time-averaged pseudo-z-scored spectral power for each trial, beginning 0.5s after the presentation of the target and ending at the completion of the trial (Subject S1) or after a maximum of 3.5s (subjects S2, S3). Kinematic calibration data consisted of the intended target direction for each trial, defined as a unit vector pointing from the center of the workspace to the target.

Intended velocity command signals were predicted from instantaneous feature activities in real-time using Equation 1,

$$\widehat{\mathbf{v}}={\mathbf{W}}^{\top }\mathbf{f}$$ (1)

where W ∈ ℝ^D×M is a decoding weight matrix mapping the D-dimensional feature vector f ∈ ℝ^D×1 onto the M-dimensional command velocity vector $\widehat{\mathbf{v}}\in {\mathrm{ℝ}}^{M\times 1}$. For subject S1, linear regression was used to find W using Equation 2:

$${\mathbf{W}}^{\top }=\mathbf{V}{\mathbf{F}}^{†}$$ (2)

where V = [v₁, …, v_N] and F = [f₁, …, f_N ]^⊤ are concatenated matrices of the N time-averaged single-trial observations of the desired movement direction and associated neural feature activity during decoder calibration (see below). The superscript “†” denotes the Moore-Penrose pseudoinverse.

For subjects S2 and S3, the Optimal Linear Estimator (OLE) [Salinas and Abbott, 1994, Kass et al., 2005, Wang et al., 2007, Chase et al., 2009] was used to find weight matrix W. In comparison to the linear regression approach used for Subject S1, the OLE provides an explicit noise model for each neural feature and is less biased than the Population Vector Algorithm [Chase et al., 2009]. Estimation of OLE decoding parameters began by first fitting an encoding model for each neural feature of the form:

$$f_i=b_0+{\mathbf{b}}_i^{\top }\mathbf{d}+\epsilon$$ (3)

where f_i is the instantaneous feature activity, b₀ is a constant offset term, d ∈ ℝ^D×1 is the intended cursor movement direction, and b_i ∈ ℝ^D×1 is the preferred direction vector relating intended cursor movement direction to neural activity for the ith neural feature. Preferred direction vectors from all features were collected into a single matrix B = [b₁, …, b_N], from which the decoding weight matrix was found according to Equation 4.

$${\mathbf{W}}^{\top }={({\mathbf{B}}^{\top }\mathbf{B})}^{-1}{\mathbf{B}}^{\top }$$ (4)

For Subject S1, initial decoder calibration was performed using 40 trials in which the cursor was automatically moved to the target (observation-based calibration). Subsequent calibration sessions consisted of 16-trial blocks of closed-loop cursor control, during which the subject was instructed to direct the cursor to the target as quickly as possible without correcting for movement errors. In order to ensure neural decoding weights changed gradually during calibration, updated decoding weights were calculated as a weighted sum of the old decoding weights and those estimated from the newly-acquired calibration data. Additional rounds of decoder calibration were performed once performance was deemed to plateau (see [Wang et al., 2013] for details).

For subjects S2 and S3, decoder calibration occurred more frequently and was commonly performed at the start of each experimental testing session. Calibration sessions were typically run in 80-trial (2D) or 96-trial (3D) blocks. As with Subject S1, initial decoder calibration was performed using an observation-based calibration procedure. Subsequent decoder calibration was performed under closed-loop control when possible, though observation-based control was used in cases where closed-loop performance using the most recent decoder was poor.

During calibration, subjects were instructed to use the identified control strategy (see Section 2.5) and to not attempt to correct for errors in cursor movements. Computer assistance [Velliste et al., 2008] was used as needed to facilitate the acquisition of closed-loop control. This assistance attenuated the component of decoded cursor velocities orthogonal to the vector between the instantaneous cursor position and the target. In general, this assistance was provided in order to aid subjects in the successful acquisition of cursor control and was removed as rapidly as possible. All evaluations of cursor control were performed on unassisted (i.e., full brain control) trials.

2.7. Characterization of brain-controlled cursor movement

Brain control performance was characterized using success rate, corrected success rate, time to target, distance ratio, and fraction of time spent at the boundary of the workspace. Success rate was defined as the number of successful trials divided by the total number of trials. The corrected success rate was defined as the expected success rate had all possible targets been presented simultaneously on each trial. To calculate corrected success rate, the success rate was re-computed with all trials where the observed cursor trajectory would have hit one of the non-presented targets before the presented target were marked as unsuccessful. The success rate and corrected success rate metrics were calculated using all trials; the remaining metrics were calculated using data from successful trials only.

Time to target was defined as the average time from the onset of brain control until contact with the target. The distance ratio (DR), quantifying the deviation of the cursor trajectory from the shortest straight-line path to the target [Simeral et al., 2011], was calculated according to Equation 5,

$$DR=\frac{1}{N}{\displaystyle \sum _{i=1}^N\frac{1}{\Vert {\mathbf{p}}_t^i-\mathbf{c}\Vert -r_t}}{\displaystyle \sum _{j=1}^{M_i-1}\Vert {\mathbf{p}}_c^{j+1}-{\mathbf{p}}_c^j\Vert }$$ (5)

where N is the number of trials, M_i is the number of time points in the ith trial, ${\mathbf{p}}_c^j$ is the position of the cursor at the jth time point, ${\mathbf{p}}_t^i$ is the position of the target for the ith trial, c is the center of the workspace, and r_t is the radius of the target. Lower distance ratios indicate straighter trajectories; a distance ratio of 1 indicates a perfectly straight trajectory. Finally, the boundary fraction, indicating the fraction of time in which the cursor was at the edge of the workspace boundary, was calculated as the number of time points in which the cursor was at a workspace boundary divided by the total number of time points. This metric provided a means with which to assess subjects’ reliance upon the workspace boundary constraint to successfully complete the task.

2.8. Comparison of cortical activity during screening and cursor control tasks

Cortical activity during the movement screening and cursor control tasks was compared in order to characterize the extent to which subjects were able to re-map attempted arm and hand movements onto intended cursor movement. Rather than use normalized spectral power, root-mean-square (RMS) signal amplitudes were utilized in order to allow a more direct comparison of signal amplitude across tasks. This analysis focused on the gamma-band (70–110 Hz) frequency range, which has been shown to be representative of the broadband increase in the spectral power of ECoG signals during movement while avoiding power line noise harmonics [Miller et al., 2007a]. Raw ECoG signals were band-pass filtered over the 70–110 Hz frequency range using a 4th order Butterworth filter. RMS signal amplitudes were then calculated for pre-movement (rest) and movement epochs for both tasks. For the screening task, the pre-movement epoch was defined as the 500ms window immediately preceding the onset of the instructed movement stimulus, while the movement epoch was defined as the 500ms window beginning 250ms after stimulus onset. For the cursor control task, the pre-movement epoch was defined as the 500ms window immediately preceding the appearance of the BCI target, while the movement epoch was defined as the 500ms window beginning 250ms after the appearance of the target. This analysis was limited to the subset of movements used as the basis for 2D control (see Section 2.5). For the screening task, this consisted of the trials where the presented visual stimulus was one of the two movements used as the basis for 2D cursor control. For the cursor control task, trials where the instructed control strategy for specific targets consisted of isolated individual movements (see Figure 2) were used. This analysis was limited to the same set of 2D cursor control trials used to characterize 2D cursor control performance (see Section 2.7).

The number of significantly-modulated electrodes, number of significantly-tuned electrodes, and depth of modulation were compared between screening and cursor control tasks independently for each subject; this analysis was restricted to the set of electrodes used for closed-loop control (see Section 2.6). Electrodes were considered modulated if their activity increased during movement, and tuned if their activity differed across task stimuli (instructed movement for the screening task, target for the cursor task). To determine the number of electrodes showing a significant increase in gamma-band modulation, RMS amplitudes were compared between the pre-movement and movement epochs for each electrode and task condition using a one-tailed Wilcoxon sign-rank test (α = 0.05). The number of significantly-tuned electrodes was determined by comparing RMS amplitudes across the two movement conditions for each task during the movement epoch using a Wilcoxon rank-sum test (α = 0.05). Finally, to compare depth of modulation, the difference in RMS amplitude between the movement and pre-movement epochs for each task was calculated for each trial and averaged across set of significantly-modulated electrodes.

2.9. Assessing contribution of cortical areas to closed-loop control

In order to determine the contribution of motor and somatosensory cortical areas to closed-loop control, cortical modulation during cursor control and the contribution of this activity to instantaneous cursor velocities were examined for Subject S3, for whom electrode coverage spanned the pre- and post-central gyri^‡. This analysis was limited to successful 2D cursor control trials.

Electrodes were classified as either motor or somatosensory based on the cortical location of the electrodes from intra-operative photographs and cortical mapping using electrical stimulation as described in [Hiremath et al., 2017]. To identify those electrodes modulated during closed-loop control, the subsets of motor and somatosensory electrodes exhibiting significant increase in gamma-band modulation were first identified for each target. Activity over the (70–110 Hz) range was averaged over the (−2 to −1 s) pre-movement and (0 to 1 s) movement epochs relative to target onset. A one-sided Wilcoxon signed-rank test was used to determine the electrodes that exhibited statistically-significant increases in cortical modulation between the pre-movement and movement epochs; subsequent analyses were limited to this electrode set. Statistical significance was determined using p-values corrected with the False Discovery Rate method [Benjamini and Hochberg, 1995, Genovese et al., 2002] (q = 0.05, nonparametric p-value threshold). In order to gain insight into whether somatosensory activity was volitionally modulated or rather the result of an efference copy from motor cortical areas, we characterized the difference in the time course of activation of motor and somatosensory electrodes. To accomplish this, single-trial time-varying gamma-band activity was smoothed and averaged across the identified electrode subsets (motor or somatosensory) for each trial, and then averaged across trials to obtain an overall temporal profile of activation of motor and somatosensory cortical areas during cursor control. The difference in the onset of gamma-band activity between motor and somatosensory cortical areas was determined by comparing the times at which activity first exceeded 50% of the maximum of each trial for the identified electrode subsets.

In order to assess the contribution of motor and somatosensory electrodes to the cursor control signal, instantaneous cursor velocities were decomposed according to Equation 6:

$$\widehat{\mathbf{v}}={\widehat{\mathbf{v}}}_m+{\widehat{\mathbf{v}}}_s={\mathbf{W}}_m^{\top }{\mathbf{f}}_m+{\mathbf{W}}_s^{\top }{\mathbf{f}}_s$$ (6)

where ${\widehat{\mathbf{v}}}_m$ and ${\widehat{\mathbf{v}}}_s$ are the motor and somatosensory contributions to the final velocity control signal $\widehat{\mathbf{v}}$, W_m and W_s are the components of the full decoding weights matrix W for the motor and somatosensory electrodes, respectively, and f_m and f_s are the instantaneous neural activity for the motor and somatosensory electrodes, respectively. We compared the direction of ${\widehat{\mathbf{v}}}_m$ and ${\widehat{\mathbf{v}}}_s$ to the target direction (calculated from the current cursor position) at each timestep, as well as the direction of ${\widehat{\mathbf{v}}}_m$ to ${\widehat{\mathbf{v}}}_s$. We also compared the normalized magnitudes ${\widehat{\mathbf{v}}}_m/\Vert \widehat{\mathbf{v}}\Vert$ and ${\widehat{\mathbf{v}}}_s/\Vert \widehat{\mathbf{v}}\Vert$ at each timestep.

3. Results

3.1. Cortical activity during motor screening

ECoG signals recorded from the sensorimotor cortex of all subjects demonstrated robust modulation during attempted arm and hand movement, even when overt movements were not possible due to chronic paralysis. Figure 3 shows time-frequency responses for all recording electrodes during selected attempted movements. Characteristic increases in the activity of the gamma/high-gamma frequency bands (40–200 Hz) and decreases in the sensorimotor rhythm (10–30 Hz), tightly time-locked to stimulus onset, was observed for a number of attempted movements. Spatial patterns of high-gamma modulation followed the expected somatotopic organization of motor (subjects S2, S3) and somatosensory (subjects S1, S3) cortices, with the centroids of activity located medially on pre- and post-central gyri for proximal arm movements and laterally for distal arm movements.

Figure 3. Averaged time-frequency responses across all electrodes during attempted movements.

Each inset plot shows the time-frequency response for a single electrode averaged across repetitions of one attempted movement. Instructed kinematic profiles are indicated by the black line in each plot. (A) Subject S1, hand flexion/extension. (B) Subject S1, elbow flexion/extension. (C) Subject S2, wrist flexion/extension. (D) Subject S2, middle finger flexion. (E) Subject S3, little finger extension. (F) Subject S3, elbow flexion/extension. Approximate location of the central sulcus (CS) as determined by electrical stimulation is shown for Subject S3 by thick black lines in (E) and (F).

Once cortical responses to attempted arm and hand movements were examined, a subset of movements were chosen to serve as the basis for cursor control using a somatotopic control strategy as described in Section 2.5. The movements serving as the basis for closed-loop control are summarized in Table 1. For subjects S1 and S3, the control strategy used for 2D control was preserved when transitioning to 3D control in order to retain any adaptation that may have occurred during 2D control. For Subject S2, a new set of movements was identified in an effort to improve our ability to extract 3 independent control signals.

Table 1. BCI control strategies.

M1, M2, and M3 indicate the attempted movements used during cursor control (see Figure 2).

	2D		3D
	M1	M2	M1	M2	M3
S1	Hand	Elbow	Hand	Elbow	Wrist
S2	Thumb	Middle finger	Thumb	Little finger	Elbow
S3	Ring/little finger	Elbow	Ring/little finger	Elbow	Wrist

3.2. Cortical control of cursor movement

The general progression of the cursor control task, including success rate and amount of computer assistance, is shown by Figure 4. Subjects began with a 4-target 2D task and progressed to an 8-target 3D task once satisfactory 2D control was achieved. Subjects S1 and S3 performed an 8-target 2D task between 4-target 2D and 8-target 3D control; this was not performed for Subject S2 in favor of transitioning to 3D control as quickly as possible. Computer assistance [Velliste et al., 2008] and decoder re-calibration were utilized as-needed in order to facilitate acquisition of brain control. In general, computer assistance was used during initial learning for a particular control strategy, while decoder calibration was used as-needed to improve control performance. Characterizations of performance, including cursor trajectories (Figures 5 and 6) and control performance metrics (Table 2) were performed on trials where satisfactory 2D and 3D control was achieved without the need for computer assistance (see highlighted regions in Figure 4).

Figure 4. BCI control performance across days for subjects S1 (A) S2 (B), and S3 (C).

BCI control success rate (black) and computer assist level (red) is plotted as a function of testing day. Each point represents a single “block” of closed-loop control consisting of 40 trials. Alternating white and gray regions mark individual days, while vertical blue lines mark the occurrence of neural decoder calibration. White bars above each panel indicate the somatotopic control strategy used for that period, while the green, yellow, and red bars indicate the task difficulty. Days without success rate data (days 16, 22, and 23 for Subject S1, days 8, 15, and 22 for Subject S2, days 6, 7, 13, and 14 for Subject S3) were planned days off. Green dots indicate the blocks used to evaluate cursor control performance.

Figure 5. Two-dimensional cursor control trajectories.

Averaged trajectories across successful non-assisted BCI trials are shown for subjects S1 (left), S2 (middle), and S3 (right). Colors of the individual trajectories correspond to their respective targets.

Figure 6. Three-dimensional cursor control trajectories.

Averaged trajectories are shown for subjects S1 (left column) and S2 (middle column), and S3 (right column). Colors of the individual trajectories correspond to their respective targets. Trajectories have been separated into front (top row), top (middle row), and side (bottom row) for the sake of clarity. Trajectories towards targets in the far plane of the workspace are indicated by dashed lines. Note that each 3D view results in the obstruction of one or more targets; while the targets themselves are obscured the trajectories are plotted as normal.

Table 2. Cursor control performance metrics.

Number of trials, success rate (SR), corrected success rate (SR_c), time-to-target (TT), distance ratio (DR), and boundary fraction (BF) are provided for selected peak performance periods for 2D and 3D control (see Section 2.7 for additional information about performance metrics). Data for time-to-target, distance ratio, movement error, and boundary fraction are presented as mean ± standard deviation. All performance metrics were computed during full brain-control trials (i.e., without computer assistance).

Subject	Task	Trials	Success rate	Success rate (corrected)	Time to target (s)	Distance ratio	Boundary fraction
S1	2D	176	0.87	0.82	2.27 ± 1.02	1.95 ± 0.74	0.00 ± 0.00
	3D	160	0.71	0.70	3.25 ± 1.31	2.45 ± 0.95	0.01 ± 0.04
S2	2D	680	0.78	0.77	0.72 ± 0.45	1.97 ± 0.98	0.01 ± 0.03
	3D	2160	0.68	0.48	0.86 ± 0.71	5.11 ± 3.96	0.17 ± 0.21
S3	2D	240	0.90	0.49	1.99 ± 1.30	3.62 ± 2.28	0.04 ± 0.09
	3D	480	0.86	0.78	1.76 ± 1.02	2.96 ± 1.52	0.09 ± 0.17

Averaged 2D and 3D cursor movement trajectories during periods of peak cursor control are shown by Figures 5 and 6, respectively. Qualitatively, 2D trajectories were relatively straight for subjects S1 and S2, while those for Subject S3 exhibited downward curvature. Curvature in the movement trajectories was more pronounced during 3D control for subjects S1 and S2 as compared to their 2D control. For these subjects, cursor movements during 3D control generally began with movement in the X–Y plane before moving in the Z-dimension.

As expected, increases in the spectral power of the gamma/high-gamma range and decreases in the sensorimotor rhythm were observed surrounding the onset of cursor movement. Time-frequency responses during closed-loop cursor control reflected the instructed control strategies; several representative examples of such responses are shown by Figure 7. For example, time-frequency responses of electrode 27 for Subject S2 exhibited increases in gamma-band activity for targets in the upper-right quadrant of the workspace (Figure 7D). This is consistent with the response of the same electrode during attempted middle finger flexion/extension (Figure 3D), and the instructed control strategy (Table 1), which associated attempted middle finger flexion with movements to the upward and right targets. Similar responses were observed for subjects S1 and S3 (Figure 7C-F).

Figure 7. Averaged time-frequency data for selected electrodes during two-dimensional cursor control.

Individual panels represent time-frequency data for a single electrode, averaged across trials, towards a single target. Responses are shown for the following: (A) Subject S1, electrode 5, (B) Subject S1, electrode 15, (C) Subject S2, electrode 5, (D) Subject S2, electrode 27. (E) Subject S3, electrode 37, (F) Subject S3, electrode 61. The layout of the time-frequency plots corresponds to the position of the targets in the workspace (see Figure 5). Dashed black lines indicate the onset of cursor control.

Performance metrics for 2D and 3D cursor control are summarized in Table 2. Success rates were 0.87, 0.78, and 0.90 for subjects S1, S2, and S3, respectively, during 2D cursor control, and 0.71, 0.68, and 0.86 during the 3D control task. For all subjects, corrected success rates were above those expected by chance (0.25 for 4-target tasks, 0.125 for 8-target tasks). Importantly, subjects were not instructed to minimize time-to-target, the straightness of cursor trajectories (distance ratio), or the amount of time spent at the edge of the workspace (boundary fraction). As a result, it is difficult to attribute differences in these metrics to differences across subjects. However, both movement error and boundary fraction increased between 2D and 3D tasks for all subjects, suggesting the difficulty of the 3D task was greater than that of the 2D cursor control task. Furthermore, the low boundary fraction values observed indicate that in most cases subjects did not rely on the workspace boundary constraints during closed loop control.

3.3. Comparison of ECoG activity during movement screening and cursor control tasks

The effectiveness of the somatotopic remapping approach was evaluated by comparing ECoG activity during the screening and cursor control tasks. This analysis focused on gamma-band RMS amplitude during movement screening for the subset of movements used as the basis for 2D cursor control (see Table 1), and that of the cursor control targets these movements were associated with (see Figure 2). For example, Subject S2 was instructed to use attempted thumb flexion to generate leftward cursor velocities and middle finger flexion to generate rightward velocities. Thus, screening activity during attempted thumb and middle finger movements during screening was compared to cursor control trials to the left and right targets. For Subject S1, post-hoc analysis of cortical activity during cursor control suggested that a change in the employed control strategy occurred over the course of 2D BCI training which resulted in electrodes active for elbow flexion being associated with upwards cursor movement. In this case, screening activity during hand and elbow flexion was compared to cursor control trials to the left and up targets. Furthermore, for Subject S3 a combination of ring and little finger movement was used to drive leftward cursor movements. In this case, screening activity during little finger movement, which generated stronger cortical activation than ring finger movement, was compared to cursor control trials to the left target.

The screening and cursor control tasks were both found to generate widespread cortical modulation resulting in a majority of the electrodes on the ECoG grid exhibiting significant modulation (Figure 8A). In contrast, the number of electrodes exhibiting significant tuning for one of the two movements (Figure 8B) was found to be smaller than the number of significantly-modulated electrodes, indicating that the modulation observed for many electrodes was not specific for a particular movement condition. Furthermore, the number of tuned electrodes was observed to increase from screening to cursor control for all subjects, suggesting that cortical activity became more specific for the instructed attempted movements as a result of brain control training. Additionally, while RMS amplitudes for the set of significantly-modulated electrodes were found to be comparable across the screening and cursor control tasks for each subject (Figure 8C), a significant increase in the depth of modulation from the screening to cursor control task was observed (Figure 8D, Wilcoxon rank sum test, p ≤ 1e − 21 for all subjects).

Figure 8. Comparison of gamma-band RMS signal amplitude during screening and cursor control tasks.

(A) Number of significantly-modulated electrodes during screening (blue) and cursor control (green) tasks. Shaded gray regions indicate the total number of valid electrodes for each subject. (B) Number of significantly-tuned electrodes during screening and cursor control tasks. (C) Gamma-band RMS amplitude during screening and cursor control tasks. Blue and green bars show data for screening and cursor tasks, respectively. Light and dark bars indicate pre-movement and movement epochs for each task. Error bars represent 95% confidence intervals. (D) Depth of modulation of gamma-band RMS signal amplitude for all significantly-modulated electrodes. Bar colors representing task condition are consistent with panels (A) and (B). Error bars represent 95% confidence intervals, while asterisks represent statistically-significant differences in depth of modulation across task conditions.

3.4. Contribution of somatosensory cortex to closed-loop control

As implanted electrodes covered somatosensory cortical areas for subjects S1 and S3, we sought to identify the potential of somatosensory cortex to contribute to closed-loop cursor control. For Subject S1, implanted electrodes were located predominately over the post-central gyrus (Figure 1B), with ECoG activity robustly modulated during both attempted movement screening tasks (Figure 3A,B) and cursor control (Figure 7A,B). This indicates that the subject was capable of volitionally modulating somatosensory cortical activity to drive cursor movement. For Subject S3, electrodes were located over both the pre and post-central gyri. To determine the contribution of somatosensory and motor cortices to cursor control for this subject, we examined both cortical modulation and the contribution of pre- and post-central gyrus electrodes to the closed-loop control signal.

Electrodes over both pre and post-central gyri showed statistically-significant increases in gamma-band activity during cursor control for Subject S3 (Figure 9A, B), with the spatial distribution of activity differing across target conditions. Overall, 44 of 60 electrodes (28 of 38 post-central, 16 of 22 pre-central) exhibited significant increases in high-gamma activity during 2D cursor control (Wilcoxon signed-rank test, p < 0.0039, FDR-corrected). Furthermore, both pre and post-central gyri were found to exhibit similar temporal profiles of activation (Figure 9C), with gamma-band activity of motor cortical areas leading that of somatosensory areas by 101.7±21.2 ms (mean ± sem, p < 1e−7, Wilcoxon signed-rank test). We examined the contribution of motor and somatosensory activity to cursor control by decomposing instantaneous cursor velocity control signals into components derived from motor and somatosensory electrodes. Velocity components from both pre and post-central electrodes were found to point in the direction of the target, indicating that target direction-specific information was contained in both electrode subsets (Figure 9D). Finally, we found that the magnitude of the somatosensory component of the cursor velocity exceeded that of the motor component (Figure 9D), suggesting that somatosensory cortical activity contributed more to cursor control ability than activity from motor cortical areas for Subject S3.

Figure 9. Contribution of somatosensory cortex to 2D cursor control for Subject S3.

(A) Average high-gamma-band activity during 2D cursor movement to targets 1 and 5. (B) Electrodes exhibiting statistically-significant high-gamma-band modulation (shown in gray) for at least one target condition. (C) Time course of high-gamma-band activation for motor (black) and somatosensory (red) electrode subsets relative to the onset of target presentation. Shaded regions indicate ± one standard deviation from the mean; vertical lines show median onset times. (D) Comparison of instantaneous control signal angular error for motor (pre-central, black) and somatosensory (post-central, red) electrode subsets. Left: histogram of angular error for the control signal component from motor electrodes. Middle: angular error of the control signal component from somatosensory electrodes. Right: difference between motor and somatosensory angular error. Histograms show counts of angular error across all timesteps, while vectors show the average control signal difference across all timepoints. (E) Histogram showing the contribution of motor (black) and somatosensory (red) electrode subsets to control signal magnitude.

4. Discussion

We investigated the feasibility of three individuals with upper limb paralysis to control an ECoG-based BCI using somatotopic command strategies. We found that subjects were capable of volitionally modulating high gamma-band activity in motor and somatosensory cortex during attempted movement of their paralyzed limb, and that the somatotopic organization of this activity was consistent with that observed in able-bodied individuals [Cramer et al., 2005, Shoham et al., 2001]. High-gamma band activity, believed to represent local neuronal population activity [Crone et al., 2006, Miller et al., 2009a], was tightly coupled to attempted arm and hand movement, consistent with previous reports of motor cortical neuronal activity recorded with intracortical microelectrode arrays in individuals with tetraplegia [Hochberg et al., 2006, Hochberg et al., 2012, Truccolo et al., 2008, Collinger et al., 2012]. Furthermore, we find that subjects were able to modulate their high-gamma-band activity using a somatotopic control strategy in order to achieve three-dimensional cursor control, and that somatosensory cortex, which natively receives sensory input, can be re-purposed to generate output signals for BCI control. Importantly, the entirety of the study, including implantation and explantation of the electrode grid, was performed without complications.

Previous studies have demonstrated ECoG-based BCI control by able-bodied individuals undergoing presurgical brain mapping [Schalk et al., 2008, Leuthardt et al., 2004, Miller et al., 2010]. Our results, extending those shown in [Wang et al., 2013], are the first demonstration of continuous and proportional three-dimensional BCI control using ECoG. We show that individuals with different movement disorders, including SCI, ALS, and brachial plexus injury, are capable of controlling an ECoG-based BCI. We believe this work suggests that the potential users of an ECoG BCI could include a broad population of individuals with motor disabilities beyond those represented in this study, such as stroke and multiple sclerosis.

We utilized a somatotopic control strategy as the basis for 2D and 3D cursor control. With this approach, we were able to identify attempted movements which generated robust high-gamma-band modulation, and use this modulation as the basis for closed-loop cursor control. By combining the somatotopic control strategy with population decoding techniques, we were better able to map ECoG modulation generated by attempted movements onto cursor velocity. Perhaps not surprisingly, the number of electrodes tuned to specific attempted movements and the depth of modulation during attempted movement were found to increase from screening to cursor control, suggesting that the somatotopic remapping approach was effective at generating the robust cortical modulation necessary for BCI control. However, though this approach was successful in allowing subjects to become proficient at BCI control, we did observe marked curvature in cursor trajectories, particularly for the 3D task. The presence of curvature in the average cursor trajectories suggests that subjects may have utilized a sub-optimal “step-wise” strategy in which different attempted arm and hand movements were made in a sequential manner. However, at no point were subjects instructed to move the cursor in a straight line to the target, nor were they instructed to acquire targets as quickly as possible, so it is unclear whether this curvature reflects an inability of subjects to make straight-line movements.

The presence of curvature in the cursor trajectories suggests that the somatotopic remapping approach may result in an increased cognitive burden on BCI users. Using this approach, subjects are required to decompose certain intended cursor movements (e.g., diagonal cursor velocities) into a graded combination of attempted arm and hand movements; the observed “step-wise” trajectories may indicate that subjects found this process burdensome, and instead relied on isolated attempted movements whenever possible. Such difficulty in decoupling BCI movements from attempted arm/hand movements could prove particularly problematic in the case of prosthetic limb control. Using the somatotopic remapping approach presented here, attempted movements would be mapped onto intended prosthetic limb endpoint velocity commands. As a result, the attempted movements used to generate prosthetic limb endpoint velocity commands may differ from the actual prosthetic limb movements that those command signals generate, potentially increasing the cognitive burden on the user. However, it has been suggested that the motor imagery used to control somatotopic-based BCIs may become second nature with extended practice [Wolpaw et al., 1991, Curran and Stokes, 2003]; further studies are necessary to investigate the extent to which such internalization is possible. Through demonstrations of translational ECoG-based BCI control have relied on variants of the somatotopic remapping approach, recent work has demonstrated that certain arm movement kinematics can be extracted from ECoG recordings [Chao et al., 2010, Shimoda et al., 2012, Nakanishi et al., 2013, Bundy et al., 2016]. However, it is still unclear if these approaches can be translated to online control.

We found that both somatosensory (subjects S1 and S3) and motor (subject S2 and S3) cortical activity can be used to control an ECoG-based BCI system. The ability of subjects to voluntarily modulate somatosensory cortical activity is particularly notable, as there is growing interest in the ability of somatosensory cortex to support BCI control [Branco et al., 2017]. Consistent with previous findings, we observe that attempted arm and hand movements elicited robust modulation of both pre and post-central gyri [Cramer et al., 2005, Shoham et al., 2001, Hotz-Boendermaker et al., 2008, Miller et al., 2010, Christensen et al., 2007, Lacourse et al., 2005, Porro et al., 1996], and that this activity is somatotopically-organized [Penfield and Rasmussen, 1950, Penfield and Jasper, 1954, Penfield and Boldrey, 1937, Merzenich et al., 1978]. It has been shown that functional magnetic resonance imaging reveals somatotopic organization of both primary motor and somatosensory cortex during hand movements, with somatosensory cortex exhibiting less overlap between cortical volumes responsive to different finger movements [Hluštík et al., 2001]. This suggests that somatosensory cortex may provide sources of activity for a somatotopic-based ECoG BCI which are more independent than those from primary motor cortex. More work is needed in characterizing the discriminability of activity in these regions during attempted movement, specifically in individuals with paralysis. Furthermore, while we find that activity from somatosensory cortex contributed to the ability of Subject S3 to control cursor movement, we observed simultaneous activation of both pre and post-central gyri. Such activation may be the result of efference copies of attempted arm and hand movements, which have recently been observed in ECoG recordings [Sun et al., 2015]. It is unclear whether simultaneous motor and somatosensory ECoG activity can be decoupled through BCI training. Nonetheless, the addition of somatosensory cortex as a source of control signals for BCI could, at a minimum, increase the signal-to-noise ratio of BCI control signals in cases of simultaneous coverage of pre and post-central gyri or enable BCI control in cases where it is not possible to implant electrodes over motor cortex, particularly in cases where motor cortex is damaged (e.g., stroke).

Though it may be tempting to compare BCI control performance across subjects, differences in the signal processing and decoding methods used over the course of the study make such comparisons difficult. These changes were made in an effort to improve brain control performance for subjects S2 and S3 beyond that achieved with Subject S1. For example, the size of the sliding window used to calculate spectral power estimates was reduced (from 300ms to 100ms) in an attempt to make the BCI system more responsive. Additionally, the decoding algorithm used was changed from linear regression to the Optimal Linear Estimator (OLE), which includes a noise model for each neural feature and has been show to be less biased than the commonly-used Population Vector Algorithm [Chase et al., 2009]. In order to determine if these changes resulted in significant improvements in BCI performance, a systematic evaluation of each methodological change would have needed to be performed with each subject. Unfortunately, the time-limited nature of the study prevented us from preforming such an assessment; future studies are needed in order to investigate the optimal signal processing and decoding methods for ECoG-based BCI systems.

While we feel that our results provide strong evidence for the feasibility of a somatotopic BCI system for individuals with paralysis, we would like to highlight several points worthy of further consideration. We have shown that subjects with different causes of movement paralysis are all able to successfully achieve 2 and 3 DoF BCI control using ECoG. However, the low number of subjects makes it difficult to draw definitive conclusions about how the source of paralysis may affect BCI control capability. To this end, it is likely that additional work investigating control quality across larger subject populations is needed. Furthermore, though the duration of electrode implantation in the presented work is longer that that of previous human ECoG BCI studies, the restriction of implant duration to less than 30 days limited the number of research questions we could study. Specifically, we chose to focus on maximizing the number of degrees of freedom that could be obtained over this period. Additional work is needed to address some of the key questions about ECoG-based BCIs, such as whether the chronic stability of ECoG implants [Degenhart et al., 2016] and local field potential recordings [Flint et al., 2013] enables long-term learning and improvements in performance, or whether the maximum number of independent control signals which can be extracted using ECoG can be increased by the use of electrode designs providing greater spatial resolution [Viventi et al., 2011].

In conclusion, we have shown that individuals with upper-limb paralysis can successfully control a BCI using ECoG with three degrees of freedom. Our results, combined with the promise of ECoG to provide robust, long-term recordings [Chao et al., 2010, Blakely et al., 2009] with relatively low hardware and software requirements, suggest that an ECoG BCI system is a viable solution for the restoration of function for individuals with paralysis. We believe that further development into novel decoding algorithms, BCI user training approaches, and fully-implantable devices with telemetry [Rouse et al., 2011], will ultimately allow for longer studies with more subjects, further facilitating the translation of this technology to the clinical realm.