Two-Dimensional Movement Control Using Electrocorticographic Signals in Humans

Abstract

We show here that a brain-computer interface (BCI) using electrocorticographic activity (ECoG) and imagined or overt motor tasks enable humans to control a computer cursor in two dimensions. Over a brief training period of 12-36 min, each of five human subjects acquired substantial control of particular ECoG features recorded from several locations over the same hemisphere, and achieved average success rates of 53-73% in a two-dimensional four-target center-out task in which chance accuracy was 25%. Our results support the expectation that ECoG-based BCIs can combine high performance with technical and clinical practicality, and also indicate promising directions for further research.

1. Introduction

Brain-computer interfaces (BCIs) convert brain signals into outputs that communicate a user’s intent [1]. Because this type of communication does not depend on peripheral nerves and muscles, it can be used by people who are severely paralyzed to communicate and interact with their environment. While recent studies have provided encouraging technical demonstrations of this new communication modality, practical applications of BCI technology are currently impeded by the limitations and requirements of the prevailing non-invasive and invasive methods.

Non-invasive BCIs use electroencephalographic activity (EEG) recorded from the scalp [1]. While these BCIs can support higher performance than often assumed, including two-dimensional control with selection capabilities [2, 3], the acquisition of such high levels of control typically requires extensive user training. Furthermore, EEG has low spatial resolution and is susceptible to artifacts from other sources. Invasive BCIs use local activity from multiple neurons recorded within the brain [4, 5, 6, 7, 8, 9]. Signals recorded within cortex have higher fidelity and might support BCI systems that require less training than EEG-based systems. However, clinical implementations of intracortical BCIs are currently impeded mainly by the difficulties in maintaining stable long-term recordings [10, 11], by the substantial technical requirements of single-neuron recordings, and by the need for continued intensive expert oversight.

Electrocorticographic (ECoG) recording from the cortical surface could be a powerful and practical alternative to current non-invasive and invasive BCI recording methods. ECoG has higher spatial resolution than EEG (i.e., tenths of millimeters vs. centimeters [12]), broader bandwidth (i.e., 0-500 Hz [13] vs. 0-50 Hz), higher characteristic amplitude (i.e., 50-100 μV vs. 10-20 μV), and far less vulnerability to artifacts such as EMG [12] or ambient noise. At the same time, because ECoG does not require penetration of the cortex, it is likely to have greater long-term stability [14, 15, 16, 17] and to produce less tissue damage and reaction than intracortical recordings.

Recent studies have shown that ECoG signals associated with imagery of arbitrary tasks can provide one-dimensional BCI control with little training [18, 19, 20, 21]. The present study extends this work to demonstrate that ECoG signals recorded from the same hemisphere can support multiple degrees of BCI control. Five patients participated (see Table 1 for clinical profiles). Each had subdural electrode arrays implanted for 7-14 days in preparation for surgery to remove an epileptic focus. The experimental protocol combined current BCI methodology [1, 22, 2] with current understanding of task-related ECoG changes [23, 24, 25, 26, 20, 27, 28].

Table 1.

Clinical profiles. All subjects were literate and functionally independent with normal cognitive capacity. The Language column specifies language laterality as determined by the WADA test (e.g., [29]). This test was not performed for subject E

Subj	Age	Sex	Hand	Lang	Grid Location	Seizure Focus
A	19	M	R	Left	Bilateral frontal and temporal strips	Diffuse right hemisphere onset
B	19	M	R	Left	Right frontal-temporal	Right inferior temporal lobe/insular region
C	21	M	L	Left	Right frontal-parietal	Right parietal
D	18	F	R	Left	Left frontal	Left frontal
E	14	M	L	N/A	Left frontal	Medial frontal lobe

The results show that people can rapidly achieve two-dimensional movement control using the ECoG activity associated with imagined or overt motor tasks. Because the acquisition of control in our study was much faster than previously reported for an EEG-based BCI [2], our results suggest that an ECoG-based BCI may allow much more rapid acquisition of two-dimensional control than do EEG-based BCIs. These findings are further evidence that ECoG could be a robust and practical alternative for clinical application of BCI technology. They also indicate promising directions for further research.

2. Methods

2.1. Subjects

The five subjects in this study were patients with intractable epilepsy who underwent temporary placement of intracranial electrodes for localization of seizure foci prior to their surgical removal (see Fig. 1 and Table 1). All patients gave informed consent. The study was conducted at Washington University Medical Center in St. Louis and Harborview Hospital at the University of Washington in Seattle, and was approved independently by the Human Studies Committees of both institutions. Prior to this study, these patients had not been exposed to a BCI system. Each patient had 26-64 subdural electrodes (configured in grids or strips) placed over the fronto-parietal-temporal region, including coverage of sensorimotor cortex, with variable coverage of other areas. The electrodes had a diameter of 4 mm (2.3 mm exposed) and an inter-electrode distance of 1 cm center-to-center (Fig. 1). Patient A had bitemporal and bifrontal strips; patients B and C had electrode grids placed over the right hemisphere; and patients D and E had grids placed over the left hemisphere (Fig. 2). Grid placements were based solely on the clinical requirements without any consideration of this study. Following array placement, each patient had post-operative anterior-posterior and lateral radiographs to verify their electrode locations.

Figure 1.

ECoG array in situ. A: Exposed brain after craniotomy. B: 8×8 electrode grid on the surface of the brain. C: Lateral x-ray image. The electrode grid and several strips are visible. D: Average brain template and electrode locations co-registered to the x-ray image.

Figure 2.

Electrode locations in the five subjects projected onto a standard brain. To facilitate comparison, the electrodes were projected from the left onto the right hemisphere for subjects A, D and E (see asterixes).

2.2. Data Collection

Each patient sat in a hospital bed about 75 cm from a video screen. In all experiments, we recorded ECoG from 16-64 electrodes using the general-purpose BCI software system BCI2000 [22]. All signal recordings were: referenced to an inactive intracranial electrode or to the scalp; amplified; bandpass-filtered between 0.1 and 220 Hz and digitized at 500 or 1200 Hz (patients C and E, respectively) or bandpass-filtered between 0.15 and 200 Hz and digitized at 1000 Hz (patients A, B, and D); and stored. The amount of data obtained and the total time in the study varied from patient to patient. It was determined primarily by the duration of the implantation and the subject’s physical state and clinical situation.

2.3. Experimental Protocol

The experimental protocol for achieving two-dimensional cursor control had three stages: feature identification, one-dimensional BCI control, and two-dimensional BCI control. The three stages are described here.

In the first stage, we collected data that allowed us to determine which ECoG features could be used for BCI control. Each subject engaged in several motor or motor imagery tasks [28] such as opening or closing the hand contralateral to the implant; protruding the tongue; moving the jaw; saying the word ”move;” shrugging the shoulders; moving the legs; moving individual fingers, and imagining each of these actions. We selected a set of tasks for each subject depending on the location of the electrode array. The number of these tasks that the subject actually performed depended primarily on the subject’s ability to maintain adequate concentration. The subject was asked to engage in each task during presentation of a visual cue (2-3 secs), and to rest while the screen was blank (1-3 sec). The different tasks were interspersed randomly and each task was repeated about 60 times.

We compared ECoG activity for the different tasks to that during rest to determine which features (i.e., amplitudes at particular locations and frequencies) differed most and thus were the best candidates for BCI control. To make this comparison, we converted the time-series ECoG data into the frequency domain using an autoregressive model (i.e., with the maximum entropy method [30]) of order 25. We derived spectral amplitudes for each 2-Hz bin from 0-200 Hz. The results were consistent with current understanding of the effects of different tasks on ECoG [23, 24, 25, 31, 32, 26, 27, 28]. Specifically, activity in mu and beta bands (i.e., 8-25 Hz) usually decreased during task execution, whereas activity in the gamma range (i.e., >35 Hz), and especially above 50 Hz, often increased with task execution. Mu/beta rhythm changes were typically spatially more widespread, and thus were detected in more electrodes, compared to gamma activity changes (e.g., Fig. 3 and [28]). These changes were usually found over anatomically relevant areas (e.g., changes in response to tongue movement occurring over inferior rolandic cortex). Changes during imagined actions were typically topographically and spectrally similar but smaller than those during actual actions. This is consistent with our previous ECoG studies [18, 33] and also with corresponding EEG studies [34].

Figure 3.

Example of an analysis comparing ECoG signals for right hand movement and rest (modified from [36]). A: Color-coded values of r² for all locations and frequencies. B: Average spectra (red for rest, green for right hand movement) (left) and r² as a function of frequency (right) for electrode 15. C: Topographical distribution (black dots indicate electrode locations on the 8×4 grid) for color coded r² values calculated for 20 Hz.

The ECoG features (i.e., amplitudes at particular locations and frequencies) with the largest task-related amplitude changes were identified as features for online control of a computer cursor. These amplitude changes were computed by calculating the coeffcient of determination (r², [35]) for each feature and between the two distributions of trial-averaged feature values for task and rest, respectively (i.e., similarly to [2]). This gave a measure for the fraction of the feature variance that was accounted for by the task, and thus gave an indication of how much control the subject had over each particular feature. We identified pairs of tasks that were independent of each other in spatial and spectral distributions and their most salient ECoG features, and then assigned each of them to control either horizontal or vertical cursor movement (as described below). While this feature selection and assignment to movement direction was somewhat arbitrary (see the Discussion for implications of this feature/task selection methodology), we strove to pick relationships that seemed most intuitive (e.g., using features that corresponded to movement/imagery of the tongue for vertical control, or features that corresponded to movement/imagery of the contralateral hand for horizontal control). These relationships were explained to the subject prior to the experiment.

In the second stage, we trained each subject first on horizontal and then vertical cursor control. We assigned one or more of the ECoG features identified above to each dimension of movement. In each trial, the subject was presented with one of two targets (i.e., on the left or right edge of the screen for horizontal control, or top or bottom edge of the screen for vertical control), and movement was restricted to the respective dimension. The cursor started in the center of the screen. The subject’s task was to modulate the assigned ECoG features so that the cursor moved to the target. Cursor movement occurred every 26 ms (St. Louis) or 40 ms (Seattle) and was defined by one of two control signals (i.e., one for horizontal movement and one for vertical movement). Each of these two control signals was calculated from 1-4 ECoG features (i.e., amplitudes in particular frequency bands at particular locations) using a weighted, linear summation. The weights were chosen manually and were usually either +1 or -1 (so as to assign the increase or decrease of feature change to the desired direction (up or down, left or right) of cursor movement). In contrast to Wolpaw and McFarland’s recent EEG study [2], these weights were not automatically adapted over the course of the experimental period. ECoG features were computed from the previous 280 ms (subjects A-D) or 64 ms (subject E). In subsequent training sessions, we then trained each subject on each dimension of cursor control using one or more tasks (e.g., training on vertical control using imagined thumb movement followed by training on vertical control using imagined shoulder movement). Subjects typically acquired accurate one-dimensional control rapidly as previously reported [18, 19, 20, 21].

In the third stage of the protocol, we conducted two-dimensional BCI experiments by combining sets of ECoG features that the subject had previously learned to control independently. In these experiments, two-dimensional cursor movement was controlled by the selected sets of horizontal and vertical ECoG features simultaneously. Depending on the subject’s condition and success with this two-dimensional movement task, we assigned different sets of the trained tasks/feature sets to the two dimensions of movement. For example, if a subject learned to control the cursor horizontally using imagined thumb or hand movement, and vertically using imagined shoulder movement, we could try two-dimensional control using imagined thumb/imagined shoulder or imagined hand/imagined shoulder movements. These initial one- and two-dimensional interactions averaged about one hour of task-related activity per subject and preceded collection of the data presented in this study. The subject’s task was to move a computer cursor from the center of the screen to a target that appeared in one of four locations on the periphery of the screen (see Fig. 4 for an example). For subjects A,B,C, and D, only the current target was displayed; for subject E, all targets were displayed simultaneously (the desired target was shown in a red color and the others in green). Each trial began with the appearance of the target. One second later, the cursor appeared in the middle of the screen and then moved with its vertical and horizontal movements controlled continuously by the patient’s ECoG features. The patient’s goal was to move the cursor so that it hit the target. The software restricted cursor movement to the workspace, i.e., the cursor was not allowed to go off the screen. The trial immediately ended when the cursor hit the area corresponding to one of the four targets (irrespective of whether the target was actually visible). If the cursor reached the correct target within a predefined maximum movement time (8-25.6 secs, depending on the subject), the target flashed as a reward. If it failed to reach a target within that time, the cursor and target simply disappeared, and the trial was registered as a miss. The number of trials that timed out was very low for all subjects: 5.7%, 4.8%, 0.0%, 0.9%, and 8.6% for the five subjects A-E, respectively. Because there were four targets and trials were allowed to time out, the expected accuracy due to chance alone was at most 25%. Whether the cursor hit one of the targets or the trial timed out, the screen was then blank for 0.26-1 sec before the next trial began. The data shown here were collected from each patient for four to twelve 3-min runs, each comprised of 26-54 trials. The runs were separated by 1-min breaks.

Figure 4.

Trial sequence for two-dimensional cursor movement. Initially, the screen was blank (rest period). Then, a target appeared in one of four possible locations on the periphery of the screen (movement preparation). Then, a cursor appeared in the center of the screen and immediately started moving as determined by the subject’s ECoG features (movement period). When the cursor reached the target or the space of one of the three other targets, the screen went blank and then, after a brief rest period, the next trial began.

2.4. 3D Cortical Mapping

We used lateral skull radiographs and the LOC package [37] to identify the stereotactic coordinates of each grid electrode. This automated procedure replicated the manual procedure described in [38]. Cortical areas were defined using Talairach’s Co-Planar Stereotaxic Atlas of the Human Brain [39] and a Talairach transformation (http://ric.uthscsa.edu/projects/talairachdaemon.html). A template 3D cortical brain model (subject-specific brain models were not available) was obtained from the AFNI SUMA website (http://afni.nimh.nih.gov/afni/suma). Each subject’s electrode locations were projected on this 3D brain model (e.g., Fig. 1-D) and activation maps (e.g., Fig. 7) were generated using a custom Matlab program.

Figure 7.

Example topographies of control for subjects D and E, calculated for all locations and for the control signals given by the frequency-band combination used online. These topographies show the color-coded correlation (as r² values) of cortical activity with vertical or horizontal movement, and thus indicate the level of task-related control of different cortical areas. Subject D used actual tongue movements for vertical control and actual hand movements for horizontal control. Subject E used imagined versions of the same actions. The traces below each topography show r² values for the locations used online (indicated by stars). Yellow bars indicate the frequency bands used online. The topographies show activity patterns over locations expected with these motor/imagery tasks (see also [28]), and are similar for actual and imagined tasks. These figures also illustrate that choice of different locations/frequencies could have yielded improved online performance (see text).

2.5. Additional Offline Analyses

We conducted additional offline analyses in which we determined whether features chosen offline would provide improved performance compared to the sets of features that were used online. In these analyses, we utilized the stepwise regression function implemented in Matlab (executed separately for vertical and horizontal movements) to select those features that were most predictive of target position. The features for these analyses were amplitudes at all locations and in four frequency bands: 8-12 Hz, 18-26 Hz, 70-100 Hz, and 110-150 Hz (i.e., a total of 64-288 features for the five subjects). Each of the 1627-5312 samples in the data sets was associated with a target value to be predicted. This target value was -1 for bottom targets, +1 for top targets, -1 for left targets, and +1 for right targets. This procedure determined, independently for horizontal and vertical movements, a set of features and associated weights. Similarly to how the manually chosen features and weights were combined to result in the control signals for online cursor control, we combined the weighted sum of these automatically determined features to produce optimized control signals offline.

3. Results

Table 2 shows how each subject used ECoG to control cursor movement. The five subjects used different actual movements (subjects B, C, and D) or imagined movements (subjects A and E) to modulate spectral amplitudes at different frequency bands and cortical locations and to thereby control the cursor. The majority of features were amplitudes of high gamma frequency (>70 Hz) recorded from electrodes over sensorimotor cortex (Brodmann’s areas 1-6). The average accuracies achieved by the five subjects ranged from 53%-73%. As Fig. 5 shows, the subjects had significant control from the beginning. Three of the five subjects (B, C, and E) improved rapidly over the short periods of training. The most rapid improvement (20% on average) was between the first and second 3-min run. To illustrate the actual trajectories of the cursor movements, Fig. 6 shows the targets and average cursor movement trajectories for each subject.

Table 2.

Two-dimensional cursor control. This table shows for each subject: the actual or imagined motor action that the subject used for control of horizontal (h) or vertical (v) movement; the cortical location (i.e., Brodmann’s area) and frequency band of the feature used for horizontal (h) or vertical (v) movement, and average target accuracy. These results demonstrate that people can use ECoG activity to control two-dimensional cursor movement

Actual Movements
Subject	Task (Direction)	Brodmann’s Area	Frequency Band	Avg. Acc.
B	Tongue Movement (v)	1	79-85 Hz	69%
	Eye Movement (h)	20	17-19 Hz
		20	17-19 Hz
		20	17-19 Hz
C	Middle Finger Movement (v)	40	70-80 Hz	73%
			90-100 Hz
			110-120 Hz
			130-140 Hz
	Shoulder Movement (h)	4	70-90 Hz
		6	70-90 Hz
		5	100-120 Hz
D	Tongue Movement (v)	6	77-83 Hz	58%
			95-101 Hz
	Hand Movement (h)	4	77-83 Hz

Imagined Movements
Subject	Task (Direction)	Brodmann’s Area	Frequency Band	Avg. Acc.
A	Imagined Jaw (v)	31	115-121 Hz	53%
	Imagined Hand (h)	47	47-53 Hz
		38	47-53 Hz
E	Imagined Tongue (v)	43	90-95 Hz	60%
			110-115 Hz
	Imagined Hand (h)	3	85-90 Hz
		2	90-95 Hz
		6	115-120 Hz
			125-130 Hz

Figure 5.

Learning curves for ECoG control of two-dimensional cursor movement using motor actions or imagery (see text).

Figure 6.

Average cursor movement trajectories for the five subjects.

Table 3 provides more detailed analyses of the control shown in Table 2 and Fig. 5. It lists the correlations (given in r²) of the horizontal and vertical control signal with horizontal and vertical target position, respectively, as well as the correlations of the control signals with the orthogonal position (e.g., horizontal control signal vs. vertical target position). These values of r² were determined similarly to the approach in [2], i.e., they were calculated based on ECoG trial averages. Thus, they indicate the fraction of the variance of the average control signal that is related to the target position. In addition, this table also shows the correlation, given as correlation coeffcient r, between individual values of the horizontal and vertical control signals. Each of these values is provided for the control signals that were derived online using the ECoG features that controlled the cursor, and for the control signals that were derived offline using an optimal set of ECoG features (see Section 2.5). Table 3 also gives the size of the target (in percent of the workspace) and the median, mean and standard deviation of the movement times. The values for movement time were calculated for all trials, i.e., including those that timed out. (As mentioned above, only very few trials actually timed out.) The values of r² represent the proportion of the control signal variance that is due to the task (e.g., movement to top vs. bottom target). Thus, they are a measure of the subject’s control over that particular ECoG signal. These correlations are high for the appropriate direction (e.g., between the horizontal control signal and horizontal target position) and low for the orthogonal direction. For example, for subject C the correlations for the horizontal and vertical control signals with horizontal and vertical target position, respectively, were 0.65 and 0.37, whereas the correlations for the horizontal and vertical control signals with the orthogonal position were only 0.01 and 0.04, respectively. Thus, these results indicate that the control signals were strongly related to the desired movement direction, and only minimally related to the opposite direction. At the same time, the correlation calculated directly between the two control signals used online was not negligible in all subjects (e.g., subject D’s correlation coeffcient was -0.40). This indicates that the x and y control signals contained correlated noise in some subjects. However, the use of the optimized set of ECoG features derived offline (see Section 2.5) typically substantially increased the correlations of the control signals with target position in the respective dimension and substantially decreased the correlations between control signals. For example, Subject B’s r² correlations for the horizontal and vertical control signal with horizontal and vertical target position were 0.21 and 0.25, respectively, whereas the correlations determined offline using an optimized set of ECoG features were 0.88 and 0.92, respectively. Subject D’s correlation r between the control signals online was -0.40, whereas the correlation derived offline using optimized features was -0.08. This indicates that all subjects had substantial and practically completely independent control over different sets of ECoG features, but that the translation of these features into control signals online was not optimal. Thus, these results suggest that control may be further optimized by improving the initial selection and possibly also by continually updating this selection (see Discussion).

Table 3.

Characteristics of each subject’s two-dimensional control. This table shows the correlations of: the horizontal ECoG control signal with horizontal target position, r²(xx); the vertical control signal with vertical target position, r²(yy); and the horizontal and vertical control signals with the orthogonal positions (r²(xy) and r²(yx), respectively). Correlation is given in r², i.e., the proportion of the control signal variance that is due to the target location (i.e., top or bottom, left or right). The following column shows the correlation coe cient r, calculated between the individual observations of horizontal and vertical control signals. For each entry, the rst value gives the correlations achieved online (on), and the second value gives the correlation determined offline (off) using a set of features determined by stepwise regression. The last several columns give: the size of a target (in percent of the workspace); the median, mean, and standard deviation of the movement times in seconds, respectively; and the speed of the movement measured in percent of screen width per second

Actual Movements
	r2(xx)	r2(yy)	r2(xy)	r2(yx)	r	Target	Movement Times (s)	Speed (%)
Subj.	on off	on off	on off	on off	on off	Size (%)	median mean std.dev.
B	0.21 0.88	0.25 0.92	0.00 0.02	0.09 0.02	-0.29 0.03	9	1.64 2.17 1.89	41.22
C	0.65 0.71	0.37 0.70	0.01 0.01	0.04 0.01	0.11 0.07	8	2.56 2.99 1.78	25.99
D	0.31 0.86	0.45 0.77	0.01 0.01	0.04 0.01	-0.40 -0.08	9	1.52 1.94 1.47	38.84

Imagined Movements
	r2(xx)	r2(yy)	r2(xy)	r2(yx)	r	Target	Movement Times (s)	Speed (%)
Subj.	on off	on off	on off	on off	on off	Size (%)	median mean std.dev.
A	0.37 0.50	0.39 0.53	0.18 0.03	0.08 0.01	-0.21 0.15	5	1.58 2.13 1.86	46.47
E	0.09 0.14	0.27 0.18	0.01 0.01	0.04 0.01	-0.06 0.23	4	2.16 2.79 2.01	73.01

The topographic interpolations of r² values in Fig. 7 show the spatial distribution of control signals for subjects D and E. Subject D employed tongue and hand movements for vertical and horizontal control, respectively, whereas subject E employed imagined movements of the same kind. (Subject E used different frequencies derived from three locations for horizontal cursor movement. The right-most topography shows the topography for the set of frequencies used at one of these three locations.) The locations used online are marked with stars. The traces below each topography show the r² values as a function of frequency for these locations (yellow bars indicate the frequency bands used online). These data again indicate that the locations/frequencies that we had chosen for online control were not optimal. For example, the locations used for horizontal movement in both subjects were not the best possible. In these examples, the actual and imagined motor actions produced activations over several motor cortical areas comparable to those in a recent ECoG motor mapping study [28]. The imagined tasks used by subject E produced patterns that are comparable to those produced by the actual tasks used by subject D.

4. Discussion

This study demonstrates that ECoG activity can support two-dimensional cursor control. In contrast to a recent study using EEG [2], two-dimensional control was achieved here using different locations on the same hemisphere. This may prove beneficial for BCI applications for patients with unilateral hemiparesis. After brief one-dimensional training similar to what we and our collaborators have previously reported [18, 19, 20, 21], all five subjects achieved this two-dimensional control within minutes. The level of control over the two movement signals and the speed of the movement reported here (see Table 3) were comparable to those that have previously been achieved after extended training using EEG in humans or in highly controlled experiments using intracortical microelectrodes in non-human primates (see Tables 1 and 2 in [2]). The time course of control acquisition is much faster than the weeks or months reported in a recent EEG study [2], and may be comparable to the rapid control acquisition suggested by anecdotal evidence for intracortical studies in monkeys. In summary, by showing that ECoG can support rapid acquisition of robust two-dimensional control without penetrating the cortex, this paper further demonstrates that ECoG is an excellent signal recording modality for BCI applications that may combine high performance with technical and clinical practicality.

While all subjects studied in this paper successfully achieved two-dimensional control, the present experimental approach has three important limitations. The first limitation is that, at present, the only subjects available for these ECoG studies are patients with epilepsy who are temporarily implanted with electrode grids prior to surgery. The second limitation relates to the current difficulty of identifying appropriate ECoG signal features. The third limitation stems from the arbitrary nature of the motor/imagery tasks used for BCI control. These three limitations are described in more detail below.

Patient volunteers who are transiently implanted with subdural electrodes for clinical evaluation offer a rare opportunity to conduct scientific studies. However, this opportunity is constrained by the clinical needs of the patients undergoing treatment. These constraints include variation in patients’ cognitive status, restricted time for experimentation, and, for our research purpose, typically suboptimal cortical coverage by the electrode array. The cognitive status of the patients is often somewhat impaired due to the medical condition that results in epilepsy [40], seizures during the course of their ECoG monitoring, and concurrent administration of narcotic medication to control their pain. In addition, the subjects’ willingness to participate is often influenced by the highly variable nature of the clinical course of diagnosis and treatment, and consequently may wax and wane unpredictably. The time available for experimentation is curtailed by the limited duration of electrode placement: two of the five subjects (B, D) had their electrodes implanted for a total of 7 days each. Post-operative recovery (2-3 days) and other factors reduced the time available for experimentation to about 3-4 days. In addition, clinical testing (e.g., electrical cortical mapping) or seizures (and subsequent post-ictal periods) often further limit available time. Because the study protocols are approved for only relatively short experimental periods per day, there were limited opportunities for experimentation in these patients. Patients A, C and E had a prolonged monitoring course (five, two, and two weeks, respectively) due to clinical requirements, which made additional testing possible. Variable cognitive status and reduced time were the most limiting factors in conducting our multi-step experimental protocol (i.e., signal identification, sequential one-dimensional training, concurrent two-dimensional control). In addition to the patient and time limitations, configuration and placement of the grids are optimized for clinical and not for BCI purposes. The configuration (i.e., 1-cm inter-electrode distance) of the clinical grids is almost certainly significantly coarser than that previous studies have suggested to be optimal (i.e., the optimum spatial sampling resolution is probably 1.25 mm [41, 12]). In fact, we often observed task-related correlations limited to only one or a few recording sites. The placement of the electrodes is dictated by clinical requirements and thus, rarely covers all relevant motor areas and is highly variable between subject. These patient-related issues greatly increase inter- and intra-subject experimental variability and significantly impair the use of a consistent and systematic experimental procedure, and will thus ultimately limit the amount of information that can be extracted using this subject population. The experience with these issues in the present study and in related studies suggests several important areas for future investigation. These include controlled animal studies that determine the long-term effects of subdural/epidural implants, that assess the difference between subdural and epidural recordings with regard to signal-to-noise ratio, and that define the optimum spacing and placement of the electrodes. These efforts should be accompanied by the development and testing of wireless telemetry systems. We expect that appropriate implementation and integration of this optimized ECoG recording platform will result in a small and potentially epidural implant with wireless transmission, which will substantially reduce the clinical risks currently associated with a large and transcutaneous implant. We anticipate that this and future studies will provide ample evidence of the utility of the ECoG platform and will thus support FDA approval of long-term human ECoG-based BCI use for the purpose of communication and restoration of mobility and functional interactions.

The second major limitation is the identification of optimal ECoG signal features. As described in Section 2.3, our first step in utilizing ECoG for real-time BCI control was the identification of the features (i.e., signal amplitudes at particular locations and frequencies) that would be most effectively modulated by the subject using a particular task (i.e., a signal identification procedure). Because there is no strong a-priori basis for making this selection, and because ECoG contains more signal features that are responsive to more tasks than does EEG, this choice is also more difficult. Moreover, ancillary analyses supported the notion reported in [20] that signal features typically change between the signal identification procedure and the real-time experiment. Traditional signal translation schemes (such as classification or regression) assume that the signal properties are the same during initial signal identification and during BCI feedback. Since this assumption is often not met, it tends to reduce performance. A good example is the generally large difference in performance between the optimized features determined offline and the set of features used online (Table 3). Full exploitation of the promise of ECoG signals will require better ways to initially identify good signal features, and subsequently track them as they evolve. One such possibility is to simply detect changes from a signal baseline (e.g., at one location per control dimension), rather than trying to identify and use specific features or sets of features. In this approach, a set of features recorded during rest is modeled, and deviations from this model (such as those associated with production of a task) are measured. These measurements can be used for device control. Initial testing [42, 43, 44] demonstrates that this methodology can be used effectively in situations in which there is little a priori knowledge about signal features (such as in the experiments discussed here).

The third limitation is related to the non-intuitive and arbitrary tasks utilized for BCI control. With an ideal brain-computer interface, a person with a motor disability would simply intend a particular movement. This intended action would be detected by the BCI system and translated into appropriate device action. In contrast, the subjects in typical human BCI experiments (such as those described in this paper) use arbitrary and non-intuitive imagery (such as imagined tongue or hand movements) to drive a computer cursor. While a recent EEG-based study [2] and this paper demonstrate that subjects can make good use of such non-intuitive imagery^‡, it appears likely that multi-dimensional control could be more e ciently achieved and potentially further improved by using more intuitive tasks such as directed actual or imagined hand movements. It has been widely assumed that only microelectrodes implanted within cortex can provide the signal fidelity necessary for effective realization of this approach. However, evidence in recent and ongoing studies [45, 46, 47, 48, 49, 50] strongly suggests that this is not the case; that, in fact, ECoG supports decoding of multidimensional movement parameters with a fidelity that is comparable to that achieved previously only with microelectrodes implanted within cortex.

In summary, in the present paper, we describe two-dimensional BCI control using ECoG in humans after minimal training, and we outline several promising directions for further research. These results should help to move BCI research closer to practical realization of powerful and reliable BCI systems for long-term use by people with severe neuromuscular disorders.