Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2009 Oct 5.
Published in final edited form as: J Neural Eng. 2008 Mar 5;5(2):101–110. doi: 10.1088/1741-2560/5/2/001

Emulation of computer mouse control with a noninvasive brain-computer interface

Dennis J McFarland 1, Dean J Krusienski 2, William A Sarnacki 1, Jonathan R Wolpaw 1
PMCID: PMC2757111  NIHMSID: NIHMS122397  PMID: 18367779

Abstract

Brain-computer interface (BCI) technology can provide nonmuscular communication and control to people who are severely paralyzed. BCIs can use noninvasive or invasive techniques for recording the brain signals that convey the user’s commands. Although noninvasive BCIs are used for simple applications, it has frequently been assumed that only invasive BCIs, which use electrodes implanted in the brain, will be able to provide multidimensional sequential control of a robotic arm or a neuroprosthesis. The present study shows that a noninvasive BCI using scalp-recorded EEG activity and an adaptive algorithm can provide people, including people with spinal cord injuries, with two-dimensional cursor movement and target selection. Multiple targets were presented around the periphery of a computer screen, with one designated as the correct target. The user’s task was to use EEG to move a cursor from the center the screen to the correct target and then to use an additional EEG feature to select the target. If the cursor reached an incorrect target, the user was instructed not to select it. Thus, this task emulated the key features of mouse operation. The results indicate that people with severe motor disabilities could use brain signals for sequential multidimensional movement and selection.

1. Introduction

Brain activity produces electrical signals that are detectable on the scalp, on the cortical surface, or within the brain. Brain-computer interfaces (BCIs) translate these signals from mere reflections of brain activity into outputs that communicate the user’s intent without the participation of peripheral nerves and muscles (Wolpaw et. al., 2002). Because they do not depend on neuromuscular control, BCIs can provide communication and control for people with devastating neuromuscular disorders such as amyotrophic lateral sclerosis (ALS), brainstem stroke, cerebral palsy, and spinal cord injury. The central purpose of BCI research and development is to enable these users, who may be totally paralyzed ("locked in," unable even to breath or to move their eyes), to convey their wishes to caregivers, to use word-processing programs and other software, or even to control a robotic arm or a neuroprosthesis.

BCIs can be either noninvasive or invasive. Present-day noninvasive BCIs derive the user’s intent from scalp-recorded electroencephalographic (EEG) activity. They are clearly capable of providing basic communication and control to people with severe disabilities (e.g., Birbaumer et. al., 1999; Muller-Putz et. al., 2005; Sellers et al., 2006). Present-day invasive BCIs derive the user’s intent from neuronal action potentials or local field potentials recorded from within the cerebral cortex or from its surface. They have been studied mainly in non-human primates and to a limited extent in humans (Chapin et. al., 1999; Wessberg et al, 2000; Serruya et al, 2002; Taylor et. al., 2002; Carmena et al, 2003; Pesaran et al, 2002; Andersen et al, 2004; Leuthart et al, 2004; Hochberg et al, 2006). These invasive BCIs face substantial technical difficulties and involve clinical risks. Recording electrodes must be implanted in or on the cortex and function well for long periods, and they risk infection and other damage to the brain. The drive to develop invasive BCI methods is based in part on the widespread conviction (Fetz, 1999; Chapin, 2000; Nicolelis, 2001; Konig, and Verschure, 2002; Donoghue, 2002) that only invasive BCIs will be able to provide users with real-time multidimensional sequential control of a robotic arm or a neuroprosthesis.

Nevertheless, in an early study (Wolpaw and McFarland 1994) we showed that a noninvasive BCI that uses scalp-recorded EEG activity (i.e., sensorimotor rhythms) can provide humans with multidimensional movement control. Furthermore, in a recent study (Wolpaw and McFarland, 2004), we showed that a noninvasive EEG-based BCI that incorporates an adaptive algorithm and other technical improvements can give humans multidimensional movement control comparable in movement time, precision, and accuracy to the control achieved by invasive BCIs in monkeys (Serruya et al, 2002; Taylor et al, 2002; Carmena et al, 2003) or humans (Hochberg et al., 2006).

In prior multidimensional studies, the BCI user (monkey or human) was presented with a single target in each trial, and the task was to move the cursor to the target. Thus, an error occurred only when the trial timed out before the target was reached. This laboratory task is less demanding than most real-world tasks, in which incorrect selections can occur and have consequences (e.g., the user selects the wrong letter or icon and must erase it). The present study emulates the more realistic real-life situation: the user is presented with multiple targets only one of which is correct, moves the cursor to a target, and then either selects it (if it is the correct target) or does not select it (if it is not the correct target). In this task, the user’s EEG provides three distinct control signals: two to simultaneously control vertical and horizontal movements, respectively; and the third to select or reject a target once it is reached. This task closely approximates real-world tasks such as using a mouse to move a cursor among the icons on a screen until it reaches the desired icon, and then pressing the mouse button to select that icon. The results show that a noninvasive EEG-based BCI can provide people with sequential as well as multidimensional control.

2. Methods

Our sensorimotor-rhythm-based BCI methodology has been fully described previously (Wolpaw and McFarland 2004; McFarland et al, 2006b) and is summarized here. The new procedures relating to sequential operation and target selection are described in detail.

Users

The BCI users were six adults, three women and three men, ages 24–56. Two of the men had spinal cord injuries (one at T7 and one at C6) and were confined to wheelchairs. All gave informed consent for the study, which had been reviewed and approved by the New York State Department of Health Institutional Review Board. Four of these users had no prior BCI experience. The two users with spinal cord injury had previously participated in other BCI studies (e.g., Wolpaw and McFarland, 2004; McFarland et al, 2005).

BCI Training Protocol and Data Collection

The user sat in a reclining chair facing a 51-cm video screen 3 m away, and was asked to remain motionless during performance. Online operation and data collection were supported by the general-purpose BCI software platform, BCI2000 (Schalk et al, 2004). Scalp electrodes recorded 64 channels of EEG (Sharbrough et al., 1991), each referenced to an electrode on the right ear (amplification 20,000; bandpass 0.1–60 Hz). All 64 channels were digitized at 160 Hz and stored for later analysis. A subset of channels located over sensorimotor cortex (see Table 1) were used to control online cursor movement and target selection online as described below. Each user completed 2–3 sessions per week. Each session consisted of eight 3-min runs separated by 1-min breaks, and each run consisted of 20–30 trials.

Table 1.

User age, disability, gender, and signal features that controlled horizontal (X) and vertical (Y) cursor movements and target selection (Z). For each feature, the scalp location (C3, C4, CP3, CP4, FC1) and the center frequency of the 3-Hz wide frequency band (in parenthesis) are given.

Name Age Disability Gender X Features Y Features Z Features
A 24 Spinal cord injury M C3 (12)
C4 (12)		 C3 (12)
C4 (12)		 FC1 (15)
C3 (12)
B 56 None F C3 (26)
C4 (26)
CP3 (26)		 C3 (26)
CP3 (26)		 C3 (26)
CP3 (22)
C 44 Spinal cord injury M C3 (12)
CP3 (12)		 C3 (24)
C4 (24)		 FC1 (24)
C2 (24)
D 38 None F C3 (26)
CP4 (26)		 C3 (26)
Cz (30)		 C3 (18)
FC1 (22)
E 43 None F C3 (12)
CP4 (12)		 C3 (12)
CP4 (12)		 C3 (12)
CP3 (27)
F 28 None M C3 (12)
C4 (12)		 C3 (12)
C4 (12)		 C3 (12)
CP3 (12)
Open in a new tab

The user first learned a one-dimensional vertical cursor movement task in which two targets appeared, one at the top of the screen and one at the bottom. In each trial, one of the targets was red (i.e., correct) and one was green (i.e., incorrect). The location of the red target for each trial was randomly determined. The cursor began in the middle of the screen and moved vertically until it reached a target, at which point the trial ended. The user’s goal was to move the cursor to the red target. If it moved to the green target, an error was registered. Vertical cursor movement was controlled by a combination of sensorimotor-rhythm features as described below. After learning this vertical movement task, the user learned a comparable one-dimensional horizontal cursor movement task in which the two targets appeared at the right and left edges of the screen, and horizontal cursor movement was controlled by a different combination of sensorimotor-rhythm features. Early in training, users typically employed motor imagery to control the cursor. As their skill developed, imagery tended to become less important.

After mastering both one-dimensional tasks, the user employed the two sets of sensorimotor-rhythm features to control both horizontal and vertical movement simultaneously. Four targets were presented, one in the middle of each edge of the screen, and the goal was to reach the single red target while avoiding the three green targets. The location of the red target for each trial was randomly determined. Targets on the top and bottom edge were 20 % of the screen in width and 10 % of the screen in height. Targets on the right and left screen edge were 10 % of the screen in width and 20 % of the screen in height. Thus each target occupied 2% of the workspace. Once a target was reached, the user employed a third set of sensorimotor-rhythm features to select the target (if it was red) or to reject it (if it was green). This completed the trial. (Users were instructed to select the target by imagining grasping it with the his/her right hand only if it was red.)

Figure 1 illustrates the sequence of events in each trial. First, the four targets appeared. One sec. later, the cursor appeared in the middle of the screen and began to move both horizontally and vertically under the control of the user’s EEG. When the cursor made contact with a target, the cursor and the other targets disappeared and the contacted target turned blue for 1.5 sec. During this period, the user employed the third set of sensorimotor-rhythm features to select or not select the target. If the target was selected, it turned yellow for 1 sec. If it was not selected, it simply disappeared and the screen was blank for 1 sec. The screen was then blank for a 1-sec. inter-trial period, and the next trial began. Thus, each trial consisted of a 1-sec period between target appearance and cursor movement, a variable period during which cursor movement occurred, a 1.5-sec post-movement selection period, a 1-sec post-trial feedback period, and a 1-sec inter-trial interval.

Figure 1.

Figure 1

Open in a new tab

Sequence of events during a trial. A: Four targets are presented simultaneously on the screen. The red target is correct and the green targets are not. B: One sec later, the cursor appears on the screen and starts to move under EEG control. C: The cursor approaches a target. D: When the cursor hits a target, the target turns blue and the other targets disappear. This indicates the start of the 1.5-sec selection phase. E: If the target is selected, it turns yellow for one sec. If the target is not selected, it disappears and the screen is blank for one sec. F: A one-sec intertrial interval precedes the next trial.

Control of Cursor Movement

To control each dimension (horizontal or vertical) of cursor movement, two EEG channels (one over sensorimotor cortex of each hemisphere) were derived from the digitized data according to a large (i.e., 6-cm interelectrode distance) Laplacian transform (McFarland et al., 1997b). The specific channels used for each subject are shown in Table 1. Every 50 msec, the most recent 400-msec segment from each channel was analyzed by a 16th order autoregressive algorithm (Marple, 1987) that was used to compute an estimation of the spectrum. Next the amplitude (i.e., square root of power) was calculated from the spectrum for 3-Hz-wide sensorimotor-rhythm frequency bands centered between 8 and 26 Hz (most commonly in the mu (8–12 Hz) or beta (18–26 Hz) frequency range). The amplitudes in these specific frequency bands from specific EEG channels constituted the signal features that conveyed the user’s intent. One or more of these features were combined to comprise the control signal (i.e., the independent variable) in a linear equation that controlled a dimension of cursor movement (McFarland et. al., 1997a). That is, if ΔV was the vertical cursor movement, Sν was the control signal for vertical movement, bν was the gain, and aν was the mean of the vertical control signal for the user’s previous performance (see below),

ΔV=bν(Sν−aν) (1)

was the function that determined each vertical cursor movement. (This form of the linear equation is used so that a and b can be defined independently of each other.) Similarly, if ΔH was the horizontal cursor movement,

ΔH=bh(Sh−ah) (2)

was the function that determined horizontal cursor movement. Movements in each dimension occurred simultaneously 20 times per sec.

For each dimension, the intercept a was defined as the average value of the corresponding signal, S, for 12 trials consisting of the three most recent trials for each of the four possible locations of the red target (McFarland et. al., 1997a). Thus, the intercept minimized directional bias, maximized the influence that the user’s EEG control had on the direction (e.g., upward or downward) of cursor movement, and helped make all targets equally accessible. The slope (or gain) b determined the magnitude of the cursor movement for a given value of (S − a). The slope was automatically selected so as to provide similar horizontal and vertical movement (i.e., the same aspect ratio as the screen resolution in pixels), and to produce cursor movement periods that typically lasted 2 to 3 sec.

Control of Target Selection

Like each dimension of cursor movement, target selection was controlled by a linear equation in which the control signal comprised a weighted combination of sensorimotor-rhythm features from channels over sensorimotor cortex, usually on the left side. The frequencies and locations of the features used for each subject are shown in Table 1. If G was the target selection signal,

G=bg(Sg−ag) (3)

was the function that determined target selection. The target was selected when the value of this function was below zero (i.e., when imagery-related desynchronization occurred), and was rejected if the value was equal to or greater than zero. The intercept a was defined as the average value of the control signal over the last 12 trials. The slope b remained at a value of 1 throughout since the selection was simply determined by whether the value of G was positive or negative.

Feature Selection and Weights

As noted above, the signal features that controlled cursor movement and target selection were amplitudes in 3-Hz wide frequency bands with center frequencies between 12 and 30 Hz, and came from EEG channels located over sensorimotor cortex. C3 and C4 were the most common channels (Table 1). For each user, channel and frequency selections at the beginning of training were based on the initial screening data (Wolpaw and McFarland, 1994). As training progressed, they were modified on the basis of results using a stepwise regression analysis (McFarland and Wolpaw, 2005). Separate regression equations were evaluated for prediction of horizontal target position, vertical target position, and target selection. During on-line performance, feature weights for these three regression equations were updated at the end of each trial with the LMS algorithm (Hayken, 1996). This continual adaptation used past performance to optimize the feature weights (Wolpaw and McFarland, 2004).

To assess the potential value of controlling cursor movement with weighted combinations of mu and beta rhythm amplitudes from channels FC3, FC1, FCz,FC2, FC4, C3,C1,Cz,C2,C4,CP3,CP1,CPz,CP2, and CP4, we calculated, in offline analyses of the data from each of the users, the correlations with target location of each amplitude singly and in weighted combinations using the multiple regression procedure from SAS (SAS Institute Inc). Parameter estimates were determined using least-squares criteria and the normal equations:

(X'X)b=X'Y (4)

where X is a m by n matrix formed from the n observations of m predictor variables (i.e., EEG amplitudes at specific frequencies and locations) and Y is the vector of n values (i.e., target positions) to be predicted. Solving for b, the vector of feature weights, yields:

b=(X'X)−1X'Y (5)

Correlation was expressed as r2, the proportion of the total variance in target location that was accounted for by the model for the two-sec cursor movement period.

We used the stepwise option as a feature selection heuristic. Briefly, a combination of forward and backward stepwise regression is implemented. Starting with no initial model terms, the most statistically significant predictor variable having a p-value < 0.01, is added to the model. After each new entry to the model, a backward stepwise regression is performed to remove any variables having p-values > 0.01. This process is repeated until no additional terms satisfy the entry/removal criteria.

Features that were selected offline by stepwise regression analysis initially were weighted according to the results of that analysis. Subsequently at the end of each trial the feature weights were updated using the LMS algorithm (Hayken, 1996) in conjunction with the prediction error for target position.

For target selection, the initial feature choice was based on evaluation of the user’s EEG during grasp imagery. Subsequent modifications in the features used and the weights assigned to them employed the same offline and online protocol described above for cursor movements.

Table 1 shows, for each user’s final sessions, the scalp locations and frequencies of the rhythm amplitudes used for each of the three control signals (i.e., vertical, horizontal, and target selection). The final results shown in Table 1, which differ markedly among users, are the products of the interactions during training between each user’s capacities and the adaptation produced by the LMS algorithm.

Evaluation of EMG activity during BCI operation

Following the completion of the primary study, 4 of the 6 users participated in an ancillary study to assess EMG activity during BCI performance. Both of the users with spinal cord injury were included in these sessions since their injuries did not preclude control of their hands and forearms. Six bipolar electrode pairs were placed on the forearm flexors, forearm extensors, and palm of each arm. For 2–3 standard sessions from each user, EMG activity was recorded continuously while the user performed the cursor movement/target selection task. Prior to each of these sessions, the user performed a maximum voluntary contraction (MVC) (i.e., by making a fist with each hand) to provide a denominator for evaluating EMG amplitude during task performance.

3. Results

For each user, performance gradually improved over the training sessions as s/he gradually gained better control over the EEG features (i.e., the rhythm amplitudes) that controlled cursor movement and target selection, and as the adaptive algorithm gradually adjusted the weights so as to vest control of cursor movement and target selection in those signal features (i.e., amplitudes in specific 3-Hz frequency bands from specific EEG channels) that the user was best able to control. As previously described (Wolpaw and McFarland, 1994; 2004), users tended to employ motor imagery to control cursor movements, particularly early in training. This imagery involved muscle groups not paralyzed in the users with spinal cord injuries. As noted, four of the six users had no previous BCI training, while two had participated in a variety of studies. For the present study, following one- dimensional and two-dimensional training, the users had 14–38 sessions (i.e., 5–15 hrs) of training on the complete move-and-select task. The data presented here are those of each user’s final three sessions, comprising 484–602 trials from each of the six users. From these data, we assessed both EEG control and the control of cursor movement and target selection that the EEG control provided.

EEG Control

We assessed EEG control during cursor movement by spectral and topographical analyses of the correlations (measured as R2) between the vertical and horizontal locations of the red target (i.e., the correct target) and the average values for the trial of the vertical and horizontal control variables (i.e., from Eq. 1 and Eq.2), respectively (Wolpaw and McFarland, 1994; Sheikh et al, 2003). Each variable correlated with its own dimension of target location and showed little correlation with the other variable’s dimension (Table 2). The users developed two independent control signals: one for vertical movement and one for horizontal movement.

Table 2.

Values of R2 for correlations of: the horizontal target position (H) with the horizontal control signal (SH); the vertical target position (V) with the vertical control signal (SV); the correct selection value (G) with the selection control signal (SG); the horizontal target position (H) with the vertical control signal (SV); the vertical target position (V) with the horizontal control signal (SH); and the horizontal control signal (SH) with the vertical control signal (SV).

User H-SH V-SV G-SG H-SV V- SH SH-SV
A 0.38 0.44 0.43 <0.01 <0.01 <0.01
B 0.31 0.37 0.50 <0.01 <0.01 0.03
C 0.18 0.33 0.07 <0.01 <0.01 <0.01
D 0.08 0.19 0.21 <0.01 <0.01 0.36
E 0.20 0.15 0.34 0.02 <0.01 0.09
F 0.14 0.11 0.31 0.04 0.03 0.30
Open in a new tab

We assessed EEG control during target selection in an analogous fashion by determining R2 for the correct selection value (coded +1 or −1) and the selection control variable (i.e., from Eq. 3). For each user, the selection control variable correlated with the correct selection value (Table 2).

Figure 2 shows for each user the topographies for the correlations (shown as R) between each of the three target dimensions (i.e., vertical location, horizontal location, and selection) and the amplitude of the EEG feature that made the largest contribution to the control signal (i.e., Eq. 1, 2, or 3) for that dimension. The correlations are shown as R rather than R2 in order to distinguish negative and positive correlations. For each signal in each user, control is focused over sensorimotor cortex. Within each user, the three control signals differ markedly in their topographies.

Figure 2.

Figure 2

Open in a new tab

Topographies for each of the six users (A–F) for the correlations (shown as R) between each of the three target dimensions (i.e., vertical location (V), horizontal location (H), and selection (G)) and the amplitude of the EEG feature that made the largest contribution to the control signal (i.e., provided by Eq. 1, 2, or 3) for that dimension. (The correlations are shown as R rather than R2 to distinguish negative and positive correlations.) For each signal in each user, control is focused over sensorimotor cortex. A user’s three topographies usually differ markedly from each other.

Figure 3 shows for User A the equations that provided each control signal and the spectral properties of that control signal. As indicated in Table 1 and Table 2, for this user vertical and horizontal movements were controlled by different and mutually independent combinations of 12-Hz activity over right and left sensorimotor cortices, and target selection was controlled by 15-Hz and 12-Hz activity over left sensorimotor cortex. It is worth noting that the FC1 15-Hz band had the largest weight in the equation that determined target selection even though, as Figure 2 shows, this feature did not have the largest univariate R value. Nevertheless, as a result of the LMS algorithm, it made the largest contribution to the bivariate equation that also included C3. McFarland et al. (2006a) discusses the complexities of evaluating multivariate models in BCI research.

Figure 3.

Figure 3

Open in a new tab

Spectral properties of User A’s vertical, horizontal, and selection control signals. From top to bottom are shown: the equations that defined these signals; the voltage spectra for these control signals; the R2 spectra corresponding to the voltage spectra; and representative examples of the time-domain signals from EEG channels that contributed to the control signals. (For the vertical or horizontal control signal, the two features used were from the same frequency band of different EEG channels, and the spectra shown are the combination of the two channels. For the selection control signal, the two features used were from different frequency bands of different EEG channels, and thus the spectra shown are for only one of the channels.) These data illustrate the sensorimotor control that enabled the user to move the cursor to the target and to select the target if it was correct.

The middle of Figure 3 shows, for the three vertical and three horizontal target levels and for the two selection levels (i.e., Select/Reject), the voltage spectra from which were derived the control signals (from Eq. 1, Eq.2, Eq.3) and their corresponding R2 spectra. At the bottom of Figure 3 are samples of EEG from locations that contributed to the three control signals. These samples are during the cursor movement period for trials in which the target was at the top or bottom or at the right or left screen edge, and during the target selection period for trials in which the correct selection choice was “Select” or “Reject.” They illustrate the strong sensorimotor rhythm control that the user employed to move the cursor to the target and then to select it. While 12-Hz mu activity changed at both locations with both dimensions of target location (e.g., Fig. 2 and Fig 3), the adaptive algorithm arrived at feature weights that gave independent vertical and horizontal control signals (Table 2).

Control of cursor movement and target selection

The EEG control summarized in Table 2 and illustrated in Figure 2 and Figure 3 gave each user significant cursor movement control. Users A–E reached the correct target in 59–88% of the trials (with 25% expected by chance); and, once a target was reached, these users correctly selected or rejected it in 71–91% of the trials (with 50% expected by chance). Every user’s performances for both cursor movement and target selection were significantly better than chance (p< 0.0001 by chi-square analysis).

Average cursor trajectories to each target for each user are shown in Figure 4. These trajectories are averages of individual trials lasting 5 seconds or less that are normalized in terms of the individual trial duration (i.e., each point is a proportion of the individual trial duration). Figure 5a shows each user’s accuracies for cursor movement, target selection, and their combination. The percentage for the combination is based on the conditional probability of correct cursor movement given a selection. This would represent the accuracy of a functioning system where rejected targets would only reduce speed but not accuracy. Given that no target was selected it could have been either the correct target or an incorrect target. However this distinction is not relevant for system performance. Figure 5b shows for each user the percentages for the three possible trial outcomes: correct target selected; no target selected; or incorrect target selected. These data make three important points. First, all the users were successful in both the cursor movement and target selection phases of the task. Second, the levels of performance on both phases differed markedly across users. (It is worth noting that the performances of the two users with spinal cord injuries were the first and third best of the six.) Third, the worst possible target outcome, selection of an incorrect target, occurred rarely in each user. In most of the trials that were not correct selections, no selection was made. These null trials waste time, but they do not create errors that then require correction. Thus, they are far preferable to incorrect selections.

Figure 4.

Figure 4

Open in a new tab

Average cursor trajectories to each target for each user for all correct cursor movement trials that reached the target within 5 sec. These trajectories are based on movements normalized by the duration of individual trials prior to averaging. The + signs delineate successive 10ths of the trial.

Figure 5.

Figure 5

Open in a new tab

Summary of cursor movement and target selection performance for each user. A: Accuracies for cursor movement (black), target selection (light gray), and their combination (dark gray). The percentage for the combination is based on the conditional probability of correct cursor movement given a selection. This illustrates that combining the cursor movement and selection tasks improved net accuracy in each user. B: The percentage of total trials in which the correct target was selected (black), no target was selected (light gray), or an incorrect target was selected (dark gray). In all users, the worst outcome, incorrect selection, is relatively uncommon, and much less common than the neutral outcome, no selection.

We also examined the average cursor movement time and accuracy of target selection for each target separately. Across the six users, average movement times were 2.5–3.5 sec. For four of the six users, selection success did not depend on target location, while two users showed slight but significant dependence.

Analysis of concurrent EMG activity

All six users sat quietly during both parts of task and made no overt movements. Figure 6 presents the results of the analysis of forearm and palm EMG during the target-selection period for the four users who participated in this ancillary study. As Figure 6 shows, EMG was low in all four users. The correlation between EMG and target type (i.e., correct or incorrect) was low in Users A and F and substantial in Users B and C. Most importantly, in all four users, the level of EMG contributed very little to the EEG control. That is, as Figure 6 shows, the R2 value for the correlation between the EEG variable (i.e., from Eq. 3) that controlled target selection and the correct selection value (coded +1 or −1) was only slightly reduced by removing the impact of EMG (i.e., by correcting for the variance common to the EEG variable and the combined activity of all six EMG channels). Thus, in all four users, EMG activity was low during target selection, and target selection was largely independent of that EMG activity.

Figure 6.

Figure 6

Open in a new tab

Summary of EMG activity and its impact during the target selection phase in each of the four users tested. A: EMG (as % of maximum voluntary contraction) for the user’s most active muscle when the target reached was correct (black) or incorrect (gray). B: The corresponding R2 values for: a multiple regression with all EMG channels (black); a regression with the EEG control signal (light gray); and a regression with the EEG control signal corrected for the EMG channels (dark gray). For all users, the EEG control signal correlated with the appropriate selection outcome and that correlation was largely independent of EMG activity.

4. Discussion

The results and their significance

The results show that people can learn to use scalp-recorded EEG rhythms to move a cursor in two dimensions to reach a target and then to select the target. Control develops over training sessions as the user gradually acquires better EEG control and as the BCI system gradually focuses on those rhythm amplitudes that the user is best able to control. The sequential two-dimensional movement control and selection demonstrated in this study is a skill that user and system gradually master together. As control improves, the motor imagery that users typically employ early in training tends to become less important and performance becomes more automatic.

As Table 1 shows, for the cursor movement phase of the task, each user acquired control over two EEG variables (i.e., Eq. 1 and Eq.2), one for horizontal movement and one for vertical movement. Furthermore, for each variable correlation with the wrong dimension of movement was very low (The correlation between control channels was rather high in D and F, but this did not result in the signals being correlated with the wrong target). The achievement of two independent movement control signals was the result of user training in combination with the LMS algorithm. Each user also acquired control over a third variable (i.e., Eq. 3) that controlled the target selection phase.

None of the users displayed overt movements during either part of the task. Analysis of forearm and hand EMG indicated that subtle changes in muscle activity were not responsible for the EEG variable that controlled target selection (i.e., Eq. 3). As Figure 6 shows, for all four users tested, EMG was low during target selection, and the correlation of the EEG target-selection variable with the correct selection outcome was only slightly reduced when the impact of this EMG activity was removed. In Users A and F, EMG correlation with the correct outcome was very low. The EMG correlations found in Users B and C are consistent with previous evidence that motor imagery (in this case imagery of a grasp response) can affect EMG activity (Dickstein et al, 2005; Wehner et al, 1984) and spinal stretch reflexes (Bonnet et al, 1997; Li et al, 2004). Thus, it is likely that the correlations noted in these two users were simply an additional consequence of the mental imagery that controlled the EEG variable. The crucial finding is that, in all four users in whom concurrent EMG was studied, the EEG variable remained highly correlated with the correct selection outcome even after correcting for the effects of EMG.

This study differs from most previous studies of two-dimensional control in that it provides multiple possible targets in every trial and thus allows the possibility of an incorrect selection. In most other studies (Chapin et al, 1999; Wessberg et al, 2000; Taylor et al, 2002; Pesaran et al, 2002; Serruya et al, 2002; Wolpaw and McFarland, 2004; Hochberg et al, 2006), only the correct target appeared on the screen and failure consisted merely of not reaching the trial within a given time period. A protocol that permits incorrect selections is more realistic, since most communication and control tasks that are encountered in the real world have the possibility of errors. For example, typical icon selection tasks based on mouse control permit selection of the wrong icon. Similarly, reach and grasp tasks permit picking up the wrong object. Thus, the present work combines our recent advances in multidimensional control (Wolpaw and McFarland, 2004) with a more realistic user task. Furthermore, the combination of two sequential control tasks, cursor movement and target selection, greatly reduces the number of incorrect selections and thereby improves the rapidity of communication.

The move and select function demonstrated here emulates the operation of a standard mouse, which allows a user to move over an icon and then select it or not select it as appropriate. Thus, the results represent a key step towards development of EEG control analogous to a computer mouse that moves among a large number of icons and selects only the desired icon.

Comparison with Previous Invasive and Non-Invasive Studies

Like our recent study of EEG-based two-dimensional movement control (Wolpaw and McFarland, 2004), most studies of movement control with activity recorded by electrodes implanted within cortex have used one-target protocols, and thus have not permitted errors (Chapin et al, 1999; Wessberg et al, 2000; Pesaran et al, 2002; Serruya et al, 2002; Taylor et al, 2002; Hochberg et al, 2006). An exception is the invasive study of Musallam et al (2004) which used four targets (but did not have a second, target-selection (i.e., confirmation) phase). They reported average success rates of 34–75% in three monkeys, a range that is somewhat lower than the target-hit accuracy range of 59–88% reported here. On the other hand, they required less time after the initial target view period: 0.1–1.2 sec vs. 2.5–3.5 sec in the present study.

The only invasive study that has used a two-phase “move and select” protocol similar to that of the present study is Carmena et al (2003), which trained monkeys to move a cursor in two dimensions to a single target and then to select, or “grasp,” it. Only one target was presented in each trial, so that full comparison with the present results is not possible. Carmena et al. (2003) used larger targets (7.7% of the workspace compared to less than 3% in the present study, taking into account the size of the target and the size of the cursor), while their movement times were slightly shorter (2.2–2.7 sec compared to 2.5–3.5 sec in the present study).

The present study applied multiple linear models to EEG features in order to control cursor movement and target selection. The LMS algorithm continually modified the model parameters on the basis of past results so as to optimize future performance. In contrast, Carmena et al (2003) applied multiple linear models to single-neuron activity to control cursor movement and target selection. Their models were constructed from unit activity recorded during actual arm and hand movements and then applied in the absence of actual movement. Thus, the applicability of their approach in people who lack normal movement control is uncertain. The methodology of the present study, which does not begin from activity recorded during actual movement, may be more readily transferable to people who are paralyzed. In addition, this study's noninvasive methodology does not require that electrodes be implanted in the brain.

Several groups have reported studies of sequential one-dimensional control using amplitude in specific EEG frequency bands. Millan et al (2004) and Pfurtscheller et al (2006) used EEG to make successive selections in a maze navigation task. Muller and Blankertz (2006) used EEG signals to make successive selections with a spelling device. Muller-Putz et al (2005) used EEG to train a user who was quadriplegic to sequentially select different components of a grasp actuated with the Freehand system (Peckham et al, 2001). In contrast to the present study, all of these systems used the same EEG features for each component of the sequential task and controlled only one dimension at a time.

Potential Improvements

The present study goes beyond previous work to show that people can use EEG features to produce three different control signals and that these signals can function sequentially as well as simultaneously. Thus, it is clear that the belief (e.g., Hochberg et al, 2006) that an EEG-based BCI cannot go beyond two independent channels of control is not correct. The limits of EEG-based control remain to be defined. It is likely that EEG-based control can be improved in speed and accuracy, and extended to more independent channels, by further improvements in signal acquisition and signal processing, in feature selection, and in the adaptive algorithm that encourages and guides user training and optimizes the translation of the chosen features into control signals. Recent studies of activity recorded from the cortical surface (i.e., electrocorticographic (ECoG) activity) suggest that gamma activity may be particularly useful for control (Leuthardt et al, 2004; Ball et al, 2004). Lower frequency gamma activity (i.e., 30–50 Hz) can be detected in EEG, and warrants careful study as a possible source of BCI control features.

Invasive methods clearly result in a better signal-to-noise ratio than EEG. This may account for the fact that invasive methods at present may require less training (e.g. Leuthardt et al, 2004). Nevertheless, the control achieved by invasive methods does improve with training (e.g. Taylor et al, 2002). The training requirements for invasive and non-invasive methods have not yet been compared in a meaningful fashion. Future developments in signal recording and analysis for both approaches will affect and clarify their relative advantages and disadvantages. It is perhaps most probable that each approach will be found most suitable for particular applications and/or individuals.

5. Conclusions

This study extends the possible applications of non-invasive BCI technology to include multidimensional movement control and sequential target selection. The results are further evidence that it may not be necessary to implant electrodes in the brain to achieve control of complex tasks, and they thereby increase the probability that BCIs will eventually become an important communication and control option for people with severe motor disabilities.

ACKNOWLEDGMENTS

We thank Theresa M. Vaughan and Gerwin Schalk for valuable advice throughout this work and Jonathan S. Carp for his comments on the manuscript. This work was supported in part by grants from NIH (HD30146 (NCMRR, NICHD) and EB00856 (NIBIB & NINDS)) and the James S. McDonnell Foundation.

REFERENCES

  1. Andersen RA, Musallam S, Pesaran B. Selecting signals for a brain-machine interface. Curr. Opin. Neurobio. 2004;14:720–726. doi: 10.1016/j.conb.2004.10.005. [DOI] [PubMed] [Google Scholar]
  2. Ball T, Nawrot MP, Pistohl T, Aertsen A, Schulze-Bonhage A, Mehring C. Towards an implantable brain-machine interface based on epicortical field potentials. Biomed. Tech. 2004;49:756–759. [Google Scholar]
  3. Birbaumer N, Ghanayim N, Hinterberger T, Iversen I, Kotchoubey B, Kubler A, Perlmouter J, Taub E, Flor H. A spelling device for the paralyzed. Nature. 1999;398:297–298. doi: 10.1038/18581. [DOI] [PubMed] [Google Scholar]
  4. Bonnet M, Decety J, Jeannerod M, Requin J. Mental simulation of an action modulates the excitability of spinal reflex pathways in man. Cogn. Brain Res. 1997;5:221–228. doi: 10.1016/s0926-6410(96)00072-9. [DOI] [PubMed] [Google Scholar]
  5. Carmena JM, Lebedev MA, Crist RE, O’Doherty JE, Santucci DM, Dimitrov DF, Patil PG, Henriquez CS, Nicolelis MAL. Learning to control a brain-machine interface for reaching and grasping by primates. PloS Biol. 2003;1:1–16. doi: 10.1371/journal.pbio.0000042. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Chapin JK. Neural prosthetic devices for quadriplegia. Curr. Opin. Neurobiol. 2000;13:671–675. doi: 10.1097/00019052-200012000-00010. [DOI] [PubMed] [Google Scholar]
  7. Chapin JK, Moxon KA, Markowitz RS, Nicolelis MAL. Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nat. Neurosci. 1999;2:664–670. doi: 10.1038/10223. [DOI] [PubMed] [Google Scholar]
  8. Dickstein R, Gazit-Grunwald M, Plax M, Dunsky A, Marcovitz E. EMG activity in selected target muscles during imagery rising on tiptoes in healthy adults and poststroke hemiparetic patients. J. Mot.Behav. 2005;37:475–483. doi: 10.3200/JMBR.37.6.475-483. [DOI] [PubMed] [Google Scholar]
  9. Donoghue JP. Connecting cortex to machines: recent advances in brain interfaces. Nat. Neurosc. 2002;5:1085–1088. doi: 10.1038/nn947. [DOI] [PubMed] [Google Scholar]
  10. Fetz EE. Real-time control of a robotic arm. Nat. Neurosci. 1999;2:583–584. doi: 10.1038/10131. [DOI] [PubMed] [Google Scholar]
  11. Haykin S. Adaptive Filter Theory. Upper Saddle River, NJ: Prentice-Hall; 1996. [Google Scholar]
  12. Hochberg LR, Serruya MD, Friehs GM, Mukand JA, Saleh M, Caplan AH, Branner A, Chen D, Penn RD, Donoghue JP. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature. 2006;442:164–171. doi: 10.1038/nature04970. [DOI] [PubMed] [Google Scholar]
  13. Konig P, Verschure PF. Neurons in action. Science. 2002;296:1817–1818. doi: 10.1126/science.1073592. [DOI] [PubMed] [Google Scholar]
  14. Leuthardt EC, Schalk G, Wolpaw JR, Ojemann JG, Moran DW. A brain-computer interface using electrocorticographic signals in humans. J. Neural. Eng. 2004;1:63–71. doi: 10.1088/1741-2560/1/2/001. [DOI] [PubMed] [Google Scholar]
  15. Li S, Kamper DG, Stevens JA, Rymer WZ. The effect of motor imagery on spinal seqmental excitability. J. Neurosci. 2004;24:9674–9680. doi: 10.1523/JNEUROSCI.2781-04.2004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Marple SL. Digital Spectral Analysis with Applications. New Jersey: Prentice-Hall; 1987. [Google Scholar]
  17. McFarland DJ, Anderson CW, Muller KR, Schlogl A, Krusienski DJ. BCI meeting 2005-Workshop on BCI signal processing: Feature extraction and translation. IEEE Trans. Neural. Syst. Rehabil. Eng. 2006a;14:135–138. doi: 10.1109/TNSRE.2006.875637. [DOI] [PubMed] [Google Scholar]
  18. McFarland DJ, Krusienski DJ, Wolpaw JR. Brain-computer interface signal processing at the Wadsworth Center: mu and sensorimotor beta rhythms. Prog. Brain Res. 2006b;159:411–419. doi: 10.1016/S0079-6123(06)59026-0. [DOI] [PubMed] [Google Scholar]
  19. McFarland DJ, McCane LM, David SV, Wolpaw JR. Spatial filter selection for EEG-based communication. Electroencephalogr. Clin. Neurophysiol. 1997a;103:386–394. doi: 10.1016/s0013-4694(97)00022-2. [DOI] [PubMed] [Google Scholar]
  20. McFarland DJ, Lefkowicz T, Wolpaw JR. Design and operation of an EEG-based brain-computer interface (BCI) with digital signal processing technology. Behav. Res. Meth. Instrum. Comput. 1997b;29:337–345. [Google Scholar]
  21. McFarland DJ, Sarnacki WA, Vaughan TM, Wolpaw JR. Brain-computer interface (BCI) operation: signal and noise during early training sessions. Clin Neurophysiol. 2005;116:56–62. doi: 10.1016/j.clinph.2004.07.004. [DOI] [PubMed] [Google Scholar]
  22. McFarland DJ, Wolpaw JR. Sensorimotor rhythm-based brain-computer interface (BCI): Feature selection by regression improves performance. IEEE Trans. Neural. Syst. Rehabil. Eng. 2005;13:372–379. doi: 10.1109/TNSRE.2005.848627. [DOI] [PubMed] [Google Scholar]
  23. Millan J, Renkens F, Mourifio J, Gerstner W. Noninvasive brain-actuated control of a mobile robot by human EEG. IEEE Trans. Biomed Eng. 2004;51:1026–1033. doi: 10.1109/TBME.2004.827086. [DOI] [PubMed] [Google Scholar]
  24. Muller K-R, Blankertz B. Toward noninvasive brain-computer interfaces. IEEE Signal Processing Magazine. 2006 Sept;:128–130. [Google Scholar]
  25. Muller-Putz GR, Scherer R, Pfurtscheller G, Rupp R. EEG-based neuroprosthesis control: a step towards clinical practice. Neurosci Lett. 2005;382:169–174. doi: 10.1016/j.neulet.2005.03.021. [DOI] [PubMed] [Google Scholar]
  26. Musallam S, Corneil BD, Greger B, Scherberger H, Andersen RA. Cognitive control signals for neural prothestics. Science. 2004;305:258–262. doi: 10.1126/science.1097938. [DOI] [PubMed] [Google Scholar]
  27. Nicolelis MAL. Actions from thoughts. Nature. 2001;409:403–407. doi: 10.1038/35053191. [DOI] [PubMed] [Google Scholar]
  28. Peckham PH, Keith MW, Kilgore KL, Grill JH, Wuolle KS, Thrope GB, Gorman P, Hobby J, Mulcahey MJ, Carroll S, Hentz VR, Wiegner A. Efficacy of an implanted neuroprosthesis for restoring hand grasp in tetraplegia: a multicenter study. Arch. Phys. Med. Rehabil. 2001;82:1380–1388. doi: 10.1053/apmr.2001.25910. [DOI] [PubMed] [Google Scholar]
  29. Pesaran B, Pezaris JS, Sahani MS, Mitra PP, Andersen RA. Temporal structure in neuronal activity during working memory in macaque parietal cortex. Nat Neurosci. 2002;5:805–811. doi: 10.1038/nn890. [DOI] [PubMed] [Google Scholar]
  30. Pfurtscheller G, Leeb R, Keinrath C, Friedman D, Neuper C, Guger C, Slater M. Walking from thought. Brain Res. 2006;1071:145–152. doi: 10.1016/j.brainres.2005.11.083. [DOI] [PubMed] [Google Scholar]
  31. Schalk G, McFarland DJ, Hinterberger T, Birbaumer N, Wolpaw JR. BCI2000: A general-purpose Brain-Computer interface (BCI) system IEEE Transactions on Biomedical Engineering. 2004;51:1034–1043. doi: 10.1109/TBME.2004.827072. [DOI] [PubMed] [Google Scholar]
  32. Sellers EW, Krusienski DJ, McFarland DJ, Vaughan TM, Wolpaw JR. A P300 event-related potential brain-computer interface (BCI): the effects of matrix size and interstimulus interval on performance. Biol Psychol. 73:242–252. doi: 10.1016/j.biopsycho.2006.04.007. [DOI] [PubMed] [Google Scholar]
  33. Serruya MD, Hatsopoulos NG, Paminski L, Fellows MR, Donoghue JP. Instant neural control of a movement signal. Nature. 2002;416:141–142. doi: 10.1038/416141a. [DOI] [PubMed] [Google Scholar]
  34. Sharbrough F, Chatrian CE, Lesser RP, Luders H, Nuwer M, Picton TW. American Electroencephalographic Society guidelines for standard electrode position nomenclature. J. Clin. Neurophysiol. 1991;8:200–202. [PubMed] [Google Scholar]
  35. Sheikh H, McFarland DJ, Sarnacki WA, Wolpaw JR. EEG-based communication: Characterizing EEG control and performance relationship. Neurosci Lett. 2003;345:89–92. doi: 10.1016/s0304-3940(03)00470-1. [DOI] [PubMed] [Google Scholar]
  36. Taylor DA, Helms Tillery S, Schwartz RA. Direct cortical control of 3D neuroprosthetic devices. Science. 2002;296:1829–1832. doi: 10.1126/science.1070291. [DOI] [PubMed] [Google Scholar]
  37. Wehner T, Vogt S, Stadler M. Task-specific EMG characteristics during mental training. Psychol Res. 1984;46:389–401. doi: 10.1007/BF00309071. [DOI] [PubMed] [Google Scholar]
  38. Wessberg J, Stambaugh CR, Kralik J, Beck PD, Laubach M, Chapin JK, Kim J, Biggs J, Srinivasan MA, Nicoleis MA. Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature. 2000;408:361–365. doi: 10.1038/35042582. [DOI] [PubMed] [Google Scholar]
  39. Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM. Brain-computer interfaces for communication and control. Clin. Neurophysiol. 2002;113:767–791. doi: 10.1016/s1388-2457(02)00057-3. [DOI] [PubMed] [Google Scholar]
  40. Wolpaw JR, McFarland DJ. Multichannel EEG-based brain-computer communication. Electroencephalogr. Clin. Neurophysiol. 1994;90:444–449. doi: 10.1016/0013-4694(94)90135-x. [DOI] [PubMed] [Google Scholar]
  41. Wolpaw JR, McFarland DJ. Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. Proc. Natl. Acad. Sci. 2004;101:17849–17854. doi: 10.1073/pnas.0403504101. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES