Neural control of cursor trajectory and click by a human with tetraplegia 1000 days after implant of an intracortical microelectrode array

Abstract

The ongoing pilot clinical trial of the BrainGate neural interface system aims in part to assess the feasibility of using neural activity obtained from a small-scale, chronically implanted, intracortical microelectrode array to provide control signals for a neural prosthesis system. Critical questions include how long implanted microelectrodes will record useful neural signals, how reliably those signals can be acquired and decoded, and how effectively they can be used to control various assistive technologies such as computers and robotic assistive devices, or to enable functional electrical stimulation of paralyzed muscles. Here we examined these questions by assessing neural cursor control and BrainGate system characteristics on five consecutive days 1000 days after implant of a 4 × 4 mm array of 100 microelectrodes in the motor cortex of a human with longstanding tetraplegia subsequent to a brainstem stroke. On each of five prospectively-selected days we performed time-amplitude sorting of neuronal spiking activity, trained a population-based Kalman velocity decoding filter combined with a linear discriminant click state classifier, and then assessed closed-loop point-and-click cursor control. The participant performed both an eight-target center-out task and a random target Fitts metric task which was adapted from a human-computer interaction ISO standard used to quantify performance of computer input devices. The neural interface system was further characterized by daily measurement of electrode impedances, unit waveforms and local field potentials. Across the five days, spiking signals were obtained from 41 of 96 electrodes and were successfully decoded to provide neural cursor point-and-click control with a mean task performance of 91.3% ± 0.1% (mean ± s.d.) correct target acquisition. Results across five consecutive days demonstrate that a neural interface system based on an intracortical microelectrode array can provide repeatable, accurate point-and-click control of a computer interface to an individual with tetraplegia 1000 days after implantation of this sensor.

1. Introduction

Tetraplegia (quadriplegia) is a condition in which willful control of the limbs is severely impaired or absent (Maynard et al 1997). Locked-in syndrome involves even more marked disability in which individuals lose all voluntary motor control with the exception in some cases of limited eye or other trace movements (Plum and Posner 1972, Laureys et al 2004, 2005, Lule et al 2009). Most individuals with tetraplegia depend on caregivers for mobility and physical interaction with their environment. The use of computers as a resource for communication and productivity is hindered by slow, unreliable or cumbersome input methods such as mouth sticks to type on a computer keyboard, EMG switches to make sequential selections in alphabet-scanning software or head/eye tracking systems to point to on-screen items in specialized assistive software. Individuals with anarthria resulting from, for example, brainstem stroke or amyotrophic lateral sclerosis (ALS) often rely on eye movements alone for communication, conveying one letter at a time to an attendant holding an alphabet board. In addition, individuals with tetraplegia may incur disability-related lifetime health care and living costs on the order of $1.8–3.2 million in the United States (National Spinal Cord Injury Statistical Center 2009). These burdens could be reduced by improving tools for communication and independence.

Many etiologies of tetraplegia or locked-in syndrome (for example, brainstem stroke and ALS) can leave cognitive and cerebral motor control structures largely intact. Since these structures contain movement signals directly related to lost functions, they are a promising source of relevant command signals. Thus, some types of brain computer interfaces (BCIs) aim to decode neural activity related to intended movements in order to provide control signals for external devices. Although movement-related field potentials can be recorded from the scalp surface using electroencephalography (EEG) (Birbaumer et al 1999, Schalk et al 2004, Wolpaw and McFarland 2004, McFarland et al 2008, Felton et al 2009, Bradberry et al 2010) and intracranially using electrocorticography (ECoG) (Leuthardt et al 2004, 2009, Felton et al 2007, Schalk et al 2007, 2008, Pistohl et al 2008, Kellis et al 2009), access to cortical signals most directly related to motor output is provided by recording the spiking activity of populations of individual neurons in cerebral motor areas. However, the need for penetrating electrodes and the requisite long-term stability of the sensor to obtain these signals have raised questions about the long term viability of this approach. A number of studies have nonetheless confirmed the validity of the concept. Spiking activity recorded from microelectrode arrays implanted in motor cortex of awake behaving able-bodied monkeys has been successfully decoded to provide real-time closed-loop neural control of cursor movement and robotic devices in two-dimensional (2D) space (Serruya et al 2002, Carmena et al 2003, Musallam et al 2004, Santhanam et al 2006, Ganguly and Carmena 2009) and three dimensions (Taylor et al 2002, Helms Tillery et al 2003, Jarosiewicz et al 2008, Velliste et al 2008). In monkeys, at least one type of sensor appears to provide neural signals for months to more than a year (Suner et al 2005). Initial successes of closed-loop control in able-bodied monkeys with this same sensor motivated our translational human pilot clinical study. A goal of this trial is to determine the potential for neural activity recorded by an intracortical microelectrode array in the motor cortex of humans with tetraplegia to provide a control signal for assistive devices. The BrainGate pilot clinical trial, conducted under an investigational device exemption from the U.S. Food and Drug Administration (FDA), was initiated in 2004 to test the safety and feasibility of a neural interface system based on an intracortical 100-electrode silicon recording array (Cyberkinetics Neurotechnology Systems, Inc.) derived from a device developed at the University of Utah (Jones et al 1992, Nordhausen et al 1994) and advanced for human applications by our group (Maynard et al 1999, Suner et al 2005, Hochberg et al 2006). Results from our first trial participants not only demonstrated the ability to record both spikes and field potentials from the array over a number of months and the ability for those with tetraplegia to control neural spiking (Hochberg et al 2006, Truccolo et al 2008), but also provided proof of concept that individuals with tetraplegia resulting from spinal cord injury could readily use motor cortical spiking for 2D control of a computer cursor years after spinal cord injury (Hochberg et al 2006). Subsequent tetraplegic trial participants were also able to use the system over a period of months to volitionally control computer cursor movements for a point-and-click interface using command signals decoded from spiking activity in motor cortex years after stroke or in advanced-stage ALS (Kim et al 2008, 2011).

These BrainGate pilot trial initial findings to date support the hypothesis that an intracortical array can serve as a BCI sensor. However, the long-term viability of chronically implanted microelectrodes and the reliability of command signals derived from spiking activity over long time periods remain to be established. Studies with animals employing other electrode types have reported tissue damage and immune responses within days or weeks (Turner et al 1999, Szarowski et al 2003, Moxon et al 2004, McConnell et al 2009; see Polikov et al 2005 for review) including cell death and morphological changes (Edell et al 1992, Biran et al 2005, McConnell et al 2009), and severe signal loss within months of implant (Moxon et al 2004; but see Vetter et al 2004). By contrast, a silicon-based array of microelectrodes projecting from a platform has been demonstrated to record 180 days in cats (Rousche and Normann 1998, Maynard et al 2000) and a version of this array identical to the one for human clinical study (Neuroport array, Blackrock Microsystems, Salt Lake City) has successfully recorded spiking signals in monkeys for at least 300 to 500 days (Suner et al 2005, Dickey et al 2009). These results suggest that this array may be suitable as a sensor for chronic human applications. The long-term participation of one person in the BrainGate trial provided the opportunity for a systematic test of a neural interface system (brain-computer interface) that utilizes signals recorded from this array years after it was implanted. The current study examines point-and-click control achieved by an individual with tetraplegia on five consecutive days on the occasion of crossing 1000 days after implant of a 100-electrode array of parylene-insulated, platinum-tipped, microelectrodes fabricated from semiconductor silicon.

2. Methods

2.1. Participant

The participant reported here, S3, was enrolled in a pilot clinical trial (Investigational Device Exemption, IDE⁹) of the BrainGate Neural Interface System¹⁰. Participants who enroll in the study have limited or no use of their hands due to spinal cord injury, brainstem stroke, muscular dystrophy, or motor neuron disease such as amyotrophic lateral sclerosis (additional information about the clinical trial is available at http://www.clinicaltrials.gov/ct2/show/NCT00912041). At the time of this study, S3 was a 56-year-old female with tetraplegia caused by brainstem stroke that occurred 12 years prior to the collection of the data presented in this report. She is unable to speak (anarthria) and has no functional use of her limbs. She has occasional bilateral flexor spasm movements of the arms; these may be initiated by any imagined or actual attempt to move. S3’s sensory pathways remain intact. She also retains some head movement and facial expression, has intact eye movement, and breathes spontaneously.

On November 30, 2005, a 96-channel intracortical silicon microelectrode array (produced by Cyberkinetics Neurotechnology Systems, Inc, and now by its successor, Blackrock Microsystems, Salt Lake City, UT) was implanted in the arm area of motor cortex as previously described (Hochberg et al 2006). One month later, S3 began regularly participating in ~1–2 research sessions per week during which neural signals were recorded and tasks were performed toward the development, assessment, and improvement of the neural interface system. This research was conducted with Institutional Review Board (IRB) approval and an IDE.

2.2. General research procedures

In anticipation of 1000 days post sensor implant in S3, a single research protocol was designed to be performed on each of the five consecutive days bridging that milestone. Our goal was to obtain a multi-day measure of the effectiveness and reliability of the control signal provided by recordings from this sensor. The prepared research protocol described below was repeated daily from August 25, 2008 through August 29, 2008, coinciding with implant days 999 through 1003. Sessions were performed in one or the other of two rooms as available at the participant’s residence. The same trained clinical trial technician oversaw all activities according to the research protocol.

During each session, the participant was seated comfortably in her wheelchair, slightly reclined from vertical and facing a 17″ flat panel monitor (34 × 27 cm, 1280 × 1024 pixels) centered slightly above eye level at a distance of approximately 56 cm. This distance provided comfortable viewing and was verified at the start of each session to ensure that on-screen images always subtended the same degrees of visual angle from the participant’s vantage point. Specifically, the screen subtended 33.8 × 27.1 degrees of visual angle and each centimeter on screen subtended approximately 1.02° of visual angle. During sessions, involuntary slow flexor spasm movements of the arms occasionally resulted in changes of arm position. We did not measure or otherwise control for these postural changes except where necessary to ensure the participant’s comfort.

The same sequence of events was executed each day. In brief, the technician used aseptic technique to connect the 96-channel recording cable between the percutaneous pedestal and the signal preamplifier and then used commercial software to view neural signal waveforms and specify time-amplitude discrimination windows (‘spike sorting’) identifying candidate spiking units for each of the 96 channels. Automated electrode impedance measurements were then acquired. Next, neural signals were recorded while the participant imagined controlling, with her right hand, a viewed cursor as it executed preprogrammed trajectories on the computer monitor. She also imagined a right hand grasp when prompted by on-screen cues. The participant was encouraged to relax and remain still while imagining performing, rather than actually attempting, these movements. The total training procedure lasted 10 to 12 min after which a pre-programmed estimation procedure was executed to train a hybrid kinematic/click state decoder (Kim et al 2011). Closed-loop assessment of the neural decoder commenced immediately with a 10 min, eight-target radial task (‘Radial-8’) followed by a 10 min sequential step tracking task for quantifying information throughput (‘mFitts1’). During this time the participant used cortical spiking activity to control the neural cursor’s continuous 2D trajectory and click. Following these tasks, the recording cable was disconnected and the participant left the room prior to end-of-session data management and documentation activities. Total participant contact time on each of these days was 1.5 to 2 h.

Details of these procedures are presented below.

2.3. Signal acquisition and spike sorting

Motor cortical activity was recorded from the microelectrode array (4.2 mm × 4.2 mm), which had been implanted via pneumatic inserter into the motor cortex arm area as described in Hochberg et al (2006). Signals from this 10 × 10arrayof 100 platinum-tipped silicon electrodes (400 µm spacing, 1. 5 mm long, ~4 µm tip diameter) were analog filtered (Butterworth band pass filter with 1st and 3rd order corners at 0.3 Hz and 7.5 kHz, respectively) and digitized by the 96-channel NeuroPort Neural Signal Processor (NSP) at 30 k samples s⁻¹ (14 bits per sample, 1 µV bit⁻¹). Signals were then filtered with a digital high-pass Butterworth filter (250 Hz 4-pole low frequency corner) prior to spike processing.

Using commercial on-line spike sorting software (Cerebus Central), the technician specified a minimum neural signal amplitude above which waveform snippets of signals were captured (48-points, 1.6 ms window). Based on continuously accumulated samples of thresholded waveforms, the technician interactively specified time-amplitude discrimination criteria to identify neural activity (‘spikes’) of putative single unit or multiple single-unit neural activity (hereafter referred to together as ‘units’). The software allowed spike waveforms of several units to be discriminated on each electrode. This sorting procedure was repeated for all 96 electrodes. During the session, timestamps of all discriminated units were stored and, during decoding, made available to the decoder software. In addition, the analog-filtered 30 kS s^–1 digitized signals were recorded from all 96 channels for offline analysis.

Data synchronization and decoding were controlled by custom software running in Matlab (The Mathworks, Natick, MA) with a nominal 100 ms time step (cycle interval). Operating in Matlab’s ‘interpreted’ mode on a Microsoft Windows XP computer yielded actual cycle intervals typically ranging from 100–120 ms (109.1 ± 4.8 ms, mean ± s.d., across all 10 closed-loop assessment tasks). On each cycle, cursor kinematics and unit spike events were sampled. Spike event timestamps were also saved with their native 33 µs resolution from the Neuroport NSP to allowed for re-interpolation of spike rates and kinematics during the filter build process (below) to reduce the effect of cycle interval variability. During closed-loop control, spiking activity was decoded on every cycle after normalizing firing rates based on the actual elapsed cycle interval.

2.4. Electrode impedance measurements

Each day, the impedance of each microelectrodewasmeasured (1 kHz stimulus, ± 50 pA max) using the built-in capabilities of the Neuroport hardware and Cerebus Central software. This test required about 1 min to perform measurements across all electrodes. Neural activity from electrodes with impedance greater than 10 MΩ was not utilized for the duration of the session. Offline, the distribution of impedance values was compared to (a) device impedances measured in saline prior to surgery using a manual impedance probe, and (b) to post-implant impedances measured approximately 12 weeks after implant using the automated impedance tester described above. The change in impedance at each electrode was normalized to the standard deviation of the sample of impedance changes across the array (i.e., z-score transformed) and the distribution of scores was examined for evidence of shifts in impedances relevant to performance results reported here.

2.5. Decoder training procedures

Next, a training data set for the decoding algorithm was recorded in six blocks of 90 s each. For all training blocks the participant was instructed to imagine controlling a preprogrammed training cursor (diameter 1.35 cm, area 0.16% of screen area) as it moved in a straight line between consecutive targets on the computer screen. Throughout training, the screen displayed one central target and eight peripheral targets (round, diameter 2.16 cm, area 0.40% of screen area) in an oval configuration spanning a large portion of the monitor. The center-to-center distance between the center and peripheral targets was 11.5, 12.5 and 13.5 cm for targets located on the vertical, diagonal and horizontal axes, respectively.

To begin each block, the cursor was located in the center target and one of four peripheral targets (0, 90, 180, or 270 polar degrees) was highlighted. The four diagonal targets were not visited during training. After an instructed delay of 300 ms, the training cursor moved with a truncated Gaussian speed profile to the center of the highlighted peripheral target where it paused for an inter-trial interval of 300 ms. The center target was then highlighted and after another 300 ms instructed delay the training cursor returned to the center with the same speed profile. This sequence proceeded pseudo-randomly through all four peripheral training targets (and the intervening center target) before a different sequence of the four peripheral targets was performed. The same predetermined pseudorandom sequence of training movements was executed for each training block on each of the five days. Peak cursor speeds at the midpoint of each movement were higher in three training blocks (2.5 s movement time between targets) relative to the other three (4.5 s inter-target time).

To provide training for the click state decoder, the above training paradigm was modified in two of the six blocks by interleaving a click training epoch after each return-to-center cursor trajectory. In these blocks, after the cursor stopped on the center target, the cursor and all targets disappeared and after 620 ms the word ‘squeeze’ was displayed on the screen. This served as the participant’s cue to imagine performing a brief hand grasp as if to execute a click a mouse. After 1.75 s the word disappeared and 620 ms later the targets and cursor re-appeared and the next out-and-back cursor trajectory started.

The foregoing kinematic and state training blocks were interleaved as follows:

1. Trajectory training (slow cursor)

2. Trajectory training (fast cursor)

3. Click state training

4. Trajectory training (slow cursor)

5. Trajectory training (fast cursor)

6. Click state training.

In all, 9 min of training data were collected in a sequence of six blocks of 90 s each. The time between consecutive blocks was less than 30 s. These training procedures were similar to those presented previously (Hochberg et al 2006, Kim et al 2008, 2011) but differed in that here there were no closed-loop training blocks. Instead, these decoders were trained with only open-loop imagined movements and the participant did not attempt neural control of cursor movement prior to each day’s assessment tasks.

2.6. Neural decoders

This study employed a hybrid kinematic/state filter that combined a Kalman filter to estimate 2D cursor velocities and a linear discriminant classifier to detect click intentions (Kim et al 2011). All sorted units with mean spike rates above 1 spike s^–1 across the 9 min of training data were included in the training set; no directional tuning criteria were applied. Cursor kinematics and spike timestamps were resampled into interpolated 100 ms bins prior to training the algorithm. Spike counts were normalized by bin duration to yield spike rates. Coefficients for the Kalman filter were computed from the four kinematic training blocks that did not contain click training epochs (6 min of data).

The linear discriminant classifier, which differentiated the intention to click from the intention to move the cursor, employed a click state ‘observation’ that incorporated the current and previous three bins of spike rates (total 400 ms). The classifier was trained from the cursor movement epochs in the four kinematic training blocks and the click (‘squeeze’) epochs in the two state training blocks. Note that the neural signals recorded during the kinematic portions of the click training blocks were not used for classifier training out of concern that the participant’s anticipation of clicking could overlap with kinematic epochs and degrade class separation. Therefore, after the classifier coefficients were estimated, a cross-validation procedure was applied to select the separation boundary between click and movement classes based on the state epochs and previously unused kinematic epochs from the state training blocks. This procedure computed the likelihood ratio of click over move for each validation sample and determined the decision boundary which maximized separation of the classes in the validation set. Building the combined hybrid kinematic/state filter took about 5 s including resampling and the cross-validation procedure.

2.7. On-line spike decoding

The hybrid kinematic/state filter was applied to real-time decoding of unit spiking activity as the participant imagined hand movements and grasp action to control a computer cursor. At each time step (nominally 100 ms; see above), spike counts were normalized by the actual elapsed software loop time to yield an ‘instantaneous’ spike rate measure for each unit. The Kalman filter velocity estimate was computed and integrated with the previous cursor position to yield a new cursor position. On every cycle the state classifier also differentiated the participant’s intention to perform a click or move the cursor. A click command was sent to the task computer whenever the estimated state transitioned from movement to the click state. The Kalman velocity estimate was also reset to zero on each cycle that the click state was detected. As described in Kim et al (2011), the relative balance of click and movement states could be adjusted by setting the prior probability of being in a movement or click state and by setting the probability of staying in a state or transitioning between states. Here, however, constant values were maintained for these settings throughout all five days.

The cursor position was bounded by the screen edges so that it remained visible at all times. When cursor position was limited by the screen boundary, the next Kalman filter estimate incorporated zero velocity as the value of the previous state. Except for this screen edge condition, the point-and-click decoder generated its kinematic and state estimates with no information about the task running on the task computer.

2.8. Radial-8 assessment task

The participant performed two tasks on each day to provide metrics quantifying closed-loop point-and-click cursor control. The first task was a ‘Radial-8’ task (supplementary figure 1 and supplementary movie, available at stacks.iop.org/JNE/8/025027/mmedia) in which one center target and 8 peripheral targets were arranged in an oval layout with the same size and positions as in the training task. Target sizes were approximately the same as desktop icons when arranged by Microsoft Windows in 15 × 12 grid. The minimum required straight-line movement distances from the start position to the target edge were 9.7, 10.8, and 11.7 cm in the Y, diagonal, and X directions, respectively. The cursor diameter was again 2.16 cm. The dimensions in this task yielded an Index of Difficulty (see next section) of 2.6, 2.7, and 2.8 bits for the three target distances, respectively.

All targets were initially gray and remained visible and selectable throughout each assessment. At the beginning of the task, the cursor was placed in the middle of the center target and then released to neural control (‘neural cursor’) for the duration of the task. A trial began when one target turned from gray to red indicating the goal location. The participant then attempted to move the neural cursor to the target and execute a click while the cursor overlapped the target by at least 1 pixel. If successful, a short beep was heard and the target colour intensified briefly before turning back to gray as the next target changed to red starting the next trial. Trial goals alternated between one of the eight pseudo-randomly selected peripheral targets and the center target such that the task proceeded in a continuous center-out-back sequence. Peripheral targets were offered without replacement until all eight positions had been presented. Trials ended in an error if no target was selected before the trial timeout period (25 s from target onset). Error trials also resulted when any gray distractor target was selected. Selection errors turned the target blue with a brief tone and the cursor was forced to the intended target position before the next trial started. The latter procedure was applied after error trials, when the end position of the cursor was uncontrolled, to ensure that the next trial assessed cursor control in the intended direction and from the designed starting point on the screen. Throughout the task, every neural click command was logged and a click sound was provided as audible feedback. Clicks on the desktop between targets did not cause the trial to end in an error.

The Radial-8 task was performed continuously without interruption for 10 min on all five days except the first, in which the task lasted only 9 min. This early termination resulted when the Cerebus Central software generated a warning message (at time 8m40 s) indicating that the Ethernet network throughput of neural data to the Matlab decoding computer was compromised due to unexpectedly high system or Ethernet activity. The technician then terminated the Radial-8 task at 9 min to ensure data integrity.

2.9. mFitts1 sequential target task

A second assessment task each day was motivated by the International Organization for Standardization (ISO) 9241-9 standard (2000, 2002) for assessing the performance of multidirectional, non-keyboard human computer interaction (HCI). The original 2D ISO task uses a radial layout with a sequence that alternates between peripheral targets on opposite sides of the circle and progresses continuously one target clockwise on each trial. Task difficulty is adjusted through the selection of movement distance (task circle diameter) and target size. However, that design tests only one trajectory from each starting point and those points are limited to the circumference of a circle centered in the workspace. Those features may not provide adequate assessment of a neural cursor that could exhibit non-uniform control in different directions and locations in the workspace. We therefore developed a modification of the ISO task in which targets of any size could occupy any position on the display, thereby assessing control along varied trajectories originating throughout the workspace.

Our modified task, mFitts1, was a sequential step-tracking paradigm in which one round red target at a time was presented on the computer screen and the participant moved the neural cursor to the target and performed a neural click to select it (supplementary figure 1 and supplementary movie, available at stacks.iop.org/JNE/8/025027/mmedia). The target was highlighted whenever the cursor was touching it to indicate that this item was selectable by a neural click. A trial ended successfully when a neural click was executed while the neural cursor touched the target. Clicks elsewhere on the desktop generated a click sound and were registered for analysis but did not otherwise affect the trial. If the target was not selected by click within 25 s then the trial ended with a timeout error. There were no distractor targets to be selected.

The sequence of target positions and sizes was determined pseudo-randomly by an algorithm designed to assess movements with uniformly distributed directions and start locations on the screen. To ensure that the designed distribution of trajectories was tested, the cursor was moved at the end of each trial to the center of the intended target as described above. This procedure was performed purely for the purpose of computing performance metrics and maintained controlled trajectory start locations following error trials. Following correct trials, the distance of this move was less than one target radius.

Task difficulty was varied not only by movement distance but also through the use of a range of target diameters. On each trial the target size was chosen pseudo-randomly from three fixed diameters (1.62 cm, 0.23% of screen area; 3.51 cm, 1.06% of screen area; 5.67 cm, 2.77% of screen area). The smallest of the target sizes was approximately the size of a standard menu bar icon (e.g. ‘Save File’) in Microsoft Word. Target locations near screen edges were adjusted to ensure that the entire target was visible. The neural cursor was a circle of diameter 0.54 cm (20 pixels, 0.025% of screen area).

The ISO 9241-9 standard takes into account the observation that, in many circumstances, the time needed to acquire targets depends on task difficulty (Fitts 1954, Welford 1960, Mackenzie 1992, Soukoreff and MacKenzie 2004, Zhai 2004). We adopted from those studies the principle that the ‘difficulty’ of each cursor movement could be quantified by an index of difficulty (ID) that incorporates the size of the target to be selected and the distance to be travelled to the target,

$$\mathrm{I}\mathrm{D}={\text{log}}_2(\frac{D}{W}+1)$$ (1)

where D is the center-to-center distance from the cursor start position to the actual end location on the selected target (where the click occurred) and W is the diameter of the target. Three different target sizes in random sequential locations yielded a range of D and W (and therefore ID) for which movement time was measured. Using this notion of task difficulty, Fitts Law has been widely applied to assess computer input device performance assuming a linear relationship between the expected movement time MT (in seconds) and task index of difficulty ID (in bits):

$$\mathrm{M}\mathrm{T}=a+b*\mathrm{I}\mathrm{D}$$ (2)

where the y-intercept a can be interpreted as the minimum possible acquisition time, and slope b specifies the additional acquisition time introduced as task difficulty increases. This formulation identifies a familiar speed-accuracy tradeoff originally embodied in Fitts Law as derived for point-to-point ballistic movements by able-bodied individuals. While application of Fitts Law per se may not be appropriate in brain-computer interfaces that do not attain able-bodied trajectory characteristics (Radwin et al 1990, Pino et al 2003), here we adopt this linear relationship for convenience with the expectation that as neural interface system performance approaches that of able bodied users, this model will approach the well characterized linear performance model embodied in Fitts Law for able-bodied subjects. The current paradigm and metrics reflect our initial effort to adopt and adjust existing HCI standards to the unique characteristics of neural HCI with the objective of quantifying progress toward the goal of achieving cursor performance on par with able-bodied computer mouse users.

On each of the five days, targets in the mFitts1 task were presented and acquired continuously for 10 min.

2.10. Quantifying cursor control

Neural control of cursor movement and click was evaluated on each day for the Radial-8 and mFitts1 assessment tasks using measures of gross task performance, cursor trajectory properties, and click accuracy.

‘Total Hit Rate’ was calculated as the proportion of successfully selected targets to the total number of targets offered in each 10 min task. Successful trials required that the cursor be moved to the target and a click executed while the cursor was placed over the target. Errors in both tasks resulted whenever a click did not occur on the target before the timeout (25 s). In addition, an error trial occurred in the Radial-8 task if a click was executed on any distractor targets. To highlight this type of error, ‘Selection Hit Rate’ measured the proportion of correct target selections to the total number of targets actually selected, excluding timeout errors in which no selection was made.

Hit rates, while common measures for selection tasks, do not account for the time required to make selections. The mean sustained rate of target acquisition was calculated by dividing the total number of correct selections by the total task time including timeout trials (Selections Per Minute).

The speed and accuracy of neural cursor movements were further quantified in the mFitts1 task using movement time MT and index of task difficulty ID adopted from the ISO 9241-9 standard as described above. For able-bodied users, MT is often taken to incorporate only the principal ballistic movement excluding reaction, homing, and selection times (Salmoni 1983, Schellekens et al 1984) while in other studies one or more of these phases are included (see Zhai 2004). Because cursor lag, inertia, homing and click decoding could represent a significant portion of the total target selection time for some neural decoding systems, performance measurements excluding one or more phases of movement may not adequately represent system throughput for BCI applications. Therefore, we use as ‘movement time’ the total selection time including reaction time, distance covering, homing, and selection action. We notate this total movement and selection time as ‘ST’ and rewrite equation (1) as

$$\mathrm{S}\mathrm{T}=a\prime +b\prime *\mathrm{I}\mathrm{D}$$ (3)

where the values of a′ and b′ differ from a and b in equation (2) due to the use of ST instead of MT. The y-intercept a′ still reflects the minimum possible acquisition time including reaction time and selection time (as well as regression modeling errors; Zhai 2004; for a dissenting view, see Soukoreff and MacKenzie 2004). Slope b′ again incorporates the additional time penalty realized when acquiring targets of increasing difficulty. For significant regressions of ST onto ID (P < 0.01), the coefficients a′ and b′ may potentially together serve as metrics for comparing performance among different computer users and input devices evaluated with this task (Zhai 2004). Note that the regression computed here and throughout the Fitts Law literature is defined only for correct trials due to the requirement for finite selection time.

All trajectories from correct trials of the Radial-8 task were plotted for each day. Mean trajectories to and from each peripheral target were also overlaid after compressing or expanding each trajectory into a standardized movement duration (the mean movement duration for each day) and after reversing peripheral-to-center trajectories so that start-of-movement events were aligned in these plots. For quantitative analysis, cursor trajectories in the random mFitts1 task were evaluated using metrics adopted from in Mackenzie et al (2001) and applied in earlier BrainGate reports (Kim et al 2008, 2011):

1. ODC: the number of Orthogonal Direction Changes in which the cursor reversed away from or back toward the target. This is a measure of how consistently the cursor moved forward to the target (best = 0),

2. MDC: the number of Movement Direction Changes away from or back toward the task axis. This is a measure of trajectory smoothness (best = 0),

3. DR: Distance Ratio is the ratio of the actual trajectory length to the ideal straight-line path (best =1),

4. ME: Movement Error measures the mean distance (offset) of the cursor from the task axis and quantifies offset of the cursor path from the ideal straight path (best = 0),

5. MV: Movement Variability measures the standard deviation of the cursor offset. It characterizes the trajectory smoothness independent of offset from the task axis (best = 0),

6. TE/h: Target Exits per Hit counts the number of times the cursor enters and then exits a target without a click being performed.

The user was free to click at any time and location on the screen. The false click ratio (FCR) quantified the accuracy of click control across all trials as the ratio of ‘false’ clicks (when the cursor did not reside on any target) to clicks on the intended target. An FCR of 0 would indicate no false clicks; an FCR of 1 would indicate an average across error and correct trials of one false click per correct target selection. While clicks on the background were not explicitly penalized in either assessment task, they did stop the cursor and penalized performance by increasing the elapsed time to select a target.

2.11. Unit analysis

Spiking ‘units’ were manually sorted online by the technician each day based on waveform shape without reference to previous sessions (i.e. there was no attempt to identify consistent units across days). Units quickly sorted in this way are not ideally suited for strict neurophysiological analysis and it is likely that some ‘units’ included waveforms of several or many co-recorded neurons. Nonetheless, the technician-sorted units comprised the base neural signals available for decoding during these research sessions.

As one measure of the quality of sorted units, we computed the mean waveform for each unit by averaging the 48-point, 1.6 ms waveforms captured with each spike event (see methods above) during the daily 10 min mFitts1 assessment task. Each unit’s waveforms were also quantified in terms of the first and second order statistics of their peak-to-peak amplitude.

No directional tuning criteria were applied during the selection of units for decoding. However, to provide a common measure of signal quality for the units used in the decoders, each unit’s spiking activity recorded during the Radial-8 task was fit with a von Mises directional tuning model that has been shown to capture well the sensitivity of spike rates to movement direction (Amirikian and Georgopoulos 2000). For each unit, the mean firing rate r associated with intended movement in direction θ is modeled as

$$r(\theta )=d+k*\text{exp}(\kappa \text{cos}(\theta -u))$$ (4)

where d and k are unit-specific regression coefficients, u is the unit’s preferred direction and κ specifies the sharpness of the tuning curve which corresponds to cosine tuning for κ = 1. Mean firing rate was calculated in the interval from 0.25–3.75 s after target onset. Units were deemed significantly tuned if the per-direction firing rates were significantly different (Kruskal-Wallis, P < 0.05) and the von Mises nonlinear model fit was significant (P < 0.05). Significantly-tuned units were characterized by their preferred direction, firing rate, and modulation depth using the fitted model. Modulation depth was computed as the modeled difference in firing rate between the preferred and anti-preferred directions as a proportion of the maximum (preferred direction) firing rate (values from 0 to 1).

2.12. Local field potential analysis

As a measure of electrode functionality we examined local field potentials (LFPs) recorded from each electrode for evidence of modulation related to imagined movements. One feature of LFPs recorded from motor cortex (as well as other regions) during actual or imagined movements is desynchronization in beta-band oscillatory activity (14–30 Hz) at movement onset relative to the pre-movement period (Donoghue et al 1998). To examine the movement-related LFP oscillatory activity across a range of frequencies, the time frequency spectrogram was computed for each electrode’s LFP signal that was recorded during the daily closed-loop Radial-8 assessment task. The original 30 kS s^–1 LFPs were decimated to 1 kS s^–1 and band-pass filtered (1–160 Hz, Butterworth 8th-order zero-phase). A segment of the signal was extracted around the appearance of the target that cued movement onset in each trial (1s before to 3 s after). For each segment, the power spectra density (PSD) was computed at frequencies from 1 Hz to 80 Hz in 1 Hz steps (Gortzel algorithm) in 512 ms overlapping windows stepped by 6 ms. The resulting spectrograms were averaged to yield a mean peri-event spectrogram for each electrode for each day. Each mean spectrogram was normalized in each 1-Hz analysis band by the in-band mean value in the 500 ms prior to target onset to yield the proportion of power in each band relative to pre-movement activity. Analysis of these data revealed a maximum event-related desynchronization for frequencies centered on ~15 Hz. The peri-event LFP from each trial was therefore band-pass filtered (10–20 Hz, 8th order Butterworth) to examine movement-related desynchronization in the beta band activity in each electrode’s LFP.

3. Results

3.1. Task performance with the neural cursor

Continuous neural control of the trajectory and click of a computer cursor was achieved on each of the five consecutive days for both the center-out-and-back Radial-8 task and the 2D step tracking mFitts1 task (figure 1). Across the five days, the rate of successful target acquisition averaged 94.9% for the Radial-8 task and 91.9% for the mFitts1 task. Once each task started, the participant had unrestricted and continuous control of the cursor direction, speed and click just as is provided by a typical computer mouse. Successful trials required that the cursor be moved to the target and a click executed while the cursor hovered over the target. Errors resulted whenever a click did not occur on the target within the allotted time. Errors could also occur in the Radial-8 task if a click was executed on any of the distractor targets. However, no incorrect target selections occurred across the five days of testing (Selection Hit Rate = 100%). Of the combined total 564 assessment trials, 526 were performed correctly (93.4%) and all 37 errors were attributable to the target not being acquired within the timeout period (table 1). Although performance varied somewhat across days, performance results indicate that the decoder conferred very good spike-based control of cursor velocity and click throughout the workspace of both tasks. On the best day (1003), continuous 2D point-and-click control yielded 100% correct target selection in both tasks with 161 targets acquired over a period of 20 min (figure 1 and table 1).

Figure 1.

Performance on the Radial-8 and mFitts1 tasks on five consecutive days in which the neural cursor was used for point and click target selection. Daily performance on the Radial-8 task was assessed as the proportion of correct target selections relative to the total number of targets selected (Selection Hit Rate, black) and as a proportion of the total number of targets presented (Total Hit Rate, blue, middle) in 10 min (9 min for Day 999). Daily performance on the mFitts1 task was similarly assessed as the proportion of successfully acquired targets relative to the total number of targets presented (Total Hit Rate, red, right) in 10 min.

Table 1.

Details of target selection performance on the Radial-8 and mFitts1 assessment tasks on five consecutive days.

		Radial-8 task					mFitts1 task
Trial day	# units	Hit rate (%)	Total trials	Correct trials	Timeout errors	Selection errors	Hit rate (%)	Total trials	Correct trials	Timeout errors
999	25	100.0	53	53	0	0	67.5	40	27	13
1000	22	98.5	66	65	1	0	100.0	63	63	0
1001	18	83.7	43	36	7	0	88.4	43	38	5
1002	26	84.8	46	39	7	0	91.8	49	45	4
1003	29	100.0	85	85	0	0	100.0	76	76	0
Total	n/a	n/a	293	278	15	0	n/a	271	249	22
Mean	24	94.9	58.6	55.6	3.0	0	91.9	54.2	49.8	4.4

Total Hit Rates were similar for the Radial-8 and mFitts1 tasks. An exception was Day 999 in which cursor performance was perfect (100%) on the Radial-8 task but decreased notably (67.5%) on the mFitts1 task. On this day the Cerebus Central data recording software reported at 8m50 s into the Radial-8 task that a high operating system load (Microsoft Windows XP) was impairing data transfer from the neural signal digitizing hardware to the decoding computer. The operator immediately stopped the task and saved the session data. No further system problems were noted so the session resumed with the mFitts1 task 1m30 s after termination of the Radial-8 task. We suspect, but cannot confirm, that the outlying performance on the mFitts1 task on this day might be attributed to an unresolved underlying system problem that persisted after the original throughput warning was noted. No system problems were noted at any other time during the 5-day study. Average selection times (reaction time + cursor translation + homing + click) for correct trials across the five days were 8.5 ± 3.7 s for the Radial-8 task and 8.7 ± 5.0 s for the mFitts1 task (mean ± s.d.). Target selection rates ranged from 2.7 to 8.6 correct selections per total task minutes (5.4 ± 1.9 sel min⁻¹) across both tasks and all five days (figure 2(a)). These selection rates included timeout error trials and were sensitive to the generous 25 s timeout which we use in order to capture data for offline analysis of episodes with less accurate control. However, due to the low incidence of error trials, selection rates increase only modestly (to 6.0 ± 1.6 sel min^–1) when timeout error trials are excluded. Selection rates were similar irrespective of task on each day (with the noted exception of Day 999) but were variable across days (table 1). Some control variability was anticipated with our current procedure of re-sorting units and recomputing new decoders each day, but more units did not always result in higher performance.

Figure 2.

The rate of target acquisition for two assessment tasks. (a) Item selection rate for the Radial-8 task (blue) and mFitts1 task (red) on each of 5 consecutive days (total correct selections divided by total task time). (b) The dependence of target selection time (measured from target appearance through click execution) on selection task difficulty for all 62 correct trials of the mFitts1 task for Day 1000. (c) Regression of selection time on task difficulty (as in (b)) for each day. The regression for Day 999 (dotted line) was non-significant. (d) The relationship between selection time and selection difficulty for all correct mFitts1 trials from the five days showing significant linear regression on the pooled data. (open circles indicate data from Day 999 for which the within-day regression was non-significant). (e) Target diameter and distance-to-target for all 22 mFitts1 error trials (circles) across the 5-day study. Short lines indicate minimum and maximum distances offered (correct and error trials) for each of the three target diameters.

While selection rates provided one measure of BCI performance, the mFitts1 task was developed in an effort to characterize speed-accuracy tradeoffs in the current version of this BCI. Presenting three target sizes in a pseudo-random sequence of movement distances resulted in a range of ID values from 0.6 bits to 4.5 bits (2.4 ± 0.9 bits). The largest IDs corresponded to the smallest target appearing at a location distant from the cursor starting point. Daily mean selection times (ST) for this task ranged from 7.0 to 11.5 s (9.0 ±1.8 s). Figure 2(b) displays for Day 1000 a linear least-squares regression (R² = 0.31, P < 0.001) showing that selection times on individual trials were dependent on task difficulty. At least one large, close target was selected in less than 1 s and selection times increased systematically with ID to greater than 15 s. Linear regressions (figure 2(c)) were significant for each day (P < 0.001; R² = 0.31, 0.28, 0.67, 0.44) except for Day 999 (see the discussion above). The performance of this system might be quantified in terms of the regression slope (b′ = NS, 2.5, 4.0, 4.9, 2.4) and extrapolated Y-intercepts (a′ = NS, 2.5, 2.0, –1.8, 1.1) together. These exhibited some variability across days, but pooling selection times across all five days (including Day 999) yielded a significant (P < 0.01; R² = 0.36) regression (figure 2(d)) that could provide an indication of this neural interface system’s average performance (a′ = 0.8, b′ = 3.3). The y-intercept a′ indicates the minimum target acquisition time for a target with index of difficulty 0 and b′ indicates the additional penalty to acquire targets of increasing difficulty.

Although the Fitts regression incorporates only correct trials due to the requirement for a finite acquisition time, one can ask how error trials related to task difficulty. In the current study, errors were not strongly dependent on distance to target (figure 2(e)). Rather, errors were distributed over a broad range of distances. On the other hand, of the 22 timeout errors recorded on the mFitts1 task in this study, 12, 6, and 4 occurred with the small, medium, and large target diameters, respectively (figure 2(e)). Hence, small target size appears to be the more important factor in failure to acquire targets.

Using equation (1) we also calculated ID for the Radial-8 task. Although in this task all target sizes were equal, the elliptical placement of targets yielded different movement distances corresponding to ID ranging from 2.3 to 2.6 (mean 2.45). The mean selection time of 8.5 s reported here for the Radial-8 task is consistent with the performance predicted for this ID using the regression model derived from the mFitts1 task (figure 2(d)). This supports other findings above that indicate performance with this neural interface system was consistent across tasks.

3.2. Cursor trajectory and click control

The center-to-peripheral and peripheral-to-center cursor trajectories, from target onset to target selection (click), for all correct trials of the Radial-8 task (figure 3) reveal that the consistently high target hit rates were achieved with variable path efficiency. On the day with the lowest target hit rate (Day 1001) few straight-line trajectories were observed. On the day with the best hit rate (Day 1003), the participant was able to more consistently achieve direct, albeit slightly curved, trajectories. The mean trajectories to each target (which include both out and back movements) are superimposed in red to facilitate comparison with other studies in which individual trajectories are not depicted. The participant achieved cursor speeds during the Radial-8 task averaging 66.7 ± 47.1 pixels s^–1 and was able to smoothly command a range of speeds from 0 to over 200 pixels s^–1 (figure 3(b)). No gain or smoothing was applied to the instantaneous velocity estimates from the Kalman filter.

Figure 3.

Neural cursor kinematics for trials in the Radial-8 task on five consecutive days. (a) Trajectories for all correct center-to-peripheral and peripheral-to-center movements (thin lines) with mean trajectories (thick lines). Gray circles (targets) and black circle (neural cursor) are drawn to scale with the outlining box demarcating display screen limits. All correct trials (‘T’) are shown for each day and span 10 min except Day 999 which was terminated at 9 min. ‘U’ indicates the number of units in the decoding filter. (b) Histogram of Radial-8 cursor speeds computed from the decoded velocity estimate at every time step (bin = 6 pix s⁻¹; 26 689 samples) in the Radial-8 task. Each colour shows the distribution from one of the five days.

No restriction was placed on the participant who often rotated her head in the horizontal plane as is common for following a cursor on a closely-placed monitor. We did not measure these movements in the current study, although the dissociation between head movement and neural cursor movement was previously described (Hochberg et al 2006).

Cursor trajectories in the mFitts1 task, which cover random distances and directions and are more akin to standard mouse-controlled cursor movements made during able-bodied control of word processing or web browsing software, were analyzed using quantitative measures presented in table 2. Across days, acquisition of the three different target sizes took an average 8.7 s with a cursor path 50% longer than the ideal straight-line path (mean Distance Ratio = 1.5). The cursor moved consistently toward the target (mean ODC = 1.6) but exhibited wavering (non-zero MDC, MV) and trajectories were offset from the ideal task axis by an average 1.46 cm (ME), or roughly three cursor diameters. The Target Exits per Hit (TE/h) of 0.31 indicates that on average 69% of all target acquisitions were achieved the first time the cursor entered the target.

Table 2.

Pointing metrics for neural cursor control in the mFitts1 task for five consecutive days. Task duration was 10 min for each day. The 5-day summary metrics (bottom) evaluate all correct trials pooled across days. (mean ± s.d.). ODC: orthogonal direction changes; MDC: movement direction changes; ME: movement error; MV: movement variability; TE/h: target exits per hit.

Trial Day	Total trials	Correct trials	Hit rate (%)	ODC	MDC	Distance ratio	ME (mm)	MV (mm)	TE/h
999	40	27	67.5	1.6 ± 1.9	3.1 ± 2.7	1.4 ± 0.6	15.7 ± 13.2	9.7 ± 6.7	0.96
1000	63	63	100.0	1.5 ± 1.9	4.0 ± 2.6	1.3 ± 0.5	12.1 ± 9.1	8.6 ± 4.5	0.41
1001	43	38	88.4	2.7 ± 2.8	4.1 ± 2.6	1.9 ± 1.3	20.3 ± 15.0	16.1 ± 11.5	0.11
1002	49	45	91.8	2.4 ± 2.4	3.4 ± 2.9	1.6 ± 0.6	19.7 ± 18.4	14.6 ± 11.1	0.29
1003	76	76	100.0	0.7 ± 1.2	3.7 ± 2.8	1.2 ± 0.3	10.2 ± 7.7	7.0 ± 4.5	0.14
5 Days	271	249	91.9	1.6 ± 2.1	3.7 ± 2.7	1.5 ± 0.26	14.6 ± 13.0	10.5 ± 8.4	0.31

This neural interface system allowed the user to generate a click at any time and any location on the screen. Clicks were well distinguished from movement epochs (figure 4(a)). During this study, no clicks were ever generated on incorrect targets during the Radial-8 or mFitts1 tasks. Clicks were generally decoded in the vicinity of the intended target (figure 4(b)). Clicks decoded while the neural cursor was between targets were termed ‘false clicks’ because they performed no target selection function. The incidence of false clicks was quantified by the False Click Ratio (FCR, table 3) where an FCR of 0 would indicate that a single click was executed to select each target and no other (false) clicks were generated. The inter-day mean FCR of 0.41 indicated less than one false click per every two successful target acquisitions. Day 1003 exhibited fewer than one false click per every 8 target selections. Another way to consider this is that the decoder determined click or movement state at every time step. Across five days of mFitts1 task there were 27 494 samples in which a click versus move decision needed to be reached. Considering 102 false clicks in this context, the decoder output was appropriate 99.63% of the time. The perfect mFitts1 hit rate on Day 1003 coincided with the fastest selection times, straightest cursor paths, and nearly the lowest TE/h (table 2) as well as the lowest False Click Rate (table 3). Hence, although there is a speed/accuracy tradeoff associated with decoding conditions on any given day (figure 2), it is possible to simultaneously achieve robust performance with regards to several important aspects of cursor control including speed.

Figure 4.

Neural click performance. (a) Demonstration of cross-validation testing during filter building on Day 1000 showing the two appended 90 s click training blocks in which imagined cursor movement was interspersed with epochs of imagined grasp (red). At every time step the decoder computed a log likelihood ratio (blue) and generated a click whenever it crossed to exceed an algorithmically-determined threshold (black line). (b) Distribution of distances between the neural cursor and the edge of target for all clicks during the mFitts1 assessment task across all five days. The green bar indicates clicks on targets; red bars indicate non-productive clicks on the desktop. 71% of all clicks were on target; of the remaining non-productive clicks, 75% were within 40 pixels of the target (red bins = 20 pixels).

Table 3.

Click metrics for neural cursor control in the mFitts1 task for five consecutive days.

Trial day	Total trials	Total clicks	Correct clicks	False clicks	FCR
999	40	64	27	37	1.37
1000	63	96	63	33	0.52
1001	43	45	38	7	0.18
1002	49	61	45	16	0.36
1003	76	85	76	9	0.12
Five days	271	351	249	102	0.41

3.3. Device and signal evaluation

Functional spiking signals recorded by the array allowed for successful cursor control across all five sessions as described above, demonstrating general integrity of the system. For further evaluation, impedance measurements were acquired from all 96 electrodes at the start of each session. The distribution of impedance values across the five sessions was bimodal (figure 5(a)). The cluster of measurements with impedances greater than 4 MΩ (3.6 on the log10 scale) were well above the expected operating range for this device. These high values came from two sets of electrodes. One set of seven electrodes was deemed non-functional because, on all five days (35 total measurements), they exhibited impedances above 4 MΩ and no identifiable (sorted) spiking activity (see figure 6); nor did they show beta suppression in the simultaneously-recorded LFP (see discussion below and figure 7). These observations were consistent with broken signal paths. Three of these seven channels had been nonfunctional since immediately after implant. The second set of high impedances derived from 18 other locations where values differed by an order of magnitude between days, exceeding 4 MΩ on some days (30 measurements) but not others. These signal paths appeared broken on days with high-impedance measurements and normal on the remaining days. These observations were consistent with variable connection quality between individual pins in the cable and pads on the percutaneous pedestal. After excluding all measurements above 4 MΩ, the distribution of electrode impedance measurements ranged from 116 kΩ to 2261 kΩ (385 ±160 kΩ, mean ± s.d.). Overall, the impedances in this study (5-day electrode means excluding > 4 MΩ) were lower (paired t-test, P < 0.001) compared to impedances measured (with different equipment) in saline prior to surgery (441 ± 210 kΩ) and in-vivo 12 weeks after surgery (438 ± 394 kΩ), but are in the range that commonly provide neural signal recordings with this sensor. The z-score transformed distribution of impedance changes relative to the 12-week in-vivo measurements (figure 5(b)) revealed a tendency toward lower impedance values except four outlier electrodes with large impedance increases (> 1.96σ).

Figure 5.

Recording array impedance measurements. (a) Histogram of impedance measurements for all 96 electrodes on five consecutive days. (b) Change in impedance (z-score) for each of 96 electrodes relative to early post-implant measurement. (c) Mean electrode impedances across five days (measured at 1 kHz). Layout corresponds to implanted 10 × 10 microelectrode array. Unlabeled black locations measured > 4 MΩ on all days and were nonfunctional. Corner electrode locations (black with label) were not connected in the 96-channel system.

Figure 6.

Properties of recorded units. (a) Number of units sorted by the technician (black), used in the filter (dark gray, middle) and significantly directionally tuned during the Radial-8 task (light gray, right) for each day. (b) Histogram of mean firing rates for each of the 183 units that were sorted during the five days (within-day means across 9 min of training data). Bins are 2 s except left bin which is 1 s to contain those units with firing rates less than 1 Hz that were excluded from decoding filters. (c) Distribution of mean waveform amplitudes for all units during each day’s mFitts1 assessment task (all technician-sorted units, black; the subset of units used for decoding, gray). (d) The mean sorted waveform (blue) from each day’s mFitts1 task shown for electrodes on the array where units were sorted in at least one session during the 5-day study. The mean daily waveforms of each second unit sorted on an electrode are also overlaid (red). At each electrode location the number of days on which a unit was recorded (i.e. the number of mean waveforms depicted) is indicated for first units (first number) and second units (second number). Gray-filled locations denote electrodes with impedances above 4 MΩ on all five days which were considered nonfunctional. Corner positions (black) were not connected in this 96-channel system.

Figure 7.

Changes in the spectral content of local field potentials associated with intended movement on Day 1000. (a) Trial-aligned mean spectrogram of the field potential recorded from one representative electrode during 10 min of the Radial-8 task. Time 0 indicates time of target presentation. Changes in power, relative to within-band power before target appearance, were computed in 1 Hz steps using overlapping windows. Colour indicates proportion of amplitude change for each frequency relative to pre-presentation baseline. Dark blue reveals suppression between 14~30 Hz evident 0.6~ 1.2after target appearncn Horizontal line marks the frequency exhibiting maximum suppression (15 Hz). (b) Single-trial field potential from the same electrode as in (a) aligned to target appearance (t= 0) and filtered (10–20 Hz, 8th order filter) to highlight changes in signal amplitude corresponding to the power suppression near 15 Hz. (c) Spectrograms as in (a) computed for each of 96 electrodes during the Radial-8 task (excluding frequencies below 2 Hz for clarity). Notations: (s) 30 electrodes with sorted units on this day, (x) seven putative failed electrodes with impedances ≫ 4 MΩ on all five days, (1) additional 8 electrodes with impedance > > 4 MΩ on this particular day, (2) 2 electrodes with impedance ~4 MΩ. Colour range identical to (a).

Mapping the 5-day mean impedances onto their corresponding electrode locations in the array (figure 5(c)) revealed clustering of the 7 non-functional electrodes along one edge of the array. Although the cause of those failures is unknown, it may prove relevant that this is the edge from which the bundle of 96 wires exits the array. Otherwise the spatial distribution of impedance values appeared unremarkable.

Single units and multiple single unit activity were sorted manually (‘sorted units’) at the beginning of each of the five sessions (figure 6). In total, 183 units were sorted during the five-day study (n = 34, 34, 37, 39, 39, respectively; 36.6 ± 2.5; figure 6(a), black). Of these, 63 were excluded from filter building and decoding because their mean firing rates recorded during the 9 min training period were less than 1 spike s^–1 The remaining 120 units (25, 22, 18, 26, 29 per day; 24 ± 4.2) contributed to daily decoding filters (figure 6(a), middle bars) and had an average firing rate of 5.3 ± 6.8 spikes s^–1 (figure 6(b)). The mean peak-to-peak amplitudes for the 183 sorted units ranged from 30.8 µV to 229.8 µV (63.1 µV ± 34.1 µV, mean ± s.d.; figure 6(c)). Amplitudes of the subset of units selected for decoding (61.2 µV ± 29.7 µV) were not significantly different from those excluded from the decoders (P = 0.29). The waveform amplitudes were relatively small but could be discriminated from background activity. As another measure of signal quality for the 120 units used in the decoders, each unit’s spiking activity recorded during the Radial-8 task was fit with a von Mises directional tuning model in offline analysis. Firing rates of 93 of these units were significantly tuned to movement directions imagined by the participant (figure 6(a), right bars; figure 8 for details). The number of directionally tuned units ranged from 10 to 29 across the five days (18.6 ± 6.8).

Figure 8.

Directional tuning of spiking activity during neural cursor control. (a) A population raster depicting all spiking events (ticks) of 33 simultaneously-recorded units (rows) during neural cursor point-and-click to one target in the Radial-8 task on Day 1000. Target onset occurred at time 0 (left vertical line) and the trial ended with a click on target at 6.18 s (right vertical line). (b) Spiking activity for one unit during all center-out cursor movements in the Radial-8 task on Day 999 (data set 2008.08.25.15.13.42_96a). For each of 8 movement directions, each spiking event is depicted by a tick with one row of ticks for each movement. Trials are aligned at target onset (time 0, red vertical line). Histograms below tally spiking events across trials and are normalized by bin size (333 ms for this analysis) to yield firing rates. This unit has a preferred firing direction of 347° (based on spiking activity between 0.25 s and 3.75 s after target onset) as summarized by the von Mises tuning curve (center). (c) Directional tuning among all significantly-tuned units in the recorded population during the Radial-8 task on each of five consecutive days. For each tuned unit, preferred direction and tuning depth from the fitted von Mises model are depicted by arrow orientation and length, respectively (top row of figures; radius represents modulation depth of 1). The number of units with significant tuning in each of the eight cardinal task directions was summed in 45° bins (bottom row).

The 183 units sorted during this 5-day study originated on 57 of the available 96 electrodes. After excluding units with mean firing rates below 1 Hz during training, the decoders utilized units from 41 electrodes (43%) of which 26 (27%) contributed units to decoding on more than one day. Electrodes from all quadrants of the array contributed sorted units, although units were more prevalent in some regions than others. The mean waveform recorded from each sorted unit during each 10 min mFitts1 task is shown at its electrode location on a map of the recording array (figure 6(d)). Mean waveforms from different days are overlaid and second units, when sorted on an electrode, are also shown. On many electrodes that recorded units on multiple days, the shape of waveforms remained fairly consistent across days even though units were sorted in each session without reference to previous days’ sorting results.

In this study, units were never sorted on 39 (41%) of the 96 array electrodes. Only 7 of these were deemed completely nonfunctional (the failed electrodes identified by impedance mapping, above). Given the ongoing discussion regarding the long-term reliability of intracortical microelectrodes, it was of interest to determine whether any evidence of neural activity could be detected in the remaining 32 locations for which we never saw viable spiking activity. We therefore examined the LFP recorded at every electrode. We reasoned that the ability to record the LFP, particularly signal changes associated with imagined movements (Hochberg et al 2006), would provide evidence to support the possibility that these electrodes were capable of recording electrical activity although no spikes were detected. For each trial of the Radial-8 task on each day, LFPs recorded at each electrode were aligned to the appearance of the target so that a trial-mean spectrogram of the LFPs could be computed. The spectrograms of many electrodes (e.g. figure 7(a)) showed strong reduction of power between 14–30 Hz about 1 s after target onset. Movement-related suppression of signal amplitude was evident in the beta-band signal (band-pass filtered 10–20 Hz for analysis) on some, but not all, single trials (figure 7(b)). LFP spectrograms at each electrode are presented in figure 7(c) for Day 1000 (see supplementary material for other days). The large majority of functioning electrodes showed evidence of beta suppression even in the absence of sorted units. The time of maximum beta-band desynchronization 1 s after target onset was consistent across the electrodes. Seven non-functional electrodes (labeled ‘x’) and eight others that exhibited high impedance on this day (>13 MΩ, labeled ‘1’) showed no movement-related modulation of the recorded signal, consistent with broken or open signal paths. Beta suppression signature on two other electrodes with high impedance (~ 4 MΩ, labeled ‘2’) may be attributable to electrical crosstalk onto these poorly connected signal paths. Hence, excluding high impedance connections on this day, nearly all of the electrodes showed evidence of movement-related beta suppression in the recorded LFP irrespective of sorted units.

3.4. Population spiking activity related to imagined movement and grasp

The population spiking activity underlying cursor control was examined offline to examine whether properties characteristic of MI neurons remained at this long post-implant (and post-stroke) interval (figure 8). Firing rates of many units were significantly tuned to a direction of intended cursor movement during the Radial-8 assessment task. The preferred directions of tuned units in the population spanned directions throughout the workspace (figure 8(c)) although the distribution was not uniform on any day (Kuiper and Raleigh tests for circular uniformity, P > 0.05). This nonuniformity is highlighted by tallying the number of preferred directions in each of eight 45° bins (figure 8(d)). The movement directions that were most prominently represented in the ensemble varied across days. On each day, however, the directional information in the ensemble firing rates was sufficient for the Kalman velocity decoder to provide neural cursor control that enabled a 93.4% mean target acquisition rate.

4. Discussion

The present study demonstrates that neuronal spiking patterns recorded through an intracortical microelectrode array can provide reliable neural interface control for a person with tetraplegia nearly two and three quarter years after the array was implanted in motor cortex. S3 showed the ability to move to and select targets with high success on each of five consecutive sessions, prospectively selected to bridge the 1000th day post-implant. Control included continuous 2D point and click in a new task that requires cursor movement to randomly located targets of different sizes, which closely mimics the use of a standard personal computer mouse. These results extend previous findings in humans with tetraplegia at shorter post-implant durations (Hochberg et al 2006, 294 days from a different participant; Truccolo et al 2008, from this participant at 90 days; Kim et al 2008, S3 between 40 and 418 days and an ALS participant between 85 and 224 days) to show that both LFP and spiking signals remain in MI, that these signals can be volitionally modulated and that continuous and state commands can be simultaneously and reliably decoded and utilized. These results help to diminish concerns that tissue reaction from a penetrating electrode will prevent detection of useful signals when measured from both a signal and functionality perspective. However encouraging, validation of these results will still require additional participants to confirm the long term capabilities of this intracortical electrode array as a sensor for brain computer interfaces. This milestone nevertheless establishes feasibility of this approach and extends results to substantially longer times for more complex computer mouse-like control.

4.1. Point-and-click cursor performance at 1000 days

Performance data show that S3 repeatedly achieved good point-and-click BCI control more than 2.7 years after sensor implant. Within each of the five sessions, 2D control of cursor direction and speed was maintained across 20 min of assessment tests including a fixed display of targets in Radial-8 and random target placements in the mFitts1 task. Under continuous control, S3 successfully reached and selected targets while never selecting an unintended target and while executing a relatively low proportion of false clicks on the desktop. One exception occurred on Day 999 in which a 100% target hit rate during the Radial-8 task degraded to a 68% hit rate in the ensuing mFitts1 task. A computer network problem occurred at this time and may have been the underlying cause in this case. Our data further show that very good performance could be achieved across all performance metrics at once using signals from this single, small cortical ensemble. For example, on Day 1003 S3 simultaneously achieved a 100% target hit rate, few false clicks (0.12 per target selection) and reasonably straight trajectories (~20% longer than a straight line). Performance metrics on this day were all among the top for any of the tested sessions.

Goal-directed movements commonly involve a tradeoff between speed and accuracy. The potential utility of a brain-computer interface will be determined in part by the balance between the target selection rate, the accuracy of the neural cursor, and the corresponding size (and therefore number) of icons that can be arrayed on the computer desktop for selection. To be effective, continuous control should be accurate throughout the workspace. We introduced the mFitts1 task to measure target selection times throughout the desktop and across a range of difficulties quantified by an Index of Difficulty that incorporated both target size and distance to target. Results show that targets of different sizes at random screen locations could be reached. The speed-accuracy tradeoff can be evaluated in part by considering how errors related to the different components of ID. In the current study, errors on the mFitts1 task were more likely on trials with smaller target diameters. However, errors were well distributed across the range of distance-to-target values presented in the task. Together, these results suggest that performance of this neural interface system could be improved more effectively by improving control of homing accuracy rather than control of cursor speed.

One objective for BCI applications will be to design neural decoders that reduce the required click decoding time without increasing the false click ratio. In the current study, the non-zero value of a′ from the mFitts1 task regression model, which represents the minimum target acquisition time when no cursor translation is required, would be comprised primarily of the participant’s reaction time and the time required to decode the user’s intention to click. We found that false clicks were generated primarily near the correct target and not randomly across the workspace. These predictive clicks may result because the participant needs to estimate timing between the cursor velocity, its arrival time on the target, and the click latency, which may be challenging. It is likely, therefore, that detecting the click intention with a lower or adaptive latency, as quantified by a lower a′, could further reduce the false click ratio. It is also possible that, in the current system, decoupling velocity and click commands by decoding these intentions from different neuronal ensembles could improve both state detection and kinematic control. Informative discrete selections might be decoded well from premotor cortex where a variety of distinct target identities can be identified quite rapidly, at least in able-bodied monkeys (Santhanam et al 2006). These more complex strategies might benefit from more than one implanted array, which has been achieved in non-human primates in our laboratory and others but has not yet been evaluated in humans.

Flexor spasm movements in this and other trial participants are frequently unavoidable and it is important to understand their influence on system performance. In addition, S3 retains some voluntary head and eye movement. Although we did not track body movements in the present study, previous analysis of video recordings indicated that physical movements in this clinical trial were not significantly correlated with on-screen cursor movements (Hochberg et al 2006). We are currently evaluating methods for tracking head, eye, and limb movements in the context of the clinical trial in order to include more thorough quantification of these factors in future studies.

Overall, the performance achieved here meets basic requirements for mouse operations to control a standard computer system, to perform simple ‘hunt and peck’ typing on a virtual keyboard, or to perform selections in specialized assistive software, all of which would be highly enabling for people unable to use their hands.

4.2. Performance variability and learning

Despite the high success rate in reaching targets on each of the five days, performance varied across sessions. S3 achieved successful performance using only 18–29 units to create the decoding filter. This is comparable to the number of units useful for decoding trajectories found for other human participants using this technology (Hochberg et al 2006, Kim et al 2011). In these data sets we included channels where it was difficult to sort individual waveforms, so that the decoder was constructed from mixtures of single- and multi-units. Nevertheless, high levels of control were achieved, indicating that neither single unit isolation nor high channel counts are essential for successful operation of the interface (see also Wood et al 2004). Because the daily sorting procedure used here resulted in a different number of units each day, some variability in performance across days should be attributable to unit count. Although the filter incorporating the highest number of units (table 1) did provide the best performance on most metrics, the total number of units in the decoder did not predict well performance outcome across the five days. Rather, performance was more closely correlated with the number of directionally-tuned units in each decoder (R² = 0.74 for Radial-8 targets per second; R² = 0.44 for Radial-8 hit rate). These results support the view that even small numbers of directionally tuned units from motor cortex can provide useful control signals in people with paralysis.

Although the best performance was achieved on the fifth day of the study, we do not believe that inter-day learning during this study contributed meaningfully to the results observed here. The first and second days also yielded excellent performance, particularly on the Radial-8 task, followed by two days of relatively reduced performance. Hence, there was no clear trend leading toward higher performance on the fifth day. In addition, although the participant has a comprehensive understanding of the behavioural tasks involved because she has performed them for years, our approach did not provide the opportunity for performance improvement through inter-day practice with fixed decoder coefficients. Rather, cursor control was assessed each day with a new decoder, with different numbers and composition of units. Our evidence documents that at least some change in the recorded population occurs each day, as demonstrated by the differences in the number of directionally tuned neurons across sessions. Thus on a particular day S3 had no prior exposure to the exact transform that mapped firing patterns from the unique population of recorded neurons to cursor movement. Hence, part of the inter-day performance variability can be attributed to the training and use of a new filter each day.

This participant, like our prior participants (Hochberg et al 2006, Kim et al 2008), achieved cursor control each day during this study using decoder coefficients computed using only a few minutes of neural signals recorded during imagined movement. We hypothesize that control using intracortical signals required no overt learning because spiking patterns contain arm motion related signals. This contrasts with EEG BCIs that have required an extended period to learn mappings between arbitrary mental imagery and the desired cursor motion and which demand significant attention during use. As a specific example, McFarland et al (2008) identified task-relevant features in participants’ EEG signals during initial sessions after which they learned to modulate those features to direct the cursor along single axes and, eventually, in 2 dimensions simultaneously. The period of this initial kinematic training was not specified but the six individuals each subsequently practiced with the same classifier for an additional 5–15 h (14–38 sessions) prior to the reported assessment sessions.

We did not attempt in this study to identify the same units each day or otherwise explicitly explore learning, but this is an important issue to be addressed in future studies. Studies of this array in monkeys (Linderman et al 2006, Dickey et al 2009) suggest that similar subsets of neurons can be tracked over longer times to reduce or eliminate performance variance resulting from changes in the ensemble membership. At least for able-bodied monkeys, performance improves for closed-loop kinematic cursor control when provided the same population of neurons day after day (Ganguly and Carmena 2009). Whether this will also work over the long term for humans with paralysis remains to be determined. Whereas such a strategy would aim to achieve more stable decoding by identifying stationary neural signals and employing longer learning periods, an alternative (or complementary) strategy would be to employ adaptive filters that identify and account for nonstationarities in the unit population or signal content. An adaptive approach could benefit from the findings reported here and elsewhere (Hochberg et al 2006, Kim et al 2008, 2011) that effective decoders can be trained with just a few minutes of intracortical data and that useful cursor control can be achieved without explicit practice even with new decoder coefficients. It remains a challenging priority to develop statistical approaches that can dynamically distinguish signal variability carrying information content from that arising from signal nonstationarity. Despite our decision to rebuild filters each day in this study’s protocol, it was nevertheless possible to quickly generate a decoding model that provided useful control day after day based only on mapping neural activity patterns occurring when the participant imagined that she was moving a cursor which was being moved automatically in a preprogrammed manner.

4.3. Comparison to extracortical sensors

Both spiking and field potentials are available as possible control signals for neural interface systems. Using spiking signals, which are only obtainable from a penetrating microelectrode, our participant continuously controlled a cursor and moved to targets, stopping and selecting at will. Intracortical electrodes can also record multi-unit activity (MUA) which may prove to be a powerful signal source (Stark and Abeles 2007). The rich information that can be derived from spiking ensembles makes these signals desirable for BCI control, but concerns about stability and reliability, as well as the durability of implanted systems, raise questions about their suitability for a practical and broadly applicable neural interface system. The ability to achieve comparable levels of control using EEG field potentials recorded from scalp electrodes would eliminate the need for implant of the sensor. Those electrodes produce a relatively impoverished neural signal, limited to summed lower frequency field potentials, that is subject to a number of artefact problems. The surface electrodes require daily application and contact quality can be unreliable within a session. Further, electrode application would require an able-bodied assistant if used by a person with tetraplegia. Intracranially-placed electrodes overcome many of these concerns. Non-penetrating intracranial electrodes, which record an intracranial EEG (ECoG), reduce artefacts and have better signal to noise ratio relative to scalp recordings. Nonpenetrating microwire arrays are another technology that can record from the cortical surface and may provide higher spatial and temporal resolution than typical ECoG grids (Kellis et al 2009, Leuthardt et al 2009). Unlike scalp EEG electrodes, implanted electrodes have the potential to become permanently placed and wireless, although this has only been achieved by one early BCI sensor technology (Brumberg et al 2010). However, those extracortical sensors require surgery with up to a 6.6% complication rate, including infection (2.5%) and bleeding (3.0%) (Van Gompel et al 2008), and have been in regular clinical use in the US for only up to 30 days at a time (the limit of current labeling) and only in persons with epilepsy and not paralysis. Recently-completed studies (in epileptic patients) of an intracranial responsive neurostimulator (RNS) using a small grid of ECoG electrodes have reported 3.7% hemorrhage and 4.7% infection rates for their fully-implanted system (Morrell 2010). By contrast, placement of an intracortical multielectrode array requires only a small cranial opening and a minimal area of sensor contact.

Our study also confirms that intracortical field potentials remain after longstanding tetraplegia, years after sensor implant. While these signals share features of ECoG signals— indeed, the intracortical sensor can capture EGoG-type signals with great fidelity—the capabilities of these LFP signals for cursor control in humans with paralysis have not been tested. However, in monkeys, intracortical LFPs contain information suitable for neural control (Stark et al 2009, Waldert et al 2009, Zhuang et al 2010) suggesting that these high resolution field potentials could provide an additional signal source that is possibly as useful as ECoG, or as a supplementary or replacement signal for intracortical spiking patterns.

At present, evaluation of the relative advantages or disadvantages amongst these various signal sources and alternative sensor types awaits further data, and it is our opinion that all sensor types (scalp, ECoG, and intracortical) will provide eventual BCI utility to the populations for whom these devices are being developed. The present results provide data supporting the proposal that penetrating microelectrode arrays of the type used here can be viable for years and are potentially useful long-term sensors.

4.4. Functional comparison to field-potential-based BCIs

At a functional level, S3’s performance appears to exceed that achieved by systems using extracortical field potentials, although differences in task design and performance metrics make comparisons difficult. Our S1 and A1 participants also achieved similarly fast and continuous target control (Hochberg et al 2006, Kim et al 2008, 2011) at earlier implant time points. A number of studies have measured pointing-task performance of BCI systems employing scalp or intracranial surface electrodes where cursor motion is controlled by field potentials in humans. One EEG study closest to the present study is a recent report by McFarland et al (2008), which arguably represents the current state-of-the art in EEG decoding to achieve real-time control of cursor trajectory with ‘click’ selection in humans. Those results illustrated that four able-bodied people and two people with spinal cord injury (but with retained arm control; Wolpaw and McFarland 2004), could perform a 2D selection task in a simplified four-target version of the Radial-8 task reported in the present study (McFarland et al 2008). In their task, one of four large on-screen targets was highlighted for selection and, after an instructed delay, the cursor at the center of the screen was released to EEG control to be moved to the target. When the cursor hit a target, the cursor and other three targets disappeared and the software entered a ‘confirmation’ epoch in which the user had 1.5 s to confirm selection of the target using the specified mental imagery; otherwise the target was deemed not selected. In either case the next instructed delay started 1 s later with the cursor automatically placed back at the center of the screen. Subjects in that study executed about 20–25 min of task time on each of three days (inferred from the text). Onscreen movement distances were comparable to the current Radial-8 task (35% and 37% of screen size, respectively) but used large targets (five times the area of our Radial-8 targets; and roughly 10 times, 2 times, and 0.72 times the screen area of our three mFitts1 targets) as well as a larger cursor (~0.25% of screen area, 11 times larger than the neural cursor used in the mFitts1 task and 50% larger than our Radial-8 cursor). With those dimensions the EEG task scored an Index of Difficulty (equation (1)) of 1.5 bits. That task difficulty contrasts with an ID of 2.3–2.6 bits for the current Radial-8 task and a range from 0.6 to 4.5 bits in our mFitts1 task. Comparing these IDs suggests that the EEG four-target task should have been less challenging than both of the tasks reported here, independent of the neural signal or signal decoding approach utilized. Despite the lower difficulty, the best 3-day mean performance reported by McFarland et al was 80% correct target selection (their figure 5(b)) compared with the mean 5-day performance of 91.9% and 94.9% for the two tasks in the present study. The subjects in the EEG study had 3%-15% errors in which a wrong target was actively selected (their figure 5(b)), compared with 0 such errors across five days of testing here, despite the fact that EEG control episodes were brief and involved preparatory periods. Results for the individual days were not reported in that EEG study so inter-day reliability cannot be compared to the results presented here.

Mean movement times (MT) reported in McFarland et al (2.5–3.5 s) were shorter than the target selection times (ST) reported here (8.7 s). Unlike the current study, MT in the EEG task excluded each trial’s reaction time (a 1 s instructed delay between target presentation and the start of cursor movement) and the constant 1.5 s selection period. When these are considered, the time expended to select targets is more comparable between studies. Trial durations in our study likely also increased because each false click stopped the cursor and required re-acceleration to get the cursor moving again. A third fundamental difference between selection times here and movement times in McFarland et al is their adoption of an algorithm to enforce inter-target movement times near the 2–3 s range using trial-by-trial adjustments to the X and Y cursor gains computed using information about the position of intended goals on each trial. Our current decoder has no information regarding intended selection locations and avoids ad-hoc gain adjustments, in favor of providing consistent speed response so that users can reliably command the cursor to accelerate, decelerate, and hover at any location on the screen. As a result, in the current study, distributions of commanded speeds indicated consistent kinematic control even without adhoc gains and in spite of new filters each day. While adequate, the peak neural cursor movement speeds in human BCIs are generally lower than the targeted training speeds and those achieved by able-bodied users. Optimizing speed models in future neural decoders could reduce target selection times substantially.

Another recent and relevant BCI study (Felton et al 2009) assessed 2D EEG-based cursor control in which dwell (500 ms) was used to selected targets in a four-location task (requiring alternating X or Y axis control) and in an eight-location task. In that study, 0 of 9 motor-impaired individuals achieved the criterion 80% target selection across five sessions on the four-location task. The eight-target task was accomplished to 80% by none of the motor-impaired subjects and only two of five able-bodied subjects.

The use of EEG signals to move cursors through 3D space has recently been reported in three able-bodied subjects and one individual with paraplegia due to spinal cord injury (T7) (McFarland et al 2010). The assessment task, which presented one target at a time in one of eight corners of a virtual 3D cube, revealed that the cursor reached the target within 7 s of onset in 93%, 78%, 76%, and 56% of all trials (for the four subjects, respectively). Although many cursor movements reached their target using trajectories limited by one of the walls (see supplemental material from that report), those results encourage further development of EEG interfaces. Indeed, that study found that ‘the first of the eight possible target locations reached by the cursor correlated strongly with the actual target location.’ In the absence of any reported hit rates or trajectory metrics, however, it is difficult to evaluate the utility of those cursor movements for BCI applications relative to the results reported here. The same difficulty applies to another recent study that examined the offline (open-loop) estimation of actual 3D hand kinematics using decoded EEG signals from able-bodied individuals (Bradberry et al 2009, 2010).

ECoG field potential recordings from the cortical surface in humans have also been decoded to reconstruct cursor trajectories with more rapid acquisition of a cursor task than with scalp EEG, but with more limited control than observed with intracortical recordings (Schalk et al 2007, 2008, Pistohl et al 2008; see review by Waldert et al 2009). Schalk et al (2008) demonstrated closed-loop cursor control by five humans using ECoG in a 2D task. Notably, participants were able-bodied epilepsy patients and only two of five performed the task with imagined rather than actual, performed movements. The cursor needed only to hit a target, requiring neither a dwell nor click, and, as with the above EEG studies, the cursor was not continuously controlled but was repositioned to the center of the workspace after the cursor encountered a target. Target hit rates were at 53% and 60% for the two subjects using imagined jaw, tongue and hand actions, well below the > 90% in the present study, even though targets were 10–20 times larger than in our study. The able-bodied subjects in that study achieved 58%, 69% and 73% hit rates. In another ECoG study, Pistohl et al (2007) reconstructed actual 2D hand movements performed by epilepsy patients by decoding subdural ECoG signals. Correlations between actual and reconstructed kinematics confirmed the availability of movement-related ECoG signals but were about half the correlation values reported when actual movements performed by monkeys were reconstructed using a decoder based on intracortical spiking activity (Wu et al 2006). These studies suggest that field potential sensors on the cortical surface can provide levels of cursor control that could potentially be useful for those unable to move. However, the quality of control appears to be considerably poorer and less reliable than that achieved by S3 in the current study. It is possible that effects of prior experience in the task, which may have been considerable for S3 (although the decoder was new each day), make comparisons to epilepsy patients who are new and brief BCI users (who have had recent surgery, and are typically on tapering doses of anti-epileptic medication) not that informative. In perhaps the only evaluation of ECoG BCI in (two) locked-in patients, no control was achieved (see Birbaumer et al 2008).

4.5. The need for, and use of, standard BCI metrics1

We applied a series of metrics that we believe to be practical measures to assess the value of BCI systems to users and to facilitate comparison across studies and different BCI systems. We quantified the tradeoff between item selection time and task difficulty using regression coefficients derived from our mFitts1 task. However, meaningful comparisons of BCI utility are complicated by the variety of participant populations, task types and measured variables across studies. Felton et al (2009) provides a useful case in point. That study quantified task difficulty in terms of bits and reported EEG-based cursor performance in terms MT-ID regression analysis similar to our ST-ID mFitts1 regression analysis. They found that 1D cursor movements to targets with a range of ID from 0.58 to 3.7 bits were achieved to criterion by 5 of 9 motor-impaired and eight of eight able-bodied subjects. Their MT-ID regression yielded average information throughput (regression coefficient b) on the 1D task between 0.340 and 0.676 bits per second (varying by subject and target direction). Ideally, these quantitative results would help predict performance of these 13 successful subjects when using the same EEG system in a slightly different task. However, when they extended the task from 1D to ‘orthogonal’ 2D control (constrained to horizontal and vertical movements) with a similar range of difficulty (0.58 and 2.59 bits), only four able-bodied subjects out of those 13 could even complete the task to criterion. Of those, only two were able to complete a diagonal 2D task of lower difficulty (ID 1.59–1.92, unconstrained simultaneous 2-axis control). Hence, information theoretic quantities characterizing performance on the 1D task did not predict the level of performance on (or even the ability to complete) the 2D tasks even with the same subjects using the same EEG BCI. This situation highlights the challenge of using MT-ID regression curves derived from one task to predict performance in another, even within the same study (and see Zhai 2004). These disparate results point out the need for standardized tasks and methodologies that could enable more robust and precise quantitative comparisons among neural interface systems while minimizing the need to interpret and compare results from dissimilar tasks. Although some of the tasks and metrics introduced here could provide a framework to improve comparisons across studies of emerging technologies, we support the consensus discussion from the 4th International BCI Conference at Asilomar where there was considerable support for the development of standardized BCI tasks by which progress in the field can be meaningfully and objectively reported.

4.6. Signal and sensor reliability

A useful neural interface will require a long lasting sensor that provides requisite signals for decades. It is widely held that tissue response will limit the duration of recording from penetrating electrode arrays and that spiking signals will be difficult to maintain for long periods of time because electrodes must remain close to neurons in order to detect their signals. Mechanical failure and device degradation are other problems that limit sensor longevity. Our results suggest that signal recording capabilities can persist for more than 2.7 years. We found that 78 channels showed LFP signals that modulated with intended action, a putative benchmark for the ability of an electrode to detect relevant neural signals at this interval. Of these functional electrodes, 72% (56) recorded spiking waveforms. Mean waveform amplitudes of the units recorded and decoded in this study were relatively small. For comparison, unit amplitudes were significantly smaller than those recorded a few months after array implantation (168.9 µV ± 156.6 µV, mean ± s.d, P < 0.001; Jan 23, 2006), although methodological differences influence this comparison. A thorough comparative longitudinal analysis is beyond the scope of the present report and will be the subject of a forthcoming study. Of the sorted units, 78% were directionally tuned, a feature relevant to creating the control signal as derived here. About 66% of electrodes that recorded directionally tuned neurons during the study failed to record discriminable spikes on at least one day, but the fact that many of these same electrodes recorded spikes on a subsequent day demonstrates their persisting capabilities. Impedance measures indicated that 7 electrodes were not functional at all. Three of these channels were nonfunctional since implant and thus never capable of recording, perhaps due to broken connections or broken electrodes as seen in previous, short-term, human studies with this device (House et al 2006). Eighteen electrodes demonstrated inconsistent measurements across days suggestive of poor pin-to-pad coupling at the pedestal connector. We anticipate that next generation wireless systems will eliminate these issues.

We observed LFP oscillations in the beta frequency band which subsided with the onset of movement. Our results are consistent with a number of studies in intact monkeys showing delay period beta activity and subsequent suppression just as movement-related spiking appears at the same site in MI (Sanes and Donoghue 1993, Murthy and Fetz 1996, Donoghue et al 1998). The exact functional significance of such beta-range modulation is not clear, but may be linked to mechanisms that shape the activity of neuronal ensembles prior to their engagement in movement (Baker et al 1999, van Wijk et al 2009, Spinks et al 2009). In the present study, beta-range LFP modulation was recorded on 24 electrodes even in the absence of spikes. LFP reflects synaptic currents summed across extracellular space potentially spanning up to 3 mm from the recording tip (Mitzdorf 1987, Juergens et al 1999; but see Katzner et al 2009) whereas action potentials reflect the activity only of units within a few hundred µm of the electrode tip (Henze et al 2000). Potential explanations for the absence of recorded spikes on otherwise functional electrodes could include neurodegeneration in close proximity to these electrodes (Edell et al 1992, Biran et al 2005, McConnell et al 2009), gliosis (Edell et al 1992, Polikov et al 2005, Griffith and Humphrey 2006, Turner et al 1999; but see Merrill and Tresco 2005), chronic inflammatory responses due to tethering or micro-motion of the array (Szarowski et al 2003, Biran et al 2005), or the random location of some tips too far from cells to record any spike potential. Alternatively, because of their higher frequency content (~250–7500 Hz), action potentials could be selectively attenuated by an increase in capacitive shunting that might result from putative degradation/delamination of the parylene insulation on the electrodes or silicone on the wire bundle (Nelson et al 2008, Nelson and Pouget 2010), loss of the Pt tip coating, or failure of wirebonds along the signal path. These are all viable issues that cannot be differentiated in the absence of additional device or histological evidence.

Although the current data do not allow us to determine the underlying reason why no spiking activity was recorded on some electrodes despite their ability to record the lower frequency, longer range LFP signals, electrode impedance measurements may prove informative. Astroglial scarring or other immune-mediated changes that might reduce recording quality have been associated with an increase in measured electrode impedances (Edell et al 1992, Merrill and Tresco 2005, Weiland and Anderson 2000). Degradation over time of wire bonds that attach wires to the array or to the pedestal connector could also increase impedance. In the current study, however, the majority of electrodes across the array showed a bias toward lower impedance from their original manufactured value while a few showed a small increase. This finding is consistent with a trend toward lower electrode impedance over the course of months observed for the similar (but not identical) ‘Utah’ array in cat cortex (Rousche and Normann 1998, Branner et al 2004). Lower impedances and smaller waveforms could be consistent with neuronal death or retraction of cellular matter from the electrodes over time (Edell et al 1992, Turner et al 1999, Merrill and Tresco 2005, Biran et al 2005, Polikov et al 2005, McConnell et al 2009). Evidence from Michigan probes implanted for 4 weeks in rats (Biran et al 2005) or 16 weeks in cat cortex has advanced the hypothesis that post-surgical glial scarring in the vicinity of electrode shafts may stabilize a few weeks after implant and be followed by a chronic neurodegeneration resulting in signal degradation (McConnell et al 2009). However, such a loss of neurons is not found in the vicinity of electrodes when a Utah array is implanted in cats (Rousche and Normann 1998, Branner et al 2004). To our knowledge, there have been no published studies reporting histological findings in monkeys after implantation of the Cyberkinetics/Blackrock or Utah microelectrode arrays. A study experimenting with different aspects of pneumatic insertion techniques for the Utah array in humans included one histology figure with electrode tracks and a neuronal stain but did not quantify neuronal density or displacement after their acute procedure (House et al 2006). In that report, neuronal soma density appears normal around several electrodes but displaced around several others. The lower impedances observed in the present study may be, alternatively, attributed to material degradation involving leakage or deterioration of insulation on the microelectrodes, sensor platform, or wire bundle to the pedestal connector. While more sophisticated impedance measurements such as impedance spectroscopy may one day help clarify such issues (Williams et al 2007), currently we cannot currently resolve these several alternative scenarios in-vivo. Nonetheless, the ability of this device to remain relatively viable for years is encouraging.

The potential longevity of this array is supported by other studies in humans and monkeys. Preclinical studies with this array showed chronic recording abilities for longer than a year in monkeys (Suner et al 2005). To date, four BrainGate pilot clinical trial participants have been implanted with this silicon microelectrode array. In addition to the present report with S3, spiking recordings were present over the many months of study in participants S1 and S2 with spinal cord injury (Hochberg et al 2006) and participant A1 with ALS (Kim et al 2008, 2011). However, these other participants exited the study, for various reasons, within 2.2 years of implant (S1: 15.9 months, personal reasons; S2: 25.9 months, failed pedestal connector; A1: 9.7 months, disease-related death). Although beyond the scope of this detailed prospective study, a retrospective longitudinal analysis across these multiple trial participants is in preparation.

In summary, it is difficult to compare our results with studies using different electrode geometries, materials, surgical implantation and stabilization methods, and species with vastly different brain sizes and immune responses (Kipke et al 2003, House et al 2006, Donoghue 2008). It is well established that penetration of the parenchyma by a rigid object will initiate a host of responses (Szarowski et al 2003, Merrill and Tresco 2005, Biran et al 2005) that could interfere with recording (Liu et al 1999, Turner et al 1999, Szarowski et al 2003, Moxon et al 2004, McConnell et al 2009). By contrast, other studies report successful longer-term recording in rats (Vetter et al 2004), guinea pigs (Williams et al 1999), cats (Liu et al 2006) and monkeys (Wessberg et al 2000, Dickey et al 2009, Suner et al 2005, Kruger et al 2010). The current findings reinforce the idea that it is possible to detect spiking signals with this array over years despite undoubted tissue damage, subsequent short and long term tissue reaction, and changes in electrode conductivity.

5. Conclusions

The current results indicate that spiking patterns derived from small intracortical microelectrode arrays can provide control signals to a person with tetraplegia that are useful in replicating actions of a computer mouse. Further, useful signals can be obtained each day more than 2.7 years after the implant is placed into the MI arm/hand area. This subject with tetraplegia and anarthria was able to use imagined hand movements to continuously move a cursor and execute clicks in order to select items anywhere on the computer monitor. In 10 assessment tasks employed across five consecutive days, 93.4% of targets were successfully selected. Although spiking units were recorded from nearly half of the electrodes, imperfections in the sensor’s integrity emerged. Nevertheless, the demonstration of functionally useful signals years after implantation provides important confirmation of the essential requirements of a BCI—that it continues to be functional despite a range of responses that may occur with any implanted device. These results, more than 12 years after the onset of tetraplegia, are presented solely from a single human subject with brainstem stroke so that any generalizations about these results to other conditions or for longer durations will require additional participants. However, they indicate that spiking neural signals from motor cortex can be harnessed to control devices and can provide mouse control operations years after injury or paralysis onset. These results help to demonstrate the durability of an implant which is undergoing continual improvement and can eventually be wireless and fully implanted (Song et al 2009). The current study motivates further development of neural prostheses which utilize chronic intracortical recording toward the control of practical assistive devices for people with paralysis or limb loss.