Clinical translation of a high performance neural prosthesis

Abstract

Neural prostheses have the potential to improve the quality of life of individuals with paralysis by directly mapping neural activity to limb and computer control signals. We translated a neural prosthetic system previously developed in animal model studies for use by two individuals with amyotrophic lateral sclerosis (ALS) implanted with intracortical microelectrode arrays. Measured more than a year post-implantation, the demonstrated neural cursor control has the highest published performance achieved by a person to date, more than double that of previous pilot clinical trial participants.

Clinical trials have recently demonstrated that people with paralysis can control computer cursors and/or robotic limbs using neural prostheses that interpret neural signals acquired from chronically implanted microelectrodes in motor cortex^1-3, scalp electroencephalography electrodes⁴, or cortical surface electrocorticography electrodes⁵. Early studies with nonhuman primates (NHPs)^6-8 guided the initial development of neural prostheses, and performance in NHPs continues to advance^9-15. In the clinical domain, increasing neural prosthetic performance is critical to move beyond proof-of-concept towards widespread adoption, and thus it is imperative to understand if and how these advances in animal models will translate to clinical populations. Through studies in NHPs, we recently developed a high-performance neural prosthesis that outperformed existing demonstrations of neural cursor control¹⁴. In this report we describe the translation of that system, as part of the BrainGate2 multi-site pilot clinical trial^*, for use by two individuals with amyotrophic lateral sclerosis (ALS) (participants T6 and T7). Using this system, the two participants achieved the highest neural cursor control performances by a person reported to date, as measured by the time required to acquire virtual targets.

To measure performance relative to that observed in previous BrainGate2 studies, participants T6 and T7 completed the same cursor control tasks previously completed by BrainGate2 participant S3. The S3 study¹⁶ represents the highest previously published cursor control performance in the BrainGate2 trial and, to our knowledge, represents the highest published human neural cursor control performance (see also Supplementary Table 1). Relative to the system used by S3, the current neural prosthesis (Fig. 1a) integrates design choices for four critical components that have demonstrated the potential to increase performance: (1) system architecture on which the neural prosthesis is implemented, (2) signal conditioning methods applied to measured electrophysiological signals, (3) decoding algorithm that maps neural activity to movement intentions, and (4) choice of behavioral actions associated with cursor control. Here we outline each of these components:

Figure 1. Comparison of neural control performance for participants S3, T6 and T7.

(a) Neural control paradigm. Broadband neural signals are recorded from an implanted microelectrode array. Signal conditioning extracts neural features, multi-unit spike counts and high-frequency local field potential power (HF-LFP), which are decoded to estimate intended cursor velocity. (b) “Radial-8” cursor trajectories: (top) mean trajectories of fifteen randomly selected trials per target and (bottom) ten randomly selected example trajectories per target. (c) Target acquisition time (mean ± 95% bootstrap confidence intervals) for “Radial-8” (S3: 278 trials, T6: 665 trials, T7: 358 trials). T6 and T7 acquisition times are significantly lower than S3 acquisition times (p < 10^-100, unpaired t-test). (d) “mFitts1” performance summary (S3: 248 trials, T6: 1072 trials, T7: 241 trials): index of difficulty was binned at 0.5 bit intervals and mean target acquisition times (mean ± 95% confidence intervals) were calculated. (e) Slope and (f) intercept for linear regression of index of difficulty vs. acquire time for “mFitts1” (95% bootstrap confidence intervals). T6 and T7 acquisition times are significantly lower than S3 acquisition times (p < 1×10^-5, analysis of covariance). T6 and T7 acquisition time includes the 500 dwell time used by these participants to select targets. Sessions shown are 224-256 and 349-387 days post-implantation for T6 and T7, respectively (T6 achieved comparable performance when tested 628, 791, and 798 days post-implantation (Supplementary Table 2)). S3 data in panels b-e re-plotted with permission from ¹⁶. Values plotted in this figure are summarized in Supplementary Table 3.

The current neural prosthesis was built on a real-time hardware and software platform designed to reduce latency and jitter from hundreds of milliseconds (S3 study)¹⁶ to 20 ± 6 ms. This advance was motivated by our previous NHP studies, which demonstrated that performance significantly increases with lower latency¹⁷ and used an earlier version of this platform to achieve high performance neural control^14,15.

The signal conditioning stage, which extracts neural spike event and local field potential features from recorded electrode voltage potentials in real-time, was modified to better adapt to the challenges of the clinical study environment (the participants' homes). These settings had considerably more electromagnetic noise than a controlled laboratory environment, potentially obscuring the features of interest. To compensate, common average referencing^1,18 and phase-preserving filtering^1,19,20 were employed to better separate neural spikes and local field potentials from background noise. In the previous study (S3)¹⁶, spikes were extracted by thresholding the spikeband (high-pass filtered) signal from each recording channel and sorting these waveforms into putative neural units. In contrast, the current study (T6 and T7) used simpler threshold crossing counts as neural features. These features demonstrated nearly equivalent decoding performance and the potential for greater stability in NHP studies^21,22 and were used in a previous clinical study¹.

In the current study we employ the Recalibrated Feedback Intention Trained Kalman Filter (ReFIT) decoding algorithm instead of the Velocity Kalman Filter (VKF) decoding algorithm previously used in the S3 study¹⁶. In a previous NHP study, we demonstrated that ReFIT outperforms VKF¹⁴.

The decoding algorithm is typically calibrated with data in which the participant is either imagining or attempting specific movements. For participant S3, data from imagined wrist movements were used¹⁶. In the current study, participants performed (T6) or attempted (T7) finger movements, instead of arm or wrist movements, based upon offline analysis demonstrating that finger movements were decoded with higher correlation (Supplementary Fig. 1).

T6 and T7 repeated two cursor control tasks previously completed by participant S3. The first task, “Radial-8”, presented fixed size targets that alternated between the center of the workspace and one of eight peripheral locations (see Fig. 1b and Supplementary Videos 1 & 2). To successfully acquire a target, T6 and T7 held the cursor on the target for 500 ms, in contrast to participant S3 who supplied a neurally-derived “click” signal when on target. T6 and T7 were able to successfully acquire and hold peripheral targets in less than half the time required by S3 (Fig. 1c and Supplementary Fig. 2). To assess the generalizability of this target acquisition time reduction, we also compared performance in the more complex “mFitts1” target acquisition task. This task used variable-sized targets that appeared at pseudorandom directions and distances from the previous target (see Supplementary Videos 1 & 2). Thus, “mFitts1” assessed target acquisition time across a range of target difficulties. Mean acquisition times suggest higher performance for T6 and T7 relative to S3 (Fig. 1d). As in previous studies¹⁶, “mFitts1” performance was summarized using a linear regression between target difficulty and acquisition time (see Fig. 1e,f and Supplementary Figures 3 & 4). The average regression intercepts for T6 and T7 were less than those of S3, suggesting a reduction in minimum acquisition time. More importantly, the regression slopes for T6 and T7 were less than those of S3 (95% bootstrap confidence intervals), indicating a significantly smaller acquisition time penalty as target difficulty is increased.

One potential real-world functional use of a neural prosthetic system for people with motor impairment is free-choice, free-paced typing²³. We tested our neural prosthetic system in such an application with participant T6 using the Dasher assistive typing interface²⁴. This software package (used with physical interfaces such as eye-gaze trackers) maps 2-dimensional cursor control onto discrete typing, and optimizes letter presentation based upon letter occurrence probabilities. T6, who had previous Dasher interface experience, was able to free-type 115 words in < 19 minutes, or approximately 6 words per minute (Supplementary Fig. 5, Supplementary Video 3).

The ReFIT decoding algorithm used by T6 and T7 was previously developed in NHP studies as refinements to the VKF decoding algorithm¹⁴, building upon previous VKF based studies with NHPs and people^16,25. To assess the specific contribution of the ReFIT decoder algorithm innovations on cursor control in the present study, we directly compared VKF and ReFIT in a block design in which participants were unaware of which control algorithm was employed (Fig. 2). Analogous to the previous NHP study and the “Radial-8” task, participants performed a center-out-and-back task. Task parameters (target size, target distance, and maximum trial duration) were chosen to facilitate a high success rate with VKF decoder control and were held constant for both VKF and ReFIT decoder control blocks. On average, ReFIT decoder algorithm based target acquisition was significantly faster than VKF based target acquisition (Fig. 2a). These findings are consistent with relative performance measured in a previous BrainGate2 study comparing similar algorithmic innovations²⁶.

Figure 2. Comparison of VKF and ReFIT neural control performance for T6 and T7.

(a) Target acquisition time, including the 500ms dwell time (performance for individual blocks shown in Supplementary Fig. 8a–b). (b–c) Two additional performance measures rely on the task movement axis, defined by the direct line path from the cursor position at the start of the trial to the target position. (b) Task direction change count is the number of times the cursor velocity in the task movement axis reversed signs. (c) Orthogonal direction change count is the number of times the cursor velocity orthogonal to the task movement axis reversed signs. Data in panels a–c represent 418 (VKF) and 377 (ReFIT) trials for T6, and 418 (VKF) and 555 (ReFIT) trials from T7. (d) Participant-reported difficulty scores within a range of 0 (effortless control) to 10 (impossible control). Data represent 4 comparison blocks (T6) and 5 comparison blocks (T7) for each decoder (individual ratings shown in Supplementary Fig. 8c). For all bar graphs the mean ± 95% bootstrap confidence intervals are shown and * indicates a significant difference between VKF and ReFIT (p<0.01, unpaired t-test). Values plotted in this figure are summarized in Supplementary Table 4. Channel counts used by each decoder are summarized in Supplementary Table 5.

Participant T7 made fewer movement direction and orthogonal direction changes (Fig. 2b, c) during ReFIT decoder control blocks, indicative of improved path quality. These differences were not significant for participant T6. However, subjective self-report suggests that T6 expended cognitive effort to compensate for relative difficulties in VKF control. After each block, participants rated control difficulty on a scale from 0 to 10, with 0 defined as no effort (cursor automatically moved to the target) and 10 defined as impossible. Participant T6 rated ReFIT decoder blocks as significantly lower difficulty (Fig. 2d). When asked for a qualitative description of the cursor control experience, T6 often commented that blocks rated with a higher level of difficulty required a cognitive strategy, such as visualizing a “ruler between the cursor and target” or imagining “a gravitational force around the target.” For VKF decoder blocks, T6 also mentioned trying to slow down cursor movements to avoid overshooting the target. In the NHP study, the presence of these overshoots were a differentiating feature between VKF and ReFIT, and contributed significantly to the longer acquisition times found with VKF. These results suggest that future animal and human studies should address the role of cognitive load and strategies in neural prosthetic control, perhaps by incorporating cognitive distractors in task designs and assessing cursor control performance degradation.

Beyond the described design choices in system architecture, algorithms, and behavioral instructions to the participants, other factors may contribute to the higher performance demonstrated by T6 and T7 relative to S3 and previous BrainGate2 study participants. One potential factor, differences in electrophysiological recording quality, was unlikely to contribute – recording quality of S3's array was similar to or better than T6's array (Supplementary Fig. 6). Another potential factor is differences in etiology of motor impairment – S3 has motor impairment due to brainstem stroke, whereas T6 and T7 were diagnosed with ALS. However, two additional BrainGate2 study participants with ALS, participants T1 and A1, achieved cursor control performance similar to or lower than that of participant S3 (Supplementary Table 1), thus it is unlikely that etiology alone could explain the difference. Another potential factor is the differing degrees of motor ability across participants. While all three participants had intact sensory pathways, S3 was not capable of functionally relevant arm movement. T7 retained limited and inconsistent finger movements and T6 was capable of dexterous finger movements. To control for differences in movement ability across participants, we conducted research sessions in which T6 actively suppressed her arm and hand movements. The resulting cursor control performance was similar to sessions in which movements were not suppressed (Supplementary Table 2 and Supplementary Fig. 7), suggesting that movement itself does not improve performance. Furthermore, although T7 had limited motor abilities relative to T6, both participants achieved similar cursor control performance. Lastly, for all three participants, multiple factors not explored in these studies might have increased performance, such as optimizing the overall amount of gain and smoothing applied to the cursor velocity. Of course, other potential differences between subjects could also lead to performance variation, as is the case with healthy subjects and skilled motor tasks (e.g., playing sports or musical instruments).

In this study, insights from NHP studies and recent clinical trials were applied to the design of a neural prosthesis and resulted in higher performance neural cursor control. The continued translation of neural prosthetic system studies from animal model to clinical research is vital for both advancing system performance and understanding real-world challenges. Incorporating advanced, high performance system design innovations, informed and iterated through clinical research, may bring neural prostheses closer to clinical utility for people with motor impairments.

Online Methods

Permission for these studies was granted by the US Food and Drug Administration (Investigational Device Exemption) and Institutional Review Boards of Stanford University, Partners Healthcare/Massachusetts General Hospital, and the Providence VA Medical Center. The three participants in this study, T6, T7, and S3, were enrolled in a pilot clinical trial of the BrainGate Neural Interface System (ClinicalTrials.gov Identifier: NCT00912041; information about the trial is available at www.clinicaltrials.gov/ct2/show/NCT00912041).

Participants

Participants were included following the inclusion and exclusion criteria of the clinical trial and informed consent was obtained from all participants. Consent to publish photos of participant T6 was obtained from participant T6.

Participant T6 is a right-handed woman, 51 years old at the start of the study, who was diagnosed with Amyotrophic Lateral Sclerosis (ALS) and had resultant motor impairment (functional rating scale (ALSFRS-R) measurement of 16). In Dec. 2012, a 96-channel intracortical silicon microelectrode array (1.0 mm electrode length, Blackrock Microsystems, Salt Lake City, UT) was implanted in the hand area of dominant motor cortex as previously described^16,27. T6 retained dexterous movements of the fingers and wrist (see Neural Control Algorithms). Data reported in this study are from T6's trial days 151, 161, 224, 228, 245, 256, 270, and 628. On all trial days, T6 achieved 94% or higher success rates for each control/task type reported. The typing video is from trial day 270.

The second study participant, T7, was a right-handed man, 54 years old at the time of the study, who was diagnosed with ALS and had resultant motor impairment (ALSFRS-R of 17). Participant T7 had two 96-channel intracortical silicon microelectrode arrays (1.5 mm electrode length, Blackrock Microsystems) implanted in the hand area of dominant motor cortex. T7 retained limited finger movements (see Neural Control Algorithms). Data reported are from T7's trial days 349, 363, 378, and 387. On all trial days, T7 achieved 86% or higher success rates for each control/task type reported.

All datasets collected were included in analyses except for those from three sessions with participant T6. One session was excluded because of administration of a sedating medication unrelated to the clinical trial prior to the session. The second session was excluded because participant T6 self-identified deviating from session instructions. The third session was excluded because participant T6 was asked to suppress movement during neural control but deviated from these session instructions.

Details of data collection from the third study participant, S3, are previously described¹⁶.

Task design

Participants engaged in center-out-and-back and random point-to-point target-acquisition tasks. Cursor velocities were controlled using a neural control algorithm (described below). Targets were acquired by moving the cursor and holding it over the target for 500 ms.

For the center-out-and-back tasks, fixed size targets alternated between the center of the screen and one of 8 locations on the periphery. The center-out-and-back task was set up with two different sets of target configurations. For performance comparisons to previous trial participant S3, target sizes and distances were scaled to match those described previously as task parameters for the “Radial-8” task¹. Radial-8 data presented are from test blocks T6-D224-B9/B10, T6-D228-B10, T6-D245-B23, T6-D256-B10, T7-363-B6/B8, T7-D387-B5/B7 (Participant-Trial Day-Block Number), composed of 665 and 358 trials from T6 and T7, respectively.

In order to compare Velocity Kalman Filter (VKF) and Recalibrated Feedback Intention Trained Kalman Filter (ReFIT) performance, target sizes were increased to permit a high success rate for both VKF and ReFIT based control to facilitate direct comparison of time to target and cursor trajectories. VKF data are from test blocks T6-D151-B6/B9/B13, T6-2013-D161-B3/B5, T7-D349-B17, T7-D363-B11, T7-D378-B105/B108/B110, composed of 418 and 418 trials from T6 and T7, respectively. ReFIT data are from test blocks T6-D151-B8/B12/B15, T6-D161-B4, T7-D349-B18, T7-D363-B12, T7-D378-B104/B106/B109, composed of 377 and 555 trials from T6 and T7, respectively. (Additional T6 data is summarized in Supplementary Fig. 8a, but was excluded from the analysis due to a failure to successfully acquire targets using the VKF.)

The random point-to-point target-acquisition task was matched to the “mFitts1” task previously run with participant S3¹. In this task, target size was chosen psuedorandomly from three fixed diameters. Subsequent target directions and distances were also chosen pseudorandomly, but were adjusted to ensure that targets appeared fully within the computer monitor workspace.

For the “mFitts1” task, Index of Difficulty for each trial was calculated as follows:

$$\mathit{\text{ID}}={\mathit{log}}_2((D/W)+1),$$

where D denotes distance from the cursor's start position to the closest edge of the acceptance window, W denotes the acceptance window (which is computed as the sum of the target diameter and the cursor radius). This formulation differs slightly from that specified in ¹ by including two provisions: first, a provision for the effective reduction in the distance to the target due to the size of the target and cursor, and second, a provision for the effective increase in the acceptance window due to the size of the cursor. This formulation allows for more robust comparison across a variety of task designs, as in ¹⁴. The “mFitts1” results for participant S3 in Fig. 1d-e, were calculated using the above formulation for comparison to the current study. These data are from test blocks T6-D224-B8/B12, T6-D228-B11, T6-D245-B21, T6-D256-B14/B17, T7-D363-B5/B7, T7-D387-B4/B6, composed of 1072 and 241 trials from T6 and T7, respectively.

Neural Control Algorithms

ReFIT and VKF decoding algorithms were previously described in detail in a nonhuman primate (NHP) study¹⁴. Neural control was continuous throughout each test block. The real-time input to both control algorithms were neural features described below. As in the NHP study, algorithm parameters were calibrated with training data collected during the same research session as evaluation of neural control performance. All training data were collected with a center-out-and-back target configuration. VKF training data were collected with either motor based control or automated open-loop control. ReFIT decoder training data were collected with VKF decoder based neural control.

In motor-based control (T6), the participant controlled the cursor's position by making physical movements with their index finger on a wireless touchpad (Magic Trackpad, Apple). T6 was not limited in her ability to span the workspace of the touchpad. Parameters of the VKF decoder in the T6 sessions reported in the main manuscript were calibrated with motor-based training data. For participant T7, early sessions attempted to use motor-based control to calibrate the VKF decoder. T7's movement ability was more limited, however, and maximally spanned a small region on the touchpad (∼1/8″-1/4″ wide), resulting in poor kinematic data. For all T7 sessions reported, VKF decoder parameters were instead calibrated to automated open-loop training data, in which the cursor was automatically moved directly towards the target by the computer (as in ¹⁶). The T6 sessions reported in the Supplementary Table 2 also used automated open-loop training data to calibrate VKF control algorithm parameters. Participants were asked to attempt (T7) or imagine (T6) finger-based movements for open-loop training. For both motor and open-loop control, the choice of finger-based movements was based upon considering various contralateral arm and hand movements and choosing movements that elicited the most neural activity (detailed in Supplementary Fig. 1). The previous NHP study¹⁴ associated cursor movements with compound arm movements by tracking end effector kinematics.

Neural control and task cuing were controlled by custom software run on the Simulink/xPC real-time platform (The Mathworks, Natick, MA), enabling millisecond-timing precision for all computations. Neural data collected by the NeuroPort System (Blackrock Microsystems, Salt Lake city, UT) were available to the real-time system with 5 ms latency. Visual presentation was provided by a computer via a custom low latency network software interface to Psychophysics Toolbox for Matlab^28,29 and an LCD monitor with a refresh rate of 120 Hz. Frame updates from the real-time system occurred on screen with a latency of approximately 13 ± 5 ms.

Neural Feature Extraction & Selection

The NeuroPort System applies an analog 0.3 Hz to 7.5 kHz band-pass filter to each neural channel and samples each channel at 30 kSamples per second. These broadband samples were processed via software on the Simulink/xPC real-time platform. The first step in this processing pipeline was to subtract a common average reference (CAR) from each channel^1,18,30. For each sample, CAR was calculated by taking the mean across all properly recorded neural channels (channels with outlier noise characteristics are not included). The CAR operation is intended to remove noise common across all recorded neural channels (e.g. Supplementary Fig. 2. in ¹⁸).

Band-pass filters split the signal into spike and high frequency local field potential (HF-LFP) bands. HF-LFPs were used in combination with spikes (hybrid decoding) as HF-LFPs can contain valuable kinematic information not always accessible if signals are restricted to recorded spikes¹⁸. To extract neural spiking activity a cascaded infinite impulse response (IIR) and finite impulse response (FIR) high-pass filter was applied. Although the IIR filter efficiently provides a sharp high-pass cutoff, it introduces a phase distortion that reduces action potential discriminability from noise. Thus, an asymmetric FIR filter was designed to reduce the impact of this phase distortion^1,19,20. A threshold detector was applied every millisecond to detect the presence of a putative neural spike. HF-LFP power features refer to the power within the 150-450 Hz band-pass filtered signal. T6 & T7 sessions reported in the main manuscript figures only used only spike-based features. The T6 sessions reported in Supplementary Table 2 and the Dasher control session used both spike and HF-LFP power features.

Neural features and spike thresholds were selected in session during parameter calibration. Neural features were sorted based upon their correlation to velocity control. Features were added one by one to the neural control algorithm and an offline assessment of directional control was used to predict online control quality (see Supplementary Modeling section from ¹⁴). This process was repeated for multiple spike detection threshold values, and the feature set selected was the one that achieved the minimum decoding error using the fewest number of channels. Neural feature values were updated with a fixed period during online control sessions. This period varied between 10-50 ms and was always matched between VKF and ReFIT.

For sessions with participant T7, neural features showed drifts in baseline rates over time. To account for these nonstationarities, baseline rates were computed de novo prior to each block, during a 30 second period in which the participant was asked simply to relax. In addition, baseline rates were continually updated over the course of the block using the following update equation:

$$b_{k,t}=b_{k,t-1}+(n_{k,t}-b_{k,t-1})\times \frac{1}{\tau },$$

where b_k,t denotes the estimated baseline rate of feature k at time t, n_k,t denotes the instantaneous rate for feature k at time t, and τ; is the adaptation time constant (30 sec). These estimated baseline rates were subtracted from the instantaneous rates before being passed into the decoding algorithms. Further, to minimize the number of terms that could be impacted by nonstationarities, the position feedback component of the ReFIT algorithm (detailed in ¹⁴) was removed for participant T7. Both modifications (baseline tracking and removal of position feedback) were also evaluated with participant T6 in the sessions reported in Supplementary Table 2.

Code Availability

Code may be made available upon request.