Virtual typing by people with tetraplegia using a self-calibrating intracortical brain-computer interface

Abstract

Brain-computer interfaces (BCIs) promise to restore independence for people with severe motor disabilities by translating decoded neural activity directly into the control of a computer. However, recorded neural signals are not stationary (that is, can change over time), degrading the quality of decoding. Requiring users to pause what they are doing whenever signals change to perform decoder recalibration routines is time-consuming and impractical for everyday use of BCIs. We demonstrate that signal nonstationarity in an intracortical BCI can be mitigated automatically in software, enabling long periods (hours to days) of self-paced point-and-click typing by people with tetraplegia, without degradation in neural control. Three key innovations were included in our approach: tracking the statistics of the neural activity during self-timed pauses in neural control, velocity bias correction during neural control, and periodically recalibrating the decoder using data acquired during typing by mapping neural activity to movement intentions that are inferred retrospectively based on the user’s self-selected targets. These methods, which can be extended to a variety of neurally controlled applications, advance the potential for intracortical BCIs to help restore independent communication and assistive device control for people with paralysis.

INTRODUCTION

Conventional assistive devices for people with severe motor disabilities are inherently limited, relying on (and thereby encumbering) residual motor function for their use. Brain-computer interfaces (BCIs) aim to provide a richer, more powerful command signal for assistive devices by decoding movement intentions in real time directly from neural activity (1–3). Intracortical BCIs have enabled people with tetraplegia to control cursors on computer screens, robotic and prosthetic arms, and other assistive devices by imagining moving their own arm (4–10).

A crucial component of a BCI is the decoder—an algorithm that estimates movement intention from neural activity (11, 12). The calibration of this decoder, which includes statistical modeling of the mapping from neural activity to movement intention, relies upon an accurate estimation of the person’s movement intention. In people with paralysis, movement intention cannot be measured directly from actual movement. Instead, it is typically estimated by asking the user to imagine that she or he is controlling the movement of an effector (for instance, a computer cursor or robotic arm) that is moved automatically to a series of presented visual targets (4–6). For continuous BCIs (ones that allow the person to control movements in continuous space), the user’s intended movement at each moment can be assumed to be a vector pointing from the current location of the effector toward the instructed target. This inferred movement intention can be regressed against the population of neural activity collected during the task to map the observed neural activity to the desired movements, thereby calibrating the decoder (4–6). After decoder calibration using this “open-loop” task (so-called because the user is not actually controlling the cursor), the decoder can be used for real-time, “closed-loop” neural control. In this mode, the user’s neural activity directly commands cursor movement with real-time feedback. By adding click decoding (6, 13) to this continuous velocity decoding and enabling text entry via a neurally controlled communication interface (14), people with tetraplegia should, in principle, be able to use any point-and-click computer application under neural control that able-bodied individuals can use with a point-and-click mouse.

Some intracortical BCI studies in monkeys have demonstrated stable neural recordings for long periods of time, permitting the use of fixed decoders (15–17). However, in many other intracortical BCI studies, particularly in humans (18), the relationship between movement intention and neural activity can change over the time scale of minutes, hours, or days because of physiological and/or recording nonstationarities in neural signals (17–23). If these nonstationarities are ignored, a decoder calibrated on data from an earlier time period will become un-calibrated, and the quality of neural control will degrade. If signal non-stationarity is expected to occur even occasionally, then successful clinical translation of BCIs requires that decoding methods are capable of compensating for it. One solution is recalibrating the decoder using data acquired during closed-loop neural control (“closed-loop calibration”) by mapping neural activity to movement intention, which can be inferred to be directly toward the presented target (7, 8, 24–28). However, even when using closed-loop decoder calibration, it would be cumbersome and disruptive to require the person to pause whatever practical BCI application he or she is using to perform a calibration task whenever signal nonstationarities occur. This strategy also limits the amount of data that can be used for decoder calibration to the amount of time the person is willing to perform the calibration task—and thereby limits the quality of the decoder [see, for example, (24)].

Instead, it would be desirable to calibrate the decoder using data collected during practical use of the BCI, in applications in which targets are not instructed. This would allow as much data as desired to be used to calibrate the decoder while eliminating the need to interrupt practical use of the BCI with explicit decoder calibration tasks. Because practical BCI applications do not have instructed targets (the user is free to select from many possible locations on-screen), it is not immediately evident how and whether movement intentions can be inferred during practical BCI use, and therefore how they could be mapped to neural signals to calibrate the decoder.

Here, we show that the decoder can be calibrated by mapping neural activity acquired during practical BCI use to movement intentions that are inferred retrospectively, based on the location of the user’s self-selected targets. We demonstrate that retrospective target inference (RTI)–based calibration produces a decoder that performs as well as a standard decoder that is calibrated using instructed targets, as measured by the typing speed and accuracy of four participants with tetraplegia, using each type of decoder in a point-and-click virtual keyboard. Combining RTI decoder calibration with two other self-calibration methods—correction of velocity bias during neural control and adaptive tracking of neural feature statistics during self-timed pauses—yielded stable neural control quality for long periods of self-paced BCI use, despite neural signal nonstationarities and without the need for disruptive recalibration tasks.

RESULTS

Four BrainGate participants with tetraplegia resulting from stroke (participants S3 and T2) or amyotrophic lateral sclerosis (ALS) (participants T6 and T7) were implanted with one to two 96-channel silicon microelectrode arrays in the hand/arm area of dominant motor cortex. Threshold-detected action potentials and/or the amount of power in the spike band were used as neural features for decoding. At the beginning of each 3- to 4-hour session (Fig. 1A), each participant performed a center-out task (the “standard calibration task”) to initialize and calibrate the Kalman filter (24). Once the standard closed-loop decoder calibration was completed, the task was switched to a neurally controlled point-and-click virtual QWERTY or radial keyboard communication interface, in which the participant was asked either to type standard words or phrases, to type their answers to questions posed by the clinical technician, or to type self-generated text. A preliminary version of these results was previously reported in abstract form (29, 30).

Fig. 1. Neural signal nonstationarities.

(A) Session timeline. Following an open-loop reference and decoder initialization block, a standard decoder was calibrated using several closed-loop center-out blocks, each lasting 3 to 5 min. Using the standard decoder, the participant then typed words, phrases, sentences, and/or paragraphs in either a QWERTY or a radial virtual keyboard. An RTI-based decoder was calibrated using only the neural data acquired during typing, and the participant continued typing using an RTI decoder until the end of the session. (B) Mean threshold crossing rates in each block of an example session (participant T7’s trial day 293), showing each channel that was used by the decoder for at least one block in the session. Blocks are labeled as in (A). Every third channel is labeled with its electrode number (in this session, 80 of 192 possible channels were selected for decoding in each block). For better visualization of the dynamic range of rate changes across blocks, rates are capped at 50 Hz (the highest actual whole-block baseline rate in this session was 68.3 Hz). (C) Directional tuning of the same channels in (B), obtained by regressing firing rates against target directions. Color represents the estimated PD, and the brightness of the color is proportional to the channel’s normalized modulation index. (The same PDs are shown in polar coordinates in fig. S3.) (D) The difference between each unit’s baseline rate in each block (“actual”) and the rate used by the decoder in that block (“used”; that is, the previous block’s baseline rate) is plotted against the difference between that unit’s baseline rate in that block and its rate in the first block (“original”), which would have been used by the decoder for the whole session if features were not being updated. (E) The angular difference between each unit’s measured PD in each block and the tuning model used by the decoder for that unit in that block (“actual vs. used”) is plotted against the angular difference between the measured PD in that block and in the first block (“actual vs. original”), which would have been the tuning model if the decoder were never recalibrated after the first block.

Mitigating nonstationarities in baseline rates

Cosine tuning curves have three characteristics that can theoretically change over time: the baseline rate, the preferred direction (PD), and the modulation depth (fig. S1). A shift in a unit’s baseline rate (fig. S1B), if ignored, would bias cursor motion toward (or opposite) that unit’s directional contribution to the decoder (31). To illustrate the prevalence and magnitude of baseline rate nonstationarities, Fig. 1B shows the baseline rates of all units used in the decoder in each block of a typical session from participant T7 (trial day 293).

To verify the utility of updating mean rate estimates, the difference between the mean rate of each unit in each block and its mean rate in the previous block (actual vs. used) was compared to the difference that would have been obtained if the decoder had not adjusted for mean rate nonstationarities and instead had used the mean rates from the first block for the entire session (actual vs. original) (Fig. 1C). In this session, the average (across units and blocks) actual vs. used mean rate difference (1.80 ± 0.12 Hz) was significantly smaller than the average actual vs. original mean rate difference (3.50 ± 0.22 Hz) (paired t test: t = 8.78, df = 639, P < 10⁻¹⁷). Thus, the previous block’s mean rates provided a significantly better estimate of the current baseline rates than did the original block’s mean rates, supporting the use of the more recent estimates by the decoder. In the self-paced typing sessions, the intervals of time between blocks of neural control could become arbitrarily long, increasing the chances of large baseline shifts between blocks as well (fig. S2). Thus, in these sessions, the estimate of the baseline rates was iteratively updated in real time between periods of neural control and frozen at the onset of the next typing block.

Because baseline rates can also be unstable during blocks of neural control, it would at first seem desirable to iteratively update the estimate of each feature’s mean rate at faster time scales during neural control as well. However, if the time constant of mean estimation is short, then mean subtraction can dampen the effects of actual neural modulation related to voluntary movement intent and cause a subsequent bias opposite the intended movement. If the time constant is sufficiently long not to cause a bias, then mean subtraction takes longer to counteract biases resulting from actual signal nonstationarities. As soon as a bias appears, the user would then have to counteract the bias by modulating their neural activity, but then the neural activity resulting from counteracting the bias would enter into the estimation of the new baseline rates. Thus, the user would have to keep modulating their neural activity to counteract the bias; that is, the bias would effectively never disappear. Instead of seeking a time constant that minimizes the negative effects of each extreme, our solution to within-block nonstationarities was to iteratively estimate and subtract out the direction and magnitude of the cursor velocity bias itself (Fig. 2). Specifically, the bias estimate was initialized to [0, 0] at the start of each block, and updated iteratively by computing an exponentially weighted running mean of all decoded velocities whose speeds exceeded a predefined threshold (Fig. 2, A and B) that included high-speed movements in the direction of the bias but excluded low-speed movements against the bias direction. This estimated bias was subtracted from the decoded velocity at each moment to command subsequent cursor movements (Fig. 2C).

Fig. 2. Bias correction.

(A) Representative example of bias estimation, from 800 s into the first typing block of participant T7’s first self-paced typing session (trial day 293). At each moment in time, the direction and magnitude of the velocity bias (red arrow) were estimated by computing an exponentially weighted running mean of all decoded velocities (grayscale dots) whose speeds exceeded the 66th centile of the speed distribution (red dashed circle) computed from the previous filter build. This threshold was empirically found to be high enough to exclude low-velocity movements generated in an effort to counteract existing biases. The shading of each dot represents time, with darker dots occurring closer to the present moment [the end of the highlighted period in (C)]. (B) Effect of bias correction at the same moment displayed in (A). The location of the cursor is represented as a black dot. The location of the (retrospectively inferred) target is a blue dot. The red arrow represents the estimated bias at that moment in time [same as in (A)]. The purple arrow indicates the decoded cursor velocity at that moment before bias correction. The blue arrow indicates the bias-corrected velocity. (C) Effect of bias correction on this entire block of typing. (Top) Velocity traces with the estimated bias (black traces) added in. The gray box indicates the time interval when individual velocity samples are displayed in (A). (Bottom) Actual cursor velocities that occurred in session, bias correction having been continuously applied in real time.

Decoder calibration using RTI

Shifts in PDs, if ignored, can result in a rotational perturbation in cursor motion (fig. S1C) (28) or a “shearing” effect on the cursor’s velocity toward and opposite its contribution to the decoder (fig. S1D). The PDs across blocks from participant T7’s trial day 293 illustrate the prevalence and magnitude of PD nonstationarities (Fig. 1C and fig. S3). To verify that the measured PD of each unit in each block is closer to the model used for that unit in the updated decoder (represented by the two-dimensional vector in the corresponding row of the H matrix; see Materials and Methods) than it would have been with the original decoder in the session, we compared the angular difference between the actual vs. the used PDs of all the units used in the decoder in each block to the angular differences that would have been obtained if the decoder were never recalibrated after the first block (actual vs. original PDs). In this example session, the mean actual vs. used PD angle difference (34.1 ± 1.5 Hz) was significantly smaller than the mean actual vs. original PD angle difference (45.5 ± 1.7 Hz) (paired t test: t = 7.67, df = 637, P < 10⁻¹³) (Fig. 1E). Thus, the current PDs are much closer to their modeled PDs in the updated decoders than they are to the original PDs in the session; that is, PDs tend to change gradually over time, supporting the utility of periodic decoder recalibration.

To keep the decoder calibrated during practical BCI use, we introduced a method by which closed-loop calibration can be applied even when the person selects his or her own targets among an unlimited array of possibilities: RTI-based decoder calibration. In RTI calibration, the users’ intended directions at each moment were retrospectively inferred on the basis of their subsequently selected targets (Fig. 3A), using some simple heuristics to determine which parts of each trajectory were most likely to correspond to true movement intent (see Materials and Methods). As with standard closed-loop calibration, these assumed movement intentions were then mapped to the neural activity recorded during typing. To verify that RTI decoder calibration preserves neural control quality despite nonstationarities in PDs, and that our heuristic assumptions about the person’s intended movement directions and times work as well as in a standard calibration task with presented targets, we compared the quality of neural control during typing using an RTI decoder versus using a standard decoder. After two to four blocks of neural typing using the standard decoder (mean, 16.4 min; range, 5.5 to 45.8 min), the data acquired during typing were used to calibrate an RTI decoder. Then, the participant was asked to type for the remainder of the session using an RTI decoder for neural control (mean, 21.0 min; range, 2.2 to 94.5 min). Across 19 sessions from all four participants, the quality of neural control, as measured by the number of correct characters typed per minute (CCPM), was as high using the RTI decoder (mean, 12.0 CCPM) as the standard decoder (11.4 CCPM); the mean within-session difference was 0.60 ± 0.58 (SEM) CCPM. Furthermore, session by session, the CCPM using the RTI decoder correlated significantly with the CCPM using the standard decoder (Pearson’s correlation coefficient r = 0.90; P < 10⁻⁶, based on a null distribution obtained by shuffling the session pairings 1 million times) (Fig. 3B). Thus, RTI calibration yields decoders that maintain neural control in each session for each participant at the same level as standard decoder calibration using explicitly prescribed targets.

Fig. 3. RTI decoder calibration.

(A) To obtain a tuning model from data acquired during neural control in a practical BCI application, such as a virtual keyboard (14), the user’s intended movement direction is retrospectively inferred to be directly toward the next selected target (white arrows). The white curve reflects the portion of the preceding cursor trajectory assumed to result from the person’s movement intent and is used toward RTI decoder calibration. The red dashed segments of the trajectory are excluded from decoder calibration. The intended direction vectors are regressed against the corresponding neural activity to calibrate the RTI decoder. (B) Typing performance using the RTI decoder versus the standard decoder, measured using the number of CCPM. Data are from 19 sessions across four participants, including 5 self-paced typing sessions (3 from participant T6 and 2 from participant T7, shown in unfilled markers). The within-session correlation coefficient r and its corresponding P value are shown in plot. (C) Typing performance using the RTI decoder versus the standard decoder, measured using the number of CSPM. For the radial keyboard, this metric can be translated into extrapolated bitrate (eBR = CSPM × log₂(N − 1)/60, where N = 8 targets). The eBR scale only applies to the radial keyboard sessions, not to the two sessions in which the QWERTY keyboard was used (*); for the QWERTY keyboard, eBR could not be computed easily because of the large variability in the size of the targets. Within-session correlation coefficient and P value are shown in plot. P values in (B) and (C) were obtained by comparing the measured value to a null distribution obtained by shuffling the pairings 1,000,000 times.

Because CCPM reflects the net typing rate, it is a practical measure of the BCI’s utility for the participant. However, CCPM does not translate directly into quality of neural control because the virtual keyboards used here permit word prediction, each selection of a word in the radial keyboard requires the selection of the right arrow, and words that are not in the dictionary require two selections per letter (14). Thus, we also computed the number of correct selections per minute (CSPM), regardless of the number of characters that resulted from those selections (Fig. 3C). For the radial keyboard, in which all of the possible targets have the same size, this metric can also be translated into extrapolated bitrate (eBR), the number of bits of information conveyed per second (17, 32) “extrapolated” to a virtual keyboard. Using CSPM, the quality of neural control was again as high using the RTI decoder (mean, 15.0 CSPM) as using the standard decoder (14.6 CSPM); the mean within-session difference was 0.41 ± 0.48 (SEM) CSPM. Session by session, the CSPM using the RTI decoder was significantly correlated with the CSPM using the standard decoder (Pearson’s correlation coefficient r = 0.96; P < 10⁻⁶). Again, these results suggest that RTI decoders perform just as well as standard decoders at maintaining typing rates while eliminating the need for disruptive calibration tasks with prescribed targets.

To verify the utility of RTI calibration in adapting the decoder to known PD shifts, we created a set of 80 simulated neurons with known PDs and a decoder whose model initially matched those PDs. Then, we shifted the PDs of 25, 50, 75, or 100% of the simulated neurons by random amounts in random directions, and tested whether RTI calibration was able to bring the decoder’s observation model closer to the actual changed PDs and rescue simulated neural control. In the 25 and 50% random perturbation conditions, RTI was always (across 20 runs of each) able to closely match the model to the shifted PDs and rescue neural control within one to two simulated 3-min blocks (fig. S4). In the 75% perturbation condition, RTI successfully rescued the decoder and simulated control in 17 of the 20 runs. In the 100% random condition, RTI usually failed, as expected; however, in 2 of the 20 runs, the cursor was able to get to four of the targets by chance in the first block (the perturbations, although large for each individual simulated neuron, happened to offset each other enough to result in a fairly low decode error). These target acquisitions allowed RTI calibration to take place and initiated a feedback loop that brought the model estimate closer to the true PDs after the first calibration, thereby allowing more targets to be acquired in the next block and allowing the next RTI calibration to improve the model estimate further, and so on; within four simulated blocks, this cascade resulted in perfect simulated control. Thus, RTI calibration robustly tracks shifting PDs in small perturbation conditions, and sometimes even in moderately large perturbation conditions, as long as at least some targets are able to be reached. Note that the simulation had no capacity for error correction or local learning (28), but instead always “aimed” directly toward the target. Thus, a BCI user might be even better at compensating for a poor decoder by using these additional strategies.

Self-paced typing

To test whether the combination of mean tracking, bias correction, and RTI decoder calibration allows for stable neural decoding for long periods of practical BCI use, we ran five longer (1 to 2+ hours) self-paced typing sessions with the radial keyboard, three with participant T6 (Fig. 4 and movie S1) and two with participant T7 (fig. S5). (Participants S3 and T2 were no longer in the trial.) After the standard decoder calibration procedure, the participants typed for as long as they wanted, pausing and unpausing the BCI whenever they desired by selecting a specific sequence of two wedges. In each of these sessions, typing rates remained as high as they started throughout the entire period of self-paced typing: there was no significant decay in typing rate over time, as measured by a linear regression between time and CSPM (Fig. 5, A and B).

Fig. 4. Example of self-paced typing session for participant T6 on trial day 668.

In the self-paced typing sessions, participants were able to pause typing when they wanted by selecting the right arrow in the radial keyboard and then the wedge containing “PAUSE.” Each pause initiated a file break and RTI decoder build, and then neural control was restored to allow the user to resume when desired, by selecting the right arrow and “UNPAUSE.” Until the unpause sequence was selected, no other wedges were active. (A) Photograph of the radial keyboard interface (left) with the PAUSE button about to be selected, and the notebook showing the text typed in this session (right). (B) Length of each block of typing, the number of CCPM and CSPM in that block, and the text entered (the vertical lines in the text of the last block indicate an “ENTER” character, which starts a new paragraph). In this session, an RTI decoder was calibrated during each of the self-timed pauses using all typing data acquired up to that point, except the last RTI decoder used only the previous three typing blocks. Note that the fastest typing rate in this session was achieved in the last typing block. The blurred words, represented by underscores in B, were redacted at the request of the participant.

Fig. 5. All self-paced typing sessions: Summary of typing rates over time.

Each session is depicted in a single hue, with darker bars indicating the time and duration of the self-paced blocks of typing in which a standard decoder was used, and lighter bars indicating the blocks in which an RTI decoder was used. Self-paced blocks of typing using an RTI decoder in that same session are depicted in bright colored bars of the same hue. (A) Three self-paced typing sessions for participant T6. (B) Two self-paced typing sessions for participant T7. (C) One session with T6 in which bias correction, feature tracking, and RTI decoder calibration were all turned off. Linear regression between time and CSPM: Pearson’s correlation coefficient r = −0.85, P < 0.001. (D) One session with T7 in which bias correction, feature tracking, and RTI decoder calibration were all turned off (black bars). Linear regression between time and CSPM: r = −0.87, P < 10⁻⁶. In this session, T7 was unable to type at all in the third block; this occurred early enough in the session to test whether neural control could be rescued by reinstating the self-calibration methods (brackets). In the first and third rescue blocks, both bias correction and interblock feature tracking were reinstated, but the standard decoder was used (dark green bars). In the second rescue block (light green bar), an RTI decoder was used that was calibrated using data from the first rescue block.

To verify that these self-calibration methods were necessary for the long-term stability of neural control, we also performed a session with each participant in which mean tracking, bias correction, and RTI decoder calibration were all turned off (Fig. 5, C and D). In both of these control sessions, the typing rates declined significantly over the same 1- to 2-hour time scales, as measured by a linear regression between time and CSPM (T6: r = −0.85, P < 0.001; T7: r = −0.87, P < 10⁻⁶; Pearson’s correlation coefficient).

Neural typing rapidly deteriorated with the self-calibration methods turned off for participant T7. We therefore used the remaining time in the session to test whether neural control could be rescued by reinstating them (Fig. 5D). In the first rescue block, both interblock mean tracking and bias correction were reinstated. Typing was again possible (CCPM, 5.41; CSPM, 11.6), and, in fact, the typing rate exceeded that of the first typing block (CCPM, 3.5; CSPM, 5.4), suggesting that neural control was already impaired by nonstationarities in the minutes between the end of the standard decoder calibration period and the end of the first typing block when the self-calibration methods were turned off. Then, an RTI decoder was calibrated using only the data collected in the first rescue block, and this decoder was used in the second rescue block; the typing rate remained high (CCPM, 8.6; CSPM, 11.1). In the final rescue block, the standard decoder was used again, with bias correction and interblock mean tracking still on, and the typing rate still remained high (CCPM, 10.0; CSPM, 13.5). Thus, in all three of these “rescue” blocks, neural control was indeed rescued by reinstating the self-calibration methods. Furthermore, mean tracking and bias correction in the first rescue block were sufficient to bring neural control back to a level that allowed RTI calibration to function properly, as judged by its ability to maintain neural control in the second rescue block.

Multiday self-calibration

Last, we tested whether the combination of mean tracking, bias correction, and RTI decoder calibration allowed for stable neural decoding across multiple days of practical BCI use. Participant T6 (participants T2, S3, and T7 were no longer in the trial) free-typed using the BrainGate BCI, pausing and unpausing the system when she desired, across six sessions spanning 42 days, without the need for any instructed-target calibration tasks after the first decoder was initialized on the first day of the series (Fig. 6). Typing speeds were maintained across the series of sessions at levels similar to or higher than the first block of the first session.

Fig. 6. Self-calibration across multiple sessions for participant T6.

Data are in the same format as Fig. 5 (the dark bar indicates the block in which a standard decoder was used, and the light bars indicate blocks in which an RTI decoder was used). The dots above the bars and the diamonds below the bars indicate typing periods during which the cursor’s speed gain or click decoder threshold, respectively, were manually adjusted by the technician; in the last two sessions of this series, there was no technician intervention once typing started. Using the self-paced radial keyboard, participant T6 typed whatever she wished across six sessions spanning 42 days, pausing and unpausing the BCI whenever she wanted, without needing to perform any explicit calibration tasks after the first day. The first block of the first session in this series (participant T6’s trial day 759) used a standard decoder calibrated earlier that day; after that, an RTI decoder was calibrated during every user-timed pause in neural control using the data acquired during the previous 20 to 60 min of typing. Each session after the first was initialized with the previous session’s last RTI decoder.

DISCUSSION

Neural signal nonstationarity (variation over time) is a major challenge for the translation of intracortical BCIs. Beyond physiological dynamics and plasticity, (apparent) changes in directional tuning and baseline rates can be large and sudden, likely largely attributable to non-physiological events, such as environmental noise and movement of the brain relative to the electrode (18). There are important differences in methodology between nonhuman primate (NHP) and human recordings that might contribute to these events being more numerous and problematic in humans (15–23). For example, NHP electrode arrays have 3-cm wire bundles, whereas human electrode arrays have 13-cm wire bundles, making them more susceptible to picking up noise before amplification; human brains are larger, as is the intracranial (epidural, subdural, and subarachnoid) space, particularly in older humans, and therefore the human brain moves more within the skull relative to the NHP brain; and most NHP recordings are conducted in a controlled, noise-reduced laboratory setting, whereas our human intra-cortical recordings are conducted in the participant’s home with many potential sources of distraction and environmental noise—a deliberate choice, because that is the setting in which BCIs will ultimately be used.

To overcome the problems caused by neural signal nonstationarities in a practical BCI use setting, we have devised and implemented a method for RTI-based decoder calibration, which maps neural activity to movement intentions that are inferred retrospectively from the user’s self-selected targets. RTI decoders performed as well as standard decoders calibrated using explicit routines with predefined targets. With the combination of RTI calibration, adaptive feature mean tracking during pauses in neural control, and velocity bias correction during neural control, participants were able to use a self-paced point-and-click communication BCI for long periods of time (~2 hours on multiple days) without degradation in neural control and without the need for disruptive calibration routines or technician intervention. In sessions with the three self-calibration methods turned off, neural control declined significantly over ~2-hour time scales.

In one session in which the ability to type rapidly disappeared with the self-calibration methods turned off, reinstating them rescued neural control and restored the person’s ability to type. In this session, the rescue block using the RTI decoder did not have higher performance than the two rescue blocks using the standard decoder that had been calibrated an hour earlier, suggesting that nonstationarities in PDs (which are mitigated by RTI calibration) did not have as catastrophic an effect on performance at these time scales as nonstationarities in baseline rates (which are mitigated by bias correction and between-block mean tracking). This result might partially be explained by the fact that PD shifts have an upper bound (180°), and perturbations in decoded PDs can partially be accommodated physiologically through re-aiming and neural plasticity (15, 28), whereas baseline shifts are unbounded and thus might not always be possible to accommodate. Although freezing the decoder’s tuning model and allowing the user to compensate for shifting PDs is a possible option, it would be preferable to relieve the user of this burden by instead adapting the decoder to shifting PDs using RTI.

The fastest point-and-click BCI-enabled typing rate previously reported by a person with tetraplegia was roughly 10 CCPM (14), sustained for a few minutes at a time, using a decoder that was calibrated using an explicit calibration task at the start of each session. Here, typing rates at least this fast, and up to ~2.5-fold faster, were sustained for much longer periods (1 to 2 hours across multiple days) without the need for intervening calibration tasks. These methods can be extended to other types of decoding algorithms and thus should provide for stable control as algorithms for neural decoding continue to evolve. They can also be extended to other point-and-click–based neurally controlled computer applications and could thereby potentially allow a BCI user to control a computer indefinitely without the need for disruptive calibration routines, an essential goal for the translation of current investigational BCIs to real-world application. With additional constraints on the assumptions of the intended movements, similar approaches could also conceivably be extended to multidimensional neural control, such as prosthetic and robotic arm reach and grasp (7, 8) or functional electrical stimulation of the person’s own limbs (33–35).

There are likely to be additional refinements that will further enhance the performance of RTI calibration. For example, the person’s intended cursor movement direction at each moment was assumed to have been directly toward the next selected target (7, 8, 24–28). However, more sophisticated methods could be incorporated that improve the estimate of the person’s true aiming direction by, for example, iteratively recomputing the aiming direction and tuning models until they converge (28) or estimating and taking into account the person’s internal model of the cursor’s expected behavior under neural control (36, 37). Also, the selection of particular segments of data to be included in filter calibration was based on a few simple heuristics in our study, but could conceivably be refined by taking into account information that can be inferred from the neural signals about the person’s attentional state, intention to move, or error signals in local field potentials (38–43). Finally, the time constants and other parameters determining the behavior of each of our methods have been hard-coded to values that were anecdotally found to work well across many sessions and several participants. Although these techniques are relatively robust to precise parameter settings, it would be beneficial to create an objective, data-driven method by which they can automatically be set for an individual user, perhaps each day, based on the recent history of the specific kind of nonstationarity that each method is intended to mitigate. Similar methods and refinements could also be extended to the self-calibration of the click decoder.

RTI provides an unobtrusive way to reap the benefits of adaptive decoder calibration, allowing as much data as desired (collected during ongoing, practical BCI use) to be added to the decoding model. When the neural signals are stable over long periods of time, continually adding more calibration data would improve the accuracy of the tuning model and enable more complex tuning models to be used without as much risk of overfitting. When the neural signals are not stable, the decoder could be continually recalibrated using only the most recent and relevant closed-loop data. This process could be aided by tracking nonstationarities in the recorded signals and selecting the optimal window and weighting of calibration data based on the history of each unit’s activity. Together, these self-calibration methods should allow the tuning model to remain accurate and up-to-date indefinitely during ongoing, practical BCI use, helping to bring intracortical BCIs closer to extended clinical utility.

MATERIALS AND METHODS

Study design

Permission for these studies was granted by the U.S. Food and Drug Administration (Investigational Device Exemption) and the Partners Healthcare/Massachusetts General Hospital (participants S3 and T2), Providence VA Medical Center (participant T7), or Stanford University (participant T6) Institutional Review Board. The four participants in this study were S3, a woman with tetraplegia and anarthria resulting from brainstem stroke; T2, a man with tetraplegia and anarthria resulting from brainstem stroke; T6, a woman diagnosed with ALS; and T7, a man with ALS (table S1). Each was enrolled in a pilot clinical trial of the BrainGate2 Neural Interface System (NCT00912041). They were implanted with one or two 96-channel silicon microelectrode arrays (Blackrock Microsystems) in the dominant hand/arm knob area of motor cortex (44), as previously described (4, 6). All four participants contributed sessions to the RTI calibration comparisons; participants T6 and T7 additionally contributed to the self-paced typing sessions (which occurred after participants S3 and T2 exited the trial); and participant T6 additionally contributed to the multiple-day self-calibration sessions (which occurred after participants S3, T2, and T7 exited the trial). The participants’ residual movement abilities varied widely.

The questions asked in this study were whether RTI decoder calibration worked as well as standard decoder calibration, and whether the suite of three self-calibration methods can maintain neural control for long periods of self-paced, practical BCI use. Neural control was assessed by CCPM or CSPM. Participants (but not the technicians running the sessions) were blinded as to whether each self-calibration method was turned on or off in each self-paced typing session. Because of the nature of the clinical trial, the frequency of research sessions that each participant contributed to this study depended on the amount of session time available relative to other ongoing BrainGate research sessions (each participated in one to three sessions per week, of which the current study was one of several concurrent studies). For the RTI versus standard decoder performance comparisons, all sessions were included in which at least one block of typing occurred using each type of decoder. Sessions of a given type began when the necessary software development was completed, and ended for each participant when he or she exited the BrainGate clinical trial or when at least two self-paced typing sessions and one control session were collected for that participant (whichever occurred first). The number of sessions for each participant thus varied across session types (table S2). For the multiday self-calibration series with participant T6, data collection ended when click decoding became unreliable, causing typing rate to decline.

Research session design

In each 3- to 4-hour recording session, neural signals were common-average–referenced (41) (fig. S6) and noncausally filtered (45), and threshold-detected action potentials and (in participant T6) the amount of power in the spike band were computed in each 20- to 100-ms bin for each channel. To calibrate the “standard” directional and click decoders, mean-subtracted neural features were mapped to movement intentions that were inferred to be directly toward the next presented target in an open-loop and then closed-loop center-out-back task (24). In closed-loop neural control, intended movements were decoded from the incoming neural features and translated in real time into the movement of the cursor using a steady-state Kalman filter (5, 46, 47). In most sessions, a linear discriminant analysis classifier running in parallel with the Kalman filter was used to decode neural cursor “clicks” (6, 13). Signal acquisition, feature preprocessing, decoder calibration, and session design are in Supplementary Materials and Methods.

After the standard decoder was calibrated, the task was switched to a neurally controlled point-and-click QWERTY or radial communication interface (14), initially using the standard decoder for neural control. Once sufficient data were acquired in the typing task, an RTI decoder was calibrated on the neural data acquired during typing. Then, the person was asked to continue typing, now using this RTI decoder for neural control. The RTI decoder was updated after every block using a sliding window of data spanning the most recent 20 min to 1 hour of free-typing.

In later sessions with participants T6 and T7, sessions began with the standard calibration tasks, and then the technician initiated the self-paced typing task, allowing the participant to control the pace of the rest of the session. The participant was able to pause typing by selecting the right arrow and then the wedge containing the function “PAUSE.” Each pause initiated a file break and an RTI decoder calibration, and then the participant could resume typing when she or he was ready by selecting the right arrow and then the wedge containing the function “UNPAUSE.” When all three self-calibration methods were turned off, as in the other self-paced typing sessions, the participant continued typing until session time ran out, or until they no longer had enough neural control to type, pause, or unpause on their own. T7 lost the ability to type early in the control session, which gave us an opportunity to test in the remaining time whether turning the self-calibration back on would help to rescue neural control. First, bias correction and between-block feature tracking were reinstated, and then, after a block of self-paced typing, an RTI decoder was built using only the data collected in that last typing block. The participant typed using this RTI decoder, paused when he desired, and then the original decoder was reinstated (with bias correction and mean tracking still on) for one last block.

Finally, in a series of sessions with participant T6 (participants S3, T2, and T7 were no longer in the trial), we tested whether these self-calibration methods allow for stable neural decoding across multiple days of practical BCI use. In the first session, the standard calibration task was used to initialize the decoder. This standard decoder was used in the first radial keyboard block. After that, an RTI decoder was calibrated during every self-timed pause, using the data acquired during the previous 20 min to 1 hour of free typing. Each session after the first was initialized with the previous session’s last RTI decoder.

RTI decoder calibration

To calibrate the RTI decoder, the person’s intended movement direction was retrospectively assumed to have been directly toward his or her next selected target; then, similar to standard closed-loop calibration with presented targets, these retrospectively inferred intended directions were mapped to the corresponding neural data. Unlike in standard calibration, however, the timing of the person’s intended movements was self-paced and therefore unknown. We estimated which time periods were most likely to correspond to the user’s intent to move the cursor with the following heuristics: (i) use only the last 5 s preceding each target selection; (ii) use only those time bins in which the cursor moved closer to the next selected target; and (iii) remove bins from calibration in which the cursor was within a certain distance or temporal window of the next selected target (Fig. 3A; Supplementary Materials and Methods).

Adaptive feature mean tracking and bias correction

Nonstationarities in baseline rates were mitigated by updating our estimate of the baseline rate of each channel based on its mean rate in the most recent block, and subtracting that rate from the ongoing rate before sending each channel’s neural data to the decoding algorithm. In the self-paced typing sessions, baseline rate and variance estimates were also updated between blocks of neural control using a recursively defined running estimate (48). Within blocks of neural control, we also iteratively estimated and subtracted out the velocity bias directly by computing an exponentially weighted running mean of all decoded velocities whose speeds exceeded a predefined threshold, set to the 66th percentile of the decoded speeds estimated during the most recent filter calibration (Fig. 2). Details of feature tracking and bias correction are in Supplementary Materials and Methods.

Statistical analysis

Typing rate was quantified as CCPM and CSPM, each measured over the entire continuous block of typing. In the free-typing blocks, the intended text was assumed to have been the final text (each selection that was undone by backspacing was assumed to have been unintended; thus, each backspace removed a character or selection from the total count when computing CCPM and CSPM). In the radial keyboard, in which all of the possible targets have the same size, CSPM was also translated into eBR, the number of bits of information conveyed per second (17, 32) extrapolated to a virtual keyboard. Two-tailed paired t tests were used to test for significant differences in the paired quantities shown in scatterplots, after confirming that the paired differences were normally distributed. Sample estimates are given as means ± SEM. P values for Pearson correlation coefficients were obtained by comparing the measured value to a null distribution obtained by shuffling the pairings 1,000,000 times.