A Non-Human Primate Brain-Computer Typing Interface

Abstract

Brain-computer interfaces (BCIs) record brain activity and translate the information into useful control signals. They can be used to restore function to people with paralysis by controlling end effectors such as computer cursors and robotic limbs. Communication neural prostheses are BCIs that control user interfaces on computers or mobile devices. Here we demonstrate a communication prosthesis by simulating a typing task with two rhesus macaques implanted with electrode arrays. The monkeys used two of the highest known performing BCI decoders to type out words and sentences when prompted one symbol/letter at a time. On average, Monkeys J and L achieved typing rates of 10.0 and 7.2 words per minute (wpm), respectively, copying text from a newspaper article using a velocity-only two dimensional BCI decoder with dwell-based symbol selection. With a BCI decoder that also featured a discrete click for key selection, typing rates increased to 12.0 and 7.8 wpm. These represent the highest known achieved communication rates using a BCI. We then quantified the relationship between bitrate and typing rate and found it approximately linear: typing rate in wpm is nearly three times bitrate in bits per second. We also compared the metrics of achieved bitrate and information transfer rate and discuss their applicability to real-world typing scenarios. Although this study cannot model the impact of cognitive load of word and sentence planning, the findings here demonstrate the feasibility of BCIs to serve as communication interfaces and represent an upper bound on the expected achieved typing rate for a given BCI throughput.

I. Introduction

ONE potential application of brain-computer interfaces (BCIs) is the restoration of communication for people with movement disabilities. The goal of such systems is to communicate text quickly and efficiently. Healthy individuals type text via a keyboard through discrete selections of keys using multiple fingers near simultaneously. This is a relatively high throughput task, and even the highest performing BCI algorithms are several times too slow to sustain such data rates [1–3]. One alternative approach is to leverage the use of a neurally-driven cursor on a virtual keyboard, akin to using a computer mouse or finger on a mobile computing device. In this manner, the interface has fewer degrees of freedom (e.g., two-dimensional movement and optionally a single discrete click), and operates with lower data rates. This approach has demonstrated feasibility in both preclinical [2–8] and early clinical [9–14] studies. The current highest sustained throughput achieved via a BCI has been with primate studies and achieves around 4–6 bits per second [2, 3, 15]. Although bitrate is an important measure of achieved performance, for a communication BCI, a more relevant measure is typing rate. However, the relationship between bitrate and typing rate is not well understood. The aim of this study was to conduct typing task experiments in non-human primates using the highest known performing BCIs to then quantify typing rate. Although this experiment is limited because monkeys do not possess a written language, such a study would provide an upper limit to the expected effective typing rate with a given level of BCI throughput.

II. Methods

Two sets of experiments were conducted in this study. They differed in the neural decoder and the resulting method of selection. The first experiment, termed dwell typing, involved only cursor velocity control [7], while the second, called click typing, included both cursor velocity control and a discrete click [3]. The aim in both experiments was to communicate prompted text as quickly and accurately as possible.

Experimental setup

Monkey protocol and behavior

All procedures and experiments were approved by the Stanford University Institutional Animal Care and Use Committee (IACUC). Experiments were conducted with two adult male rhesus macaques (L and J), implanted with 96 channel Utah electrode arrays (Blackrock Microsystems, Salt Lake City, UT) using standard neurosurgical techniques. Electrode arrays were implanted in arm motor regions of primary motor cortex (M1) and dorsal premotor cortex (PMd), as estimated visually from local anatomical landmarks as previously described [1]. Monkey L was implanted with a single array at the M1/PMd border on 2008-01-22. Monkey J had two arrays implanted on 2009-08-24, one in M1 and the other in PMd. All available channels were used from both monkeys. Monkeys had been previously trained to make point-to-point reaches in a 2D plane with a virtual cursor controlled by the contralateral arm or by a neurally-driven decoder, as diagrammed in Figure 1a. They exhibited no learning behavior to the task during these studies. The arm ipsilateral to the implanted hemisphere was gently restrained at all times during these experiments. The contralateral arm was left unbound and typically continued to move even during neurally-controlled sessions. This approach was preferred because it minimized any behavioral, and thus neural, differences between the training set (arm control sessions) and the testing set (neural prosthesis sessions). This animal model was selected because we believe it more closely mimicked the neural state a human user would employ when controlling a neural prosthesis in a clinical setting [16]. This model has previously demonstrated comparable performance to the dual arm-restrained animal model, where the contralateral arm is additionally restrained [7].

Fig. 1. Experimental Setup.

a Diagram of monkey performing typing task using experimental neuroelectrophysiology rig. b Diagram of virtual typing task. Workspace was divided into a 6 × 6 grid of 4 cm square targets. Gray cursor was under control of monkey. Monkeys were presented with yellow targets and a green prompted target to navigate to. Dashed lines denote the acquisition region boundaries–not shown to the monkey. c Layout of the 6 × 6 keyboard. Special keys were: ⇑ - shift, ⊔ - space, ⇐ - delete, ↲ - return, EOM (bottom left) - triplet of keys to select in sequence to note “end of message”.

Experimental hardware

The virtual cursor and grid of targets were presented in a 3D environment (MSMS, MDDF, USC, Los Angeles, CA) [17] as previously reported [2]. Hand position was measured with an infrared reflective bead tracking system (Polaris, Northern Digital, Ontario, Canada) polling the acquisition region at 60 Hz. Neural data was processed by the Cerebus recording system (Blackrock Microsystems, Salt Lake City, UT) and made available to the behavioral control system within 5 ± 1 ms. Spike counts were collected by applying a single negative threshold, set to −4.5 × root mean square of the voltage of the spike band of each neural channel using the Cerebus software’s thresholding algorithm [18]. Behavioral control and neural decode were run on separate PCs using the Simulink Real-Time/xPC platform (Mathworks, Natick, MA) with communication latencies of 3 ms. This system enabled millisecond-timing precision for all computations. Visual presentation was provided via two LCD monitors with refresh rates at 120 Hz, yielding frame updates within 7 ± 4 ms. Two mirrors, setup as a Wheatstone stereograph, visually fused the monitors into a single 3D percept for the monkey, although all cursor motion was constrained to two dimensions [19]. Datasets are referenced by a monkey and date format where a dataset from Monkey J would be JYYYYMMDD.

Decoder design

Both of the decoders used in this study have been previously described and demonstrated [3, 7], but are briefly described below.

ReFIT Kalman Filter

The cursor movement decoder used in this task was the Recalibrated Feedback Intention-Trained Kalman Filter (ReFIT-KF) [7, 20]. The ReFIT-KF algorithm uses a two-step training protocol that corrects the errors made during an initial block of closed-loop control to arrive at a final high-performing decoder. The ReFIT-KF has demonstrated the highest sustained throughput [2] and longest sustainable performance [15] of any single BCI decoder to date.

To build the decoder used in this study, 500 trials of center-out and back trials were performed under a hand-controlled cursor to targets in eight directions in a 12 cm radius as previously reported [7]. This initial dataset was used to build a first pass decoder, which was then tested online to collect another training set for which the kinematics were corrected, yielding the final decoder that was used for the remainder of the experimental day.

HMM Click Decoder

A hidden Markov model was used to decode the intention of click during the second set of experiments. Details of this decoder were described in a prior study [3]. A click decoder helps avoid the dwell time necessary in a cursor movement only BCI when selecting targets, yielding performance gains of up to 50%. Running in parallel with the ReFIT-KF, these two decoders achieved the highest throughput of any BCI under any control modality. In general, the HMM is trained by separating the hand-controlled reaches into two states: move and stop. The stop state is the period of time where the cursor is dwelling over the target. These labelled states are used to build the transition model and the corresponding neural time bins are used to estimate the emissions model. Together with the ReFIT-KF, these two decoders enable two-dimensional cursor control and a click signal, akin to using a conventional computer mouse.

Typing task

The typing task was setup in a grid layout. The task was similar to that used in a prior study that optimized keyboard-like tasks [2]. The grid was laid out in a 6×6 grid with 4×4 cm acceptance regions, where letters and symbols were assigned to specific “keys” in the workspace as shown in Figure 1b. The arrangement of letters was derived from a Metropolis keyboard layout [21, 22] as shown in Figure 1c. The monkeys were not shown this mapping and saw only yellow dots in a grid layout. However, the prompting of these targets was done in a specific sequence as dictated by the text to be copied. The prompted target was lit in green, and the monkey’s goal was to navigate the cursor to that green target and select it while avoiding selection of a yellow target. Monkeys were rewarded with a liquid reward for each successful trial. Selection of a yellow target was considered an error and resulted in a failed trial and no liquid reward. Every error was required to be corrected by the selection of the “delete key” before resuming the text sequence, just as done in a conventional keyboard. In this fashion, the task simulated the typing of words and sentences that a human participant would perform.

There were two methods for selecting a letter, depending on which experiment was being conducted. In dwell typing experiment with only the ReFIT-KF and 2D cursor control, targets were selected by dwelling on them for the required 450 ms hold time. In the click typing experiment, the HMM click decoder controlled symbol selection by detecting the intention to select the target under the cursor.

The passages for the two experiments were different and selected from articles from the New York Times (i.e., dwell [23] and click [24] typing). For the purpose of this study, capitalization and punctuation was ignored. These experiments were repeated on more than one day, but only one representative day from each control modality is shown from each monkey. From the monkeys’ points of view, these experiments were effectively identical to prior experiments where only bitrate was measured [2, 3, 15], differing only in that targets were prompted in a non-random fashion. For clarity and simplicity of presentation, as these three prior studies contained over a million trials between them and confer significant confidence in the tasks and decoding paradigms, we elected here to show only a single representative day of typing tasks under each decoding modality.

Metrics

There are three relevant metrics in this study. Each will be discussed briefly below.

Typing rate

Typing rate is the primary measure used in this study and the most intuitive because of its familiarity and applicability. Conventional typing rate is measured as the effective characters per minute transmitted when copying text. These approaches have varying ways to account for uncorrected errors (e.g., a misspelled word). However, we did not utilize any of these in this study as we continually prompted the delete key until all errors were corrected before resuming. Thus, the stream of text transmitted was always correct, except for the most recent sequence of errors.

Typing rate can be described via the following equation:

$$T=\frac{S_c-S_i}{5t}$$

where T represents typing rate in words per minute, S_c represents the correct number of symbols/keys transmitted (including spaces and deletes), S_i is the incorrect symbols/keys transmitted (that would then have to be deleted), and t the elapsed time. The division by five is the convention used to convert characters (including spaces) per minute into words per minute [25]. Note that this measure is independent of information theoretic influences such as dictionary size and channel code. Any coding method can be used so long as the output is meaningful English, though the most common method used is the single-symbol channel coded QWERTY keyboard.

Achieved bitrate

Achieved bitrate represents the effective throughput of the system under a single-symbol channel code, previously described in detail [2]. A single-symbol channel code is the most conventional form of communication channel used by people when they type (i.e., a QWERTY keyboard). Although multi-symbol channel codes are more efficient and resistant to errors and considered best practice in electronic communication channels, they are not practical or easy to use by people. In a single-symbol channel coded keyboard, the delete button is used to correct errors one symbol/letter at a time. The equation for achieved bitrate under this coding scheme is below:

$$B=\frac{{\text{log}}_2(N-1)\text{max}(S_c-S_i,0)}{t}$$

where B is the achieved bitrate in bits per second (bps) and N is the number of selectable symbols on the interface (including delete key). As with typing rate, the other three parameters are the same: S_c is the correct number of symbols, S_i is the number of incorrect symbols, and t is the elapsed time. The N − 1 term accounts for the reduction in dictionary size by one because the delete key does not produce a symbol, but instead is a marker which edits the stream of transmitted symbols. The max function is necessary because bitrate cannot be negative, and this demonstrates that a 50% success rate (i.e., where S_c = S_i) conveys no net information. This metric has been previously derived in a slightly different, but mathematically equivalent form termed “practical bit rate” in an EEG-based BCI study [26].

Information transfer rate

The most popular metric in the field of BCIs is information transfer rate and was first introduced nearly thirty years ago [27]. It is a measure of channel capacity and thus always overestimates the throughput of the communication channel. Further, it is not a practical measure because of the many assumptions it makes [28], including infinite length channel codes. The equation for information transfer rate is shown below:

$$\mathrm{I}\mathrm{T}\mathrm{R}=\frac{{\text{log}}_{2}N+P{\text{log}}_{2}P+(1-P)\text{log}\frac{1-P}{N-1}}{t}$$

where ITR is the information transfer rate in bps, N is the number of selectable symbols, P is the probability of correctly selecting a symbol (i.e., $\frac{S_c}{S_c+S_i}$), and t is the elapsed time. Comparisons between ITR and achieved bitrate appear towards the end of the text.

III. Results

Both the dwell and click typing experiments were successful in simulating typing under the varying control modalities. Performance on the dwell typing experiment with both monkeys is shown in Figure 2. Monkey J achieved an average rate of 10.0 words per minute (wpm), completing the prompted article in about three hours. Monkey L achieved an average of 7.2 wpm and completed about two thirds of the article in two hours before voluntarily stopping. Part of the reason for Monkey L’s lower performance is that he has lower overall behavioral motivation and BCI throughput than Monkey J. An example video of Monkey J using the BCI to transmit text with the dwell typing experiment is shown in Figure 3, which is a still frame from Supplementary Video 1.

Fig. 2. Dwell typing performance.

a Plot of dwell typing experiment of Monkey J. Discontinuities seen after 100 minutes are when the monkey took voluntary pauses. Data from dataset J20111021. b Histogram of dwell typing performance from the same dataset in Monkey J. Dot at top notes the average typing rate of 10.0 wpm. c Same plot as in panel a, but for Monkey L, from dataset L20111103. d Same plot as in panel b, but for Monkey L, with an average typing rate of 7.2 wpm.

Fig. 3. Example of dwell typing.

Still frame from Supplementary Video 1 showing performance of dwell typing task of Monkey J. Data is from J20111014. Monkey only saw yellow and green dots, letters were overlaid in post-processing. The particular decoder used in this video had been held constant for over thirty days from a prior study [15]. The trial shown in this frame is trial 157, where the period key is prompted.

Performance on the click typing experiment with both monkeys is shown in Figure 4. Monkey J achieved an average rate of 12.0 wpm, completing all but the last two paragraphs of the article in just over an hour. Monkey L achieved an average rate of 7.8 wpm in just over an hour and half, stopping at the sentence prior to where Monkey J stopped. As the click typing experiment is faster paced and more strenuous, the monkeys (Monkey J in particular) are less willing to perform the click typing task for as long [3]. Example videos of Monkey J using the BCI to transmit text under the click typing experiment appear as Supplementary Videos 2 and 3.

Fig. 4. Click typing performance.

a Plot of click typing experiment of Monkey J. Data from dataset J20120411. b Histogram of click typing performance from the same dataset in Monkey J. Dot at top notes the average typing rate of 12.0 wpm. c Same plot as in panel a, but for Monkey L, from dataset L20120501. d Same plot as in panel b, but for Monkey L, with an average typing rate of 7.8 wpm.

Combining these datasets with data collected from these same monkeys using the same decoding methods from earlier studies [2, 3, 15], we related typing rate to achieved bitrate, shown in Figure 5a. This plot shows the average typing rates and bitrates achieved by both monkeys using both experiments. A regression line fit to this data (with an assumed intercept at the origin), yielded a slope of 2.69 (95%CI: 2.43 − 2.96). Thus, the relationship between typing rate in words per minute and achieved bitrate in bps can be approximated by the following equation:

$$T=2.7·B$$

Fig. 5. Metrics comparison.

a Plot of typing rate vs achieved bitrate. The achieved bitrate measures are from prior studies [2, 3, 15]. Circles denotes Monkey J. Squares denote Monkey L. Hollow marker denotes dwell typing experiment. Filled marker denotes click typing experiment. The regressed line is of the form T = 2.7 · B. b Plot of achieved bitrate and ITR as a function of success rate. The y-axis does not have ticks because only the relationship of the lines is important in this plot and absolute bitrate varies as a function of dictionary size. Achieved bitrate is the linear piecewise line and ITR is the curved line. Note that the ITR curve is always greater than or equal to the achieved bitrate line. This plot is for N = 5. Intercept points are at $\frac{1}{N}$ (chance rate) and $\frac{N-1}{N}$.

Finally, we mathematically modeled the relationship between achieved bitrate and information transfer rate. For a dictionary size of five, this relationship is plotted in Figure 5b. The precise shape of the ITR curve and the slope of the achieved bitrate line vary with dictionary size, but the general features do not change for all tasks with N > 2. Consistent with the notion that ITR is a measure of capacity, we see that it is strictly higher than achieved bitrate regardless of the BCI communication accuracy (i.e., success rate). The ITR curve touches zero at the chance level: $\frac{1}{N}$. However, achieved bitrate is zero at all values at or below 50% success rate, reflecting the fact that no symbols are effectively transmitted at these success rates because of the constant need to select the delete key. The other point at which achieved bitrate and ITR are equivalent is when the success rate is $\frac{N-1}{N}$. Note that the relationship to ITR is not one-to-one for success rates at or below 50%. This implies that a single measurement of ITR cannot well predict typing rates at low success rates.

IV. Discussion

This study demonstrates the feasibility of BCIs as communication devices for transmitting text. To our knowledge, the typing rates achieved in this study represent the highest of any BCI under any control modality. We also found that text can be communicated uninterrupted for hours via a single BCI decoder without any changes. The click typing experiment was conducted 2.5 (Monkey J) and 4 (Monkey L) years post surgical implantation, serving as an additional existence proof that these electrode arrays can facilitate high-performing BCI control for years. We note that this study is limited by the use of monkeys who were only acquiring prompted targets. The results presented here do not account for the cognitive load of word and sentence formation that a person would encounter. However, for a copy typing task as is commonly used for measuring typing performance, this may be a close approximation. At the least, this study provides an upper limit for the expected typing rate for a given achieved bitrate. A more accurate measure of the relationship between achieved bitrate and effective typing rate would best be performed by human participants with comparable high-performing BCIs. Nevertheless, this study builds additional evidence towards the promise of BCIs for building neural prostheses for people with motor disabilities.

Biographies

Paul Nuyujukian (S05-M13) received the B.S. degree in cybernetics from the University of California, Los Angeles in 2006. He received an M.S. and Ph.D. degrees in bioengineering and M.D degree from Stanford University in 2011, 2012, and 2014, respectively. He is currently a postdoctoral scholar in the departments of Neurosurgery and Electrical Engineering at Stanford University. His research interests include the development and clinical translation of brain-machine interfaces for neural prostheses and other neurological conditions.

Jonathan C. Kao (S13) received the B.S. and M.S. degree in electrical engineering from Stanford University in 2010. He is currently pursuing his Ph.D. degree in electrical engineering at Stanford University. His research interests include algorithms for neural prosthetic control, neural dynamical systems modeling, and the development of clinically viable neural prostheses.

Stephen I. Ryu received the B.S. and M.S. degree in electrical engineering from Stanford University, Stanford, CA, in 1994 and 1995, respectively, and the M.D. degree from the University of California at San Diego, La Jolla in 1999. He was a postdoctoral fellow at Stanford in Neurobiology and Electrical Engineering from 2002–2006. He completed neurosurgical residency and fellowship training at Stanford University in 2006. He was on faculty as an Assistant Professor of Neurosurgery at Stanford University until 2009. Since 2009, he has been a Consulting Professor of Electrical Engineering at Stanford University. His neurosurgical clinical practice is with the Palo Alto Medical Foundation in Palo Alto, CA. His research interests include brainmachine interfaces, neural prosthetics, minimally invasive neurosurgery, and stereotactic radiosurgery.

Krishna V. Shenoy (S87-M01-SM06) received the B.S. degree in electrical engineering from U.C. Irvine in 1990, and the M.S. and Ph.D. degrees in electrical engineering from MIT, Cambridge in 1992 and 1995, respectively. He was a Neurobiology Postdoctoral Fellow at Caltech from 1995 to 2001 and then joined Stanford University where he is currently a Professor in the Departments of Electrical Engineering, Bioengineering, and Neurobiology, and in the Bio-X and Neurosciences Programs. He is also with the Stanford Neurosciences Institute and is a Howard Hughes Medical Institute Investigator. His research interests include computational motor neurophysiology and neural prosthetic system design. He is the director of the Neural Prosthetic Systems Laboratory and co-director of the Neural Prosthetics Translational Laboratory at Stanford University. Dr. Shenoy was a recipient of the 1996 Hertz Foundation Doctoral Thesis Prize, a Burroughs Wellcome Fund Career Award in the Biomedical Sciences, an Alfred P. Sloan Research Fellowship, a McKnight Endowment Fund in Neuroscience Technological Innovations in Neurosciences Award, a 2009 National Institutes of Health Director’s Pioneer Award, the 2010 Stanford University Postdoctoral Mentoring Award, and the 2013 Distinguished Alumnus Award from the Henry Samueli School of Engineering at U.C. Irvine.