A brain-computer typing interface using finger movements

Abstract

Intracortical brain computer interfaces (iBCIs) decode neural activity from the cortex and enable motor and communication prostheses, such as cursor control, handwriting and speech, for people with paralysis. This paper introduces a new iBCI communication prosthesis using a 3D keyboard interface for typing using continuous, closed loop movement of multiple fingers. A participant-specific BCI keyboard prototype was developed for a BrainGate2 clinical trial participant (T5) using neural recordings from the hand-knob area of the left premotor cortex. We assessed the relative decoding accuracy of flexion/extension movements of individual single fingers (5 degrees of freedom (DOF)) vs. three groups of fingers (thumb, index-middle, and ring-small fingers, 3 DOF). Neural decoding using 3 independent DOF was more accurate (95%) than that using 5 DOF (76%). A virtual keyboard was then developed where each finger group moved along a flexion-extension arc to acquire targets that corresponded to English letters and symbols. The locations of these letter/symbols were optimized using natural language statistics, resulting in an approximately a 2× reduction in distance traveled by fingers on average compared to a random keyboard layout. This keyboard was tested using a simple real-time closed loop decoder enabling T5 to type with 31 symbols at 90% accuracy and approximately 2.3 sec/symbol (excluding a 2 second hold time) on average.

I. Introduction

It is estimated that 1.7 percent of Americans live with some form of paralysis [1]. Intracortical brain-computer interfaces (iBCIs) offer the prospect of motor and communication restoration for people with severe speech and motor impairments [2], [3]. In previous studies from our group, we demonstrated communication rates of approximately 9 words per minute (WPM) using a system based on selecting keys using a 2D cursor [2] and approximately 18 WPM using a system based on decoding handwriting [3]. For an able-bodied person, typing on a standard QWERTY keyboard can lead to throughput of 70+ WPM and is preferred by many people over handwriting. However, there is currently no BCI interface for typing based on individual finger movements, possibly because decoding the continuous movement of multiple finger groups has only recently been demonstrated in monkeys [8], [9] and humans [10]. Studies involving users of assistive technology have revealed a need for a diverse collection of communication interfaces to cater to individual needs of each person [11].

There are several different possible approaches to developing an optimized BCI keyboard. While it may be ideal to decode attempted finger presses on a standard QWERTY keyboard, finger movements are usually accompanied by subtle wrist and arm movements which can lead to limited accuracy for decoding neural activity associated with these discrete movements. An alternative approach is to identify independent degrees of freedom that correspond to the flexion-extension movements of different finger groups, mapping keys to the locations of these finger groups along the flexion-extension axis and assigning characters to keys using the statistics of the English language. Continuous finger movements displayed in real time allow the user to employ visual feedback about the selected keys, potentially leading to higher accuracy.

In the present study, we show a proof-of-concept BCI keyboard based on individual finger movements and demonstrate its closed loop performance. This work introduces an approach for personalized BCI interface development based on simultaneously optimizing both the task structure (keyboard layout) and the neural decoder. Of the three important components of any BCI system (neural recording, decoding and task), most past research has focused on improving each component in isolation from the others. While there has been some recent work on co-optimization of neural recording and decoding [6], our approach of designing the keyboard based on the ability to decode different movements presents a relatively new direction.

II. Methods

A. Neural Recordings

Data were collected from a participant (T5) enrolled in the BrainGate2 pilot clinical trial (ClinicalTrials.gov NCT00912041). T5 is a 70 year old right-handed man with C4 AIS C spinal cord injury. Two silicon microelectrode arrays, each consisting of 96 1.5-mm long electrodes (192 channels total) were placed in the ‘hand-knob’ area of the left precentral gyrus. Threshold crossings (threshold −4.5 RMS of the noise) captured the multi-unit population activity and were used for all the analyses presented.

B. Finger and keyboard animations

The task was implemented in 3D with Unity software (https://unity.com/), which was used to display a graphical rendering of a right hand. Each finger was constrained to move along a single flexion-extension direction. The hand was shown with the palm facing the participant so that finger movements did not overlap. Open loop trials involved animating the fingers along a predetermined trajectory; during closed loop trials, finger position was decoded from neural activity. The keyboard was developed by placing keys in the 3D space, with each key lying along the flexion-extension arc of a particular finger. Details of character assignments to these keys is shown in Sec. III-B.

C. Open-loop trials

The ability to decode different finger combinations was evaluated using open loop data as described in Sec. III-A. The open loop trials were randomized across target gestures and cued by the animated fingers starting from rest, slowly moving toward a target position (1-2s) and holding at that position for 1-2s before going back to rest. Classification accuracy was evaluated using a Naïve Bayes decoder that used the total number of threshold crossings in neural activity from the movement and hold periods. Individual and simultaneous movements of two finger groups were evaluated, with 10-fold cross-validation.

D. Closed-loop trials

Similar to the standard 2D-cursor control task [2], the participant attempted to apply maximum force to the finger groups he intended to move (similar to a ’joystick’ control). The neural activity generated during this attempted movement was used to estimate the the finger velocity. Specifically, the output scores from a logistic classifier prior to classification were used to estimate the velocities. An additional class was used for no movement. The following operations were sequentially applied at each step during closed loop decoding: (i) log-probability for different classes was estimated from the neural activity (binned at 15ms) (ii) the log-probabilities were causally smoothed with a half-Gaussian filter (σ = 300ms), (iii) the most probable class was identified, (iv) the velocity was set to a fixed magnitude (with sign corresponding to the class) if the probability of the most probable class exceeded 0.6, (vi) velocity was smoothed with a half-Gaussian filter (σ = 75ms) and (vii) finally, this velocity was used to update the finger position. Several values were empirically specified through manual quasi-optimization.

Each trial began by highlighting the target key, resetting all finger positions to neutral, and waiting for two seconds before allowing movement toward the target. At each update step, distance between each key and the corresponding finger position was evaluated if the finger is far from idle, and the nearest key was selected over the entire keyboard. Keys were typed by dwelling over them for two seconds. Trials were successful if the correct target character was selected, and unsuccessful if either the trial timed out (at 10 seconds) or an incorrect character was selected.

Neural activity and intended finger movement directions were logged continuously, iteratively updating the decoder and immediately using it for closed loop decoding. Collecting almost all training data in closed loop minimizes the impact of distribution shift between open loop and closed loop data.

III. Results

We first analyzed open-loop data to investigate whether grouping five fingers into a smaller number of finger groups resulted in higher decoding accuracy. Subsequently, we designed a virtual keyboard using natural language statistics for typing with three finger groups. Finally, we demonstrated closed loop typing with this virtual keyboard.

A. Three-finger grouped decoding is more accurate than five-finger grouped decoding

Linear decoding analysis showed that the movement of up to three finger groups could be accurately classified in our participant. T5 attempted flexion/extension movements of individual fingers or a group of fingers on the right hand (contralateral to the arrays). For individual finger movements, the linear decoder achieved 71% cross-validated accuracy, with confusion between identical movements on nearby fingers (Fig 1A). This reflects the bio-mechanical structure of the hand, as nearby fingers have correlated movement and share similar muscles. While the performance for decoding individual fingers was above chance, near-perfect accuracy and independent control of finger movements is required for accurate typing. An improved accuracy of 95% was observed by combining nearby fingers, resulting in three finger groups consisting of thumb, index-middle and ring-little fingers (Fig 1B). Next, we designed a BCI keyboard for typing with these three finger groups.

Fig. 1.

Classification analysis with confusion matrices showing accuracy of (A) five individualized finger movements and (B) fingers grouped into three groups. Movements are grouped by flexion and extension (orange colored boxes represent flexion or extension and blue box outline groups the other).

B. Keyboard design for three-finger group typing

For the keyboard with three finger groups, there are six directions of movement as each finger group can be flexed or extended from the neutral rest position. The keyboard is designed for 31 symbols (26 letters, space (indicated by symbol >) and punctuation “?”, “.”, “,”, and “’”) to match our recent handwriting-decoding study in the same participant [3]. First, roughly equal number of keys were assigned to each of the six directions. More keys were assigned to the thumb as it could be controlled with highest accuracy.

Next, characters were assigned to the keys using the statistics of the English language. The relative frequency of different symbols was measured using the Brown text corpus of 57,340 sentences ( [7], Fig. 2A). Notably, the space (>) had the highest frequency, and some of the punctuation symbols had higher frequency than letters. The symbols were assigned to keys using a “greedy” method, with the highest frequency symbols assigned to keys with the smallest distance from the corresponding finger’s rest position. As there are many keys with the same distance from rest, key assignment was based on the effector with the highest controllability (thumb movements, before index and middle finger movements, before ring and little finger movements; flexion before extension movements). The resulting assignment is shown in Fig. 2B. High frequency symbols such as “E”, “T”, “A”, space (“>”) are only a small movement away from the corresponding rest position. This assignment results in roughly equal frequency of occurrence in each direction of motion.

Fig. 2.

Keyboard design. (A) Unigram frequencies of all symbols from the Brown Corpus. (B) Designed keyboard. Colors indicate finger groupings, with index-middle (green) and ring-little (yellow) tied together. Keys (circles) lie along the flexion-extension movement axis of the corresponding finger, with key color indicating finger assignment. Staggered locations of keys on the same finger group allows a unique selection even though two fingers are constrained to move together. (C) Mean distance per character across sentences from Brown corpus for the optimized keyboard (black histograms) and different random keyboards (red histograms). Lines indicate Gaussian fits to each histogram. Minimum and maximum distance of keys indicated as a reference.

Under the assumption of constant velocity throughout a trial and independence of individual fingers, typing rate is proportional to average distance travelled per character. For a particular keyboard layout, the average distance per character was computed for all characters in a sentence and the distribution across all sentences in the Brown corpus was evaluated. The distribution for the optimized keyboard (black, Fig. 2C) was substantially smaller than the distribution for different random keyboard layouts (red, Fig. 2C). Overall, this exercise of optimizing keyboard layout suggests that task optimization, based on the decoding properties in BCI participants may lead to substantial improvements in performance.

C. Closed loop decoding performance

This ‘three finger’ virtual typing interface was tested during real-time closed loop decoding, using sequences of characters forming English sentences. Each character was displayed for two seconds (preparatory period) before the participant could move the virtual fingers. Characters were selected by dwelling for two seconds over the corresponding key. Failure resulted if a wrong character was chosen or the trial timed out after 10 seconds. While the decoding performance was evaluated with a number of different modifications, results are reported here from a single research session. Across 161 characters, the participant (T5) achieved 90% accuracy with 4.3s on average per trial for successful key presses (including 2s dwell time, Fig. 3A, B, C, E). The time spent per character increased linearly with the distance from rest position, with most targets located close to rest in the optimized keyboard (Fig. 3D).

Fig. 3.

Closed loop typing performance. (A, B, C) Example trials showing finger positions for typing different letters. The finger group corresponding to the target key is flexed or extended until one of the fingers is over the key. The target and decoded sentences are displayed at the bottom of each figure. (D) Time per trial (including the two second hold time) vs distance of the target key from rest position. Green dots indicate successful trials and red dots indicate failure (either due to timeout of 10 seconds and selecting the wrong key). (E) Finger trajectories for the three finger groups. Black lines indicate target finger positions and yellow lines indicate the decoded finger positions. Green/red squares indicate successful/failed trials respectively. The target character is indicated above each trial. Vertical lines indicate the two second wait period at the beginning of each trial.

IV. Conclusions and future work

We present a 3D virtual keyboard for typing with three finger groups and test the communication rate with simple decoding techniques. We optimized the arrangement of symbols so that high frequency characters were closer to the neutral position, and each combination of finger, position and direction represented letters with roughly equal probability in the English language. The current finger movement performance, if combined with a perfect ‘click’ signal [2] instead of a two second hold time, would translate to roughly 21 correct characters per minute typing speed (number of correctly typed characters / time).

Multiple avenues of future improvement are possible. First, some aspects of the current task could be further optimized for speed, e.g., reducing the two second wait time before movement and/or the two second hold time for selection. Second, a novel nonlinear decoding architecture may lead to shorter target acquisition times [3], [8]. Third, the distances to target keys could be reduced for lower acquisition times by either incorporating bimanual typing (as the neural recordings in T5 represent both contralateral and ipsilateral movements [12]), or having a second degree of freedom (corresponding to abduction / adduction) for digits with strong tuning such as the thumb. Finally, higher communication rates could be achieved by incorporating more natural language structure with additional functions such as optionally ‘autocompleting’ words or characters using a discrete click signal.

In addition to a useful communication interface, the presented typing interface is a test-bed for answering many fundamental questions. What is the optimal balance between the number of degrees of freedom per finger vs total number of fingers? Does performance improve with practice when typing with BCI keyboard? If performance does improve with training, is this due to greater familiarity with the symbol locations on the keyboard or due to ‘more dexterous movements’ such simultaneous preparation and/or execution? With the newly introduced system it should be possible to address these, and many other questions.