Real-Time Linear Prediction of Simultaneous and Independent Movements of Two Finger Groups Using an Intracortical Brain-Machine Interface

SUMMARY

Modern brain-machine interfaces (BMIs) can return function to people with paralysis, but current upper extremity BMIs are unable to reproduce control of individuated finger movements. Here, for the first time, we present a real-time, high-speed, linear BMI in nonhuman primates that utilizes intracortical neural signals to bridge this gap. We created a non-prehensile task that systematically individuates two finger groups, the index finger and the middle-ring-small fingers combined. During online brain control, the ReFIT Kalman filter could predict individuated finger group movements with high performance. Next, training ridge regression decoders with individual movements was sufficient to predict untrained combined movements, and vice versa. Finally, we compared the postural and movement tuning of finger-related cortical activity to find that individual cortical units simultaneously encode multiple behavioral dimensions. Our results suggest that linear decoders may be sufficient for BMIs to execute high-dimensional tasks with the performance levels required for naturalistic neural prostheses.

ETOC

Nason et al. present a real-time brain-machine interface for controlling the simultaneous and independent movements of two groups of fingers in nonhuman primates. These techniques can be used to restore naturalistic control of paralyzed hands and enable a deeper understanding of how motor cortex represents dexterous finger behaviors.

Graphical Abstract

INTRODUCTION

Neural prostheses have the potential to return independence to many people with neurological disorders or injuries. In human clinical trials, laboratories have restored use of computers, self-feeding, and prosthetic hands using implants to translate electrophysiological signals into user intent (Ajiboye et al., 2017; Memberg et al., 2014; Nuyujukian et al., 2016; Pandarinath et al., 2017; Wodlinger et al., 2015). Of greatest interest to people with cervical-level spinal cord injury is the return of hand and arm function (Anderson, 2004). Although this has motivated many groups to study neural prostheses for hand control, only a few have been translated to use with people and none have been translated to full-time use outside of the laboratory. Functional electrical stimulation provides an avenue for outputting the intentions of the user to their natural limbs (Kilgore et al., 1989, 2008; Memberg et al., 2014; Smith et al., 2005), with commercial solutions having already existed such as the FreeHand System. Unfortunately, they typically rely on some external motion or myoelectric commands from residual functional muscles, which require learning and are generally unnatural to use.

This has driven many groups to use brain-machine interfaces to extract hand prosthesis control signals from a more natural source. In humans, various studies have attempted to characterize the relationship between finger movements and electrocorticography activity (Chestek et al., 2013; Hotson et al., 2016; Kubánek et al., 2009). However, that relationship was insufficiently strong to enable quick classifications or fully dexterous continuous movements. Some groups have used intracortical microelectrodes in monkeys to record activity patterns on the order of single neurons to investigate a more concrete connection between them and finger behaviors (Baker et al., 2009; Mollazadeh et al., 2011). These studies suggest such a relationship reliant on intracortical recordings is stronger than with electrocorticography, but classification of which finger is moving by itself does not provide enough understanding of that relationship to predict the quality of continuous control.

The ability to precisely control the positions (individuated placements of the hand degrees of freedom) of all individual fingers is a key characteristic of dexterous hand use in primates. As such, many groups have attempted to relate many, if not all 27, degrees of freedom (DoFs) within the hand to neural activity during reach-to-grasp tasks offline by continuously predicting hand posture (combined arrangements of different hand degrees of freedom; Aggarwal et al., 2013; Bansal et al., 2011; Okorokova et al., 2020; Vargas-Irwin et al., 2010). These studies found continuously predicting postures exceeded the performance of predicting the movement velocities, which contrasts with brain-machine interface studies for arm reaches. Additionally, although these studies showed very high offline correlation between firing rates and behaviors for many of those DoFs, most, if not all, of the DoFs presented showed highly correlated trajectories and may not have truly been independent. This makes it unclear how well this neural activity corresponds to those DoFs individually or if all the DoFs are moving so similarly that anything with a similar time course will correlate well. Further, without evaluating online control of individual DoFs, the applicability to intuitive and naturalistic neural prostheses is uncertain.

Dexterous hand use in primates and humans has two classes of movement: prehensile (for grasping objects; Napier, 1956; Santello et al., 1998) and non-prehensile (for manipulating objects). While all of these studies have made great strides towards the ability to predict the intended movements of the hand, they have focused almost exclusively on prehensile movements (Napier, 1956; Santello et al., 1998), which has a clear importance towards the goal of returning hand function to those without it. There is a substantial gap in the literature surrounding the neural representation of continuous non-prehensile movements and whether that representation can be leveraged in a brain-machine interface. One study trained a primate to use a joystick to control non-prehensile movements of a virtual hand, eventually using an online decoder to move the virtual hand online (Rouse, 2016). However, it remains unclear how much of the brain control depended on the monkey’s intentions to perform non-prehensile hand movements or if the monkey continued to think about its arm mapped to the virtual hand as the control system for the task.

Generally, the algorithms for online control of neural prostheses assume linear relationships between primary motor cortex neural activity and either the position and velocity, or expected muscle activations of prosthetic movements. Variants of Kalman filters (Ajiboye et al., 2017; Gilja et al., 2012; Malik et al., 2011; Wu et al., 2004), ridge regressions (Collinger et al., 2013; Mulliken et al., 2008; Wodlinger et al., 2015), and Wiener filters (Ethier et al., 2012; Koyama et al., 2010; Sachs et al., 2015) have been used to control arms, hands, and fingers online. Linear online decoders are promising candidates for an out-of-laboratory clinical neural prosthesis due to their computational simplicity and high prediction performance. However, with limited quantities of recording electrodes, covariances between nearby neural signals, and increasing numbers of DoFs required for finger control, linear decoders may be unable to accommodate multiple independent degrees of freedom.

It has been noted by several groups that the same neurons can covary with substantially different behaviors, which could make the prediction of finger movements particularly difficult. For example, primary motor cortex can simultaneously encode information about upper extremities, fingers, and speech, independent of body laterality (Cross et al., 2020; Diedrichsen et al., 2013; Heming et al., 2019; Jorge et al., 2020; Stavisky et al., 2019, 2020; Willett et al., 2020). As tasks increase in complexity, linear models may be unable to discriminate between neural states without sampling greater quantities of relevant neurons. Therefore, it is valuable to characterize the limits of linear models in discriminating neural states with truly simultaneous movement of independent DoFs.

Here, we show for the first time, fine, independent, and simultaneous online control of two systematically individuated groups of fingers within one hand to acquire two targets, one each for the index finger and the middle-ring-small (MRS) fingers, in a non-prehensile task using linear Kalman filters and an intracortical brain-machine interface in nonhuman primates. With intention-based retraining of the Kalman filters, we find that online brain control improves significantly. Then, we find that the magnitude of individual neural activations to particular non-prehensile movements, whether they correspond to movements of one group or combined movements of both groups, can be well predicted by the weighted sums of the most similar movements. This suggests that neural representations of continuous, non-prehensile finger movements are related by linear combinations of similar movements, enabling us to accurately predict untrained finger movements offline using ridge regression. Finally, we characterize the similarity between postural and movement tuning of cortical spiking activity to fingers to find that the preferred movement direction is rarely associated with the preferred posture, regardless of the beginning posture.

RESULTS

Linear Two-Finger Decoding in Real-Time

We first sought to validate that linear decoder models could individuate two systematically separated finger dimensions moving independently and often simultaneously throughout their entire ranges of motion. To do so, we trained two adult male able-bodied rhesus macaques, monkeys N and W, to perform a two-target two-finger task by using a manipulandum (Vaskov et al., 2018). The manipulandum consisted of two flat surfaces, one for each finger group, where each surface was free to rotate about a hinge at the metacarpophalangeal joints and was sized according to the finger group being used to push it (index vs. MRS). This task is illustrated in Figure 1A, with a drawing of the manipulandum included. Although the monkeys were presented with two one-dimensional targets, they can be visualized as one two-dimensional target in a two-dimensional space of % index flexion versus % MRS flexion, as shown in Figure 1B, without making any assumptions about cortical representations of these finger groups. For real-time decoding, targets were presented in a center-out style, in which every other target presented was located at rest. Non-rest targets were pseudo-randomly chosen from the postures in Figure 1B, then a magnitude of movement was pseudo-randomly chosen between +20%, +30%, or +40% flexion or extension from rest. These movement magnitudes can be imagined by further flexing each posture as shown in Figure 1B, as shown in Supplementary Table 1, and are realized on the vertical axes in Figure 2A and B and Figure 3A and B. This resulted in 19 total target combinations without IE+MF and IF+ME postures (3 magnitudes for 6 different postures with a central rest target; see STAR Methods). During the task, we synchronously recorded neural activity using 96 channels of implanted Utah silicon microelectrode arrays (Blackrock Microsystems, Salt Lake City, UT, USA) from the hand area of primary motor cortex in each monkey (implant photographs in Figure 1D). Each experimental day, we collected a training dataset in which the monkey controlled the virtual hand using the manipulandum while synchronously recording 300–1,000Hz spiking band power (SBP). We have previously shown that SBP is well correlated with the firing rate of the largest amplitude single unit or units on an electrode and typically results in higher decoding performance than threshold crossing rate (Nason et al., 2020). Then, we trained a Kalman filter as detailed in the methods to predict fingertip velocities in real-time and tested it in closed-loop by actuating the virtual hand according to the predictions.

Figure 1. Experimental description.

(A) The monkey was seated in front of a screen displaying a virtual hand with his left hand placed in a manipulandum. Positions of the index and middle/ring/small (MRS) finger groups were measured by the manipulandum (right side of A) synchronously with the neural activity. The position measurements or the decoded finger positions were used to actuate the virtual hand, depending on the stage of the experiment. Targets were pseudo-randomly presented in a center-out pattern based on the postures in B or in a pattern where target positions were pseudo-randomly placed along each finger’s dimension, not separated by more than 50% of the range. (B) Two-dimensional space for visualizing the hand movements. Postures shown are at +30% compared to rest, which is at 50% between full flexion and full extension. I – index finger group, M – MRS finger group, F – flexion, E – extension, R – rest. (C) An example tuning curve from monkey N illustrating an SBP channel tuned to index extension and MRS flexion movements. Asterisks indicate significant difference from the average activity across the experiment (two-sided two-sample Kolmogorov-Smirnov test, p < 0.001, corrected for false discovery rate). (D) Photographs of monkey W’s and monkey N’s intracortical Utah microelectrode array implants. Both implants were in right hemisphere. * indicates arrays used in this study. A – anterior, L – lateral, CS – central sulcus.

Figure 2. Two-finger closed-loop Kalman filter decodes using spiking band power (SBP).

(A, B) Example closed-loop prediction traces from monkeys N and W using the standard Kalman filter, respectively. Targets are represented by the dashed boxes, internally colored to indicate the targeted finger with a border color representing whether the trial was acquired successfully (green, red if not). “I” means the index finger group and “MRS” means the middle/ring/small finger group. The mean path efficiency of the trials displayed in each window is presented at the top right. (C) Statistics for all closed-loop two-finger Kalman filter trials for monkeys N (left) and W (right). The red lines indicate the means, which are numerically displayed above each set of data, along with standard deviation. The statistic for each trial is represented by one dot in each plot, split into columns per monkey. “Succ Rate” means the percentage of total trials that were successfully acquired in time and “Path Eff” means path efficiency.

Figure 3. Two-finger closed-loop ReFIT Kalman filter decodes.

(A, B) Example closed-loop prediction traces from monkeys N and W, respectively, using the ReFIT Kalman filter. Targets are represented by the dashed boxes, internally colored to indicate the targeted finger. “I” means the index finger group and “MRS” means the middle/ring/small finger group. The mean path efficiency of the trials displayed is presented at the bottom right. (C) Statistics for all closed-loop two-finger ReFIT Kalman filter decodes for monkeys N (left) and W (right). The red lines indicate the means, which are numerically displayed above each set of data, along with standard deviation. The statistic for each trial is represented by one dot in each plot. “Succ Rate” means the percentage of total trials that were successfully acquired in time and “Path Eff” means path efficiency. (D) Statistics of each type of velocity reorientation for ReFIT training with monkey N (see STAR Methods). “N” means velocities for each finger were negated if not pointing to that finger’s target, “R” means the velocities were rotated in the two-dimensional finger space towards the target, and “B” means both reorientations were used by concatenating velocities modified by N and R and repeating the neural activity. Asterisks indicate significance (p < 0.01, two-tailed two-sample t-test).

We found that the monkeys could successfully control the movements of both finger groups independently in real-time using a linear Kalman filter. Figure 2A and B show predicted finger traces from monkeys N and W using the SBP Kalman filter, respectively. The sections of predictions displayed were chosen specifically to show a wide variety of targets. These traces demonstrate that the monkeys could individuate the two fingers independently with smooth and controlled effort using the brain-machine interface. Figure 2C shows the closed-loop statistics for the two-finger SBP Kalman filter. Regarding path efficiency, monkey N’s paths to targets were 61% efficient on average, less than the average 76% efficiency when controlling the virtual hand using the manipulandum. Monkey W acquired targets with an average efficiency of 53% using the decoder, also less than the average 63% efficiency when using the manipulandum. With respect to acquisition time, monkey N reached the target in an average 0.98s with an average orbiting time of 0.86s, greater than the average 0.51s time to target and 0.35s average orbiting time in manipulandum control. Monkey W achieved an average 1.3s time to target with a 1.0s average orbiting time using the Kalman filter, also higher than the average 0.79s time to target and 0.65s orbiting time when using the manipulandum. For completeness, Supplementary Figure 2 illustrates the same for threshold crossing rate rather than SBP for monkey N.

Out of interest for applications to brain-machine interfaces, we attempted to maximize closed-loop, two-finger decoding performance using the state-of-the-art recalibrated feedback intention-trained (ReFIT) Kalman filter (RFKF; Gilja et al., 2012; Vaskov et al., 2018). The RFKF training procedure occurs following the monkey’s usage of the original Kalman filter. It assumes that the neurons controlling the Kalman filter’s predictions represented the intention of the monkey to optimally bring the fingers to the targets, regardless of the directions of the predictions. Then after reorienting predictions to match the presumed intentions of the monkey, the linear model is retrained. As discussed in the methods, the two interpretations of our finger task result in two frameworks for retraining: rotation of the net velocity in two-dimensional finger space to back-calculate each finger’s intended velocity (similar to the original ReFIT method) or independent negation of each finger’s velocity if it is moving away from the target. The statistics comparing these two methods and a third combining both are shown in Figure 3D for monkey N. Ultimately, there were few statistical differences between the recalibration methods (p-values for the following order of comparisons: rotation and negation, negation and both, rotation and both; p = 0.34, p = 0.72, p = 0.62 for path efficiency comparisons; p = 0.013, p = 0.80, p = 0.064 for time to target comparisons; p = 0.0044, p = 0.036, p = 0.81 for orbiting time comparisons; two-tailed two-sample t-test).

For monkey N, the RFKF made a substantial improvement in decode performance over the standard Kalman filter. Figure 3A shows closed-loop prediction traces from monkey N using the SBP RFKF. In comparison to the Kalman filter, monkey N’s ReFIT Kalman filters significantly improved prediction performance (68% versus 61% path efficiency, 0.73s versus 0.99s time to target, 0.28s vs 0.86s orbiting time, all significant with p < 0.001, two-tailed two-sample t-test). Additionally, the predictions are less oscillatory when attempting to stop on a target, and when it is oscillating, the amplitude is generally smaller than the standard SBP Kalman filter, aligning with what was previously reported (Gilja et al., 2012, 2015; Vaskov et al., 2018). To showcase this, Supplementary Video 1 presents median performance from monkey N’s usage of the SBP RFKF in real-time with a general comparison to manipulandum control. Supplementary Video 3 presents exemplary performance of the SBP RFKF including the IE+MF and IF+ME postures.

For monkey W, the RFKF did not improve performance over the standard Kalman filter (54% versus 53% path efficiency, p = 0.12, 1.23s versus 1.3s time to target, p = 0.069, 1.17s versus 1.0s orbiting time, p = 0.013, two-tailed two-sample t-test; Supplementary Video 2).

To explain this result, we analyzed the consistency of each monkey’s SBP channels between the three modes of controlling the virtual hand: manipulandum control, Kalman filter control using SBP, and RFKF control using SBP. After z-scoring the SBP used to generate all of the data in Figure 2 and Figure 3, we recomputed three linear regression matrices: one between SBP and the finger kinematics during manipulandum control, the second between SBP and the intention-corrected finger kinematics during Kalman filter control, and the third between SBP and the intention-corrected finger kinematics during RFKF control. From the regression coefficients trained for velocity, we estimated the preferred direction of each normalized SBP channel based on the vector [C_n,I, C_n,MRS] for coefficients C and channel n. Figure 4A illustrates the preferred directions for the three manipulandum-control channels with the highest magnitudes in each monkey. These illustrative channels demonstrate that the channels that are most impactful for predicting kinematics do not substantially change their encoding preferences between manipulandum control and brain control modes (Figure 4B and C). For monkey W, however, there is small variation in the tuning preferences, which may have impacted his capability of using the RFKF to exceed the performance of the Kalman filter. We believe these variations are the direct result of substantially lower motivation to perform the task compared to that of monkey N.

Figure 4. Analysis of changes in SBP channel tuning through different stages of online virtual hand control.

(A) Preferred directions for the three normalized SBP channels with the highest manipulandum-control magnitude on one day for each monkey. Solid arrows with square markers are the preferred direction and magnitude for each channel during manipulandum control (Manip.), dashed arrows with asterisk markers are the same during Kalman filter control (KF), and dotted arrows with circle markers are the same during ReFIT Kalman filter control (RF). (B, C) Changes in preferred direction (B) and magnitude (C) of the 5 highest-magnitude manipulandum control channels for each of the 6 days for each monkey.

Cortical Neurons Show Specificity to Individual Contractions

We found it surprising that such a complex hand task could be captured so well by a linear decoder. To look at whether these linear relationships hold within individual neurons, we constructed tuning curves of individual units across all eight finger postures at just the +30% magnitude to best guarantee consistent behavior and corresponding cortical activity. On one exclusive and representative day for each monkey, they performed the center-out task in manipulandum control for at least 30 continuous minutes while SBP and broadband activity were recorded synchronously. The broadband activity was spike sorted using Offline Sorter (Plexon, Dallas, TX, USA) to extract the firing rates belonging to sorted units. 388 trials for monkey N and 819 for monkey W were processed. We calculated the mean firing rate for each type of movement and plotted tuning curves. Illustrative tuning curves are displayed in the left plots of Supplementary Figure 3.

Initially, we characterized the tuning preferences of the 115 sorted units across both monkeys, which are summarized in Supplementary Table 2 and Supplementary Table 3. First, in Supplementary Table 2 regarding specificity of neural activation to flexion and extension, we found 42 of the 115 units showed specificity to one of the finger flexors or finger extensors, but not the other, suggesting cortical representation of distinct muscle contractions. Second, in Supplementary Table 3 regarding specificity to individuated finger group movements, we found that 27 of the 115 units showed tuning to only one finger group. Among the units that showed specificity to one group, there were approximately equal quantities tuned to movements of the index and MRS finger groups. Most units showed Gaussian-distributed firing rates about the mean firing rate for each movement, with 46 of the 115 units having at least one movement for which the normalized activations were not normally distributed (p < 0.001, two-sided one-sample Kolmogorov-Smirnov test, corrected for false discovery rate).

In addition to performing all tuning analyses with the standard sorted units, we also included tuning analyses using the 300–1,000Hz spiking band power (SBP). We have previously shown that filtering spiking signals in the 300–1,000Hz band provides a signal that is highly correlated with the activity of the highest signal-to-noise ratio (SNR) units on an electrode (Nason et al., 2020). This is particularly effective for electrodes with low SNRs, as the 300–1,000Hz band was found to balance the tradeoff between signal and noise power (Irwin et al., 2016). Therefore, we constructed tuning curves from this band, which enabled us to perform the same analyses on sharper tuning curves from primarily multiunit electrode recordings. This resulted in more sources of unit activity than the limited quantities of sortable units on a Utah microelectrode array. As such, we obtained nearly 40% more tuned neural features, from 59 of 115 tuned sorted units to 82 of 192 tuned SBP channels, though there may be substantial overlap in information content between sorted units and SBP. In an anecdotal investigation, we also found that the cross-correlation of SBP features is greater than that of sorted units, likely due to the similarity in background noise (biological in the form of low-amplitude neural spikes but possibly also thermal in origin; Lempka et al., 2011). However, the effect of this appears minute, as the tuning curves for these channels that are included in the left plots of each pair in Supplementary Figure 3 as well as Supplementary Table 2 and Supplementary Table 3 showcase similar preferences to sorted units. Most channels showed specificity to one muscle group and one finger group, with more representation of MRS movements than index movements in monkey N and vice versa in monkey W. Lastly, similar to the sorted units, almost all SBP channels showed Gaussian-distributed power levels about their respective means for all movements across all trials (14 of 192 channels were not normally distributed, p < 0.001, two-sided one-sample Kolmogorov-Smirnov test, corrected for false discovery rate).

Finger-Tuned Neural Activity Is Linear

To investigate how the neural activity relates to similar finger movements, we began with a classical cosine tuning analysis. Of the 92 tuned sorted units and 148 tuned SBP channels, 60 and 94, respectively, had significantly correlated sinusoidal fits (p < 0.05, Pearson’s correlation, corrected for false discovery rate). While sinusoidal tuning of a neuron to two-dimensional movements of a single limb may be reasonable (Georgopoulos et al., 1982), it may not be fundamental (Todorov, 2000), so the same rationale may not apply to movements of multiple limbs, or in our case, fingers. We have demonstrated that a linear Kalman filter enables online separation and individuation of two finger dimensions, so we sought to characterize the goodness of fit of several linear models.

First, we visually analyzed some tuning curves (like those in Figure 1C and Supplementary Figure 3) and noticed that the neural activity for any particular movement appeared to have a similar activation level for the most similar movements (the closest movements as plotted in Figure 1B and C). This motivated us to calculate how well the neural activity for each movement could linearly predict the neural activity for each other movement, independent of any cosine fits (see STAR Methods). Figure 5 illustrates this, where the activity on each grid’s vertical axis is regressed from all of the other activities as indicated on the horizontal axis. In these grids, more yellow (or blue) cells indicate greater (or more negative) regression coefficients for the activities on the horizontal axis to predict the activity on the vertical axis. Generally, we found that the activity corresponding to most movements could be strongly predicted by the activity of its most related movements (the yellow off-diagonal bands with substantial significance asterisks). For example, the SBP activity corresponding to monkey N’s MRS-only flexion (MF) can be predicted significantly by the activities corresponding to index extension/MRS flexion (IE+MF) and index & MRS flexion (IF+MF). This suggests a number of potential linear models that explain the relationships between similar behaviors of different finger groups, three of which are investigated in Supplementary Figures 3 and 4. Overall, this suggests that cortical units represent the combined movements of multiple finger groups in a similar way to how those same units represent the individual movements of those finger groups. However, the optimal combinations of similar movements are to unequally weight their contributions (as illustrated in Figure 5 and Supplementary Figure 5), explaining why the average model in Supplementary Figure 4 failed to significantly predict any tuning curves.

Figure 5. Predictability of the activity of each movement from the activities of other movements.

The top and bottom rows of axes represent monkeys N and W, respectively, and the left and right columns of axes represent SBP and sorted unit results, respectively. A cell in each set of axes is colored based on the predictor’s regression coefficient (horizontal axis) when predicting the activity of a movement (vertical axis). Asterisks indicate statistical significance (based on each coefficient’s t-score, p < 0.001).

Linear Models Generalize to Predict Untrained Finger Movements

From a decoding perspective, the similarity in neural activations between individual and combined movements may shed some light on the training data requirements for a high performance, multidimensional hand neural prosthesis. Specifically for our two-finger task, in addition to addressing the linearity of neural activity, might a decoder model need to be trained on both individual and combined finger group movements, or is the neural activity for one of the types of movements sufficient to generate a model that encompasses both?

To answer that question, we split the trials (1,018 from monkey N and 819 from monkey W) into two subsets: one representing individual finger group movements only and the other only combined movements of both finger groups. Then, we trained linear regressions with ridge regularization on each set of trials to predict finger positions using exclusively sorted units or SBP and tested them on the corresponding sorted units or SBP of the untrained set of trials. For example, we first trained regressions on individual movements of the index or MRS finger groups and used the trained algorithm to decode the combined movements of the index and MRS finger groups, and vice versa. Figure 6 illustrates the averaged predicted traces from split training for all trials of a given movement overlaid on the averaged predicted traces for that same movement using a decoder trained on all trials with cross-validation. Both monkeys’ traces show that the split-trained decodes mostly overlay the full-trained decodes, which suggests that finger individuation information in the population’s neural activity is preserved across movements, even for untrained behaviors.

Figure 6. Offline ridge regression decoding of all SBP channels, trained on either individual or combined finger group movements.

The central trace with shaded region are the average predicted behavior from all trials of the indicated movement ± standard deviation, aligned in time by movement onset (vertical gray line). Individual movements (two left columns) were decoded using a regression trained on combined movements (right two columns), and vice versa for combined movements (“Split Decode” dashed traces). The “Full Decode” solid traces represent the average decode given the full dataset to train the regression, with cross-validation. Blue traces correspond to the index group and yellow traces correspond to the MRS group. The yellow or blue lines near the top of each plot indicate significant differences between the mean predicted positions based on a bootstrap analysis on the differences (>95% one-sided confidence interval).

However, there are differences during some of the predictions (greater than a one-sided 95% confidence interval of a bootstrap analysis on the errors). Differences in these predicted traces align with the results found during the tuning curve analyses, where the neural activity of particular movements requires unequal contributions of the component movements. As such, by the assumption of linearity, there should be some loss in performance associated with testing on untrained behaviors, which is shown in Figure 6. Numerically, for the two monkeys ordered as (N, W), the split-trained SBP decoders achieved (90.5%, 82.4%) of the variance accounted for (VAF) by the full-trained SBP decoder and (95.1%, 90.9%) of the correlation coefficient of the full-trained SBP decoder. For the sorted unit decoders (Supplementary Figure 6), the monkeys achieved (85.9%, 82.7%) of the VAF by the full-trained sorted unit decoder and (92.7%, 90.9%) of the correlation coefficient of the full-trained sorted unit decoder. We included the full set of averaged decodes for the eight movements using sorted units for both monkeys in Supplementary Figure 6. To avoid any confounds from averaged traces, we also included plots of the individual trials’ decodes used to calculate the averages in Supplementary Figures 7–8.

Misaligned Postural and Movement Tuning in Finger-Related Cortical Units

It has been well described previously that finger postures can be predicted online from cortical units better than their movements (Aggarwal et al., 2013; Bansal et al., 2011; Okorokova et al., 2020; Vargas-Irwin et al., 2010), but our brain-machine interface used in Figure 2 and Figure 3 demonstrated high prediction performance for movements, not postures. This raises an intriguing question related to how finger-related cortical processing circuits and units simultaneously represent movement and posture. Here, we provide a brief investigation into the similarity between cortical tuning to finger postures and cortical tuning to finger movements within our two-dimensional task.

To increase the range of postures the monkey explored at a variety of velocities, we modified the center-out task to be randomized, as illustrated in Figure 1A. In this random task, the starting and ending target postures for each trial were randomly determined along the ranges of motion of each finger group, not exceeding a separation of 50% of the range of motion. On one exclusive day for monkey N and across two consecutive days for monkey W, we compared the SBP movement tuning curves between the center-out and random tasks, with examples shown in Figure 7A. We found that many of the movement preferences did not substantially change between tasks (11/40 normalized SBP channels with the highest trained linear regression weights for monkey N and 18/40 for monkey W were significantly different in preferred direction, 1000-iteration bootstrap, p < 0.01, corrected for false discovery rate). Of those that were significantly different (four examples in Figure 7A), the preferred movement visually appeared similar, indicating that any significant differences in preferred movement are likely to be small.

Figure 7. Comparison between postural and movement tunings of SBP during the random task with manipulandum control.

Dashed traces in all plots represent zero z-score. (A) Tuning curves for three channels from each monkey. Asterisks indicate significant differences in preferred direction determined via 1,000-iteration bootstrap, p < 0.01, corrected for false discovery rate. (B) Movement tuning overlaid on postural tuning. The color at each location, or posture, is the SBP activity for the channel above extracted from the target holding period (i.e. at zero velocity). The white line represents the preferred postural direction. The solid black trace represents that channel’s SBP activity for movements in each direction, smoothed across 10% of the trials. The black scale bar represents 0.5 z-score and the black line from center indicates the preferred movement direction. (C-E) Statistics of the plots in (B) for the 40 most impactful channels to a linear regression decoder. (C) Comparison between preferred movement and postural directions. Each line represents one channel. (D) Difference in angle between the preferred movement and postural directions. Each dot represents one channel. The width of the violin at each angle difference indicates smoothed relative density. n.s. means not significant, s. means significant with p < 0.01, corrected for false discovery rate. (E) Magnitude of difference between the maximum or minimum activity across all postures and rest. The numbers on top represent the median difference. Asterisks indicate statistical difference, p < 0.01, one-sided two-sample t-test. (F) Eight channels of z-scored electromyography across all postures during the hold periods.

To compare movement tuning with postural tuning, Figure 7B overlays the movement tuning curve on a two-dimensional postural tuning surface, where each two-dimensional point is colored based on the mean activity of that channel when the finger groups were stationary at the indicated posture. In those examples, many show alignments between the preferred movement direction (black line out from center) and the preferred postural direction (white line out from center), but surprisingly, most do not show such alignment. Figure 7C and D present some statistics for the 40 normalized SBP channels with the highest trained regression weights for each monkey. 28 of monkey N’s 40 channels and 14 of monkey W’s 40 channels had a difference between the preferred posture and preferred movement greater than 30°, despite the distributions of differences appearing normal about 0°. Furthermore, 31 of monkey N’s 40 channels and 19 of monkey W’s 40 channels had significantly different preferred directions between posture and movement (1,000-iteration bootstrap, p < 0.01, corrected for false discovery rate), affirming misalignment in the preferred posture and the preferred movement direction of a given cortical unit.

Investigating these example postural tuning surfaces further suggests that the strongest cortical representation of the postures occurs at the limits of the range of motion. To characterize this, for each of the same 40 channels from each monkey, we calculated two differences: the first between the maximum activity across all postures and the activity at rest, and the second between the activity at rest and the minimum activity across all postures. Figure 7E illustrates these differences, indicating that resting activity is much closer to the minimum than the maximum activity across all postures (p < 0.01, one-sided two-sample t-test). This outcome suggests that cortical SBP channels are generally positively tuned towards their preferred posture and increase in activity as a posture changes from a resting state towards its maximum magnitude. This outcome is further emphasized in Figure 7F, which plots the smoothed electromyographic activity recorded from bipolar intramuscular electrodes within monkey N’s forearm across all postures for each finger group. These traces are generally maximized at the edges of the range of motion with lower activity levels near 50% flexion, suggesting that our estimated resting posture is relatively close to the monkey’s true resting posture.

DISCUSSION

Modern hand neural prostheses have not yet been able to reproduce individuated finger movements across their entire ranges of motion. Here, we demonstrated that a linear Kalman filter with or without intention retraining is adept at decoding individuated, non-prehensile movements of an index finger and a middle-ring-small finger group in real-time with high-performance, given sufficient number of tuned units. We then showed that both monkeys could maintain or substantially improve their control performance using the ReFIT Kalman filter, which trained consistent weights per input channel with the prior decoder models. To address how this complex task can be explained with a linear decoder, we found that the neural activity of any particular movement can be reliably predicted from the activity of its related movements via a weighted sum, providing an explanation for our high performance linear decoding model. We validated this claim by showing that combined finger group movements can be predicted with a decoder model trained on individual finger group movements and vice versa. Only a slight performance loss was realized when comparing predictions of decoders trained on the full set of behaviors and decoders trained only on trials representing individual or combined finger movements. Finally, using our non-prehensile task allowed us to begin to uncover how cortical neural activity simultaneously encodes information about postures and changing postures, suggesting that preferred movement directions and preferred postures are frequently misaligned.

The complexity of our 2D task proposes interesting questions regarding the interpretations and intentions of the monkeys performing it. By analyzing the tuning preferences of cortical spiking activity during manipulandum, Kalman filter, and RFKF control modes, it seems that cortical spiking activity was very consistent between all three. Any changes in decoder weights can likely be attributed to differences in the training data between control methods rather than neurons altering their tuning. Additionally, we proposed two frameworks for retraining linear Kalman filters based on the monkeys’ intentions, one assuming each finger is its own task for two total objectives and the other assuming both fingers belong to one two-dimensional task. While both improved decoding performance substantially, neither strongly outperformed the other despite making clearly different assumptions about the monkeys’ interpretations of the task. As such, we think these data suggest two possible truths: the nuances of intention-based recalibration methods for the prediction of continuous two-finger movements will not significantly impact performance, or the optimal recalibration scheme for multiple finger movements remains unknown.

While others have found that decoding continuous postures of the hand achieves greater performance than decoding the movements (Aggarwal et al., 2013; Bansal et al., 2011; Okorokova et al., 2020; Vargas-Irwin et al., 2010), we were nonetheless able to achieve high performance by decoding the movements directly. The non-prehensile task explored here enabled us to begin to uncover the relationship between posture and movement representations in single cortical units. We surprisingly found that a unit’s preferred posture is not necessarily related to that unit’s preferred movement direction. From a decoding perspective, we hypothesize that this could explain the high-performance control the monkeys achieved using the Kalman filters. If the SBP activity can simultaneously represent movement along one behavioral dimension and posture along an orthogonal behavioral dimension, even simple decoders like linear regressions can parse each channel to provide two pieces of helpful information. Substantial additional work will be necessary to make such conclusions, particularly because the implants used in our study heavily under-sample cortical populations.

After determining that we could predict the level of cortical activation during combined movements as weighted sums of the activations during individual movements, independent of any cosine fits, we validated this finding by predicting untrained, combined finger movements using linear regressions trained only on individual finger movements. We did not expect the positions predicted by the split decoder and the positions predicted by the full decoder to be so similar, which they were in many cases. We propose a few potential explanations. First, this could be attributed to using behaviors of different amplitudes (±20%, ±30%, and ±40% from rest, see Behavioral Task in Methods) for each direction, such that when plotting just the middle amplitude, the differences in predictions were negligible. Second, since the tuning curves showed graded activations to several related movements, combinations of the contributions from different units and SBP channels may be sufficient to predict movements of greater magnitude. Third, finger-related neural activity may have a minor non-linearity, either between units and SBP channels or with its relationship to the behavior, which has been reported previously (Naufel et al., 2019).

Our real-time decoding results suggest that linear models can accurately fit the movements of two independent fingers across their full ranges of motion from cortical spiking activity. Our previous study originally suggested that this may require nonlinear models (Vaskov et al., 2018). However, at that time, the monkeys had not been trained to voluntarily individuate their finger movements in manipulandum control mode and were not penalized for making undesired movements. In this manuscript, the monkeys received substantial additional training without using the brain-machine interface and with the necessary penalties for incorrect movements to voluntarily individuate their fingers, indicating linear models are sufficient for fitting such movements. Linear decoding of individuated finger movements has major implications for clinical neural prostheses. It has been previously suggested by unconstrained finger movements that primary motor cortex linearly predicts movements of the hand and fingers in monkeys (Aggarwal et al., 2013; Ethier et al., 2012; Kirsch et al., 2014; Okorokova et al., 2020) and humans (Ajiboye et al., 2017; Wodlinger et al., 2015) when many of the DoFs may move along very similar trajectories. Our results extend these to show that linear models can accurately predict the movements of two well-separated and independent finger DoFs across their continuous ranges of motion. On the front of individuation, some other groups have demonstrated that intracortical arrays can classify movements of individuated digits in humans (Bouton et al., 2016; Jorge et al., 2020). Our methods and results, taking advantage of able-bodied primates and the precise tracking of finger movements with the manipulandum, bring the state-of-the-art in finger individuation from discrete decoding to the realm of continuous decoding. Finger-related neural activity in primary motor cortex recorded with human-grade Utah microelectrode arrays can be sufficient to individuate at least two systematically separate DoFs using linear models. Further, linear individuation of two DoFs within the hand, as shown here, suggests the possibility of linear models sufficiently individuating more of the 27 DoFs within the hand, as has been hinted for low-magnitude movements (Kirsch et al., 2014).

To close the gap between the capabilities of hand neural prostheses and the natural hand, further work is needed to completely characterize the cortical representation of non-prehensile movements as well as how cortex manages movements that are simultaneously prehensile and non-prehensile (i.e. playing a guitar or rapidly solving a puzzle cube). Co-contraction of finger-related muscles makes characterizing the relationship between motor cortex activity and muscle activity difficult (Häger-Ross and Schieber, 2000; Lang and Schieber, 2004). While we have briefly investigated how cortical spiking activity and the electromyography of finger muscles corresponds to our primarily kinematic non-prehensile task, further investigation is required to uncover the impact variable forces have on the relationships characterized here. Usage of the thumb is particularly central to this question, which will be difficult to address with macaques because of the restricted functionality of their thumbs compared to those of humans.

In terms of clinical viability of a neural prosthesis, where usage outside of a laboratory, hospital, or rehabilitation environment is the ultimate goal, the computational simplicity and generality of linear models make them promising solutions. Decoders such as the Kalman filter, Wiener filter, or ridge regression (Collinger et al., 2013; Ethier et al., 2012; Malik et al., 2011) require surprisingly few computations per iteration, opening the possibility of implementation on portable or implantable devices. Additionally, our results suggesting that the activities of individual movements can be linearly combined into the activities of more complex movements hint that decoders may not need to be trained on the full suite of behaviors that they will be used to predict. Instead, training decoders on orthogonal behaviors that span the full behavioral space (such as a center-out task), with representative neural activity, may be all that is required. This may cut the 5–10 minutes of decoder training time drastically, potentially streamlining the daily calibration of an outside-of-laboratory neural prosthesis. Generally, the results presented here suggest that naturalistic hand and finger neural prostheses with many DoFs may be close to clinical translation using simple linear decoder models and simple training procedures.

Though the results presented here show that linear decoding models can predict the movements of individuated fingers with high performance, the decoders were unable to achieve the level of precision and control of the able-bodied hand. This is despite the monkeys partially moving their fingers during brain-control mode, hypothetically to assist with controlling the virtual fingers. To bridge that gap, decoders may need to account for a nonlinear relationship between cortical activity and behavior. Several nonlinear neural networks have been tested for brain-machine interfaces (Hosman et al., 2019; Pandarinath et al., 2018), though few have transitioned to testing online. An early online recurrent neural network (Sussillo et al., 2012) showed that neural architectures have promise in online decoding. However, due to the heavy computational requirements of neural networks per online prediction update, they must be optimized and significantly compressed before being considered for out-of-laboratory, portable, and implantable brain-machine interfaces. For example, the online recurrent neural network presented previously (Sussillo et al., 2012) required an estimated 144,000 to 225,000 multiplication operations per prediction update for their two-dimensional center-out arm reaches task, which is substantially more than the 505 multiplication operations required by a two-dimensional steady-state Kalman filter similar to what was used in this work (Malik et al., 2011). Therefore, it is valuable to characterize the limits of linear models in discriminating neural states with truly simultaneous movement of independent DoFs, as was presented with fingers in this work.

Importantly, the finger-related neural tuning models give insight into how primary motor cortex may represent the activity of a variety of related movements. Most neural units presented in this study did not show nonlinear specificity to one movement, but graded tuning to several related movements. In fact, most of the tuning curves fit the classical cosine tuning model demonstrated for arm reaches (Georgopoulos et al., 1982) and finger movements (Georgopoulos et al., 1999). Despite directional tuning with arm reaches fitting logically (the angles and magnitudes of movement can all be referenced to one limb in a radial task), we do not believe cosine tuning can represent the movements of multiple fingers. The physiological assumptions are broken by our task with two independent finger dimensions that have their own relatively independent muscles. Our task employs what are essentially two limbs (or fingers) traversing their own spaces with any given neuron capable of being tuned to both limbs, making musculoskeletal models seem like the more relevant explanation for the underlying neural activity (Todorov, 2000). While it is intuitive to conclude that the cosine tuning model expands to our two-finger task, we think our regression models (LR and LRO above) better explain how neurons can simultaneously encode movements of multiple independent limbs, though further investigation is required to investigate the role of sensory feedback in these consistent and linear relationships.

This raises a major question: how far can this weighted-average model be extended as we consider more DoFs? Considering the addition of the thumb, which is critical to hand use in primates and humans, the regressions may simply require fitting the new similar movements (i.e. thumb flexion in the case of predicting IF+MF+ thumb flexion). Further investigation will be required to determine if the regression model will hold in this case, as perhaps the addition of DoFs will require an exponentially increasing number of required component movements. For example, for IF+MF+thumb flexion, perhaps all of IF, MF, thumb flexion, IF+MF, IF+thumb flexion, and MF+thumb flexion may be required for accurate activity prediction. On the surface, these conclusions may appear to contrast with previous findings that cortical units strongly represent muscular synergies within the hand during awake use and during cortical microstimulation (Overduin et al., 2014, 2015; Saleh et al., 2010). We hypothesize that our findings complement this literature, where the shortcomings of our linear models (i.e. reduced performance when predicting combined finger movements with a model trained on individual movements) could possibly be better addressed with synergistic modeling. An alternative hypothesis is that the linear relationships we found between finger dimensions may arise from an alternative operating mode of cortex during non-prehensile movements, which may require more focused control to execute. In both instances, further work will be necessary to compare independent vs. synergistic cortical models related to finger movements (Kirsch et al., 2014; Mollazadeh et al., 2014).

In our work and those just mentioned, it is clear that finger-related cortical units often encode information about multiple digits. Several groups have investigated how neurons tuned to multiple behaviors respond to tasks requiring those behaviors for different extremities and across various species (Cross et al., 2020; Diedrichsen et al., 2013; Heming et al., 2019; Jorge et al., 2020; Stavisky et al., 2019, 2020; Willett et al., 2020). One way to explain this is that the neural activity underlying separate and combined behaviors can be explained by multiple orthogonal subspaces. Here, we have evidence to suggest that linear models can effectively combine at least two finger subspaces that are substantially related. Provided sufficient quantities of neurons representing those subspaces, we think compound movements may continue to be well represented by linear combinations of their component subspaces, up to the full dimensionality of the hand and beyond to the entire motor system.

STAR METHODS

Resource Availability

Lead Contact

Further information and requests for resources and reagents should be directed to and will be fulfilled by the lead contact, Cynthia Chestek (cchestek@umich.edu).

Materials Availability

This study did not generate new unique reagents.

Data and Code Availability

• A subset of the data used in this study can be found on the Chestek Lab website as of the date of publication (https://chestekresearch.engin.umich.edu/data-and-resources/). Due to the size of the complete set of data, additional data used in this study will be shared by the lead contact upon reasonable request.

• All original code has been deposited at https://doi.org/10.7302/jnkz-az17 and is publicly available as of the date of publication. DOIs are listed in the key resources table.

• Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.

KEY RESOURCES TABLE.

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Experimental models: Organisms/strains
Macaca Mulatta	World Primates, Inc. (Miami, FL, USA)	N/A
Software and algorithms
Matlab R2012b and R2018a	The Mathworks, Inc.	https://www.mathworks.com/products/matlab.html
Simulink/xPC Target 2012b	The Mathworks, Inc.	https://www.mathworks.com/products/simulink-real-time.html
Code and data to reproduce analyses and figures	Original code	https://doi.org/10.7302/jnkz-az17
Other
Cerebus Neural Signal Processor	Blackrock Microsystems, LLC.	https://www.blackrockmicro.com/products/#data-acquisition-systems
Utah microelectrode arrays	Blackrock Microsystems, LLC.	https://www.blackrockmicro.com/products/#electrodes

Experimental Model and Subject Details

Nonhuman Primates

All procedures were approved by the University of Michigan Institutional Animal Care and Use Committee.

All experiments were conducted with two monkeys (male, macaca mulatta). Monkey N was age 7 to 8 years and weighed between 12.7 and 16.8kg (weight range due to the extended COVID-19-related experiment break) during the period of data collection. Monkey W was age 8 years and weighed between 13.1 and 14.1kg during the period of data collection. The subjects were fed a standard laboratory animal diet with supplemental fresh fruits and vegetables. When possible, the animals were pair-housed, and were always provided frequent access to a variety of enrichment, including food puzzles, chewing toys, and television.

Method Details

Implants

We implanted two male rhesus macaques (monkey N age 7 to 8, monkey W age 8 at the time of data collection), with Utah microelectrode arrays (Blackrock Microsystems, Salt Lake City, UT, USA) in the hand area of primary motor cortex, as described previously (Irwin et al., 2017; Vaskov et al., 2018). Pictures of the implants are illustrated in Figure 1D. Only motor cortex arrays were used in this study. Monkey N’s motor cortex arrays are two 64-channel arrays, explaining why they are smaller in area than his sensory array. Monkey N was between 117 days and 708 days post-cortical implant for all data analyzed. Monkey W was between 78 days and 100 days post-cortical implant for all data analyzed.

In a separate surgery, we implanted monkey N with chronic bipolar intramuscular electromyography recording electrodes (similar to PermaLoc™ electrodes, Synapse Biomedical, Inc.). After induction of anesthesia, the monkey was positioned supine for easy access to his left arm. A single radial-volar incision was first used to access flexor muscles of the deep and superficial compartments of the forearm, following which a single dorsal-ulnar incision was used to access the extensor muscle of the forearm. For each muscle of interest, intra-operative neural stimulation of the muscle was performed to isolate finger-related and wrist-related actions of interest. Electrodes were secured intramuscularly using non-absorbable monofilament suture at a location in close proximity to the entry point of the innervating nerve. In instances where on neural stimulation, better isolation of intended muscular movement occurred at a site distal to the identified neural entry point, an additional electrode was secured at this distal site. Isolated muscles included the flexor digitorum profundus-index (1x near the nerve entry point, 1x distal near the wrist), flexor digitorum profundus-MRS (1x), flexor pollicis longus (1x, unused in this study), flexor carpi radialis (1x), flexor carpi ulnaris (1x), extensor digitorum communis (1x), extensor indicis proprius (1x), extensor carpi radialis brevis (1x), and extensor pollicis longus (1x, unused in this study). After all electrodes were secured to the muscles of interest, electrode wires were tunneled proximally to the upper arm using an incision on the posterior upper arm, posterior to the elbow, ensuring sufficient redundancy and laxity on the lengths of the wires to account for motion at the elbow. An additional interscapular incision was used as an exit site for the tunneled wires which were then connected to the standard PermaLoc™ connector. All incisions were closed in a layered fashion using absorbable sutures. Following the implantation of these electromyography recording electrodes, the monkey persistently wore a Primate jacket (Lomir Biomedical, Inc.). Monkey N was 36 days post-arm-implant for all electromyography data analyzed.

Feature Extraction

All processing was done in Matlab versions 2012b or 2018a (Mathworks, Natick, MA, USA), except where noted.

Threshold crossing rates were processed and synchronized in real-time during the experiments (see the subsequent section for a description of data flow). We configured the Cerebus neural signal processor (Blackrock Microsystems) to extract voltage snippets that crossed a −4.5 times the root-mean-square (RMS) threshold, customized to each channel. Then, these waveforms were streamed to a computer running xPC Target version 2012b (Mathworks), which logged the source channel of each spike and the time of each spike’s arrival relative to all other real-time experimental information. Both monkeys had 96 channels of threshold crossing rate data analyzed, though for closed-loop decoding, channels were masked to those that were not clearly disconnected and had contained morphological spikes during the experiment or at some time in the past (see SBP section below for reasoning).

We also extracted sorted unit firing rates for offline analyses. We imported the relevant broadband (0.1Hz – 7.5kHz sampled at 30kSps) recordings into Offline Sorter version 3.3.5 (Plexon, Dallas, TX, USA). Then, we high-pass filtered the recordings with a 4-pole Butterworth filter with a cutoff frequency set to 250Hz. To sort clear units, we used the threshold level determined during the experiment by the Cerebus at −4.5RMS, then eliminated clearly artifactual threshold crossings. Then, we sorted the remaining spikes crossing the threshold individually and in combination with principal component analysis, clusters as determined by k-means or Gaussian mixture model clustering (as implemented in Offline Sorter), and visual inspection. The spike timings of each sorted unit were then re-synchronized with the experimental data offline. After all sorting, monkey N had 47 units and monkey W had 68 sorted units.

Spiking band power was also acquired in real-time by the same experimental system as threshold crossing rates. We configured the Cerebus to band-pass filter the raw signals to 300–1,000Hz using the Digital Filter Editor feature included in the Central Software Suite version 6.5.4 (Blackrock Microsystems), then sampled at 2kSps for SBP. The continuous data was streamed to the computer running xPC Target, which took the magnitude of the incoming data, summed all magnitudes acquired in each 1ms iteration, and stored the 1ms sums as well as the quantity of samples received each 1ms synchronized with all other real-time experimental information. This allowed offline and online binning of the neural activity with 1ms precision. As with threshold crossing rate, we masked channels for closed-loop decoding to those that were not clearly disconnected and had contained morphological spikes during the experiment or at some time in the past, as SBP could possibly extract firing rates of low-SNR units remaining represented on such channels (Nason et al., 2020).

Experimental Setup

The experimental apparatus used for these experiments is the same as described previously (Irwin et al., 2017; Nason et al., 2020; Vaskov et al., 2018). Briefly, the monkeys’ Utah arrays were connected to the patient cable (Blackrock Microsystems) and raw 0.1Hz-7.5kHz unfiltered broadband activity at 30kSps, 300–1,000Hz activity at 2kSps, and threshold crossings at a −4.5RMS threshold were extracted from the neural recordings by the Cerebus for storage. The 2kSps and threshold crossing features were streamed to the xPC Target computer in real-time via a User Datagram Protocol packet structure. The xPC Target computer coordinated several components of the experiments. It binned threshold crossings and SBP in customizable bin sizes, coordinated target presentation, acquired measured finger group positions from one flex sensor per group (FS-L-0073–103-ST, Spectra Symbol, Salt Lake City, UT, USA), and transmitted finger positions along with target locations to an additional computer simulating movements of a virtual monkey hand (MusculoSkeletal Modeling Software) (Davoodi et al., 2007). Task parameters, states, and neural features were stored in real-time for later offline analysis.

Behavioral Task

We trained monkeys N and W to acquire virtual targets with virtual fingers by moving their physical fingers in a more complex version of the two-finger task we published previously (Nason et al., 2020). During all sessions, the monkeys sat in a shielded chamber with their arms fixed at their sides flexed at 90 degrees at the elbow, resting on a table. The monkeys had their left hands placed in the manipulandum described previously (Vaskov et al., 2018). Each monkey sat in front of a computer monitor displaying the virtual hand model and targets described previously. The monkeys were trained to move one finger group independent of another finger group. Monkey N preferred to perform the task with his index finger individuated from his MRS fingers, but monkey W preferred his index and middle fingers to be individuated from his ring and small fingers. The fingers of the virtual hand were split into an index finger group and a MRS finger group for both monkeys, where the fingers of each group moved together. Monkey N’s index finger actuated the virtual index finger and his MRS fingers actuated the virtual MRS fingers. In monkey W’s case, the movements of his index and middle finger actuated the virtual index finger while the movements of his ring and small fingers actuated the virtual MRS fingers. We later validated anecdotally that there was no impact on performance due to the mismatch between monkey W’s preferred finger split and the split of the virtual fingers by training monkey W to split his fingers in the same way as monkey N (index and MRS) and the virtual hand.

Each trial began with one spherical target appearing along the one-dimensional movement arc of each finger group, for a total of two simultaneous targets. Each target occupied 15% of the full arc of motion of the virtual fingers, except where indicated otherwise. Targets were presented in one of two patterns.

The first pattern represents a classical center-out-and-back pattern when viewed from the two-dimensional behavioral space illustrated in Figure 1B (Georgopoulos et al., 1982). Every other target was presented at a rest position, 50% between full flexion and full extension. The non-rest targets were pseudo-randomly selected from the postures in Figure 1B and a magnitude of movement was pseudo-randomly chosen (+20%, +30%, or +40% from rest). IE+MF and IF+ME postures did not have a +40% movement magnitude, as that was too far of a split in the finger groups to be performed reliably.

The second pattern represents random targets by pseudo-randomly placing targets for each finger group. First, a pseudo-random finger separation fsep was generated between −50% and +50% of the range of motion of each finger group. Then, a pseudo-random central position cpos was generated in the range of abs(fsep)/2 to 1-abs(fsep)/2. The index finger group was given a target at cpos+fsep/2 and the MRS finger group was given a target at cpos−fsep/2.

The presentation order of the targets was random, though the same order was repeated across sets of trials. Approximately one year after data collection, several experiments with a true-random order of the targets validated no change in performance, including Supplementary Video 3. For a successful trial, the monkey was required to move the virtual fingers into their respective targets and remain there for 752ms continuously in manipulandum control or 502ms continuously in brain control, except where otherwise noted. The additional 2ms was an artifact of a minor but long-standing bug that required targets be held for 2ms longer than requested. Upon successful trial completion, the monkeys received a juice reward. Closed-loop decoding experiments used center-out target patterns without the IE+MF and IF+ME posture styles, as the data was collected prior to conception of those postures. At a later time, we trained and tested a ReFIT Kalman filter on the full set of targets in the center-out style (including the IE+MF and IF+ME targets). Supplementary Video 3 validates that monkey N maintained high-performance despite the addition of these two posture styles. Tuning analyses used both center-out target patterns and random target patterns.

Neural Activity Normalization

All of our tuning analyses used normalized neural activity. For each sorted unit or SBP channel, we computed the mean, standard deviation, and root-mean-square activity level across the entire experiment in 300ms bins. Then, we eliminated all trials that were not center-to-out (unless the analysis used the random target pattern), that were unsuccessful, or had a duration longer than 3.0s for monkey N, indicating possible distraction, struggle, or other factors that may confound tuning analyses.

For each trial, we estimated the onset of native movements by the following procedure. First, for all finger groups that were not in the target at the beginning of the trial (which could happen in the random target pattern), we found each finger group’s main movement time, calculated as the first point in time at which the finger group passed 20% of the distance from its starting position to its ending position. Then, we found each finger group’s movement onset time, calculated as the point in time of the maximum jerk, or maximum change in acceleration, for each finger group prior to that group’s main movement. The trial’s movement onset time was selected as the earliest movement onset time across the active finger groups, where an active finger group is one which starts outside of its corresponding target. For trials in which all finger groups had starting positions within their corresponding targets, all finger groups were considered active. We found this procedure matched estimated movement onsets visually well for nearly all observed trials.

Given the movement onset, then for each included trial, we extracted the activity for each sorted unit or SBP channel from the period of time that maximized tuning depth for each analysis for each monkey, as the analyses depended on representative tuning curves. In all cases, we analyzed 300ms of activity about movement onset with a starting time between −100 to +100ms relative to movement onset. The specific starting times used for each analysis are detailed in that analysis’ methods section below. We then normalized these measurements by subtracting the unit’s or SBP channel’s mean then dividing by the standard deviation. This results in one normalized level of activity for each unit or SBP channel for each trial.

Computation of True and Predicted Tuning Curves

We calculated tuning curves for each monkey using one isolated experiment during which monkey N performed 2,130 and monkey W performed 1,836 manipulandum control trials of the center-out-and-back task, with monkey W performing the task with targets encompassing 16.5% of the range of motion of each finger group. We eliminated trials that were not center-to-out, that were unsuccessful, and, for monkey N only, that had a duration longer than 3.0s, then calculated the mean activity for each movement direction. For each monkey, we selected a start time relative to movement onset for the 300ms analysis window that maximized the tuning depth, measured as the difference between maximum and minimum mean activities across all movements. We found this start time to be 50ms after movement onset for monkey N and 100ms prior to movement onset for monkey W. Then, we further eliminated all trials that had movement onset times less than the analysis window’s start time, leaving 1,025 trials for monkey N and 819 trials for monkey W. All of these trials were used for the ridge regression analyses (see section below), while only trials in which the monkeys moved their fingers ±30% from rest were used for tuning analyses to guarantee consistency in behavior and ideally consistency in neural activity (all of monkey W’s trials were ±30% from rest). We did not use the ±40% trials as we did not offer ±40% targets for the IE+MF nor the IF+ME postures due to difficulty of acquisition, leaving ±30% the greatest magnitude of movement with targets presented for all postures.

To compute sinusoidal tuning curves, we first fit the amplitude, phase, and offset by linear regression as was done by others (Chestek et al., 2007). However, we found that this resulted in few sinusoidal tuning curves that correlated significantly with the measured tuning curves (26 of 148 modulated SBP channels and 31 of 92 modulated sorted units were significantly correlated). We anecdotally found this was a result of the period of the optimal tuning curves being unequal to 360 degrees. Therefore, to better represent sinusoidal tuning curves, we fit the amplitude, period, phase, and offset by using Matlab 2018a’s fminsearch function minimizing the total squared error over a maximum 1,000,000 iterations. These tuning curves yielded the results included above.

We tested three models of predicting the trial-averaged firing rates of all units corresponding to some movement given the trial-averaged activity of the units for other movements. The first naïvely assumes that the firing rates corresponding to one movement is the average of the firing rates of its component movements (i.e. the activity of IF+MF is the average of the activity of IF and the activity of MF). The second and third assume that the firing rate n of some movement m is a weighted sum of the firing rates corresponding to the other movements along with some constant offset c₀. This can be modeled by the following linear equation for one channel with an arbitrary k number of movements:

$$n_m=c_0+\sum _{i\in [1,k],i\ne m}{c}_i{n}_i$$

Let n_m,u be the firing rate from neural unit u for movement m. Then N_m is the vector of firing rates from all neural units corresponding to movement m and N_D is the training data matrix containing firing rates from all neural units corresponding to all of the movements that are not movement m:

$$N_m=\left[\begin{array}{c}{n}_{m,1}\\ n_{m,2}\\ ⋮\end{array}\right]$$

$$N_D=\left[\begin{array}{ccccccc}{n}_{1,1}& n_{2,1}& \cdots & n_{m-1,1}& n_{m+1,1}& \cdots & 1\\ n_{1,2}& n_{2,2}& \cdots & n_{m-1,2}& n_{m+1,2}& \cdots & 1\\ ⋮& ⋮& \cdots & ⋮& ⋮& \ddots & ⋮\end{array}\right]$$

As such, we can solve for the vector of coefficients for one movement m by solving the following linear regression equation using the data given by all valid units:

$$C_m=N_D^T{\left(N_D{N}_D^T\right)}^{-1}{N}_m$$

The three models we tested assumed that n_m is composed of some amounts of its component movements, which are n_m−1 and n_m+1 in this formulation. The first model (Avg) assumes n_m is the average of n_m−1 and n_m+1, requiring no regression to learn weights, as illustrated in 817 equation (1) below:

$$n_m=\frac{n_{m-1}+n_{m+1}}{2}$$ (1)

The second model (LR) assumed that n_m is composed of some learned weighted amounts of n_m−1, n_m+1, and a learned offset, as illustrated in equation (2) below

$$n_m=c_{m-1}\cdot n_{m-1}+c_{m+1}\cdot n_{m+1}+c_0$$ (2)

where c_i are weights learned by regression. Finally, the third model (LRO) assumed that n_m is composed of some learned weighted amounts of n_m−1, n_m+1, a learned offset, and the firing rates corresponding to the opposite movement (included as a regression method due to the weak yellow diagonal four diagonals off of the main diagonal in the grids of Figure 5), as illustrated in equation (3) below:

$$n_m=c_{m-1}\cdot n_{m-1}+c_{m+1}\cdot n_{m+1}+c_{m+4}\cdot n_{m+4}+c_0$$ (3)

where c_m+4 is the weight learned for the opposite movement, in this case +4 due to there being eight movement directions in our center-out task. We performed these same procedures using SBP in place of the firing rates.

We predicted the neural activity for each movement based on the two regression models with leave-one-out cross-validation on the neural features. The cross-validated linear models for predicting sorted unit activity were trained only on the other sorted units within each monkey. The same was done for SBP channels. Sorted units and SBP channels that had no significantly tuned movements according to a two-sample two-tailed Kolmogorov-Smirnov test (p > 0.001) and sorted units with mean firing rates under 2Hz were excluded from analyses. This left a total of 148 SBP channels (92 from monkey N and 56 from monkey W) and 92 sorted units (35 from monkey N and 57 from monkey W). Significance of the predicted tuning curves was determined using the p-values associated with correlation against the null hypothesis that a predicted tuning curve was not significantly correlated with its measured tuning curve, corrected for false discovery rate.

Computation of Movement and Postural Tuning

We calculated movement and postural tuning curves for monkey N using one isolated manipulandum-control experiment and for monkey W using two isolated manipulandum-control experiments across two consecutive days. After eliminating unsuccessful trials, we found that analysis window start times of 100ms after movement onset for monkey N and 0ms after movement onset for monkey W optimized tuning depth. Eliminating trials with movement onset times less than each monkey’s analysis start time resulted in 1,353 center-out trials and 870 random trials for monkey N and 452 center-out trials and 352 random trials for monkey W.

To compare center-out and random tuning curves, we computed the preferred direction of each channel during each task via the following procedure. First, we normalized each channel’s SBP activity across each set of trials in 100ms bins, then we computed the Kalman filter observational model (the C matrix in the section below) for each task. We used the coefficients trained for each finger group’s velocity to estimate each channel’s preferred movement direction within each task, then found the difference in angle between the preferred movement directions. To determine significance, we performed a bootstrap analysis similar to that performed previously (Chestek et al., 2007). Briefly, we took the normalized SBP activity from each task computed before, then resampled equal quantities of samples from the center-out and the random set of trials twice. We trained two Kalman filter observational models and calculated each channel’s difference in preferred movement direction between the two trained models. We performed this procedure 1,000 times to estimate the null difference distribution for each SBP channel, then calculated the p-value of each channel’s true difference in preferred movement direction between center-out and random tasks based on those distributions. We performed false-discovery rate corrections across the 80 channels with the largest velocity coefficients across monkeys, 40 from each.

The postural tuning surfaces were computed from the normalized SBP activity during the target acquisition periods. For monkey N, the activity was extracted from 252ms to 752ms after onset of the target acquisition period. For monkey W’s center-out trials, the activity was extracted from 102ms to 302ms after onset of the target acquisition period (which was 302ms in duration), and for his random trials, the activity was extracted from 252ms to 502ms after onset of the target acquisition period. These time periods were chosen based on the task parameters to maximize the amount of SBP activity analyzed but keep the true finger velocities nearest to 0. We sorted all trials into postural bins of width 10%, ranging from 0% to 100% flexion of each finger group, based on the mean posture of each trial during the same analysis periods detailed above. Each channel’s SBP activity was averaged across all trials belonging to each postural bin, generating the postural tuning surface. The generated postural tuning surfaces were plotted using Matlab’s pcolor function and linearly interpolated between neighboring postures. The preferred postural direction for each channel was calculated from the Kalman filter observational model’s two trained coefficients for finger position.

To determine if the preferred postural direction was significantly different from the preferred movement direction, we performed a 1,000 iteration bootstrap analysis similar to what was detailed previously. Instead of computing preferred postural and movement directions from two different resamples of the random task data, the bootstrap preferred postural and movement directions were computed from the same set of resampled data to estimate the null difference distribution.

Decoding of Neural Activity

We executed two different algorithms to predict finger movements from neural activity.

Closed-Loop Kalman Filtering

First, to assess neural prosthetic performance in application, we gave the monkeys visual feedback of the decoders’ outputs during the behavioral task. For each closed-loop experimental session, the monkeys began by completing at least 350 trials with the virtual hand controlled directly by the movements of the manipulandum. The monkeys were required to acquire and hold the targets for 752ms continuously for a successful trial with a 10s trial timeout. The behavioral data (i.e. one-dimensional positions per finger group) were measured synchronously with the non-normalized neural features by the xPC Target computer. Then, we trained a standard position/velocity Kalman filter on this data binned at 32ms (which we found superior to other tested bin sizes), as described previously (Irwin et al., 2017), using Matlab version 2012b (Mathworks). For predictions of two finger dimensions, the Kalman filter assumed a kinematic state of one position and one velocity for each group:

$$x_t=\left[\begin{array}{c}{P}_I\\ P_{MRS}\\ V_I\\ V_{MRS}\\ I\end{array}\right]$$

The Kalman filter predicts the state at each timestep based on an optimal combination of two different predictions. The first is a prediction made based on the state of the previous timestep, and the second is a prediction made based on a comparison between the measured neural activity and that predicted by the predicted kinematics of the current timestep. This can be summarized by the following equations:

$${\widehat{x}}_{t\mid t-1}=A{\widehat{x}}_{t-1}$$

$${\widehat{x}}_t={\widehat{x}}_{t\mid t-1}+K_t\left(y_t-C{\widehat{x}}_{t\mid t-1}\right)$$

where y_t is a vector of neural features at the current time step, K_t is the Kalman gain balancing how much the neural activity should contribute to the final prediction, A is the state transition matrix, and C is a linear regression trained to convert kinematics to neural features. Training of these matrices was performed as described previously (Irwin et al., 2017) but extended to account for both index and MRS. The A matrix was fit to take the following form:

$$A=\left[\begin{array}{ccccc}1& 0& 1& 0& 0\\ 0& 1& 0& 1& 0\\ 0& 0& A_{V_I{V}_I}& A_{V_{MRS}{V}_I}& 0\\ 0& 0& A_{V_I{V}_{MRS}}& A_{V_{MRS}{V}_{MRS}}& 0\\ 0& 0& 0& 0& 1\end{array}\right]$$

After training the standard Kalman filter, we computed the Kalman filter’s predictions in real-time to actuate the virtual hand independent of the monkeys’ physical movements. For successful acquisition, the monkeys were required to acquire and hold the targets for 502ms continuously with a 10s trial timeout for monkey N or a 30s trial timeout for monkey W. The positions displayed with the virtual fingers were computed by adding the predicted velocity to the previous time step’s predicted position, with an initial position taken as the true position of the fingers at the time the Kalman filter began running. Monkey N executed some brain control trials with his physical fingers restricted from movement and others without restriction. Monkey W did not have his finger movements restricted during brain control mode. Both monkeys used both the standard Kalman filter and the dual-stage training ReFIT Kalman filter. To train the ReFIT Kalman filters, the monkeys used the standard Kalman filter in closed-loop for at least 250 trials with their fingers unrestricted. Then, we used one of three intention-training paradigms to train the ReFIT Kalman filter.

1. The first is a two-dimensional form of the ReFIT Kalman filter we previously proposed (Vaskov et al., 2018), where we negated the incorrectly predicted one-dimensional velocity for each finger and set the velocities of fingers within their targets to zero before recomputing the regression matrices. This option assumes the monkey viewed the task as two separate one-dimensional problems and is highly related to the visual cues.

2. The second training method is very similar to the original ReFIT Kalman filter training method (Gilja et al., 2012). In the two-dimensional behavioral space illustrated in Figure 1B, we rotated the net two-dimensional velocity towards the target’s center for every trial and calculated the component index and MRS velocities from the net velocity. This option assumes the monkey viewed the task as one two-dimensional problem and is highly related to our two-dimensional space in which the two one-dimensional targets are mapped to one two-dimensional target.

3. The third training method combines the first two. We concatenated the recalibrated predictions from each method to create a 2n × d predictions matrix, where n is the number of timesteps in the training data and d is the dimensionality of the behavior (5 in this case). Then, we concatenated one copy of the n × f neural feature matrix (f being the number of features) to match the size of the first dimension of the predictions matrix to recompute the regressions. This option assumes the monkey viewed the task as some combination of two one-dimensional tasks and one-two dimensional task, which reflects some of the behavior seen in Supplementary Figure 1.

After training the ReFIT Kalman filter, we again computed the predictions in real-time using the new model and delivered visual feedback of the predictions to the monkey by actuating the virtual hand.

We additionally sought to validate that inclusion of the IF+ME and IE+MF posture styles did not result in poor closed-loop control performance. Supplementary Video 3 illustrates exemplary but near typical performance from monkey N using the ReFIT Kalman filter (type 2) trained and tested on the center-out task including IF+ME and IE+MF targets. The differences between the ReFIT Kalman filter used in Supplementary Video 3 and the ReFIT Kalman filter described previously are that the sum of the previous time step’s predicted position and the current time step’s predicted velocity was used to update the positions of the virtual fingers as well as the positions in the Kalman state, there were 0 bins of lag included online (Irwin et al., 2017), and during ReFIT retraining the velocities were zeroed when inside the target.

Stability of Trained Kalman Filter Parameters

To compare the least-squares regression parameters that best represent the neural activity in each type of virtual hand control (manipulandum, Kalman filter, and RFKF), we computed preferred movement directions for each SBP channel. First, we normalized the SBP activity during each control method, then computed the regression weights that would transform the kinematic measurements into normalized SBP activity. In the cases of the Kalman filter and RFKF, the predicted kinematics were intention-corrected via rotation (type 2) as described in the previous section prior to regressing. The preferred movement directions for each SBP channel under each control method were extracted from the index and MRS velocity regression parameters.

Open-Loop Ridge Regression

When trying to gauge the generalization of linear models to untrained finger behaviors, we used ridge regression (with a regularization term λ = 0.001) to predict continuous finger positions as it does not depend on iteration for continued stability. The neural activity was binned every 100ms and 10 total bins were used per feature: one for the current time step and one for each of the previous 9 timesteps for a total 1s of neural data. Neural activity from before each trial’s beginning was assumed to be 0 for sorted units and each channel’s root-mean-square value for SBP, as had been done previously (Chestek et al., 2007).

After sorting all trials as previously described, we split them into two sets: individual and combined finger group movements. Then, we trained a regression model on one set and tested on the other. Additionally, we performed 10-fold cross-validation on the full dataset to gauge information loss without training on the full set of behaviors. To compare these, we performed a 100,000 iteration bootstrap analysis on the errors between the two decodes for all time points in each experiment. Then, if the error for any averaged sample was greater than the upper one-sided 95% confidence interval resulting from the bootstrap analysis, the split decode was deemed significantly different from the full decode. Correlation coefficients were computed as Pearson’s r between the predicted and measured behavior, and variances accounted for were computed according to the following equation:

$$VAF=\left(1-\frac{\mathrm{var}(y-\widehat{y})}{\mathrm{var}(y)}\right)$$

where y is the measured ground truth behavior, $\widehat{y}$ is the predicted behavior, and VAF is the variance of y accounted for by $\widehat{y}$. The variance accounted for percentages, as written in the results, were taken as the ratio between the variance accounted for by the split-trained decoders divided by the variance accounted for by the full-trained decoder, multiplied by 100. The same computation was executed for the correlation coefficient ratios.

Performance Metrics

To evaluate performance, we used four metrics. First, success rate was computed as a percentage of the total number of trials for which the monkeys successfully acquired each trial’s targets before the trial timed out (10s).

Second, path efficiency was computed as the straight-line distance between the starting position and the nearest point in the target in two-dimensional space (Figure 1B) divided by the total distance that was travelled in two-dimensional space until the first instance both finger groups were simultaneously in their respective targets. We implemented this metric to estimate the efficiency of all movements, though it could be used to assess the simultaneity of combined finger group movements. We have included Supplementary Figure 1 to indicate the average behavioral trajectories in two-dimensional space and demonstrate how simultaneous were the monkeys’ multi-finger movements. Trials in which both finger groups were never in their respective targets simultaneously were given a path efficiency of 0%. Trials where the starting positions were inside the targets and successful trials immediately following failed trials were excluded from path efficiency analyses to avoid artificially increasing path efficiency.

Third, time to target was computed as the total time between a trial’s beginning and the first instance both finger groups were simultaneously in their respective targets. Trials in which both finger groups never simultaneously reached their respective targets were given a time to target equal to the trial timeout period (10s). Successful trials immediately following a failure and trials where all finger groups began inside their respective targets were excluded from time to target analyses to avoid artificially decreasing mean time to target.

Fourth, orbiting time was computed as the time between the first instance both finger groups were simultaneously in their respective targets and the end of the trial, minus the hold time. Orbiting time is a measure of the stopping ability of a controller, where the optimal orbiting time of 0s means the target was acquired and held beginning with the first instance both fingers were in their respective targets. To avoid artificially lowering the mean orbit time, unsuccessful trials that had a corresponding time to target later than 9 seconds into the trial were excluded from orbiting time analyses. Trials that did not have an instance where both finger groups were simultaneously in their respective targets, trials immediately following failed trials, or trials where the starting positions were inside the targets were also excluded from orbiting time analyses as these types of trials could artificially decrease the mean orbiting time.

We opted to exclude the hold time from the calculations so that the computed metrics better represented the time taken to reach the targets. Note that we required the monkey to hold targets for longer time periods when in manipulandum control mode so that there would be sufficient representation of neural data corresponding to stopping when training the online decoder.

Quantification and Statistical Analysis

Statistical analysis for closed-loop decoding experiments (Linear Two-Finger Decoding in Real-Time in the Results section) used a two-sided two-sample t-test between trial statistics in Matlab with significance level p < 0.001. Statistical analysis for determining the normality of neural activations for finger movements (Cortical Neurons Show Specificity to Individual Contractions in the Results section) used a two-sided one-sample Kolmogorov-Smirnov test in Matlab with a significance level p < 0.001, corrected for false discovery rate. Statistical analysis for determining movement tuning (Finger-Tuned Neural Activity Is Linear in the Results section) used a two-sided two-sample Kolmogorov-Smirnov test compared to the mean activity across the experiment in Matlab with significance level p < 0.001, corrected for false discovery rate. Statistical analysis for fitting cosine and regression tuning curves to measured tuning curves (Finger-Tuned Neural Activity Is Linear in the Results section) used the p-values associated with Pearson’s correlation computed in Matlab with a significance level p < 0.05, corrected for false discovery rate. Statistical analysis for determining differences between the split-data decoders and the full-data decoders (Linear Models Generalize to Predict Untrained Finger Movements in the Results section) used a 100,000 iteration bootstrap analysis on the errors between the two decoders in Matlab with a significance level of greater than a one-sided 95% confidence interval. Statistical analysis for determining differences in tuning curves between the center-out and random tasks (Misaligned Postural and Movement Tuning in Finger-Related Cortical Units in the Results section) used a 1,000 iteration bootstrap analysis on the preferred directions for each channel determined via regression with a significance level of p < 0.01, corrected for false discovery rate among the 40 most-strongly-tuned channels across both monkeys (80 total). Statistical analysis for determining differences between preferred posture and preferred movement direction (Misaligned Postural and Movement Tuning in Finger-Related Cortical Units in the Results section) used a 1,000 iteration bootstrap analysis on the preferred postures and movement directions for each channel determined via regression with a significance level of p < 0.01, corrected for false discovery rate. Statistical analysis for determining differences between maximal and minimal posture-related activations of channels (Misaligned Postural and Movement Tuning in Finger-Related Cortical Units in the Results section) used a one-sided two-sample t-test with a significance level of p < 0.01.

Supplementary Material

3

Two-Finger Simultaneous Brain Control Using ReFIT Kalman Filter and SBP for Monkey N, Related to Figure 3

Video illustrating monkey N’s capability of controlling the virtual hand using the manipulandum or the ReFIT Kalman filter predicting from spiking band power. The left side of the video shows manipulandum control mode, and the right side of the video shows brain control mode. Average and trial target acquisition times represent the total time from target presentation to target acquisition, less the hold time (750ms in manipulandum control mode, 500ms in brain control mode). The videos are each played at full speed.

4

Two-Finger Simultaneous Brain Control Using ReFIT Kalman Filter and SBP for Monkey W, Related to Figure 3

Video illustrating monkey W’s capability of controlling the virtual hand using the manipulandum or the ReFIT Kalman filter predicting from spiking band power. The left side of the video shows manipulandum control mode, and the right side of the video shows brain control mode. Average and trial target acquisition times represent the total time from target presentation to target acquisition, less the hold time (750ms in manipulandum control mode, 500ms in brain control mode). The videos are each played at full speed.

5

Exemplary Two-Finger Simultaneous Brain Control Using ReFIT Kalman Filter and SBP for Monkey N, Related to Figure 3 and STAR Methods

An exemplary video illustrating monkey N’s capability of controlling the virtual hand using the manipulandum or the ReFIT Kalman filter predicting from spiking band power. This video additionally includes targets requiring the splitting of the fingers, i.e. index flexion with middle-ring-small extension or index extension with middle-ring-small flexion. The left side of the video shows manipulandum control mode, and the right side of the video shows brain control mode. Average and trial target acquisition times represent the total time from target presentation to target acquisition, less the hold time (750ms in manipulandum control mode, 500ms in brain control mode). The videos are each played at full speed.

Table 1.

Abbreviations

Abbr.	Complete Phrase	Description
SBP	Spiking Band Power	Mean-absolute value of the 300–1,000Hz filtered intracortical voltage recording
MRS	Middle/Ring/Small	One of the finger dimensions investigated, in which the middle, ring, and small digits moved together
R	Rest	A posture between flexion and extension where the monkey’s digits were both in a resting state
IF	Index Flexion	A posture in which the index finger group was flexed and the MRS finger group was at rest
IF+MF	Index Flexion + MRS Flexion	A posture in which both the index and MRS finger groups were flexed
MF	MRS Flexion	A posture in which the index finger group was at rest and the MRS finger group was flexed
IE+MF	Index Extension + MRS Flexion	A posture in which the index finger group was extended and the MRS finger group was flexed
IE	Index Extension	A posture in which the index finger group was extended and the MRS finger group was at rest
IE+ME	Index Extension + MRS Extension	A posture in which both the index and MRS finger groups were extended
ME	MRS Extension	A posture in which the index finger group was at rest and the MRS finger group was extended
IF+ME	Index Flexion + MRS Extension	A posture in which the index finger group was flexed and the MRS finger group was extended

HIGHLIGHTS.

• Simultaneous and independent brain-machine interface control of two finger groups

• Cortical tuning between manipulandum and brain-machine interface use is consistent

• Linear decoders can predict untrained finger movements

• Cortical units simultaneously encode multiple kinematic dimensions