Decoding of intended saccade direction in an oculomotor brain-computer interface

Abstract

To date, invasive brain-computer interface (BCI) research has largely focused on replacing lost limb functions using signals from of hand/arm areas of motor cortex. However, the oculomotor system may be better suited to BCI applications involving rapid serial selection from spatial targets, such as choosing from a set of possible words displayed on a computer screen in an augmentative and alternative communication (AAC) application. Here we describe a chronic intracortical BCI in monkeys to decode intended saccadic eye movement direction. Using activity from multiple frontal cortical areas, we could decode intended saccade direction in real time with high accuracy, particularly at contralateral locations. Accurate decoding was evident even at the beginning of the BCI session; no extensive BCI experience was necessary. High-frequency (80–500 Hz) local field potential magnitude provided the best performance, even over spiking activity, thus simplifying future BCI applications. Most of the information came from the frontal and supplementary eye fields, with relatively little contribution from dorsolateral prefrontal cortex. Our results support the feasibility of high-accuracy intracortical oculomotor BCIs that require little or no practice to operate and may be ideally suited for “point and click” computer operation as used in most current AAC systems.

1. Introduction

Brain-computer interfaces (BCIs) utilizing electrodes implanted in cerebral cortex have shown great promise for restoring lost motor function to severely paralyzed individuals. To date, research efforts involving intracortical BCIs have mostly involved implants in the hand/arm area of motor cortex in an effort to control movement of a computer cursor or robotic arm/hand, initially in non-human primates [1–5] and more recently in quadriplegic human participants[6,7]. Human movement capabilities extend far beyond movements of our hands and arms, and different portions of the motor and premotor cortices are specialized for different motor tasks. Accordingly, one would expect that motor cortical areas other than the hand/arm area may be better suited for many BCI tasks. For example, speech motor cortical regions may be better suited than the hand/arm area for controlling movements of a speech synthesizer [8].

Saccades are amongst the fastest and most accurate voluntary movements we make in our everyday lives. Plus, saccades are relatively simple compared to limb movements. They are discrete (step-like), ballistic, and highly stereotyped eye movements. The neural representation of saccade has the potential of direct endpoint decoding [9–11]. Thus, the motor and premotor cortical regions devoted to saccade control may be ideally suited for BCI-driven spatial target selection. Spatial target selection is central to our interfacing with a computer. We choose desired letter keys or menu items. Thus, a BCI capable of making rapid target selections would be of great utility for restoring communication capabilities in individuals suffering from near-total paralysis (as in locked-in syndrome, and the later stages of amyotrophic lateral sclerosis and multiple sclerosis) via an augmentative and alternative communication (AAC) software package.

In the current study, we investigated the use of the saccadic eye movement system for a BCI task involving selection of one of multiple spatially distributed targets on a computer screen (hereafter referred to as an oculomotor BCI; see also [12,13]), by predicting the intended saccade directions prior to saccade execution. Two monkeys were implanted chronically with multielectrode arrays in three cortical regions: dorsolateral prefrontal cortex (PFC), supplementary eye field (SEF), and frontal eye field (FEF). PFC, SEF, and FEF are frontal areas that have been implicated in spatial short-term memory, saccade planning, and saccade generation [14–19]. Saccades can be elicited by low-current electrical stimulation in both SEF and FEF [9–11], and FEF in particular has been shown to contain a vector representation of saccade endpoints [10], making these areas ideal for real-time prediction of saccade plans.

In addition to demonstrating proficient control of the oculomotor BCI, we performed a number of offline analyses to characterize the amount of saccade-related information contained in different brain regions and in different neural activity measures. Previous studies have used a number of different neural activity measures for BCI purposes, including multiunit/single-unit firing rates, local field potentials (LFPs) in different frequency bands, or hybrid signals [2,18,20–24], with differing conclusions about which is the most effective for decoding. Here we systematically test a variety of neural activity measures to assess their suitability for decoding intended saccadic eye movement directions.

2. Materials and Methods

2.1. Monkeys and implants

Two adult male monkeys (monkey C, Macaca fascicularis, 9kg, monkey J, Macaca mulatta, 11kg) were handled in accordance with National Institutes of Health guidelines and the Massachusetts Institute of Technology Committee on Animal Care. Prior to the recording and BCI sessions, the monkeys were trained to perform a memory-guided saccade task (described further below), which required them to hold the location of one out of six randomly chosen visual targets in memory during a delay period and saccade to the target location to receive a liquid reward. After they became proficient at the task, three 32-channel microelectrode arrays (Blackrock Microsystems) consisting of 32 microelectrodes spaced 400 μm apart and 1 mm in length were implanted unilaterally in dorsolateral prefrontal cortex (PFC), supplementary eye field (SEF), and frontal eye field (FEF) in each monkey. The implant sites were determined prior to surgery using structural magnetic resonance imaging and anatomical atlases. One monkey received implants in the left hemisphere and the other in the right hemisphere, the implant locations for both monkeys are sketched in figure 1.

Figure 1.

Implant locations for both monkeys. Ps: principal sulcus; As: arcuate sulcus; PFC: dorsolateral prefrontal cortex; SEF: supplementary eye field; FEF: frontal eye field.

2.2. Brain-computer interface paradigm

The BCI paradigm is a modified version of the memory-guided saccade task (figure 2A). Monkeys were required to fixate on a dot in the center of the screen, and were briefly (350 ms) presented with one out of six targets (evenly spaced apart at 12.5-degree eccentricity). Following the cue presentation, the monkeys had to keep the cued location in working memory for a 750 ms delay epoch while maintaining central fixation. At the end of the delay epoch, the fixation dot disappeared, signaling the end of enforced fixation. From here on, the task differed depending on which of two phases of the session was being performed: eye-control trials or BCI-control trials.

Figure 2.

BCI paradigm and decoding scheme. A: BCI paradigm. Each trial was initiated by central fixation. A cue randomly chosen from 6 possible locations appeared briefly on screen, followed by the delay epoch, during which the monkeys must hold the target location in working memory and neural signals were collected. The response epoch differed depending on the trial type. During eye-control trials, monkeys had to make saccades to the correct target location to receive liquid reward, and a green highlight was then placed on the correct target. If monkeys made saccades to a wrong target location, a red highlight was placed on that target followed by a 3-second time-out. During BCI-control trials, a neural decoder instead selected the target location and drove the same feedback as in the eye-control trials. Every session consisted of 600 (for monkey C) or 300 (for monkey J) successfully performed eye-control training trials before the BCI trials were initiated. Epoch durations are given in milliseconds below panels. B: Saccade trajectories during eye-control trials, one session from each monkey, color-coded by the cued target locations. Saccade trajectories are highly stereotyped, with very little variance across repeated saccades to the same target location.

In the first phase, eye-control trials, the monkeys performed a saccade to the target location held in memory when the fixation dot was removed. Saccade trajectories from example sessions for both monkeys are shown in figure 1B, color-coded by the cued target locations. In our experiments, both monkeys completed their saccades typically in less than 20 ms from saccade initiation. If the saccade was made to the cued location, the target was presented with a green highlight and a water reward was delivered. If the saccade was made to any other location (which happened very rarely in practice), the chosen target was presented with a red highlight and reward was withheld. 600 successful eye-control trials were required in the first phase for monkey C before progressing to the BCI-control trials phase.

Once the required number of eye-control trials had been successfully performed, a decoder was trained using data gathered from successfully performed eye-control trials with known saccade targets, and the session entered the BCI-control phase. The BCI-control trials of the task were similar to eye-control trials, except that target selection and the resulting reward were controlled by a neural decoder that predicted intended saccade location based on delay period neural activity. During BCI-control trials, monkeys were allowed to move freely following the disappearance of the fixation dot and/or the display of the BCI-chosen saccade direction. Reward was contingent only on decoder output. In other words, the trained decoder came ‘online’ and replaced overt eye movements for target selection in the BCI-control trials. Subsequent offline analysis indicated that training a decoder with more than 300 eye-control trials did not result in a substantial improvement in accuracy (average offline decoding accuracy of 10 sessions for 200 training trials: 62.97%, 300 trials: 67.81%, 400 trials: 69.66%, 500 trials: 69.96%, 600 trials: 70.66%, combined for both monkeys, using linear discriminant analysis decoder described below). Therefore only 300 eye-control (decoder training) trials were used for monkey J. Both monkeys were given the similar number of total trials per session. Since the number of eye-control trials differed (600 for monkey C, 300 for monkey J), the number of BCI-control trials per session differed, and was at least 700 for monkey C, and 1000 for monkey J.

2.3. Neural signal processing

2.3.1. Recording

From each electrode, we acquired both threshold-crossing spike waveforms and LFPs using a multichannel data acquisition system (Cerebus, Blackrock Microsystems). Spiking thresholds were set manually for each channel, and a spike was registered if the voltage trace crossed the threshold. LFPs were extracted with a fourth order Butterworth low-pass filter with a cut-off frequency of 500 Hz and recorded at 1 kHz. The choice of a relatively high cut-off frequency (500 Hz, compared to LFP cut-off frequencies of 200–300 Hz or lower in many studies) was motivated by pilot findings of significant saccade direction information in the 200–500 Hz range. Eye movements were monitored using an infrared eye tracking system (EyeLink 1000, SR Research) at a sampling rate of 1 kHz.

2.3.2. Online BCI paradigm

Electrode signals to be used for online decoding were collected during the first 720 ms of the delay epoch of the paradigm, leaving the final 30 ms of the delay epoch to allow for transformation of the electrical signals into decoder input and transmission of the decoder output to the computer display. High-frequency LFP was extracted online in Matlab (MathWorks) for every channel, by first performing common average referencing for each array, then applying a third order Butterworth bandpass filter with a passband from 80 to 500 Hz. Each channel’s band-passed LFP was transformed to provide inputs to the decoder by simply summing its magnitude (absolute value) across the 720 ms recording period. The 80–500 Hz range was chosen based on pilot results from monkey C, which indicated that this range provided the most information for decoding of intended eye movement direction. After completion of eye-control trials at the start of each session, channel selection was performed to help determine the most informative channels and reduce dimensionality of the input data structure for the decoder, using one-way analysis of variance (ANOVA, p < .05) with respect to saccade directions on data collected during eye-control trials. Only this subset of significantly informative channels was used for decoding the intended saccade target for the rest of the session, as offline analysis indicated that such channel selection generally enhanced decoder computational speed and performance for a variety of inputs and decoders (unpublished observation).

2.3.3. Offline neural measure comparison

In addition to the online BCI, a number of offline analyses were performed in order to compare different measures derived from the recorded electrical signal during the first 720 ms of the delay period as potential decoder inputs. A total of 11 measures were extracted from the recorded neural signals, grouped into three signal types: continuous LFP magnitude measures; discrete, spike-related measures based on threshold crossings and waveform shapes; and hybrid measures that combine the continuous and discrete measures.

Continuous LFP magnitude measures were extracted from 6 different commonly studied frequency bands: 1–4 Hz (delta), 4–8 Hz (theta), 8–13 Hz (alpha), 13–30 Hz (beta), 30–80 Hz (gamma), and 80–500 Hz (including high gamma¹). To calculate the LFP magnitude measure for each frequency band, the recorded electrical signals were filtered with a third order Butterworth bandpass filter with corresponding frequency bands, then the magnitudes of the filtered LFP signals were summed over the delay period to form one decoder input per recording channel. The LFP signal magnitudes were then log-transformed to obtain a more Gaussian-like distribution of inputs to the neural decoders described in the next subsection.

Three different discrete measures were obtained from the detected threshold crossing waveforms. To obtain single-unit (SU) spike counts for each electrode, the waveforms were manually sorted offline into single units using principal component analysis (Plexon Offline Sorter). Individual electrodes were found to have between 0 and 4 single units. Spike counts during the delay period for all detected single units acted as input to the SU decoder. Some threshold-crossing waveforms did not satisfy waveform shape and/or cluster distinctness requirements for identification as a distinct single unit. These waveforms, termed non-isolable spikes, were included in a second discrete signal measure that augmented SU spike counts with a count of non-isolable spikes for each electrode. This measure will be referred to as SU+. Finally, multiunit (MU) spike counts were formed for each electrode by summing all threshold-crossing waveforms occurring on that electrode (essentially pooling together all SU and non-isolable spikes).

In addition to these continuous and discrete measures, two hybrid measures were analyzed. Both utilized LFP magnitude in the 80–500 Hz range since this range was shown to provide the best decoding performance of the continuous measures. For the first hybrid measure, the continuous measure for each electrode was supplemented with SU spike counts for each identified SU. For the second hybrid measure, the continuous measure was supplemented with MU spike counts for each electrode.

For each signal measure, an ANOVA (p < .05) was applied to select informative channels to be used as input, and a leave-one-out cross-validation procedure was used to characterize the accuracy of an LDA decoder (described below) using the selected channels as input to the decoder. The first 300 eye-control trials were used for this analysis, in which 299 trials were used to train the decoder and the remaining trial was used to test it. This procedure was repeated 300 times (omitting a different trial from the training set each time), and decoder accuracy was computed as the average percentage of correctly decoded trials across these 300 iterations.

2.3.4. Spectrotemporal analysis of saccade direction information

To obtain a finer-grained analysis of the spectrotemporal contents of LFPs on different electrode channels, we used multi-taper methods [25] to compute a spectrogram for each channel during the delay period of each eye-control trial. Multi-taper parameters were: 5 tapers, half time-bandwidth product of 3 (TW = 3), and window size of 500 ms stepped every 20 ms. The explained variance by intended saccade direction for each time-frequency point of the spectrogram was then computed with a one-way ANOVA as a measure of the information about saccade direction in time-frequency resolved neural signals.

2.4. Decoders

2.4.1. Online BCI paradigm

The 80–500 Hz LFP magnitude signals measured during training trials were first used to identify electrode channels containing information about intended saccade direction (one-way ANOVA on delay-period activity, with respect to the cued saccade target directions; p < .05). The 80–500 Hz LFP magnitude on these p significant channels was then used as training data to compute the probability that observed p-dimensional data vectors X = (x₁, …, x_p) on each test trial belonged to each possible class (target direction). These probabilities were computed using a linear discriminant analysis (LDA) classifier (or decoder), which assumes neural activity for each class approximates a multivariate Gaussian distribution with a class-specific mean μ_k, but common covariance structure Σ across k = 1, …,6 target directions [26]:

$$P(X\mid k)=\frac{1}{\sqrt{{(2\pi )}^p\mid \mathit{\sum }\mid }}{e}^{\left(-\frac{1}{2}{(X-{\mathrm{\mu }}_k)}^T{\mathit{\sum }}^{-1}(X-{\mathrm{\mu }}_k)\right)}$$

Estimates of the Gaussian means μ_k and the covariance matrix Σ were calculated from training trials. On validation trials, the saccade direction with the highest probability P (X|k) was selected as the decoder’s estimate ŷ of the intended saccade direction. Note that, because target direction had uniform probability P(k) and the normalization constant P(X) is the same for all directions, this decision rule is identical to selection based on the posterior probability using Bayes Theorem:

$$P(\widehat{y}=k\mid X)=\frac{P(X\mid k)P(k)}{P(X)}$$

Selecting the direction with the highest posterior probability is equivalent to selecting the predicted class based on log likelihood ratios as is typically done in LDA. Thus for our decoding purpose, this is an equivalent formulation as the classical LDA. LDA variants have been previously used offline to decode reaching directions [21] and saccade directions [19,27]. An LDA decoder was chosen for the current BCI because of its simplicity, computational efficiency, and optimal performance based on pilot results from monkey C.

2.4.2. Offline decoder comparisons

To confirm that our conclusions were not dependent on the specific decoder used, we also performed offline analyses comparing LDA to four other decoders: naïve Bayes classification, linear regression, artificial neural network classification, and artificial neural network regression.

Like LDA, a Naïve Bayes classifier computes the posterior probability of each class (saccade target), selecting the class with the highest posterior probability [21]. However, it makes a distinct set of assumptions—that all features approximate Gaussian distributions with class-specific means and variances (loosening the common covariance assumption of LDA), but are independent from each other (i.e., zero covariance):

$$P(X\mid k)=\prod _{i=1}^p\frac{1}{\sqrt{2\pi {\sigma }_{ik}}}{e}^{-\frac{{(x_i-{\mathrm{\mu }}_{ik})}^2}{2{\sigma }_{ik}^2}}$$

Estimates of the Gaussian mean μ_ik and variance σ_ik for k = 1, …,6 target directions and for each feature i were calculated from training trials. As with LDA, on validation trials the target direction with the highest posterior probability was selected as intended saccade direction.

Linear regression decoding has been applied in several prior BCIs [6,20], and provides a continuous output, unlike the discrete classifiers described above. Regression coefficients mapping neural activity to the horizontal and vertical components of saccade target directions were estimated from eye-control trials using least squares:

$$\begin{gathered} B_h={(X^{T}X)}^{-1}{X}^T{y}_h \\ {B}_v={(X^{T}X)}^{-1}{X}^T{y}_v \end{gathered}$$

Where X represents the neural signal used as input and the y’s represent the horizontal (y_h = cos θ) and vertical (y_v = sin θ) components of the saccade direction θ. On validation trials, these coefficients were used to decode a continuous saccade direction:

$$\widehat{\theta }={tan}^{-1}({\widehat{y}}_v/{\widehat{y}}_h),{\widehat{y}}_v=X{B}_v,{\widehat{y}}_h=X{B}_h$$

Which was then discretized into the nearest one of the six possible target directions.

Artificial neural networks (ANN) have been successfully used to decode direction of intended movement in the reaching literature [20,28]. Here we tested both a discrete-output ANN that classified each saccade direction as a distinct class (ANN-c), and a continuous-output ANN that similarly decoded the horizontal and vertical components of saccade target directions like the regression decoder described above (ANN-r), both using a fully connected 3-layer feed-forward network consisting of one input layer, one hidden layer, and one output layer. The input layer consisted of nodes representing selected neural measures as input for the network. The hidden layer contained the same number of nodes as the input layer, each computing a weighted sum across input nodes (where x_i is the activation of the ith input node, and w_ij is the connection weight from input node i to hidden node j):

$$z_j=\sum _{i\in \text{input}}{x}_i{w}_{ij}$$

Then passing its output through a logistic activation function:

$$y_j=\frac{1}{1+e^{-z_j}}$$

The output layer differed between ANN-c and ANN-r decoders. For ANN-c, the output layer consisted of six output nodes with logistic activation functions, each representing one of the six saccade target directions, and the network was trained using the cross entropy loss function:

$$L=-\frac{1}{N}\sum _{n=1}^N\sum _{k=1}^6[y_{n,k}\mathit{log}{\widehat{y}}_{n,k}+(1-y_{n,k})\mathit{log}(1-{\widehat{y}}_{n,k})]$$

Where ŷ_n,k represents the k^th (out of 6 directions) output node’s activation level for the n^th (out of N) training sample, and y_n,k represents the desired output (1 if the node represents the correct saccade direction, 0 otherwise). For ANN-r, the output layer consisted of 2 nodes with linear activation functions, representing the horizontal and vertical components of the saccade direction. The ANN-r network was trained using the sum of squared error loss function:

$$L=\frac{1}{2}\sum _{n=1}^N\sum _{a=1}^{\{\mathit{hori}.,\mathit{vert}.\}}{(y_{a,n}-{\widehat{y}}_{a,n})}^2$$

Where ŷ_a,n is the activation level of the a^th output node (representing either the horizontal-axis or vertical-axis component) corresponding to the i^th (out of N) training sample, and y_an represents the desired output.

Both ANN-c and ANN-r were trained using backpropagation in conjunction with conjugate gradient descent optimization, during which errors were propagated throughout the neural network according to their respective loss functions. For ANN-c, intended saccade direction was decoded as the output node with the highest activation level in the output layer, while for ANN-r intended saccade direction was computed from its horizontal and vertical components, and then discretized into the nearest one of the six possible target directions.

Two versions of each decoder type were trained: one using the magnitude of the LFP in the 80–500 Hz frequency range, and one using the SU+ spike counts. Only channels with saccade direction information (one-way ANOVA, p < .05) in training trials were used as inputs for decoders. A leave-one-out cross-validation procedure using the first 300 eye-control trials was applied to characterize the accuracy of each of the 10 resulting decoders (5 decoder types x 2 input types), and decoder accuracy was computed as the average percentage of correctly decoded trials across these 300 iterations.

2.4.3. Mutual information

In addition to decoding accuracy, we also evaluated decoder performance using the mutual information (MI) between the decoder-predicted and actual saccade directions. MI measures how much information the decoder predictions Ŷ (including both correct and incorrectly decoded ones) can provide about true saccade directions Y. The entropy of the actual saccade direction distribution Y is:

$$H(Y)=-\sum _{k=1}^{6}P(y=k){log}_{2}P(y=k)$$

And the entropy of the actual saccade direction distribution Y conditional on decoded saccade direction distribution Ŷ is:

$$H(Y\mid \widehat{Y})=-\sum _{k=1}^6\sum _{c=1}^{6}P(\widehat{y}=c,y=k){log}_2\frac{P(\widehat{y}=c)}{P(\widehat{y}=c,y=k)}$$

Finally, the mutual information between Y and Ŷ is:

$$MI(Y,\widehat{Y})=H(Y)-H(Y\mid \widehat{Y})$$

MI is measured in bits, and for our six-class paradigm ranges from 0 to log₂ 6 ≈ 2.58 bits. All probabilities were computed directly from the empirical data values.

To illustrate the distinction between decoding accuracy and MI, suppose we have a 2-direction decoding task, and one hypothetical decoder can achieve 50% accuracy if it outputs the same direction all the time, without knowing anything about the data. Its MI would be 0 bits, indicating that it provides no information about the true direction. On the other hand, if another hypothetical decoder always predicted the direction opposite of the correct one, it would have a decoding accuracy of 0%, but its MI would be 1 bit, indicating a perfect relationship (anti-correlation) between the decoded and the true directions. Comparing the MIs between these two hypothetical decoders reveals that the second one had knowledge of the underlying data structure (albeit completely flipped from the true one). In this extreme toy example, the MI provides a more accurate assessment of the decoders from the perspective of model validation, and can serve as a useful complementary measure for decoder comparison. In practice, we found that MI and accuracy provided essentially identical results.

2.4.4. Comparison of saccade information in different brain regions

To assess how much each brain region (FEF, SEF, PFC) contributed to decoder performance, we performed offline analyses in which LDA decoders were generated for each brain region separately. For each brain region, decoders were constructed for each of the 6 continuous and 3 discrete neural measures described above. To assess whether delay-period results generalized to the peri-saccadic period, this analysis was carried out separately using data from: (i) the first 720 ms of the delay epoch, and (ii) 100 ms before to 100 ms after saccade onset. A leave-one-out cross-validation procedure using the first 300 eye-control trials was applied to characterize the accuracy of each decoder for each brain region in each epoch. Unlike previous analyses, no channel selection using ANOVA was performed since it would result in no usable channels for some areas (for instance area FEF in monkey C did not always have usable spikes during some sessions due to its low unit count). All channels were provided to the LDA decoders.

3. Results

3.1. Online BCI performance

Prior to the BCI sessions, both monkeys were proficient at the memory-delayed saccade task and were at near-perfect performance. Therefore, any incorrectly decoded trials were attributed to decoder error rather than behavioral error on the part of the monkey. Based on pilot results from monkey C, we determined that LFP power in the 80–500 Hz range was likely to provide the most informative neural measure for decoding purposes (see related analyses in next subsection). This measure was thus used for online performance in the BCI sessions.

The decoding accuracies for 10 consecutively recorded BCI sessions were 61.46% with SD ± 6.48% (MI: 1.03 ± .19 bits) for monkey J, and 71.84 ± 3.24% (MI: 1.42 ± .11 bits) for monkey C, both highly above chance (16.67%). Since both monkeys received implants unilaterally, we expected BCI performance to be better for targets in the visual field contralateral to the implant due to a stronger contralateral visual representation in the implanted areas of frontal cortex [10,29,30]. Figure 3 shows the breakdown of decoding accuracy for targets in contralateral and ipsilateral visual fields. Monkey J received implants in the right hemisphere, and decoding accuracy for contralateral targets was 76.64 ± 6.91%, while ipsilateral target decoding accuracy was 46.53 ± 13.55%. Monkey C received implants in the left hemisphere, with contralateral decoding accuracy of 87.7 ± 2.05% and ipsilateral decoding accuracy of 55.95 ± 5.3%. Contralateral targets had significantly higher decoding accuracies than ipsilateral targets for both monkeys (monkey C: p = .002, monkey J: p = .002, two-sided Wilcoxon signed-rank test). The confusion matrices in figure 3 show that the decoders tended to confuse targets in the ipsilateral visual field for both monkeys. For instance, the 180° target for monkey C was usually confused with the 240° target by the decoder, and the 0° target was confused with the 60° target by the decoder for monkey J.

Figure 3.

BCI performance summary. Top row: average BCI performance for contralateral and ipsilateral saccade targets across all 10 sessions, as well as all targets combined, for each monkey. Decoding accuracies of contralateral targets are significantly higher than ipsilateral targets, indicated by a star. Dashed line indicates chance-level accuracy. Bottom row: confusion matrices showing, for each monkey, the full distribution of decoder-predicted saccade directions for each actual cued direction. Each row (actual direction) is normalized to sum to 100%, so plotted values represent the percentage of trials for a given actual direction that each possible direction is predicted. Contralateral targets are highlighted in a dashed red line, and the percentage of trials for each (actual, predicted) target pair is labeled in gray.

3.2. Effect of practice

To investigate the effect of practice on BCI performance, we compared online decoding accuracy during the early BCI phase with the late BCI phase of the same session. The first 25% of BCI-control trials were extracted, and their online decoding accuracies were compared with performance of trials from the last 25% of the BCI-control trials within session, for all sessions. Monkey J performed better during the early BCI phase (early: 73.29 ± 5.96%, late: 52.16 ± 7.46%, p = .002, two-sided Wilcoxon signed-rank test), while monkey C’s performances remained stable over the BCI phase (early: 71.36 ± 4.92%, late: 71.91 ± 3.73%, p = .56, two-sided Wilcoxon signed-rank test). This difference in performance stability is likely related to the change in neural signals within the same session. High-frequency LFP signals from channels selected for decoder input changed by 1.97 ± 1.69% on average from early to late BCI phase for monkey C, while the signal changed by 3.16 ± 3.75% on average for monkey J. The signal drift of monkey J was significantly more pronounced than that of monkey C (p ≪ .01, two-sided Wilcoxon rank sum test), and likely contributed to its performance decrease within the same session. Across sessions, decoding accuracies did not change significantly for both monkeys (average difference of online decoding accuracies between the first and last 3 sessions for monkey C: 4.5%; monkey J: −1.76%). In another analysis to quantify BCI performance change, linear regression was performed on online BCI performance against day of the session, and the regression slopes were not significant (monkey C: p = .30, monkey J: p = .97), Together these results suggest that—unlike in many other BCI paradigms (see Discussion)—our oculomotor BCI paradigm did not result in improved performance with continued usage.

Inspection of eye movement traces revealed that monkey C gradually stopped making overt saccades to the cued location during BCI-control trials, while monkey J continued to produce overt eye movements in BCI-control trials even though these movements were not required to obtain reward. This difference is likely due to a slight paradigm difference between the two monkeys: the fixation dot remained on after the end of the delay epoch for monkey J (though fixation maintenance was not required), while for the second monkey the fixation dot promptly disappeared at the end of the delay epoch and there was a longer delay until reward feedback.

3.3. Neural measure comparison

The BCI literature has not reached a consensus regarding which measures of neural signals work best for decoding purposes. Studies using single-unit (SU) spikes, multiunit (MU) spikes, and LFPs have all shown promising results [18,21,24,27,31]. To compare neural signal types in the current BCI paradigm, we calculated leave-one-out cross validation accuracies in offline analysis, for LDA decoders trained using 11 different signal measures, including LFP power in 6 frequency bands: 1–4 Hz (delta), 4–8 Hz (theta), 8–13 Hz (alpha), 13–30 Hz (beta), 30–80 Hz (gamma), and 80–500 Hz (including high gamma); 3 spike-related measures: firing rates of sorted SU spikes (referred to as SU), sorted SU spikes supplemented by non-isolable spikes that did not satisfy waveform shape and/or cluster distinctness requirements to be identified as a distinct single unit (referred to as SU+), all spikes pooled together for each electrode (referred to as MU); and two hybrid measures that combine LFP and spike components: 80–500 Hz and SU+, 80–500 Hz and MU. Analyses were restricted to eye-control trials to avoid potential signal changes from monkeys adapting to the BCI paradigm. Qualitatively similar results were obtained for both monkeys; their decoder accuracies were combined in the analyses presented here.

Table 1 indicates the number of identified single units in each implanted area for each monkey. There are large differences between the number of isolated single units on each array and subjects. PFC contained the highest number of single units in monkey C, while for monkey J it was SEF. Offline decoder accuracies using LDA for all 11 signal measures are summarized in figure 4. Among the LFP measures tested, 80–500 Hz band power had the highest leave-one-out cross validation accuracies (82.32 ± 4.52%, MI 1.79 ± .15 bits), significantly higher than any other continuous or discrete measures (p ≪ .01 for all pairs, two-sided Wilcoxon signed-rank test). For the spike-related measures, SU+ had the highest decoding accuracies (62.02 ± 14.44%, MI 1.11 ± .42 bits), and statistical comparison of SU+ with SU revealed that decoding accuracies were significantly higher (p ≪ .01, two-sided Wilcoxon signed-rank test) when non-isolable spikes were included, as in the SU+ measure. Hybrid models that combined 80–500 Hz LFP with either SU+ or MU had decoder accuracies that were nearly identical to decoder accuracy when using only the 80–500 Hz LFP band (‘80–500 Hz LFP/SU+’: 82.29 ± 6.39%, MI 1.79 ± .21 bits, ‘80–500 Hz LFP/MUs’: 82.32 ± 5.66%, MI 1.79 ± .20 bits), suggesting that any additional information from spike-related measures was largely if not completely redundant with information in the 80–500 Hz LFP measure.

Table 1.

Daily single-unit counts in each implant region, summed over all electrodes on the same array, and averaged across sessions.

	PFC	SEF	FEF
Monkey C	33.1 ± 3.04	4 ± 0.94	0.5 ± 0.97
Monkey J	3.5 ± 1.08	42.2 ± 2.97	5.5 ± 1.65

Figure 4.

Offline decoding performance comparison of 6 continuous, 3 discrete, and 2 hybrid neural activity measures. Analysis is restricted to eye-control trials to avoid signal changes caused by monkeys adapting to the BCI paradigm. Leave-one-out cross validation accuracies were computed with LDA decoder. Results are combined for both monkeys, 10 sessions per monkey. Continuous signals are the raw magnitudes of the given frequency bands. Discrete signals consist of three types of firing rates: ‘SU’, firing rates of sorted single-units only; ‘SU+’ firing rates of single-units, augmented with the rate of all non-isolable threshold-crossing spikes as a distinct decoder input; ‘MU’, firing rate of multiunits (all threshold-crossing spikes on the same electrode). Hybrid signals combine continuous and discrete measures: ‘80–500 Hz/SU+’, high-frequency LFP and SU+; ‘80–500 Hz/MU’, high-frequency LFP and MU. Dashed line indicates chance level accuracy. Decoding accuracy of the 80–500 Hz band is significantly higher than all other signals (p ≪ 0.01, two-sided Wilcoxon signed-rank test) except hybrid signals.

3.4. Decoder comparison

The analyses described thus far utilized LDA decoders. To ensure that our results are not specific to the decoder used, in additional offline analyses we compared performance of LDA decoders to four other decoder types: discrete artificial neural network classification (ANN-c), continuous artificial neural network regression (ANN-r), naïve Bayes, and linear regression decoders. Figure 5 summarizes each decoder’s accuracy using the 80–500 Hz LFP signal (left panel) and SU+ spike count signal (right panel), and table 2 shows the corresponding mutual information values. Among different signal types, 80–500 Hz LFP performed the best for all five decoders, indicating that it is a robust and informative signal, independent of the specific choice of decoder.

Figure 5.

Offline decoding performance comparison of different decoders. Results from leave-one-out cross validation are shown separately for each monkey (A: monkey C; B: monkey J), 10 sessions per monkey. 80–500Hz LFP: 80–500Hz raw band power. ‘SU+’: firing rates of single-units, augmented with the rate of all non-isolable threshold-crossing spikes, as a distinct decoder input. Dashed lines indicate chance level accuracy.

Table 2.

Mutual information (bits) of different decoders for 80–500 Hz LFP, and SU+ inputs.

	Monkey C		Monkey J
	80–500 Hz LFP	SU+	80–500 Hz LFP SU+
LDA	1.74 ± .12	.72 ± .09	1.82 ± .16	1.50 ± .20
ANN-c	1.62 ± .13	.64 ± .10	1.61 ± .16	1.42 ± .19
ANN-r	1.37 ± .14	.47 ± .10	1.26 ± .15	1.19 ± .19
Naïve Bayes	1.33 ± .19	.40 ± .07	1.32 ± .12	1.13 ± .16
Regression	1.58 ± .14	.69 ± .09	1.57 ± .15	1.37 ± .14

Of the five decoders tested, LDA consistently performed best for both continuous and discrete signals. While this is consistent with previous results showing optimal performance of simple linear models in decoding neural spiking data [32] we do not wish to make any general claims about the optimality of LDA. It might be the case, for example, that larger training datasets (compared to our few hundred trials) might support more complex, nonlinear models. Our results do suggest that LDA—which is simple and computationally inexpensive—produces reasonably good decoding accuracy with a relatively small number of training observations.

3.5. Brain region comparison

To determine how saccade information is distributed across the brain regions studied here, LDA decoders utilizing signals from only one brain region (FEF, SEF, or PFC) were constructed and evaluated using leave-one-out cross-validation to compute decoder accuracy offline. Figure 6 provides the decoder accuracies for each of the 6 continuous and 3 discrete neural signal measures considered herein. Decoder accuracies provided in figure 6A are for data collected during the first 720 ms of the delay epoch; those in figure 6B are for data collected during the 200 ms period centered around saccade initiation (response epoch). For monkey C, SEF provided the best decoding performance for all signal types, both during the delay epoch and the response epoch. For monkey J, SEF provided the best decoding performance for the three discrete signal measures, whereas FEF provided the best performance for the continuous signal measures; this was true for both the delay and response epochs.

Figure 6.

Offline decoding performance comparisons of implant regions during (A) the first 720 ms of the delay epoch and (B) the response epoch (100 ms before saccade onset to 100 ms after saccade onset), for different signal types. The first 300 trials during each session are used to obtain leave-one-out cross validation (decoding performance) for 9 signal types: 1–4 Hz band LFP; 4–8 Hz band LFP; 8–13 Hz band LFP; 30–80 Hz band LFP; 80–500 Hz band LFP; ‘SU’, firing rates of sorted single-units only; ‘SU+’, firing rates of single-units, augmented with the rate of all non-isolable threshold-crossing spikes, as a distinct decoder input; ‘MU’, firing rate of multiunits (all threshold crossings on the same channel, unsorted).

The number of identified single-units appears not to correlate with the decoding performance of high-frequency LFP in each implant region. PFC and FEF in monkey J both have relatively few identified single-units (see table 1), their decoding performances are worse than that of SEF when using discrete signals as input, but become equal or even better than SEF when using high-frequency LFP. This is also reflected in SEF in monkey C. Despite having much lower single-unit counts than PFC, SEF still outperforms PFC regardless of which input signal is used for monkey C.

3.6. Spectrotemporal analysis of saccade direction information

To examine the exact spectrotemporal distribution of information conveyed in LFPs about saccade direction, we computed multitaper power spectrograms for each electrode channel during the delay period of each eye-control trial, and computed how much of the data variance can be accounted for by saccade direction (using a one-way ANOVA) at each frequency and time point. Figure 7 shows the explained variance in the spectrograms of 4 example channels, with the first two panels corresponding to two electrode channels in monkey J and the last two panels to electrode channels in monkey C. Generally speaking, information about saccade direction appears to be spread across a broad frequency range, and different frequency ranges appear to encode information in different electrode channels. For example, the first channel from the left in figure 7 has most of the explained variance about saccade direction in frequencies above 300 Hz, while the last channel has the highest explained variance roughly between 80 and 200 Hz, particularly toward the end of the delay epoch. These factors may explain why the broad 80–500 Hz range provided the best decoder performance of all the neural measures compared (figure 4).

Figure 7.

Saccade direction information during the delay epoch. The percent of variance explained by saccade direction in the spectrogram data of 4 example channels (2 from each monkey) are shown here. Time is plotted relative to start of delay epoch; the entire delay epoch is plotted here. The ranges (minimum–maximum) of explained variances for all 4 spectrograms from left to right are: 0–18, 0–20, 0–30, and 0–10, respectively. Channel source from left to right: monkey J – channel 19 (SEF), monkey J – channel 64 (FEF), monkey C – channel 44 (SEF), monkey C – channel 60 (SEF).

4. Discussion

In contrast to previously developed BCIs that focus on replacing limb motor functions, the current study is one of the first to explore the potential of BCIs that involve microelectrodes implanted in the oculomotor system (see also 9, 10). Our results indicate that intended saccade directions can be predicted in real time with reasonably high accuracy (average of 66.7% correct in a 6-target task; 82.2% for contralateral targets), in the absence of any overt movement, from high-frequency LFPs recorded in frontal cortical regions, most notably SEF and FEF. This lays the groundwork for future BCIs that tap into the oculomotor system in order to perform tasks involving rapid serial selections from a set of spatially distributed targets. Such a system would be enormously useful for individuals suffering from locked-in syndrome since it could be used to rapidly navigate computer software, including AAC applications for restoring speech capabilities. We envision that an oculomotor BCI can be used either as a stand-alone system, or in combination with limb and/or other BCIs to provide an extra channel of control for AAC or for controlling movements of an external camera.

In addition to demonstrating the feasibility of an oculomotor BCI, we performed offline analyses aimed at identifying which oculomotor regions and neural activity measures provide the most information regarding intended saccade direction. While PFC, SEF, and FEF have all been implicated in voluntary oculomotor control, they have not been recorded in parallel and directly compared in the same experimental sessions before. We found that, overall, SEF and FEF provided the most informative signals for saccade target prediction. For both monkeys, neural spike rates from SEF were more informative than those from FEF and PFC, during both the memory delay epoch and the response epoch. Measures of LFP magnitude from SEF contained more information than those from FEF and PFC in one monkey, and slightly less than that from FEF in the other monkey, potentially due to differences in signal quality or individual variability. As oculomotor regions heavily implicated in saccade planning and execution, FEF and SEF share similarities, but also exhibit notable differences in their neural properties. Both regions have been shown to exhibit broadly similar saccade related activities [10,11,33], while SEF is also implicated in motor sequence planning [34,35] and shows stronger sequence effects than FEF [35,36]. SEF exhibits higher anti-saccade activities than pro-saccades [37]. FEF has long been known to encode saccade endpoints in vector form in retinocentric space [10,33], while SEF appears to encode saccade endpoints in multiple coordinate systems, and has been proposed to be involved in coordinate transformation [33,38]. FEF exhibits stronger connectivity than SEF with lower-order saccade generating structures [33,39], and requires less current to elicit saccades than SEF under electrical stimulation [33]. These findings indicate SEF is likely upstream from FEF, and more involved in the preparation rather than the motor production aspects of saccade generation. The high decoding performance in SEF even during the response epoch is notable given the above consideration. Our results suggest SEF plays a role no less significant than FEF in the execution of saccade, and can be a rich source signal for oculomotor BCIs.

Of the three brain regions studied, we found the least information regarding saccade direction in PFC for both monkeys. PFC has been implicated in memory-guided saccades, working memory, decision processes related to saccades[15,40,41]. PFC neurons also exhibit directional tuning [42]. Its neural signals have been used to decode saccade targets successfully offline [18,19] - authors in [18] targeted PFC and FEF together, and authors in [19] targeted specifically PFC. However, its exact role in saccadic tasks is less understood than that of FEF and SEF, and the current findings suggest that it contains relatively little stable information regarding saccade direction in the delayed saccade task.

To identify the best neural measures for decoding intended saccade direction, we compared decoding performance for a set of discrete (spike-related), continuous (LFP-related), and hybrid neural signal measures. The reaching literature lacks consensus regarding whether single-neuron spike rates, multiunit spike rates, LFPs, or combinations of these signals are best for decoding purposes. Although spike rates have generally been used in arm movement BCIs [2,20], conflicting results exist concerning whether spike rates or LFPs provide better signal measures for decoding. For example, while authors in [23] found spike rates to be superior to LFPs (including 100–200 Hz, 200–400 Hz bands) in the primary motor cortex (M1) and ventral premotor cortex (PMv) for decoding 3D arm end point and reach kinematics, authors in [21] reported that in M1, LFPs (< 150 Hz) can be equally or more effective than spike rates at decoding arm trajectories, and authors in [43] found what they called multiunit activities between 300 Hz and 6000 Hz in M1, dorsal premotor cortex (PMd) and PMv to be the best signal for decoding reach and grasp related parameters. Furthermore, some studies suggest better decoding performance can be achieved by combining spiking activity and LFPs into a hybrid measure [21,23,43].

Our results indicate that relevant decoding information is restricted to neither a specific frequency band nor a narrow temporal window within the delay period. Of the 11 neural measures compared, magnitude of the LFP in the relatively broad 80–500 Hz band (which we refer to as high-frequency LFP) provided the richest information of saccade direction intention across a range of decoders. The high-frequency LFP used herein differs from those found to be the most informative in other LFP-based decoding studies, which have reported a wide variety of frequency ranges including sub-ranges of 0–150 Hz [24], combinations of sub-ranges below 190 Hz [22], 25–90 Hz [27], 50–300 Hz [18], 70–200 Hz [44], 76–150 Hz [45], and 200–400 Hz [31]. Together with the current results, these findings suggest that the optimal neural activity measure for BCI purposes depends on the exact recording configuration used and brain areas implanted.

Our high-frequency LFP signals include the high gamma band (often defined as 70–150 Hz, though the upper bound can range up to 250 Hz). Prior studies indicate that power in the high gamma band is strongly correlated with population spiking activity near the recording electrode [46–50]. The superior performance of high-frequency LFPs compared to other neural measures in our study may be attributable to their success at capturing the population spiking activity, including the smaller spikes that are not picked up during the thresholding and spike-sorting process. This is corroborated by two results in our study. First, offline decoding accuracies using spiking signals improved when non-isolable spikes were also included (‘SU’ vs. ‘SU+’ in figure 4 and 5), indicating that useful signals were discarded as ‘noise’ during the spike sorting process. Second, for both monkeys, combining spiking activity with high-frequency LFPs does not result in more information than what already exists in the high-frequency LFPs (figure 4), indicating redundant information in the spiking signals and high-frequency LFPs. Relatedly, in an analysis not included in figure 4, performance of an LDA decoder using high-frequency LFP (average decoder accuracy 82.32% with SD ± 4.52%) was compared to performance using LFP in a more-traditional definition of the high gamma range (80–200 Hz, average decoder accuracy 69.87% ± 5.11%). The better performance of the high-frequency LFP indicates that substantial saccade information exists in the 200–500 Hz frequency range, also evidenced in the explained variances in this band (figure 7), perhaps because this range captures some aspect of population spiking activity not evident in the high gamma range.

It should be noted that, in one monkey, offline decoding performances in SEF using a discrete signal (‘SU+’) are comparable to those using high-frequency LFPs (figure 6A, monkey J). However, when all three implant regions are pooled together, high-frequency LFPs still provide better performance (figure 5B, monkey J). The contribution of individual implant regions to decoding performance do not appear to be additive, and potentially reflect synergistic interactions between all regions.

A major advantage of using LFP signals compared to spike-related signals in BCIs is their efficiency. They can be recorded at much lower sampling rates than spiking signals (≤ 1 kHz vs. ≥ 20 kHz), allowing use of lower cost, lower power recording configuration. Feature extraction can be performed very rapidly, requiring only bandpass filtering and rectification, in contrast to spiking signals, which require filtering, thresholding, waveform feature extraction, and spike sorting. In addition, spike sorting often requires end-user manual adjustment, which is obviated by using continuous signals. Thus, high-frequency LFP signals may allow for faster, lower cost, and easier to use BCIs.

To the best of our knowledge, there is only one other example of an eye-movement-based BCI in the current research literature. Graf and Andersen [13] used SU spikes collected from five individually adjustable microelectrodes located in a different part of the saccade system, the lateral intraparietal cortex (area LIP). An average decoder accuracy of 32.2% for a delayed saccade task involving 8 targets (chance level 12.5%) was reported. Although this is lower than the average decoder accuracy of 66.7% in our 6-target BCI paradigm (chance level of 16.7%), differences in the recording configuration, signal selection, and decoders involved in the two studies prohibit strong conclusions regarding the relative suitability of LIP for an oculomotor BCI as compared to the frontal cortical regions implanted here.

It is notable that, in contrast to most arm/hand area BCI studies, we did not see a significant performance increase with practice using the oculomotor BCI. Relatedly, when using a decoder with fixed parameters in their LIP-based oculomotor BCI, Graf and Andersen [13] found a steady decrease in performance over time rather than an increase that would be indicative of motor learning. Together these findings suggest that the delayed saccade task may not be amenable to inducing plasticity in the oculomotor system. This may be because performance is more limited by intrinsic constraints, such as the laterality of representation, in the oculomotor system, or it could be due to the relative sparse feedback compared to the continuous feedback in most arm/hand BCIs. Graf and Andersen [13] did note performance improvements within the same session with practice when using an adaptive decoder that continuously updated its parameters during BCI trials, but the degree to which these improvements reflected the non-stationarity of neural signals [51] versus changes in neural firing properties resulting from motor learning remains unclear.

Although our results provide an initial “proof of concept” for an oculomotor BCI involving implants in SEF and FEF, key limitations need to be addressed before such a system could be considered for human implantation. First, although our BCI’s performance was well above chance, it is not high enough for effective use of (for example) an AAC application to restore speech communication. This suggests the need for more electrode channels in order to obtain an acceptable accuracy level, particularly in the information-rich SEF and FEF regions. Second, we saw no evidence of performance improvement with practice, indicating that the delayed saccade paradigm may not be well-suited for capitalizing on motor learning processes that lead to improvements in performance in many limb-related BCIs. Third, our findings indicate that contralateral targets are more accurately predicted than ipsilateral targets, suggesting that bilateral implantation of FEF and SEF would provide improved performance across the entire visual field over unilateral implantation.