Toward High Performance, Weakly Invasive Brain Computer Interfaces Using Selective Visual Attention

Abstract

Brain–computer interfaces have been proposed as a solution for paralyzed persons to communicate and interact with their environment. However, the neural signals used for controlling such prostheses are often noisy and unreliable, resulting in a low performance of real-world applications. Here we propose neural signatures of selective visual attention in epidural recordings as a fast, reliable, and high-performance control signal for brain prostheses. We recorded epidural field potentials with chronically implanted electrode arrays from two macaque monkeys engaged in a shape-tracking task. For single trials, we classified the direction of attention to one of two visual stimuli based on spectral amplitude, coherence, and phase difference in time windows fixed relative to stimulus onset. Classification performances reached up to 99.9%, and the information about attentional states could be transferred at rates exceeding 580 bits/min. Good classification can already be achieved in time windows as short as 200 ms. The classification performance changed dynamically over the trial and modulated with the task's varying demands for attention. For all three signal features, the information about the direction of attention was contained in the γ-band. The most informative feature was spectral amplitude. Together, these findings establish a novel paradigm for constructing brain prostheses as, for example, virtual spelling boards, promising a major gain in performance and robustness for human brain–computer interfaces.

Introduction

Brain–computer interfaces (BCIs) use neural activity patterns to control external devices, such as virtual spelling boards or prosthetic arms (Lebedev and Nicolelis, 2006). Their success is tightly linked to the ability to extract rapidly and reliably detailed information based on the patient's recorded brain activity (Lebedev and Nicolelis, 2006; Ifft et al., 2012).

For this purpose, BCIs have been constructed using noninvasive or invasive recording technologies. Noninvasive technologies comprise EEG, near-infrared spectroscopy, MEG, or fMRI. Whereas MEG and fMRI are impractical for everyday use, EEG and near-infrared spectroscopy allow healthy subjects to control spellers (McFarland and Wolpaw, 2011), to operate actuators (Blankertz et al., 2010; Gomez-Rodriguez et al., 2011), and to play games (Blankertz et al., 2010) without surgical procedures. However, poor spatial resolution and low signal-to-noise ratio of EEG and near-infrared spectroscopy are major limiting factors for noninvasive technologies: information transfer rates (ITRs) maximally reach ∼25 bits/min (van Gerven et al., 2009), even when choices are made from multiple targets. This performance corresponds to a spelling speed of only five letters/min, making routine communication at least difficult.

Invasive recordings go along with substantially higher performance (Schwartz, 2004; Wolpaw and McFarland, 2004; van Gerven et al., 2009; Moran, 2010). Neural signals recorded intracortically from the macaque monkey's motor cortex allowed to decode intended movements (O'Doherty et al., 2009; Andersen et al., 2010) and to control a robotic arm (Wessberg et al., 2000; Velliste et al., 2008). The ITRs for such recordings reach 144 bits/min for binary decisions (Santhanam et al., 2006). In contrast to motor-related signals, neural correlates of selective visual attention were rarely used to control invasive BCIs, despite the importance of attention-dependent modulations of P300-evoked potentials (Brunner et al., 2011) or of steady-state visually evoked potentials for noninvasive BCIs. One exception is a matrix speller using electro-corticogram (ECoG) signals from human patients, which achieved ITRs of ∼60 bits/min (Brunner et al., 2011) or ∼12 letters/min. However, matrix spellers require multiple stimulus presentations to identify a persistently attended character and are therefore an inherently time-consuming approach. Spatial selective attention bears the potential to solve this problem if it can be deliberately and quickly shifted between numerous possible locations in visual space. This raises the question of whether intracranial recordings from visual areas allow to distinguish the direction of attention in a fast and spatially precise way. Here we investigated this question for weakly invasive recordings with high-resolution, epidural ECoG arrays in macaque monkeys performing a covert attention task. We found that the ITR can be increased to 242 bits/min in 200 ms decoding intervals, going along with an almost error-free classification, hence promising typing speeds for virtual keyboards in the range of one character per second.

Materials and Methods

Behavioral paradigm and visual stimulation.

Two male rhesus monkeys (Macaca mulatta) were trained to a shape-tracking task (Fig. 1C) (for a detailed description of the stimuli, see Taylor et al., 2005). During training and recording, the monkeys sat in a primate chair with the head restrained. Visual stimuli were presented on a 21” CRT screen with a frame rate of 100 Hz and a distance of 81 cm from the monkey's eyes.

Figure 1.

Experimental paradigm and recording. A, B, Size and position of the two morphing stimuli on the computer screen for the far (A) and the close condition (B). C, Time course of a trial with four morph cycles. Shown are only the shapes of the target sequence. Gray shading represents the response window for this trial, and the blue region represents the time window T_F. D, Schematic drawing of a cross section of the array. E, Sketches of the position of the electrodes in relation to visual areas V1 and V4. STS, Superior temporal sulcus; IOS, inferior occipital sulcus; LS, lunate sulcus.

The monkeys were presented with two sequences of morphing shapes. They started a trial by fixating a central fixation point and pressing a lever. After 650 ms, two shapes were presented. In conditions Far_F and Far_M (recorded in Monkey F and Monkey M, respectively), stimuli were placed in different hemifields, at both sides of the fixation point (stimulus center 0.9° below the horizontal meridian and 2.9° aside the vertical meridian). The stimuli covered a region with a diameter of ∼4° (Fig. 1A). In condition Close_F, two smaller stimuli (diameter ∼0.8°, center-to-center distance ∼1.35°, eccentricity 1–1.5°) were placed in the same hemifield (Fig. 1B). One of the sequences (target sequence) was cued within the first 200 ms of stimulus presentation by green coloring of the shape that faded to white within the subsequent 400 ms. After a static shape presentation of 1300 ms, both shapes morphed through a sequence of different shapes, referred to as S2–S6 (Fig. 1C) according to their sequential position. The monkeys had to release the lever at the reoccurrence of the initial shape in the target sequence. The response window ranged from 400 ms before the end of a morphing cycle to 750 ms after the beginning of the next cycle. The time of reoccurrence of the initial shape was randomly selected from S3 to S6. The other shape sequence served as a distracter. A reoccurrence of the initial shape in the distracter sequence had to be ignored. A correct response was rewarded with a small amount of fruit juice. If the monkeys broke fixation (rectangular fixation window of ±0.75° for conditions Far_F and Far_M, round fixation window with a radius of 0.5° for condition Close_F), or responded too early or too late, the trial was aborted without reward.

In general, epidural recordings have a limited resolution because they average signals from neuronal populations with spatially distributed receptive fields. Hence, our motivation for the Close experiment performed in Monkey F was to investigate whether at least some information about the direction of attention can be decoded with epidural recordings if stimuli are closely spaced.

For retinotopic mapping, small white squares (0.4° × 0.4°) were flashed at different positions in the lower right quadrant of the visual field while monkeys were engaged in a fixation task and kept their gaze within ±0.75° around the fixation point.

Surgical preparation.

All surgeries were done under general anesthesia and sterile conditions. After the monkeys mastered the fixation task, they were implanted with a head post and a thin gold ring placed between the conjunctiva and the sclera of one eye for measurement of gaze direction using the indirect search coil method (Bour et al., 1984). After completion of the subsequent training for the shape-tracking task, the monkeys were implanted with an array of flat platinum-iridium electrodes placed epidurally over temporal cortex (mainly V4) and neighboring parts of V1/V2 close to the lunate sulcus (Fig. 1E). The connector of the array was embedded in acrylic cement on top of the skull. Stereotactic positioning of the array was based on structural magnetic resonance images obtained for each animal from a 4.7 T Bruker Biospec scanner. The precise location of the implanted electrode array was estimated postoperatively by the stereo-tactic coordinates determined during implantation, their comparison with structural magnetic resonance images obtained after implantation, and morphological confirmation in one of the monkeys.

Recordings.

For epidural recording of cortical field potentials, electrode arrays (36 electrodes in Monkey M and 37 electrodes in Monkey F) were used (Fig. 1D,E). They were manufactured of Teflon-coated platinum-iridium (90Pt/10Ir) wires (diameter 50 μm, Science Products) and a 0.1 mm thick silicone foil (Goodfellow). Each electrode consisted of platinum-iridium wire several centimeters long. One end of the wires was uninsulated and bent into a loop with a diameter of 210–220 μm. These wires were inserted with a regular spacing of 3 mm into the silicone foil with the uninsulated loop positioned in parallel to the foils' surface (and hence to the dura). At the foils' side opposite to the dura, the wires were glued with silicone (Dow Corning 734) to the silicone foil and extended several centimeters over the arrays' border. Their ends were soldered to a connector (Binder). The electrodes' impedances were typically 25 kΩ at 100 Hz. Two reference electrodes (platinum-iridium wire, 150 μm diameter) were placed frontally. In Monkey F, a third reference was attached to the backside (i.e., directed to the bone) of the electrode array (0.1 mm platinum-iridium foil, 4.5 mm diameter). For recording, the array's connector was connected to a multichannel head stage. Recordings were referenced to the latter electrode in Monkey F and to a frontal electrode for Monkey M. Signals were amplified (×40,000 in Monkey F, ×30,000 in Monkey M, bandwidth of 1–150 Hz) and recorded at a sampling rate of 1 kHz.

All surgical and experimental procedures were performed in accordance with the European Communities' Council Directive of November 24, 1986 (86/609/EEC) and with the regulations for the welfare of experimental animals issued by the Federal Government of Germany and had been approved by the local authorities.

Dataset seconds and data preprocessing.

In each of the three datasets (Far_F, Far_M with distant stimuli, and Close_F with near stimuli), trials that terminated with a correct response were divided into two classes: class A (attention on the one shape sequence) and class B (attention on the other shape sequence). For one of the monkeys, some trials had to be rejected because the signal occasionally saturated the amplifier.

The recorded field potentials were high-pass filtered with a digital filter (Butterworth IIR filter, cutoff frequency 0.65 Hz at 3 dB, forward and backward filtering to avoid phase shifts) to eliminate DC offset. Furthermore, the 50 Hz component corresponding to the line current was removed from the measured data by the following procedure. The 50 Hz line current signal was recorded during all trials. For each trial, a template for the interference was calculated by cutting the time series of the trial into ∼20 ms chunks that were time-locked to the zero-crossings of the 50 Hz signal. These chunks were averaged to yield a template, which was subsequently subtracted from the original time series, again time-locked to the zero-crossings of the line current.

For the datasets Far_F and Far_M, the current source density (Gevins, 1984) with unit V/m² was computed to suppress the effect of the common reference and to minimize spatial smearing (Nunez et al., 1997): For each time bin, the second spatial derivative of the field potentials was calculated with the Laplacian operator (Perrin et al., 1987) using Gaussian radial basis functions for interpolation (Moody and Darken, 1989).

Data analysis and feature extraction.

For each electrode η, the preprocessed signals ν_{η, j}(t) with j denoting the trial number and t the time after trial onset were convolved with complex Morlet wavelets ω(t, f₀) (Kronland-Martinet et al., 1987) to obtain the wavelet coefficients a_{η, j}(t, f₀) via the following:

The spacing of frequency bands was logarithmic between 5 and 200 Hz, chosen as f₀(k) = Ω^k−1f₀(1) for k = 1, …, 17 frequency bands starting at f₀(1) = 4.84 Hz. For a sufficiently tight coverage of frequency space, we set Ω = 1.26. Because of the wavelet's overlap in frequency space, data from two neighboring frequency bands are not completely independent (for ease of notation, we do not explicitly state the dependence of the variables on frequency f₀, trial number j, and electrode index η where possible).

Based on the complex wavelet coefficients a(t), three features were extracted from short time epochs in individual trials: average spectral amplitude, average phase coherence, and average phase difference. By introducing the amplitude α and phase Φ, we can write the wavelet coefficients a as a = αexp(iΦ). For obtaining mean spectral amplitude A, we averaged α over a time interval from t₀ to t₁:

Using averaged data instead of the full set of all wavelet coefficients over time led to a reduced complexity for the training of the classifiers, which could be handled computationally. For obtaining phase coherence C and phase difference Ψ, we averaged phase differences between electrodes η and η′ in the complex domain:

giving us a complex number c from which C and Ψ were extracted as absolute value and angle via the relation c = C exp(iΨ), respectively. For electrodes η and η′, Ψ_n,n′ represents the mean phase difference of the phase difference distribution, whereas C_η,η′ is inversely related to the width of this distribution. The phase coherence assesses the stability of the phase difference between two recording sites, yielding C = 1 if the phase differences are constant over a certain time interval, and C = 0 if the phase differences are equally distributed.

Coherence has a bias b that depends on the number N of independent samples (degrees of freedom), which is in good approximation $\frac{\sqrt{\pi }}{2\sqrt{N}}$ for N > 10 (Sun and Yang, 1992, their Equation 8). Because of the size of the wavelet in the time domain, we get correlated values after spectral decomposition of the signal, thus reducing degrees of freedom. The bias is then mainly determined by the half-width of the wavelet's envelope. For Morlet wavelets with frequency f and scaling parameter k₀, analyzing a signal over a period T₀, this bias becomes $b\approx \frac{1}{2}\sqrt{\frac{k_0}{T_{0}f}}$, which we verified for our data. We always used k₀ = 6 in our analysis, and $T_0=T-\frac{2k_0\sqrt{2}}{2\pi f}$ denotes the part of the full analysis window of size T, which was outside the Cone-of-Influence at the window's borders.

In our analysis, this bias contains no extra information for decoding the state of attention because we only compared coherence values with the same degrees of freedom (i.e., analysis windows of equal length) via a discrimination measure.

In total, a set of M electrodes allows one to analyze each of the (M² − M)/2 different electrode pairs for the two data features average phase coherence and average phase difference. In the following, we will leave out the term “average” when referring to these features.

To investigate whether different directions of spatial selective attention can be reliably distinguished based on short data epochs in individual trials, we classified the recorded data according to the condition of attention using support vector machines (SVMs) (Schölkopf et al., 2000; Schölkopf and Smola, 2001), which deliver performances superior to linear methods. For implementing the SVMs, we used the widely used libsvm software package (Chang and Lin, 2011), which provides convenient data preprocessing routines automatically searching the parameter space of possible SVM realizations while calibrating the input data to match the abilities of the classification algorithm. Radial basis functions were chosen as kernels for classification.

Parameters of the SVM are always optimized on training data only, whereas performance is tested on separate data. We divided each of our three datasets into separate test and training data by a blockwise decomposition. The data were first separated into six consecutive blocks of trials (except for Close_F, where only five blocks were used) with approximately the same number of trials per block. Each block contains the successful, consecutive trials, which sometimes extend over consecutive recording days. One of these blocks was used as the test dataset. The remaining blocks were used as a pool for assembling the training dataset, for which 50% of all trials in that pool were selected randomly. This reduction of training trial numbers by a factor of 2 was actually required by restrictions on the available computing power and led to a reduction of processing time by a factor of 4. The selection was done such that the proportion of trials for the two classes of attention in the full dataset was conserved (i.e., chance levels p_chance were constant). After classification of the selected test block, the next block was taken as test data, and all other blocks as the new training data pool. This procedure was repeated until all data had been classified in a one-block-leave-out strategy, and the resulting performances (percentage of correctly classified trials) were averaged over all blocks, yielding the classification performance p_c. This kind of trial segmentation was necessary because the datasets were recorded in a block design, where attention was directed to the same location in a series of subsequent trials. Without this procedure, slow nonstationarities, lasting over several trials, would cause an artificial improvement in classification performance.

The chance level p_chance of an uneducated guess was computed as the percentage of occurrences of the most frequently presented class in all trials. An increase (decrease) in performance above chance level was considered to be significant if the probability to obtain an equal or higher (equal or lower) performance by drawing from a binomial distribution around p_chance was <0.001.

Because of the differences in chance levels, we also computed the entropy I (measured in bits) as a measure for the information obtained from the signals. I allows a direct comparison of the classification power between the three datasets and is computed via the following:

Results

To investigate the potential of neuronal correlates of covert visual attention in ECoG signals as a signal source for BCIs, we examined how reliable the direction of attention to one of two spatially separate stimuli in the visual field could be extracted from these signals. Tables 1 and 2 contain detailed information about the behavioral performance in the experiment. A typical example of data obtained from the epidural ECoG recordings is given in Figure 2A. It shows the average spectral power distribution along the course of the trial for an attended stimulus, normalized to the background activity before stimulus onset (for details, see Fig. 2 legend). A transient response to stimulus onset with a broad distribution over frequencies was followed by a sustained increase in power, which was confined to the γ frequency range between 30 and 100 Hz. As described previously (Taylor et al., 2005), the mean power of this sustained γ-band response averaged over several hundred trials was strongly enhanced in the attended condition compared with the nonattended condition (Fig. 2B), being visible even in single-trial local field potential (LFP) traces (Fig. 2C).

Table 1.

Details about the distribution of trials and behavioral performance^a

	Correct trials (attentional/nonattentional)	Mean response time (ms)	Behavioral performance (%)
FarF	4480 (2023/2457) over 15 d	1261	61.4
FarM	5209 (2639/2570) over 16 d	1380	78.8
CloseF	960 (546/414) over 5 d	1288	77.1

^a The first column shows the number of successful trials used for the presented analyses, recorded over the listed number of days. The two ensuing values in parentheses break down the total number of trials into the two attentional conditions. The second column lists mean times of the behavioral response relative to the beginning of the morph cycle leading to the target shape. The third column reflects the behavioral performance of the animals and shows the amount of correctly performed trials (disregarding eye-movement errors) during the whole experiments.

Table 2.

Details about the significance levels for the results^a

	pchance (%)	psig (p < 0.001) (%)	diff50% (%)	diff75% (%)	diff95% (%)	diff99% (%)
FarF	54.8	57.1	±2.8	±2.4	±1.2	±0.6
FarM	50.7	52.8	±2.6	±2.2	±1.1	±0.6
CloseF	56.9	61.8	±5.8	±5.1	±2.7	±1.5

^a The first column contains the chance level of producing a correct answer with an uninformed guess. The second column gives threshold levels of significance for deciding whether a performance value is significantly different from chance level with an error probability p < 0.001. The remaining columns list maximum lower bounds for differences in performance from which two given performance values p_c1 and p_c2 are considered to be significantly different (p < 0.001). diff_50% is for a range of classification performances p_c1 in 0%–75%, diff_75% for 75%–95%, diff_95% for 95%–99%, and diff_99% for 99%–100%. If the second performance value p_c2 is, by the amount specified in the table, either lower or higher than p_c1, the corresponding performance difference is significant.

Figure 2.

Basic LFP features. A, Time-frequency plot showing the average spectral power of one electrode over the time course of a trial, in condition Far_F. Neuronal data were recorded over temporal cortex and showed the largest classification performance in dataset Far_F. Spectral power (A²) was normalized to the baseline activity before stimulus onset via (A²/A^2,baseline) − 1. B, Corresponding difference plot between the attentional conditions (attention inside the receptive field − attention outside the receptive field). C, Raw LFP data for the attention inside (red) and the attention outside the receptive field condition (blue). Gray lines between B and C indicate the morph cycle from which the data were taken.

Classification of attentional state

Because BCIs need to act in real time, we used short data episodes taken from single trials to estimate which stimuli the animals attended. For classification, we considered three “features” of the LFP data, each of them averaged over a short time interval T. These were spectral amplitudes A_{η, j}(f₀) for electrode η and frequency band f₀ in trial j, as well as phase coherences C_η,η′,j(f₀) and phase differences Ψ_{η,η′, j}(f₀) for electrode pairs (η, η′) (for definitions, see Materials and Methods). Chance levels (p_chance) and their corresponding threshold values for performances (p_c) significantly higher than chance level (p < 0.001) are listed in Tables 1 and 2.

High classification performance with a single electrode or electrode pair

Classification performance for single electrodes (or single electrode pairs) was estimated using data from time interval T_F comprising the 1400 ms immediately preceding the monkey's response window. Figure 3A–C shows the classification performance based on mean spectral amplitude A over T_F for all single electrodes, using all 17 frequency bands extracted between 5 and 200 Hz. In all datasets, temporal (T) recording sites showed the highest classification performance. Electrodes corresponding to occipital areas (O) allowed still good classification but with performances that were considerably lower.

Figure 3.

Classification performance on data from single electrodes and single electrode pairs. A–C, Results for classification based on spectral amplitude A for the stimulus configurations: A, Far_F; B, Far_M; C, Close_F. Performance values that were not significant (i.e., p > 0.001) are shown in pale blue font. The color scale on the right represents classification performance. Classification performance based on phase coherence C (D–F) and panels based on phase difference Ψ (G–I), for all single electrode pairs in the stimulus configurations: D, G, Far_F; E, H, Far_M; F, I, Close_F. Performance is coded according to the color bar shown to the right of each panel. The white box represents T–T interactions; the green box, O–O interactions; and the cyan boxes, O–T interactions. All other interactions include electrodes over regions that were not driven by the stimuli. For all panels, 17 frequency bands between 5 and 200 Hz from the time interval T_F were used.

Classification performance based on the stability of the phase difference between two recording sites as measured by the phase coherence C for the same frequencies and time intervals T_F is shown in Figure 3D–F for all electrode pairs. The highest performances in all three datasets were achieved with T–T pairs, whereas they were clearly weaker for the O–O pairs and for the T–O pairs, particularly in the Far condition. Similar results were obtained from classification on the actual size of the phase difference Ψ. Figure 3G–I shows that attention not only modulated the precision of phase locking between two electrodes but also caused the signals to lock at different phase differences.

Maximum performances obtained from, respectively, the best electrode or best electrode pair, are summarized in Table 3. For all three measures and datasets, high classification performance up to a maximum of 94.8% was achieved. All listed values were significantly different from chance level. This finding demonstrates that all three features allowed good classification performance for the location of attention. Interestingly, our control experiment for nearby stimuli, originally expected to be most difficult for distinguishing the direction of attention because of the very small and closely spaced stimuli, consistently showed the best classification performance in all three measures.

Table 3.

Maximum classification performances on 17 frequency bands between 5 and 200 Hz on single electrodes (or single pairs of electrodes) over temporal cortex, using time interval T_F^a

	A	Ψ	C
FarF	79.3/0.258	82.7/0.329	79.7/0.265
FarM	83.2/0.347	81.8/0.315	67.9/0.094
CloseF	92.7/0.609	94.8/0.691	94.2/0.667

^a Values are given as %/bit.

Classification performance improves with number of electrodes/electrode pairs

Next, we investigated how performance depends on the number of electrodes used for classification. For this, we combined data from single electrodes (or electrode pairs) successively into multidimensional data vectors and classified them with SVMs: We did this analysis for each of the three features (amplitude, coherence, and phase difference) separately. Starting with the best electrode (or electrode pair), further data were added with decreasing rank in the single electrode (or electrode pair) classification performances (again, each entry included all 17 frequency bins between 5 and 200 Hz) until values from 25 electrodes (or electrode pairs) were combined. For each of the three features, classification performance improved with increasing number of electrodes (or electrode pairs) and saturated after adding 5–15 electrodes (or electrode pairs) (Fig. 4).

Figure 4.

Changes in classification performance with number of signals used. Classification on spectral amplitude A (black), on phase coherence C (blue), and on phase difference Ψ (red) is shown for all three datasets: A, Far_F; B, Far_M; C, Close_F. One to 25 electrodes (or electrode pairs) were subsequently included into the classification analysis, according to their rank defined by their single electrode (or electrode pair) classification performance. Again, datasets included all 17 frequency bands between 5 Hz and 200 Hz from time interval T_F.

Maximal performances achieved by one of the combinations comprising one to 25 electrodes (or electrode pairs) and the performance for the full sets of 25 combined electrodes (or electrode pairs) are listed in Tables 4 and 5, respectively. Best classification rates were observed for the feature “amplitude” (Fig. 4).

Table 4.

Maximum classification performances achieved by combining data from up to 25 single electrodes (or electrode pairs)^a

	AMax	ΨMax	CMax
FarF	93.3/0.639	87.1/0.439	91.3/0.567
FarM	95.3/0.726	87.1/0.445	87.1/0.445
CloseF	99.0/0.905	98.2/0.856	97.9/0.839

^a Values are given as %/bit. Classification was based on 17 frequency bands between 5 Hz and 200 Hz and time interval T_F.

Table 5.

Classification performance obtained from the entire set of the 25 best electrode (or electrode pairs)^a

	A25	Ψ25	C25
FarF	92.7/0.616	86.1/0.412	91.0/0.557
FarM	94.2/0.680	85.4/0.400	86.3/0.424
CloseF	98.9/0.899	98.1/0.850	97.5/0.818

^a Values are given as %/bit. Classification was based on 17 frequency bands between 5 Hz and 200 Hz and time interval T_F.

Information about attentional state is maximal in the γ-band

So far our analysis took signals from a broad frequency spectrum between 5 and 200 Hz into account to identify the direction of attention. To investigate how signals in different frequency bands contribute to classification, we estimated for each of the three features (amplitude, coherence, and phase difference) the classification performance based on each of the 17 single frequency bands individually. We built three sets of electrodes (or electrode pairs) containing the 25 electrodes (or electrode pairs) with highest ranks based on spectral amplitude (Set_A), on phase coherence (Set_C), and on phase difference angle (Set_Ψ), respectively. The spectral analysis of classification performance for these sets (Fig. 5) revealed that the majority of the information lies in the frequency range >30 Hz. Peak values occurred predominantly between 60 and 80 Hz and came close to the values observed when providing the SVM with data from all frequency bands together. Hence, there was little, if any, nonredundant information outside the γ-band available, which could be exploited for classification by SVMs. Similar as for classification based on all frequency bands together, classification performance based on individual frequency bands showed reduced performance if only single electrodes instead of electrode (or electrode pair) sets were considered (for amplitude data, see Fig. 6A–C). Highest classification performances based on only one frequency band for single electrodes (or electrode pairs) are listed in Table 6. As for the electrode (or electrode pair) sets, maximal information was contained in the γ-band between 61 and 97 Hz. Again, if the two stimuli were closely spaced, highest classification performances were observed (up to 91.8%). Comparison with the highest classification results obtained for single electrodes (or electrode pairs) based on the full signal spectrum (Table 3) revealed a loss in performance between ∼1% and 7%.

Figure 5.

Discriminability of the attentional condition based on data from single frequency bands. Classification performance on spectral amplitude A (black, Set_A), phase coherence C (blue, Set_C), and phase difference (red, Set_Ψ), for the stimulus configurations: A, Far_F; B, Far_M; C, Close_F. All data were taken from the period T_F. The green line represents the chance level; and the orange line represents the significance level with a p value of 0.001. Peak performance for all datasets and features was always between 60 and 80 Hz.

Figure 6.

Discriminability of the attentional condition based on data from single frequency bands. The panels show classification performance on spectral amplitude A in dependence on frequency band and electrode (or electrode pair) index according to the color bar to the right, for the stimulus configurations: A, Far_F; B, Far_M; C, Close_F. Electrode(-pair)s are ordered according to descending performances. Differences of chance levels result in different background color levels in the plots. Classification performance for each frequency band is indicated in the corresponding axis label. All data were taken from the period T_F.

Table 6.

Maximum performance for classification based on one frequency band with a single electrode (or electrode pair), using time interval T_F^a

	A	Ψ	C
FarF	76.2/0.202 @77 Hz	80.8/0.288 @61 Hz	77.5/0.224 @77 Hz
FarM	79.0/0.258 @77 Hz	77.9/0.238 @77 Hz	66.1/0.076 @77 Hz
CloseF	91.8/0.577 @97 Hz	87.5/0.443 @77 Hz	87.8/0.451 @97 Hz

^a Values are given as %/bit.

Figure 7 illustrates the good separation of single-trial data between different conditions of attention for all three features if the two most informative individual frequency bands from the best electrode (or electrode pair) are considered. The red and blue data clouds for the two attentional conditions were well separable, making classification even for simple linear classifiers easy.

Figure 7.

Discriminability of the attentional condition based on data from single frequency bands. The graphs represent the data clouds for the two attentional conditions (in red and blue). The rows represent dataset Far_F (A–C), dataset Far_M (D–F), and dataset Close_F (G–I). The columns represent the features spectral amplitude A (A,D,G), phase difference Ψ (B,E,H), and phase coherence C (C,F,I). Each dot represents one trial. For the scatter plots, the two most informative frequency bands of the best electrode (or electrode pair) for the corresponding feature were selected. Classification performance for each frequency is indicated in the corresponding axis label. All data were taken from the period T_F.

High-performance classification is also possible with short time windows

Using relatively large time windows of 1400 ms resulted in classification performances, which can already saturate at ∼90–99% correct. For increasing the bit rate of a BCI based on visual attention, it might thus be possible to decrease the size of the time window without reducing performance by the same factor. Based on the sets of 25 electrodes (or electrode pairs) described above (Set_A, Set_C, and Set_Ψ), which had shown best performance for the respective feature, we analyzed time windows from 1400 ms down to 20 ms to quantify how performance scales with its length. The maximal performance over all window positions was selected for each window size. Figure 8A–C shows that degradation of classification performance was rather small. Substantial degradation was only found for window lengths of ≤200 ms. For Monkey F, performance was maximal at a window size of 1000 ms, achieving 99.9% for the feature “amplitude” and closely spaced stimuli. With decreasing window size classification, performance dropped faster for coherence and phase difference values than for amplitude A. However, performance expressed as ITR in bits per minute (Fig. 8D) was highest for very short time window sizes (i.e., for 30–60 ms), showing that the reduction in time necessary for performing the classification outweighed the loss in classification performance. Based on amplitude values from 25 electrodes, the ITR for these time window sizes reached values between 260 and 580 bits per second.

Figure 8.

Classification performance in dependence on time window size. Results are displayed as percentage correct for the features “spectral amplitude” A (black), “phase difference” Ψ (red), and “phase coherence” C (blue) for the configurations: A, Far_F; B, Far_M; C, Close_F. Data from 25 electrodes (or electrode pairs) were used for the frequency bands between 30.6 and 193.6 Hz for time windows ≥200 ms width (i.e., 200, 400, 700, 1000, 1200, 1400, and 1600 ms). For window sizes <200 ms (i.e., 20, 24, 30, 34, 40, 50, 74, 86, 100, and 150 ms), only frequency bands for which the corresponding wavelets were fully fitting into the window were taken. D, Same data displayed as performance measured in bits per minute for the configurations: Far_F (solid line), Far_M (dashed-dotted line), and Close_F (dashed line), but only for spectral amplitude A. For all these plots, time windows ≥200 ms were stepped in 100 ms intervals, and time windows <200 ms were stepped with half their size. We analyzed data from a time interval starting 1700 ms before the beginning and ending 100 ms after the beginning of the response window. The mean response times in the three conditions were outside the analysis window (mean response time relative to the end of the analysis window: Far_F, 171 ms; Far_M, 290 ms; and Close_F, 198 ms). For all possible windows fitting within this time interval, the classification performances were calculated and their individual maxima for each window size is shown. D, The vertical blue line represents the border below it would not have been possible to achieve a continuous repositioning of the focus of attention within the used time window. The orange line represents the significance level (p < 0.001), and the green line represents the chance level.

Classification performance is modulated with morph cycle

Analyzing classification performance within sliding 200 ms time windows allows to investigate the dynamics of attentional modulation of LFP signals along the time course of the trial (Fig. 9). It turned out that classification rate changed dynamically along the trial. It was lowest around the time when the morphing shape reached one of the original shapes that were used to construct the morphing sequence (at t = 0 in Fig. 9A). Because for the target shape, the mean response was 298 ms before this state, these were times during the morph cycle where the monkey had already identified the stimulus as being or not being the target shape. In contrast, during the transition between two shapes A and B, classification performance for attention was large. Over these periods, the stimulus was highly ambiguous but began to provide information about the shape B into which it will morph. Here the monkey needed to identify and hence to attend the upcoming shape to respond if necessary. To test whether this dynamic reflects solely the dynamics of shape deformation, we compared this result with the time course for the first morph cycle. Although the dynamics of the stimulus were the same as for the subsequent morph cycles shown in Figure 9A, the shape S2 at the end of the cycle could never have been the target shape. We found that classification performance is much lower (Fig. 9B, compare equivalent periods shaded in blue). This result was similar for the Far_F condition but not for the Far_M condition where performances were comparable.

Figure 9.

Time courses of classification performance during the trial. Performance was computed for subsequent, 200 ms time windows whose centers are marked by the dots. The solid black line represents performance on spectral amplitude, and the dotted lines represent classification performance on feature combinations (black-red for A and phase difference Ψ, black-blue for A and phase coherence C, and red-blue for Ψ and C). The frequency bands between 30.6 and 193.6 Hz, and all temporal electrodes (or the corresponding electrode pairs) were used. The orange lines represent the significance level (p < 0.001), and the green lines represent the chance level. A, The origin of the time axis is centered at 1400 ms (one morph cycle, time course of cycle indicated below graph) before the behaviorally relevant stimulus (target) appeared. B, First morph cycle from shape S1 to shape S2, with time axis relative to trial onset. A, B, The blue shading represents the middle of a morph cycle.

Finally, we investigated whether combining information from different features of the LFP data increases classification performance. For reasons of computational complexity, we investigated these aspects for reduced sets containing only the temporal electrodes. We found that classification based on phase coherence C or phase difference Ψ, combined with spectral amplitude A, did not surpass classification performance on spectral amplitude alone (Fig. 9). Combining C and Ψ provided an improved classification performance compared with the individual features (data not shown) but lower classification performance than any combination, including the spectral amplitude.

Discussion

The results showed that it is possible to decode the allocation of attention to one of two morphing shapes based on epidural field potential recordings over visual cortex. Classification was performed rapidly and reliably on single trials in this framework. Signal features supporting high classification performance included the spectral amplitude of γ-band activity between ∼30 and 100 Hz as well as the phase coherence and the phase difference between signals from different electrodes in the same frequency range. Most informative recording sites concentrated at temporal cortex-containing area V4. Using more than one electrode (or electrode pair) improved classification considerably and allowed very high performances up to 99% within analysis windows of 1400 ms width. Using shorter analysis windows caused an only moderate reduction in classification performance. Correspondingly, the ITR was maximal for windows of 30–60 ms, reaching up to 580 bits/min. The time course of classification performance based on 200 ms windows revealed a characteristic modulation paralleling the putative attentional requirements of the task.

Thus, recordings with chronically implanted epidural electrode arrays over visual cortex provide signals that are highly informative about states of spatial visual attention. With respect to stimulus shape, a high and attention-dependent level of discriminability was previously shown for epidurally recorded V4 and V1 field potentials (Rotermund et al., 2009) and for not simultaneously recorded populations of single units (Zhang et al., 2011). The finding that such high amounts of information were obtained from single trials within only a few hundred milliseconds suggests that this type of recording is an interesting source of signals for high-performance BCIs. The remarkable performance was achieved despite using epidural field potential recordings, which are commonly expected to deliver a low signal-to-noise ratio and assumed to be less specific than subdural or intracortical recordings. Clearly, the epidural technique is less invasive and associated with smaller risks. Therefore, it may be more acceptable if patients are either in a state in which opening of the dura should be avoided, or if the expected improvements of life quality by a BCI do not justify possible risks by a more invasive approach.

A second important aspect with respect to BCIs was the temporal stability of the ECoG measurements and the analyzed features over extended periods of time. Although the datasets to train the SVMs and test datasets were not from interleaved blocks or trials, and mainly from different recording days, classification performance and ITR were high. In contrast, in numerous studies using action potential data, (e.g., Wu and Hatsopoulos, 2008; Li et al., 2011), the variability of recorded responses required retraining of the decoding algorithms in each session.

A third important factor critical for BCIs is a sufficient ITR. In our setting, we observed nominal performances of up to 139, 204, and 242 bits/min (depending on the dataset) when decoding from 200 ms time windows, which seems the shortest time scale on which attention can be deliberately switched (Wolfe et al., 2000). With a 1000 ms time window, ITRs decreased to 40.5, 46.3, and 58.5 bits/min. However, these lower ITRs were accompanied by a higher reliability of a single classification, which might be preferable in settings where incorrect decisions require a large effort from the user to be canceled. The most informative feature was spectral amplitude, which can be extracted rapidly, and on-line, from recorded data. The excellent separability of the data (as shown in Fig. 7) is a further advantage because linear classifiers can be used without much loss in performance. This becomes important if decoding algorithms need to be adapted on-line (Rotermund et al., 2006; Li et al., 2011), which is achieved faster for linear algorithms. Furthermore, successful decoding in short time windows allowed observing rapid fluctuations in the time course of classification performance, which were maximal during the transition between two shapes. At these times, an ideal observer would most accurately monitor the stimulus for figuring out whether it will morph into the behaviorally relevant shape. In addition, performance was lower if the morphing cycle was terminated by a shape that never could have been the target shape. Together, these observation suggest that the classification dynamics is linked more strongly to fast variations of attentional demand during stimulus morphing than to the actual stimulus dynamics. Given such changes of task demands, highly trained animals are well known to modulate their attentional efforts correspondingly (Ghose and Maunsell, 2002). In addition, behavioral performance and neuronal activity patterns have been shown to be influenced by fast and uncontrolled fluctuations of attention (Cohen and Maunsell, 2010). The ability to classify successfully even in short windows enables BCIs to identify such fluctuations of selective attention. This can help to identify and selectively access short periods providing highly reliable estimates for the focus of attention.

A fourth and unexpected observation was the high spatial resolution of decoding the focus of attention based on epidurally recorded ECoGs. The good performance for small and very close stimuli suggests that the classification does not require the two neuronal stimulus representations to be located far apart from each other or even in different hemispheres. A possible reason for the surprising discernibility are the particularly strong effects of spatial selective attention on neuronal activity patterns, which occur if target and distracter stimuli are located within the same receptive fields, strongly competing for being processed (Moran and Desimone, 1985; Reynolds et al., 1999; Lee and Maunsell, 2010). Evidence for such enhanced competition between stimuli has not only been described for single-cell recordings, but also for fMRI data (Pessoa et al., 2003), which are tightly related to the here investigated γ-band activity (Logothetis et al., 2001). Hence, it is plausible that, in the present case, strong competition for the same neuronal substrate between closely spaced stimuli enforces more distinct activation patterns compared with conditions with no competition for the same recorded neuronal population. If this high resolution extends to stimulus constellations with more than two stimuli, it would become possible to gain more than one bit of information for each attentional selection of a stimulus by displaying more than only two potential targets of attention simultaneously.

This raises the question whether visual areas, such as V4, are the optimal site for decoding the direction of attention with high spatial resolution. It has been proposed that the direction of spatially selective attention derives from the occulomotor system programming ocular movements (Rizzolatti et al., 1987). In line with such a premotor theory of attention, local electrical stimulation in the frontal eye fields, an area tightly involved in saccade planning, causes attention-like modulations of stimulus-driven activity in retinotopically corresponding locations of area V4 (Moore and Armstrong, 2003). Furthermore, attending a certain stimulus results in γ-band synchronization of the corresponding locations of the two areas (Gregoriou et al., 2009), indicating their enhanced interaction (Womelsdorf et al., 2007).

Therefore, the occulomotor system might also provide signals that allow for the decoding of the direction of attention. However, it has been shown that allocation of spatial attention and planning of a saccade are not the same processes but can be dissociated in physiological (Lebedev and Wise, 2001; Juan et al., 2004; Lebedev et al., 2004) as well as psychophysical (Hunt and Kingstone, 2003) investigations. In particular, the attention-dependent synchronization with V4 neurons is observed selectively for FEF visual neurons but not for movement or visuomovement neurons (Gregoriou et al., 2012). These findings suggest that neuronal activity related to saccade planning is not necessarily representing the direction of attention and therefore not a preferable choice for decoding the direction of attention.

Decoding the focus of attention depended on the number of recording sites. For each of the three features (amplitude, coherence, and phase difference), it improved with increasing number of electrodes (or electrode pairs) and saturated after combining 5–15 electrodes (or electrode pairs) (Fig. 4). We attribute this early saturation in classification performance to two factors: (1) single electrode performance was already >80% correct, leaving only limited space for further improvements; and (2) because of the layout and placing of the electrode array, not more than ∼10 electrodes received stimulus-driven responses from temporal cortex, which contained the recording sites most strongly modulated by attention.

Comparison of classification performance achieved for the three features revealed comparable performances for single electrodes (or electrode pairs) (Tables 3 and Table 6). Using multiple electrodes (or electrode pairs) showed an advantage for the feature “amplitude” in the γ-band for the set of very similar stimuli in our experiment. This effect was most pronounced for analysis window sizes ∼200 ms. The lack of gain in classification performance achieved by combining different features might be attributed to two basic effects: (1) information among different features can be highly redundant; and (2) providing more feature dimensions with a less than linear increase in contained information introduces additional noise, which will make learning of a suitable classifier difficult. Because combinations rather degraded performance, it cannot be excluded that the model-free approach with an SVM, which can only provide a lower bound on information content, combined the different features also in a destructive way. Therefore, advances in understanding the dynamics of cortical processing and how they express in the three different features might allow for model-based approaches, which use possibly existing nonredundant information.

A prototypical application for an attention-controlled BCI application is the virtual spelling board (Blankertz et al., 2011; Treder et al., 2011) using EEG recordings and stimuli consisting of a matrix with several symbols. The user selects one of these symbols by directing covert attention to it (Treder et al., 2011). This approach can be most useful for patients who suffer from an inability for volitionally controlled eye movements. Unlike EEG recordings, ECoG recordings have a high spatial resolution and show stimulus specificity (Yoshor et al., 2007; Jacobs and Kahana, 2009). Given the speed, decoding accuracy, and high spatial resolution we already achieved with our straightforward approach based on γ-band signals from ECoG recordings, it appears feasible to realize a virtual speller with the whole set of characters and based on covert attention. The most important limiting factor would be the time to shift attention between targets rather than the time window required for decoding the attended stimulus. Because the focus of attention can be shifted volitionally 3–5 times in a second (Wolfe et al., 2000), spelling speeds of hundreds of characters/min would be possible.

In our current setting, we trained a different SVM for each subsequent time window after stimulus onset. This is adequate for estimating information content but impractical for an application where a continuous on-line classification is intended. If construction of a corresponding “global” classifier should not be possible, a virtual spelling board application is still feasible if the stimulus has a well-defined onset (e.g., by showing a blank screen before displaying the matrix of letters). After this onset, our current classification scheme can be used with different SVMs on subsequent time intervals. Classification results would then be pooled for determining the currently selected letter. The advantage of assessing the dynamics of attention over classification on the whole presentation interval is that intended states of sustained attention will possibly be better distinguished from meaningless transient states. Such dynamical differences might indicate whether a user wants to accept or to cancel the current selection.

Together, the present work suggests that epidural field potential recordings can provide surprisingly detailed information about cognitive states, such as the direction of covert attention with high speed and accuracy. This provides a basis for new high-performance BCI applications based on these states, which provide the information transfer rate required for effective applications.