Neural mechanisms of training an auditory event‐related potential task in a brain–computer interface context

Abstract

Effective use of brain–computer interfaces (BCIs) typically requires training. Improved understanding of the neural mechanisms underlying BCI training will facilitate optimisation of BCIs. The current study examined the neural mechanisms related to training for electroencephalography (EEG)‐based communication with an auditory event‐related potential (ERP) BCI. Neural mechanisms of training in 10 healthy volunteers were assessed with functional magnetic resonance imaging (fMRI) during an auditory ERP‐based BCI task before (t1) and after (t5) three ERP‐BCI training sessions outside the fMRI scanner (t2, t3, and t4). Attended stimuli were contrasted with ignored stimuli in the first‐level fMRI data analysis (t1 and t5); the training effect was verified using the EEG data (t2‐t4); and brain activation was contrasted before and after training in the second‐level fMRI data analysis (t1 vs. t5). Training increased the communication speed from 2.9 bits/min (t2) to 4 bits/min (t4). Strong activation was found in the putamen, supplementary motor area (SMA), and superior temporal gyrus (STG) associated with attention to the stimuli. Training led to decreased activation in the superior frontal gyrus and stronger haemodynamic rebound in the STG and supramarginal gyrus. The neural mechanisms of ERP‐BCI training indicate improved stimulus perception and reduced mental workload. The ERP task used in the current study showed overlapping activations with a motor imagery based BCI task from a previous study on the neural mechanisms of BCI training in the SMA and putamen. This suggests commonalities between the neural mechanisms of training for both BCI paradigms.

1. INTRODUCTION

Interruption of the output of the central nervous system by injury or disease may lead to the locked‐in state (LIS), in which the ability to communicate is impaired or lost but consciousness is preserved (Pels, Aarnoutse, Ramsey, & Vansteensel, 2017; Plum & Posner, 1972; Storm et al., 2017). A variety of augmentative and alternative communication strategies exist, which can be used by individuals suffering from LIS. Affected individuals may overcome this communication impairment by using a brain–computer interface (BCI), usually controlled via electroencephalogram (EEG) components. Modern BCIs offer control over a variety of applications and can be used independently at home and may be used for mental state monitoring (Holz, Botrel, Kaufmann, & Kübler, 2015; Juel, Romundstad, Kolstad, Storm, & Larsson, 2018; Käthner et al., 2017).

Unfortunately, approximately one third of first‐time users are unable to achieve control over the BCI (Blankertz et al., 2010). Improved understanding of the processes underlying successful BCI usage may lead to optimisations of BCI training for helping users with low initial performance and to the development of novel or improved paradigms (e.g., using images of faces for improved classification in the visual P300 BCI Kaufmann, Schulz, Grünzinger, & Kübler, 2011). In the current study, both aspects were investigated to unravel neural mechanisms involved in training to acquire control of an EEG‐based auditory BCI.

Event‐related potential (ERP)‐based BCIs rely on ERPs associated with attention to stimuli. For example, P300 BCIs take advantage of the effect that the ERPs elicited by different stimuli, which may be visual, auditory, or tactile, vary with attention allocation. In the classic visual P300 BCI, introduced by Farwell and Donchin (1988), rows and columns of a letter matrix are illuminated in a random pattern. Attended rows and columns elicit a different ERP response than unattended ones. This difference can be easily and reliably detected by respective algorithms and was shown to be stable across years even in patients with amyotrophic lateral sclerosis (ALS) (Holz et al., 2015). Importantly, it has been shown repeatedly, in healthy volunteers and those with motor impairment alike, that training can improve performance in a non‐visual ERP‐based BCI (Baykara et al., 2016; Halder et al., 2016; Halder, Käthner, & Kübler, 2016). Although ERP paradigms are particularly convenient to restore communication for persons with severe paralysis (Nijboer et al., 2008), to date no study has examined the neural mechanisms of acquiring control in ERP‐based BCI.

Non‐visual ERP‐BCIs are of particular interest for restoring communication for persons with impaired gaze control or vision (Riccio, Mattia, Simione, Olivetti, & Cincotti, 2012). In addition to paradigms transferring the classical sequential P300 speller design to the auditory domain, with words as stimuli instead of illuminating the rows and columns (Furdea et al., 2009), streaming paradigms were implemented with different auditory stimuli on each ear (Hill & Schölkopf, 2012) or based on affective stimuli (Onishi, Takano, Kawase, Ora, & Kansaku, 2017). In direct comparisons, auditory BCIs allow for similar performance as BCIs based on tactile stimuli (Halder, Takano, & Kansaku, 2018).

Previous studies investigating the neural mechanisms of BCI control found the supplementary motor area (SMA) to be active in both motor imagery and slow cortical potential (SCP) BCI tasks. In Hinterberger et al. (2005), the SMA was the largest active cluster during the time window immediately before an SCP trial. In Halder et al. (2011), the SMA differentiated between high‐ and low‐aptitude motor imagery BCI users. The SMA has also been shown to be active during real‐time feedback (Zich et al., 2015). Marchesotti et al. (2017) confirmed the role of SMA during motor imagery tasks, but also pointed out the contribution of areas outside the sensorimotor cortex to the BCI task, in particular the posterior parietal cortex and insular cortices.

The current study had the following hypotheses: (h1), training will increase performance with an auditory P300 BCI, replicating the findings in Baykara et al. (2016). (h2) Brain activation measured with functional magnetic resonance imaging (fMRI) will differ between attended and ignored auditory stimuli in the regions known to be involved in the generation of the P300 (supramarginal and frontal gyri and temporal regions associated with auditory target processing; see Halgren et al., 1995 and Linden et al., 1999). (h3) Comparing brain activation pre‐training versus post‐training with a P300 ERP‐based BCI will reflect the effects of training, particularly in brain areas involved in generating the P300 ERP. In addition, we performed two exploratory analyses: (e1) analysis of the effect of performance on brain activation comparing successful with unsuccessful learners; (e2) analysis of overlapping activation between performing motor imagery for controlling a sensorimotor‐rhythm (SMR) BCI and attending to target stimuli in the current ERP‐based auditory BCI may reveal a general task network in the brain.

2. METHODS

2.1. Participants

Ten healthy participants (six females, mean age 25.51, range 19.93–34.83) were recruited for two fMRI sessions, one pre‐BCI training (t1) and one post‐BCI training (t5), and three sessions of EEG‐based auditory P300 BCI training (t2, t3, and t4). Participants were compensated with €8 per hour. All participants gave written informed consent, and the study was carried out in accordance with the declaration of Helsinki (2013).

2.2. Procedure

In all sessions of the experiment (t1 − t5), the participants were presented with the same auditory stimuli. The illusion of directionality of auditory stimulation was evoked using interaural time difference and interaural level difference as described in Käthner et al. (2013). Stimuli were the same as in Simon et al. (2014). Those were duck‐, bird‐, frog‐, gull‐, and pigeon‐sounds, and they were arranged on positions of a circle from left, middle left, front, and middle right to right, and presented using stereo headphones. A virtual 5 × 5 symbol matrix was coded with the five animal sounds (seen in Figure 1a,b). This matrix served to select one of the 25 letters (y and z occupied the same cell). To select a letter (target), attending to one animal sound was required for its row and for its column. The other sounds had to be ignored (non‐targets). The matrix was not displayed to the volunteers (thus called virtual matrix), neither during the fMRI nor the EEG sessions.

Figure 1.

Description of symbol selection procedure. (a) The five sounds were modified to appear to originate from specific positions on a half‐circle around the participant's head. (b) Each sound was associated with a row and a column in the virtual matrix of letters. In Step 1 (green), the participants selected a row in the virtual matrix by attending to one of the sounds. For example, attending to the bird sounds selects the row of letters F–J. The ERP elicited by the target (attended sound) differs from the ERPs by the non‐targets (ignored sounds). This ERP signature is detected by the BCI. In Step 2 (blue), the participants select the column. For example, attending to the gull sound selects the letter I (red) from the previously selected row of letters (F–J). (c) Selection of one letter requires 57.75 s if all row and column stimuli are repeated 10 times, as during the calibration measurement, in randomised order [Color figure can be viewed at http://wileyonlinelibrary.com]

The current target, the switch from a row to column selection, and the end of the selection process were announced with a pre‐recorded voice as in Halder, Käthner, & Kübler (2016). The animal sound associated with the current target was included in the announcement (e.g., if the current target letter was the I, the following statement was announced: “To select I first attend to the bird and then to the gull. First the bird...”, see Figure 1c). During the switch from a row to column, the volunteer was reminded which animal to attend to for the column. After presentation of all stimuli (end of trial), the user was informed which letter was selected by the BCI (e.g., “I was selected”).

Each participant performed five sessions. One fMRI session was performed before the EEG training (t1) and one after the training (t5). The fMRI sessions consisted of three runs of auditory stimulation identical to the stimulation performed in the EEG sessions but without online classification of the selected letter. Additionally, three EEG sessions (t2−t4) were performed, which consisted of three calibration runs, which were identical to the stimulation performed during the fMRI sessions and nine training runs with online feedback. The words spelled during the training were identical in the three EEG sessions. However, the sequence was shifted by three words in each session; see Table 1 for an overview of the experimental design and sequence of words.

Table 1.

Overview of session protocols for fMRI and EEG measurements. The first three words (AGMSY) were spelled during EEG‐BCI calibration (t2−t4) and fMRI (t1, t5) sessions without feedback. During EEG measurements t2−t4, nine additional words (permuted across sessions) were spelled with feedback

	fMRI pre	EEG‐based auditory P300 BCI training			fMRI post
	t1	t2	t3	t4	t5
	Test run	EEG setup	EEG setup	EEG setup	Test run
	Localizer				Localizer
No feedback	AGMSY	AGMSY	AGMSY	AGMSY	AGMSY
	AGMSY	AGMSY	AGMSY	AGMSY	AGMSY
	AGMSY	AGMSY	AGMSY	AGMSY	AGMSY
		Calibration	Calibration	Calibration
Feedback		VARIO	TUMBI	UMBIT
		GRUEN	RUBI	PHLEX
		HUNGER	VALERI	VIRAGO
		TUMBI	UMBIT	VARIO
		RUBI	PHLEX	GRUEN
		VALERI	VIRAGO	HUNGER
		UMBIT	GRUEN	TUMBI
		PHLEX	HUNGER	RUBI
		VIRAGO	TUMBI	VALERI
MRI only	Field map				Field map
	Anatomy				Anatomy

2.2.1. Auditory EEG BCI training

The three sessions of auditory EEG BCI (t2−t4) were implemented on separate days (time between sessions 2–7 days).

At the beginning of every session, calibration data were collected from spelling the letters on the diagonal of the letter matrix (AGSMY) three times. Repeated presentation of the auditory stimuli optimises the ERP signal‐to‐noise ratio until a ceiling effect occurs. Therefore, during calibration, to select one letter, participants had to attend to 20 auditory stimuli (i.e., 10 repetitions for each row and each column, respectively). Based on offline analysis of the calibration data, the number of stimulus repetitions was optimised individually to optimise classification accuracy and to avoid ceiling effects, which would slow down communication without additional benefits (see Baykara et al. (2016) and Halder, Käthner, & Kübler (2016) for details). The number of repetitions was set to the repetitions needed to reach 70% accuracy, plus three additional repetitions. If 70% accuracy was not achieved during calibration, 10 repetitions were used. Collection of the calibration data required approximately 10 min.

After calibration, the participants wrote nine pre‐defined character strings (copy spelling task) composed of five letters each (VARIO, GRUEN, HUNGER, TUMBI, RUBI, VALERI, UMBIT, PHLEX, and VIRAGO). The words were chosen to counterbalance attention to each of the five different auditory stimuli. The first session started with the word VARIO, the second with TUMBI, and the third with UMBIT.

In all spelling tasks, the system paused between row and column selections for 2 s and between letter selections for 12 s. Stimulation duration was 150 ms with an inter‐stimulus interval of 287.5 ms (437.5 ms stimulus onset asynchrony [SOA]). This resulted in a maximum time of 57.75 s for selection of one letter (see Figure 1c for the visualisation of the letter selection procedure).

2.2.2. Functional magnetic resonance imaging experiment

Both fMRI measurements (t1 and t5) consisted of a test run to establish whether the stimuli could be heard by the participants despite the scanner noise, a short localiser measurement, and a measurement of the blood oxygen level‐dependent (BOLD) response during three repetitions of the five letters AGMSY. Overall, a sequence of 50 target and 200 non‐target stimuli were presented per letter. For each letter, 90 scans were performed (temporal resolution [TR] of 2 s and 180 s in total).

A requirement of the P300 speller is to present stimuli in rapid succession (i.e., rapid‐presentation event‐related fMRI design; see Dale, 1999). In order to accommodate for the overlap of the haemodynamic response, stimuli were jittered to allow deconvolution for estimation of the haemodynamic response to each stimulus. Optseq2 (http://surfer.nmr.mgh.harvard.edu/optseq/) was used to find the optimal sequence of events to optimise differentiation between target and non‐target responses (parameters: event response window between 2 and 18 s, resolution 0.5 s, 30000 searches per letter). In contrast to the EEG‐BCI training sessions, additional short time‐segments without stimulation (null‐events) were added, allowing the haemodynamic response to return to baseline. The sequence was optimised on a per‐letter basis, as successful selections are evaluated on a per‐letter basis also in the EEG BCI task.

The stimuli were presented with an SOA of 500 ms. This was an additional difference to the EEG measurements. The reason for this difference was that the sequence of stimuli the SOA needed to be a factor of the TR for the optimisation with optseq2. The SOAs smaller than 500 ms were found to lead to better information transfer rates (ITRs) as in Käthner et al. (2013); therefore, different SOAs for the EEG and fMRI measurements were used.

Stimuli were presented with the Presentation software (Neurobehavioural Systems, Inc., Berkeley, CA, http://www.neurobs.com) via pneumatic headphones.

Following the auditory task, a field map and anatomical scan were acquired. Both fMRI sessions were conducted in an identical manner on separate days. The average time between first and second fMRI measurements was 17.7 days (SD 3.6, range 14–22).

2.3. Data recording

2.3.1. EEG recording and online classification

In the three auditory P300 BCI training sessions (t2, t3, and t4 in Table 1), EEG was recorded at a sampling rate of 256 Hz, a high‐pass filter of 0.1 Hz, and a low‐pass filter of 30 Hz. Sixteen electrodes were placed at locations AF7, FPz, AF8, F3, Fz, F4, C3, Cz, C4, CP3, CPz, CP4, P3, Pz, P4, and POz. The ground electrode was placed at the AFz location and the reference electrode at the right earlobe. The EEG was recorded with a g.USBamp (g.Tec GmbH, Austria). Data recording, signal processing, and stimulus presentation were performed with BCI2000 (Schalk, McFarland, Hinterberger, Birbaumer, & Wolpaw, 2004) on a Hewlett‐Packard ProBook 6460b with a dual‐core CPU, 4 GB of RAM, and a 64‐bit Windows 7 operating system.

After collection of the calibration data, a model of the current participant's ERP was created with stepwise linear discriminant analysis using p value <0.1 for the forward step and p value >0.15 for the backward step and a maximum of 60 iterations. Features were extracted from a window of 800 ms post‐stimulus after applying a moving average filter with a width of 20 samples and subsampling by a factor of 20; see Halder et al. (2010) for detailed descriptions.

With the calibrated model, the data of the copy spelling task were classified online by applying the model to the 800 ms EEG segment following each stimulus presentation. This yielded classifier outputs that were summed for each stimulus type (number of classifier outputs per stimulus type dependent on the number of stimulus repetitions chosen individually for each participant; see Section 2.2.1 for motivation). Based on the summed scores, the row and the column with the highest scores were selected, and the letter at the cross section was considered to be the current selection. This selection was conveyed to the participant via auditory feedback (see Table 1 for the words used in the copy spelling task).

2.3.2. fMRI acquisition

In both fMRI sessions (t1 and t5 in Table 1), images were acquired using a Siemens Magnetom Skyra 3T whole body scanner equipped with a Siemens 32 channel headcoil. The structural (T1) images were recorded with a TR of 1.9 s, echo time of 2.26 ms, a voxel size of 1 × 1 × 1 mm³, a resolution of 256 × 256 pixels, and 176 ascending slices. The functional (T2) images were recorded with a TR of 2 s, an echo time of 33 ms, a voxel size of 3.5 × 3.5 × 3.5 mm³ , a resolution of 64 × 64 pixels, and 27 interleaved slices.

2.4. Data analysis

2.4.1. EEG analysis

EEGLAB was used to load the BCI2000 data into Matlab (Delorme & Makeig, 2004). Artefact subspace reconstruction was then applied to detect broken channels and remove noise and bursts from the EEG recording. If a broken electrode was detected, it was replaced using spherical interpolation of the neighbouring electrodes. The data were segmented into windows of −200 to 800 ms around the presentation of stimuli. Finally, any segments with extreme amplitudes, trends, joint probability, and kurtosis were rejected.

A t test for independent samples was used to compare the full matrix of amplitude values (channels × samples) between t2 and t4 for each participant individually. This yielded a set of t‐values that was then averaged across all participants. The t‐values of comparisons with p value ≥0.001 were set to zero. The reason for using t‐values instead of EEG amplitudes was to make the apparent difference in the change of the target response from t2 to t4 less dependent on the absolute value of the feature under consideration. The peak amplitudes of the ERPs were defined as the maximum amplitude between 200 and 800 ms at Cz. The latency was defined as the time in milliseconds between the stimulus onset and the peak maximum.

Since the selection times were adapted individually for each participant, accuracies cannot be compared between the participants. Therefore, ITR was calculated according to Wolpaw, Birbaumer, McFarland, Pfurtscheller, and Vaughan (2002) for each participant and session to assess the success of the training (h1).

A repeated measures analysis of variance (ANOVARM) was calculated to determine whether there was an effect of session. Post hoc, t tests for paired samples were used to determine pairwise differences between sessions in performance and ERP characteristics. α =0.05 (two tailed) was considered significant.

2.4.2. fMRI analysis

The fMRI data were analysed with the statistical parametric mapping (SPM) version 12 toolbox (Wellcome Trust Center for Neuroimaging, London, United Kingdom; Friston et al., 1994) in Matlab 2016b. First, the images were converted from the digital imaging and communications in medicine to the SPM format (*.nii). Second, realignment to the chronologically first scan was performed with the least squares approach and a six parameter (rigid body) spatial transformation. Slice timing was corrected with the middle scan as the reference. Anatomical images were co‐registered to the mean of the functional images by means of mutual information. Then the anatomical images were segmented to the tissue probability maps supplied with SPM12 and regularised to the International Consortium for Brain Mapping space template for European brains (Mazziotta et al., 2001). The functional images were then normalised on the basis of the parameters determined for this segmentation. Finally, all functional images were smoothed by a full‐width‐half‐maximum Gaussian kernel at 8 × 8 × 8 mm³.

First‐level analysis on the individual data was performed using general linear models with the realignment data from pre‐processing as additional regressors. The high‐pass filter was set to 128 s, and serial correlations were accounted for using an auto‐regressive model. In SPM, each run (spelling the letters AGMSY; a total of three runs) was modelled as a session. The targets and non‐targets were modelled as separate conditions for each stimulus type (10 conditions: each of the five sounds once as target and once as non‐target). Contrasts were defined for all targets as one with rest as zero, all non‐targets as one with rest as zero, and all targets as one versus non‐targets as minus one with rest as zero. In one version of the first‐level analysis, the conditions were defined as events with zero second duration. In the second version, an epoch for the condition from 10 to 16 s was chosen based on the time courses of the event‐related data. Both were analysed using the second‐level analysis in a full‐factorial 2 (session)  × 2 (attention)  × 5 (stimulus) ANOVARM within subject designs with equal variances for all variables. To determine whether the difference between attended and ignored sounds was reflected in the fMRI, higher brain activation related to attended stimuli was compared between Sessions t1 and t5 using a t‐contrast (h2). Similarly, higher brain activation following ignored stimuli was determined by contrasting ignored with attended targets. To determine the training effects, the positive effect for session was calculated using only the attended stimuli once for t1 > t5 and once for t1 < t5 (h3). Depending on the contrast, the haemodynamic response may result in a decreased signal. Thus, it was specifically interesting to find the regions where the absolute BOLD signal change decreased in t5. The following criteria were used to differentiate decreases in activations from stronger negative signal change in the t1 > t5 contrast: the regions with positive signal change at t1 were considered to decrease in activation from t1 to t5. The regions with negative signal change at t1 were considered to have stronger absolute signal change at t5 compared to t1.

To determine the effects of successful learning (e1), we performed an additional second‐level analysis in a full‐factorial 2 (session) × 2 (performance) × 5 (stimulus) ANOVARM within subject designs and visualised the main effect (F‐value) of performance. Participants were assigned to a performance group based on whether the average ITR of t3 and t4 increased by at least 1 bit/min compared to t2.

Labelling of the anatomical regions was determined with the SPM anatomy toolbox (Eickhoff et al., 2005).

The SPM toolbox MarsBaR (version 0.44) was used to extract peri‐stimulus time histograms for anatomically defined regions of interest (ROIs) based on finite impulse response models (Ollinger, Shulman, & Corbetta, 2001). The temporal resolution was set to 2 s. Data were extracted from ROIs defined in the automatic anatomic labelling atlas (Tzourio‐Mazoyer et al., 2002).

Next, conjunction analysis was performed to determine the overlap between brain activation during a motor imagery task (data from Halder et al. (2011)) and the auditory attention task used in the current study (e2). A second‐level analysis combining 17 participants from Halder et al. (2011) and the 10 participants from the current study was performed using a two samples t test assuming independence and unequal variance. The two contrasts (motor imagery vs. rest with attended auditory stimuli vs. ignored auditory stimuli) were then combined in a conjunction analysis.

3. RESULTS

3.1. h1: Effects of EEG‐BCI training

Average online accuracy increased from 60% in Session t2 to 73% in Session t3 and then decreased to 65% in Session t4. Due to the approach of adapting stimulus repetitions to avoid ceiling effects, there was no change of accuracy across sessions (ANOVARM: F_{2, 18} = 2.43, p = 0.12). The accuracies must be viewed in combination with the selection times per letter, which decreased from an average of 43.08 s in Session t2 to 36.57 s in Session t3 and to 35.76 s in Session t4 (ANOVARM: F_{2, 18} = 6.95, p = 0.005). Consequently, the differences between Sessions t2 and t3 (t₉ = 2.38, p = 0.04) and Sessions t2 and t4 (t₉ = 3.67, p = 0.005) were significant. The ITR increased from 2.87 bits/min in Session t2 to 4.61 bits/min in Session t3 and 4.16 bits/min in Session t4 (ANOVARM: F_{2, 18} = 4.99, p = 0.02). Post hoc pairwise comparisons (t tests for paired samples) yielded significant differences between Sessions t2 and t3 (t₉ =  − 2.71, p = 0.02). This confirmed hypothesis h1; see Figure 2. Based on an individual level, performance increased from t2 to t4 in seven of ten participants. All seven reached accuracies above 70%.

Figure 2.

Accuracy of letter selection, information transfer rate (ITR), and selection time in seconds averaged over all participants for each of the three sessions separately. Vertical lines were placed over the bars if the difference was significant according to a t test for paired samples. A large star indicates a significance p < 0.01. Error bars show the SE of the mean

The mean amplitude of the maximum peak at Cz increased from 2.66 μV (SD 1.32, range 1.17–5.41) in Session t2 to 3.47 μV (SD 2.8, range 0.72–10.55) in Session t4. Latencies of the peak increased from 444 ms (SD 179, range 258–668) in Session t2 to 446 ms (SD 204, range 219–703) in Session t4. The changes in amplitude (ANOVARM: F_{1, 9} = 2.38, p = 0.15) and the changes in latency (F_{1, 9} = 0.00, p = 0.96) were not significant. ERPs for t2 and t4 and a graphical representation of this data are shown in Figure 3. The analysis of the t‐values (see EEG analysis in Section 2) revealed that amplitudes on frontal channels decreased and amplitudes on central channels increased from t2 to t4.

Figure 3.

Changes in the response to the target stimulus were visualised using t‐values from the comparison of the responses during t4 compared to t2 in a matrix of all channels and a time window from −200 to 800 ms. Topographies of these t‐values at four time points illustrate decreases in amplitude on frontal channels and increases in amplitude on central channels. The time course of the EEG amplitudes in μV was similarly affected by the training (see curves for Fz, Cz, and Pz comparing target responses during t2‑t4). Amplitudes on Cz increased (not significant according to repeated measures ANOVA), and latencies of the peaks were not affected [Color figure can be viewed at http://wileyonlinelibrary.com]

3.2. h2: Differentiability of attended and non‐attended stimuli in the fMRI

Stronger brain activation to attended stimuli as compared to non‐attended stimuli was found in both fMRI sessions (see Figure 4a), confirming hypothesis h2. The first cluster of activation was found in the left and right putamen and the left and right precentral gyrus. This cluster additionally included activation of the inferior frontal gyrus (IFG) also covering the Brodmann area (BA) 44. A second cluster of activation was localised around the middle cingulate cortex (MCC) and SMA in the posterior medial frontal cortex. A third cluster was localised at the right superior temporal gyrus (STG) with a corresponding fourth cluster on the left side. This included the activation of BA 22. Activation to attended stimuli was weaker in the left hemisphere than in the right hemisphere. Further significant activation was found in the supramarginal gyrus (BA 40). In Figure 4b, target versus non‐target contrasts are shown for t1 and t5 separately. The strongest activations during t1 were again found in the putamen, precentral gyri, MCC and SMA, and the right supramarginal gyrus (BA 40).

Figure 4.

Contrast between target and non‐target (unc. p < 0.001, N = 30) brain activations averaged across Session t1 and Session t5 (a). Target versus non‐target differences for Session t1 and Session t5 separately (b). Differences between target versus non‐target (unc. p < 0.001, N = 30) between Session t1 and Session t5 (c). Regions with stronger brain activations during Session t1 (before training) are shown in blue, and regions with stronger activations (both unc. p < 0.05, N = 30) during Session t5 (after training) are shown in red. Separate colour bars representing t‐values are shown for each panel. Images are shown superimposed on a canonical brain image (ch2better template) [Color figure can be viewed at http://wileyonlinelibrary.com]

3.3. h3: The effects of training on brain activation

The second‐level contrast comparing brain activation between t1 and t5 (see Figure 4c) revealed stronger activation during t1 in the precentral gyri (including BA 44), superior temporal gyri (BA 22), SMA and MCC, and stronger activations during t5 in the superior frontal gyrus and the middle occipital gyrus (BA 19 [cytoarchitectonic area hOc4lp]).

An investigation of the BOLD signal change over time showed that the effect that appeared as a deactivation in the contrasts shown in Figure 4c was in fact a stronger rebound at t5. After an initial peak, the signal at t5 decreased stronger in amplitude than at t1. To localise these non‐event‐related changes in the data, the difference between t1 and t5 was investigated using epochs between 10 and 16 s (see Figure 5). The contrast showed increased activation only before the training (t1 > t5). This analysis revealed that frontal areas decreased in activation across sessions, in particular the superior medial gyrus. In the supramarginal gyrus, STG, and MCC (not shown in Figure 5), a stronger rebound effect was found at t5 compared to t1. Furthermore, the t1 > t5 contrast was more pronounced in the right hemisphere. In summary, these changes confirmed hypothesis h3.

Figure 5.

Regions with stronger brain activations during Session t1 (before training) using epoch‐based first level analysis (10‑16 s after stimulus presentation, second level contrast with unc. p < 0.01, N = 30) in the top row. Mid and bottom rows show the regions with positive and negative signal changes in this time period. BOLD time courses extracted from the superior medial gyrus and supramarginal gyrus (bottom). The colours indicate the signal changes in %. Positive changes above 0.004% at t1 are shown in red, and negative changes below −0.004% at t5 are shown in blue. The mean signal change was calculated from 10 to 16 s. Responses to attended (target) stimuli are shown with continuous lines, and responses to ignored (non‐target) stimuli in dashed lines. Responses at t1 are shown in orange and responses at t5 in yellow. The shaded area of the curves indicates the extent of the SE. Time courses shown were based on the event‐related analysis [Color figure can be viewed at http://wileyonlinelibrary.com]

3.4. e1: Influence of performance on brain activation

Seven participants increased their performance and were assigned to the group of “learners,” three participants did not increase and were assigned to the group of “nonlearners.” The analysis of the effect of performance (see Figure 6) suggests that successful learning was related to activation in the STG, postcentral cortex, the calcarine sulcus, and lingual gyrus in the occipital lobe.

Figure 6.

The main effect of performance on brain activation (unc. p < 0.01, N = 30). The participants were split into a group of learners (N = 7) and non‐learners (N = 3). This split was introduced as an additional factor in the second‐level analysis of the epoched data. STG: superior temporal gyrus [Color figure can be viewed at http://wileyonlinelibrary.com]

3.5. e2: Overlap of activation during SMR and P300 BCI control

The conjunction analysis (see Figure 7) revealed substantial significant (p <.001; cluster size 30 voxels) overlap of activation for the SMR BCI and the P300 BCI in the left and right inferior frontal gyrus, the left and right putamen, and the SMA and MCC. Smaller clusters of overlapping significant activation were found in the left precentral gyrus and left middle frontal gyrus.

Figure 7.

The top row shows the brain activation from attended versus ignored targets in the current study in shades of white to red and the brain activation from motor imagery versus the rest in shades of green to blue (Halder et al., 2011). Both contrasts were thresholded with an FWE corrected p < 0.05 and a cluster size of 30 voxels. The bottom row shows the conjunction of motor imagery and attention to auditory stimuli. The conjunction was thresholded with an uncorrected p < 0.001 and a cluster size of 30 voxels. MFG: middle frontal gyrus; SMA: supplementary motor area; MCC: middle cingulate cortex, putamen; IFG: inferior frontal gyrus [Color figure can be viewed at http://wileyonlinelibrary.com]

Tables with coordinates and activation values of all analyses can be found in the Supporting Information.

4. DISCUSSION

Ten participants were trained with an EEG BCI (t2, t3, t4; h1) and brain activation associated with attending auditory target stimuli as compared to non‐targets using fMRI (t1, t5; h2) was investigated. The effect of training was reflected in significant changes in the brain activation of areas associated with the P300 ERP component (h3, e1). Finally, there was significant overlap between previously recorded motor imagery data and the current data (e2).

4.1. EEG BCI performance

Training with the EEG BCI (h1) led to improved ITRs in seven of ten participants. These seven participants achieved above 70% accuracy, which is the minimum accuracy for meaningful communication (Kübler et al., 2001). In Baykara et al. (2016), seven of the eight participants using the same BCI as in the current work reached above 70% and also the ITR was higher (3.88 bits/min over all sessions in the current study compared to 5.33 bits/min in Baykara et al., 2016). The participants in both studies reached the highest ITRs in the second EEG BCI training session (4.61 bits/min in the current study and 5.90 bits/min in Baykara et al., 2016). In both cases, the EEG data were recorded with 16 channels using the BCI2000 software and processed using the same signal processing pipeline. Thus, one can assume that the difference in ITR was related to inter‐individual differences.

The decrease in performance ITR from t3 to t4 may be indicative of an inverted u‐shaped (Yerkes & Dodson, 1908) profile, which would suggest that there are no lasting training effects or that training may even deteriorate the performance. However, other studies with the auditory P300 BCI do not support this assumption. Baykara and colleagues also observed a decrease in performance from the second to the third EEG training session but after the third session performance stabilised (Baykara et al., 2016). In a further study over three sessions with healthy participants, the strongest increase in ITR occurred between the first and second sessions (Halder, Takano, et al., 2016). In both studies as in the current one, most of the learning occurred between the first and second training sessions, which are compatible with Logan's instance theory of memory and learning (Logan, 2002). Thus, the training paradigm should be adapted after the second session to achieve further improvements. In our study with motor impaired end‐users, performance drops were visible in the fourth session (two cases) or not at all (one case) in five sessions of training Halder, Käthner, and Kübler (2016). Thus, the adaptation of the training paradigm may have to occur later when training motor impaired end‐users.

An analysis of the full feature matrix used in the current study indicated that the training led to increased amplitudes of the target response on central and decreased amplitudes on frontal channels. This increased the overall dynamic range of the minimum to maximum amplitudes of the event‐related response to the attended stimuli, which we assume was the cause of the improved classification accuracies.

Relative performance increases were successfully replicated (in the training period t2, t3, and t4) and the performance increases followed the same pattern as in Baykara et al. (2016); that is, strong increase after the first training, slight decrease from training 2 to training 3. Thus, we conclude that the training effect itself was replicated in the current study even if the absolute performance levels were lower.

4.2. fMRI assessment of effects of attention to target stimuli

The contrast between attended and ignored stimuli showed higher brain activation for the attended stimuli (h2) in frontal and precentral areas (BA 44), MCC and SMA, STG (BA 22), and supramarginal gyrus (BA 40). Strong subcortical activations were found in the left and right putamen.

The brain areas primarily responsible for generating the P300 appear to be the STG and supramarginal gyrus. This was determined using intracerebral recordings. Early sensory components of ERPs are generated in the STG and the later P3 component in the supramarginal gyrus (Halgren et al., 1995). In a study with a late stage ALS patient, Bensch et al. (2014) found strong auditory ERP responses in the STG. In Horovitz, Skudlarski, and Gore (2002), the activity in the supramarginal gyrus was shown to strongly correlate with ERP amplitude. Linden et al. (1999) proposed that the supramarginal gyrus is primarily responsible for the detection of the target. The supramarginal gyrus may form a network with frontal and precentral areas that is used for saliency detection. McCarthy, Luby, Gore, and Goldman‐Rakic (1997) attribute this network of inferior parietal lobule and frontal gyri to working memory, which is needed to maintain attention to the current target in the BCI task.

In Linden et al. (1999), notably, also the SMA was involved in this network. Other studies indicate that SMA is not only active during motor tasks but is also involved in a broad range of non‐motor functions (Nachev, Kennard, & Husain, 2008). A common factor among these tasks appears to be the need for sequential processing (Cona & Semenza, 2017). In line with this view, the area at the junction of SMA and anterior cingulate has been found to be activated in covert auditory attention tasks (Benedict et al., 2002). In a combined EEG‐fMRI study by Eichele et al. (2005), EEG‐correlated activations during an oddball task were found in the STG, supramarginal gyrus, and frontal cortex but not the SMA. This may indicate that this area may be activated on a different timescale or that it does not correspond to an ERP on the scalp, but is indicative of the cognitive task that is being performed. The fact that some areas may be active during the oddball task in the fMRI but not correlated to the EEG was also suggested by Horovitz et al., (2002). The authors differentiated between EEG‐correlated activity and fMRI activity that differed from baseline but was uncorrelated with the EEG (indicating that certain brain regions that are activated in the oddball task were reflected in the fMRI data but not the ERP data). In studies not focused on ERPs, activity in the putamen and MCC has been shown to play a role in auditory spatial attention (Wu, Weissman, Roberts, & Woldorff, 2007).

The current findings and the literature allow the following interpretation: the ERPs underlying control of a P300 BCI were in fact reflected in the current fMRI data (IFG, STG, and supramarginal gyrus) and were accompanied by non‐P300 but task‐specific activations that were also found in previous studies (SMA, MCC, and putamen).

4.3. Effects of the training on fMRI assessed brain activation

The difference in brain activation between t1 and t5 was investigated to determine the effects of the training with the EEG BCI (h3). Research with animal models has demonstrated plasticity of the auditory cortex in adult primates. Recanzone, Schreiner, and Merzenich (1993) trained monkeys to discriminate frequency differences. The training led to an increase in the area representing the trained frequency range as compared to controls that received the same auditory stimulation, but were trained with a tactile discrimination task. In a study with human volunteers, Jäncke, Gaab, Wüstenberg, Scheich, and Heinze (2001) showed that changes in the brain activation measured with fMRI were dependent on the success in the auditory discrimination task. Thus, it seems probable that the activity in the human auditory cortex will exhibit changes as a consequence of training. These changes may be reflected in increased or decreased activation (Ohl & Scheich, 2005). A decrease in activation may indicate the recruitment of a more spatially confined network for the trained task, whereas the initial activation by the untrained task may be broader. Interestingly, in the current study, most regions decreased in activation after the training. One explanation is that this may be an indication of a recruitment of a more confined network due to the training (Ohl & Scheich, 2005). This turned out to be incorrect, as an investigation of the time courses showed that what appears as a negative response in the activation maps was due to a more pronounced rebound of the haemodynamic response after the training (see Figure 5). Due to the rapid stimulus presentation used in the current study, the activation of the brain regions could not return to baseline every time a new stimulus was presented. Thus, it is probable that the negative response visible in the activation maps and the stronger rebound in the time courses indicates stronger oscillations of the haemodynamic response due to the more focused attention to the targets at the end of the training. To examine this effect in more detail, an analysis of the epoch between 10 and 16 s after stimulus presentation was performed. Brain activation during t5 compared to t1 was increased (according to the aforementioned interpretation of stronger rebounds) in the STG and supramarginal gyrus, the area believed to be primarily responsible for the generation of the P3 (Halgren et al., 1995). The STG was also one of the regions that differentiated between “learners” and “non‐learners” in the current study. The analysis also revealed activation differences between t1 and t5 in frontal areas, in particular the superior medial gyrus. In contrast to the previous regions, the time courses revealed that the frontal activation did in fact decrease from t1 to t5 and this decrease was not due to a rebound. Lesion studies with humans that have sustained injuries to the superior medial gyrus suggest that this region is involved in working memory tasks and particularly spatially oriented processing (du Boisgueheneuc et al., 2006). A study using transcranial direct current stimulation came to the conclusion that the superior medial cortex influences inhibitory control, for example when choosing from a range of possible actions and inhibiting responses in certain cases (Hsu et al., 2011). In the current study, this could correspond to the attention to the target stimulus and ignoring the non‐target stimuli. Thus, it may be concluded that the higher cognitive components of the BCI task (remembering which stimulus to attend to, attending this stimulus, and ignoring the others) became less demanding with training.

According to the current analysis, the activity in the IFG also decreased with training. This was unexpected due to the apparent involvement of the IFG in the generation of the P300 found in other studies (Linden et al., 1999). In general, the IFG appears to be involved in semantic processing of words (Thompson‐Schill, D'Esposito, & Kan, 1999). This particular function may require a stronger recruitment of the IFG initially, which may decrease in activity after the training because the participants need less effort to identify the auditory stimuli.

Interestingly, in our exploratory analysis of the effect of performance, we found activations in the lingual gyrus in the occipital lobe, which has been shown to be active during reading (Mechelli, Humphreys, Mayall, Olson, & Price, 2000). Since no visual information was shown to the participants during the fMRI measurements, this effect may be related to semantic processing such as memory of the current target letter. Another interesting observation from the fMRI epoch analysis was the strong right hemispheric dominance. This is the case for general attention tasks, as reviewed by Coull (1998) and was also found by Eichele et al. (2005) for brain activation during oddball tasks. In the current study, of the non‐P300 specific region, only the neural activity in the MCC increased due to the training (putamen and SMA remain constant). This may indicate an increase of the participants' ability to direct auditory spatial attention to task relevant stimuli (Wu et al., 2007).

4.4. Overlapping activation across BCI paradigms

Strong overlapping activations between motor imagery data from Halder et al. (2011) and P300 data (e2) of the current study were found in a cluster ranging from the SMA to the MCC and a cluster including the putamen and the IFG. In their recent review, Camilleri et al. (2018) have described the function of the putamen as part of a network of interacting subcortical processing loops that are involved in controlling human interaction with the environment via motor responses (motor imagery BCI) and in processing information from the environment (P300 BCI). In the same review, SMA and MCC were assigned higher cognitive functions such as sensation and action preparation (motor imagery BCI) and working memory and attention (P300 BCI). It is also worth noting that the SMA was one of the regions found to be most often activated across the different tasks that were analysed in the review by Camilleri et al. (2018). This may indicate that the conjoint activation marks a network, processing task components, that are common for motor imagery and the current BCI task, similar to, for example, the multiple‐demand network (Müller, Langner, Cieslik, Rottschy, & Eickhoff, 2015) and the extrinsic mode network (Hugdahl, Raichle, Mitra, & Specht, 2015). The final region involved in both BCIs was the IFG encompassing both BA 44 and 45. These areas have been shown to be relevant for semantic processing. In addition, this region encompasses the insular cortex. Together with the SMA and MCC, the insular forms the so‐called “saliency network” (Menon & Uddin, 2010). The insula is involved in responding appropriately to different stimuli, which includes detection of salient events and attention switching (required for the P300 BCI task) and also access to the motor system (required for the motor imagery BCI task). Thus, the current data suggest that P300 and motor imagery BCI conjointly activate the “organisers” and the sensation and action groups of the multiple demand network (see figure 6 in Camilleri et al., 2018).

4.5. Role of SMA

The SMA has been shown to play a role in SCP and motor imagery BCIs in humans and in studies with monkeys controlling a neuronal interface (Carmena et al., 2003; Halder et al., 2011; Hinterberger et al., 2005; Marchesotti et al., 2017). Interestingly, neural activation in the SMA has been found to be correlated with performance in auditory attention tasks, similar to the previous findings relating to motor imagery BCI performance (Seydell‐Greenwald, Greenberg, & Rauschecker, 2014). SMA activation has also been found in spatial attention tasks. Classically, the SMA is thought to be part of attentional control systems (Posner & Petersen, 1990), whereas Hopfinger, Buonocore, and Mangun (2000) speculated that the function of the SMA may be to analyse the stimulus for the features of the target. The current result provides further support for the notion that the role of the SMA during BCI tasks is not strictly motor related (see Nachev et al., 2008 for a review), because the current auditory BCI did not involve a motor task‐component. It is also interesting to note that no increase in the activation of the SMA was found due to the training, although performance increased over time, which also indicates a more supervising role of the SMA than a direct involvement in task execution. Cautiously, one may speculate that in those BCI users with high performance, such monitoring is no longer necessary to the extent seen in less good performers. Thus, SMA activation may not increase linearly with performance, but rather asymptotically.

As mentioned previously, the SMA is involved also in SCP and motor imagery BCI tasks. This suggests that there are universal components of BCI aptitude, and the role of the SMA may primarily be to monitor proper task execution. Currently, based on observations from available data, there has been no indication that P300 BCI and motor imagery BCI aptitude are correlated. Thus, it is not probable that the SMA is the only factor that determines performance. For example, emotional stability has recently been shown to be predictive of P300 BCI performance (Hammer, Halder, Kleih, & Kübler, 2018). Thus, one may suggest that the SMA is involved in higher order control or monitoring of task instantiation of various types of BCI, but areas linked to task execution may have an equally strong impact on performance and will vary depending on the BCI input signal.

5. LIMITATIONS

The low number of sessions is a clear limitation of the current study. A longitudinal design would be necessary to define approaches to increase the performance of the participants beyond the current level of improvement. A second limitation is the small sample size in the current study of 10 participants only. A larger sample size would enable us to fully explore the differences between learners and non‐learners (currently limited to the exploratory analysis e1). Unfortunately, such a study would require considerable resources. The design of the current study already required a total of 50 sessions. Finally, we artificially limited performance to 70% accuracy to avoid a ceiling effect often seen in our previous studies (Baykara et al., 2016; Halder, Käthner, & Kübler, 2016; Halder, Takano, et al., 2016). However, this may have a negative influence on the behaviour or mood of participants if they perceive the lack of improvement in accuracy as a lack of improvement overall. We could have set the limit of achievable accuracy higher, that is, between 90 and 100%, but this would increase the probability of ceiling effects.

6. CONCLUSIONS

The current study has demonstrated that training with a BCI affects not only behavioural variables but also related brain activation. For the first time, an overlap of activation could be identified between brain regions that are active during motor imagery and P300 BCI tasks, thus BCI independent interventions focusing on training these brain regions may be a key for improving the performance of end‐users with low initial aptitude (Botrel, Acqualagna, Blankertz, & Kübler, 2017). An alternative (or complementary) approach to activate such brain regions may be stimulation methods such as transcranial direct current stimulation; see Zich et al. (2017). Aptitude prediction and conjunction analysis of P300 and motor imagery data further highlighted the importance of the SMA for BCI control, thus training and/or stimulation of this region may lead to increased performance. Based on the current study, no conclusions can be drawn concerning the effects of long‐term training with BCIs (Saeedi, Chavarriaga, & Millan, 2017), as only three sessions were conducted.

In summary, training with a P300 BCI was shown to increase activation in superior temporal and supramarginal gyri and decrease activation in frontal regions, indicating that training improved stimulus perception and processing and reduced mental workload requirements. The SMA may contribute to higher order task coordination necessary in motor imagery and ERP‐BCIs. These findings highlight the potential value of developing well‐designed training protocols and making the user experience as engaging as possible to facilitate focused attention (Jeunet, N'Kaoua, & Lotte, 2016; Kosmyna & Lécuyer, 2017) in order to support fast acquisition of successful BCI communication. Last but not least, optimised training may resolve the issues of BCI inefficiency (Kübler et al., 2014).

Supporting information

Appendix S1: Supplementary Information

Halder S, Leinfelder T, Schulz SM, Kübler A. Neural mechanisms of training an auditory event‐related potential task in a brain–computer interface context. Hum Brain Mapp. 2019;40:2399–2412. 10.1002/hbm.24531