An auditory Brain-computer interface (BCI)

Introduction

Patients with severe motor paralysis need muscle-independent communication technologies. Communication is of utmost importance for the quality of life of such patients (Bach, 1993). Especially when patients have neurological or muscular diseases, such as amyotrophic lateral sclerosis (ALS), which leads to motor impairment as the disease progresses, the most horrifying outlook of the patients is the so-called locked-in state (LIS), in which only residual voluntary muscular control remains, rendering communication difficult or even impossible. In the complete locked-in state (CLIS) all voluntary muscular control is lost. Brain-computer interfaces (BCIs) are systems that translate brain activity into signals controlling external devices (Birbaumer et al., 2006). BCIs can maintain communication in severely paralyzed patients (Birbaumer et al., 1999; Kübler et al., 2001).

Many CLIS patients have compromised vision, and may not be able to use a visually-based BCI. Thus BCIs on the basis of other sensory modalities need to be explored. Usually the auditory system is uncompromised in these patients.

The feasibility of an auditory BCI has been investigated in few studies using different EEG input signals, e.g., P300 evoked potential and slow cortical potentials, which will be briefly reviewed. Sellers and Donchin tested healthy volunteers and patients with ALS with a 4- choice P300 BCI. Patients were presented either visually or auditorily or both with the words “yes”, “no”, “pass”, and “end” (Sellers and Donchin, 2006). The patients’ task was to focus their attention on either “yes” or “no”. The authors were able to show that a target probability of 25% was low enough to reliably elicit a P300 and that this response remained stable over a period of 12 sessions in healthy volunteers as well as in ALS-patients.

Hill and co-workers (Hill et al., 2005) attempted to classify P300 evoked responses that occurred in response to two simultaneous presented auditory stimulus streams. Both streams constituted an auditory oddball paradigm. To choose one of two possible targets (binary decision), the participant had to focus on either one of the streams. When attention was focused on the target stimuli (e.g., by counting them), EEG responses to target stimuli and standard stimuli could be classified. Although variation between participants existed, classification results suggested that it was possible for a user to direct conscious attention, and thereby modulate the event-related potentials that occur in response to auditory stimuli reliably enough, in single trials, to provide a useful basis for an auditory BCI.

Slow cortical potentials (SCPs) constitute another component of the EEG which can be used as a BCI input signal (Birbaumer et al., 1999). Hinterberger and colleagues compared learning to regulate the SCP amplitude of the EEG by means of visual or auditory feedback or a combination of the two modalities (Hinterberger et al., 2004). Three groups of 18 participants each underwent three sessions of SCP training. More than 70 % correct responses were achieved by six participants with visual feedback, by five participants with auditory feedback and only two participants with combined feedback. The average accuracy in the last session was 67 % in the visual condition, 59 % in the auditory condition and 57 % in the combined condition. Thus, in this study, visual feedback was superior to the auditory and combined feedback, but BCI control could be also achieved with auditory feedback.

An auditory BCI which uses sensorimotor rhythm (SMR) as an input signal has not been tested. Most adults display SMR in the EEG recorded over sensorimotor cortices (Niedermeyer and Lopes da Silva, 2005). The major component of SMR is often called the mu rhythm and is in the alpha band (8–12 Hz), usually accompanied by changes in synchronization in the beta band (13–25 Hz). The SMR amplitude increases or decreases related to motor movement. More important for BCI research, SMR amplitude also changes as a function of motor imagery. A recent study with visual feedback of SMR amplitude with four paralyzed ALS patients showed successful SMR regulation for all patients (Kübler et al., 2005). Performance was over 70 % in all patients (Kübler et al., 2005), and thus sufficient for communication (Perelmouter and Birbaumer, 2000). This makes the SMR-based BCI a good candidate for patients. However, whether the SMR-based BCI is feasible when auditory feedback of SMR amplitude is provided remained an open question.

In the present study, we compared BCI performance based on auditory or visual stimuli of the SMR amplitude in healthy volunteers. We aimed at answering two questions: First, can healthy participants achieve the same level of performance with auditory feedback as with visual feedback? Second, are there differences in learning as a function of feedback modality? Additionally, we were interested whether psychological variables, namely mood and motivation, would influence performance.

Materials and Methods

Participants

Participants were 16 students from the University of Tübingen (6 men, 10 women) with no history of neurological or psychiatric disorder, who were divided in two groups. Participants were paid 8 € per hour and all were naïve with regards to BCI training. The average age of the participants was equal in both groups (auditory feedback group (26.75 ± 4.13); visual feedback group (24.13 ± 2.64)). Participants gave informed consent for the study, which had been reviewed and approved by the Medical Ethical Committee of the University of Tübingen and the National Institutes of Health (NIH). Each participant sat in a reclining chair facing a computer screen and loudspeakers and was asked to remain motionless during performance. After an initial evaluation defined the frequencies and scalp locations of SMR rhythm activity (see Table I), each volunteer participated in three two-and-a-half hour sessions on separate days within one week with one or two day breaks in which they learned to regulate their SMR rhythm amplitude. Participants were instructed to imagine the very movement (e.g. right hand or feet movement imagery) which resulted in greater SMR amplitude differences during the initial evaluation. Each session consisted of three blocks of ten approximately two-minute runs (130 seconds) divided by one minute breaks. Each run consisted of 23 trials.

Table I:

Initial electrode position, initial frequency band from which participants were provided with feedback and R² (the proportion of the single-trial variance that is due to the target location) for the first training session.

auditorv feedback				visual feedback
participant	initial electrode	initial frequency	R2	participant	initial electrode	initial frequency	R2
A1	C4	23.5–26.5	0.35	V1	CP3	7.5–10.5	0.23
A2	CP3	22.5–25.5	0.25	V2	C4	10.5–13.5	0.31
A3	Cz	18.5–21.5	0.45	V3	CP4	10.5–13.5	0.30
A4	C3	10.5–13.5	0.45	V4	CP4	10.5–13.5	0.17
A5	C4	10.5–13.5	0.25	V5	C4	22.5–25.5	0.21
A6	Cz	8.5–11.5	0.23	V6	C3	10.5–13.5	0.19
A7	C3	10.5–13.5	0.31	V7	C4	10.5–13.5	0.41
A8	CP3	16.5–19.5	0.21	V8	CP3	10.5–13.5	0.54

Visual feedback

During each trial in the visual feedback condition, participants were presented with a cursor on the left edge of the screen and a target consisting of a red vertical bar that occupied the top or bottom half of the right edge of the screen. The cursor moved steadily across the screen with its vertical movement controlled by SMR amplitude. Motor imagery reduced or desynchronized SMR amplitude and moved the cursor down toward the bottom target, while no imagery led to SMR synchronization and moved the cursor upward (top target). The participants’ task was to move the cursor so that it hit the target when it reached the right corner. During a reward period the target flashed yellow if hit. Specifically, cursor control is a continuous function of spectral amplitude at a pre-defined SMR frequency (see Table I). For one-dimensional movement we use a single regression function:

$$\Delta {\text{V}=\text{b}}_{\text{v}}\left({\text{S}}_{\text{v}}-{\text{a}}_{\text{v}}\right)$$ (1)

where S is the control signal (weighted sum of features), a is the estimated mean of the control signal, and b is the gain term that controls the size of the cursor step. If subjects correctly modulated their EEG they would tend to hit the correct target. In contrast, if they could not control their EEG the cursor would move up and down randomly and either target would be selected equally often.

Auditory feedback

Feedback of the SMR amplitude was realized by either harp or bongo sounds. SMR desynchronization was represented by bongo sounds and synchronization by harp sounds. The sounds (in .wav format) were taken from a commercially available CD (100 spectacular sound FX volume 8) and were adapted to have the same length and maximal loudness with Cool Edit Pro 2.0. The loudness of the sounds corresponded to the amount of either desynchronization or synchronization of SMR. During each trial the target was presented by a voice-recorded instruction through the loud speakers. When motor imagery or SMR desynchronization was required the instruction was ‘Aufgabe Bongos’ (which translates to ‘task bongos’), and for SMR synchronization ‘Aufgabe Harfe’ (which translates to ‘task harp’).

When participants produced sounds according to the task requirement, they were reinforced by a voice saying “richtig!” (‘correct!’), when they failed the voice said “falsch” (‘incorrect’). One could argue that participants with their eyes closed would resemble CLIS patients without vision. On the other hand, the eyes open condition would be more comparable to the procedure used for the group trained with visual feedback. Thus, participants were asked to keep their eyes open and focus on a fixation cross presented in the center of the computer screen.

In the visual feedback mode, the cursor moved up and down randomly and either target would be selected equally often when no modulation of SMR occurred. With auditory feedback the following problem arises: how to give auditory feedback when no SMR modulation occurs? There are two possibilities to resolve this problem: First, no classification no sound (= silence), and second, equal presentation of both sounds. As no data were available as to whether silence would be better than simultaneous presentation of both sounds, we conducted the following pilot study.

Pilot study for feedback of unclassifiable SMR modulation

Twenty healthy participants (14 women, 6 men; age range: 18 – 53) had to learn how to move a computer mouse in a predefined direction. Moving the mouse in one direction produced an increase in bongo sounds, whereas a movement in the other direction produced harp sounds. Each volunteer participated in 16 blocks of 20 trials each. In each block the degree of the axis in which the mouse was supposed to be moved was randomly changed between 0 and 360 degrees (see Figure 1). The more the direction of the mouse movement corresponded with the target direction the more the harp or bongo sounds increased in loudness. Movement around the center of the axis corresponded to unclassifiable SMR modulation.

Figure 1:

Schematic of one of sixteen possible angles (a) in which the mouse was supposed to be moved. When, for example, the mouse was moved in a 30 degrees angle to the upper right, harp sounds were produced. When the mouse was moved in a 210 degrees angle to the lower left, bongo sounds were produced. Participants had to figure out the correct angle by trial and error with the aids of auditory feedback.

Participants were divided in two groups. In the “silence” group no sounds were provided when the mouse moved around the center of the axis and in the “sounds” group both harp and bongo sounds were presented. Before each trial a computer-generated voice indicated the target (‘task harp’ or ‘task bongo’). Each trial lasted 3 s.

We found no difference in performance measured as percentage of correct responses (mouse movement in the required direction) between the “silence” group (mean performance = 83.65 %; SD ± 2.48) and the “sound” group (mean performance = 82.27 %; SD ± 2.48). Thus, we arbitrarily chose to present both sounds when SMR amplitude modulation was not classifiable.

Feedback trials

The timing of trials during both the visual and auditory feedback was identical. The inter-trial interval was 2.5 seconds. At the beginning of each trial the target was presented within 1.25 seconds. Then either visual or auditory feedback was provided for 2.5 seconds, followed by a reward period of 1.25 seconds (see Figure 2).

Figure 2:

Timing of the trial design in both the auditory and the visual feedback group.

EEG recording

Scalp electrodes (in a cap) recorded 16 EEG channels (FP1, FP2, F3, Fz, F4, T7, T8, C3, Cz, C4, CP3, CP4, P3, Pz, P4, Oz) (right-ear reference; bandpass 0.01 to 70 Hz; sampling rate 160 Hz). After every session, training data of one session were analyzed and evaluated for possible alterations of the electrode position or frequency band where most task related SMR modulation occurred.

Psychological variables

Regulation of SMR amplitude involves a learning process. Learning processes are influenced by psychological variables such as mood and motivation. To investigate the relations between mood, motivation and performance, each participant filled out two questionnaires prior to each session. First, current mood was measured using a subscale of a German inventory to assess quality of life (Skalen zur Erfassung der Lebensqualität, SEL; Averbeck et al., 1997). The questionnaire consisted of 10 statements which had to be rated on a 5-point Likert scale to the extent that they applied to the participant in the current situation. Second, motivation was measured using the Questionnaire for Current Motivation (Rheinberg et al., 2001), which was adapted to the situation of BCI training. On a 7-point Likert scale participants rated items that assessed four different components of motivation: 1) mastery confidence, which indicated how much confidence a participant had that the training would be successful, 2) fear of incompetence, which indicated how much a participant feared to fail in the training, 3) interest, which indicated how interested the participant was in the training and 4) challenge, which indicated how challenging the participant considered the training.

Data analysis

Accuracy in each block was defined as the percentage of hit targets or achieved sounds. To compare BCI performance as a function of modality a 9×2 repeated measures ANOVA was calculated with blocks (9) as within and feedback modality (2) as between subject factors. To investigate learning in each group, linear trends were calculated for individual data. Multiple regression analysis was performed to investigate the predictability of each psychological parameter (mood, mastery confindence, fear of incompetence, interest and challenge) on performance. Since each participant provided data points on three sessions, the effects of each psychological parameter were adjusted for the effects of session.

Results

Group data

Performance for both groups are presented in Figure 3. The 9×2 repeated measures ANOVA revealed a main effect of feedback modality (F8,1 = 8.326; p < .05) and a significant interaction between block and modality (F8,1 = 3.757; p < .001), but no main effect of blocks. With an average performance of 74.1 % (SD ± 13.45) the visual feedback group performed better than the auditory feedback group (55.96 %, SD ± 10.26).

Figure 3:

Average accuracy per block for the visual feedback group (solid line) and the auditory feedback group (dashed line).

Simple pairwise comparisons of blocks between the groups revealed that in the first block the visual feedback group performed significantly better than the auditory feedback group (t = −6.384; df = 14; p < .001), whereas performance did not differ in the last block (t = 0.447; df = 14; p = 0.447). From block 6 on, performance did not differ between groups (all comparisons in blocks 6–9 ns; uncorrected for multiple comparisons). Performance increased from the first to the last block in the auditory feedback group (t= 0.043; df =14; p<.05), but not in the visual feedback group (t= 1.135; df= 14; p=.275).

Individual data

Performance in the first and last block for each participant is listed in Table II. To investigate individual learning we calculated linear trends with percentage of correct responses as a function of blocks for each participant (Table II). Six participants of the auditory and 2 of the visual feedback group had significantly increasing linear trends. Three participants of the visual feedback group had significantly decreasing trends.

Table II:

Performance of all participants in the first and the last block. F-values and P-values indicate significance of linear trends as calculated with linear regression analysis.

auditory feedback				visual feedback
participant first block last block linear trend				participant first blocklast block linear trend
A1	55.29	77.65	F1/7= 29.34	V1	62.17	78.8	F1/7= 11.24
			p= 0.001				p= 0.012
A2	44.71	52.94	F1/7= 0.094	V2	61.31	48.70	F1/7= 3.22
			p= 0.094				p= 0.116
A3	46.47	76.47	F1/7= 7.43	V3	90,0	99.13	F1/7= 10.78
			p= 0.030				p= 0.013
A4	51.76	78.24	F1/7= 15.81	V4	77.4	58.26	F1/7= 21.27
			p= 0.005				p= 0.002
A5	51.76	38.24	F1/7= 5.11	V5	82.17	68.70	F1/7= 10.57
			p= 0.058				p= 0.014
A6	40.59	45.88	F1/7= 1.79	V6	88.7	83.00	F1/7= 0.41
			p= 0.223				p= 0.543
A7	59.41	85.29	F1/7= 7.23	V7	83.93	79.14	F1/7= 0.69
			p= 0.031				p= 0.434
A8	47.65	59.41	F1/7= 11.07	V8	89.3	52.60	F1/7= 21.57
			p= 0.013				p= 0.002

According to Perelmouter and Birbaumer (2000) in a BCI with binary choices 70% accuracy is necessary for communication. In the last block 4 participants of the auditory feedback group and 4 of the visual group achieved such high a performance.

Psychological Data

All participants but one (AFA in session 1) filled out the questionnaires before each training session. Thus, three data points could be obtained for each participant, except AFA.

In Figure 4, performance per session is plotted against ratings of mood, mastery confidence, fear of incompetence, interest and challenge assessed prior to each session.

Figure 4:

From top to bottom performance is plotted against mood, mastery confidence, fear of incompetence, interest and challenge. Left column: data from the visual feedback group, right column: auditory feedback group. Diamonds represent the ratings prior to session one, triangles prior to session two and squares prior to session three.

Multiple regression analysis showed a main effect of mood (b= .498; p<.05), mastery confidence (b= .578; p< .05) and fear of incompetence (b= −.616; p< .05) on performance in the visual feedback group. This indicates that higher scores of mood and mastery confidence were related to better performance. In contrast, higher ratings of fear of incompetence were related to worse performance.

In the auditory feedback group only a main effect of fear of incompetence was found (b= .470; p< .05) indicating that higher scores of fear of incompetence were accompanied by better performance. Mood, mastery confidence, interest and challenge were unrelated to performance.

Since participants V4, V5 and V8 had a significant decreasing trend in their performance in the visual feedback training, we explored the relations between their mood, motivation and performance. V4 started with more than 70 % correct in the first block. Her mood worsened and her fear of incompetence increased in every training session. Similarly, V8 started with an accuracy of 89% and mood worsened and interest decreased with training. In V5 no such relations were found. Note, that psychological data were acquired before each training session.

Discussion

To summarize, in 3 training sessions visual feedback of SMR amplitude led to better average BCI performance than auditory feedback. This was due to high performance at the beginning of training in the visual feedback group. After 3 training sessions performance was the same in both groups. Thus, auditory feedback required more training, but lead to approximately the same level of performance at the end of training. We conclude, that a two-choice BCI based on auditory feedback is as feasible for communication as a BCI based on visual feedback, provided sufficient time for learning is allowed.

These results raise important questions: First, why does the visual feedback result in initially better BCI performance or in other words, why is it more difficult to learn with auditory feedback? The superiority of visual feedback is compatible with a study of Lal and colleagues (1998) who compared the effect of feedback modality (visual, auditory or combined) on the ability to regulate blood pressure. Participants could lower their systolic blood pressure more effectively using visual or combined feedback. It was suggested that attention processes may be in competition during the auditory feedback (Lal et al., 1998; Hinterberger et al., 2004). The superiority effect of visual feedback was also found by Hinterberger and colleagues (2004). The authors suggested that the auditory stimuli used as feedback for SCP amplitude changes might be distracting. From the results of the present study we also speculate that the lower initial performance with auditory feedback may be due to an increased demand for attentional resources in the auditory feedback group as compared to the visual feedback group.

Second, why did we see a high performance in most participants at the beginning of visual feedback training and then see a decline in performance in three participants? During the first ten runs (approximately 21 minutes) of the first visual feedback session six out of eight participants already showed a performance above 70 % correct. Five participants even showed a performance above 80 % correct during these initial runs. These results indicate that with visual feedback, participants have strategies immediately available to regulate SMR whereas auditory feedback seems to retard learning. The decline in performance is not expected, but might be explained by the results of the psychological data (see below).

Third, psychological variables were pointed out as important factors for feedback learning: The operant conditioning approach to regulation of physiological responses emphasised motivation as playing an important role in biofeedback learning (Miller, 1982; Yates, 1980; Kübler et al. 2001). Acquiring extensive control over brain responses constitutes a difficult, often frustrating or even boring task. The SMR based BCI task requires participants to rapidly shift between an activated, or aroused state (mu desynchronization) and a relaxed, idling state (mu synchronization). Desynchronization requires persistent effort and hence, persistent motivation (Miller, 1982). However, for relaxing, or “thinking of nothing” as required for synchronization, motivation that is too strong may disturb the correct responses. Symptoms of depression are also known to hamper executive function necessary to sustain attention during BCI training (Porter et al., 2003). To investigate whether BCI performance would be influenced by mood and motivation, we assessed both variables before each training session. In the visual feedback group better mood and greater mastery confidence were related to better performance and higher fear of incompetence with decreased performance. These results could not be found in the auditory feedback group. A remarkable finding was that three participants in the visual feedback group showed decreasing trends in performance. We might speculate that as performance for these three participants was already high at the beginning (77.4 %, 82.2 % and 89.3 % in the first block) the remaining training sessions were experienced as boring or stressful as indicated by a decrease of mood and interest and an increase of fear of incompetence as the training progressed. From the psychological and performance data provided in this study no causal relationships can be derived. However, we may speculate that when participants are successful from the start of training and perceive that they can control the feedback signal, mood and motivation increases as the training progresses, whereas if performance is high from the start the fear of failure in further sessions or a loss of interest may hamper performance and further learning. In the auditory feedback group the fear to fail in the training increased as learning progressed, which may indicate that participants fear to lose the competence acquired with training.

The relation between motivation, mood, quality of life and BCI performance in severely paralyzed patients is currently under investigation in our laboratory. Our unpublished psychological data of patients show that patients are highly interested and motivated throughout the training. This may be because most disabled participants entering our BCI study are highly aware of the communication problems in late-stage ALS and see their involvement in our studies as a way of anticipating and compensating these problems. Fourth, what do the results mean for disabled persons with impaired vision? For visual feedback we can compare the data from our healthy participants with four disabled participants with amyotrophic lateral sclerosis (ALS) in the study of Kübler and colleagues (2005) who showed a performance ranging from 55.4 to 61.9 % correct during the same amount of initial 10 runs with exactly the same methods. However, after sufficient training also the disabled participants performed higher than 75 % correct. In another study with slow cortical potentials Kübler and colleagues (2003) found that healthy participants showed much faster learning during the initial sessions than disabled participants with ALS. Thus, it seems that although participants with ALS learn slower than the healthy participants with visual feedback, they can achieve the same performance with sufficient training.

Fifth, it would be very interesting to investigate the learning process of disabled participants with impaired vision who are trained with auditory feedback. Since auditory feedback results in slower learning for the healthy participants one could speculate that disabled participants may have even more problems with an auditory BCI. However, one might also speculate that disabled BCI users, with impaired vision, have developed a better sense of hearing.

Another hypothesis is that there might be an intrinsic difference in the amount of information we can extract from the visual versus the auditory modality. Healthy people rely heavily on guiding their actions through the visual system. Therefore, maybe the visual system supports our learning better than the auditory system. Thus, it would be interesting to test blind, but otherwise healthy people, with the auditory feedback. We would hypothesize that they perform better than healthy participants with auditory feedback and as good as healthy participants with visual feedback.

We conclude that the results of this study are encouraging for the development of auditory BCIs, but their feasibility for LIS and CLIS patients remains to be shown. Furthermore, it seems that psychological factors such as mood and motivation are related to BCI performance.