Investigating the role of combined acoustic-visual feedback in one-dimensional synchronous brain computer interfaces, a preliminary study

Abstract

Feedback plays an important role when learning to use a brain computer interface (BCI), particularly in the case of synchronous feedback that relies on the interaction subject. In this preliminary study, we investigate the role of combined auditory-visual feedback during synchronous μ rhythm-based BCI sessions to help the subject to remain focused on the selected imaginary task. This new combined feedback, now integrated within the general purpose BCI2000 software, has been tested on eight untrained and three trained subjects during a monodimensional left-right control task. In order to reduce the setup burden and maximize subject comfort, an electroencephalographic device suitable for dry electrodes that required no skin preparation was used. Quality and index of improvement was evaluated based on a personal self-assessment questionnaire from each subject and quantitative data based on subject performance. Results for this preliminary study show that the combined feedback was well tolerated by the subjects and improved performance in 75% of the naïve subjects compared with visual feedback alone.

Introduction

The electroencephalography (EEG)-based brain computer interface (BCI) is intended to be a system that seeks direct interaction between the human brain and machines, aiming to augment human capabilities by enabling people (especially those who are disabled) to communicate and control devices by merely “thinking” or expressing intent. One of the main goals of BCI researchers is to build a communication pathway for the human brain that does not depend on its standard output channels, such as nerves and muscles.1–3

Such systems can be realized in two ways, ie, an externally paced mode (synchronous BCI) or an internally paced mode (asynchronous BCI). Synchronous BCI requires the subject to achieve a specific mental state in response to an external event within a predefined time window, whereas in asynchronous BCI the subject is free to intend a mental state or a specific thought with no time restraint.4 Both methodologies make use of classified EEG signals; synchronous BCIs make use of oscillatory EEG activity5 and/or slow cortical potential shifts6 and, in asynchronous BCIs, various types of event-related potentials are used.7 The two most commonly used EEG oscillations are the Rolandic μ rhythm in the range of 7–13 Hz and the central beta rhythm above 13 Hz. Both originate in the sensorimotor cortex. These rhythms are not only linked to voluntary motor intentions, but as many recent studies have confirmed, they are also linked to the mental imagination of movements.8,9

Typically, visual feedback is given to the user, and the common control task is movement of a cursor on the screen of the BCI machine.10–12 In this scenario, it is recognized that feedback plays an important role when subjects are learning to control their brain signals,10,13 particularly for naïve subjects (never trained in BCI before).14–16

However, only a handful of studies have addressed the real role of visual feedback in BCIs.10 Furthermore, to our knowledge, the use of other modalities, such as auditory feedback, has not been studied in Rolandic μ rhythm-based BCIs. Given the broadly accepted importance that auditory stimuli have in the human brain, in particular the ability of the brain to locate sound events in space,17 we hypothesized that auditory stimuli could improve the performance in a lateralized Rolandic μ rhythm BCI task.

Moreover, with a few notable exceptions, most current BCI research evaluates the performance of BCI systems, and not the preference or comfort of the subject.14,16 The aim of this preliminary study was to compare the performance and preference of naïve subjects during a simple monodimensional control task with and without the aid of additional auditory feedback.

Materials and methods

The number of electrodes required for a full EEG recording and the related necessary subject preparation (which may include scalp scrubbing and abrasion) can be a significant burden in BCI use, particularly for a naïve subject. Therefore, in order to make the BCI session more comfortable and enjoyable for the subject, we reduced the number of EEG electrodes. We placed only three electrodes in the standard 10–20 positions of C3, C4, and Cz, referred to a reference electrode on the left ear lobe. Our custom EEG hardware allows use of dry electrodes kept in position with a drop of standard collodion applied directly on the surface of the electrode without skin preparation.18–20 However, in this study, in order to increase subject comfort, we avoided the use of collodion, and instead used standard brass electrodes kept in position with standard adhesive conductive paste which is washable with water. This allows a very quick set-up with an average set-up time per subject of only 2 minutes, including the time required to locate the electrode positions on the scalp via head measurement.

EEG signals were sampled at 512 Hz then processed by the BCI2000 AR signal processing module to drive both the visual and the auditory feedback.1 The auditory feedback is not provided by the default BCI2000 application module, so a new module has been compiled, as described later (the source code of the new compiled module is available for download as an additional file).

BCI task

The BCI task for this experiment was the standard one-dimensional left-right control task. The subject was instructed to imagine left and right hand movements in order to move a ball to a target randomly positioned on the screen.1 The moving ball (ie, the visual feedback) was driven by processed and classified EEG signals, ie, C3-Cz and C4-Cz.

Auditory feedback was implemented by shifting the frequency of a pure tone in relation to the position of the ball (visual cue) on the screen. The frequency was initially set at 600 Hz at the starting ball position (screen center). The frequency increased when the ball approached the target and decreased if it moved away from it. The volume of the sound was also panned between stereo speakers according to the direction of the ball’s movements. Additional auditory feedback was given for target hits. This was a 500 Hz tone for 200 msec for a correct hit and a 150 Hz tone for 200 msec for a miss. The stereo speakers were placed on two stands beside the armchair used for the experiment. In order to allow further customization, the volume panning and the frequency shifts can be regulated using our BCI2000 software module. In our experiment the value for frequency shifts was set to 10 Hz and the panning percentage to 5%, ie, each percent value of the ball movement corresponds to a volume panning of 5% and a frequency shift of 10 Hz.

The experiment was performed in a soundproof booth with dimmed light. Interaction between subject and operator was minimized, allowing feedback and communication only during the inter-trial intervals. Each trial duration was set to three minutes. There was no limitation on the number of targets displayed. To produce a reasonable number of measurements, the maximum feedback duration was fixed to 10 seconds, ie, if the subject was not able to hit the a target in the allowed 10 seconds, the attempt was classified as “aborted” and the next target displayed. We tested this protocol in 11 subjects, three of whom were well trained in BCI, nine being male and two being female. None of the subjects had a known history of neuropsychiatric illness. The average age of the subjects was 29.5 (range 24–43) years. Of these subjects, ten were right-handed and one naïve subject was left-handed.

Experimental protocol

Subject preparation

The subject’s head was measured using a standard head measurement tape. The C3, C4, and Cz locations were marked using a soft makeup pencil, and the electrodes applied without performing subject preparation. The task was explained, the subject was seated, and the electrodes were connected to the system. This phase (including explanation of the task) did not last more than 3 minutes. The naïve subjects were randomly divided in two subpopulations (Sa and Sb). Subjects assigned to population Sa were presented with the standard visual feedback available from BCI2000, and subjects belonging to subpopulation Sb were presented with the new combined visual/auditory feedback. To allow comparison between techniques using the same subjects, the type of feedback was changed once the subject scored greater than 60% of correct hits, or if the subject was fatigued. The following description of the BCI protocol phases are for Sa subjects. For Sb, phase 3 and 4 were reversed.

Familiarization task and parameter adjustment

Prior to the experiment, the quality of the EEG data was checked by performing the classic eyes open/close test (alpha wave reactivity).20,21 In all experiments, the subject was instructed to avoid blinking as much as possible. For the naïve subjects, a “familiarization” trial was performed. This trial also constitutes the training for both the subject and the classifier. Therefore, only the visual feedback was used in this first run. During this task, in order to assist the subject to concentrate better on imagination of the movement, a physical movement was required. The subject was instructed to depress a hand-held button to the side of the target displayed on the screen.

In order to avoid the subject attempting to control the ball, or being distracted by the moving object, the moving ball was not displayed in this familiarization task. The target was only lit in the case of correct hits. The subject was also instructed to focus on the feeling of movement rather than the movement itself in order to recall that brain activity during the imaginary task. It is well known that imagining the kinesthetic action rather than simple visualization of the movement is a stronger mental state, and hence improves the quality of the classified signals.5

The linear classifier available in BCI2000 was used to learn the relationship between physical movements and EEG response. At the end of the familiarization task, the adjusted offset and normalized gain values (parameters of the BCI2000 classifier) were saved for use in other tasks. The initial parameters for the classifier channel matrix were set to the default value of −1 and 1 on the 12 Hz frequency bin for the C3-Cz and C4-Cz channels, respectively.1

Visual feedback control tasks

In this phase, ten runs with the standard cursor task available in the BCI2000 package were performed, with a resting period of about a minute between trials. Each trial length was fixed to 3 minutes with a maximum feedback duration time of 10 seconds. As mentioned, the classifier for this task was not allowed to adapt values because these were fixed from the familiarization task.

Visual/auditory control tasks

In this phase, ten runs with an inter-trial break of one minute were performed using the enhanced application module capable of playing the auditory feedback in combination with the visual feedback. After a minimum of ten runs, a further arbitrary number of runs of either kind of feedback was performed. These were performed only on request. In practice, 90% of subjects were so enthusiastic that they requested several additional runs and asked to return the following day to play more runs using the auditory feedback.

Questionnaire

On completion of the experiment, subjects were asked to grade nine questions by assigning a score between one and ten. Questions were related to the feeling of the subject during the experiment (see Table 1). There was no time constraint for answering the questions, and the questionnaire was completed immediately following the experiment while the subject was still in the soundproof booth.

Table 1.

Questions list

Question	Grade weight explanation
1. Did you get tired because of the experiment?	1, not tired at all; 10, very tired
2. Did you enjoy the experiment?	1, not at all; 10, very much
3. Did you find the experiment comfortable?	1, not at all; 10, very much
4. Did you feel that you could control voluntarily the movement of the ball?	1, not at all; 10, perfect control
5. Did you find the sound helpful or disturbing?	1, not at all helpful, very disturbing indeed; 10, very helpful, necessary rather than disturbing
6. Did you feel frustration during the experiment?	1, no frustration; 10, a lot of frustration
7. Did you feel frustration during the experiment with only the video feedback?	1, no frustration; 10, a lot of frustration
8. Did you feel frustration during the experiment with the audio feedback?	1, no frustration; 10, a lot of frustration
9. How much likely you will agree to be test subject for this kind of experiment again?	1, not very likely (never again); 10, very much likely

Quantitative data analysis

Subject performance was assessed by computing the average of correct hits, the average percentage of incorrect hits, and the average percentage of timed out trials (aborted) achieved during phase 3 and phase 4. These were compared with the threshold of a 50% hit rate which represents random guesses.22 Computed data were compared across phase 3 and 4 for each subject in order to estimate the real contribution of the added auditory feedback. It is worth highlighting that for subjects with performance close to the 50% “random guess value”, a high number of “aborted” trials (in some cases more than 30% of the total trials) were recorded. These subjects also reported that the experiment was frustrating. Further details on these subjects are given in the Results section.

New application module and BCI2000 settings

The auditory feedback was added to the standard application module from the BCI2000 package by transforming the visual horizontal ball coordinate (x) and its differential (dx) value (previous position – calculated new position) into auditory frequency displacement and a volume pan percentage value. The frequency displacement value was added to the starting frequency (600 Hz for this experiment) if the new calculated x value could move the ball towards the correct target, or else subtracted for movements in the opposite direction. Volume was panned according to the direction of the new calculated x value. The starting frequency value was made available in the BCI2000 operator module as a configuration value customizable by the end user. The general loudness of the sound was adjustable by the subject in according to his/her comfort.

It is worth highlighting that the new x value was calculated by the signal processing module in accordance with the relative power spectra values of the EEG signals C3-Cz and C4-Cz for 30 different evaluations in 0.5 second time windows. It is known that physical and imaginary hand movements have a focus of activity in the μ rhythm bandwidth around C3 and C4,4,5,23 hence we used the frequency bin centered at 12 Hz, 3 Hz wide, initially weighted at −1 and 1, respectively. It is beyond the scope of this paper to give a full explanation about the meaning of these parameters because some are peculiar to the BCI2000 software package (a full explanation and tutorials about the use of this software are also available online at http:\\www.bci2000.org), and experimental parameters, binary, and source codes of the new realized module are available for research purposes upon request.

Results

In this section, data from trained and untrained trials are presented. We would like to highlight that EEG data recorded across the entire subject sample show the normal EEG patterns of an awake adult. An example of the sample data showing a clear lateralized (in this case on C3 represented in bold) μ rhythm in the time domain is shown in Figure 1. It is noteworthy that the phenomenon is much more evident in the frequency domain, in Figure 2. This is a direct comparison between EEG signals acquired from C3 (solid line) and C4 (dashed line), during the execution of left imaginary movement following the displaying of a left target (top panel) and the execution of right imaginary movement following the displaying of a right target (bottom panel). A large change in the signal power at 12 Hz and its first harmonic at 24 Hz can be observed for the change from left to right tasks associated with the electrodes located on the left and right (C3 and C4).

Figure 1.

Electroencephalographic example workspace as recorded from one subject showing clear lateralized μ rhythm pattern.

Figure 2.

Lateralized μ rhythm in the frequency domain. C3 solid line, C4 dashed line in a (A) left imaginary task and in a (B) right imaginary task.

Results for well-trained subjects

Given the high percentage of correct hits achieved by these subjects there was no need for a familiarization task. Instead, a single run with visual feedback was performed to confirm good subject performance. Results achieved from trained subjects for the visual feedback and for the combined audiovisual feedback are shown in Table 2. Results from the questionnaires are shown in Table 3. It is evident from the results in Table 2, columns 2 and 3, that for trained subjects, the presence of audio feedback did not affect their performance. This is almost constant throughout the experiment. However, the presence of audio feedback was welcomed by well trained subjects (Table 3, question 5).

Table 2.

Visual feedback results from trained subjects

Subject	Highest performance ever achieved	This run (visual feedback) single run	This run (combined audiovisual feedback) best achievement	Combined audiovisual feedback average (10 trials)
1	95%	83.2%	87%	81.2%
2	92.5%	78%	79%	75.6%
3	85%	71%	83.5%	72.9%

Table 3.

Questionnaire result from well trained subjects

Question	Grade subject 1	Grade subject 2	Grade subject 3
1. Did you get tired because of the experiment?	3	4	5
2. Did you enjoy the experiment?	8	8	8
3. Did you find the experiment comfortable?	8	8	8
4. Did you feel that you could control voluntarily the movement of the ball?	8	7	7
5. Did you find the sound helpful or disturbing?	8	9	9
6. Did you feel frustration during the experiment?	4	5	5
7. Did you feel frustration during the experiment with only the video feedback?	6	6	7
8. Did you feel frustration during the experiment with the audio feedback?	4	5	5
9. How much likely you will agree to be test subject for this kind of experiment again?	9	9	10

Results for naive subjects

Results obtained from naïve subjects are summarized in Table 4. Columns 4 and 6 show that average performances for subjects belonging to either Sa or Sb are close to the random guess (50%). Detailed analysis of the BCI session log for each subject show that, in some cases, the number of aborted trials (a trial is classified as aborted when it does not score a correct hit nor a wrong one) was larger than the number of valid ones. Possible reasons behind the high number of aborted trials are discussed in the next section. In general terms, combined audiovisual feedback led to a performance increase of 75% in subjects from both populations, but the magnitude of improvement in most subjects was very small.

Table 4.

Result from naïve subjects

Subject	Population	Best score visual feedback	Average visual feedback	Best score combined audiovisual feedback	Average combined audiovisual feedback
4	Sa	49%	48%	51%	48%
5	Sa	51%	49.5%	52%	50%
6	Sa	53%	46.6%	57%	52.1%
7	Sa	50%	38.8%	87%	74.1%
8	Sb	67%	63.1%	72%	66.2%
9	Sb	58%	54.4%	62%	53.1%
10	Sb	57%	51.4%	60%	52.7%
11	Sb	47%	43%	52%	46.8%

Note: Bold type indicates results of best subjects.

Despite the inconclusive results obtained in this preliminary experiment, it is worth underlining the results for two subjects, one per population, ie, subject 7 (belonging to Sa) and subject 8 (belonging to Sb). Both had performance increases with the combined audiovisual feedback. For example, subject 7 never achieved a result better than a random guess with solely visual feedback but received an 87% score when the audio feedback was added.

Quantitative analysis of the questionnaire (Table 5) shows that subjects enjoyed the experiment and would happily return for more recordings (average answer to question 9 scores 9.28). While the audio feedback was not rated as being as helpful as expected (average answer to question 5 scores 5.43), subjects expressed a greater feeling of frustration during audio feedback alone. Moreover, given that it is possible to infer from the low score of the average answer to question 4, naïve subjects generally did not feel in control of the cursor.

Table 5.

Average grades per question (naïve subjects)

Question	Average grade (naïve subjects)
1. Did you get tired because of the experiment?	5.86
2. Did you enjoy the experiment?	7.43
3. Did you find the experiment comfortable?	7.00
4. Did you feel that you could control voluntarily the movement of the ball?	4.86
5. Did you find the sound helpful or disturbing?	5.43
6. Did you feel frustration during the experiment?	6.14
7. Did you feel frustration during the experiment with only the video feedback?	6.43
8. Did you feel frustration during the experiment with the audio feedback?	4.43
9. How much likely you will agree to be test subject for this kind of experiment again?	9.28

Discussion and conclusion

In conclusion, combined use of auditory and visual feedback can increase performance in some naïve subjects and it can ease the sense of frustration due to the feeling of not being in control of the visual cue. The most important outcome is that our experimental set-up was welcomed and all the naïve subjects enjoyed the experiment and agreed to be subjects for future BCI experiments.

The results in Table 4 show that our naïve subjects had great difficulty in performing better than a random guess. This result is not likely due to the general instrumentation, given the quality of the data recorded (eg, the data depicted in Figure 1). Further analysis of the BCI session log showed that, in most cases, the number of trials aborted often represented the majority of trials per session. We performed a cross-analysis of the answers to questions 4 and 6 with the related BCI logs for the subjects when the percentage of aborted trial was greater than 50%. These subjects (patients 4 and 11) reported poorer results and a feeling of frustration. Other reasons behind the poor performances of our naïve subjects may be: the minimal EEG montage of only three electrodes, and given that it is not always true that the sensory-motor cortex area lies exactly under the C3 and C4 positions, we may not have recorded a high enough μ rhythm signal-to-noise ratio; the classifier had little data to learn from; and, as mentioned earlier, the classifier was learning only during the single physical trial.

The two abovementioned issues certainly contributed to the high number of aborted trials; the classifier was probably functioning correctly, but the calculated adapted weight was too small and the cue speed was too slow for the time allowed. Moreover, probable imprecise identification of the sensory-motor cortex made the classification even more difficult (poor μ rhythm signal-to-noise).

In future work, we plan to improve the performances of naïve subjects without increasing the burden related to subject preparation. We are designing a distributed headset-style EEG electrode capable of covering the area around the identified C3 and C4 areas with seven electrodes each, with one electrode in the center of the identified position and six electrodes equally distributed at a 1 cm radius from it. With this montage, we hope to increase the μ rhythm signal-to-noise available for classification. Furthermore, we will extend the training (or familiarization task) to allow for improved classification.