Heading for new shores! Overcoming pitfalls in BCI design

Abstract

Research in brain-computer interfaces has achieved impressive progress towards implementing assistive technologies for restoration or substitution of lost motor capabilities, as well as supporting technologies for able-bodied subjects. Notwithstanding this progress, effective translation of these interfaces from proof-of concept prototypes into reliable applications remains elusive. As a matter of fact, most of the current BCI systems cannot be used independently for long periods of time by their intended end-users. Multiple factors that impair achieving this goal have already been identified. However, it is not clear how do they affect the overall BCI performance or how they should be tackled. This is worsened by the publication bias where only positive results are disseminated, preventing the research community from learning from its errors. This paper is the result of a workshop held at the 6th International BCI meeting in Asilomar. We summarize here the discussion on concrete research avenues and guidelines that may help overcoming common pitfalls and make BCIs become a useful alternative communication device.

Introduction

Brain-Computer Interfaces (BCIs) have proven promising for a wide range of applications ranging from communication and control for motor impaired users, to gaming targeted at the general public, real-time mental state monitoring and stroke rehabilitation, to name a few [1, 2]. Despite this promising potential, BCIs are scarcely used outside laboratories for practical applications. Several publications and reviews have pointed out some of the roadblocks faced on the translation of BCIs -using either invasive and non-invasive techniques-towards practical applications [3, 4]. These include safety and biocompatibility of invasive approaches, as well as wearability and ergonomics of non-invasive recording techniques. In the later case, the main reason preventing electroencephalography (EEG)-based BCIs from being widely used is arguably their poor usability, which is notably due to their low robustness and reliability, as well as their often long calibration and training times. Although, BCIs based on P300, Steady-State Visual Evoked Potentials (SSVEP) generally require shorter training periods [9, 10].

As their name suggests, BCIs require communication between two components: the user’s brain and the computer. In particular, users have to volitionally modulate his/her brain activity to operate a BCI. The machine has to decode these patterns by using signal processing and machine learning. So far, most research efforts have addressed the reliability issue of BCIs by focusing on command decoding only [5,6,7,8]. While this has contributed to increased performances, improvements have been relatively modest, with correct mental command decoding rates being still relatively low and a reportedly significant amount of study participants unable to achieve satisfactory BCI control [1, 9, 10, 11]. Thus, the reliability issue of BCI is unlikely to be solved by solely focusing on command decoding [12, 13].

We argue that limiting factors on the successful translation of BCI systems stem from inadequate practices commonly followed by the research community. This paper summarizes the discussions on this topic held at the workshop “What’s wrong with us? Roadblocks and pitfalls in designing BCI applications” held on May 30th 2016 at the 6th International Brain-Computer Interface Meeting in Asilomar, CA, USA. The workshop was attended by 35 people of different backgrounds who identified methodological aspects worth of improving. Although we discussed here mainly the cases of non-invasive approaches, most of the topics addressed are relevant to the BCI research field as a whole. The paper is organized around four main topics, each of which represents a major component of any closed-loop BCI system. (i) End users. So far, little focus was on put on the inclusion of user-requirement aspects when designing BCI solutions. The need of end-user (group) specific tailoring of the system, of course, requires additional resources, which limits the possibility of successful transfer into marketable applications. However, design and evaluation of BCI systems should go beyond the “plain” decoding of neural signals and should take into account all components of the brain-machine interactions. Including, but not limited to feedback, human factors, learning strategies, etc. [12, 13, 14]. (ii) Feedback and user training. BCI researchers seem to neglect the fact that these systems consist of a closed loop between the human and the machine. In spite of this, the human is typically considered as a static, compliant entity that always perform what the experimenter asks for in a precise and consistent way. In reality, subject’s behavior is determined by his/her understanding of the task, abilities, motivation, etc. Therefore, both feedback and training instructions should be designed such as to maximize the interplay among them and help the user to learn a proper strategy to achieve good BCI control skills. (iii) Signal processing and decoding. Real end user application environments are much noisier, dynamic and provide multi-sensory impressions to the user, leading to drastic changes in EEG signals as compared to well-controlled lab environments or during BCI calibration situations. As result, lab-based BCIs potentially fail in real-life contexts. (iv) Performance metrics and reporting. There is a lack of clear metrics to assess the effective performance of a BCI system. It is generally acknowledged that these metrics should comprise both machine and human factors altogether (e.g. decoding accuracy and usability), but it is not clear how to weigh them. This in turn means that it is also not clear on how to explicitly report BCI results. This prevents the field from having a scientific discussion based on proper reporting and interpretation of results.

These topics and related pitfalls and challenges are further elaborated below. The aim of this paper is to start a discussion on strategies to improve current research and evaluation practices in BCI research. The vision is that open discussion on pitfalls and opportunities eventually leads to the development of systems that end-users can operate independently.

End-user-related issues

Several roadblocks within the aim of transferring BCI technology to end-user’s homes were addressed and discussed in our workshop. We suggest improvements in the areas of end-user selection, BCI system selection and discuss the potential of a common BCI end-user database. Furthermore, we point out how to reach the circumstances under which the user can be successfully included during the BCI application design process and thereby contribute to the integration of human factors in the usability of a BCI application (e.g. by following a user-centered design, UCD).

Identifying a potential BCI end-user

The choice of a potential candidate for using a BCI-based device is not trivial since not every end-user per se is suited. Kübler and colleagues stated several internal and external preconditions for successful BCI use [15]: (1) A potential candidate should be interested in and (2) in need of assistive technology, (3) be able to give informed consent, (4) be skilled concerning the cognitive abilities required to control the BCI system and (5) be supported by family members or caregivers. Already proper judgement of an end-user’s interest might be challenging (both for his/her relatives/caregivers as well as for outsiders like us as researchers). We might be tempted to believe that the enthusiasm we experience after implementing a working BCI system should also be felt by the end-user. But even though generally high satisfaction with the currently available BCI systems was reported, a clear demand for BCI improvements was also articulated by end-users [16] and caregivers [17]. Some of these aspects concern technological and practical aspects such as usability and ergonomics of the setup. For instance, the use of electrode cables and electrode gel as well as the general design of the BCI cap has been criticized. Furthermore, end-users reported not being able to use a BCI system on a regular basis as they would not be supported by family members or caregivers [16]. Several reports of end-user outcomes for BCI [18-22] show that successful translation of BCI systems not only depends on technology-related matters but also on personal, transactional, and societal challenges. Some of them are imposed by the healthcare system leading to a lack of external support for potential BCI end-users. Including healthcare providers in the BCI application design process might raise awareness of end-users needs. Additionally, their perspective might be valuable concerning the identification of potential BCI end-user groups.

On the other hand, there are single reports of successful BCI system transitions to end-users’ homes [23-25]. End-users in these studies not only fulfilled internal prerequisites for successful BCI use, but were also supported by caregivers or family members and thereby share beneficial external preconditions. Additionally, they had something else in common: Incentives to use the BCI were tremendously high as end-users were enabled by the BCI system to do something another assistive technology would not have allowed them. While the end-user reported by Sellers and colleagues [24], ran his research lab using the BCI system for up to 8 hours a day, the end-user reported by Holz and colleagues used the system for artistic self-expression which gave her back a lost communication channel. Therefore, it is our challenge as researchers, to create BCI applications that really increase incentives for potential BCI end-users.

Identifying a potential BCI system

As the choice of a BCI system that fits the end-user might be time consuming and frustrating for the end-user, several approaches feasible at bedside were suggested within the BCI community to facilitate this process. These approaches can be sub-divided into physiological and behavioral approaches.

Physiological approaches to predict BCI performance with one specific BCI system

The possibility of finding markers to predict future performance may help to rapidly screen suitable users to a given BCI system. A neurophysiological predictor within the sensorimotor rhythm (SMR) based on recordings of two-minutes of resting state data was found to be significantly (r=.53) correlated with SMR BCI performance [26]. Also gamma activity during a one-minute resting state period showed significantly correlations with later motor imagery performance [27]. The resting state EEG network (two minutes assessment) can be a predictor for successful use of a SSVEP based BCI [28]. However, these studies have only involved able-bodied subjects. Further research with subjects with motor impairments is required to evaluate transferability of results.

One challenge to be faced is the assessment of resting state activation with a reduced number of electrodes and in clinical samples to realize bedside data assessment. To achieve this goal, advice from neuroscientists might be helpful as this interdisciplinary approach could facilitate choosing a reliable screening method. Recent works both with able-bodied subjects and one end-user with locked-in syndrome have proposed methods to assess performance variation within a session and during runtime. They may help to better assess the suitability of a given BCI approach, and to provide user-specific support during the training periods [29,30].

Concerning the P300 BCI, resting state heart rate variability (HRV) which is an indicator of self-regulatory capacity was found to explain 26% of the P300 BCI performance variance [31]. Other screening paradigms basically rely on using standard neuropsychological paradigms that assess the same EEG correlations the BCI system relies upon. One easily applicable paradigm is the oddball paradigm [32]. P300 amplitudes in the oddball paradigm measured at frontal locations were found to correlate up to r=.72 with later P300 BCI performance [33]. P300 amplitude responses to oddball paradigms were also successfully utilized to decide which stimulation modality -among auditory, visual or somatosensory- would most likely lead to best classifiable results [34].

Behavioral approaches to predict BCI performance with one specific BCI system

The assessment of behavioral predictors of BCI performance is usually based on either questionnaires or button-press response tasks. Concerning psychological predictors of BCI performance, the willingness and ability to allocate attention towards a task was found to explain 19% of variance when using an SMR BCI [35]. Also the ability to kinesthetically imagine motor movements was significantly correlated with later SMR BCI performance [36]. Finally, the user’s spatial abilities, as measured using mental rotation tests, were also found to correlate to mental-imagery BCI performance, both for motor and non-motor imageries [37]. In P300 BCI use, one specific component of attention, the ability for temporal filtering was found to predict P300 BCI performance in people diagnosed with Amyotrophic Lateral Sclerosis (ALS) [38]. Motivation as measured with a visual analogue scale was reported to explain up to 19% of variance in P300 based BCI applications [39, 40].

In conclusion, the identification of potential BCI performance predictors has not been exhaustively investigated so far [7]. A standardized screening procedure including easily applicable instruments and methods should be agreed upon. One such suggestion was already proposed for the identification of potential BCI users for a rapid serial visual presentation keyboard [41]. In case researchers could collaboratively and interdisciplinary contribute to the identification of reliable predictor variables, end-users might be equipped fast with the BCI system best suited for their individual needs.

Advantages of an end-user database

Another step towards identifying methods for fast and reliable BCI system selection might be sharing a common database [42, 43] in which selected information about end-users and the BCI systems that fit them are merged. Possible information of interest might be diagnoses and assessed predictor variables (see above). An end-user database might also be based on level of function rather than diagnosis as a primary descriptor. Similar to the International Classification of Functioning, Disability and Health (ICF)¹ that have been developed for specific patient groups [44, 45, 46], an end-user database could describe functional limitations, and include a full range of diagnoses that lead to needs for potential BCI use.

Furthermore, experience reports by caregivers, BCI experts and people with additional backgrounds who work with an end-user might add crucial information which allow to deduce the choice of the best BCI system for a user by evaluating previous success stories. Additionally, potential dead ends once detected could help others to prevent wasting time and effort on a less efficient BCI based approach.

Of course, ethical issues must be taken into account as highly detailed reports about end-users including their diagnoses and their BCI experiences might threaten end-users anonymity which must be prevented. However, after having gathered a sufficiently big and meaningful database, for which criteria must be agreed upon, data mining approaches can be used to support the decision making process. Not only physiological data (brain activation) but also behavioral data (questionnaire data) could be integrated and utilized to support the best possible fit between end-users and specific BCI systems. To reach this goal, collaborations between BCI researchers, clinicians, caregivers and family members all over the world is crucial, as well as common efforts to decide which information should be shared and how. This would require standard protocols and procedures to collect physiological or behavioral data, such that they can be compared between different research and clinical groups, different EEG recording devices, or BCI processing pipelines for instance. Alternatively, this could also be the opportunity to design data analysis tools dedicated to compare BCI performances or EEG signals collected with different devices or BCI processing pipelines, in order to enable such comparisons.

The user-centered design (UCD) and why it is rarely implemented

In case a BCI user and the appropriate BCI system were selected, the system still needs to be adjusted to the end-users’ needs as otherwise, long-term use remains elusive. One procedure that was successfully implemented [23] is the user-centered design, UCD [14]. The end-user should be involved as early as possible by being asked to express needs and requirements the BCI based assistive technology (AT) is supposed to fulfill. Methods to elicit involvement and feedback from potential BCI users have been proposed [47]. End-users should be encouraged to actively influence the process of their BCI system being adjusted. First trials of using the technology are to be evaluated and the technology or features of it should be accustomed to the user’s requirements. This process is to be iterated such that by trying new solutions and re-adjusting the system, the end-user receives an AT device that truly supports him or her in daily life. Another key factor suggested by UCD is the multidisciplinary approach. In case an end-user is in a medical condition requiring professional care, close interaction between family, caregivers and medical staff tremendously facilitates the process of transferring an AT to the end-user’s home as not only the needs and requirements of the end-user can be taken into account, but also the device should be adjusted to the needs and requirements of the ones who support the end-user. Furthermore, a multidisciplinary approach would allow for improvement of end-user training. For example, by taking into account experts opinions on how to create learning paradigms that promote success but also facilitate BCI use for the end-user (See next section).

To mention just one example of UCD-based development: The aim of the FP7 ABC project was to develop an interface for individuals with cerebral palsy (CP) that improves independent interaction, enhances non-verbal communication and allows expressing and managing emotions [48]. As described above, as first step the potentially most useful BCI system was identified [49]. Then, based on the end users preferences and demands, a BCI-operated communication board was implemented [50]. To provide a better training experience (see next section) and educate end user on how to use BCI technology, a game-based training environment was implemented [51]. BCI-based communication only works when end users understand the working principles and relevance of their own active involvement. Not providing clear and detailed information to end users will become one major pitfall. Finally, cooperative/competitive elements were added to the game-based user training [52]. Users can now train together with friends and not alone in front of the computer. Each developmental step was discussed with end users and caregivers. Development, however, was not linear. Advances in the end users understanding of BCI technology, let to the redesign of several aspects of the system over and over again.

A crucial point to be addressed when applying UCD is the resources it requires. So far, only rare cases of long-term independent BCI use were reported [23-25], which might be due to limited resources available. Not only is the hardware expensive, also experts need time to adjust the system properly. And after UCD based BCI system transfer to an end-users home, we can report results of this one single case. Thus, awareness for the value of case studies must be increased. Even though for fundamental research the claim of generalizability of results is a valid one, this claim cannot be transferred to a BCI context, in which per definition, every end-user might present different needs and requirements [53]. As a consequence, researchers are faced with the conundrum of how to provide the possibility of permanently refining of the system depending on the (changing) conditions of the user given the available resources. The truth is that currently few research groups can afford such process due to lack of funding programs in this type of long-term, high-risk research, as well as little interest from publishing venues as outcomes appear to be incremental. Some alternatives to improve in this respect is to develop better, formal strategies to support the interface adaptation to the user. These strategies can be based on the typification of the experience from previous studies, along with the characterization of the users for which they worked. This process can identify which parts of the process can or not rely on automatic processes (using strategies for co-adaptation, online performance evaluation, shared control, etc.) and devise guidelines for evaluating potential design alternatives. Finally, the engagement of other actors, besides the research community, specifically focused on translational aspects of the technology is a no-brainer for creating the conditions of successful design of BCI solutions.

Feedback and training

BCI control is known to be a skill that must be learned and mastered by the user [54, 55]. Indeed, user’s BCI performances (i.e., how accurately his/her mental commands are decoded) become better with practice and BCI training leads to a reorganization of brain networks as with any motor or cognitive training [54, 55]. Therefore, to ensure a reliable BCI, users must learn to successfully encode mental commands in their brain signals, with high signal-to-noise ratio. In other words they should be trained to produce neural activity patterns that are as stable, clear and as distinct as possible. With poor BCI command encoding skills, even the best signal processing algorithms will not be able to decode commands correctly. Unfortunately, how to train users to encode these commands has been rather scarcely studied so far. As a consequence, the best way to train users to successfully encode BCI commands is still unknown [1,7, 54]. Worse, as we argue in this paper, current user training approach in BCI are actually even inappropriate, and most likely one of the major causes of poor BCI performance, and high BCI deficiency rates. In the following, we present recent evidences explaining why they are inappropriate, and present open research questions that would address this fundamental limitation.

Current BCI user training approaches are inappropriate

One common approach used in non-invasive BCI training consists in asking users to perform mental imagery tasks i.e., the kinesthetic imagination of body limb movements [56]. These tasks are mapped used to encode commands (e.g., imagining left or right hand movements to move a cursor in the corresponding direction). Meanwhile, users are being provided with some visual feedback about the mental command decoded by the BCI [54]. However, currently used standard BCI training tasks and feedbacks are extremely simple, based on heuristics, and provide very little information. Typical BCI feedback indeed often consists of a bar displayed on screen whose length and direction vary according to the EEG signal processing output and the decoded command. Is that really the best way we can train our users to gain BCI control? To answer that question, we have studied the literature from the fields of human learning, educational science and instructional design [11, 57]. These fields have indeed studied across multiple disciplines, e.g., language learning, motor learning or mathematical learning, what are the principles and guidelines that can ensure efficient and effective training approaches. We have then compared such principles and guidelines to the training approaches currently used in BCI. In short, we have shown that standard BCI user training approaches do not satisfy general human learning and education principles ensuring successful learning [11, 58, 59]. Notably, typical BCI feedback is corrective only, i.e., it only indicates users whether they performed the mental tasks correctly. Oppositely, human learning principles recommend providing explanatory feedback by indicating what was right or wrong about the task performed [58]. BCI feedback is also usually unimodal, based only on the visual modality, whereas exploiting multimodality is also known to favor learning [60]. Moreover, training tasks should be varied and adapted to the user’s skills, traits (personality or cognitive profile) and states [58]. They should also include self-paced training tasks, to let the user explore the skills he/she has learnt. BCI training tasks, in contrast, are fixed over time and users, synchronous and repeated identically during training. Finally, and intuitively, training environments should be motivating and engaging, whereas standard BCI training environments may not favor engagement, and appear to be boring after prolonged used as often informally reported by users. As mentioned in the previous section, feedback and training tasks design may as well take into account the user’s preferences and psychological profile to maximize his/her motivation. Note that an engaging environment does not necessarily have to be a visually appealing but complex and charged visual environment (like most modern video games), as it may overload the user. Unfortunately, there are many other training principles and guidelines not satisfied by classical BCI training [11, 57].

We have also shown that in practice, standard feedback and training tasks used for BCI are also suboptimal to teach even simple motor tasks [61]. In particular, we studied how people could learn to do two simple motor tasks using the same training tasks and feedback as those given to motor imagery (MI) BCI users. More precisely, we asked subjects to learn to draw on a graphic tablet a triangle and a circle (the correct size, angles and speed of drawing of these two shapes being unknown to the subject) that can be recognized by the system, using a synchronous training protocol and an extending bar as feedback, like for standard MI-based BCI training. Our results show that most subjects (out of N=53 subjects) improved with this feedback and practice (i.e., the shapes they draw are increasingly more accurately recognized by the system), but that 17% of them completely fail to learn how to draw the correct shapes, despite the simplicity of the motor tasks [62]. This suggests that the currently used training protocols may be suboptimal to teach even a simple skill. As such, and although not a definitive proof, this could suggest such protocols are in part responsible for the poor BCI control achieved by some users, and thus should be improved.

Therefore, both theory and practice suggest that current BCI training approaches are likely to be suboptimal. This a bad news for current BCI systems and users, but a good news for BCI research and BCI future: it means there is a lot of room for improvement, and many exciting and promising research directions to explore, as described in following section.

Open research questions in BCI user training

As shown in the previous section, evidence converges towards indicating that current BCI user training approaches are inappropriate. A promising direction to change them in a relevant way would be to make them satisfy principles and guidelines from human learning theories and educational science.

At the level of the training tasks we propose our BCI users, educational science recommends providing varied, adaptive and adapted training tasks. This raises a number of currently unanswered questions. How to design varied and relevant training tasks (e.g., by taking into account the end user objective and needs, as mentioned in UCD above)? What should these tasks train? In order to design adapted training tasks, we should also find out about how the user’s profile impact BCI learning and performances. Some recent research results go into that direction and are worth being further studied (35, 62, 63, 64]. To design adaptive training tasks also requires adjusting the training tasks sequence to each user over time. How to do so to ensure an efficient and effective learning? The BCI community could learn on this topic from the field of Intelligent Tutoring Systems (ITS), which are tools specifically designed for digital education, to find an optimal sequence of training exercises for each user, depending of this user’s skills, traits and states [62].

At the level of the feedback provided, educational science recommends to exploit multimodal feedback and to provide feedback that is explanatory rather than purely corrective, which it is so far for BCI. This raises the question of whether we can exploit other feedback modalities (e.g., auditory, tactile) for training, and how? Recent results show that complementary tactile feedback can enhance motor imagery BCI performances for instance [65, 66], which confirms this is a promising direction to explore. Designing an explanatory feedback for BCI is currently very challenging given the lack of fundamental knowledge on motor imagery and on BCI feedback [67]. For instance, why mental commands are sometimes erroneously recognized? Which feedback content providing to the user? Which feedback presentation should be used to represent this content? These are crucial research questions that the BCI community will have to answer to design appropriate feedbacks for BCI, and thus to efficiently and effectively train our BCI users.

Finally, at the level of the training environment, improving BCI training requires to design motivating and engaging training environment. How to do so? How to keep BCI users being motivated and engaged in the training? Some recent works showed the positive impact of video games and virtual reality on BCI training and performance [51, 52, 68]. There is now a need to understand why it is so, and to formalize these approaches to ensure the next generation of BCI training approaches will be motivating and engaging at all times, for all types of users.

From a non-technical and non-scientific point of view, all this research also depends on how to report such studies on BCI user training. Indeed, many different training tasks, feedbacks and environments could be tested and explored. Not all of them will lead to improve BCI training efficiency or improved BCI performances. However, it is essential to know what works and what does not work to deepen our knowledge on BCI training. It would also be really inefficient, especially considering how costly (in time and money) BCI experiments on user training can be, if several research groups were to try the same experiments without knowing that other groups have already tried before but that it failed. This all points to the necessity to publish negative results in BCI research, to ensure an efficient research and to ensure access to all available relevant knowledge.

Signal processing and decoding

As mentioned before, research in signal processing and decoding algorithms have played a significant role in the field of BCI in the past two decades. First attempts to gain BCI control were based on operant conditioning [69]. The BCI decoding algorithm was pre-defined and users had to learn by trial-and-error to modulate brain rhythms (for example, by controlling a cursor to hit targets). By introducing machine learning (ML) and pattern recognition (PR) methods the burden of learning was shifted towards the machine. The parameters of the BCI decoding algorithm were adapted to the user’s individual EEG signatures [70]. PR refers to methods that translate characteristic EEG features (patterns) into messages. ML refers to methods that optimize feature and PR parameters. According to the guiding principle let the machine learn, followed by some groups, a number of ML and PR methods have been applied [6, 71]. Some improvements in gaining BCI control have been made. However, the goal of yielding good control for everyone was not achieved. Today, a co-adaptive approach is most common. First the PR methods are trained to detect user specific patterns. Then feedback training, based on the trained PR model output, is used to reinforce the EEG pattern generation. By repeated application of the two steps the user and the machine mutually co-adapt. More recent approaches include online co-adaptation and transfer learning [72- 77]. In the former case BCI model parameters are adapted and iteratively updated during feedback training. In the latter case models are transferred between users and/or between days.

The above discussion mostly applies to BCIs that are based on the detection of changes in spontaneous EEG. PR performance in BCIs that are based on the detection of evoked potentials (EP) is robust. Since properties of EP signatures are time-locked to visual, auditory or sensorimotor stimuli, increased detection performance can be achieved. In the vast majority of healthy users detection rates higher than 90% are achieved in the case of SSVEP in less than 30 minutes of time [9]. The performance, however, drops considerably when moving out of the lab and into end users’ homes [49]. Some studies have shown EP-based BCIs that yield good performance with subject-independent classifiers, i.e., without calibration at all for a new user [78, 79]. However, so far these approaches have mainly been tested in offline studies and with healthy users only. It is therefore still unknown whether this results generalize to end-users. Moreover, performances are still better when exploiting subject-specific data and user training, even for P300-based BCI [79, 80].

Significant improvement in performance was achieved by using co-adaptation. However, fundamental issues of BCI learning were not tackled. ML/PR were mostly treated separately without considering all components in the BCI feedback loop [12, 13]. As a matter of fact, typically linear time invariant models are used to translate non-stationary and inherently variable EEG signals. This means that BCI models may perform well for limited periods of time - BCI experiments typically will not take more than one hour - but not for extended time or 24/7 use.

Statistical PR methods such as linear discriminant analysis (LDA) or support vector machines (SVMs) are most commonly used [6]. Feature extraction and decoding parameters are optimized based on collected EEG data. Since data collection is time consuming, optimization is typically based on a limited amount of data. It is therefore essential to estimate the generalization of selected models. Methods such as regularization, shrinkage or cross validation may prevent over fitting to the data [81-84]. Overfitting means that PR models memorize the training data. Due to EEG non-stationarity, this may lead to suboptimal performance on unseen data. Another important prerequisite is to select PR models based on the properties of the selected EEG features. Hence, the probability density function of features and of the PR framework should match for optimal performance. For example, using power spectral density (PSD) estimators with LDA will likely fail, since LDA assumes Normally distributed data, which PSDs are usually not. Computing the logarithm of the PSD features will make them more Normal-like, thus increasing the LDA classification performances. This leads to the most crucial aspect: features. Optimal performance requires the proper choice of features. For instance, ERP and MRCP-based BCI typically reach better performances when using time-domain features, whereas SSVEP and mental imagery-based BCIs should use PSD or similar features [85, 86, 87]. Preprocessing and decoding algorithms should also be selected depending on the amount of available training data and EEG channels [85, 87]. Typically, the more training data, the more complex the algorithms can be. For multi-channel mental imagery-based BCIs, a very common pre-processing approach is the common spatial pattern (CSP) and its evolutions [85,88,89]. Despite recent improvements in feature extraction and preprocessing algorithms (see [71] for a review), there are still a large number of users that are not able to control a BCI (BCI inefficiency). At least not in the short term. And there is not enough experience with long-term training! Which features are most informative and lead to enhanced performance? There is no answer to this question at this time. More basic research and a better understanding of brain functioning is required to be able to answer this question. Some attempts to enhance the interpretability of brain oscillations are already ongoing (e.g. [90,91]).

Obviously, BCI end users that are able to reliably generate EEG patterns achieve higher PR performance. Typical mental tasks used to encode messages for imagery-BCIs are hand and feet motor imagery combinations. It turns out that a user-specific combination of “brain-teaser”, i.e., tasks that require problem specific mental work (e.g. mental subtraction or word association), and “dynamic imagery” tasks (e.g. motor imagery or spatial navigation) significantly enhance BCI performance in healthy [92-93] as well as in users with disability [94-95]. In the latter case binary classification accuracy was up to 15% higher when compared to a classical motor imagery task combination [94]. These results again emphasize the need to consider the different components in the BCI feedback loop and their interplay.

Another important aspect that impacts on PR and ML are contamination artifacts. Artifacts are interference signals that share some of the characteristic features of EEG and can produce misleading EEG signals or destroy them altogether. When developing BCIs, one has to take care that cortical signals are used for communication and control and not artifacts [96-98]. Muscle artifacts, for example, are highly correlated to the user’s behavior and have much higher amplitudes. This means that they are also easier to detect and may impact on PR performance. If the aim is to establish communication for an end user and muscular activity is voluntarily generated then this approach may be suitable. However, the nature of the signal used for control should be clearly identified by the developers, i.e. this communication device does not qualify as BCI if no brain activity is used; alternatively, it may correspond to a hybrid BCI if both muscle and EEG signals contribute for control [99-101]. Artifact or not an artifact? Labeling and characterization of artifacts is one major issue. Often, the performance of artifact removal algorithms is derived from comparisons to human ratings (e.g. [97,102]). Scorings between humans, however, may vary to a large extent. Finding a common definition and elaborating scoring guidelines may help creating an artifact database that serves as benchmarking for algorithms validation. One idea is to setup a web-based artifact-scoring tool where experts from all over the world can score EEG signals and contribute to the building of such a comprehensive database. However, as said before, not only EEG data and scoring information should be shared, but also source code and protocols. Only that would allow an open and transparent validation of new methods and lead to intense scientific discussion.

Performance metrics and reporting

Another pitfall that hinders advancement of BCI technologies lies on the methods used to assess the quality of the developed systems. To a large extent, the research community has mainly focused on evaluating the performance of the decoding engine and applied mainly PR and ML performance metrics. A widely used metric is the accuracy of the classifier (true positive rate, TPR)². Other metrics also take into account the effect of misclassifications (false positive rate, FPR) and estimating the specificity (1-FPR) of the decoder as well as the true and false negative classification (TNR and FNR, respectively) [103]. Other metrics, inspired from information theory have also been proposed by analyzing the BCI as a communication channel between the brain and the controlled device [104]. However, these metrics have their limitations as it has been already pointed out in previous works [105-108].

Besides these limitations, they are often poorly used in an important proportion of BCI related literature. One important characteristic of most BCI studies is that they rely on a small quantity of data, both in terms of the number of subjects that participate in the experiments as well as the number of trials recorded for each of them. This clearly limits the conclusions that can be drawn from these studies. Nevertheless, BCI published works rarely address nor discuss the population effect size and often overstate the impact of their findings by implying they would generalize to greater populations. Since the interpretation of the numerical values depends on the amount of data available, it is important to consider the chance level performance [109]. The less data available, the higher the chance level or the broader the confidence interval of the performance estimation. This means for a limited amount of data the chance level performance may be very high. This in turn leads to an overly optimistic interpretation of the results. Moreover, special care should be taken when choosing the statistical tests used for comparing different experimental conditions [83,110,111], and the decoding performance evaluation should specifically take into account the characteristics of the dataset used (in terms of number of trials and class distribution).

In addition, these metrics typically evaluate the output of the classification function against a set of labelled samples acquired during a calibration period (i.e. supervised learning paradigms). Therefore, these metrics rely on the assumption that the brain activity that is fed to the classifier during operation follows the same distribution than the one observed during calibration. This assumption is widely recognized as incorrect and often brushed off by arguing that EEG signals are non-stationary, we claim that there is a more fundamental reason for changes in the neural activity during BCI operation. As a matter of fact, the BCI system is inherently a closed loop system and, in consequence the feedback it provides is one of the causes of signal changes in the user’s brain activity. Therefore, even though offline evaluation of the classifier performance may be useful for preliminary evaluation of different decoding methods it is not enough to assess the performance of a BCI and performance evaluation during online usage is necessary.

The aforementioned aspects concern exclusively the evaluation of the BCI decoding engine. Thus, they only reflect one element in the BCI loop and do not provide information about how suitable the system is for its user. Several studies have shown that human factors and user characteristics influence performance [39, 112, 113,114]. In consequence, proper evaluation of the BCI system cannot be limited to decoding metrics, and should also include efficiency/effectiveness metrics in the human-computer interaction sense [115,116], as well as explicit assessment of the human factors (cognitive workload, sense of agency, among others), e.g., through the use of questionnaires [117,118]. This implies the need for combining both quantitative objective performance measures with subjective, qualitative assessments based on self-reporting, and highlights the fact that BCI performance may not be comprehensively reflected by a single figure [14]. Nevertheless, The appropriate way to weigh in these different metrics is strongly application dependent and remain one outstanding challenge in the field.

Note, that introducing mechanisms that support users to control BCIs (for example word prediction or evidence accumulation for decision making) positively impact on the performance. It is essential to identify the impact of such mechanisms when computing chance performance levels [50], and properly evaluate the contribution of the BCI to the achievement of the task.

Last but not least, it is necessary to improve the way BCI research is reported in peer-reviewed publications. At the current state, other research groups cannot independently replicate most experiments. Mainly due to the lack of a complete description of the methods used for processing the signals (from signal acquisition and conditioning to feature selection and classification), as well as the specific instructions given to user. We exhort the community to be more careful in the design and reporting of their experiments. As mentioned above, experimental studies are typically performed with small populations of subjects that do not match the intended BCI users (e.g. graduate students), thus their results may not be applicable to a large population. Besides these issues, the BCI field also suffers from problems inherent to the scientific literature, as is the bias to publish only positive results that prevents the research community of learning from its errors [119]. As a countermeasure for these weaknesses we proposed the need of clear a guidelines for good reporting practices. Even though we can adopt guidelines from related fields (c.f., [111, 120-122]) some aspects inherent to BCI systems may require specific new guidelines to be drawn (e.g. procedures for decoder calibration, evaluation performance on small populations, among others). Such guidelines can be endorsed by the BCI society encouraging authors and reviewers to follow them. Importantly, a great amount of information about these systems can be obtained from qualitative data, good practices for collecting, reporting and analyzing it can be extremely helpful to the research community. Last but not least, encouragement for reporting negative BCI results, e.g., through dedicated peer reviewed journal special issues can be also beneficial. However, reporting of these results should come from rigorous and unbiased studies that yield results that are strong enough to refute a hypothesis and help to design follow up studies. Only in these cases they can be useful to advance the field towards more robust and reliable systems.

Conclusion

The design of BCI systems reliable and robust enough to allow independent use is a challenging task involving significant advances in diverse domains. Researchers in the field have gone a long way demonstrating the feasibility of extracting meaningful information from neural signals to control external devices. However, there are still several roadblocks to surmount for these systems to be successfully deployed to their intended end-users. We have summarized in this paper several methodological aspects that need to be taken into account in order to achieve this goal. A first aspect corresponds to the need of better ways integrate the user’s needs and preferences into the design of the BCI solution. Several actions have been suggested along this line (see Figure 1), which require intense and interdisciplinary collaborations in order to go beyond the existing success stories of single cases of independent BCI use, towards formal methods for effective translation from the research lab into real applications. Experience from other cases in neurotechnologies can serve as model or inspiration for the BCI case. One model to be used as a starting point on how the design process might be optimized is Scherer’s Matching Persons and Technology Model [123,124] that examines the psychotechnological interplay of device and end-user from the perspective of optimizing user outcomes.

Figure 1.

User related topics to avoid roadblocks in transferring BCI systems from the lab to the end-users home.

In addition, at the user training level, there are also a number of pitfalls we have to overcome. Indeed, currently used training protocols are most likely highly suboptimal, both from a theoretical and practical point of view. It is therefore necessary to conduct fundamental research to understand what BCI skills users are learning, how they are learning, and what makes them fail or succeed at BCI control. It is also necessary to try to go towards adapted and adaptive training tasks, as well as towards explanatory feedback, to ensure successful BCI skills acquisition.

Digital signal processing, pattern recognition and machine learning are essential components of modern BCIs that allow for brain-computer co-adaptation. Dealing with artifacts and brain signal non-stationarity, i.e., finding appropriate features, are issues that have the largest potential to increase performance and need closer attention. Sophisticated machine learning methods s may help tackling these issues and contribute to the further advancement of BCI. However, the highly multidisciplinary nature of BCI research makes it very difficult for researchers in the field to know enough of each discipline involved to avoid all methodological flaws associated to each of them (e.g., statistical flaws or protocol design with confounding factors). This has lead to a number of BCI publications with biases, confounding factors, and contradictory results that may slow down progress in the field. We identify a need within the field to be more cautious on the application and evaluation of these methods. This can be improved through published guidelines and more interdisciplinary participation in BCI related events and research projects.

Last but not least, it is crucial for the BCI community to fully embrace its interdisciplinary nature and effectively engage all possible stakeholders on its scientific events, and development projects. It should be widely acknowledged that successful translation of these technologies go beyond scientific research and should involve interest groups (e.g., patient associations), insurance and public health representatives, and the private sector so as to identify clear user needs, as well as strategies to support research and development of BCI solutions in a sustainable way.

Biographies

Ricardo Chavarriaga, Ph.D. is a senior researcher at the Defitech Foundation Chair in Brain-Machine Interface, Center for Neuroprosthetics of the École Polytechnique Fédérale de Lausanne (EPFL), Switzerland. He holds a PhD in computational neuroscience from the EPFL (2005). His research focuses on robust brain-machine interfaces and multimodal human-machine interaction. In particular, the study of brain correlates of cognitive processes such as error recognition, learning and decision-making. As well as their use for interacting with complex neuroprosthetic devices.

Melanie Fried-Oken, Ph.D. is a clinical translational researcher in Augmentative and Alternative Communication (AAC) and Brain-Computer Interface (BCI). She provides expertise about assistive technology for persons with severe communication impairments who cannot use speech or writing for expression. She is a Professor of Neurology, Pediatrics, Biomedical Engineering and Otolaryngology at the Oregon Health & Science University (OHSU); Director of OHSU Assistive Technology Program, and clinical speech-language pathologist. She received her Ph.D. from Boston University in 1984 in Psycholinguistics; her M.A. from Northwestern University in Speech-Language Pathology, and B.A. from the University of Rochester. She has written over 40 peer-reviewed articles, numerous chapters and given over 200 national and international lectures on AAC for persons with various medical conditions. She has been PI on 16 federal grants and co-investigator on six federal grants to research communication technology, assessment and intervention for persons with developmental disabilities, neurodegenerative diseases, and the normally aging population.

Sonja Kleih, Ph.D. received her M.Sc. in Psychology and investigated the effects of motivation on BCI performance. She received her PhD in Neuroscience in 2013 from the University of Tübingen, Germany. Since then she worked as a Post-Doctoral researcher in Prof. Andrea Kübler’s BCI lab at the University of Würzburg, Germany. Her research interests are BCI based rehabilitation after stroke, user-centered design and psychological factors influencing BCI performance.

Fabien Lotte, Ph.D. obtained a M.Sc., a M.Eng. and a PhD in computer sciences, all from INSA Rennes, France, in 2005 (MSc. and MEng.) and 2008 (PhD) respectively. His PhD Thesis received both the PhD Thesis award 2009 from the French Association for Pattern Recognition and the PhD Thesis award 2009 accessit (2nd prize) from the French Association for Information Sciences and Technologies. In 2009 and 2010, he was a research fellow at I2R, Singapore, in the BCI Laboratory. Since January 2011, he is a tenured Research Scientist at Inria Bordeaux, France. His research interests include BCI, HCI and signal processing.

Reinhold Scherer, PhD. received his M.Sc. and Ph.D. degree in Computer Science from Graz University of Technology (TUG), Graz, Austria in 2001 and 2008, respectively. From 2008 to 2010 he was postdoctoral researcher and member of the Neural Systems and the Neurobotics Laboratories at the University of Washington, Seattle, USA. Currently, he is Associate Professor at the Institute of Neural Engineering, BCI Lab at TUG and member of the Institute for Neurological Rehabilitation and Research at the clinic Judendorf-Strassengel, Austria. His research interests include BCIs based on EEG/ECoG, statistical and adaptive signal processing, mobile brain and body imaging and robotics-mediated rehabilitation.