Workshops of the Sixth International Brain–Computer Interface Meeting: brain–computer interfaces past, present, and future

Abstract

The Sixth International Brain–Computer Interface (BCI) Meeting was held 30 May–3 June 2016 at the Asilomar Conference Grounds, Pacific Grove, California, USA. The conference included 28 workshops covering topics in BCI and brain–machine interface research. Topics included BCI for specific populations or applications, advancing BCI research through use of specific signals or technological advances, and translational and commercial issues to bring both implanted and non-invasive BCIs to market. BCI research is growing and expanding in the breadth of its applications, the depth of knowledge it can produce, and the practical benefit it can provide both for those with physical impairments and the general public. Here we provide summaries of each workshop, illustrating the breadth and depth of BCI research and highlighting important issues and calls for action to support future research and development.

Introduction

Brain–computer interfaces (BCI) (also referred to as brain–machine interfaces; BMI) are, by definition, an interface between the human brain and a technological application. Brain activity for interpretation by the BCI can be acquired with either invasive or non-invasive methods. The key point is that the signals that are interpreted come directly from the brain, bypassing sensorimotor output channels that may or may not have impaired function. This paper provides a concise glimpse of the breadth of BCI research and development topics covered by the workshops of the 6th International Brain–Computer Interface Meeting.

History and distinctives of the BCI Meeting Series

The individual meetings of the International Brain–Computer Interface Meeting Series have occurred approximately every three years, with a goal of bringing together BCI researchers from around the world. The first International BCI Meeting was held in 1999, with 50 scientists from 22 laboratories attending [1]. The growth of the BCI Meetings has paralleled the astonishing growth of BCI research itself, with ever larger meetings in 2002 [2], 2005 [3], 2010 [4], and 2013 [5–7]. The Sixth International BCI Meeting was held 30 May–3 June 2016 at the Asilomar Conference Grounds in Pacific Grove, California, USA. The 2016 BCI Meeting was attended by 400 participants from 26 countries, representing 188 laboratories and organizations. The 2016 Meeting was the first to be organized under the direction of the newly established BCI Society and offered a registration discount for BCI Society members. Approximately one-third of the BCI Meeting registrants were BCI Society members, with many more opting to join the Society after the BCI Meeting.

In the opening session, Dr Jon Wolpaw, the president of the BCI Society, spoke about the mandate from NIH that led to the creation of the First International BCI Meeting in 1999. The BCI Meeting was to be held at an isolated location to keep participants together, it was to have a large number of young people to grow the field, and it was to have a highly interactive format. These characteristics, along with the diverse background of attendees, have become distinctive of the BCI Meeting series. The BCI Meetings seek to bring together representatives of all the diverse fields required for successful BCI research, development, and translation into commercial products. The BCI Meeting is attended by engineers, physicians, computer scientists, federal funding representatives, clinical rehabilitation specialists, neuroscientists, psychologists, speech-language pathologists, BCI users, caregivers, entrepreneurs, and many others. Progress in BCI research and development, and especially the creation of useful, appropriate applications, requires interaction and, indeed, close collaboration between people from many of these backgrounds. While BCI sessions are becoming common at many conferences, the diversity of disciplines at the BCI Meetings is unique.

The 2016 BCI Meeting registrants identified themselves as 40% students, 12% postdocs, 12% early career, and 37% established researchers. There were also two BCI users, one of whom assisted in testing a BCI for communication, the other who tested a BCI for walking. The Asilomar Conference Grounds supports the interactive nature of the BCI Meetings by providing common meals, housing for all BCI attendees, and a beautiful environment for casual conversation and networking.

With a theme of ‘BCI: Past, Present, and Future,’ this Sixth BCI Meeting built on the knowledge of past BCI research and the accomplishments of the established researchers who clear their calendars to attend the BCI Meetings. The Meeting provides a detailed overview of the present state of BCI development, and the workshops and interactions fuel the future of BCI research, development, and translational efforts.

Organization of workshop summaries

The workshops of the BCI Meeting have evolved since the first Meeting, when each participant was part of a single multi-day workshop on one of six topics (Definitions, Components, Invasive Methods, Signal Analysis, Signal Translation, and Applications). Workshops now occupy three time slots so that each attendee can participate in workshops on multiple topics. Each workshop is still intended to be an interaction between attendees, but most now seek to provide in-depth knowledge on a specific topic instead of the comparisons or competition between diverse methods that was often the focus of the broader workshops of the earliest meetings.

At the first BCI Meeting, BCI applications were limited to communication/computer access, control of prosthetics, robotics or functional electrical stimulation, monitoring alertness, and controlling a flight simulator. While maintaining these applications, major BCI applications now include stroke rehabilitation, entertainment, assessment of disorders of consciousness, and research tools for study of neuroscience. Many BCIs are now intended not only as tools for people with physical impairments, but also as treatments for physical or cognitive impairments. Further, the ever-growing variety of BCI applications increasingly includes applications for those without disabilities.

All workshops were proposed by members of the BCI community. The Program Committee helped to merge the more common topics, at the same time promoting newer or under-discussed topics to produce a workshop list that covers a wide breadth of BCI research and development. This report contains summaries of these individual workshops, grouped by themes. Each summary lists the organizers and all additional presenters. The summaries provide an introduction to each topic, key points of the presentations and discussion, and resources for further study. Active participation by attendees in workshop discussion is one of the most valued aspects of the BCI Meeting Series experience, and is, of course, impossible to capture in print. But the summaries do present the conclusions and consensus opinions reached through such discussion, as well as calls for action and future research.

The breadth and diversity of the workshops made grouping by topic difficult since overlapping themes run through many of the workshops, yet each has its distinct focus and flavor. Overlap was both common and desirable to create the greatest interest among attendees. For example, the workshops ‘Restoration of Upper Limb Function through Implanted Brain–Computer Interfaces’ and ‘Noninvasive BCI-control of FES for Grasp Restoration in High Spinal Cord Injured Humans’ demonstrate workshops on different approaches to restoration of limb function.

Three themes were selected for organizing this report, though other groupings could easily be proposed. The first theme presented here contains workshops focused on BCIs for specific applications or treatment groups. This theme reflects the greater awareness that has developed of the diverse populations that can benefit from BCI. This theme is first represented by a workshop on BCIs for assessment of disorders of consciousness, followed by a set of workshops on recovery of function through therapeutic intervention after stroke or through control of robotics or functional electrical stimulation systems. Several additional workshops cover BCIs for unique populations – ranging from children with motor or neurodevelopmental disorders to the healthy adult population.

The next theme presented here contains a group of workshops that enabled attendees to concentrate on specific signals or technology for advancement of BCIs, showing that signals and technology are still a popular topic. The first workshop included in this section explored specific types of brain signals whose intricacies can present challenges as well as opportunities for BCI research. This workshop is followed by workshops that discussed specific algorithms and emerging signal analyses that can lead to improved BCI function. The theme ends with the consideration of specific hardware considerations for future BCI developments.

Translational and commercial issues in BCI development encompass the final theme for the workshop summaries. Communication applications predominate in this section. They were a major discussion topic in the Applications workshop at the first BCI Meeting and seem closest to being translated commercially. However, the workshops of this theme show the greater interest and awareness in practical translational issues and pathways to commercial success that have developed as the field of BCI research matures. Several workshops also considered the translational issues embodied in deployment of implanted BCI systems. Many of the challenges faced by non-invasive BCIs and implanted BCIs are remarkably similar as we seek to create practical, usable devices that can be deployed for effective and affordable use within the health care system.

Overall, the varied workshops present the startling and ever-growing breadth of BCI applications and user populations, with common themes emerging to inform advancement of BCI performance and applications via active collaborations across disciplines.

BCIs for specific populations/applications

BCIs for assessment of locked-in and patients with disorders of consciousness (DOC)

Organizer: Christoph Guger

Presenters: Christoph Guger (g.tec medical engineering GmbH); Damien Coyle (Ulster University); Donatella Mattia (Fondazione Santa Lucia); Marzia De Lucia (Lausanne University Hospital); Leigh Hochberg (MGH/Brown University/Providence VAMC); Betts Peters (Oregon Health & Science University); Chang S. Nam (North Carolina State University); Quentin Noirhomme (Brain Innovation BV); and Jitka Annen (Université de Liège).

The cognitive function of patients with DOC are currently diagnosed with tools like the Coma-Recovery Scale Revised (CRS-R) [8] which categorized patients as in (1) coma, (2) vegetative state (VS)/unresponsive, (3) wakefulness state (UWS), or (4) minimally consciousness state (MCS). Both patients with DOC and those who are locked-in (LIS) or completely locked-in (CLIS) experience variations in cognitive functions, making an objective system to describe their functions desirable. BCIs have the potential to provide objective descriptions of remaining brain functions based on the classification of recorded EEG, evoked potentials and EEG analysis maps. Furthermore, BCIs can provide communication for some of these patients.

BCIs for this type of application use either motor imagery or evoked potentials. For a motor imagery design, patients are asked verbally to perform certain imagined motor movements that will produce different event-related desynchronization (ERD)/event-related synchronization (ERS) patterns. The BCI system classifies the associated EEG data and reports the accuracy with which it can separate the EEG associated with different motor imagery instructions. For an evoked potential design, an auditory oddball paradigm is used to produce a P300 response or a mismatch negativity (MMN). Semantic paradigms can also be used to produce an N400 or P600 response. Because most DOC patients lack reliable vision, BCIs using vibro-tactile displays for an odd-ball task are also important.

University Hospital Liége is using fMRI, EMG, and EEG with auditory/vibro-tactile paradigms to assess DOC and LIS patients. The University of Ulster uses auditory guided motor imagery, having developed auditory feedback to improve accuracy [9]. The Oregon Health and Science University is running motor imagery BCIs with LIS patients to enable communication, and also developed a BCI-based screening test for vision. g.tec developed a system called mindBEAGLE that runs AEP-P300, VT-P300, and motor imagery based BCI paradigms for assessment and also for communication. The system is currently being tested at 10 sites with acute, sub-acute and chronic patients [10, 11]. Cliniques Center Hospitalier Universiteire Vaudois developed an AEP-based method to predict the outcome of acute TBI patients with above 80% accuracy [12] that is being evaluated with four partners in Switzerland. Massachusetts General Hospital is using AEP-P300, VT-P300 and motor imagery in acute-TBI patients in the intensive care unit to test if patients are able to understand conversations and to enable them to communicate. Fondazione Santa Lucia is using AEP-based paradigms to test MMN, P300, N400, and P600-based paradigms [13]. North Carolina State University is developing BCI algorithms for assessment and communication with DOC and LIS patients.

BCIs have the potential to provide an objective marker of whether DOC and LIS patients can perform certain experimental paradigms. If the BCI system gives 100% accuracy, then the patient both understood the instructions and performed the task correctly. Therefore, the patient is assumed to be able to follow conversations. However, if the accuracy is 0%, then the situation is not clear. The patient might not have understood the instructions, might have been unable to do the task or the BCI interpretation of the brain activity involved in the task may not be accurate. Repetition of the assessment provides insight into daily fluctuations or medication effects and helps to plan treatment or visiting schedules. The assessment also provides a first step to understanding whether patients will be able to communicate. A positive assessment leads to a next step in which a BCI can be used to answer Yes/No questions or as a spelling system.

Brain–computer interface based motor and cognitive rehabilitation after stroke

Organizer: Donatella Mattia

Presenters: Dr Donatella Mattia (Fondazione Santa Lucia); Dr Floriana Pichiorri (Fondazione Santa Lucia); Dr Natalie Mrachacz-Kersting (Aalborg University – AAU); Dr Sonja Kleih (University of Würzburg); and Andrea Kubler (University of Würzburg).

EEG-based BCIs can support neuronal plasticity after stroke in both the sub-acute and chronic stages. While positive effects on post-stroke motor rehabilitation have been demonstrated, there have been only a few randomized controlled trials. The increasing synergy between rehabilitation medicine and neuroscience is producing a radical change in neurorehabilitation, consisting of a ‘copernican’ revolution from a patient-centered to a brain activity-centered perspective in designing rehabilitation interventions. For this perspective, BCIs can provide an instantaneous window into brain activity and mechanisms which underpin functional recovery. The vision is that BCIs can not only enable direct control of a device (e.g. robot) to restore or improve patient performance, but also feedback (to patients and therapists) about the ongoing brain changes associated with BCI-driven exercises. Accordingly, BCIs can fill two roles: as a device to rehabilitate; and as a decision-making guide for intervention.

Several non-invasive BCI-based approaches are currently being studied to promote functional motor and cognitive recovery after stroke. A sensorimotor rhythms-based BCI combined with realistic visual feedback of the upper limb supports hand motor imagery practice in subacute stroke patients [14, 15]. Following a randomized control trial at the Santa Lucia Foundation in Rome on efficacy [16], this BCI-assisted rehabilitative intervention is being used in a rehabilitation ward for a large clinical trial. Trial goals are to determine duration and frequency of intervention, follow-up of clinical and neuroplasticity-related benefit, and standardization of neurophysiological intervention outcome measures. An ‘associative’ BCI for lower limb motor rehabilitation provides timely coupling between brain commands and afferent response signals through detection of movement-related cortical potentials (MRCP) combined with functional electrical stimulation (FES) [17]. Clinical efficacy in a cohort of chronic stroke patients has been shown [18] and this BCI is now being tested in acute patients. Preliminary data also show promising results for cognitive (attention and memory function) rehabilitation assisted by BCI-mediated neurofeedback in chronic and sub-acute stroke patients [19].

Group discussion produced priority directions for further BCI development for stroke rehabilitation. Future efforts should not concentrate exclusively on either the development of more effective decoding algorithms or the integration of evidence-based clinical principles to harness brain plasticity through task-dependent experience. A ‘hybrid’ approach pursued by multidisciplinary teams will best fulfill the complexity of the rehabilitation requirements. Further, the development of new algorithms to decode motor/cognitive ‘intentional’ signals should be physiologically driven instead of data driven. This will enable incorporation of the growing knowledge on brain reorganization after stroke damage. Such development will synergistically advance neuroscience questions relevant to translation of BCI into practice, such as identifying determinants of response-to-treatment and tailoring/shaping interventions according to patients’ clinical and neurophysiological characteristics.

Crucial questions affecting clinical use of successful BCI systems include the timing of intervention delivery, adaptability (without harm) to patient compliance, integration/interactions with conventional treatment, and intervention efficacy. The consensus was that individual sessions should be relatively short, with an intensive rehabilitation regimen preferred over long sessions. Future clinical trials should use individualized rehabilitative goals and establish sensitive efficacy metrics (e.g. minimally clinically relevant differences) instead of typical BCI performance metrics.

BCIs for stroke rehabilitation

Organizer: Christoph Guger

Presenters: Christoph Guger (g.tec medical engineering GmbH); José del R. Millán (École polytechnique fédérale de Lausanne – EPFL); Donatella Mattia (Fondazione Santa Lucia – FSL); Junichi Ushiba (Keio University – KU); Surjo R. Soekadar (University Hospital Tübingen – UHT); Vivek Prabhakaran (University of Wisconsin-Madison – UWM); Natalie Mrachacz-Kersting (Aalborg University – AAU); and Kyousuke Kamada (Asahikawa Medical University – AMU).

Worldwide, stroke is the leading cause of long-term disability and 30–50% experience very limited recovery. In just the USA alone, there are 800,000 new stroke cases annually, and the numbers are increasing. This workshop featured presenters from eight worldwide institutions (acronyms in the presenter list above). All of them have either an international or national BCI-based stroke rehabilitation program.

Motor imagery-based BCIs are well suited for stroke rehabilitation because these systems are able to capture movement imagination or movement attempts and immediately trigger a real movement via an actuator: functional electrical stimulation (FES), nerve stimulation, prosthetic device or exoskeleton). BCI systems for stroke rehabilitation measure either the event-related desynchronization (ERD)/even-related synchronization (ERS) (EPFL, FSL, UHT, g.tec, UWM, KU, AMU) or use motor-related cortical potentials (MRCP) (AAU). Closing the brain activity/physical response loop through use of the BCI produces central nervous system (CNS) plasticity that leads to restoration of normal brain function or a relocation of the functional control to undamaged areas of the brain.

UHT developed an ERD/ERS-based BMI system that triggers a real hand movement using an orthosis, and could show significant improvements in a group study [20] as further data is collected. FSL is using a virtual BCI-controlled avatar to provide visual feedback, and has shown improvements [16]. EPFL uses a BCI-FES device to produce motor movements [21]. g.tec uses a combination of a first-person view avatar with FES stimulation of the corresponding body parts (hand or leg) in a system called recoveriX [11]. UWM enables motor movement via a BCI-FES device and also triggers a tongue stimulator for enhanced feedback [22]. KU uses a BCI-robotic device to generate the movements and showed effectiveness in a group study [23]. AMU uses the BCI system for recovery after stroke, but also after neurosurgery in acute patients. AAU [18] showed improvement with an MRCP-based system with peripheral nerve stimulation.

Successful rehabilitation of chronic or sub-acute patients requires pairing attempted or imagined motor movement with feedback based on brain activity to form a closed-loop system. The usage of a virtual avatar activates the mirror neurons that are tightly coupled with the sensorimotor cortex. The actuator produces limb movement which also activates proprioceptive feedback when the patient imagines or attempts the movement, which also activates the motor cortex. Finally, the BCI picks up all these changes in the EEG signal and triggers in real-time the next movement.

Further studies will show the degree and speed of motor improvements. Especially for acute patients, it is important to show that BCIs provide additional or faster improvement as compared to conventional therapies. BCIs also provide numeric feedback on accuracy that can be used to motivate and coach the patient.

Therapeutic applications of BCI technologies

Organizer: Dennis McFarland

Presenters: Dennis McFarland (National Center for Adaptive Neurotechnologies); Janis Daly (University of Florida); Chadwick Boulay (Ottawa Hospital Research Institute); Muhammad Parvaz (Icahn School of Medicine at Mount Sinai); and Michael Luhrs (Maastricht University).

BCI technology can restore communication and control to people who are severely paralyzed. There has been speculation that this technology might also be useful for a variety of diverse therapeutic applications [24]. This workshop considered possible ways that BCI technology can be applied to motor rehabilitation following stroke, Parkinson’s disease, and psychiatric disorders. These diverse applications all share a reliance on state-of-the-art neuroimaging and signal processing technologies. At the same time, each presents a series of unique challenges.

Dennis McFarland described several ways that BCI technologies have been used for development of therapeutic applications. These include the traditional neurofeedback paradigm, EEG-based imagery enhancement, closing the sensorimotor loop, training task preparation, and state-dependent training. While several of these paradigms have been designed for rehabilitation of motor disorders, others, such as state-dependent training, potentially have broader application. Even for the well-characterized motor system, much remains to be learned about the role of its various parts in terms of the signals generated and their potential relevance for rehabilitation. At the same time, there is great potential for modifying the activity of brain regions that could result in therapeutic benefit, provided that we acquire the necessary knowledge.

Janis Daly described the process of rehabilitation of motor function post-stroke. She noted that methods that appear to work in some patients are ineffective in others, a phenomenon that requires explanation. This will require a better understanding of the brain signals we use and the nature of individual differences.

Chadwick Boulay described his research that shows how Parkinson’s patients undergoing deep brain stimulation surgery can learn to control the amplitude of their subthalamic beta oscillations (a biomarker for disease severity) using a virtual reality BCI. He discussed how BCI technologies might be used to improve function in this group by down-conditioning pathological signals to induce adaptive plasticity in the underlying networks. It should not be taken for granted that the best signals to use in a therapeutic BCI are those with the strongest correlation with disease state. Similarly, we should be careful not to choose signals simply because they enable accurate volitional control.

Muhammad Parvaz described the altered response to emotion-provoking stimuli that occurs in cocaine addiction. These individuals have a blunted reaction to normally positive stimuli and a reaction to cocaine stimuli which actually intensifies during the initial period of abstinence. He discussed how BCI technologies might be applied to facilitate addiction recovery. There is a critical need to compare the outcomes of this intervention with those from mainstream pharmacological and cognitive-behavioral interventions to provide comparative metrics that will further guide evidence-based clinical decision-making.

Finally Michael Luhrs described how feedback during fMRI imaging can be used to alter the activity of well-localized structures that may not be readily assessable to non-invasive electrophysiological recordings.

There was a general consensus that it is not known at present what specific neural signals might be employed and how best to use these. This area is only just beginning to be explored, and at this point it may be best for researchers to explore many different possibilities.

Clinical applications of brain–computer interfaces in neurorehabilitation

Organizers: An H. Do; Marc Slutzky; and Zoran Nenadic.

Presenters: An H. Do (University of California, Irvine); Marc Slutzky (Northwestern University); Surjo Soekadar (University Hospital Tübingen); Zoran Nenadic (University of California, Irvine); and Charles Liu (University of Southern California, Los Angeles).

A significant challenge for clinical neurorehabilitation of conditions such as stroke, spinal cord injury (SCI), and traumatic brain injury (TBI) is the lack of a satisfactory means to restore lost motor functions [25, 26]. New and effective techniques are needed to fill this gap and provide meaningful functional restoration to the affected patient population. BCIs have increasingly been studied as one such means. In particular, BCIs may serve as neuroprostheses to replace lost motor function in those with complete paralysis. Alternatively, BCIs may act as tools that facilitate neural repair mechanisms to improve residual motor functions in patients with partial paralysis. However, BCI systems are not yet used in mainstream rehabilitation. This workshop examined the means by which BCIs can eventually be deployed in clinical practice.

BCI-controlled neuroprostheses decode neural signals into control signals for external prosthetic devices (e.g. FES, robotic exoskeleton, etc.) [27–31]. Several BCI-controlled neuroprosthetics for both upper and lower extremities have been developed using both invasive and non-invasive recording methods. Although there are preclinical studies in humans and early phase clinical trials of BCI-controlled neuroprosthetics, there are still no Phase III/pivotal trials for these systems to demonstrate safety, efficacy at reducing disability, and reliability. High system complexity and the potential need for human implantation present significant challenges to definitive large-scale clinical trials. Neurosurgeons specializing in neurorehabilitation and functional neurosurgery will be critical partners in the design, maturation, and clinical testing of such systems.

BCIs can also be used as tools to elicit neural repair mechanisms. The underlying biological mechanisms are still incompletely understood and are generally believed to center around Hebbian learning. For example, applying sensory feedback as a part of BCI operation can upregulate input into the post-stroke sensory and motor cortices, and subsequently enhance motor cortex output [32]. Alternatively, Hebbian learning may also be elicited by simultaneously activating the primary motor cortex (via BCI control) and lower motor neurons (via functional electrical stimulation) [33, 34]. Some Phase I/II studies have demonstrated that BCI-based rehabilitation is potentially safe and may be efficacious in reducing disability [33, 35].

Despite the potential for clinical application, the lack of definitive Phase III clinical trials confirming that BCIs are safe and effective at mitigating disability after neurological injury has prevented the use of BCIs in clinical neurorehabilitation practice. Further, once successful Phase III trials have been completed, it will still be necessary to obtain regulatory approval. In addition, it will be critical to secure interest amongst physicians and patients, as well as the willingness of medical insurance payers to reimburse for their use. Without a reimbursement scheme, the likelihood that BCI systems will be adopted in clinical practice will be low.

BCI devices for inducing neural repair mechanisms and BCI-controlled neuroprostheses are not yet well defined, and work is needed to develop appropriate device designs and operating protocols for clinical deployment. Nevertheless, the BCI research community should consider these clinical science gaps, as well as regulatory and commercialization challenges, while developing BCI systems. Incorporation of regulatory and deployment strategies in long term BCI research plans will speed adoption into clinical practice.

Restoration of upper limb function through implanted brain–computer interfaces

Organizer: Jennifer Collinger

Presenters: A. Bolu Ajiboye (Case Western Reserve University, Louis Stokes Cleveland VA Medical Center); Richard Andersen (California Institute of Technology); Jennifer Collinger (University of Pittsburgh, VA Pittsburgh Healthcare System); Robert Gaunt (University of Pittsburgh); and Takufumi Yanagisawa (Osaka University Medical School).

This workshop brought together various research groups currently conducting clinical trials of implanted BCIs with the goal of restoring upper limb function lost after injury or disease. Most are operating under Investigational Device Exemptions (NCT00912041, NCT01849822, NCT01964261, NCT01364480, and NCT01894802). To date, most studies have used intracortical microelectrodes implanted in motor cortex to extract velocity-based information to control computer cursors or robotic arms [27, 28, 36–38]. Posterior parietal cortex is an alternative cortical target that contains information about movement goals [39] and trajectories [40]. Recent work in humans confirmed that goal, trajectory, and hand shape information can be decoded during motor imagery [41, 42]. Future participants will have arrays in both posterior parietal and motor cortex for direct comparison of the potentially complimentary contributions of the two areas.

Two primary methods of restoring limb function include reanimation of a person’s own arm through functional electrical stimulation (FES), or replacement of arm function using a robotic prosthetic arm. Ultimately, people with spinal cord injury would prefer to have function restored to their own limb [43]. However, for others, a prosthetic arm may be more appropriate. Robotic arms enable repeatable and reliable output, allowing development to focus on the BCI. An intracortical BCI has enabled an individual with tetraplegia to control a robotic arm in 10 simultaneous and continuous dimensions (including translation, orientation, and hand-shape) that led to significant improvements in upper limb function [27, 38]]. Others have shown use of an intracortical BCI for tasks such as drinking from a cup lifted by a robotic arm [28].

Integration of BCI with FES has a number of challenges, including variable end effector dynamics that can be posture or time-dependent, particularly as muscles fatigue. Previous work has shown that intracortical BCI can enable control of a realistic, dynamic arm model [44], and also that single joint movements can be decoded from motor cortical activity [45]. In parallel, an implanted FES system for restoring hand and arm movement is in development [46, 47]. Preliminary results from an investigation of BCI control of implanted FES were also discussed.

Electrocorticography (ECoG) BCIs record neural activity from electrodes placed on the surface of the cortex. ECoG has enabled classification of different hand postures [48, 49] and control of endpoint velocity [50] during real-time prosthetic control. Although ECoG provides less detailed information about movement than intracortical recordings, it may be more stable [51]; however, longer-term studies are needed.

A number of challenges must be solved to move the technology forward. First, most BCIs lack somatosensory feedback, which will be essential for restoring natural upper limb function. Current clinical trials using intracortical or mini-ECoG stimulation of the somatosensory cortex suggest that sensations can be generated in hand-related areas and that stimulation parameters can modify the intensity of the sensation [52, 53]. Another challenge is developing decoding models that account for neural changes during object manipulation [38]. Object manipulation requires control of fingertip force in addition to the kinematic parameters typically decoded by the BCI. We also discussed the potential of computer vision or autonomous robotics to assist BCI users and improve performance [54, 55]. Many challenges remain in the transition of such implanted BCI technology out of the laboratory and into the homes of patients.

Non-invasive BCI-control of FES for grasp restoration in high spinal cord injured humans

Organizers: Gernot Müller-Putz and Rudiger Rupp.

Presenters: Gernot Müller-Putz (Graz University of Technology); Joana Pereira (Graz University of Technology); Patrick Ofner (Graz University of Technology); Andreas Schwarz (Graz University of Technology); Rüdiger Rupp (University Hospital in Heidelberg); and Matthias Schneiders (University Hospital in Heidelberg).

The bilateral loss of hand-grasp function associated with a complete or nearly complete lesion of the cervical spinal cord severely limits an individual’s ability to live independently and retain gainful employment. Any functional improvement is highly desirable not only from the patient’s viewpoint, but also economically. Neuroprostheses for motor function based on Functional Electrical Stimulation (FES) provide a non-invasive option for improvement of upper extremity function [56]. In particular, hybrid-FES systems consisting of FES and active orthotic components are effective in restoration of everyday manipulation capabilities [57].

EEG-based BCIs offer a valuable component of a neuroprosthetic user interface with the major advantage of operation independent from residual motor functions. Further, motor imagery (MI)-based BCIs have enormous implications for providing natural control of a grasping and reaching neuroprosthesis, especially for individuals with high spinal cord injury (SCI), by using volitional signals from brain areas directly involved in upper extremity movements.

The workshop first summarized the state of the art in non-invasive grasp neuroprosthesis and hybrid BCI [58–60]. Subsequently, the current possibilities of non-invasive BCI-controlled neuroprostheses were presented [61–65], with an emphasis on application for everyday activities in individuals with high SCI [66, 67]. Many motor imageries currently used for BCI control are unintuitive and therefore impractical for real-life applications. A major workshop focus was therefore the identification of more realistic control commands. Research results on this topic were presented, and we discussed neural correlates behind goal-directed movements [68], recent progress in non-invasive movement decoding [69, 70], and time domain classification of different reach and grasp movements [71]. The workshop then covered the steps necessary for successful neuroprosthetics use, including the characteristics of the neurological status of the innervation of muscles of potential end users of neuroprostheses with SCI. The lessons from clinical work with neuroprosthetic users were of greatest interest. Most patients have a C4 or C5 level incomplete lesion, resulting in partly preserved arm motor functions. This points to the need for hybrid-BCI approaches to merge BCIs with traditional user interfaces. For half of the patients, functional electrical stimulation does not activate the hand and arm muscles because of muscle denervation caused by damage of spinal cord motor neurons. Finally, the technical and neurophysiological concept of a neuroprosthesis with surface electrodes was introduced and a workshop participant volunteered to be involved in a nice demonstration of the neuroprosthesis showing two grasp patterns, the palmar and lateral grasp respectively.

Future research will elaborate on the possibility of decoding intended complex movements of the whole arm and hand from non-invasive EEG. Studies already show that individual portions of complex movements can be decoded, e.g. the intention to move to a goal, the movement itself, or single grasps. The challenge will be combining these decoders and transitioning to attempted or imagined movement. Showing feasibility in individuals with SCI is another major challenge. The combined controller could either operate an FES-based neuroprosthesis or a robotic arm, depending on the degree of lower motor neuron damage and the capabilities of future FES-systems.

BCI research and development for children

Organizer: Disha Gupta

Presenters: Disha Gupta (Burke Medical Research Institute/Weill Cornell Medical College); Patricia Davies (Colorado State University); William Gavin (Colorado State University); Scott Makeig (Swartz Center for Cognitive Neurosiene, UCSD); Walid Soussou (Wearable Sensing LLC); and Jewel Crasta (Colorado State University).

Established BCI applications largely focus on neurological disorders [72, 73], traumatic brain injuries [74], or strokes [35, 75, 76] in adults. Emerging applications generally engage healthy adults to showcase working BCI systems [75, 77, 78]. The focus on adults is natural because of their well-characterized EEG and the relative simplicity of acquiring robust data from them. However, BCIs may also be useful for children – for treating neurodevelopmental disorders (e.g. autism, attention deficit hyperactivity disorder [ADHD]), neurodegenerative disorders (e.g. spinal muscular atrophy [SMA]) or orthopedic injuries given limited alternative avenues for therapeutic intervention. BCI-based replacement or enhancement of impaired function has the possibility to improve the quality of life of these children and even to prevent the progression of the disorder. Indeed, BCI has been shown to be effective as a potential treatment in ADHD [79]. However, translating adult BCI applications to pediatric applications [80] is not straightforward, especially in neurologically impaired groups, with challenges characteristic to pediatric neurophysiology research. Workshop discussion focused on defining and addressing challenges to effective and successful pediatric BCI applications.

• Brain reorganization: The ongoing development [81, 82] of a child’s brain makes it more likely to undergo extensive brain circuit reorganization depending on timing and location of brain injury [83–85].

• EEG signals: Injury may produce spatially and/or spectrally atypical evoked responses and oscillatory data features, with limited age-specific normative EEG data available. Neurodevelopmental consequences of injury may arrest, delay, or eliminate specific EEG features [86–89], with heterogeneous effects [90–92].

• Source localization: Subspace-decomposition and source-localization methods could be valuable tools for objectively reducing data dimension, augmenting signal-to-noise ratio, reducing spatial overlaps, and identifying weak and atypical features [93, 94]. Age-specific generic head-models, perhaps from the NIH pediatric MRI initiative [95, 96], could be useful. Subject-specific head-models could increase accuracy and information value, but also pose challenges to combining data across children.

• Experimental paradigms: Obtaining responses time-locked to stimuli can be challenging in younger children with cognitive and behavioral disorders. Passive ERP paradigms and advanced signal processing methods will be required to achieve maximum efficacy. Communicating experimental paradigm requirements and behavioral expectations to children with specific impairment/age/cognitive abilities may be difficult or impossible. Engaging children’s attention may require packaging cue, stimulus, and feedback presentations within a game with rewards designed to maintain focus. In children, EEG features can also be affected by the modality and nature of the stimulus, the child’s psychological and physiological state at the time of testing, and individual developmental differences in cortical maturation, as per the additive model [97].

• EEG acquisition: For these children, high-density, wet, wired EEG systems involving long training sessions are not ideal. Sensory sensitivities to gel, abrasion, and headgear may require extensive desensitization. Wires may pose risk of injury and create artifacts. Although often not available for pediatric head sizes, dry, active, and/or wireless headsets [98–100] could mitigate some of these obstacles, and may be more robust to movement and electrical artifacts [101]. Software could also provide real-time artifact rejection and subspace decomposition [102–104].

In summary, the challenges of BCI use are increased for younger children, especially those with brain injury or neurodevelopmental disorders. Pediatric BCI applications cannot be direct translations of adult studies, and can only achieve success with active and collaborative attention to the above challenges. Scientific and funding communities should nurture pediatric BCI R&D in parallel with adult applications.

Passive BCI and neuroadaptive technologies

Organizer: Thorsten O. Zander

Presenters: Patrick Britz (Brain Products GmbH); Martijn Schreuder (ANT Neuro); Mike Chi (Cognionics); Laurens R. Krol (Technische Universität Berlin); Lena Andreessen (Technische Universität Berlin); and Thorsten O. Zander (Technische Universität Berlin).

Today’s interaction with technology is asymmetrical in the sense that (1) the operator has access to any and all details concerning the machine’s internal state, while the machine only has access to the few commands explicitly communicated to it by the human, and (2) while the human user is capable of dealing with and working around errors and inconsistencies in the communication, the machine’s flexibility in that regard is still very limited [105]. With increasingly powerful machines, this asymmetry has grown and is still growing, but our interaction techniques have remained the same. This presents a clear communication bottleneck: users must still translate their high-level concepts into machine-mandated sequences of explicit commands, and only then does a machine act [106].

However, during such asymmetrical interaction, the human brain is continuously and automatically processing information concerning its internal and external context, including the environment and ongoing events. Passive BCIs can access the information in this brain activity in real time so that the machine can interpret it and thus generate a model of its operator’s cognition [107, 108]. This model can serve as a predictor to estimate the operator’s intentions, situational interpretations, and cognitive state, e.g. emotions, enabling the machine to adapt to them, essentially responding to the user without having received any form of explicit communication. Such adaptations can even replace standard input entirely [109].

This neuroadaptive technology is specifically relevant to auto-adaptive experimental designs, but also opens up paradigm-shifting possibilities for technology in general, addressing the issue of asymmetry in human–technology interactions and relieving the above-mentioned communication bottleneck.

In a moderated discussion, workshop participants from academia and industry reflected on how and where this technology could and should be applied. In particular, since neuroadaptive technology promises to support general human–technology interaction, the discussion focused on applications of general interest. Applications suggested included adaptive learning environments, audio and video tagging, and adaptive automation. Emotion detection could provide a particularly powerful basis for neuroadaptivity: systems with real-time access to what the user experiences positively or negatively can use that information to, for example, learn what makes the interaction more enjoyable, and adapt accordingly.

Common obstacles for general-purpose BCI and thus general-purpose neuroadaptive technology, are hardware limitations and the need for calibration. Larger amounts of cross-context data may help in finding ways to reduce calibration times, perhaps even to zero. Because of this, and also to stimulate the work in this young field, workshop participants agreed that data and algorithm repositories are part of the way forward. Several such repositories already exist, including one at http://www.bnci-horizon-2020.eu and one hosted by the Community for Passive BCI Research (http://www.passivebci.org), which includes both a repository, as well as a communication platform for researchers to exchange experiences.

Furthermore, given the likely presence of i.a. muscle activity and external noise sources during non-experimental BCI use, it is all the more important to make existing BCI models robust against such non-brain influences, and to validate them neuroscientifically. We should link the correlated cognitive processes identified by the model to known neuroscientific findings. Such approaches may also provide new (e.g. interaction-related) neuroscientific findings themselves.

Finally, it was noted that ethical questions must also be considered: what are the ethical consequences of the generation and storage of short- and long-term cognitive and emotional user models?

BCIs for artistic expression

Organizers: Anton Nijholt and Chang S. Nam.

Presenters: Femke Nijboer (Leiden University); Loic Botrel (University of Wuerzburg); Vojkan Mihajlovic (Holst Center /imec, Wearable Health Solutions); Anne-Marie Brouwer (TNO Behavioral and Societal Sciences); Tim Mullen (QUSP Lab); Grace Leslie (MIT Media Lab); Jose Contreras-Vidal (University of Houston); Angela Riccio (Fondazione Santa Lucia); Chang S. Nam (North Carolina State University); and Anton Nijholt (University of Twente).

Artists have been using BCIs for artistic expression since the 1960s [110]. Both interest and opportunities to explore BCI creativity are now increasing because of the availability of affordable BCI devices and software that eliminates the need to invest extensive time in getting the BCI to work or tuning it to their application. Designers of artistic BCIs are often ahead of more traditional BCI researchers in ideas on using BCI in multimodal and multiparty contexts, where multiple users are involved, and where robustness and efficiency are not the main matters of concern.

This workshop was intended for BCI researchers who are interested in non-clinical BCI applications, in particular applications that invite users to play and to be creative with a BCI. The workshop addressed an audience that is interested in research to investigate non-traditional, challenging, and entertaining interactions and in research on using BCI as a channel that allows artistic expression of creativity, moods, and emotions. This workshop presented current (research) activities in BCIs for artistic expression and identified research areas of interest for both BCI researchers and artists/designers of BCI applications. The workshop originated from a special issue of the journal Brain–Computer Interfaces devoted to ‘Arts and Brain–Computer Interfaces’ [111]. Both the special issue and this workshop highlighted that users of artistic BCI technology can be the artists who compose art in real time using BCI signals, performers, audience members or even full audiences using BCI technology together. Often this is done in a multimedia, multimodal and multi-brain context [112, 113]. Current artistic BCI environments allow users to play with and modify animations and musifications, and there are examples of BCI control of instruments and tools for artistic expression and exploration [114].

The workshop included short talks, a performance, questions and answers, and general discussion. Presentations described providing ALS patients with painting tools for home use [115], an initiative to organize a design competition, and a neuro-catwalk fashion show displaying designs of attractive and artistically satisfying BCI headsets. Research was presented on what goes on in the brain of a juggler and whether that information can be visualized or sonified to make a performance even more attractive [116]. What goes on in the brains of readers of fiction? Can we distinguish between reading ‘neutral’ texts versus reading ‘emotional’ texts [117]? Interactive fiction where a reader’s emotional state is used to select the next episode in a narrative is one of the possible application areas.

Another presentation reported on the emotional and esthetic processes produced in the brain while observing, experiencing, and producing art. Many examples, often taken from the annual Mozart & the Mind festival in San Diego, were presented in which musicians and researchers from cognitive neuroscience and neurotechnology team up to create BCI music performances [118]. Workshop participants also experienced a music improvisation by Grace Leslie incorporating flute, electronics, and brain activity, introducing and illustrating her concept of ‘introspective expression’ (e.g. http://www.graceleslie.com/Vessels).

The role of intention and the role of control during artistic expression using BCI [119] emerged as an area for future discussion. Can lack of robustness and the presence of artifacts play a positive role in creation, performance and experience of an artistic BCI? Should BCI be considered as a tool, similar to a paintbrush, or can it be used to create new forms of artistic expression? Further investigations of such questions are included in plans for a follow-up workshop on ‘BCIs for Artistic Expression’ at the 7th BCI Meeting.

Studying learning with BCIs

Organizers: Aaron Batista; Steven Chase; Jose Carmena; and Byron Yu.

Presenters: Marc Scheiber (University of Rochester); Ben Engelhard (Hebrew University and Princeton University); Aaron Batista (University of Pittsburgh); Karunesh Ganguly (University of California San Francisco); and Vivek Athalye (University of California, Berkeley and Champalimaud Institute).

The neural mechanisms whereby we gain new knowledge and expertise are still largely unknown. We particularly lack information about how synaptic plasticity can lead to new patterns of activity among a network of neurons that control behavior. BCIs offer distinct advantages for studying the neural basis of learning. In a BCI, we record directly from all the neurons that impact behavior (that is, the movement of a computer cursor or a robotic device), and we can trigger learning simply by providing animals with novel mappings from neural activity to behavior. Learning is a widespread neural phenomenon, affecting many brain areas and pathways. Current technologies do not allow us to directly monitor all learning-related changes, but using the BCI approach, the effects of those changes must be observed in the activity of the neurons that we record because only those neurons impact behavior. Thus, a BCI can provide new insight into the neural basis of classical phenomena in motor and cognitive learning, including adaptation, exploration, rapid re-learning, interference, and skill learning [120]. In this workshop, we discussed insights into the neural mechanisms of learning that arise from BCI control.

Monkeys can rapidly learn to control the activity of a small group of neurons. In one study [121], up to four neurons were selected to control the vertical position of a cursor on a computer screen. Remarkably, monkeys could rapidly find ways to co-modulate the neurons whether or not the neurons had similar or different ‘preferred direction’ tuning, and whether or not they were situated near each other within the motor cortex. In a more recent study [122], a monkey learned to use 16 arbitrarily selected neurons to control four different grasp shapes of a virtual hand. To some extent, monkeys thus can learn to combine the activity of arbitrarily selected motor cortex neurons for BCI control.

Some new BCIs can be learned more rapidly than others [123]. The key difference is whether good control of the BCI requires the monkey to exhibit new patterns of neural activity (these are learned slowly) or whether the animal can control the BCI simply be re-using pre-existing patterns of neural activity in new ways (these can be learned more rapidly.)

In now-classic studies of BCI learning [124, 125], it was found that animals can learn to control arbitrary BCI mappings. A closer analysis of those data is revealing the neural strategies at play during BCI learning. Initially, animals modulate the activity of individual neurons independently while they search for neural activity patterns that provide them with good control of the BCI. As skill develops, further refinement involves the coordinated modulation of the components of neural activity that are shared among many neurons.

Rodents that engage in BCI learning will ‘replay’ the newly learned neural activity patterns while they sleep [126]. This replay occurs during the slow-wave portion of sleep, and it is highly predictive of learning, in that when it occurs, the animal is more likely to improve its task performance the following day.

In new studies by Ben Engelhard, Elion Vaadia, and their colleagues, they showed that when animals learned to modulate the activity of a single neuron, there were changes both in the activity of other neurons, and in the interactions among pairs of neurons, even though those changes were not required for the learned behavior. These effects were well-explained by a neural network model in which plasticity is modulated by a global reward signal.

Taken together, these four examples show that by recording the activity of populations of individual neurons, and providing ‘neurofeedback’ via a BCI paradigm, we are beginning to see the neural population changes that underlie learning.

Advancing BCI research through specific signals or technology

Exploiting cognitive processes for brain–machine interaction

Organizers: Iñaki Iturrate (École polytechnique fédérale de Lausanne – EPFL) and Ricardo Chavarriaga (EPFL).

Presenters: Benjamin Blankertz (Technische Universitat Berlin); José del R. Millán (EPFL); and Richard Andersen (California Institute of Technology).

BCIs often rely on neural correlates of motor processes for the direct control of external devices. These systems, however, can also rely on other cognitive processes naturally elicited during the interaction with the device such as attentional processes [127], conscious processing [128] and mental workloads [129].

These signals are linked to the task being executed, and thus provide a natural way of boosting the interaction with the machine [130]. Furthermore, they may also increase the user’s sense of embodiment with the machine; that is, their sense that the BCI is an extension of their own body [130]. In the field of human–machine interaction, this embodied interaction has been shown to decrease the task workload while simultaneously increasing the user’s acceptance of the system [130]. Furthermore, the decoding of these signals can provide high-level information crucial to solving the task in a more efficient way, assuming the low-level execution of the task is dealt with by the device. Like embodied interaction, this concept of shared-control strategies [131] has been shown to decrease the task workload by relieving users of the effort of dealing with low-level planning.

A classic example of a cognitive signal is the P3 component of event-related potentials (ERP). Since its advent as an interaction signal in the late 1980s [132], recent works have pushed forward its use under more complex cognitive tasks. Rapid serial visual presentation (RSVP) is one such example, where several images (both distractors and targets) are sequentially shown in a central location to avoid gaze shifts [133]. Recently, P3 signals have also been correlated with different levels of cognitive processing [134] and the successful detection of such levels in single trials opens the door for BCIs for out-of-the-lab applications [134].

Another promising ERP-like cognitive process for BCIs is that of error processing [135]. Error-related signals are evoked after the device executes incorrect commands, and recent studies have shown how they can be decoded in single trials [136, 137]. Furthermore, they are present in a wide range of contexts and seem to share a common neural generator, despite their variability across different tasks [138]. A natural way of including these signals, also termed error-related potentials [139], is that of using them as a way of canceling incorrect selections made by the BCI [140]. Alternatively, they have been recently used as a way of teaching different devices how to solve motor tasks via reinforcement learning algorithms [141, 142].

Asynchronous signals during motor planning are another option. Neural activity preceding actions in motor tasks predicts the onset of self-paced movements [143, 144]. For example, it can provide a more natural control of neuroprostheses by providing a quicker response to the user’s desire to move. In a different scenario, these cognitive processes can also be exploited in applications for able-bodied users such as improved response time for self-paced decisions of braking and steering during car driving [144].

Implanted BCIs can also exploit cognitive processes, such as neural spiking activity that encodes high-level information to improve interaction. In particular, several studies have demonstrated how the parietal cortex not only decodes the motor imagery of body limbs [145], but also goal locations [146] or hand shape representation (i.e. grasping types) [42].

Decoding speech processes using intracranial signals

Organizer: Christian Herff

Presenters: Tanja Schultz (University of Bremen); Dean Krusienski (Old Dominion University); Jon Brumberg (University of Kansas); Emily Mugler (Northwestern University); David Conant (University of California, San Francisco); James O’Sullivan (Columbia University); Zac Freudenburg (University Medical Center Utrech); Christian Herff (University of Bremen); and Stéphanie Martin (EPFL).

Speech provides a natural and fast means of communication that is mostly unharnessed by current BCIs. As a communication method, direct decoding of brain activity related to intended speech would be a massive breakthrough for BCI research. Advantages of intracranial recordings over scalp recordings include high spatial and temporal resolution recordings of cortical activity during the speech process without contamination by motion artifacts. This enables in-depth analysis of the complex dynamics of speech processes. High-gamma activity, which can be reliably measured by intracranial recordings, provides localized information about cognitive processes [147], including speech production and perception [148].

This workshop presented the current state-of-the-art in decoding of speech processes in intracranial signals. Using regularized Linear Discriminant Analysis segmental features can be classified with high accuracies in overtly produced continuous speech [149]. Analyzing the utilized classification models enables the investigation of the spatial topography and temporal dynamics for the manner and place of phoneme production. Another study demonstrated that these articulatory gestures are insensitive to within-word context [150], while the classification of phonemes is influenced by co-articulation. Despite the influence of within-word context, consonant phonemes can still be decoded from electrocorticographic (ECoG) activity with accuracies up to 36% (chance level 7.4%) [151]. To investigate speech production further, another group recorded ECoG activity in parallel with videos of the mouth and ultrasound imaging of the tongue [152]. After extracting parameterized features such as lip opening and the position of specific points on the tongue surface, a direct mapping between motor cortex activity measured with ECoG and articulator movement was calculated. Besides articulator control, it was also shown that the duration of words strongly influenced the high gamma response in ECoG recordings [153]. This finding emphasizes the importance of equal word lengths for comparisons and classification studies. Another study showed that automatic speech recognition technology [154] can decode ECoG activity during continuous speech into a textual representation [155]. The generative models used in this approach are also useful to investigate spatio-temporal regions of high discriminability between different phonemes. This decoding is not entirely based on the speech perception of one’s own voice. Neural activity only from temporal offsets prior to phone voicing and thus associated with speech planning and production yielded phoneme accuracies up to 40% (chance level 6%). Reinforcing these findings that speech perception might not be necessary for decoding of speech, another study showed that the spectral dynamics of imagined speech can be reconstructed from ECoG activity [156]. Additionally, discrimination between imagined word-pairs during speech imagery was presented [157].

In addition to interpretation of speech production, ECoG also enables in-depth analysis of speech perception. A study investigating the cocktail-party phenomenon [158] showed that deep-neural networks can decode the attended speaker in a multi-speaker listening task. This finding has direct applications in hearing-aids, which could then amplify only the attended speaker instead of the entire acoustic scene.

In conclusion, it was shown that intracranial signals are ideally suited for the investigation of speech processes and might therefore be a promising new direction to restore communication. In the following vivid discussion, the group decided to establish a mailing list to foster future collaboration and the exchange of findings, experiments, and data. Participation is invited on https://mailman.zfn.uni-bremen.de/cgibin/mailman/listinfo/neurospeech.

Novel application fields for auditory BCIs

Organizers: Michael Tangermann and Martin G. Bleichner.

Presenters: Michael Tangermann (Universität Freiburg); Martin G. Bleichner (Universität Oldenburg); Disha Gupta (BURKE Rehabilitation & Research, USA); and and Benjamin Blankertz (TU Berlin).

Real-time decoding of brain signals is one of the strengths of BCI systems. Auditory BCIs can not only establish communication and control for patients [159], but also support basic research on auditory perception and auditory processing. Thus, auditory BCIs can be valuable tools to investigate spatial and temporal auditory attention, music-, word-, and language-processing. Further, auditory BCIs can serve as a building block for novel developments in hearing aids or for cognitive training and rehabilitation for an aging society.

Most workshop participants stated a background in basic BCI research (47%) while the other participants had either a clinical background, worked in industry/R&D or in neuroscience research. The purpose of the workshop was to discuss opportunities and challenges of novel applications for auditory BCIs.

The two most widely used auditory BCI experimental paradigms are steady-state auditory evoked potential, SSAEP [160], and auditory event-related potentials paradigms, aERP [161, 162], and both paradigms benefit when spatial auditory attention is utilized. Brain signals collected under these paradigms can be decoded through standard machine learning approaches for oscillatory [163] and ERP signals [164]. Auditory BCIs support a variety of applications beyond communication and control of devices.

Auditory BCIs can serve as a building block for enhancing the spatial selectivity of hearing devices to identify the audio target of interest to the hearer [158, 165] and selectively amplify those signals. This novel research field builds upon new developments in (mobile) ear EEG recordings [166, 167], which allow for single trial decoding of spatial attention with a small number of electrodes that are located at or around the ear. Dr Gupta has a novel clinical research application using auditory BCI with ear EEG as a tool for cognitive assessment and rehabilitation of autistic children. In-ear EEG is used here as an undisturbing way to record EEG in children to increase their compliance. Auditory BCIs have recently been proposed as a tool to support the rehabilitation of language deficits after stroke [168]. For aphasic stroke patients, a spatial auditory BCI [169] can help patients overcome naming deficits. Auditory BCIs can also bridge between BCI and music research, as they enable a continuous analysis of the brain activity associated with the temporal dynamics of music processing by individual human listeners [170–172].

The discussions among participants identified several challenges to be resolved to push auditory BCIs forward. General improvements in classification performance and paradigm effectiveness are desirable. These improvements would be facilitated by a better understanding of the neurophysiological basis of those processes exploited by auditory BCIs. Additionally, improvement of the clinical relevance and the reliability of auditory BCI systems will eventually be important problems, although other challenges must be overcome before they can be considered to be of the highest priority.

Overall, auditory BCIs are perceived as a rapidly growing research area within the field of BCI, not only for traditional communication applications, but with varied novel clinical and non-clinical application areas on the horizon.

ECoG decoding for BCI

Organizer: Nick Ramsey

Presenters: Nathan Crone (Johns Hopkins University); Mariana Branco (University Medical Center Utrecht); Alan Degenhart (University of Pittsburgh); and Tetiana Aksenova (Atomic Energy and Alternative Energies Commission).

In recent years, cortical surface electrodes, electrocorticogram (ECoG), have become of great interest for BCI as an approach to provide permanent implantable electrodes for BCI. ECoG signals are being studied both from human participants (patients with epilepsy or paralysis) and some non-human participants.

Decoding hand movements requires adequate coverage of relevant cortex (M1 and/or S1 hand region) in terms of topography and electrode density. All the presented studies utilized the high frequency broadband (HFB) power as the most informative feature in ECoG signals [173] for extracting brain function. HFB from multiple electrodes enables decoding of individual finger movements for continuous movements of a robotic arm [50, 174, 175]. HFB can also enable decoding of discrete movements such as alphabet sign language gestures with a goal of developing a BCI for communication [176]. Moreover, gestures as well as hand trajectories can be decoded equally well from S1 and M1, which suggests that even when M1 is affected by pathology (e.g. stroke), S1 may still provide a source of control signals for BCI decoding.

Research in people with paralysis [50] yields similar results to research with epilepsy patients, indicating the validity of BCI development studies in people with epilepsy. In three paralyzed patients implanted with ECoG grids for a maximum of 30 days [177], 2-D and 3-D cursor control has been achieved with promising performance levels (85% and 75% correct hits respectively). Further, a 64-channel BCI implant for patients with tetraplegia (‘WIMAGINE, [178]) will soon be ready for testing. This system, consisting of ECoG electrodes on the body of a transcranial device with epidural recording and wireless transfer capabilities, allows for continuous processing of brain signals, and is intended for control of an exoskeleton.

Overall, decoding limb trajectories and discrete gestures using HFB signals and various electrode grid configurations from sensorimotor cortex yielded quite promising results. However, decoding ECoG for detailed hand or limb movements is not yet at a level that is needed for safe robotic arm control. Decoding sign language for communication may be feasible sooner (now at 75% correct classification of four gestures). Extensive discussions addressed the challenges lying ahead, which include (1) determination of the optimal electrode configuration and placement (only one third of cortical gray matter is accessible at the surface), (2) improving decoding to a level minimally required for real BCI robotic arm control, and (3) the hardware required to handle large numbers of electrodes and high-volume data flows. In addition, a barely touched upon critical issue is the performance of ECoG BCI when the user is not deliberately generating control signals. Preventing high rates of false positive detections, which would render an ECoG BCI implant almost useless, may prove to be a highly challenging endeavor. Progress is further expected with research on microwire grids, network dynamics within and across regions (an advantage of ECoG over micro arrays), and better understanding of the exact nature of cortical movement representation in terms of biomechanics and sensorimotor integration, resulting in neurobiologically informed (constrained) decoding possibilities. Moreover, it is expected that inducing adaptive processes to sharpen the brain response (plasticity) will result in elevated system performance.

Understanding state change and its impact on BCI performance

Organizers: Brent Lance and Tzyy-Ping Jung.

Presenters: Brent Lance (US Army Research Laboratory); Li-Wei Ko (National Chiao-Tung University); Avinash Singh (National Chiao-Tung University); Yufei Huang (University of Texas, San Antonio); Dongrui Wu (DataNova, LLC); Vernon Lawhern (US Army Research Laboratory); and Tzyy-Ping Jung (University of California, San Diego).

The performance of BCI classification algorithms is strongly affected by variations in neural signals driven by changes in the state of the individual using the BCI. As a result, BCI performance tends to be highly variable from session-to-session, drastically limiting the utility and acceptance of BCIs in both medical and non-medical domains.

This problem is compounded by a lack of appropriate data from subjects using BCIs over long periods. While there have been a few studies looking at the variability of P300 ERPs over extended time frames [179], the long-term within-individual variability in many neural signals relevant to BCIs is unknown, as is how that variability affects BCI performance. The US Army Research Laboratory (ARL) is collecting long-term BCI data in collaboration with several universities and startups by developing a video game with embedded BCI paradigms [180]. These BCI paradigms will use a free-to-play-based model, where players are provided in-game rewards for attempting to use the BCIs. The project goal is to obtain 200 hours of neural game-playing data over six months from each of 30 subjects. Another long-term BCI pilot study from National Chiao Tung University (NCTU) in Taiwan and the University of San Diego (UCSD) reports that preliminary results show significant changes in the EEG that correlate with subject-reported fatigue scales.

Novel classification algorithms may also create BCIs that are robust to state change over time. Deep Learning algorithms are a family of machine learning approaches based on stacked layers of neural networks that have recently shown drastically improved performance in domains including computer vision and language processing. Deep Learning has shown promising potential for several BCI problems [181–188]. Another approach, Active Learning, is a method for training or adapting classifiers by identifying highly informative unlabeled data, requesting labels for that data, and incorporating the newly labeled data into the classifier. While many BCI paradigms are not amenable to active learning approaches, some event-related paradigms can incorporate these approaches [18–190].

An alternative approach focuses on extracting novel EEG feature representations to better distinguish and characterize neural states, enabling classifiers to better adapt to changes in those states. Deep Learning can be used for learning feature representations from EEG [191]. An alternative representation would be to aggregate source-localized EEG data into regions of interest, then extract spatiotemporal features from each region.

The discussion identified as particular problems the difficulty of identifying and labeling underlying state from fluctuations in EEG data and of maintaining subject motivation over the course of the experiment. One key suggestion was to examine existing research on pharmacological EEG, i.e. studies which show significance differences in EEG when subjects are exposed to different pharmacological compounds. These studies of artificially induced state change may provide a good starting point for identifying potential hypotheses and experimental confounds for studying naturally occurring state change.

In conclusion, it may be possible to increase the robustness of BCIs during long-term usage through improved understanding of the variability of BCI-relevant neural signals over time and through improved BCI algorithms that are robust to that variability.

Improving BCI usability through transfer learning methods

Organizers: Michael Tangermann and Pieter-Jan Kindermans.

Presenters: Michael Tangermann (University of Freiburg); Pieter-Jan Kindermans (Technical University of Berlin); Hiroshi Morioka (Technical University of Berlin); and Alexandre Barachant (Burke Medical Research Institute).

Originating from the field of machine learning, the transfer learning (TL) concept [192] has recently been adopted by the field of BCI. The goal of transfer learning is to share information across related tasks. In the field of BCI, it typically describes the transfer of information either between sessions with the same user (session-to-session TL), between different users (subject-to-subject TL), or between similar BCI tasks.

Transfer learning can (1) reduce the calibration time of an online BCI, (2) improve overall classification performance, and (3) provide information to better understand the underlying data, the learning problem and the structure of the feature space (although this third application has not yet truly been utilized).

These three goals are linked by the question: ‘What exactly can successfully be transferred between users or sessions?’ The spectrum of transferable knowledge is rather wide. It ranges from traditional knowledge (experimental design, parameters, and protocols), over machine learning knowledge (hyperparameters of machine learning methods and trained ML decoders), to the transfer of features or even raw data. Most TL discussions focus on transfer of machine learning knowledge, but the advent of deep learning and bigger data sets in BCI may soon make the sharing of raw data beneficial for our community.

Most workshop attendees had a background in machine learning. They met to (1) assess the state of the art in TL for BCI, (2) categorize existing TL approaches into a systematic framework, (3) discuss which TL approaches may be most appropriate for the specific BCI application the attendees work on, and (4) identify current bottlenecks preventing a wider application of TL concepts to BCI. Inspired by two overview presentations discussing the most common and best performing TL methods in BCI [193–196]; and two presentations on novel TL approaches [197, 198], discussions in the plenum and in specialized subgroups revealed the following:

• ERP-BCI paradigms benefit greatly from current TL concepts with many opportunities for increased use [199]. On the other hand, current TL approaches seem less effective for ERD/ERS-BCI paradigms.

• TL can provide benefit not only through transfer of the actual signal of interest (e.g. the target ERP), but also through transfer of knowledge about typical background activity (noise, artifacts, etc.). However, little information is yet available on the situations and learning problems that will achieve the most benefit from the transfer of noise information compared to the transfer of signal information [200].

• Non-traditional data representations and feature spaces may provide additional benefit for TL. For example, the representation in covariance space or the information contained in subspace components/sources may be especially suitable for TL.

• Comparison of published TL approaches is impeded by differences in datasets, evaluation procedures, performance metrics, and terminology. The group discussed how a data repository for TL problems would need to be organized to allow for objective benchmarking of TL algorithms.

• Interdisciplinary workshops between the machine learning and BCI research communities would accelerate the adoption of the rapidly increasing number and variety of TL algorithms by the BCI community.

Overall, realization of the great potential of TL, especially for ERP BCIs, will be accelerated by standardization of procedures and data formats to facilitate data sharing to maximize TL efficacy.

Deep learning and other machine learning and signal processing methods for analyzing EEG in BCI paradigms

Organizer: Chuck Anderson

Presenters: Chuck Anderson (Colorado State University); Elliott Forney (Colorado State University); Dean Krusienski (Old Dominion University); Yalda Shahiari (University of Rhode Island); Damien Coyle (Ulster University); and Nick Waytowich (Columbia University and US Army Research Lab).

Many advanced data analysis methods have been developed for EEG pattern recognition, but few have resulted in BCI performance that surpasses what is achieved with simple linear methods. The recent success of Deep Learning methods for difficult problems of image and speech recognition and similarities between such data and EEG signals suggest that Deep Learning might contribute to BCI advances. In this workshop, classical approaches to EEG signal processing and classification were summarized and compared with recent BCI methods using deep learning.

Signal processing methods for BCI have included spatial and frequency-based filtering, common spatial patterns, ways of selecting subsets of channels and time samples, averaging multiple trials, canonical correlation analysis, linear detrending, and common average reference [201]. Graph theory can also be applied to use data to estimate the functional connectivity in the brain. Methods based on undirected methods include coherence, correlation, phase, and directed models such as Granger causality and adaptive directed transfer functions. These methods have shown interesting results regarding brain functional connectivity pattern changes in several neurological impairment models such as patients with ALS, a particular type of mice model of schizophrenia, and patients with Parkinson’s disease.

A motion trajectory prediction (MTP) based BCI can decode real and imagined 3-D hand movements to six targets from EEG. Most MTP BCI studies report the best decoding accuracy when a 0.5–2 Hz bandpass filter is applied to the EEG whereas recent results show that theta (4–8 Hz), mu (8–12 Hz), and beta (12–28 Hz) bands are more robust for MTP when the standard approach bandpass filtered time-series is replaced with time-varying bandpower values for a specified EEG band [201–203].

Nick Waytowich summarized applications of convolutional neural networks (CNN) to data from five BCI paradigms. By performing minimal pre-processing of the data and using regularization techniques to limit the complexity of the trained neural networks, results were obtained that surpassed those of conventional methods in four of the five paradigms. Deep neural networks usually require lots of data, so there are possible advantages of training such methods using data from many subjects.

Chuck Anderson, from Colorado State University, illustrated a way to interpret what is learned by the first layer of a small CNN applied to P300 data. Finally, the performance of several classifiers, including CNNs, was demonstrated on an ipython notebook with EEG data from asynchronous mental tasks. The demonstration showed that the CNNs performed better than an autoregressive modeling approach and the common linear discriminant analysis method.

Discussions following the presentations focused on potential advantages and limitations of Deep Learning methods for BCI. The need for lots of data could limit the applicability of Deep Learning, unless models can be trained on data from many subjects. The interpretation of what deep neural networks learn from EEG data may lead to new hypotheses of brain function during complex cognition.

Algorithms and performance using implanted devices

Organizers: Steven Chase; Aaron Batista; Jose Carmena; and Byron Yu.

Presenters: Steven Chase (Carnegie Mellon University); Josh Merel (Columbia University); Hansjoerg Scherberger (German Primate Center); Yuxiao Yang (University of Southern California); and Paul Nuyujukian (Stanford University).

Even the simplest of movements engages millions of neurons across multiple brain regions. However, to date, intracortical neural prostheses base their actions on the activities of, at most, several hundred neurons, typically from a single brain area. Given this bottleneck in neural output, one might assume that the major limits on BCI control stem from limitations in the information available in the recordings themselves. However, impressive improvements in performance have been demonstrated over the past decade, despite relatively constant numbers of recorded neurons over this time. This suggests that decoding algorithm design may still be a limiting factor in device performance. What are the ultimate limitations on neural prosthetic control? What algorithms would allow those limits to be attained? In this workshop, we discussed our current understanding of the algorithmic principles that are critical for improved BCI control. These talks focused on three parallel directions.

• Harnessing subject learning. Complementary talks, ‘Provably optimal design of intracortical BCI decoding algorithms’ and ‘Leveraging user and decoder learning for BCIs’ discussed formal methods for including subject learning as part of the design loop. The first emphasized that a physical control system standpoint can lend insight into the reasons that some decoding algorithms outperform others [204], and introduced a rigorous definition of BCI usability by linking to ideas from optimal control theory. The second talk presented a unifying framework for considering decoder calibration and subject learning in parallel [205, 206]. Both agreed that an understanding of limitations in learning will be critical for optimizing BCI performance.

• Information in the grasping network. Current state-of-the-art BCI control is just beginning to tackle the control of high-degree-of-freedom robotic hands [27]. The talk ‘Decoding of grasping movements from the primate parietal and frontal cortex’ presented some recent experimental results on the nature of the information conveyed in various parts of the grasping network [20]. The results show a consistent change in the grasp representation across cortical brain areas, and suggest organizing principles by which information might be extracted from each.

• Engineering robust control. The final talks of the session focused on engineering design principles that improve behavioral performance. First, ‘Highrate control-theoretic BMIs’ presented evidence that feedback rates matter, and discussed techniques for enabling nearly continuous feedback control [208]. The final talk, ‘Intracortical Communication BMIs’ discussed recent work transitioning high-performance decoding algorithms from the lab to the clinic [209, 210], and some differences between humans and macaques that might impact robust control. It is currently unknown whether these differences relate to overall differences in brain architecture between the two species, or to the particulars of an individual’s disease state. This will be an important point for the design of next-generation decoding algorithms.

The clinical efficacy of BCI technologies depends in large part on improvements in the algorithms that decode the user’s motor intent. Future work combining basic scientific studies of the motor system with engineering improvements in signal extraction will be required before the ultimate performance limits of these devices are reached.

Combining BCI with non-invasive brain stimulation techniques

Organizers: Aureli Soria-Frisch (Starlab Barcelona SLU) and Laura Dubreuil (Neuroelectrics Inc. USA).

Presenters: Giulio Ruffini (Neuroelectrics); Ricardo Chavarriaga (EPFL); Theodore Zanto (UCSF); and Surjo Soekadar (EKUT).

The combination of transcranial Current Stimulation (tCS) techniques with BMI is a topic of increasing interest for neurorehabilitation of motor and cognitive functions. While early works with tCS reported on a mere improvement of BCI feature values without overall increase in communication performance [211], various later studies prove that adding tCS techniques can improve general BMI performance and therefore its value as a rehabilitation tool [32, 212, 213]. Hence, neuroplastic changes strongly related to re-learning of impaired functions in rehabilitation become facilitated by non-invasive multi-level electrotherapy such as tCS when applied during BMI-based rehabilitation [32]. In addition, tCS can selectively enhance the activity of physiologically targeted brain areas, an interesting property for BCIs. However, use of tCS involves parameters including stimulation type, site, and session schedule [214] that have to be adapted in each therapeutic intervention to optimize the rehabilitation outcome. This workshop covered the main insights on the combination of tCS and BCI technologies for rehabilitation and cognitive improvement, including tCS principles, the combination of EEG signals with tCS, BCI–tCS rehabilitation protocols, and an overview of the benefits, disadvantages and difficulties of the BCI–tCS combination.

Ruffini’s presentation on the main principles of electrical brain stimulation going from the neuron to the brain level can be downloaded from http://wiki.neuroelectrics.com/index.php/Learning-materials (July 2016). Presentation of the main principles of tCS action [214] ensured a common basis among workshop attendees to explore more advanced topics in tCS like those related to its rehabilitation applications. In some work [211], a single-session use of tDCS appears to modulate individual neural correlates, but does not lead to a significant increase in the overall BCI communication performance. As discussed after the talk, possible causes of this might be the single intervention session and the simplicity of the particular protocol.

More advanced protocols have been used in later works with more successful outcomes. For example, the combination of transcranial alternating current stimulation (tACS) and EEG in cognitive rehabilitation produced improvements through neuroplastic changes in different neural regions. Specific brain rhythms are affected by tACS, mainly in the stimulation frequency, and these effects may be modified through the adjustment of the stimulation parameters [213]. One of the open issues remains the removal of the EEG artifact produced by the tACS. This issue has evolved from simple protocols to modern advanced approaches, in which the usage of amplitude-modulated tACS avoids the aforementioned artifact [215]. Advanced principles and methods for adjusting stimulation protocols include methodologies for targeting particular brain regions through the application of optimized multi-site montages, which improve the tCS focality [216].

All invited talks were followed by vivid discussions. The participants underscored the workshop’s scientific depth and its practical value for researchers, clinicians, and rehabilitators. As a summary, the main message for attendees was that tCS protocols are more complex than the traditional tDCS at motor cortex and are worth exploring for the improvement of BCI, both in communication and rehabilitation applications. This involves the selection of alternative stimulation types, multi-site montages optimized to target particular areas, and correct parameterization of protocols. We expect the workshop to boost the design and, more important, the efficacy of studies and therapeutic interventions based on innovative tCS techniques in the near future. Long-term effects of tCS and its combination with BCI is an open research question. This question and the improvement of montages and protocols in general are the most challenging research questions for the future.

Haptic guidelines for BCI research

Organizer: Mounia Ziat

Presenters: Mounia Ziat (Northern Michigan University); Jan van Erp (TNO); Manuel Cruz and Darrel Rohit Deo (Stanford University); and Samir Menon (Stanford University).

Although some BCI studies have used the haptic modality [217–219], haptic-BCI research has received relatively little attention from the BCI community in comparison to visual or auditory BCI. This workshop was designed to provide guidelines for BCI researchers to include haptic modalities in their research by exposing them to the possibilities of haptic technology from simple cutaneous actuators to complex exoskeleton robots. After background presentations on the psychology and physiology of touch [220], the attendees had the opportunity to hear experts in the field present an overview of haptics actuation, state-of-the-art of haptics in BCI and neuroscience research, and live-demonstrations of some haptic devices. Two devices were brought to demonstrate the potential of haptic feedback for motor imagery and P300. One device uses the neck [221, 222] as a potential site for a tactile P300 speller paradigm. Several neurological diseases and conditions such as ALS or LIS are often accompanied with ocular motility disorders [223] that can in some cases make the usage of a visual P300 speller tedious if not impossible. Using a free channel such as touch could be beneficial. Placement of haptics on the neck or the head could not only provide directional information but would still be present as an intact organ even when peripheral nerves had been damaged. Another device provides skin deformation feedback, which can potentially be interpreted as a substitute for proprioception in cases where natural proprioception is distorted or lost, such as in amputation and stroke. Skin deformation feedback provides cutaneous shear forces that tangentially stretch the skin and is often applied to the fingertip due to its high density of mechanoreceptors, which makes it highly attuned to tactile stimulus [224, 225]. It has the ability to communicate direction and magnitude information, which can be used to convey trajectory information of a BCI-controlled prosthesis. This may aid in motor imagery techniques by using skin deformation as a substitute for proprioception to better envision intended movement.

The workshop concluded with a group discussion about possible alternatives and several specific challenges that face researchers who would like to combine haptics with BCI applications. Among these challenges are the cost of the hardware and the intimate nature of touch. For instance, an area such as the neck could be perceived as off-limits for tactile stimulation. Hands-on demonstrations were very well received and helped to understand the current technological limitations and barriers exposed during the workshop. Attendees responded positively to the device around their neck, mentioning that the stimulation is pleasant and similar to a neck massage, rather than being noxious and invasive, as some had expected. Finally, both wearable BCI and haptics technologies need to be discrete and not imposing to be socially acceptable, but also easy to use and connect. In summary, the workshop allowed the attendees to expand their knowledge of BCI to the area of haptics and try alternative technology that could help advance the field.

Out of the lab – acquiring high quality EEG during mobile application

Organizers: Reinhold Scherer (Graz University of Technology); Stefan Debener (University of Oldenburg); Martijn Schreuder (ANT Neuro); and David Ojeda (Mensia Technologies).

‘Get out of the lab and into the real world!’ has become a major aim of BCI research and of EEG research in general. There is a growing interest in using BCI technology for neurorehabilitation (e.g. stroke rehabilitation [16, 226, 227]) or as assistive technology devices (e.g. for persons with cerebral palsy [228]). An aspect of vital importance in this context is signal quality. Recording high-quality data under ambulatory or mobile conditions is highly challenging, since (movement) artifacts, as well as dynamic environments, make it very difficult to analyze the data. Recent advances in EEG hardware [229–231] and software development have pushed the boundary, allowing the acquisition of good-quality signals during movement. However, a vast number of BCI studies do not report on using online artifact removal or detection methods. These are, however, crucial to ensure that BCIs decode signals originating in the brain and not signals resulting from overt behavior or other correlated noise sources.

This workshop was split into two parts. The first part included presentations that introduced a minimum of common terminology, a review of available sensor technologies and some recent innovative hardware and software techniques and technologies. A common approach to remove artifacts is blind source decomposition. Either the EEG time series or the EEG power density spectrum is decomposed (e.g. FORCe, [232] or by using principle component analysis [233]) into source components (SCs) and SCs with certain features are removed. One issue with this approach is that SCs are removed that do not fit the picture of ‘clean EEG’. Thus, the cleaning may become a self-fulfilling prophecy. A very successful approach to detecting artifacts is the use of covariance matrices and Riemannian geometry [234, 235]. Innovative hardware solutions include artifact suppressing mobile amplifiers [236] and flex-printed, unobtrusive electrodes such as the cEEGrid array [167], which can be combined with a wireless amplifier (SMARTING; http://www.mbraintrina.com) and conventional mobile phones. First results and comparison with standard electrodes are very promising.

The second part of the workshop was focused on gaining hands-on experience with hardware and software tools. Several mobile EEG systems from different manufacturers were available, providing hands-on experience with several applications, including smartphone-based readiness potentials during free walking, unobtrusive long-term EEG acquisition with cEEGrid electrodes, and wireless motor imagery neurofeedback.

The workshop concluded with a group discussion on hands-on experiences, open questions, and next steps. Next steps include data and code sharing, as well as ‘crowd-scoring’. Crowd-scoring requires a web-based platform that allows experts from all around the world to score and annotate EEG signals and share their expertise. This will provide a sound basis to evaluate signal processing methods. Recently, promising solutions for EEG data sharing (hppt://www.eegstudy.org) and standardized event labeling (http://www.hedtags.org) have been developed. To conclude, things are moving forward, and novel and innovative solutions for recording high quality EEG during mobile applications are being developed.

Translational and commercial issues

A framework for considering the voice of the users of BCI rehabilitation devices

Organizer: Denise Taylor

Presenters: Denise Taylor (Auckland University of Technology); Nada Signal (Auckland University of Technology); Mads Jochumsen (Aalborg University); Sylvain Cremoux (LAMIH (CNRS-UVHC)); and Imran Niazi (New Zealand College of Chiropractic).

Providing effective rehabilitation is a lynchpin in achieving independence for people with disabilities. With the significant investment in the development of medical technologies for rehabilitation, it behooves us to apply an understanding of rehabilitation processes alongside an understanding of the patient perspective in the design and development of such technologies. This workshop highlighted how clinician and user perspectives can and should influence the design and implementation of BCI devices for neurological rehabilitation. Discussions with engineers and clinicians revealed a gap between (1) the engineers’ understanding of disability, rehabilitation, and patient experiences, and (2) the clinicians’ understanding of the possibilities of engineering. This workshop challenged both engineers and clinicians to extend their understandings and consider how this might influence design and implementation of medical devices. Fundamental principles of rehabilitation, including motor learning, experience-dependent neural plasticity, intensity, dose, repetition, salience, specificity and the use of feedback were discussed. How these principles were expressed in a traditional rehabilitation setting was discussed, along with how they could be expressed in conjunction with a BCI rehabilitation device. It is clear that misconceptions of the principles of rehabilitation negatively influence device design. For example, engineers may fail to understand the complexities of movement repetition, which is not simply repetition of the same movement over and over again.

The importance of understanding the lived experience of users was highlighted. The International Classification of Functioning (ICF), Disability and Health Framework was presented, with a discussion around how this tool can help to consider the wider experiences of people with disabilities (http://who.int/classifications/icf/en/). Within the ICF framework, functioning and disability are conceived as multi-dimensional concepts relating to the body functions and structures of people, the activities they do, the areas of life in which they participate, and the factors in their environment which affect these experiences. The underlying model is one where contextual (environmental and personal) factors interact with the individual with a health condition and determine the level and extent of that individual’s functioning.

A third strand of the workshop addressed the requirements for clinical trial-level evidence in rehabilitation devices. The importance of well-designed, appropriately powered trials that adhered to the highest level of quality control was emphasized. It is at this level of evidence that clinicians determine the effectiveness or otherwise of an intervention. The use of CONSORT guidelines for randomized control trials were discussed (http://consort-statement.org/). The workshop concluded that a shared understanding was needed to facilitate the design of effective BCI devices for rehabilitation that will be implemented in rehabilitation practice and in the daily lives of those with disabilities. This shared understanding can be built through collaborations, interactions in workshops such as this, cross-training between engineers and clinicians, use of tools such as the ICF, and interactions with people with disabilities.

Pathways to effective BCI communication and computer interaction for people with disability

Organizer: John Simeral

Presenters: Shangkai Gao (Tsinghua University); Theresa Vaughan (National Center for Adaptive Neurotechnologies); Beata Jarosiewicz (Brown University); Frank Willett (Case Western Reserve University); Chethan Pandarinath (Stanford University); and Vikash Gilj (University of California, San Diego).

Individuals with paralysis or communication disorders resulting from ALS, brainstem stroke, high cervical spinal cord injury, or other neurological impairments may benefit from BCI-enabled human–computer interaction (HCI) and augmentative and alternative communication (AAC). BCI platforms relying on different human brain signals recorded through EEG, ECoG, or intracortical electrodes can provide command signals for these assistive devices [237, 238]. Other signal acquisition technologies such as fMRI [239, 240] and NIRS [241, 242] are possible, but are less well-characterized in individuals with motor disability.

This workshop promoted collaborative communication among investigators spanning traditionally disparate BCI domains to share best practices for clinical translational research, strategies for improving BCI performance, understanding user needs and expectations, and effective design and deployment of unattended in-home BCI systems for individuals with severe motor disability. Despite actual and inferred differences among BCI technologies and methods, investigators share common motivation and challenges in translation from investigational BCI systems to independent home use [243].

Maximizing online communication rates is a priority across domains. BCI communication performance has improved for BCIs using evoked potential SSVEP, P300, ECoG, and intracortical 2-D cursor-based spellers. An SSVEP BCI with a novel ‘joint frequency-phase modulation’ to detect evoked potentials in EEG [244] achieved 60 selections-per-minute in non-invasive online spelling tasks, surpassing rates previously reported for other BCIs. Although achieved by healthy subjects, this communication rate provides a challenging and motivating benchmark. Participants with ALS using an intracortical BCI at home have achieved average point-and-click virtual typing rates up to 30 correct selections per minute [245]. Further iBCI performance gains may result from incorporating neural dynamics models developed in nonhuman primates [247] and confirmed in two people with ALS [248] into novel kinematic decoders. Performance is also sensitive to classification and decoding parameters that vary across users. A novel feedback simulator for cursor iBCIs optimizes critical kinematic parameters by modeling control characteristics of the user in addition to system noise and decoder behavior [248].

Translating BCIs for in-home unattended use is another common priority. The EEG P300 research community has broad experience with home use of BCI by disabled users in the USA and Europe [249–252]. This includes the EU BackHome project to develop, install, and evaluate EEG P300 BCIs in users’ homes [252] and the Wadsworth Center’s more than 60 evaluations and dozens of installations (not all ALS) of P300 BCIs ([250]; unpublished data). All BrainGate intracortical BCI research sessions are also performed in participants’ residences. An automated iBCI self-calibration method enabled trial participants with tetraplegia to achieve on-screen cursor-based letter selection rates of 15 selections-per-minute spanning more than a month, with no explicit recalibration required [251]. These diverse multi-patient, at-home BCI experiences provide benefit across BCI research domains.

Ultimately, BCI effectiveness must be evaluated relative to user priorities and available alternative assistive technologies. A recent international survey evaluated interest and acceptance regarding BCI technologies spanning EEG, ECoG and iBCI (and a commercial eye tracker) among 156 prospective users with quadriplegia from spinal cord injury [254]. Irrespective of the underlying sensor technology, survey respondents were concerned with esthetics and the burdens of daily maintenance and technician intervention. Respondents also prioritized autonomous, unobtrusive and safe BCI solutions, suggesting that pathways to effective BCI communication probably involve miniaturized wireless system implementations.

BCI implants: medical, ethical, regulatory and commercial issues

Organizer: Nick Ramsey (University Medical Center Utrecht)

Panel Participants: Jon Wolpaw (Wadsworth Center); Jane E. Huggins (University of Michigan); Leigh Hochberg (Brown University); Spencer Kellis (California Institute of Technology); Eran Klein (University of Washington); Scott Stanslaski (Medtronic PLC); and Takufumi Yanagisawa (Osaka University).

This workshop was designed around discussions with an expert panel on several linked topics, with active inclusion of the workshop participants. Discussions were introduced by 2–3 panel members briefly stating their stance on one of the three topics described below, and a lively discussion among audience and panel members ensued.

The first topic addressed health issues involved in implanting BCI systems in patients. The discussion included issues regarding temporary implant for research and permanent implant for restoration of communication and motor control (including robot limbs). Overall, the surgical risks were regarded as low, given modern surgical practices. Issues include risk assessment and approaches to minimize them. Accurate communication of risks to potential participants was acknowledged as an area of importance, since the understanding of the risks of implant can be both under- and over-estimated. The general consensus was that temporary implant research is needed for BCI to move forward, but that care must be taken to minimize the actual risks involved, and that those risks should be clearly explained to potential participants.

The second topic focused on when surgery is justified for acquiring a BCI. This topic dealt with the arguments against and in favor of implanting. The questions addressed included when a person qualifies for an implant and what particular clinical populations are the target populations. The general notion was that the severity of a disability determines whether a non-invasive solution may suffice, or whether an implant would better meet the needs. Implants were considered most appropriate for those with the greatest impairment. Neither the panel nor the audience voiced principal objections to implants. However, it is clear that for this emerging field, regulations and guidelines need to be established to protect beneficiaries of the multiple technologies being developed. Regarding the needs that can be met with implants and how they differ from non-invasive solutions, the benefits of invisibility of the system and permanent availability were emphasized. Permanent availability is most important for those who are severely paralyzed and cannot setup a system themselves, but who have an ongoing need for a means of communicating and calling a caregiver.

Finally, the panel discussed what research is needed to bring implants to market. Clearly, research is required in terms of medical efficacy, clinical trials, and comparisons between implants [243]. Although it is early for multi-center research, it is highly important to conduct such endeavors once the opportunity arises. Issues that need to be resolved include required numbers of patients, duration of inclusion, and clinical implant indications. Activities relevant for promoting implantable BCI also concern regulatory and reimbursement hurdles. Engaging industry and identifying market parameters on which to base investments in implantable devices will be crucial for clinical application. BCI researchers will have a significant role in defining these issues.

BCI implants are still at an early stage and the path towards clinical application is not yet clear. However, lessons may be drawn from the evolutionary development of the cochlear implant [255], which started with single electrode implants suitable only for sensing loud noises and developed into current systems enabling telephone conversations. The path-to-market for cochlear implants took some 25 years from first implants in the early 1960s, to commercial production in the late 1980s. BCI implantation should likewise begin by providing immediate practical benefit and evolving toward more complex applications.

Does BCI mean business for augmentative and alternative communication?

Organizers: Femke Nijboer (Leiden University); Melanie Fried-Oken (Oregon Health & Science University); Theresa M. Vaughan (National Center for Adaptive Neurotechnologies); and Douglas K. R. Robinson (Université Paris-Est Marne-la-Vallée).

BCI researchers are beginning to translate BCI technology for augmentative and alternative communication (AAC). Stakeholders, including experts from neuroscience, engineering, computer science, and AAC; current and potential consumers (e.g. people with locked-in syndrome, progressive neuromuscular disorders and spinal cord injury) and their families; industry and the general public, have participated in structured conversations about how to bring BCI and AAC to market and to society over the past five years [256]. This workshop asked BCI researchers to explore the challenges, opportunities, and bottlenecks for (1) design criteria for improved BCI usage and competitiveness with alternatives; (2) technology transfer routes that are strong and sustainable; and (3) improved methods for increasing the number of children and adults with complex communication needs who can use a BCI.

The workshop took a multi-stakeholder perspective as an entrance point to address these challenges, exploring multi-dimensional factors that shape the technology transfer process beyond technical dimensions. The 30 participants formed three discussion groups, each addressing separate topics. While a detailed write up of the workshop is in process, the following provides a snapshot of each group discussion.

1. The real cost of developing and regulating complex implanted devices: while binary switches may provide low-hanging fruit early in development, the cost of complex devices may limit those who can afford to develop, demonstrate and deploy them.

2. The minimum level of functionality that would mean (successful) business: this led to a discussion about the broader BCI market and the role for non-medical BCI demonstrators and products. The group concluded that much can be done with lower functionality as evidenced by an emerging supply chain of companies providing such devices [257].

3. The most desirable BCI functions for AAC: here, the group looked at the current state-of-the-art, described limitations based on the experience of the multi-disciplinary team of participants and practitioners, looked at what would be the most desirable functions, and the timeline that these would or could be available. Specific discussions about the uptake of non-invasive BCIs with high-functionality were explored.

Overall, the groups suggested a list of issues most likely to influence effective technology transfer: costs; standards and design for manufacturing; business aspects (e.g. venture capital, development, and marketing); regulations and norms; user values and establishing consensus measures across stakeholders; differences in healthcare systems; and the need to create a healthy innovation landscape for both invasive and non-invasive BCIs that deliver societal and economic value.

Technological implant developments

Organizer: Erik Aarnoutse

Presenters: Erik Aarnoutse (Brain Center Rudolf Magnus, University Medical Center Utrecht); Scott Stanslaski (Medtronic Neuromodulation); Masayuki Hirata (Osaka University); Fabien Sauter-Starace (CEA-LETI Clinatec); and Arto Nurmikko (Brown University).

The advantages of using implant technology for BCI are clear (high-quality signal, permanent availability, minimal expertise for operation, self-agency, and esthetics), but is accompanied by strict requirements of technology, ethics, and regulation.

The field of BCI implants combines research from material science, electrical engineering, neuroscience, neurosurgery, but also regulatory affairs. These different levels of research were discussed by speakers from different perspectives (academia, industry) and different parts of the world (Europe, USA, Japan) in presentations on four state-of-the-art, fully implanted BCIs: Activa PC+S [258]; WIMAGINE [178, 259]; W-HERBS [260, 261]; and the Brain-Implantable Chip (BIC) [262, 263]. Insights from the first study bringing BCI implants to the home, the Utrecht Neuroprosthesis (UNP), gave a perspective from the technology-transfer point of view.

Human research is being done with Activa PC+S [258], a deep brain neurostimulator with sensing capability, which was developed by Medtronic as a tool that gives insight into the chronic nature of deep brain disorders. The device can stream signals out so that closed loop systems and algorithms can be developed. The first major success in this area has been a locked-in patient using the device for communication at home in the UNP study [264].

Lessons on development constraints of implants (safety, biocompatibility, operability, manufacturing and quality management) have been learned from the WIMAGINER^® experience [178, 259]. A full ECoG-based BCI platform from implantable device to a four-limb exoskeleton was developed to prove the feasibility of controlling complex effectors. The clinical research protocol is now approved in France.

Another study with humans has been done with a wired ECoG system (HERBS) [260, 261], with patient-specific micro-ECoG electrodes using 3-D printing and development of the wireless, fully implanted W-HERBS. Animal tests evaluated longevity of the system over six months.

Developed in an academic environment, the BIC [262, 263] enables a broadband neural recording from a multi-electrode array in a non-human primate brain by converting these signals to a digital stream of infrared light pulses for transmission through the skin. Many of the lessons learned during development involve management of quality systems for design and manufacturing, which is not within the academic tradition.

Erik Aarnoutse brought the perspective of bringing this technology to the home of the user in the UNP study. Many regulatory efforts where needed for ethical approval, e.g. continued use of the system after the clinical trial is finished, which every patient can demand, requires a post-marketing surveillance plan. Also, minimal invasive surgery, essential for acceptance of the technology by users and clinicians, required validation of prelocalization with fMRI [265]. The first results are very promising.

An important conclusion of this workshop was that the technological, but especially regulatory hurdles of bringing BCI implants to the market are large. The W-HERBS and BIC are not approved for human investigations yet; the Activa PC+S and WIMAGINE are approved only for investigational use. None of the mentioned implants are approved for commercial sale. The UNP study shows, however, that it is feasible to bring the technology of BCI implants to the home of the user. A larger public acceptance will provide impetus for bringing implantable BCIs to the market, but the combined expertise of industry and academia is necessary to accomplish this.

What’s wrong with us? Roadblocks and pitfalls in designing BMI applications

Organizers: Ricardo Chavarriaga (EPFL)

Presenters: Sonja Kleih (University of Würzburg); Fabien Lotte (Inria Bordeaux); and Reinhold Scherer (Graz University of Technology).

Despite impressive progress in BMI research, including tests involving their intended end-users, effective translation from proof-of-concept prototypes into reliable applications remains elusive. Indeed, current BMI systems, using either invasive or non-invasive techniques, cannot yet be used by end-users and their caregivers with the same level of independence as other assistive technology, but require frequent (e.g. weekly) support from BMI experts.

Multiple factors create roadblocks that hinder this transition. This workshop identified pitfalls in the BMI development process, and defined concrete actions to reduce their impact. A brief summary of the discussions is presented here, while a more detailed report can be found elsewhere in this issue [266].

Many research efforts ignore the fact that a BMI is, by definition, a closed loop system where human and machine constantly interact. A common pitfall is to devote research efforts solely to optimizing the decoding engine instead of simultaneously studying and enhancing the interaction with the human. Improving training paradigms and feedback is essential to help the user find appropriate strategies to achieve BMI control [267]. Moreover, user requirements and preferences must be considered from early stages of the design process [268].

Another pitfall is the signal vulnerability to artifact contamination, especially with non-invasive recording techniques. Multiple methods have been proposed to remove artifacts, especially those due to eye movements [269]. However, some end-users (e.g. people with cerebral palsy) may exhibit other types of artifacts that are more challenging to remove [270]. Similarly, real-life applications are generally noisier than laboratory conditions [271]. The creation of repositories archiving datasets from different types of end-users could advance the state-of-the-art in robust methods for real-life BCI systems. A crowd-sourcing approach for labeling signal contamination may provide a valuable asset to benchmark approaches for artifact management.

Another issue concerns the metrics used to estimate the BMI performance [272]. Performance assessment should go beyond machine learning/classification metrics, and include efficiency/effectiveness metrics in the human-computer interaction sense [273]. Strong evidence shows that human factors influence performance. Thus, researchers should explicitly assess these factors (cognitive workload, sense of agency, among others) in the experimental design [274, 275].

Finally, improvements are needed in the way BMI research studies are designed and reported. Experiments are often conducted in small populations, without involving intended end-users, or lack appropriate controls to identify experimental confounds. Further, studies often use inappropriate statistical tests [276–279]. These design flaws limit the ability to generalize conclusions. As in other fields, publications must provide complete method descriptions to allow replication of experiments. Similarly, negative results must also be reported to enable the research community to learn from its errors [280]. We propose the preparation of guidelines for good reporting practices that the BCI Society can endorse and encourage authors and reviewers to follow. While some guidelines from related fields can be adopted (e.g. [278, 281–283]), specific aspects inherent to BMI systems should be considered as well (e.g. procedures for decoder calibration, evaluation performance on small populations, among others). Similarly, we propose periodic special issues of peer-reviewed journals dedicated to reporting negative BCI results, obtained with rigorous and unbiased studies. These steps will advance the field towards more robust and reliable systems, enabling the leap from research laboratories to real world applications.

Conclusion

As represented in these workshops, BCIs are being used in an ever-expanding variety of application areas. The decades of BCI research and development are bearing fruit as a number of applications approach commercial readiness, requiring an increased focus on regulatory and usability aspects of system design. At the same time, the expansion of BCIs into new application areas is highlighting the importance for BCI advancement of increased understanding of the complexities, variability, and underlying mechanisms of the brain signals being interpreted. Thus, despite the increasing maturity of the field, BCI research remains a collaborative endeavor. No one area of training can capture the entirety of the knowledge needed for improving BCI performance. Collaboration is still needed between engineers, clinicians, and basic scientists. Thus, students seeking to enter BCI research should not expect to become an expert in all areas affecting BCI function, but should acquire sufficient exposure to many areas to facilitate productive collaborations while maximizing their expertise in one or two areas of training in which they can make individual contributions. The value of big data collaborations involving data sharing and pooled expert analysis of data was discussed in several workshops as an important step to understanding the complexities of brain activity with regard to BCI performance. The workshops of the BCI Meeting Series provide a venue to initiate such efforts and an opportunity to build collaborations to advance BCI research and development to support the emergence of practical BCI products for deployment as clinical treatment options or tools for personal expression for people, both with and without physical impairments.