The combination of brain-computer interfaces and artificial intelligence: applications and challenges

Abstract

Brain-computer interfaces (BCIs) have shown great prospects as real-time bidirectional links between living brains and actuators. Artificial intelligence (AI), which can advance the analysis and decoding of neural activity, has turbocharged the field of BCIs. Over the past decade, a wide range of BCI applications with AI assistance have emerged. These “smart” BCIs including motor and sensory BCIs have shown notable clinical success, improved the quality of paralyzed patients’ lives, expanded the athletic ability of common people and accelerated the evolution of robots and neurophysiological discoveries. However, despite technological improvements, challenges remain with regard to the long training periods, real-time feedback, and monitoring of BCIs. In this article, the authors review the current state of AI as applied to BCIs and describe advances in BCI applications, their challenges and where they could be headed in the future.

Introduction

With the large explosion in technology, the border between humans and machines has begun to narrow. Our spectacular science fictions describing “mind control” have gradually come true with the help of machines. The frontiers of these new techniques are brain-computer interfaces (BCIs) and artificial intelligence (AI). Experimental paradigms for BCIs and AI were usually developed and applied independently from each other. However, scientists now prefer to combine BCIs and AI, which makes it possible to efficiently use the brain’s electric signals to maneuver external devices (1).

For severely disabled people, the development of BCIs could be the most important technological breakthrough in decades (2). BCIs, which represent technologies designed to communicate with the central nervous system, and neural sensory organs can provide a muscle independent communication channel for people with neurodegenerative diseases, such as amyotrophic lateral sclerosis, or acquired brain injuries (3). The history of BCIs is intimately related to the effort of developing new electrophysiological techniques to record extracellular electrical activity, which is generated by differences in electric potential carried by ions across the membranes of each neuron (4). The methods of detecting different types of brain signals can be classified as invasive or noninvasive (5). Invasive recording systems include electrocorticography (ECoG), microelectrode arrays (MEAs), and so on (6). Noninvasive BCIs including electroencephalography (EEG), magnetoencephalography, functional magnetic resonance imaging (fMRI), and functional near-infrared spectroscopy, do not carry risks of tissue damage and can be implemented rather easily (7). With the help of these electrophysiological techniques, BCIs can be quickly applied to ‘read’ the brain to record its activity and decode its meaning and to ‘write’ to the brain to manipulate activity in specific regions and affect their function (8). However, the development of BCIs has limitations. Although we have obtained much information from multiple extracellular electrodes, this large amount of information cannot be transferred efficiently (9). Neuroscientists cannot unambiguously discern a person’s intentions from the background electrical activity recorded in the brain and match it to the actions of robotic arm (10). The reason for this limitation is that the neural correlates of psychological phenomena are inexact and poorly understood (11). Fortunately, recent advances in AI methodologies have made great strides, verifying that AI outperforms humans in decoding and encoding neural signals (12). This provides AI a great opportunity to be to an ideal helper in processing signals from the brain before they reach the prostheses.

AI is a set of general approaches that uses a computer to model intelligent behavior with minimal human intervention, eventually matches and even surpasses human performance in task-specific applications (13). When AI works within BCIs, internal parameters are provided to the algorithms constantly, such as pulse durations and amplitudes, stimulation frequencies, energy consumption by the device, stimulation or recording densities, and electrical properties of the neural tissues (14). After receiving the information, AI algorithms can identify useful parts and logic in the data and then simultaneously produce the desired functional outcomes (15). Although these studies remain largely in the preclinical research domain, the continued development may highlight clinically actionable changes in BCIs.

At the dawn of technological transformation, a tendency to combine BCIs and AI has also attracted our attention. Here, we review current applications with a focus on the state of BCIs, the role that AI plays and future directions of BCIs based on AI (Figure 1).

Figure 1.

Schematic description of BCIs based on AI. With the help of AI to process signals, the applications of BCIs have been extended greatly, including cursor control, auditory sensation, limb control, spelling devices, somatic sensation, and visual prosthesis. The circuit can be described as follows. First, micro-electrodes detect signals from the human cerebral cortex and send them to the AI. Second, the AI takes charge of signal processing, which includes feature extraction and classification. Third, the processed signals are output to achieve the abovementioned functions. Finally, feedback is sent to the human cortex to adjust the function. BCIs, brain-computer interfaces; AI, artificial intelligence.

Applications of BCIs based on AI

Applications in cursor control

Early studies focused on the control of personal computer mouse cursors for paralyzed patients through BCIs with high feasibility (16,17). The fundamental components of a cursor control BCI include a sensor to record neural signals, a decoder to interpret movement intentions, and a computer cursor that interacts with the external environment (18).

A pioneering study published in 2000 by Kennedy and colleagues first demonstrated that an invasive BCI device with a special electrode implanted into the outer layers of the human neocortex could be decoded to drive a cursor on a computer monitor (19). Studies in nonhuman primates have shown that cursor control BCIs can achieve multidimensional neural integration with two or more degrees of freedom (20). One-dimensional (1D) cursor control is achievable using EEG with event-related desynchronization, using decision trees sequentially string selections together to make a final choice (21,22). Two-dimensional (2D) cursor control can be achieved using techniques such as fMRI or EEG (23). Recent work in 2017 reported the development of a high-performance, invasive BCI for communication, using two algorithms to translate signals into point-and-click commands: the ReFIT Kalman Filter for continuous two-dimensional cursor control and a Hidden Markov Model-based state classifier for clicking (24). By providing at least 2D neural control of the computer cursor and a parallel selection method such as a click, the user can not only type in self-selected characters but also use native computer application with the cursors, just like a healthy person can with a mouse (25). Moreover, this BCI improved the communication rate to 32 letters/min, making cursor control more efficient (26).

On the basis of scalp EEG, ECoG and synchronous evoked potentials, many BCI systems for cursor control have been developed, such as the P300 matrix speller and the rapid serial visual presentation method (27). The Brain Gate group conducted the first human trial of a motor BCI starting in June 2004, which recorded signals from a Blackrock 96-channel MEA implanted in the arm area of M1 in a patient with tetraplegia following cervical spinal cord injury (28). They achieved two-dimensional movement of a cursor on a screen and subsequently used this “neural cursor” to direct the movement of a robotic limb (28).

Applications in neuroprosthetics and limb rehabilitation

The task complexity of BCI studies rapidly has advanced from 2D and 3D control of a cursor on a computer screen (29) to the control of more natural behaviors, such as reaching and grasping (30), self-feeding (31), and bimanual arm movements (32). To observe a person with tetraplegia picking up a cup of coffee using a BCI-controlled robotic arm is spectacular. This rapidly advancing technology works by implanting an array of electrodes either on or in a person’s motor cortex, which is the brain region involved in planning and executing movements (33). Then the activity of the brain is recorded while the individual engages in cognitive tasks, such as imagining that they are moving their hand, and is used to command a robotic limb (33).

In addition, many therapeutic strategies have been developed to help stroke patients regain some function in the affected limb. However, approximately 80% of all stroke survivors with upper limb motor deficits do not benefit from these approaches (34). The method of lower limb rehabilitation using BCIs has been explored recently (35). The overall principle is thought to be that closing the loop between the cortical activity of motor intention and movement, thereby producing afferent feedback activity, might restore functional corticospinal and corticomuscular connections (36).

Early researchers of neuroprosthetics implanted electrodes into a monkey’s brain and measured the signals from the electrodes. The monkey used a joystick to control a robotic arm. Eventually, the researchers changed the controls so that the robotic arm was being controlled only by the signals coming from the electrodes, not the joystick (37). Invasive and noninvasive BCI systems have also been used to enable neural control of a robotic arm in humans (38,39). The use of implantable electrodes allows patients to control movements with several degrees of freedom, enabling them to make more complex and functional movements. However, approaches that use noninvasive systems provide limited control, and most complex movements rely on the AI of the robot (40). To classify motor-related signals specifically for BCI applications, Nurse and colleagues developed a generalized approach that takes advantage of a stochastic machine-learning method (41). Importantly, their classifier does not need to rely on the use of extensive a priori data to train the BCI. Their algorithms outperformed other methods on the Berlin BMI IV 2008 dataset and demonstrated high levels of classification accuracy when tested on datasets derived from EEG signals (41).

In addition, studies have shown that the use of BCIs can be effective in chronic stroke motor rehabilitation and have demonstrated that stroke patients without residual movement benefit from BCIs for their cortical and subcortical reorganization including functional and structural connectivity (42). The learning of neuroprosthetic control has been shown to reshape cortical networks and trigger large-scale modifications of the cortical network, even in perilesional areas (43). Machine learning algorithms can make more gradual decoder modifications (44) and produce a robust neuroprosthetic performance that can be maintained despite nonstationary neural inputs and changes in context (45). The development of AI and computer deep learning will provide a new way to decode neural signals with extremely fast, never before-seen speeds.

Applications in somatosensation

For treating patients with paralysis, movements are highly dependent on somatosensory feedback, especially proprioceptive and tactile feedback (46). Mechanoreceptive afferent signals in our skin convey information about the location of contacts (47) as well as the forces exerted on the skin when we grasp an object (48). Loss of proprioception largely eliminates the ability to plan the dynamics of limb movements (49). Given the importance of somatosensation, the development of bidirectional BCIs is essential. Activation of sensors on the prosthesis could be transferred to activate neurons with the corresponding receptive fields (36). We can explore the link between the pressure exerted on objects by the prosthesis and the appropriate magnitude of intracortical microstimulation (ICMS) pulses more efficiently with the help of AI (50). Random forests in noninvasive sensorimotor rhythm BCIs have been shown to be practical and convenient nonlinear classifiers (51). In addition, somatosensory evoked potential-based BCIs have been extensively investigated using the Fukunaga-Koontz transform-based feature extraction method with a performance improvement from 70% to 75% (52).

One approach to restore somatosensation through BCIs is studying relevant natural somatosensory coding with the help of recorded systems such as ICMS and attempting to mimic the nervous stimulation of healthy people (36,53). Tucker Tomlinson and Lee E. Miller observed that a monkey’s perception was altered in a predictable manner by delivering ICMS coincidently with force pulses applied to a monkey’s hand (54). However, the perception was not accurate enough because it is currently impossible to activate large numbers of neurons individually and independently (54). More combined technologies are needed before we can achieve restoration of the somatosensation of humans with BCIs.

Applications in auditory sensation

The most common and oldest way to use a BCI is a cochlear implant (55). The cochlear implant is a successful example of an afferent interface, used to restore hearing in over 200,000 patients worldwide according to the U.S. Food and Drug Administration since December 2010 (https://www.nidcd.nih.gov/about/strategic-plan/2012-2016/science-capsule-cochlear-implants). A cochlear implant bypasses the nonfunctioning part of the ear, processes sound waves into electric signals and directly stimulates the sensory epithelium of the basilar membrane to simulate auditory stimuli (56). These implants may be placed into the cochlear nerve or pons junction when the auditory nerve has been damaged; the cochlear nucleus, such as the lateral foramen of Luschka; or into the inferior colliculus, a part of the distal sensory circuitry (57). A better resolution of sensory input is achieved by providing it to the cortex rather than the auditory nerve (57).

However, even if cochlear implantation enables some children to have restored hearing and attain age-appropriate speech development after sensory loss, the outcomes are highly variable between patients and are difficult to predict (58). A computational model combined with a complete finite element model and synthetic structures has been designed to provide the missing information linking each of the surgical variables to their effect on the excitation of the nerves based on µ computed tomography images. The metric for the usefulness of the stimulation protocol was calculated and used to rerun the simulations with better parameters (59). A random-forest regression model using clinical data (e.g., duration of deafness and age at surgery) was applied for predicting word recognition scores in postlingually deaf adults with a high accuracy of 95.2% (60). Presurgical neural morphological data obtained from fMRI have been used in linear multivariate ranking support vector machines for building and validating predictive models of speech-perception improvement after surgery (61). The findings suggest that neural systems that are unaffected by auditory deprivation best predicted postsurgical speech-perception outcomes (61).

Applications in speech synthesizers

Patients with tetraplegia and anarthria can communicate in real-time with the help of neural point-and-click control derived from intracortical neural activity (62). With the development and proliferation of mobile communication equipment such as mobile phones, tablet, and other touch-screen devices, researchers have developed many kinds of virtual keyboards with more efficient text entry capabilities, including DASHER, which is driven by gestures and has been tested as a BCI communication equipment based on one-dimensional EEG control (63). Such devices interpret neural activity that occurs while people silently mouth words and then use this information to generate synthetic speech sounds (64).

Locked-in patients with amyotrophic lateral sclerosis can control the variations in their slow cortical potentials to operate an electronic spelling device at a rate of approximately 2 characters per minute (65). To further improve the communication rate, techniques for decoding speech directly from the cortex should be explored. One study examined the potential to decode words from an ECoG recorded from Wernicke’s area, but the accuracy was modest (66). In a recent study, researchers used a two-stage decoder to accurately reconstruct a speech spectrogram (67). Stage 1 of the decoder is a bidirectional long short-term memory (bLSTM) recurrent neural network, decoding articulatory kinematic features from continuous neural activity recorded from the ventral sensorimotor cortex, superior temporal gyrus, and inferior frontal gyrus. Stage 2 is a separate bLSTM for decoding acoustic features from the decoded articulatory features from stage 1. Then, the audio signal is synthesized from the decoded acoustic features (67). Unlike using a cursor to spell, transferring neural activity into speech directly is more efficient (64). The strategy may be an important next step in realizing speech restoration for patients with paralysis.

However, the speed of communication using this electronic spelling device is much slower than normal speech (64). Alternatively, patients with even minimal muscle control still preferred communicating with more conventional assistive devices such as eye gaze trackers or binary switches that can provide higher communication rates (67). Even so, the two-stage bLSTM-based decoder has been a breakthrough in brain-computer interfacing.

Applications in optical prosthetics

Visual prosthetic development has one of the highest priorities in the biomedical engineering field (68). Complete blindness from retinal degeneration arises from diseases such as Leber’s congenital amaurosis or age-related macular degeneration, which causes dystrophy of photoreceptor cells (69). The procedure of visual prostheses can be described as follows. First, the prosthesis detects light emanated from sources or reflected from surfaces in the physical environment of the implant patient. Second, the light is transduced into an artificial stimulus. Third, the artificial stimulus is delivered to the retina and evokes a response (70). Rudimentary vision can be achieved by converting images into binary pulses of electrical signals and delivering them to the visual cortex (70).

For visual prosthesis design, the electrode array is critical. Prosthesis electrode arrays need to adapt to different optimal stimulus locations, stimulus patterns, and patient disease states (71). To find the best prosthesis electrode array for a patient, BCIs based on AI are indispensable. A new electrical stimulation tuning system constructed with a generalized nonlinear model framework and autoencoder has been developed and was suggested to estimate electrical stimuli equivalent to a given natural visual stimulus when introduced to lateral geniculate nucleus cells (72). A new kind of thin, flexible multielectrode polymer probe and a robotic insertion approach for inserting flexible probes were developed by Neuralink (73). This technology is suitable for visual prosthesis and may provide ideal retinal stimulating methods that have the flexibility to match the curvature of the retina without placing significant mechanical pressure on the retina (74).

However, there are things to note about the development of visual prostheses. To date, sensations created have been in the form of bright spots referred to as phosphenes or visual perception patterns (75). Artificial neural networks which are powerful learning tools that provide a great deal of flexibility and scalability may provide new methods for mimicking the natural visual system.

Discussion and conclusions

The present review highlights current research in the BCI field based on AI, which has grown rapidly over the last 15 years (76-78). The combination of BCIs and AI offers a powerful way to investigate brain function by providing direct knowledge and control of neurons controlling behavior, which will help scientists know more about the human brain and promote developments in rehabilitation medicine (8). One of the biggest advantages machine learning may confer on BCI is the ability to achieve real-time or near-real-time modulation of training parameters and subsequent adjustments in response to active real-time feedback (46). Furthermore, algorithms learn from previous data and guide users towards decisions on the basis of what they have done in the past (79). Patients and healthy subjects alike often show large variability, or even inability, of brain self-regulation for BCI control, known as BCI illiteracy (79). Adaptive machine learning methods can help participants who suffer from BCI illiteracy to gain control of the system, combining supervised techniques and unsupervised adaptation (80).

Despite the reported successes and breakthroughs in this field, there still exist some problems. First, most studies have focused on the recovery of motor ability, and the use of BCIs and AI for cognitive training is still at a very early stage (81). Second, clinical BCI applications are still very limited, and some important issues need to be solved before BCIs could be considered effective systems for rehabilitation in clinical settings. For example, stimulating electrodes with smaller diameters are needed (82). Third, machine learning algorithms learn to analyze data by generating algorithms that can rarely be predicted and comprehended in the real world, which leads to problems of unknown process between a person’s thoughts and the technology acting on their behalf (83).

As technologies that directly integrate the brain with computers become unprecedentedly complex, various ethical and social challenges that merit further examination and discussion will also arise. For instance, some forms of BCIs are likely to be expensive, posing questions of affordability and feasibility for people with severe disabilities to access them as assistive technology (84). In addition, BCIs with a decision-making device into human’s brain with AI software that autonomously adapts its operations raise questions about human autonomy by (85). Brain information as digitally stored neural data can also be exploited by others with sufficient computational power to make inferences about our memory, intentions, conscious and unconscious interests, and emotional reactions (86). Moreover, reports have surfaced about a minority of people who undergo deep-brain stimulation for Parkinson’s disease becoming hypersexual, or developing other impulse-control issues (85,86).

In conclusion, BCI based on AI is a rapidly advancing field of interdisciplinary integration of medicine, neuroscience, and engineering. Although most studies that have evaluated AI applications in BCIs to date have not been vigorously validated for reproducibility and generalizability, the goal of these devices is to improve the level of function and quality of life for people with paralysis, spinal cord injury, amputation, acquired blindness, deafness, memory deficits, and other neurological disorders. The capability to enhance normal motor, sensory or cognitive function is also emerging and will require careful regulation and control. Further technical development of BCIs, clinical trials and regulatory approval will be required before there is widespread introduction of these devices into clinical practice. The development of this technology must trigger a revolution in medicine.

Supplementary

The article’s supplementary files as

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.