Online BCI Typing using Language Model Classifiers by ALS Patients in their Homes

Abstract

The P300 speller is a common brain-computer interface system that can provide a means of communication for patients with amyotrophic lateral sclerosis (ALS). Recent studies have shown that incorporating language information in signal classification can improve system performance, but they have largely been tested on healthy volunteers in a laboratory setting. The goal of this study was to demonstrate the functionality of the P300 speller system with language models when used by ALS patients in their homes. Six ALS patients with functional ratings ranging from two to 28 participated in this study. All subjects had improved offline performance when using a language model and five subjects were able to type at least six characters per minute with over 84% accuracy in online sessions. The results of this study indicate that the improvements in performance using language models in the P300 speller translate into the ALS population, which could help to make it a viable assistive device.

Introduction

The P300 speller is a common brain-computer interface system that can provide a means of communication for “locked-in” patients, such as those with amyotrophic lateral sclerosis (ALS).[1] While this system was initially developed almost 30 years ago, it is not widely used in part because typing speed and accuracy are below those desired by the ALS population.[2] The system utilizes one’s electroencephalogram (EEG) to facilitate language communication by detecting stimulus evoked potentials and converting them into virtual commands in the form of typing. Non-invasive electrodes yield undesirable signal noise, necessitating many stimulus trials and sophisticated computation. This decreases the speed of typing and calls for several system optimization studies. Approaches at system optimization include varying the dimensions of the character matrix,[3–5] optimizing system parameters,[6,7] and employing various signal processing methods.[8–11]

Recent studies have shown that performance using the system can be improved by utilizing knowledge of natural language to improve signal classification,[12] similar to methods used in other language processing domains such as speech recognition.[13] Simple n-gram language models have been the mainstay of BCI studies, beginning with a naïve Bayes model,[14,15] and later using a partially observable Markov decision process [16] and a hidden Markov model.[17,18] However, recently a particle filter (PF) algorithm made possible the use of more complicated language models by eliminating the requirement for sampling over entire state spaces, which was shown to have superior results, yielding a 56% average increase in bit rate over traditional methods in offline trials.[19] This method approximates distributions by projecting samples through a state-space language model based on the observed EEG signals. The system then determines the most likely output by finding the state that attracts the highest number of samples.

While the preliminary results using language models have been promising, but they have largely been tested on healthy volunteers in a laboratory setting. Using healthy subjects is advantageous for validating and testing these systems because they are more accessible than ALS patients; the ALS population is significantly smaller than the healthy population and patients generally have trouble traveling, requiring a prototype system to be mobile enough to transport to a patient’s home. Using healthy subjects also alleviates the research burden on the ALS population as they often are easily fatigued and have severe health concerns so constant experimentation would be both impractical and unethical. Because many ALS patients do not have cognitive deficits,[20] it is often presumed that healthy subjects’ EEG signals reflect those of ALS patients and, therefore, that improvements in healthy subject BCI performance will predict improvements in the ALS population. However, some ALS patients have deficits that can affect BCI performances such as lack of gaze control, blurry vision, and unstable neck muscles hindering the ability to keep the head steady. The patients’ environments can pose additional challenges with medical equipment such as bilevel positive airway pressure (BiPAP) machines and ventilators that can affect electrode connectivity and introduce electrical noise. Moreover, the stimulus evoked responses, including visual evoked signals and the p300 response may not be identical in healthy controls and affected subjects. Because of these differences, it is important to test BCI systems on the target population in order to verify that improvements seen in healthy subjects truly reflect a more effective system for ALS patients.

The goal of this study was to perform initial validation of the methods that have been developed in healthy subjects within the target patient population and to identify issues unique to ALS patients that should be addressed in order to make the P300 speller a viable augmentative and assistive communication (AAC) system. A mobile setup was constructed by loading a desktop and monitors onto a cart which was transported to patient homes. Subjects used a version of the P300 speller with language models [19] that was previously validated on healthy volunteers to type in both online and offline modes. Results were then compared with those previously reported using healthy subjects to demonstrate how the healthy subject results translate into the target population and to identify issues unique to ALS patients that need to be addressed in a final system.

Materials and methods

Subjects

Six patients participated in this study. Five of the subjects (subjects A, B, D, E, and F) were diagnosed with ALS and the sixth (subject C) had brainstem necrosis caused by radiation therapy to treat a tumor. Two of the subjects (D and E) required a second visit because of technical problems with the equipment that prevented recording of EEG signals during the initial visit. All subjects consented to participate and the study was approved by the UCLA institutional review board. None of the subjects had gross cognitive deficits (although formal comprehensive testing was not immediately available) and all used non-BCI means of communication (table 1).

Table 1.

Description of ALS subjects participating in this study, including their function ratings (ALS-FRS) and current communication methods.

Subject	Description	ALS-FRS	Communication Method
A	Subject in on ventilator with only gaze control.	2	Eye tracker
B	Subject is able to walk and speak with difficulty.	24	Talking
C	Subject is able to speak with difficulty, has control of arms, but cannot walk.	18	Talking
D	Subject is able to walk with walker, can speak, but has trouble supporting head while sitting.	28	Talking
E	Subject is able to speak and move head with minimal difficulty, uses a BiPAP mask, has some control of one hand, but cannot move legs.	11	Talking, head mouse
F	Subject is paralyzed from the neck down and has no ability to speak.	8	Head mounted laser

Data Collection

EEG data was recorded using g.tec amplifiers, active EEG electrodes, and an electrode cap (Guger Technologies, Graz, Austria). Signals were sampled at 256 Hz, referenced to the left ear grounded to AFz, and filtered using a band-pass of 0.1–60 Hz and a notch filter from 58–62 Hz. The electrode set consisted of 32 channels placed according to a previously published configuration.[6] The system used a 6 × 6 character grid, row and column flashes, a stimulus onset asynchrony of 125 ms, and famous faces stimuli.[21] The hardware was loaded onto a cart, which was then transported to patients’ homes so they could use it in their home environment. Subjects were moved to the location most comfortable for them to best simulate realistic use of the system (Bed for subject A, kitchen chair for subject B, electric wheelchair for subjects C, D, and E). The application was then displayed on a monitor approximately 1 meter in front of the subject’s face.

Each subject first participated in an initial offline experiment, consisting of three five-minute trials where the subject copied a given 10 character phrase with 10 repetitions per character and without feedback. The data from these training phases was then analyzed using a standard Stepwise linear discriminant analysis (SWLDA) classifier (see below) and, if the classifier was successful in finding classification features in the subject’s signals, a testing phase followed. During the testing phase, subjects were instructed to pick a phrase that they wished to type, which they indicated to investigators using their standard communication method. They then used the system in online mode with the results being displayed in real time. Because of the amount of time required for setup and teaching the patient how to use the system, as well as offline training, fatigue was a concern during the online session. Subjects were told to use the system only as long as they were comfortable, which resulted in different amounts of online data between subjects. For comparison, a set of 10 healthy volunteers with normal or corrected to normal vision between the ages of 20 and 35 underwent the same protocol in a controlled laboratory setting.

BCI2000 was used for data acquisition and online analysis.[22] Offline analysis was performed using MATLAB (version 8.6.0, MathWorks, Inc, Natick, MA).

Classifier

SWLDA is a classification algorithm that uses least squares regression to select a set of signal features.[23] It iteratively adds significant features and removes the least significant features until either the target number of features is reached or a state where no features are added or removed is achieved.[11] The dot product of the feature vector and user’s EEG signal then determines a score for a given stimulus response. Using the score distributions for attended and non-attended signals, the likelihood of a response resulting from an attended signal can be determined.[15]

The PF method combines these likelihood probabilities with prior knowledge about language structure to decide the most probable character given the observed signal. This is accomplished by estimating the probability distribution over possible outputs,[19] which is created by sampling a batch of possible realizations of the model called particles. These so-called particles independently move through states in the language model based on their transition probabilities. Once a character selection is made, particles are resampled based on weights derived from observed EEG responses, effectively replacing lower probability with higher probability realizations. The algorithm then computes a histogram to estimate a probability distribution over the possible output strings after the particles have moved through the model.

Dynamic classification was implemented by setting a threshold probability rather than a set number of flashes to determine when a decision should be made.[15] The classifier then selected the character that satisfied the highest probability. In offline analysis, the value of the threshold probability was varied between 0 and 1 in increments of 0.01, the corresponding speeds, accuracies, and bit rates were determined, and the threshold probability that maximized the bit rate was selected.

Evaluation

Evaluation of a BCI system is two-fold: the ability of the system to achieve the desired result (i.e., accuracy) and the amount of time required to reach that result must both be accounted for. Selection accuracy can be used as a surrogate for measuring system efficacy. We define accuracy by the percentage of final output characters that match the target string. Selection rate (SR) measures the speed of the system by taking the inverse of the average time required to make a selection.

Because there is a tradeoff between speed and accuracy, information transfer rate (ITR) (in bits/min) is also used to evaluate BCI systems. It has been shown that ITR overestimates the information rate in BCI typing because it assumes that all characters are equally probable and independent of previous selections, which do not hold when typing natural language.[24,25] Furthermore, ITR assumes that all errors are uniformly distributed across all possible characters, which is not true in the presence of a language model which biases the system towards more probable characters. Nevertheless, it is a standard metric used for evaluation of P300 communication systems, so we include it here in addition to the more appropriate accuracy and selection rate metrics. Because of the violated assumptions, however, this value should only be used as a basis for comparison to other language-based systems and not as an accurate assessment of the true information.

The ITR is found by first calculating the bits per symbol, B, the amount of information transmitted per selection when taking into account the accuracy and the number of possible selections [26]:

$$B={\log }_{2}N+\mathit{\text{Acc}}{\log }_2\mathit{\text{Acc}}+(1-\mathit{\text{Acc}}){\log }_2\frac{1-\mathit{\text{Acc}}}{N-1}$$

where N is the number of characters in the grid (36) and Acc is the selection accuracy. ITR can then be found by multiplying the selection rate by the bits per symbol. Significance was tested using Wilcoxon rank-sum tests. In offline analysis, the optimal threshold probability was determined independently per subject. However, this optimization was impractical for online experiments, so a previously reported value of 0.95 was used for all trials.[18]

Results

Offline Performance

In retrospective analysis of the training data using the offline version of the particle filter classifier, five of the subjects were able to achieve above 90% accuracy using fewer than 40 flashes on average (figure 1). When optimizing the selection threshold, all subjects other than subject C had accuracies above 80% with selection rates of at least 10 characters/minute (table 2). The ITR values for these five subjects ranged from 34.15 (subject D) to 61.19 (subject B).

Figure 1.

Offline accuracy versus average number of stimuli (i.e., row/column flashes) used for classification. As the selection threshold is increased in offline analysis, the number of stimuli used for classification increases, increasing the selection accuracy. The results of the five ALS patients are plotted along with the average and standard deviation for a set of 10 healthy subjects.

Table 2.

Offline selection rates, accuracies, and information transfer rates (ITR) for each ALS subject along with the average results from a group of healthy subjects using the same system when using two different classifiers: the standard linear discriminant analysis with dynamic stopping (DYN) or particle filtering with a language model (PF).

Subject	SR (selections/minute)		ACC (%)
	DYN	PF	DYN	PF
A	6.33	9.65	83.33	96.67
B	9.68	13.68	93.33	93.33
C	4.66	5.54	46.67	53.33
D	7.54	9.98	76.67	80.00
E	8.53	11.88	90.00	96.67
F	8.42	11.92	96.67	96.67
Average	7.53	10.38	81.11	86.11
Healthy Average	9.78	11.97	95.00	96.00

Using the full signal, the SWLDA classifier was not able to find sufficient features in subject C’s data in order to detect P300 signals. When manually inspecting the signal, P300 responses were seen in the parietal and occipital electrodes (figure 2). Limiting the classifier to these channels, the classifier was able to achieve 53.33% accuracy with a selection rate of 5.54 characters/minute, both substantially lower than the other subjects.

Figure 2.

Average signal response for attended (solid) and non-attended (dashed) stimuli in the parieto-occipital channels for subject C.

Overall, the full set of six subjects averaged 10.38 characters/minute with an accuracy of 86.11%, resulting in an ITR of 43.96 bits/minute using the particle filter classifier. All three values were higher than those found using a standard SWLDA classifier with dynamic stopping (7.53 characters/minute, 82.22%, and 28.88 bits/minute, respectively). The differences in selection rate and ITR were significant (p=0.03 and p=0.03, respectively), but the difference in accuracy was not significant (p=0.13). The values found using the particle filter algorithm were lower than those achieved by healthy subjects (11.97 characters/minute, 96.00%, and 57.00 bits/minute), but none were significant (p=0.25, p=0.19, and p=0.15, respectively using Wilcoxon rank-sum tests). Excluding subject C from analysis resulted in a higher average selection rate (11.35 characters/minute), accuracy (92.67%) and bit rate (50.78 bits/minute). The selection rate and ITR were still lower than the values for healthy subjects, but none of the differences between the data sets were significantly different (p=0.50, p=0.40, and p=0.31, respectively).

Online Performance

The four subjects who were able to use the system in online mode were all able to type using the system with at least 84% accuracy, with three (subjects A, B, and D) having perfect accuracy. The subjects typed different numbers of characters based on the length of the phrase chosen, with subjects A, B, D, E, and F typing nine, 15, 18, 50, and 19 characters, respectively. There were 11 incorrect selections across all subjects of which five (45.5%) were automatically corrected by the PF algorithm. Selection rates varied from 5.96 characters/minutes (subject D) to 12.61 characters/minute (subject E), resulting in ITR values from 30.79 bits/minute to 60.08 bits/minute.

The average accuracy for ALS patients (95.64%) was slightly higher than that found in healthy subjects (94.21%) using this system and the average selection rate (9.90 characters/minute) was lower than that reported for healthy subjects (11.16 characters/minute), but neither difference was statistically significant (p=0.31 and p=0.16, respectively). The lack of significance for the selection rate is likely due to the small sample size and a large variance in performance as subject D had a lower selection rate than any of the healthy subjects and the selection rates for subjects A and E were lower than all but one of the healthy subjects. Similarly, the average ITR value for ALS patients (46.26 bits/minute) was lower than that reported for healthy subjects (52.27 bits/minute), but not significantly (p=0.16).

Discussion

BCI typing with a language model classifiers results in improved performance over that generally reported in the literature in subjects who are the intended users of such devices, similar to the improvements seen in healthy subjects. The online typing speeds achieved using the classifier implemented in this study (5.96–12.61 characters/minute) are also higher than those previously reported in the ALS population using other language models. In a 10 subject ALS study, Mainsah et al. [27] reported an average selection rate of 1.7 characters/minute with an accuracy of 80% using a standard classifier. Incorporating dynamic stopping and a language model using naïve Bayes, subjects were able to increase average typing speeds to 3.5 characters/minute with a nonsignificant decrease in accuracy. In a pair of previous experiments using the rapid serial visual presentation BCI communication system with a similar naïve Bayes language model, a total of seven patients had individual typing sessions that ranged from 0 to 3.7 correct characters/minute depending on the subject and the likelihood of the target string.[28,29] The online bit rates for the ALS patients in this study were (46.07 bits/minute) also higher than those reported from healthy subjects using simpler language model based classifiers such as naïve Bayes or hidden Markov models, which generally range from 20.83 bits/minute to 42.31 bits/minute.[18,30,31]

Of the five subjects who successfully used the system in online mode, one subject (subject B) had comparable performance to the healthy volunteers, while the other four were lower. The four subjects who had lower performance all used headrests that pushed against their occipital electrodes. Because the electrodes used in this study were not flat against the head, this pressure could cause them to turn, reducing their connectivity over time. Subject D in particular moved his head to focus on each new character, which caused shifting in the electrodes and resulted in noise for the first few stimuli for each character. Additionally, subject E used a BiPAP mask continuously through the experiment, which required shifting electrodes to avoid the straps. While no obvious differences were noticed in the signals, the slight change in location and additional pressure could have affected connectivity. Nevertheless, these subjects were able to use the system to successfully communicate in an online setting. However, the obstacles unique to the target patient population should be taken into account when choosing hardware for a production system as they can possibly affect how well the system translates into a realistic environment.

While five subjects were able to successfully use the system in online mode, there was one subject for whom the classifier could not detect P300 signals, preventing online use. This subject reported subjective blurry vision and difficulty focusing his gaze, which prevented him from using eye tracking systems. After manually choosing the channels with the best signal, however, offline analysis showed that communication could still be possible. Identifying the set of channels that can best discriminate between attended and non-attended stimuli is therefore highly important for the creation of a usable P300 speller system. Even after channel selection in this case, performance was lower than necessary for a reasonable communication system. It is possible that altering the interface could have improved the SNR for this subject, improving the system’s performance. Increasing the size and spacing of the targets, for instance, could make stimuli easier to distinguish. Creating a general system that can work for the majority of subjects allows for mass production, reducing costs and making the system more accessible to the ALS community. However, it is important to identify ways that the system can be customized in order to reach those subjects with additional concerns.

Limitations and future directions

The main limitation of this study is the relatively low amount of online data recorded for each subject, with each subject using the system online for only one to five minutes. Because of the small amount of data involved, the performance estimates for the target population presented here are likely less precise than those for the healthy subjects. The reason for this limitation is the time was spent during each session for setup, instructing the user on how to use the system, and collecting 30 offline trials, resulting in fatigue. Presumably, serial studies within subjects would yield more online data for further analysis, which is required for future studies. Reducing the number of channels used by the system could also decrease the amount of time required for setup.[32] In order to focus on online trials, future studies can also reduce the amount of training data required or remove the training session completely and instead learning the subjects’ EEG features online.[17,33]

While several of the subjects needed assistive devices to use a computer, none of them were fully “locked-in,” so speech or commercially available AAC devices such as a head mouse or eye tracker were effective. While BCI systems have some potential advantages over these devices (eye trackers, for instance, may need to be recalibrated when lighting changes in a room), additional work is needed before BCI systems can be considered to supplant existing AAC devices, including assessment of relative performance, both with respect to character selection rate and accuracy in the target environment. Currently, BCI solutions are the only option for fully “locked-in” patients. There are additional challenges in using the P300 speller for these subjects as the lack of gaze control will hinder the strength of the stimulus response.[34] Future studies should pay particular attention to this subpopulation as they are the patients who can be helped the most.

Alternative BCI systems have been developed which address the gaze dependence challenge by presenting all stimuli in the center of the screen, removing the need to shift gaze.[35] Other systems remove the dependence on vision entirely by eliciting P300 signals through either tactile or auditory stimuli.[36,37] While these systems are generally slower, they can potentially be viable for subjects with vision deficits that prevent usage of the P300 speller (such as that reported for the one subject in this study who could not use the speller in online mode). The language model and PF algorithm used here are independent of the graphical interface, so they can be directly implemented in these alternative systems. Future studies should explore this integration to determine whether similar gains can be obtained using these systems.

While the results of this study show that current methods can work as a means of communication for ALS patients, the current system needs to be modified in order to be a truly viable communication method. The hardware used consisted of a montage of 32 electrodes, which takes significant time to set up and can be prohibitively expensive. The current software is run on a desktop computer, which is overly cumbersome to move around so that subjects can communicate in different rooms. Also, the current implementation does not provide a means for users to initiate or terminate communication, limiting autonomy. Future studies should include tailoring the system to the target population by minimizing the required hardware and giving the subjects more control so that less assistance is needed in order to use the system.

This study serves as a demonstration of how the results using this system translate to communication performance by the targeted used population. However, the results are for a single, brief session and do not reflect the benefits and concerns that are encountered during long-term use. During the session, users expressed some challenges with getting used to the system, which could have adversely affected results and would not be present during actual use. Subject C, for example, said that he had difficulty finding the target letter in time because he was used to a QWERTY keyboard layout rather than the alphabetical one used here. Also, subject E mentioned that he modified his strategy between testing and training because it helped him concentrate. The language model used in this study was also trained on generic English text and is likely not the most accurate representation of the target strings that ALS patients will use the system to type. Tailoring the language model to ALS patients in general or individual subjects can further improve results. However, there are also additional concerns with long-term use such as fatigue and habituation, which may reduce the effectiveness of this system. Longitudinal studies within the ALS population need to be conducted in order to accurately gauge the utility of BCI communication systems.

Conclusion

This study indicates that the improvements in performance using language models in the P300 speller translate into the ALS population, which could help to make it a viable assistive device. However, the home environment and ALS subjects pose additional challenges, which should be considered when developing technology for use in an AAC device.

Table 3.

Online selection rates, accuracies, and information transfer rates for each ALS subject along with the average results from a group of healthy subjects using the same system. Subject C did not participate in the online trial because the classifier was not able to successfully distinguish between attended and non-attended signals.

subject	SR (selections/minute)	ACC (%)
A	9.24	100.00
B	11.62	100.00
C	N/A	N/A
D	5.96	100.00
E	10.06	94.00
F	12.61	84.21
Average	9.90	95.64
Healthy average	11.16	94.21

Biographies

William Speier received a B.S. in biomedical engineering and applied mathematics and an M.S. in computer science from Johns Hopkins University, and a Ph.D. in biomedical engineering from UCLA. He is currently a postdoctoral scholar in the neurosurgery department at UCLA and is involved in several areas of research, including medical informatics, neural signal analysis, and brain–computer interfaces.

Nand Chandravadia is pursuing a neuroscience B.S. at UCLA and is currently a research assistant in the UCLA neurosurgery department. His research interests include brain-computer interfaces, brain mapping, and neuroadaptive technologies.

Dustin G. Roberts received a B.S. in biochemistry from UCLA in 2014 and is pursuing an M.D. at the UCLA David Geffen School of Medicine with an interest in neurosurgery. Currently, his research interests include brain-computer interfaces, neuroimaging analysis, and neurosurgical interventions for pain relief.

Shrita Pendekanti is an undergraduate student pursuing a B.S. in neuroscience at UCLA and is currently a research assistant in the UCLA neurosurgery department. Her research interests include functional mapping, brain-computer interfaces, morphological studies, and meta-analyses of brain function.

Nader Pouratian received a B.S. and Ph.D. in neuroscience and an M.D. from UCLA. He is an Associate Professor of neurosurgery, neuroscience, and bioengineering at UCLA. His research focuses on multimodality brain mapping and signal processing including both invasive and noninvasive human neurophysiology to improve neurosurgical safety and efficacy and to develop novel and improved methods to restore communication and function to patients with neurological impairments.