A plug-and-play brain-computer interface to operate commercial assistive technology

Abstract

Purpose

To determine if a brain-computer interface (BCI) could be used as a plug-and-play input device to operate commercial assistive technology, and to quantify the performance impact of such operation.

Method

Using a hardware device designed in our lab, participants (11 with amyotrophic lateral sclerosis, 22 controls) were asked to operate two devices using a BCI. Results were compared to traditional BCI operation by the same users. Performance was assessed using both accuracy and BCI Utility, a throughput metric. 95% confidence bounds on performance differences were developed using a linear mixed model.

Results

The observed differences in accuracy and throughput were small and not statistically significant. The confidence bounds indicate that if there is a performance impact of using a BCI to control an assistive technology device, the impact could easily be overcome by the benefits of the assistive technology device itself.

Conclusions

BCI control of assistive technology devices is possible, and the performance difference appears to be very small. BCI designers are encouraged to incorporate standard outputs into their design, to enable future users to interface with familiar assistive technology devices.

Introduction

Brain-computer interfaces (BCIs) are intended to enable people with severe physical impairments to communicate and to operate technology without moving their bodies [1,2]. However, BCI performance has yet to reach either user expectations [3] or the level of performance available through commercial assistive technology (AT) [4]. To address this issue, researchers have begun incorporating AT techniques such as word prediction into BCIs (see e.g. [5]).

This incorporation can take three forms. In the first approach, BCIs are implemented as stand-alone AT devices with specific, limited functionality. In this approach, research time and effort is often spent in re-creating functionality already available through commercial products (see e.g. the creation of a custom programmable infrared controller in [6]). Though the resulting technology may be more tailored to the BCI application than a commercial product, existing AT products are likely to be more powerful than a research-driven design due to the different design focus. Another approach which has gained recent popularity is integrating existing AT products into BCIs (see e.g. [7–10]). This second approach has the advantage of allowing access to the capabilities of proven products, though the effort and expense required for integration grows with the number of products integrated. Because of these costs, some products are likely to remain un-integrated, limiting user choice.

A third approach, which we recommended [11], is to have the BCI act as a traditional input device through the principle of input emulation. In assistive technology design, input device emulation has been emphasized as a design criterion, because input emulation can maximize interconnectivity with special input devices [12], and provide a way to accommodate the changing needs of users [13]. For individuals with progressive conditions such as amyotrophic lateral sclerosis (ALS), who often experience changes in their ability to interact with their environment, this ability could be quite important. For these users, the availability of multiple access pathways to control existing AT devices could lower adoption barriers to BCIs.

This manuscript focuses on the practicality of a BCI keyboard that can replace physical keyboards. Such a keyboard must connect through the same universal serial bus (USB) port that the physical keyboard uses and also provide sufficient function for practical use. Performance categories include: 1) text generation accuracy and 2) text generation speed. Sufficient text generation accuracy, in the BCI field, is typically defined as 70% [14]. Sufficient text generation speed depends on competing AT options.

The P300 BCI design [15] is appropriate for keyboard replacement because it provides direct access to multiple options, frequently a visual letter grid as in Figure 1. In this design, rows and columns of the grid flash in a random order and the user mentally counts flashes of the desired option (letter, number, etc). Flashes of the desired option produce a P300 or “oddball” response in the user’s electroencephalogram (EEG), while flashes of rows and columns without the desired option produce little response. The desired option can be determined from the EEG after several complete sets of flashes. For people with vision impairments, P300 BCIs using auditory presentation of options are under development [16]. Paired with USB output emulation, a P300 BCI may be used as a keyboard replacement. In this study, our lab’s Multi-purpose BCI Output Device (MBOD) [11] was used to provide the USB output capability.

Figure 1.

A grid used to control a Dynawrite text-to-speech device. Users attend to the selection of their choice, noting when the selection flashes (currently, the fourth row is flashing).

While the technical capability of the MBOD to interface with a variety of devices was explored in [11], it remains to be shown that the resulting system can be operated without impacting performance of the BCI itself. BCI systems have typically been designed for self-contained use, though several BCI systems allow access to other programs running on the same computer (e.g. [8,10]). Using a BCI to interface with a standalone AT device presents at least three potential barriers to usability that are absent when simply interacting with the BCI. First, attention must be split between the AT device and the BCI display, even though one or both devices may be designed for uninterrupted attention. At a minimum, this operation mode could be considered a secondary task, which is known to affect the amplitude and shape of the P300 response on which the BCI depends [17]. If of sufficient magnitude, these changes would make it more difficult for the BCI to determine which option the user desires to select.

A second barrier is increased task difficulty due to timing requirements. In the field of text entry, it has been shown that shifting attention between devices requires approximately twice as much time as shifting attention between areas on the same device [18]. While both of the times in [18] were considerably shorter than the inter-character interval used in the P300 BCI, attention shift is only one task that must be performed in that interval. Accordingly, requiring an attention shift between AT device and BCI could be said to increase the task difficulty, possibly compromising performance.

A third barrier is that operation requires gaze shift, which might be a problem for some potential BCI users. For example, people with very advanced ALS have been shown to have difficulty controlling eye gaze systems [19]. This barrier is not addressed in this manuscript, as eventual adjustments to display modalities or workstation configurations could reduce or eliminate this requirement (see the limitations section for further discussion).

This study evaluates the performance impact of using the MBOD to operate commercial devices, as compared to simply generating text within the BCI itself. While the MBOD itself is a stopgap measure designed for research rather than commercial applicability, the results should generalize to any standalone P300 BCI keyboard using a traditional monitor for display. Our research goal was to determine if an unmodified commercial assistive technology can be functionally operated through plug-and-play replacement of a physical keyboard by a BCI keyboard, without unduly affecting BCI performance.

Methodology

Participants

Initial testing was performed with participants without amyotrophic lateral sclerosis (ALS). Three participants experienced with P300 BCIs were recruited from the Wadsworth Center BCI Group participant list based on schedule availability. Novice participants without BCI experience were recruited first-come-first-serve through public postings at the University of Michigan. Inclusion criteria were: age 18 or older, able to read text on a computer screen, and able to understand participation instructions. Exclusion criteria were: inability to give informed consent, uncontrolled neck movements interfering with electroencephalogram (EEG) recording, photosensitive epilepsy, or open sores on the scalp. After data collection from participants with ALS (see below), additional participants without ALS were recruited for age-matched data. For these participants, an additional inclusion criterion was added: age within one year of a study participant with ALS.

In all, 29 control participants (without ALS), 3 experienced [20] and 26 novices, were recruited. Twenty-two control participants, 13 men and 9 women, completed the 3 session protocol with a range of 3–29 days between first and last sessions. Seven novice participants were excluded from analysis for not completing the protocol: reasons given included headache and nausea after participation and frustration with text generation accuracy and speed (one each), while some participants simply did not respond to further scheduling attempts. Mean age of control participants who completed the protocol was 43years (range 18–79).

Participants with ALS were recruited from the University of Michigan Motor Neuron Disease Clinic. Participants with ALS had additional recruitment criteria of weakness affecting a hand or arm. Functional impairments were measured with the ALS functional rating scale-revised (ALSFRS-R), validated for telephone administration [21]. The ALSFRS-R rates function such that a score of 48 indicates no functional impairment while a score of zero indicates an individual who has loss of speech and purposeful limb movement; and requires a ventilator and feeding tube.

Thirteen novice participants with ALS were recruited. Eleven participants with ALS, (7 men and 4 women) completed the protocol. ALSFRS-R scores ranged from 18 to 41 with mean 28±7. Two participants with ALS were excluded from analysis for not completing the protocol. BCI accuracy was low (17%) after initial configuration for a 63 year-old participant with ALSFRS-R of 19. The individual did not want to repeat the configuration. A 56 year-old participant with ALSFRS-R of 11 had accuracy of 97.3% in the first session, but dropped out when accuracy in the second session was 50%. Mean age of participants with ALS completing the protocol was 62 years (range 45–78).

Participants either signed consent forms approved by the appropriate institutional review boards (IRBs) or gave consent and had a caregiver sign on their behalf.

Experimental Setup

Participants wore an EEG cap and sat approximately 0.8 meters from a 17-inch monitor showing the BCI display. The BCI was used to operate three devices: (1) the BCI itself in standalone mode, (2) a commercial communication aide¹, and (3) a separate laptop computer² running a commercial interface evaluation program³.

The communication aide device was below the BCI display; the laptop computer was beside the BCI display. Center-to-center distances between the BCI and communication aide displays and the BCI and computer displays were about 0.2 meters (14°) and 0.5 meters (32°) respectively. All BCI display variations contained the alphabet plus appropriate space, punctuation or command selections in a grid with 6 rows and 6 columns, as shown in Figure 1.

Experimental Design

The complete protocol involved 3 sessions of 1.5–2 hours, starting with about 30 minutes of time to set up the BCI. In the first session, BCI configuration data was collected for each participant. The configuration was determined from EEG data recorded while the user focused on each character in “THE QUICK BROWN FOX” for 15 complete sets (sequences) of stimuli (flashes). Each flash highlighted a row or column for 31.25 milliseconds with a pause of 125 milliseconds before the next row or column flashed. Time between characters was 3.5 seconds. The P300_GUI tool distributed with BCI2000 was used for configuration with settings of least squares weight selection, 800-ms EEG segments, and sample rate decimation of 20. For testing, the number of sequences for a participant was set to one greater than the number predicted to provide maximum BCI accuracy for that participant. This set the selections-per-minute for the participant since the BCI only selects a letter after the prescribed number of sequences; the average was 6.5 sequences; the average output was 4.1 selections/minute.

After configuration, and during each following session, the participant used the BCI to reproduce a different sentence on each of the 3 devices; the device currently in use will be referred to as the “target” device. To require participants to interact with the target device and not merely use the BCI display, we provided feedback only through the target device; letters selected using the BCI did not appear on the BCI screen unless the BCI was the target device (standalone mode).

Regardless of target device, participants used a backspace selection to correct mistakes, so the time and number of selections to reproduce sentences varied realistically. To limit user frustration, we restricted the time spent on any sentence to 15 minutes, ending the sentence early if it was not complete.

In each target device, participants were asked to reproduce a 23-character sentence which was printed in 46-point Arial font and taped to the top of the BCI display. For the communication aide device, participants were also asked to make a special selection which made the device speak the text that had been produced. A different sentence was used for each target device, and a different set of sentences was used in each session. The nine sentences came from the sentence bank of the Compass interface evaluation software⁴. Participants reproduced the sentences in the same order, but device order was counterbalanced across sessions to minimize bias from fatigue, learning or sentence variation.

Equipment

During BCI use, EEG from electrodes F3, Fz, F4, T7, C3, Cz, C4, T8, CP3, CP4, P3, Pz, P4, PO7, Oz, and PO8 was sampled at 256 Hz with a 16-channel electrode cap⁵ and a g.USBamp⁶ with reference and ground on the right and left mastoids respectively. Data were recorded and processed using BCI2000⁷.

The BCI was connected to the USB port on the communication aide or computer with the Multi-Purpose BCI Output Device (MBOD) [11] providing USB compatibility. An MBOD communication program on the BCI computer interpreted the non-standard BCI2000 output and transmitted it through the MBOD to the target device as standard USB keyboard codes. Thus, BCI2000 was recognized by these devices as a standard plug-and-play USB keyboard, making these devices BCI-operable without modification.

Data Analysis

BCI performance measurements were 1) text generation accuracy, and 2) corrected text throughput, as measured by the BCI-Utility metric [22].

Determining BCI selection accuracy (Figure 2) required an accurate record of selections intended by the participant. Because participants were instructed to correct errors, the result of previous selections determined the next intended selection. A list of intended targets was created by examining the target sentence and the actual selections. If a participant informed us of an error of intent, such as losing place in a sentence or accidentally attending the wrong character, the intent was recorded according to the participant’s report. Once the intended targets were determined, accuracy was calculated as the percentage of attempted selections producing the intended result. The selection commanding the communication aide device to speak the completed sentence was included in accuracy calculations.

Figure 2.

Example sentence, actual letters selected, target text including error correction, and correctness of each selection (1 indicates correct, 0 indicates incorrect). Note that if no incorrect selections were made, Sentence and Target would be a perfect match and the backspace (◀) would not appear. In this example, 26 of 29 selections were correct, producing an accuracy of 90%.

Statistical analysis was performed for both participant groups (with ALS and controls) in one cohort. Ninety-five percent confidence intervals for the text generation accuracy and throughput were calculated using a linear mixed model [23]. Fixed effects that were known to affect BCI performance, or the underlying P300 waveform on which the BCI depends, were also included in the model. These factors included diagnosis of ALS [24], number of sequences of stimuli (e.g. [15]), and participant age (e.g. [25]). Accuracy variations by participant and correlations among observations on the same participant were expected, so participant was included as a random effect. Because each device was tested during each session, device was included as a repeated measures factor within session. The same model was used to calculate confidence intervals for throughput as measured by BCI-Utility, except that number of sequences was not included as a fixed effect because the term appears directly in the formula for throughput. Age was then discarded from both models due to correlations with diagnosis and a dramatic difference in Akaike Information Criterion, a goodness-of-fit measure. Statistical analyses were performed with SAS⁸.

To address the age correlations, a second analysis was performed as above, but with only data from people with ALS and their age-matched controls. In this model, age was included as a fixed effect.

Results

Data Characteristics

Of the 306 sentences attempted by the participants who completed the study, 214 were completed without errors remaining. Another 35 sentences were completed, but errors were left in the text, in contradiction with the instruction given to the participants. With corrections, completed sentences took a mean of 31.2 selections in 7.67 minutes. Sixteen participants (4 with ALS) completed at least one sentence without needing corrections, i.e. 100% accuracy. Three participants did not finish 23 characters of typing with one accidentally skipping a letter, one running out of session time, and one participant stopping early out of frustration with low accuracy.

The per-participant accuracy averages for each device are shown in Figure 3.

Figure 3.

Average accuracy by device for each participant. Top) Control participants; Bottom) Participants with ALS. Error bars shows standard deviation across sessions.

Statistical Analysis

In the accuracy analysis, the effect of device was far from significance (p ~ 0.99). Neither diagnosis nor number of sequences of stimuli were found to be significant in this study (p ~ 0.15 and 0.06 respectively), though number of sequences trended toward significance. This relationship is likely to be real but weak; the number of sequences was set based on training accuracy, which is an estimate of BCI performance, albeit a poor one. The effect of session was not significant (p ~ 0.13), though considerable day-to-day variation was observed. As can be seen in see Figure 3, where the error bars indicate day-to-day variation, these day-to-day variations were typically larger than across-device variations, though no predictable trend was seen across participants.

In the throughput analysis, the fixed effect of diagnosis was found to be significant (F_1,31=4.27, p < 0.05), with participants with ALS scoring on average 0.74 corrected selections/minute slower than able-bodied controls, a difference of about 25% of the mean throughput for the control group. Fixed effects for session and device were far from significance (p > 0.3 for session, p > 0.95 for device). Relevant means and 95% confidence intervals are shown in Table 1. A plot of BCI accuracy versus ALSFRS-R appears in Figure 4.

Table 1.

Means and 95% confidence intervals for effects of device and diagnosis, all participants.

Accuracy (%)
	Estimate	LB	UB
BCI	82.1	77.6	86.6
Laptop - BCI	−0.05	−3.40	3.30
Comm. - BCI	−0.15	−3.57	3.29
ALS - Control	−5.45	−13.1	2.20

Throughput (corrected selections/minute)
	Estimate	LB	UB
BCI	2.75	1.97	3.14
Laptop - BCI	0.004	−0.29	0.29
Comm. - BCI	−0.03	−0.30	0.24
ALS - Control	−0.74	−1.48	−0.01

Figure 4.

Average BCI accuracy vs. ALS-FRS-r.

The age-matched analyses produced similar results, which are listed in Table 2. In the age-matched accuracy analysis, device was far from significance (p > 0.35). Number of sequences was found to be related (p < 0.05) Age approached significance (p ~ 0.09), while diagnosis appeared unrelated (p ~ 0.86).

Table 2.

Means and 95% confidence intervals for effects of device and diagnosis, only participants with ALS and age-matched controls.

Accuracy (%)
	Estimate	LB	UB
BCI	82.6	77.8	87.4
Laptop - BCI	−0.68	−4.49	3.12
Comm. - BCI	−2.47	−5.91	0.97
ALS - Control	−0.75	−9.75	8.24

Throughput (corrected selections/minute)
	Estimate	LB	UB
BCI	2.79	2.33	3.24
Laptop - BCI	0.001	−0.37	0.37
Comm. - BCI	−0.14	−0.45	0.18
ALS - Control	−0.74	−1.60	0.11

In the age-matched throughput analysis, the fixed effect of device was still far from significance (p > 0.6). Of the other fixed effects, only diagnosis approached significance (p ~ 0.08).

Discussion

The speed and accuracy at which a BCI could become useful to an individual depends on the speed and accuracy of alternative interfaces, most of which are faster than current BCIs [2]. For users with few interface options, a plug-and-play BCI as a keyboard replacement could reduce accessibility barriers by allowing any keyboard-operable device to be BCI-operated. The MBOD is a stopgap measure to study this functionality with current BCIs. A plug-and-play BCI would enable participants with other options to alternate between interfaces controlled by physical movements and by a BCI depending on their disease status, fatigue level, and preferences. This would preserve resources invested in a current functional system while maintaining its availability if the user’s impairments increase. Likewise, AT providers could prescribe a BCI as an alternative access method for familiar commercial AT. If BCIs cannot interface with AT, then the functionality that the AT provides to someone with a progressive impairment may be lost when the person’s physical condition progresses to the point that they need a BCI. In effect, their disablements would be increased due to the unnecessary incompatibility of two technologies both designed to enable people with impairments.

The results of this study support the hypothesis that an unmodified, mainstream commercial assistive technology can be functionally operated through plug-and-play replacement of a physical keyboard by a BCI keyboard. While some participants reported difficulty with screen positioning, particularly with the communication device, the results show that for most users the performance was extremely similar across devices. The confidence bounds on differences in accuracy and throughput due to device are relatively tight compared to inter-participant differences. For example, the bounds for difference in accuracy due to device are approximately ±3.5%, whereas Figure 3 indicates the standard deviation of mean BCI accuracy across participants is 14%. Within-participant differences in daily performance are also larger than device differences– the mean accuracy difference between session one and two was 5%, which lies well outside the confidence bounds for device differences. Even in the worst-case, where the true value for device differences lies close to the lower confidence bound, the boost from assistive technology may be larger than the difference in BCI performance caused by interfacing with the AT device; in [5], predictive spelling increased corrected output speeds by 1.5 characters/minute, which would more than offset the worst-case 0.3 selections/minute penalty predicted by this study (see Table 1).

One participant in the non-ALS group, K118, had dramatic device differences. The participant’s accuracy on the communication device was within the range of other able-bodied controls, but three standard deviations below the mean of the others in the standalone BCI device. We considered excluding the individual’s data from analysis based on lack of effort when using the other devices, as the participant reported wanting to hear the communication aide speak the sentence. This highlights the growing interest in the impact of motivation on BCI accuracy [26,27]. Individuals with impairments who need a BCI to accomplish their top priority tasks should be more motivated than research participants. The apparent importance of motivation supports incorporating plug-and-play capabilities into commercial BCIs to maximize the number of BCI-operable tasks, since users may be less successful with a BCI that limits or reduces the tasks they can perform.

The single cohort analysis indicated a reduced throughput for participants with ALS, though that sample showed significant correlation between age and diagnosis because many of the early participants were young, able-bodied people. The age-matched analysis did not show statistically significant reduction in throughput by the usual criteria, but it is worth noticing that the estimate itself remains almost identical. These results suggest that there may in fact be an average throughput difference between people with ALS and controls, at least when the number of sequences of stimuli are optimized based on accuracy.

This reduced throughput for participants with ALS is a concern because these are people from the target user population. It is interesting to note that some participants with lower ALSFRS-R scores had BCI accuracy as high as any controls. However, participants with ALS who had lower accuracies also had lower ALSFRS-R scores (Figure 4). This may indicate that BCI performance is affected by a particular subset of ALS symptoms, not overall ALS progression as measured by the ALSFRS-R. One likely candidate is the cognitive impairment associated with ALS, which appears to impair verbal fluency [28–30]. However, the number of individuals with ALS in this study is too small to address the question of which subset is important, if any.

Another way to handle alternation between interfaces is implementing a hybrid BCI, such as that presented in [31]. A primary goal of hybrid BCIs is to address the issue of fluctuating user conditions using a control fusion approach, with online systems determining which input device (BCI or physical interface) is providing the most useful control at any given moment. While the system in [31] is capable of operating most computer software, it cannot interface with standalone AT devices. If the ability to interface with such devices could be added to hybrid BCI systems, the result could be an elegant method for handling multiple inputs to existing AT devices.

Study Limitations

Participant motivation was not measured, and motivation has recently been shown to affect BCI performance [26,27].

Although BCIs are intended for operation without physical movement, several recent studies may indicate that the most common P300 BCI “keyboards” utilize eye movements [4,32]. Plug-and-play BCIs that provide a separate technology interface and therefore a separate display are particularly susceptible; however careful design should reduce this reliance. Minimizing BCI display size is an obvious step, since a small display requires less eye movement, provides more positioning options, and provides similar accuracy to a large display [33]. Users who want to operate computer software can eliminate the separate display by connecting the plug-and-play BCI back to the BCI host computer; a configuration tested successfully using the MBOD. Eventually, BCIs could use a transparent screen or retinal projection so the display appears at the same location as the target device [34]. Alternatively, a plug-and-play BCI could use auditory presentation of options as in [16], or a visual BCI could operate an AT device with auditory feedback.

Conclusions

A plug-and-play BCI can be used as a functional keyboard replacement to operate AT and mainstream technology for people with ALS and controls. Differences in average accuracy and throughput between devices appear to be quite small, and can be offset by the advantage of using AT.