TeleBCI: remote user training, monitoring, and communication with an evoked-potential brain-computer interface

Abstract

Brain-computer interfaces (BCIs) are a movement-independent form of augmentative and alternative communication (AAC) for individuals with amyotrophic lateral sclerosis (ALS). The rare utilization of such devices in the homes of patients stems from a number of factors, one of which is the complexity of providing training and support for users. This paper describes the teleBCI interface used to train the patient and facilitator in the operation of a virtual keyboard using an evoked potential BCI. Fifteen patients with motor neuron disease and their communication partners were included in the study, participating from their homes while receiving remote support from the research team. Patient/caregiver teams completed 8 sessions each of P300 BCI training virtually with the researcher. As they participated in subsequent training sessions, participant teams required less help to complete physical, computer, and BCI-specific tasks associated with device use. A subset of users experienced improved performance over sessions, progressing to utilize the full functionality of the speller and communicate with a nurse partner over a telemedicine interface. Perceptions of device utility varied with accuracy of the BCI system. In the management of ALS, the integration of telemedicine provides new opportunities for care delivery, including how BCI-AAC are deployed and used.

1. Introduction

Amyotrophic lateral sclerosis (ALS) is characterized by the degeneration of both upper and lower motor neurons. Only about 25-33% of patients have speech dysfunction at onset of symptoms, but it is estimated that 80-95% of individuals with ALS will be unable to effectively use natural speech at some point prior to death [1] a physical limitation associated with poorer quality of life (QoL) [2]. When speech becomes unintelligible, patients may choose to use augmentative and alternative communication (AAC), a set of procedures and processes for maximizing functional and effective communication [3] that have been shown to facilitate a positive change in QoL [4, 5].

Speech generating devices can use multiple inputs for control, such as mechanical switch, eye-gaze, or electroencephalogram (EEG) in the case of a brain-computer interface (BCI). Individuals with ALS using AAC technology do so on average for more than two years - substantially longer for those on mechanical ventilation – and continue to do so until around one month prior to their deaths [6].

BCI technologies in particular offer the possibility of eye-independent communication through the user’s intentional modulation of brain activity. An early demonstration of this featured the self-regulation of slow cortical potentials to translate thoughts into “yes” and “no” commands [7]. Since that time, one of the most widely implemented BCI control signals is the ‘oddball’ paradigm, where a target stimulus is presented infrequently and randomly within a larger group of non-target stimuli. By responding to the presentation of only target stimuli by, in the case of this study, counting the target stimuli as they appear, a target-specific evoked potential (EP) can be measured at the scalp. The P300 is a positive component of this EP, generally the most prominent feature of the oddball evoked response, originating at the centro-parietal midline with latency of approximately 300 milliseconds [8]. Other components such as the N170 and N400f are elicited during the process of face recognition [9], the use of which has recently been employed for BCI use [10, 11]. The ‘P300 speller’ is a BCI paradigm that makes typing selections based on the evoked responses to a target stimulus.

Despite substantial evidence of P300 BCI efficacy from laboratory demonstration, a smaller body of research exists for utility of BCI-AAC devices in the homes of end users. This disconnect may occur due to physical, cognitive, and behavioral deficits of end users, device limitations (inaccuracy, limited functionality) and the AAC environment (funding, professional support, caregiver availability) [12]. In recent years, more research teams are designing BCI systems intended for use outside of the lab without substantial researcher oversight. The BackHome system has been developed by a consortium of European researchers to achieve just this [13]. This team used a user-centered design approach to create a system that prioritizes BCI efficacy, functionality, and ease of use. Brain Painting, an application within BackHome where users express their art through the P300-controlled digital painting, has demonstrated long-term stability when used in the homes of patients for up to 5 years [14] and has been shown to have a positive impact on the user’s competence, adaptability, and self-esteem [15].

BCI, like other AAC methods, requires adaptation to the individual user, with ongoing training for the user and their facilitators, lack of which can cause helplessness [12]. AAC facilitators assist with day-to-day AAC needs, and ongoing facilitator support is critical to AAC success [16]. Limited demonstrations have shown that BCI training can be achieved remotely [17, 18, 19], and that the system parameters can be monitored and updated to maintain long-term use [20, 18]. Few of these have included assessments of patient and clinical team satisfaction [19], or quantitative monitoring of progress in training on the device. A recent study described an extensive trial of independent home use of BCI in individuals with ALS, which demonstrated that users can be trained to use the spelling device to maintain communication and QoL [21]. They showed that with proper screening and training, users could learn to operate these devices independently. However, this training was conducted by a member of the research team over the course of 2-6 home visits, an intensive effort that would either require local providers or place a substantial burden on the research team. As BCI systems enter the homes of users, more information is needed on the most effective ways to train and sustain BCI use.

The expanding practice of telemedicine in ALS care provides a clear opportunity for integration of BCI-AAC training, monitoring, and communication. Many individuals with ALS rely on AAC technologies for basic communication, including those who might receive care via telemedicine. For these individuals, a virtual visit to the clinic may be limited to indirect interactions between patient and clinical team, as mediated by a caregiver. Knowing the importance of communication in the QoL of patients with ALS and the demonstrated failure of BCIs when implemented as AAC in patients with no alternative communication options [22], a strong case can be made for the proactive, sustained training of users and their communication partners in AAC-BCI via telemedicine [1, 23].

The primary aim of this study is to develop and test a videoconference-compatible brain-computer interface communication module (teleBCI) in patients who possess limited verbal communication. The remainder of this paper describes the teleBCI interface used to train the patient and facilitator in the application and operation of a virtual keyboard using an EP-based BCI. The study assessed the efficacy of teleBCI training, communication with the BCI, and perceptions of its usability from the user and facilitator.

2. Methods

2.1. Subjects

Three participant groups were enrolled in this study: AAC-BCI users, their BCI facilitators (together referred to as “participant teams”), and clinician nurses. Clinician nurses were members of a multidisciplinary ALS care team. Potential patient participants were recruited from the clinical population and through use of a regional mailer. Patients were required to have a diagnosis of definite, probable, probable laboratory-supported, or possible ALS by revised El Escorial research criteria [24], primary lateral sclerosis (PLS, upper motor neuron involvement only), or progressive muscular atrophy (PMA, lower motor neuron involvement only). Eligible subjects were also required to have limitations in their capacity for useful speech, upper extremity function, or both, have a facilitator able to perform the study procedures, and have access to a reliable in-home internet connection. Consent procedures were performed in the home of the participant, following the research protocol approved by our Institutional Review Board (IRB).

2.2. teleBCI

The teleBCI system (Figure 1) was set up by the researcher in the home of the participant team. It included a Windows 10-based laptop with integrated webcam and microphone, a second patient-facing monitor on which was mounted a Tobii eyeX eye tracking bar (Tobii, Sweden), and an 8-channel g.Nautilus cap (Guger Technologies, Austria), with gelled g.Scarabeo active electrodes for recording EEG. The videoconferencing software Adobe Connect (Adobe, USA) allowed for two-way audio and video transmission, screen sharing, and remote computer control by the research team. Data synchronization (Box Inc., USA) allowed for programs, settings, and study data to be shared bi-directionally between the researcher and participant team. BCI2000 [25] was used for data acquisition, computation, and feedback of biosignals. EEG data was recorded from electrodes Fz, Cz, P3, Pz, P4, PO7, Oz, and PO8 at 250 Hz, 8th-order Butterworth filtered between 0.5 and 60 Hz, and 60 Hz notch filtered before being saved to file. Simultaneously, BCI2000 synchronized the capture of 2-dimensional position of detected eye gaze at a rate of approximately 30Hz.

Figure 1.

Components of the teleBCI system.

The typing applications used in the study were based off the checkerboard P300 speller, described previously [26, 27]. The speller was a 5x8 grid that flashed in groups of 4 or 5 non-contiguous icons. Two online spelling systems were used. The training program consisted of copy spelling task with a grid of 35 active icons, including each letter of the alphabet, a space character, and the numbers 1-8. The training program generated labeled data that could be used to create and update the classifier. The notepad speller allowed for free spelling and produced unlabeled data. It also contained additional icons for deleting a letter, undoing the previous selection, clearing, saving, and speaking the current notepad text. Three accessories were unique to the notepad speller: text-to-speech, prepared text via saved phrases, and text prediction. To create text prediction functionality, a Windows-based text auto-completion program (TypingAssistant, Sumitsoft, China) updated a list of 8 words that could be completed from the current word fragment, which the user could select by choosing the corresponding number.

A single P300 trial included the selection of one character. Individual icons of the stimulus grid were initially flashed 10 times per trial; if the subject achieved 100% accuracy (4/4 trials copied correctly during a run), the number of flashes could be reduced to increase typing rate. Similarly, the subject could advance to the notepad speller if they achieved 100% accuracy on a training run. For subjects P1-P10, stimuli were flashed for 100 ms followed by an inter-stimulus interval of 100 ms. In a revision of the protocol to enhance the random quality of target stimuli, subjects P11-P15 used a modified speller with a stimulus-on time of 120ms, and off time between 60 and 300 ms. Interstimulus interval settings were not modified for individual subjects over the training period. For the two iterations of the speller system, a trial lasted 18 (P1-P10) or 27 (P11-P15) seconds and a run consisted of multiple (4 in the case of copy spelling tasks) trials.

Two user-facing programs, created in MATLAB (Mathworks, USA), were utilized by the participant team during training. The Guide included textual and graphical direction in the ten steps performed in the training session: 1) turning on the computer; 2) logging into the videoconferencing portal; 3) calibrating the eye tracker; 4) fitting the cap; 5) applying electrodes; 6) opening BCI2000; 7) checking impedances; 8) performing a P300 run; 9) updating classifiers; 10) cleaning/charging/storing the system. The time spent on each step was recorded in the guide log. The Classifier program displayed a list of all the training data files, presented top to bottom on the display, ranked by the standard difference between target and non-target EEG means averaged across channels. This allowed the user to visualize files of higher discriminibility and, with the help of the researcher, select a subset from the current and previous sessions to build a classifier. The pipeline for classification was standardized. Data was segmented into 1200 ms epochs following stimulus presentation. It was determined that for the sampling rate (250 Hz) and block sizes (25 for subjects P1-P10 and 15 for subjects P11-P15) of EEG acquisition, the delay between timing of the stimulus code marker and an optical sensor on the monitor corresponding to that stimulus code was 450 ms and 300 ms, respectively. This delay was due mainly to bluetooth transmission of data packets, and was used to correct stimulus onsets in the data file. Data segments were low pass filtered at 20 Hz and then down-sampled by a factor of 12. Epoched data for individual channels were concatenated so to produce a n_stimulus × (n_channels * n_timepoints) matrix, along with an associated (n_stimulus× 1) label vector for target and non-target stimuli. This was passed to a stepwise linear regression function which calculated a set of weights for separating the data from the two classes. These weights and their associated channels and times were saved for online use within the linear classifier module of BCI2000.

2.3. Protocol

The first study component was a home visit, during which the researcher conducted the intake interview, which included the Edinburgh Cognitive and Behavioural ALS Screen (ECAS) [28] when possible and the ALS Functional Rating Scale – Revised (ALSFRS-R) [29]. The ALSFRS-R ranged from 0-48, and for descriptive purposes we defined bulbar (sum of questions 1-3, range 0-12), and motor (sum of questions 4-9, range 0-24) sub-scores of this assessment.

The researcher then guided the participant team through the steps of the teleBCI Guide. Users were instructed how to start the system and connect to the videoconference. They were shown how to position the monitor to allow for eye tracking and clear viewing of the whole screen, which was typically at a distance of 18-24 inches, eyes aligned with the center of the monitor. Depending on the seating used, the monitor was either positioned on a table or attached to a floor stand (Rehadapt, Charlottesville, VA, USA and Daessy, Richmond, BC, Canada). The floor stand could be rolled under a bed or wheelchair and adjusted for different heights and head angles. The researcher guided the user to create an eye tracking calibration profile that could be recalibrated as needed in subsequent sessions. The cap size for the EEG system was selected based on the circumference of the head according to the manufacturer’s sizing chart. The wireless transmitter connected to the cap was either attached to the back of the head, or if the user was reclining, detached from the cap and laid at the side. The participant team was shown how to apply gel to the electrodes and check the impedances that were displayed in a list on the computer by the BCI2000 software. The target impedance was less than 50 kΩ for each channel. In practice, the active electrodes of the g.Nautilus system are not required to meet as rigorous impedance levels as passive electrodes, which was one reason why these systems were chosen for this study. In cases where data demonstrated significant artifact, users were shown how to remove a channel from online processing for the session. Users performed training runs of copy spelling, after which the classifier was updated and used in the subsequent run. At the end of the session, the researcher demonstrated the procedures for removing, cleaning, and storing the system.

Other than the differences in system timing for subjects P11-P15 described above, all subjects began training using a visual P300 system with ten stimuli repetitions per trial. If the visual paradigm elicited a weak or non-existent P300 response that produced classification accuracies close to random, the subject was transitioned to an audio-based version of the system, the performance outcomes of which are not evaluated here. Except for the first run of the first session, all BCI tasks were performed online with the user receiving feedback of the selection after each trial. Other aspects of device use were adjustable as needed to change the type of speller used, the accessories to aid with communication, and the number of P300 sequences encompassing a single trial.

At the end of the first session, the participant team completed the Assistive Technology Device Predisposition Assessment (ATDPA) form of the Matching Person and Technology (MPT) assessment [30]. This tool has the participant team rate the functionality of the device to their communication needs in 12 domains, rated on a 1-5 scale, with a total score of 60 indicating maximum expected benefit.

The following 8 weekly training sessions were remotely guided by the researcher through the telemedicine interface. During each session, the participant team set up the system and initiated the BCI training program. The goal of each session was to complete at least 4 P300 runs in the training program. Each run consisted of 4 copy spelling trials, using four-letter words chosen at random from a predefined word bank. The actual number of trials completed was subject to user preference and vigilance. As users gained proficiency, they advanced from the training program to the notepad speller, where users were able to spell their own words/phrases while using accessories for auto-completion, text-to-speech, and saving phrases. Phrases completed using the notepad speller were confirmed with the subject at the end of the run for accuracy. At the end of each teleBCI session, the participant team completed the post-session log, where they described help received from the researcher and their confidence in the procedures. The EEG, eye tracking data, and computer logs were saved locally and synchronized to the research Box account.

The endpoint of the study involved communication of the patient and an ALS nurse using the BCI notepad speller over the telemedicine portal. Participants who were not able to achieve usable accuracy with the device (>70% correct selections in the training program) did not complete this interaction. The ALSFRS-R and ATDPA were re-administered at the final home visit.

2.4. Assessments

One of the challenges of executing strictly-regimented research in this population is the customization needed to adapt BCI to users of different functional abilities. The protocol design and evaluation in this study therefore utilized key aspects of user-centered design described by Kübler et al. [31]. Assessment of device performance used traditional metrics such as accuracy and communication speed, as well as the usability of the system and satisfaction of the BCI user and communication partner. The structure of the training sessions allowed for collection of the following repeated measures related to the participant team’s experience.

1. Setup proficiency - The steps of teleBCI operation were grouped into three categories: physical tasks, (application and maintenance of cap and electrodes), non-BCI computer tasks (videoconferencing and general computer navigation), and BCI tasks (calibration of the eye-tracker, BCI2000 configuration and utilization, updating classifiers). The Guide log reported the time spent on each of these tasks. At the end of the session, both the participant team and researcher reported whether help was offered during completion of each task (yes/no) and the level of confidence in team self-sufficiency (a 4-item Likert scale ranging from ‘not at all confident’ to ‘very confident’). Additionally, the type of interaction between the researcher and the participant team, whether in-person, phone, or videoconference was recorded by the researcher.

2. Data quality - Impedances achieved during cap setup, error and variance of eye gaze, and the amplitude and latency of evoked potentials were captured for each session.

3. TeleBCI use Communication accuracy and effective bit rate, the latter derived from bits per selection and divided by trial length, were tracked over session. The calculation of bits per selection is given in [32] as $lo{g}_2(N)+Plo{g}_2(P)+(1-P)lo{g}_2(\frac{1-P}{N-1})$, where N is the number of possible choices per trial (40 for a 5x8 speller) and P is the probability of a correct selection. Because the timing and number of flashes, stimulus modality, and type of speller all factor into the communication rate, the effective bit rate was chosen to provide a common comparison across the different system iterations. Also collected were the types of spellers and different accessories used.

4. BCI Perceptions - The ATDPA assessed users disposition towards the P300 BCI AAC after the first and final sessions.

2.5. Analysis

Repeated measures aggregated over sessions relating to setup proficiency, data quality, and teleBCI use, were analyzed using Spearman’s rank correlation with session number as the independent variable. Significant results are reported with correlation coefficients and significance values. To describe changes in the relationship between perceived benefit and user experience, we estimated a linear model, with the change in summed ATDPA scores as the response variable, and participant confidence, time burden, and performance with the device as predictor variables. The difference in summed ATDPA scores could range from negative to positive 44, with endpoints representing extreme decrease or increase in perceived benefit, respectively. The aggregate confidence scores in the post-session surveys, average time to set up the device, and average BCI accuracy were used as independent predictors of this change.

3. Results

3.1. Study cohort

Fifteen patient and caregiver pairs were included in the study (11 male patients). The patient group had a mean±SD age of 63.7±8.2 years. 13 patients had a diagnosis of ALS, one had PLS, and one had PMA. A full listing of relevant demographics is given in Table 1. The median time since symptom onset was 63 months, with a range of 12-142 months. Nine patients completed an ECAS for this study or in the past year, of which one tested below the cutoff for cognitive impairment, and three (including the one with cognitive impairment) had scores indicative of behavioral impairment. The mean total, bulbar, and motor scores of the ALSFRS-R were 20.1/48, 5.6/12, and 6.2/24 at initial assessment, which decreased to 17.6/48, 4.9/12, and 5.1/24 at study end. Patients performed study procedures at a median distance of 64.5 miles from the location of the ALS clinic. Two nurses, with combined clinical experience of 30 years in the care and management of individuals with ALS, engaged with a subset of subjects in teleBCI-facilitated communication at the conclusion of the study.

Table 1.

Patient demographics. TSSO - Time since symptom onset (months), FRS_b - sum of bulbar items 1-3 of the ALSFRS-R (range 0-12) at study initiation/end, FRS_m - sum of motor items 4-9 of the ALSFRS-R (range 0-24) at study initiation/end, ECAS_c - cognitive score of the ECAS with cutoff for impairment at <105, ECAS_b - behavioral score of the ECAS with cutoff for impairment >0

Code	TSSO	FRSb	FRSm	ECASc	ECASb
P01	63	6/6	8/7	110	0
P02	36	0/0	0/0
P03	13	2/2	20/15	109	0
P04	122	0/0	0/0
P05	12	9/9	4/5	113	1
P06	66	10/9	14/14	120	0
P07	142	2/2	1/1
P08	39	8/7	1/1	109	0
P09	27	7/4	2/1
P10	81	7/5	0/0
P11	24	0/0	18/12	98	1
P12	24	10/9	3/1	107	1
P13	73	11/11	9/7	115	0
P14	124	7/7	12/12	127	0
P15	64	5/2	1/1

All 15 participant teams completed the initial session and 8 teleBCI training sessions. Three subjects transitioned to an audio version of the P300 speller because of ineffective visual evoked responses. For subject P04, this occurred at the initial session, after determining the subject’s inability to orient their gaze to the task on the computer screen. For subjects P02 and P07, this occurred at sessions 7 and 5 after demonstration of evoked responses that were non-discriminable between target and non-target trials on the visual P300 task. Evaluation of the audio-based P300 spelling system is not discussed here, rather we focus on the results of the visual P300 task, which were available for all subjects except P04.

3.2. Setup proficiency

Sessions typically lasted 60-90 minutes. The setup time, or steps 2-7 of teleBCI procedures, improved from a mean of 39.7±13.3 minutes in the first session to 18.8±8.4 minutes in the last session. As shown in Figure 2A, session number demonstrated a negative correlation with completion time for physical tasks (Spearman’s ρ = −0.22, p = 0.0150), non-BCI computer tasks (ρ = −0.35, p = 0.0001), and BCI tasks (ρ = −0.37, p < 0.0001).

Figure 2.

Progression of teleBCI training over the initial in-person and eight remote sessions. Assessments are averaged within three groups: physical tasks, non-BCI computer tasks, and BCI tasks. Boxes extend from the first to third quartiles of data, with median as an open circle. Thin lines extend to the furthest data point within 1.5 times the interquartile range, and individual markers outside this range are outliers. A. Time to complete tasks. B. The fraction of tasks reported by team to have received researcher help. C. Confidence in task self-sufficiency reported by the participant team.

Participant teams reported researcher guidance for physical and computer tasks tapering to a low value by the fourth session, while assistance with the BCI tasks remained relatively high (Figure 2B). Teams still required assistance on an average of 84% of BCI tasks during the final training session, while only requiring assistance with 40% of physical tasks and 30% of computer tasks. Session number was negatively correlated with reported assistance during physical tasks (ρ = −0.39, p < 0.0001), non-BCI computer tasks (ρ = −0.40, p < 0.0001), and BCI tasks (ρ = −0.28, p = 0.0012). The researcher at first session reported a higher proportion of physical (6.7% greater), computer (13.3%), and BCI (10.7%) tasks requiring their guidance compared to the reports of participant teams. This difference changed direction by the second session, and by the 5th session the researcher reported less often than participant teams that assistance was needed (40.0%, 20.0%, 9.3% fewer tasks needed guidance, according to the researcher.) In the final four sessions, perceived help with physical tasks elicited the largest disconnect between reporting groups.

Similarly, participant confidence to complete physical and non-BCI computer-based tasks unaided saturated at a median response of ‘very confident’ by teleBCI session 2 (Figure 2C). Session number was significantly correlated with participant-reported confidence on non-BCI computer tasks (ρ = 0.41, p < 0.0001) and BCI tasks (ρ = 0.37, p < 0.0001). The difference in perceived confidence also varied between reporting groups over time. The researcher was initially less confident of the teams’ ability to complete physical, computer, and BCI tasks unassisted, demonstrating a mean disparity in confidence scores (on a Likert scale with range 1-4) of −1.33, −0.96, −0.92. Discrepancies in confidence ratings between groups declined over sessions, approaching a minimum at session 4 (0.10, −0.19, 0.00), and maintained at relatively low values through the last session.

3.3. Data quality

Overall, electrode impedances increased with session number, and the average number of invalid electrodes (those defaulting to an impedance value of 501 kΩ) per subject increased from 0.14 (out of 8) in the first session, to a maximum of 2.14 in the second to last session. Excluding the influence of invalid electrodes, the mean impedance did not change significantly throughout the study, ranging from 22 kΩ in the first session, to 29 kΩ in the fifth session, and falling to a minimum of 20 kΩ in the seventh session. Two cap failures occurred: one due to battery malfunction, and the other a sustained difficulty reducing noise in the EEG signal. Caps were exchanged in person (n=1) or via mail (n=1).

Eye tracking occurred in all sessions of visual P300 speller use, with the exception of 6 sessions in which either P300 spelling trials were not completed or the eye tracker had a technical error, usually related to firmware updates, that prohibited gaze capture. The eye tracker was calibrated with a median gaze error of 1.4 cm from target center, although for individual users, this ranged from 0.8 cm to 3.9 cm. Median gaze error, as well as the variance of gaze around the target and the percentage of each trial that contained invalid gaze data, were all highly correlated (p<.001), however none changed over the course of the study.

Mean evoked responses to target and non-target stimuli from selected EEG channels for all subjects who used the visual P300 speller are shown in Figure 3. Three subjects transitioned to the audio-based interface after achieving inadequate EEG discriminability with the visual system. For P04, this occurred at the first visit, as oculomotor control was compromised. The transition occurred on the 7th visit for P02 and the 5th visit for P07, hence the visual P300 evoked potential data available for those early sessions showing little appreciable target response. In general, the largest evoked responses to the target stimulus occurred in channels Fz and Cz. In channel Cz, the latency to the peak target response was 229.1±32.7 ms. The amplitude of this peak response ranged from 0.99 to 11.45 μV.

Figure 3.

Common average reference evoked responses at channels Fz, Cz, Pz, and Oz to target (black) and non-target (grey) icons of the visual P300 speller. Number of trials included in the average response is shown on the right of the plot.

3.4. TeleBCI use

Home visits in addition to the first and final study sessions were required on two occasions: once for transitioning the patient from the video to audio P300 interface, and once to troubleshoot cap issues in person. The remainder of the sessions took place over the videoconferencing platform, with supplementary phone calls, often done to initiate the session, occurring in 8/15 participants by the final session. By the end of the 8 weeks of training, 9/15 patients achieved 100% accuracy in at least one P300 run, warranting the use of the notepad speller during at least one session (Figure 4). Use of notepad accessories scaled with the increasing use of this speller, with mean co-utilization rates of 46.8%, 63.5%, 41.3%, for text-to-speech, text prediction, and prepared text functions. Seven of these subjects were able to use the notepad speller to communicate with the nurse via telemedicine in the final session. The phrases with the highest effective bit rate spoken by each user, which ranged from 11-74 bits per minute, are given in Table 2. To separate the effect of word/phrase auto-completion, the selection bit rate, which varied based on the length of the trial due to differences in sequence repetition number and the interstimulus interval, ranged between 11.8 and 29.6 bits/selection across subjects. Individuals who were able to save and recall phrases, such as P08, who typed the first letter “H”, selected the number to autocomplete the phrase, and then spoke it, achieved relatively high communication rates.

Figure 4.

Use of the teleBCI transitioned from the trainer to the notepad (bordered bars), with those utilizing the notepad also using the speller accessories (open bars) at an increasing rate.

Table 2.

Participants and their best completed phrase during their clinical interaction, the number of selections to complete the phrase, the time (in seconds) per selection, selection bit rate in bits/minute per correct selection (sBR), and the effective bit rate in bits/minute (eBR).

ID	Phrase	Selections	Time/Selection	sBR	eBR
P01	HELLO_AGAIN”	9	18	17.74	23.65
P03	YES_AND_THANKS^”	20	10.8	29.57	23.65
P08	HELLO_[3-LETTER NAME]”	3	14.4	22.17	73.92
P10	NIC\_D2F	8	18	17.74	11.09
P13	A_WONDERFUL_DAY^”	11	21.6	14.78	22.85
P14	HELLO_[3-LETTER NAME]”	3	16.2	19.71	65.70
P15	NO	2	27	11.83	11.83

^{^-}save phrase, ” - speak phrase. Participant P10 was attempting to communicate ”NICE_DAY”.

The group did not experience a significant effect of session number on online P300 accuracy (p = 0.18) nor bit rate (p = 0.08), though device performance was highly variable between participants. Lack of group-wide improvement masked the gains seen by those who did eventually achieve control. In the 5 users who obtained an average accuracy of 70% during the last two sessions, a significant training effect was observed. A fitted linear model predicted a 2.6% improvement in accuracy (p = 0.018, Figure 5A) and 0.6 bit per minute increase in bit rate per session (p = 0.013, Figure 5B) for this high performing group.

Figure 5.

Accuracy (left) and bit rate (right) of P300 BCI performance, by training session. Box plots show all participants (∇, n=14) and just those with an average of ≥ 70% accuracy by the last two runs (Δ, n=5). Boxes extend from the first to third quartiles of data, with median as an open circle. Thin lines extend to the furthest data point within 1.5 times the interquartile range, and individual markers outside this range are outliers.

3.5. BCI Perceptions

Fourteen participants completed the ATDPA at the first and final sessions (Figure 6). The mean total score across participants was 44.1 at the first administration and 38.9 at the final administration. Dyad confidence (item 3) was one of the items that showed improvement. Items associated with device burden (BCI fitting into routine, having stamina to use, having the required support, and fitting in the home) all saw small group-wide drops from initial to final session. Certain items, such as the expectation of the BCI to meet communication goals and the perceived benefit of the device (items 1 & 2 of the assessment) were negatively influenced by poor performance. For individual responders, changes in total ATDPA scores ranged from −28 (less likely to use BCI at final visit) to 13 (more likely to use BCI at final visit). The stepwise linear regression procedure retained accuracy, time, and confidence predictors in the final model (F-statistic vs. constant model: 5.53, p-value = 0.017). every point increase in BCI accuracy experienced during the trial resulted in an estimated 0.26-point positive influence to ATDPA at follow-up.

Figure 6.

Initial (left column) and final (right column) responses for the 12 questions of the ATDPA (n=14).

Nurse and patient participants engaging in telehealth encounters agreed that the accessories for BCI communication, namely text-to-speech, word prediction, and prepared text, were critical to device usability. Both patient and nurse participants responded that these tools were either ‘very helpful’ or ‘somewhat helpful.’

4. Discussion

Studies that document the experience of end-users of BCI in the home are a rare exception in the BCI literature [21, 33, 14, 18, 19]. These studies describe the variety of challenges faced by the users who would benefit from a BCI system and the extent of customization needed to meet their communication needs. Training of the participant team, particularly the BCI facilitator, was identified as critical to sustained home BCI use. [14, 21]. Miralles et al. [13] describe a number of critical aspects of facilitator training which need to be improved, many of which were observed by the research team in this study. An emphasis was placed on a user-friendly interface with clear and simple steps, troubleshooting support using non-technical terms, and simple methods for performing system quality checks.

This study utilized a telemedicine model for training a group of AAC end-users and their caregivers in the setup and use of a P300 brain-computer interface. Multiple positive outcomes of training on the participant dyad were recognized, including reduced time to perform tasks, and greater confidence in procedures. Although the system functioned reliably when coached by a researcher, some participant teams continued to require ongoing guidance by the end of the study. This indicates that certain aspects of the system must be simplified without sacrificing performance.

In our study, the positive effects of training were most apparent in the capability and confidence of the caregiver to set up and operate the devices. Users saw significant improvements over the training sessions in multiple areas, including time to complete the three system task groups. Use of BCI-specific computer components, such as measuring impedances or updating the classifiers, remained of highest difficulty for individuals to complete without the help of the researchers. Further improvement may come from automating more of these tasks or at least making them more streamlined. All patients were, by the nature of study self-selection, quite technology oriented. BCI facilitators, often caregivers or family, came with a variety of technology experiences. Anecdotally, prior experience of the facilitator with Windows-based computer navigation allowed for a quick learning experience.

Training tended to follow a pattern of high user confidence in the first session (where the researcher was physically present), followed by a steep drop at the first remote visit. Over the course of eight sessions, the researcher became comparatively more confident in the abilities of the participant team, and underestimated the amount of help the team received from researcher input. At the end of the study, the majority of dyads were receiving help from the researcher to complete BCI-related tasks and there was a substantial disconnect in the perceived help needed to operate the system between the participant team and researcher. Still, both groups became more confident in the team’s ability over the study sessions.

Despite increasing confidence, we did not observe significant improvement in BCI accuracy or speed over the course of sessions, however when segmenting out users who eventually gained control of the system at an adequate level, improvement was observed to occur. This effect can likely be attributed to use of progressively robust classifiers from a greater pool of training data. Subjects were not screened for ability to use a BCI before enrollment. Therefore, we had a significant portion of subjects who were not able to gain control over the spelling functions of the P300 system by the end of the training period. This is in line with our previous research [27] showing that the nature of the disease, whether a result of physical or cognitive changes, makes successful control of a BCI less likely in ALS patients than healthy controls. The individuals who were able to communicate with the nurse using the BCI-AAC system did so during the final session. By this time, most had prepared and saved some phrases with which to communicate with the nurse.

As a result of the mixed performance, we saw a net increase in the perceived burden of teleBCI (ATDPA items of fitting into routine, having stamina to use, having the required support, and fitting in the home) from initial to final visits (Figure 6). Although this negative trend was driven by individuals who did not gain control over the BCI, this result indicates that improvements to device usability still need to be made to reduce burden on the user and family. Those who did achieve a level of control necessary to be able to communicate with the nursing team tended to rate their experience with the BCI higher and the additional accessories as very helpful.

4.1. Technical Considerations

The deployment of a complete BCI system in the homes of users presented the research team with their own learning curve. A major factor which allowed for this study to occur was the use of study-specific computers. The standardization of the computer in this study was a necessary limitation to be able to have control over the system and its components. Standardization of hardware and drivers as well as the software components for videoconferencing, email, and data transfer significantly reduced the complexity of this endeavor. Still, unplanned system and software updates, most notably of the eye tracker, caused substantial delays. The anti-virus software installed in compliance with university policy occasionally caused erratic system behavior, requiring one in-person visit and one teleconference visit to troubleshoot.

Issues with the electrode cap also arose, resulting in a mailed replacement for a faulty battery, and an in-home visit to diagnose poor quality recordings. Although not as widely utilized an indicator of data quality for active electrode EEG systems, we used electrode impedance to indicate the quality of electrode and gel application. The increasing number of electrodes with poor connection observed over sessions suggests a potential weak point in the training scheme. Beyond the initial training, not much additional guidance was given for inspecting the cap before use and making sure electrode contacts were clear. Either additional training for caregivers is needed to maintain EEG data integrity, or alternative EEG hardware that is effective for non-technician operators must be developed.

4.2. Limitations and Future Work

A limitation of this work was that the study for the most part did not include a period of independent use. Although this was planned for if users 1) achieved adequate communication with the BCI and 2) would benefit from the BCI as their primary form of communication, only one of the 9 users who eventually utilized the full functionality of the notepad speller continued using the device independently in their home following the conclusion of the study. This team continued to use the device in their home a total of 8 additional sessions over the next 7 months. Independent testing by end-users requires further modifications to the system for enhanced performance and ease of operation, some of which are described in a recent trial performed by Wolpaw et al [21].

Another limitation was that the training process, although offering the capability for remote involvement, excluded one individual who was not able to be enrolled because they were in a rural location and relied on a cellular connection for internet connectivity. The training procedures also placed a significant time burden on the researcher which was not possible to sustain for more than 4-5 subjects at a time. As shown in the results, many of the physical and non-BCI computer tasks could be learned within 2-4 sessions, while the majority of participants reported receiving guidance from the researcher on the BCI tasks at the last session. Creation of a more streamlined user interface with fewer technical hurdles (checking impedances, creating and loading the classifier) may result in a shorter training period to reach proficiency.

The discrepancy between stimulus presentation timing for subjects P1-10 and P11-15 introduces a confounding factor in the interpretation of results between these groups. Beginning with subject P11, the randomization of the interstimulus interval and the resulting lengthening of a single trial from 18 to 27 seconds could be expected to cause differences in evoked responses, accuracies, and bit rates, though none of these systemic trends were observed in this limited sample.

The device was also cumbersome, with many peripheral components compared to – for example – an all-in-one eye tracking AAC solution. Size reduction could be accomplished by exchanging the floor/table mount with a wheelchair attachment, and eliminating the need for two screens by utilizing an all-in-one model with integrated touchscreen. Gaze tracking was only peripherally utilized in this study as a dual measure of device setup proficiency and to monitor the visual attention of the user during the task. As deployed in non-patient settings, eye tracking, could be incorporated into the classification of communication intent through a hybrid BCI system [34, 35]. Single monitor solutions with integrated eye tracking would also likely reduce the complexity of setup.

4.3. Conclusions

This paper describes at-home training in P300 BCI over the course of multiple sessions, using a videoconferencing interface to facilitate interactions between the researcher and users, and cloud-based data sharing for monitoring training progress. This system was tested in 15 subjects experiencing significant limitations to their upper extremity function or speech secondary to ALS, PLS, or PMA. Over the course of the 8 training sessions, users improved on their time to setup the device and perform the spelling task. Seven subjects achieved sufficient proficiency with the basic speller system to use the feature-rich notepad speller to communicate with the study nurses over the platform. When segmenting users into high- and low-performing groups, high performers achieved a significant rate of improvement in P300 spelling accuracy of 2.6% per session.

Overall, participants found that the device was more demanding of their time, space, and stamina than they had originally anticipated. Furthermore, survey responses from the researcher showed a much higher perceived level of task independence than that perceived by the participant team. These findings demonstrate that the expectations of both groups need to be laid out and revisited throughout the training process.

The strength of this approach to BCI training is the integration of BCI components into the telemedicine infrastructure. This pairing holds the potential for a speech pathologist/assistive technology professional to remotely judge the efficacy of current devices, adjust parameter settings, and recommend alternative solutions. Furthermore, such integration would allow patients using AAC to directly interact with the clinical team when engaging in care via telemedicine.