Enhancing Communication for People in Late-Stage ALS Using an fNIRS-Based BCI System

Abstract

Objective

Brain-computer interface (BCI) based communication remains a challenge for people with later-stage amyotrophic lateral sclerosis (ALS) who lose all voluntary muscle control. Although recent studies have demonstrated the feasibility of functional near-infrared spectroscopy (fNIRS) to control BCIs primarily for healthy cohorts, these systems are yet inefficient for people with severe motor disabilities like ALS.

Methods

In this study, we developed a new fNIRS-based BCI system in concert with a single-trial Visuo-Mental (VM) paradigm to investigate the feasibility of enhanced communication for ALS patients, particularly those in the later stages of the disease. In the first part of the study, we recorded data from six ALS patients using our proposed protocol (fNIRS-VM) and compared the results with the conventional electroencephalography (EEG)- based multi-trial P3Speller (P3S). In the second part, we recorded longitudinal data from a patient in the late locked-in state (LIS) who had fully lost eye-gaze control. Using statistical parametric mapping (SPM) and correlation analysis, the optimal channels and hemodynamic features were selected and used in linear discriminant analysis (LDA).

Results

Over all the subjects, we obtained an average accuracy of 81.3% ±5.7% within comparatively short times (i.e., <4 sec) in the fNIRS-VM protocol relative to an average accuracy of 74.0% ±8.9% in the P3S, though not competitive in patients with no substantial visual problems. Our longitudinal analysis showed substantially superior accuracy using the proposed fNIRS-VM protocol (73.2% ±2.0%) over the P3S (61.8%±1.5%).

Significance

Our findings indicate the potential efficacy of our proposed system for communication and control for late-stage ALS patients.

I. INTRODUCTION

SEVERE neuromuscular impairments characterized by the locked-in and completely locked-in states (LIS and CLIS) are linked to several disabilities, including amyotrophic lateral sclerosis (ALS), one of the most common adult-onset motor neuron diseases [1]. As paralyzed patients with ALS gradually lose all voluntary motor control, assistive communication tools can substantially improve their quality of life and facilitate their care by providing a certain amount of independence. Brain-computer interfaces (BCIs) have demonstrated promising outcomes in establishing communication tools for these patients [2]-[4].

Evidence from previous research suggests that while patients in the early stages of ALS can successfully communicate through eye-tracking devices, BCIs are candidate alternative communication tools in more advanced ALS as they rely on neural controls independent of muscle movements [3], [5], [6]. However, particularly in the late stages of ALS, patients lose all types of residual motor control, including eye- gaze and vital autonomous movements such as respiratory and bulbar functions, and gradually, enter LIS and CLIS, which leads to total immobile status despite certain preserved levels of brain function [1]. Hence, many conventional non-invasive BCIs that rely on eye-gaze control and alternative solutions, such as eye-tracking systems fail to provide successful and efficient communication in the late stages of ALS [1], [6], [7].

Electroencephalography (EEG)-based BCIs are the most researched non-invasive BCI systems to establish communication for ALS patients as they can be used conveniently at patients’ bedsides. Among these systems, the P300-based oddball paradigm is the most commonly used design [8]. This paradigm exploits the P300 event-related potential (ERP), a positive response evoked following a rare target stimulus within a train of non-target stimuli, as a control signal for communication purposes through a spelling application known as the P3Speller (P3S) [8]. In the standard P3S protocol, a 6 ×6 matrix of letters is used in which each row/column is randomly intensified and the subject is instructed to mentally count the intensifications of target characters while ignoring non-target intensifications. Most common P3S-BCI systems rely on visual stimuli for the ALS patient population, as the feedback from these patients regarding auditory and tactile oddball systems is mostly reported to be tedious or monotonous [9]. However, in the later stages of the disease when the patients face ocular problems such as dual vision or impaired gaze control, standard visual ERP-based BCIs fail to perform reliably and efficiently. For example, in a study conducted by Kübler et al. [6] CLIS patients could not maintain sufficient control of a P3S-BCI. Another attempt at employing the standard P3S paradigm with a post-stroke LIS patient failed to consistently reach acceptable performance, which led the authors to change the paradigm by splitting the letter matrix and iterating the process multiple times [10].

Despite the poor user experience of auditory ERP-based BCIs, they remain a rare solution in the later stages of LIS as auditory ERPs remain intact [1], [11]. However, achieving reliable performance in late-stage LIS remains a challenge. In a study conducted by Kubler et al. [12] four ALS patients including three in the LIS failed to control an auditory-based P3S, either due to the severity of their paralysis or possible cognitive dysfunction caused by neuronal loss.

In addition to ERP-based BCIs, other candidates for BCIs relying on EEG in the later stages of ALS include paradigms based on sensory-motor rhythms (SMR), the rhythmic active-ities recorded over the sensorimotor cortex and modulated during motor imagery and actual movements. Although these systems have been reported to reach satisfactory outcomes particularly in binary classification tasks and in the earlier stages of ALS [2], successful uses of motor imagery-based BCIs in advanced ALS are rarely reported [1], [6], [7]. The ‘extinction of thought’ hypothesis was proposed as an explanation for CLIS patients’ inability to use BCI systems. This hypothesis suggests there is a loss of connection between goal-oriented imagery and feedback for CLIS patients. This disconnection strongly affects the patients’ learning process, and thus, their abilities to effectively control BCIs [6]. Furthermore, evidence points to the disruption of SMR-related neural oscillations for ALS patients [13] that might degrade their overall BCI performance. Nonetheless, a few successful studies have recently examined adding a non-motor modality, including vibro-tactile stimulation [14] and mental subtraction [15], to the conventional motor imagery task to improve binary classification performance over that achieved using a standalone motor imagery task. However, extending these binary designs to spelling applications for late-stage ALS patients remains a challenge.

Although a variety of EEG-based BCI systems have been investigated to enhance communication autonomy in late-stage ALS, none provide stable and reliable communication means for these cohorts [6], [16]. However, in addition to the existing EEG-based BCIs, functional near-infrared spectroscopy (fNIRS) has shown potential as a novel control signal for non-invasive BCI applications [17], [18]. Most fNIRS signals used in BCIs mirror cerebral oxygenation alterations in response to either the activation of the motor cortex through motor imag-ination tasks [19], [20] or the activation of the prefrontal and frontal cortices through mental tasks such as mental arithmetic operations [21], [22]. These BCIs have also been investigated as potential solutions to the open research problem of facilitate-ing communication for late-stage ALS patients [17], [18], as fNIRS systems have several advantages over EEG-based BCIs. First, fNIRS integrates information from the cortical surface by penetrating the skull, which provides higher spatial resolution than EEG (i.e., fNIRS~1 cm vs. EEG~3 cm). Second, fNIRS compensates for EEG’s low signal-to-noise ratio (SNR), likely due to its low sensitivity to artifacts. Third, it measures hemodynamic-vascular responses as another essential neural property rather than EEG’s electrical responses. Furthermore, because of its lower cost and portability compared to other neuroimaging methods measuring vascular activities, include-ing functional magnetic resonance imaging (fMRI), fNIRS is potentially advantageous for BCIs, either as a sole input modality or in combination with EEG for use at patients’ bedsides. The first investigation of an fNIRS-BCI system for ALS patients was performed by Naito et al. [17], examining 40 ALS patients of whom 17 were in the CLIS. Communication was established based on a series of “yes/no” questions and patients performed either mental arithmetic calculations or mental singing. Despite acceptable average accuracy rates in ALS patients, the authors concluded that the fNIRS-BCI applicability for CLIS was low, linking it to either lack of motivation or lowered brain activity in very late stages of the disease. Later on, Gallegos-Ayala et al. [18] presented a case study of an fNIRS-BCI to classify hemodynamic responses from an ALS-CLIS patient in response to correct and incorrect statements. Classification results significantly above chance level strongly support fNIRS as a possible candidate for communication for the ALS-CLIS patient and opened new doors for further expansion of fNIRS-based BCIs for CLIS- ALS patients. Despite a few studies showing the potential of fNIRS-BCIs as binary communication tools for late-stage ALS, there are still ongoing challenges on the feasibility and practicality of these communication systems for these cohorts [17], [18]. Thus, considerable work is still required to advance existing binary-choice fNIRS-based BCIs for ALS patients, particularly those in the advanced stages. Extending binary fNIRS-BCI systems to create practical spelling tools in the later-stages of ALS is a challenge that, to date, no fNIRS study has addressed.

Following our previous work assessing the feasibility of an fNIRS-based communication system with healthy subjects [23], in this study, we expanded our work to examine the proposed framework on ALS patients, emphasizing the late- stage of the disease. As only a few studies have shown the potential for fNIRS-based binary communication in the late- stages of ALS, the first aim of this study was to examine the feasibility of an fNIRS-BCI system for this cohort through a new spelling platform. Given that conventional P3S-BCIs fail in the final stages of the disease due to ocular problems in these patients, we investigated whether, through a novel extension of the standard spelling paradigms, metabolic-fNIRS information gained using our proposed system could compensate for this shortfall. The next aim of the study was to examine the longitudinal performance of our proposed system in the later stage of the disease when the patients lose their eye-gaze control to assess the feasibility of its further development for daily use.

II. METHODS

A. Subjects

Six participants with ALS (age 57.0±15.7 years, one female) were recruited for this study (See Table I). Their functional rating scale-revised (ALSFRS-R) score was 11.6±9.5 (Mean±SD) on a 48-point scale, where 48 represents normal function in activities of daily living (ADL) and 0 represents a complete loss of function [24]. All participants had at least some post-secondary education. Three patients (ALS- 1, 2, and 4) had gastrostomies as well as tracheostomies. ALS-1 was in the late-stage LIS with no objective means of communication due to the loss of all muscle control, including eye movement. The only form of communication for this patient was an idiosyncratic and error-prone yes/no pupil dilation only his caregiver could subjectively read. This approach deteriorated over the course of the recordings to the extent that it lost reliability as a means of communication. Two other patients with artificial ventilation (ALS-2 and 4) used eye-tracking devices to communicate with others. ALS-3 could still move his index finger and make non-verbal sounds to sustain minimal communication abilities. The ability to talk in the patient ALS-5 was intact, though non-facial muscle movements were lost. Similarly, ALS-6 could communicate through normal speech and barely move a joystick with one hand. All participants were tested in either their homes or care centers. The study protocol was approved by the Institutional Review Board (IRB) of the University of Rhode Island (URI) and all subjects provided informed consent or assent for the study and received financial compensation.

TABLE I.

Participant’s Demographic Information

Subject No.	Age	Sex	ALSFRS-R (max 48)	Education Level	Means of Communication
ALS-1	29	M	0	College degree	No reliable means
ALS-2	55	M	4	Graduate degree	Eye-tracking
ALS-3	70	M	14	Some post-secondary	Non-verbal sound
ALS-4	67	M	7	College degree	Eye-tracking
ALS-5	69	F	23	College degree	Verbal
ALS-6	52	M	22	Some post-secondary	Verbal
Mean±SD	57.0±5.7	-	11.6±9.5	-	-

B. Experimental Protocol

In this study, we developed a new Visuo-Mental paradigm. The proposed paradigm is an adapted version of the conventional oddball paradigm and relies on a combination of visual tasks and mental (arithmetic) operations, which additionally provoke hemodynamic changes in the pre/frontal areas [22], [25]. Incorporating the mental calculation can hypothetically compensate for ALS patient’s impairments in the visual task, in particular in the later stages of the disease, when the patients lose fine eye-gaze control. This study contained two parts; in the first part (part I) of the study, all subjects participated, first, in a familiarization session to get trained on the experimental protocol and then participated in the main experimental session. Subjects were exposed to visual stimuli through a 23” LCD monitor with the help of a holder at their bedside. Each experimental session consisted of three runs (run 1, run 2, and run 3) as described below.

In the first run, for comparison with our proposed paradigm, subjects performed the standard P3S paradigm in which a picture of a celebrity face was superimposed over letters at intensification time, as flashing familiar faces enhances corresponding ERPs and improves BCI performance [26]. Following the standard P3S protocol, a 6 × 6 matrix of letters was used in which each row/column was randomly intensified for 93.75 ms followed by a 62.5 ms inter-stimulus-interval (ISI). Each subject was instructed to mentally count the intensifications of target characters while ignoring non-target ones in the mode. In total, for the P3S paradigm, there were 14 target characters with 10 row/column flashes (20 trials in total) for each.

In the next two runs (run 2 and run 3), subjects performed our proposed paradigm, known hereafter as the Visuo-Mental (VM) paradigm, where the celebrity picture was replaced with a 2 × 2 matrix of digits (Fig. 1 right). The subjects were then instructed to perform mental calculations using the matrix of numbers intensified over the target character. The calculation included a simple addition/subtraction either diagonally (at the first flash) or vertically (at the second flash) within the intensified matrix followed by doubling the higher result. The stimulation time was set to 300 ms, and the inter-stimulation interval (ISI) was set to six seconds. The ISI in the VM paradigm was higher than that for the P3S to allow for the emergence of intrinsically slow hemodynamic responses. In total, for the VM paradigm, there were 14 target characters with one row/column flashes (single-trial) for each character.

Fig. 1.

Left: The fNIRS montage. Right: The proposed Visuo-Mental (VM) paradigm.

In the second part (part II) of the study, we longitudinally recorded four additional test sessions from the subject in the late-stage LIS (ALS-1), in successive months (in total, five sessions over five months) to examine the usability of our proposed system as the patient’s disease progressed over time and the patient entered the CLIS.

C. Data Acquisition

While, for the P3S paradigm, we only recorded EEG data as a comparative reference, in the VM paradigm, we recorded both hemodynamic (fNIRS) and electrophysiological (EEG) activities simultaneously, using a montage containing fNIRS optodes (Fig. 1-left) and EEG electrodes on a single cap as shown in supplementary materials. Although the focus of this work was on developing an fNIRS-based communication system, we also investigated how incorporating EEG features associated with the VM paradigm to the fNIRS features could potentially enhance the performance. However, considering the low signal-to-noise (SNR) characteristics of EEG, the single intensification of the visual stimuli in the VM task was not expected to provide sufficient additional information relative to the multi-trial setup of the standard P3S paradigm. Thus, this study focused primarily on the fNIRS-based BCI, and the analyses/results associated with EEG and the hybrid (EEG+fNIRS) data have been reported in the supplementary material, to avoid distraction from the main focus of this work.

fNIRS data were recorded using the NIRScout system (NIRx Inc.) with two near-infrared wavelengths (760 nm and 850 nm) and digitized at a sampling rate of 7.81 Hz. NIRScout relies on the continuous wave properties of light passing through the tissue and the scalp. As depicted in Fig. 1-left, we used six emitters and five detectors (for a total of 14 fNIRS channels) to cover frontal and prefrontal areas reported to mirror a wide range of mental tasks including mathematical operations [22], [25]. The emitters were located at F3, Fz, F4, AF3, AF4, and Fpz, while the detectors were placed at F1, F2, AFz, Fp1, and Fp2, according to the Modified Combinatorial Nomenclature (MCN) positioning system.

EEG data were recorded using the g.USBamp amplifier (g.tec Medical Technologies) with a 256 Hz sampling rate. Following the optimal channel set suggested by Krusienski et al. [27] for the P3S setup, we chose eight EEG channels: Fz*, Cz, P3, Pz, P4, PO7, PO8, and Oz, with FAF2, the nearest position to Fz in the 128-channel montage, denoted as Fz* due to the fNIRS optode located at Fz. All experimental protocols, data acquisition, and stimulus presentations were controlled using BCI2000 and NIRStar software.

D. Signal Analysis

EEG data were bandpass filtered at 0.5–30 Hz. For the P3S paradigm, data were segmented into epochs of 800 ms post-stimulus and then downsampled to 20 samples (with the factor of 11) [28] for later use as the classification features.

fNIRS data were band-pass filtered at 0.01–0.2 Hz to mit-igate physiological noises caused by respiratory and cardiac activities [29]. The modified Beer-Lambert Law was used to calculate changes in the concentrations of oxygenated hemoglobin (HbO2) and deoxygenated hemoglobin (HbR) using recorded alterations in the reflected light attenuation. The exported data were then up-sampled to 256 Hz for synchronization with the EEG data and segmented into three different time-window: [0–2], [0–4], and [0–6] sec post-stimuli, for both target (attended) and non-target (unattended) stimuli.

To select the optimal fNIRS channel set, Statistical Parametric Mapping (SPM) [30] was conducted using a General Linear Model (GLM)-based predictive model, Y = Xβ + ε. We then obtained least-square estimates of β weights, which quantify the contribution of defined hemodynamic response function (HRF) as basis functions to the time-series Y, where Y is a matrix of hemodynamic data and X is the design matrix, constituted by the total number of HRF models. Multiplying β by X produces maximally accurate approximations of hemodynamic time series using linear combinations of designmatrix columns, with e which is the “error” or “residual” term” minimized through the GLM model. In this study, channel optimization was done separately on the training data set in which the corresponding target and non-target HbO2 epochs were used to extract the coefficients, followed by the SPM method to calculate the t-scores indicative of the discrimination level between the target and non-target hemodynamic classes. For further analysis, we then selected two channels with the highest absolute t-score as the two most representative channels for each patient.

The following features (13 in total) were then extracted from each epoch: the integral of the HbO2, HbR, HbT (HbT=HbO2+HbR), (HbO2-HbR) concentration changes, the maximum HbO2 concentration change, the maximum absolute change in HbO2 concentration, four down-sampled values of HbO2 (with a factor of 128), and the three slopes between those samples. The features were extracted following our previous work on the healthy cohort [23], and other fNIRS studies of mental arithmetic tasks [22], [25], [31]. Through a Pearson’s correlation analysis, six best features (per channel) were then selected for further analysis, as the optimal fNIRS features.

E. Classification

Seventy percent of the data was used for training, and the rest was kept as a testing set. As the target and non-target samples were unbalanced (i.e., there were two targets for every ten non-targets), we chose the first 70% targets and non-targets as the training set to ensure both groups had similar representation in the training set and avoid bias. We used linear discriminant analysis (LDA) to evaluate performance as LDA is a common method for characterizing hemodynamic responses, especially those associated with mental arithmetic tasks [31].

F. Performance Evaluation

Performance was evaluated using three metrics: accuracy (Acc), sensitivity (Sen), i.e., true positive rate (TPR), and specificity (Spe), i.e., true negative rate (TNR). In the fNIRS- VM classification, we evaluated performance using different numbers of optimal features, with the two most correlated features used first as the initial feature set. Then, the feature number increased step-by-step to include all optimal features.

The longitudinal evaluation of the proposed system on the patient in late-stage LIS (ALS-1) was performed in which receiver operating characteristic (ROC) curves [32] were constructed using 20 different thresholds (to classify target vs. non-target), changing stepwise (with steps of 5%) from 0 to 1 across three time windows ([0–2], [0–4], and [0–6] sec). For each threshold value, class labels were re-evaluated and new sensitivity (TPR) and false positivity rate (FPR=1-specificity) measures were calculated. Thus, each threshold resulted in a new pair of TPR and FPR values represented by a point in the ROC space. These points constitute the ROC curve whose area under the curve (AUC) was then calculated to evaluate the performance for each feature set and each session/window. Considering that AUC value of 0.5 is associated with random chance estimation and one with ideal performance, we used the criteria of the sum of the AUCs over time windows to determine the feature set resulting in the highest performance. The reported performance metrics, in the following sections, are the results obtained from the feature sets belong to the highest AUC sum.

III. RESULTS

Here, we present our results in two parts. In part I, the fNIRS-VM results for all the ALS patients (one session per patient) are reported. Then, in part II, we discuss data collected longitudinally from our only patient in the late-stage LIS (ALS-1). Results from the EEG-VM and Hybrid-VM data appear in the supplementary material.

A. fNIRS Responses (Part 1)

Fig. 2 illustrates the averaged normalized post-stimuli HbO2 responses for the target (solid lines) versus non-target (dashed lines) epochs for all the ALS patients (the corresponding HbR plots are presented in the supplementary material—Fig S3). For each patient, the two left plots represent the four leftmost pre/frontal fNIRS channels, i.e., AF3-Fp1 (chi), AF3- AFz (ch2), AF3-F1 (ch3), F3-F1 (ch4), while the middle, and the right plots, respectively, show Fpz-Fp1 (ch5), Fpz- Fp2 (ch6), Fpz-AFz (ch7), Fz-AFz (ch8), Fz-F1 (ch9), Fz- F2 (ch10), and the four rightmost pre/frontal channels, i.e., AF4-Fp2 (ch11), AF4-AFz (ch12), AF4-F2 (ch13), and F4- F2 (ch14). Due to the intrinsically slow fNIRS response, each plot displays 12 seconds of the data (~2 × ISI = 12 sec) after stimulus onset to display a complete hemodynamic time evolution pattern over larger segments of data. Task performance, however, is considered within the yellow-colored portion (ISI = 6 sec) representing the segment considered for further classification analysis. In most channels, we typically observed substantial evoked deflection in the hemodynamic patterns for the target epochs compared with the non-targets, particularly in the first six seconds, which is the segment used in our analysis. However, the hemodynamic patterns and active channels’ topography differed among patients. In ALS-1, the patient in late-stage LIS, we observed two main hemodynamic patterns. One pattern was ascending, with a maximum peak at 3.55±1.66 sec post-stimulus in the target epochs; the other pattern was descending approaching a minimum 6.07±2.55 sec post-stimulus. In ALS-2, the initial dip is typically absent or very short, while a single peak occurred at an average of 6.38±1.71 sec post-stimulus. The left pre/frontal channels, i.e., AF3-Fp1 (ch1), AF3-AFz (ch2), AF3-F1 (ch3), F3-F1 (ch4), and AF4-Fp2 (ch11) illustrate this pattern for ALS-2. In ALS- 3, we observed generally weaker hemodynamic responses than other patients and no consistent pattern across channels. ALS- 4 also demonstrated clear target responses in most channels. However, while several frontal channels (e.g., AF3-F1 (ch3), F3-F1 (ch4), Fz-F1 (ch9), Fz-F2 (ch10), AF4-F2 (ch13), and F4-F2 (ch14), showed positive evoked deflections with the average peak at 5.05±0.27 sec post-stimulus, the other channels mostly showed descending patterns. In ALS-5, all channels, except Fpz-Fp1 (ch5), began with an initial dip, on average, approaching a minimum at 1.28±0.52 sec post-stimulus in the target epochs followed by a rise peaking at 6.83±0.80 sec post-stimulus. ALS-6 had a similar pattern to ALS-2 across the channels, with a shorter initial dip, approaching a minimum at 1.40±1.08 sec post-stimulus and a rising average peaking at 5.32±0.71 sec.

Fig. 2.

Averaged normalized HbO2 levels for targets (solid lines) versus non-targets epochs (dashed lines) for all ALS patients. For each patient, there are seven plots, i.e. two left, three middle, and two right representing the pre/frontal fNIRS channels (denoted by numbers based on the order represented in Fig.1- left). Each plot shows the hemodynamic response of two paired channels (in the neighboring cortical region), in red and blue, for 12-seconds post-stimulus segments. The colored area is the main time segment where the mathematical task was performed by the subjects.

Fig. 3 illustrates the 3D t-statistic topographical maps, for each patient, generated through SPM analysis over HbO2 responses in [0–6] sec post-stimulus windows to demonstrate the statistical difference between target and non-target epochs over the head. Note that SPM analysis was performed only on the training sets to select the two most representative channels, based on their statistical salience, prior to the feature extraction. In ALS-1, the SPM analysis revealed the highest absolute t-score for channels Fpz-Fp2 (t-score=8.20) and F3-F1 (t- score=5.61). In ALS-2, all channels showed positive t-scores representing higher positive deflection of target hemodynamic responses compared to non-targets, with the highest t-score at channel Fz-AFz (t-score= 19.65). In ALS-3, we did not observe any substantial discrimination in most channels except in Fpz-AFz (t-score=−13.69) and F3-F1 (t-score=−10.95) with negative t-values which support the HbO2 patterns demonstrated in Fig. 2 for this participant. ALS-4 had general higher positive t-scores in the frontal channels, with the highest scores at AF4-F2 (t-score= 12.76) and F3-F1 (t-score=9.13), while he showed corresponding highly negative t-scores in the prefrontal channels Fpz-AFz (t-score=−10.59) and AF4-AFz (t-score=−10.55). ALS-5 showed relatively higher negative t-scores at AF3-Fp1 (t-score=−11.50) and AF4-Fp2 (t-score=- 9.45) compared with other channels. For ALS-6, as expected from the averaged HbO2 patterns (Fig. 2), positive t-scores were observed in all frontal and prefrontal channels with the highest t-score at F3-F1 (t-score= 13.05).

Fig. 3.

3D t-statistic topographical maps, for each patient, generated through SPM analysis over HbO2 concentrations in [0–6] sec post-stimulus windows. Each channel’s t-statistic appears as a colored segment.

B. Classification Performances (Part I)

Table II provides detailed classification performance metrics, accuracy (Acc), sensitivity (Sen), and specificity (Spe), for the VM paradigm using fNIRS features alone (fNIRS- VM) over the [0–2], [0–4], and [0–6] sec post-stimulus time windows in comparison with the standard P3S paradigm for each subject. Over all subjects, the accuracy (81.3%±5.7%), sensitivity (72.9%±13.3%), and specificity (82.9%±7.0%) were greatest in the [0–4] sec window compared with all the other time windows in the fNIRS-VM protocol. The average accuracy in the first 4 sec time window was about 7.0% higher than the P3S protocol, with an average accuracy, sensitivity, and specificity of 74.0%±8.9%, 58.0%±14.0%, and 78.0%±7.6% respectively. In general, the fNIRS-VM classification accuracies were over 70.0% for all subjects in all time windows, except for ALS-3 in the [0–6] sec window (Acc=68.0%). Notably, two subjects, ALS-4 and ALS-1, had superior classification accuracies in the fNIRS-VM protocol than the P3S in all time windows. ALS-4’s maximum accuracy was 92.0% within the [0–4] sec window in the fNIRS- VM protocol, with 75.0% sensitivity and 95.0% specificity, compared to 66.0% accuracy, 42.0% sensitivity, and 71.0% specificity in the P3S. ALS-1 achieved a maximum accuracy of 76.0% with respective sensitivity and specificity of 62.5% and 79.0% in both the [0–2] and [0–4] sec windows, compared to 60.0% accuracy, 44.0% sensitivity, and 64.0% specificity in the P3S protocol. ALS-2, ALS-5, and ALS-6 had comparable performance outcomes in both the fNIRS-VM and the P3S protocols. ALS-3, however, had higher performance in the P3S with 84.0% accuracy compared with that of 78.0% maximum accuracy achieved within the [0–4] sec window in the fNIRS- VM protocol.

TABLE II.

Performance Metrics for the VM Paradigm Using FNIRS Features Alone (FNIRS-VM) Over Three Post-Stimulus Time Windows in Comparison With the Standard P3S Paradigm for Each Subject

Protocol		ALS-1			ALS-2			ALS-3			ALS-4			ALS-5			ALS-6			Average ± SD
	Sec╲%	Acc	Sen	Spe	Acc	Sen	Spe	Acc	Sen	Spe	Acc	Sen	Spe	Acc	Sen	Spe	Acc	Sen	Spe	Acc	Sen	Spe
fNIRS-VM	0–2	76.0	62.5	79.0	76.0	62.5	79.0	72.0	75.0	71.0	90.0	75.0	93.0	78.0	50.0	83.0	76.0	75.0	76.0	78.0±5.7	66.7±9.3	80.2±6.7
	0–4	76.0	62.5	79.0	86.0	75.0	88.0	78.0	88.0	76.0	92.0	75.0	95.0	78.0	50.0	83.0	78.0	88.0	76.0	81.3±5.7	72.9±13.3	82.9±7.0
	0–6	72.0	62.5	74.0	86.0	50.0	93.0	68.0	50.0	71.0	82.0	75.0	83.0	82.0	50.0	88.0	72.0	88.0	69.0	77.0±6.6	62.5±14.4	79.8±8.9
P3S	–	60.0	44.0	64.0	81.0	69.0	84.0	84.0	71.0	86.0	66.0	42.0	71.0	83.0	74.0	85.0	71.0	48.0	75.0	74.0±8.9	58.0±14.0	78.0±7.6

C. Classification Performances (Part II)

Fig. 4 illustrates classification accuracies for both the fNIRS-VM and the P3S protocols in all five sessions’ data recorded from the only late-stage LIS patient (ALS-1). Table III provides three performance metrics of sensitivity, specificity, and AUC values of ROCs, constructed using stepwise changing of labeling threshold for each window across different sessions (shown in Fig. 5). Overall, in all time windows across all sessions, fNIRS-VM accuracies were superior compared with the P3S accuracies for this patient.

Fig. 4.

Classification accuracies in the fNIRS-VM protocols (gray shaded) in three time windows ([0–2], [0–4], and [0–6] sec) compared with the accuracy (white) of the standard P3S for the longitudinal data recorded from the late stage LIS patient (ALS-1) over five sessions.

TABLE III.

Performance Metrics for the Late-Stage LIS Patient (ALS-1), Comparing FNIRS-VM and P3S Across Longitudinal Recording

Protocol		Session 1			Session 2			Session 3			Session 4			Session 5			Average ± SD
	Sec╲%	Sen	Spe	AUC	Sen	Spe	AUC	Sen	Spe	AUC	Sen	Spe	AUC	Sen	Spe	AUC	Sen	Spe	AUC
fNIRS-VM	0–2	62.5	73.8	0.52	87.5	73.8	0.61	50.0	76.2	0.50	87.5	69.0	0.77	50.0	71.4	0.54	67.5±16.9	72.9±2.4	0.59±0.11
	0–4	62.5	73.8	0.53	75.0	71.4	0.61	62.5	73.8	0.56	75.0	69.0	0.71	37.5	83.3	0.48	62.5±13.9	75.2±3.6	0.58±0.09
	0–6	62.5	66.7	0.53	75.0	71.4	0.57	75.0	71.4	0.64	75.0	61.9	0.64	50.0	69.0	0.53	67.5±10.0	71.4±5.8	0.58±0.06
P3S	–	44.0	64.0	–	50.0	63.0	–	60.0	64.0	–	41.0	66.0	–	41.0	69.0	–	47.2±7.2	65.2±2.1	–

Fig. 5.

The ROC curves for the fNIRS-VM protocol in three time windows for all five recording sessions from the late-stage LIS patient (ALS-1).

Over all sessions, the accuracy (73.2%±2.0%) was greatest in the [0–4] sec window (with sensitivity and specificity of 62.5%±13.9% and 75.2%±3.6% respectively) compared with all the other time windows in the fNIRS-VM protocol. The average accuracy in the first 4 sec time window was about 11.0% higher than the P3S protocol, with an average accuracy, sensitivity, and specificity of 61.8%±1.5%, 47.2%±7.2%, and 65.2%±2.1%, respectively (Fig. 4 and Table III).

All AUC values shown in Fig. 5 exceeded 0.5, indicating higher than chance performance in all time windows and all sessions. The maximum AUC was observed in the [0–2] sec window, with an average of 0.59±0.11. Despite disease progression for ALS-1, which was expected to cause AUC to decline over time, the AUCs fluctuated over the sessions and did not demonstrate a consistent descending pattern.

IV. DISCUSSION

In this study, we developed a new fNIRS-based BCI system in concert with a proposed Visuo-Mental paradigm to facilitate communication for ALS patients, particularly those in the later stages of the disease. The proposed paradigm is an adapted version of the conventional oddball paradigm and relies on a combination of visual tasks and a set of mental (arithmetic) operations. The calculation task is reported to be reflected in pre/frontal hemodynamic changes [22], [25] and temporal ERP-EEG and spectral-EEG power features [33] utilized, either in a single- or multi-modal manner, to classify data epochs as belonging either to the target or non-target class. As the few existing fNIRS studies on ALS patients have mainly examined binary protocols [18], we have extended these binary paradigms to an oddball-based platform in which the row/column intensification of a matrix of alphanumeric letters can potentially be expanded further to a spelling protocol. Such an approach will be especially helpful for ALS patients in later stages of the disease when existing eye-tracking systems and most conventional EEG-based BCIs become impractical.

Notably, ALS-1, who was in the late-stage LIS state without gaze control, had highly encouraging longitudinal performance relative to his P3S outcomes. Among our visually intact patients, our results demonstrated acceptable accuracies (above the chance level of 70% [34]) in most time windows. However, the proposed fNIRS-VM system was not competitive with conventional P3S, as these patients could effectively use standard EEG-based BCIs and/or eye-tracking systems. It is noteworthy to mention that ALS-4, who had no substantial ocular problems, achieved considerably better fNIRS-VM performance than P3S performance. This result may relate to head positioning difficulties that cause the patient to lose an efficient view of the letters, which also degraded his performance with his eye-tracking system. Interestingly, this issue did not affect his fNIRS-VM performance, since the VM paradigm relies more on the mental task and less on visual competence than the P3S. On average, our results showed ALS patients achieved comparable performance utilizing the proposed fNIRS-based BCI to existing relevant BCI studies used with healthy cohorts. These outcomes are promising and demonstrate the feasibility of fNIRS-based BCI systems to restore communication and control for people with severe motor deficits.

Our findings of hemodynamic deflections during mathematical operations for ALS patients align with the hemodynamic patterns associated with arithmetic calculations reported on healthy cohorts [35]. In a preliminary spelling application, Schudlo et al. [22] reported an average accuracy of 77.4% when one letter out of three was selected through a mental arithmetic operation. However, this performance was achieved through a protocol that included 20 seconds for the mathematical task followed by 12 seconds of rest. Power et al. [21] used the same timing (20 seconds for the task separated by 12 seconds of rest) in a similar mental arithmetic task, where they reported an average accuracy of 72.6%. However, in practical BCI applications, this paradigm is highly time-consuming and therefore inefficient in daily routines, particularly for spelling applications. While our proposed VM paradigm only relies on single-trial responses, we achieved acceptable performance in comparatively short time windows, even within [0–2] sec. This rapid capture of hemodynamic responses can be attributed to the detection of the initial dip at the beginning of the hemodynamic activities in response to the mental aspects of our proposed paradigm, also reported in other studies [36]. Although fNIRS-based BCI studies predominantly focus on features related to the elevation of hemodynamic responses, very few studies utilize the initial dip to characterize hemodynamic responses. Among this handful of studies, Khan and Hong [37] were the first to report acceptable accuracy using features extracted from the initial dip, utilizing [0–2] sec of post-stimulus hemodynamic activity for a four-command decoding BCI paradigm. Given that proper fNIRS feature selection can decrease the necessary post-stimulus window size (down to 2.4 sec), which is needed to reach acceptable accuracy levels [38], our stepwise selection of optimal features might have contributed to our acceptable performance within relatively short time windows. The accuracies achieved in the fNIRS-VM paradigm by the late-stage LIS patient showed that the cognitive dysfunctions, reported to be parallel with the motor degenerations in the pathogenesis of ALS disease [6], [33], did not play a role in eliminating pre/frontal cortical hemodynamic activities to the proposed mental task. This result aligns with the implications noted by Fomina et al. [39], who posit that maintaining neural modulations associated with goal-oriented thinking tasks is present in the advanced stages of ALS. Therefore, although some cognitive dysfunctions may be associated with the progress of the disease, higher cognitive processing (e.g. mental calculations) evoke sufficient hemodynamic responses in late-stage LIS for potential employment as translational means to advance existing BCI-based communication systems for these cohorts. Moreover, despite the inattentiveness reported in some alternative P3S-based BCI designs, such as auditory oddball systems [9], [12] for late- stage LIS patients, the rapid (300 ms) matrix intensification in our proposed VM paradigm helps keep subjects alert, and thus, reduces the possibility of inattentiveness. While this short intensification period may not necessarily allow subjects to capture all the digits in an overlaid matrix, it evokes sufficient responses to support acceptable classification performance, as it augments more cognitive components, including arousal, memory (retrieving the rapidly presented digits), and decisionmaking (picking a calculation strategy or deciding between adding diagonally or vertically) than the core mental calculations can alone. We additionally provided instructions for when subjects could not retrieve all the digits, asking them to replace missing digits with pre-assumed numbers and perform some calculation without regard for mathematical accuracy. This way, while we reduce the (emotional) pressure for accurate calculations on the subjects, we ensure engagement of the cognitive components needed for classification procedure.

Analyzing EEG-VM performances showed that while including later ERP components and spectral features associated with the proposed single-trial (single row/column inten-sification) VM paradigm provided comparable results to the multi-trial P3S, the inclusion of the EEG features associated with mental arithmetic task did not improve performance when compared with the P3S (Table S2). In particular, employing these features did not improve performance for the late-stage LIS patient (ALS-1), meaning they could not compensate for this patient’s gaze control issues as the fNIRS features could. Moreover, the hybrid features provided only a slight performance improvement over fNIRS features alone, meaning that simple concatenation of fNIRS and EEG features did not provide considerable gain over single modal fNIRS features. Thus, advanced fusion methods are needed for future hybrid investigations.

Further modifications to our experimental design can be made to simplify the mathematical task, potentially using repetitive subtraction, a proven effective basis for fNIRS tasks [22], [25]. The ISI can also potentially be shortened, as classification performances were acceptable over shorter time windows (i.e., [0–2] sec). The other limitation of this work was a relatively high variance in the sensitivity measure in the VM paradigm, which was related to the low number of targets in our testing data sets. We were limited in the number of characters to-be-spelled, as more letters would have made the run too long and tiresome for the patients. The shorter ISI helps include more letters in each run (in the same amount of time), and thus, allows for more to-be-spelled characters.

Another important limitation of this study was lack of adaptation of multi-distance methods in our recording system (NIRx) to cancel physiology-based systemic and non-task related interference in the fNIRS data due to the passage of light through the superficial layers of the head [40]. This is mainly done using short source-detector separation channels while recording the data [41].

Implementation of more efficient classification techniques, including artificial neural networks (ANN) can also improve performance. ANN has been reported to properly capture hemodynamic activities in mental arithmetic tasks compared to alternative classification methods [35]. Moreover, since our evaluations were limited to offline analysis, the next step of this work will be extending the proposed system in a real-time spelling protocol with a larger sample of patients in the late stages of LIS. Additionally, as locked-in syndrome is not confined to ALS, employment of the developed system in diseases with similar symptoms, including minimally conscious state (MCS) can provide a new level of autonomy for a broader range of non-communicative patients.

V. CONCLUSION

Broadly translated, our findings indicate the feasibility of developing BCI systems relying on fNIRS data for communication and control in ALS patients, particularly, those in the later-stages of their disease. The oddball-based matrix (row/column) nature of the stimulus presentation in our proposed paradigm provides the grounds for an extension to a spelling system that is lacking among most previously developed binary fNIRS-based BCI systems tested on ALS patients. In addition, our single-trial design relies less on gaze control and more on the mental tasks, unlike the standard P3S design with multiple trials which makes the visual task particularly challenging to accomplish in late-stage LIS. Furthermore, our results demonstrated acceptable performances in a relatively short time window (<4 sec) indicating the efficacy of short event-related hemodynamic changes in the future real-time fNIRS-based BCI communication systems. Finally, the rapid (300 ms) matrix intensification in our proposed paradigm helps keep subjects alert and thus reduces the possibility of inattentiveness.

Overall, the outcomes from this study suggest a new direction toward developing fNIRS-based BCI protocols for late-stage ALS. The proposed experimental paradigm can be further extended to a real-time spelling system, bringing advantage to LIS patients who have lost their fine eye-gaze control.