EEG-Controlled Functional Electrical Stimulation Therapy With Automated Grasp Selection: A Proof-of-Concept Study

Abstract

Background: Functional electrical stimulation therapy (FEST) is a promising intervention for the restoration of upper extremity function after cervical spinal cord injury (SCI). Objectives: This study describes and evaluates a novel FEST system designed to incorporate voluntary movement attempts and massed practice of functional grasp through the use of brain–computer interface (BCI) and computer vision (CV) modules. Methods: An EEG-based BCI relying on a single electrode was used to detect movement initiation attempts. A CV system identified the target object and selected the appropriate grasp type. The required grasp type and trigger command were sent to an FES stimulator, which produced one of four multichannel muscle stimulation patterns (precision, lateral, palmar, or lumbrical grasp). The system was evaluated with five neurologically intact participants and one participant with complete cervical SCI. Results: An integrated BCI-CV-FES system was demonstrated. The overall classification accuracy of the CV module was 90.8%, when selecting out of a set of eight objects. The average latency for the BCI module to trigger the movement across all participants was 5.9 ± 1.5 seconds. For the participant with SCI alone, the CV accuracy was 87.5% and the BCI latency was 5.3 ± 9.4 seconds. Conclusion: BCI and CV methods can be integrated into an FEST system without the need for costly resources or lengthy setup times. The result is a clinically relevant system designed to promote voluntary movement attempts and more repetitions of varied functional grasps during FEST.

Functional electrical stimulation therapy (FEST) is an approach that has shown considerable promise in improving functional outcomes after spinal cord injury (SCI).¹ FEST uses trains of electrical pulses that produce coordinated contractions of muscles specifically selected to facilitate functional movements (eg, reaching and grasping a cup from a table). After patients attempt the desired movements for a few seconds, a therapist manually triggers the stimulation and guides the limb to ensure that the resulting movements are natural and of good quality.² In individuals with SCI, FEST has been used successfully to restore several functions including walking,^3–7 standing,^8,9 and reaching and grasping.^2,10–15 In the context of upper limb rehabilitation (reaching and grasping), FEST has resulted in some of the largest improvements in arm and hand function reported to date for individuals with SCI at different stages of recovery (ie, acute and chronic).^2,10,16–19 A detailed review of FEST for the restoration of voluntary movement can be found in Nagai et al.²⁰

Even though the mechanisms of action of FEST are not fully understood, factors that have been hypothesized to play a role include spinal synaptic plasticity²¹ and natural patterns of somatosensory and proprioceptive feedback coupled with voluntary motor commands.² In contrast to other forms of electrical stimulation, such as neuromuscular electrical stimulation (NMES), the emphasis on voluntary movement attempts and massed practice of functional movements are thought to be central to the effectiveness of FEST.^2,21

The objective of this proof-of-concept study was to develop a novel FEST system designed to further emphasize the key elements of this type of therapy through the integration of two technological solutions. First, a brain–computer interface (BCI) was used to trigger the functional electrical stimulation (FES) by detecting the patient's intention to move. This approach is intended to promote coupling between the stimulation and genuine attempts at voluntary movement. In addition, this triggering strategy does not require the presence of residual voluntary movement after paralysis or the availability of another signal indicative of voluntary movement (eg, electromyographic activity). Second, a computer vision (CV) system was used to identify the target object and automatically identify the appropriate grasp type to be produced by the FES. This reduces the need for a therapist to manually adjust device configurations (including electrode placements as well as stimulation intensities and sequences) to produce different hand movements, a process that can consume a significant portion of a therapeutic session that includes multiple targeted movements. Our intent is to increase the number of movement repetitions that can be accomplished in a session while covering the desired range of functional hand grasps. The system was developed with an explicit focus on short-term clinical applicability, such that all of its components require minimal setup and technical expertise.

Materials and Methods

Setup

FES system

We implemented a neuroprosthesis for grasping using two 4-channel programmable Compex Motion (Compex, Switzerland) noninvasive electrical stimulators,¹³ with each unit producing two different movements. The Compex Motion stimulators allowed us to have unrestricted placement of stimulation electrodes, fully programmable stimulation parameters and sequences, the capability of combining multiple stimulators in series, and the ability to use multiple modalities for triggering the situation (eg, buttons, potentiometers, physiological signals, and a BCI), providing a high level of flexibility suitable for implementing experimental neuroprosthetic systems, like the one presented here. The term neuroprosthesis refers to an electrical stimulation system that facilitates functional movements. Specifically, the device was designed to facilitate lumbrical grasp and power grasp, both produced by stimulator 1, and precision pinch and lateral pinch, produced by stimulator 2. Details on the stimulation used can be found in Figure 1. The programming of the neuroprosthesis was designed to trigger a preprogrammed sequence of movements consisting of grasp for 8 seconds followed by hand opening to release (3 seconds). The order in which each muscle was stimulated was executed automatically by the programming of the sequence. A splitter cable was used to allow for both stimulators to stimulate the same muscles (eg, both neuroprostheses required electrodes on the extensor digitorum communis muscle). Both stimulators were configured to respond to a signal provided by the BCI and CV system (described below), which allowed selecting between the stimulators and the two different stimulation sequences (ie, precision vs lateral pinch or palmar grasp vs lumbrical grasp). The system's software allowed only one stimulator to be active at any given time.

Figure 1.

Experimental setup. The computer vision (CV) system and brain–computer interface (BCI) were implemented as two independent systems that could communicate with each other through TCP/IP (transmission control protocol/Internet protocol) networking protocols. The CV system first identified objects placed on a table in front of the participant, determined the grasp necessary to manipulate it, then provided the required grasp to the BCI system. The BCI configured the functional electrical stimulation (FES) system to produce the intended movement. The FES system was implemented using two four-channel electrical stimulators (each capable of producing two different grasps). The BCI used a single EEG electrode. Triggering of the stimulation sequences started when the BCI was activated (event I in figures b, c, d, and e). Activation and deactivation of the stimulation was done gradually by ramping up (or down) its intensity. The facilitated movements included palmar grasp (b), lumbrical grasp (c), lateral grasp (d), and precision pinch (e). All stimulation sequences lasted 8 seconds and ended by facilitating hand opening. This position (opened hand) was sustained for 3 seconds after which the system would return automatically to an idle state, waiting for the next BCI activation.

FES configuration process

The location and stimulation intensity of each electrode was determined at the beginning of the experimental session. Potential electrode sites were tested by increasing the stimulation intensity in 1 mA increments and noting the resulting motor response. At each step, participants were asked if the stimulation was uncomfortable or painful. If the desired motor response was not observed, the electrodes were repositioned and tested again.

BCI system

The BCI developed for this study used one unipolar electroencephalography (EEG) channel, which was bandlimited (3 Hz and 30 Hz) and amplified using a biosignal amplifier (QP511; Grass-Telefunken, Germany) prior to its acquisition (USB-6363; National Instruments, USA). The system's graphical interface and signal processing were developed in LabView (National Instruments, USA).

The BCI continuously monitored changes in signal power, estimated using the root mean square value of a 125 ms window of the squared EEG. This signal was further bandlimited (high-pass, 0.1 Hz cutoff frequency) and processed with a moving average filter resulting in a smoothed version of the power signal. Activation of the BCI took place whenever the power of this power estimate was sustained below a predetermined threshold. This well-documented reduction in power, often referred to as event-related desynchronization (ERD), is typically observed within the alpha (8–12 Hz) and beta (13–30 Hz) EEG frequency ranges prior to, during, and while imagining voluntary movement.²² Both the activation threshold as well as the duration over which the power had to be decreased were set heuristically by the experimental team.²³

Before the experiments took place, the BCI had to be configured for each participant. To do this, each individual was asked to press a button 80 times with their (self-identified) dominant index finger while sitting comfortably and relaxed. Wrist movements were used when active finger flexion was unavailable. The pace of the button activation was set by the participants who practiced prior to the data collection to ensure that there were at least 4 seconds between consecutive switch activations. Together with the switch activations, EEG signals were band-pass filtered (0.05–40 Hz) and recorded (SynAmps RT; Compumedics, USA) at a rate of 400 samples per second simultaneously from six electrodes (F3, Fz, F4, C3, Cz, and C4 of the 10–20 electrode placement system). The recorded EEG was segmented, with each segment containing 2 seconds prior and 0.8 seconds after each switch activation. Trials that were visibly contaminated by motion artifacts were eliminated from further processing. Identification of the electrodes and EEG frequency bands displaying ERD was performed following the procedure found in Pfurtscheller and Aranibar.²² Briefly, a bank of overlapping band-pass filters between the frequencies of 4 Hz to 30 Hz was applied to the EEG recordings. The filters had a bandwidth of 2 Hz and their center frequencies were separated by 1 Hz. The filtered signals were squared, smoothed (1 second moving average filter), and averaged. The first 500 milliseconds of the average were used to calculate a baseline of every spectral component and the remaining samples were expressed as percentages of this power baseline. Statistical validation of the observed power changes for each was performed using a t statistic (p = .05) bootstrap (500 bootstraps); we resampled our data set and validated the power change 500 times for each data point (time). This process allowed us to identify the electrode(s) displaying ERD. Based on the results of this calibration process, a single electrode and frequency band was selected and used in the rest of the experiment.

CV system

The purpose of the computer vision system was to identify the target object, which determined what FES grasping pattern was most appropriate. The images were acquired using a standard RGB webcam (HD Pro Webcam; Logitech, USA) with 1920 × 1080 resolution. Image processing was performed in Matlab (version 2014a) as follows. First, the objects were segmented from the rest of the image using background subtraction. A background image consisting of the empty table was captured at the beginning of each trial. Once the trial began, the background image was subtracted from the image of the object on the table using an absolute difference. The resulting image was converted to grayscale, thresholded, and denoised using image dilation to obtain the mask of the object as seen in Figure 2. The edges of the mask were refined through an edge detection process on the corresponding region of the original image.

Figure 2.

Background subtraction used to detect the mask of the object: (a) image of the object, (b) image of the background, and (c) mask after background subtraction.

The resulting masks were used to train an object classifier. Eight objects were used, consisting of an eraser, roll of tape, cup (handle visible), pencil, book, cellphone, soda can, and tennis ball. Each object was associated with one of four hand grasps (precision pinch, lateral grip, lumbrical grasp, and palmar grasp), with two objects for each type of grasp. A training set for the classifier was created using 50 images of each object placed randomly in the image boundaries, resulting in 400 images in total. The following image features were used as the inputs to the classifier:

• Ratio of length and width of the object. The longest axis of the object was taken as its length and the perpendicular axis as the width. The ratio of length to width was used as the feature, to reduce sensitivity to scaling.

• The ratio between the perimeter and area of the object mask. This allowed the system to be invariant to scale and rotation and capture information regarding the size and shape of the object.

• The Hough transform. Canny edge detection was applied to the mask, followed by a Hough transform.²⁴ The Hough graph was summarized by summing along its distance dimension and fitting a third order polynomial to the resulting curve. The features were the polynomial coefficients, as well as the overall summation of the Hough graph over both its distance and angle dimensions.

• The curvature of the mask. The point-wise extrinsic curvature²⁵ of the mask's contour was computed through a transformation to Fourier space. The mean and standard deviation were extracted and used as features. The curvature reflects the characteristic shape and contour of the object.

These features were used to train a Random Forest classifier with an ensemble of 100 trees. The Random Forest estimated the probability of each class. If no object had a probability higher than 0.6, a secondary classifier was used to make the final determination. In that case, Speeded Up Robust Features (SURF)²⁶ were used to detect sparse interest points among a set of corner points extracted by the Harris–Stephens algorithm.²⁷ These interest points were then matched to entries in the training set,^28,29 and the best matching image was used to establish the object type.

CV+BCI+FEST integration

The classification result of the CV system was transmitted to the BCI using TCP/IP (transmission control protocol/Internet protocol)communication (ie, both systems remained independent from each other). This value allowed the BCI to determine which of the two stimulators to activate and the necessary stimulation sequence. That is, integration of the BCI and FES system was achieved using two digital signals, each connected to one stimulator, produced by the BCI. The value of this 5-volt transistor-transistor-logic (TTL) signal determined which of the two possible grasps (stimulation sequences) was to be produced by the FES system when triggered. Finally, activation of the BCI by the users' intention to move initiated the stimulation (Figure 1). The link between the BCI and the FES system could be enabled/disrupted by the experimenters at any moment through the BCI's user interface.

Experimental sequence

In each experimental trial, the CV system first captured an image of the background (ie, the table without any objects on it). Next, one of the objects—selected using a list randomized prior to the experiment—was placed on a table, which allowed the CV system to classify it. Once the object was classified, the CV system informed the BCI on the grasp to facilitate. An experimenter would then ask participants to place their hand close to the object in preparation for grasping it with the movement assisted by the FES system. The link between the BCI and the FES system was activated and the participants were asked to attempt to grasp the object. More specifically, they were instructed to imagine the kinesthetic experience of grasping the object and avoid actual movement. They were allowed to attempt the movement as long as they needed to activate the BCI (ie, there was no time-out limit), at which point the FES would be activated. This interaction design was adopted to mirror the actual procedure followed during therapy for restoration of voluntary movement after paralysis using FES. In this case, the rate at which commands are issued, a common performance metric for BCI systems, does not have the same relevance as when using BCI technology as an assistive device. The trial ended when the participants grasped, lifted, and released the object with the artificially produced movement. This was repeated 10 times for each of the eight objects.

Results

Participants

Five individuals with no neurological conditions and one man with a complete cervical SCI (self-reported C6 level, AIS B), able to use his arms and wrists with no hand movement, took part in this study. The age of the participants was 32.3 ± 9 years (mean ± SD), and four were men (Table 1). All of the participants identified themselves as right-handed.

Table 1.

Participant demographics and results

BCI configuration

Electrode C3 was identified for all participants as the most viable site to implement the BCI. Four participants used EEG frequencies in the alpha range (8–12 Hz) and two in the beta (13–30 Hz) range. The specific bandwidths used for each individual are shown in Table 2.

Table 2.

Brain–computer interface (BCI) performance figures

Classification accuracy of the CV system

The average classification accuracy of the CV system, defined as the percentage of correctly identified objects (and consequently the movement required to grasp it), was 90.8% when calculated using data from all participants. The accuracy achieved during the trial with the participant with SCI was 87.5%. Figure 3 shows a confusion matrix describing the CV system's classification performance.

Figure 3.

Confusion matrix of the computer vision system. The four categories are each of the hand grasps used in the experiment. Each column displays the result of the computer vision (CV) system, and rows show the actual grasp required for any given object. Values along the diagonal of the matrix display correct classifications. Elements outside the diagonal can be used to understand misclassified items. Results are displayed as number of trials, with corresponding percentages in parentheses.

BCI activation latencies

The average latency for BCI activation for all participants was 5.9 ± 1.5 seconds. This value was 6 ± 1.6 seconds when calculated without including the results from the participant with SCI, who had an individual latency of 5.3 ± 9.4 seconds. Individual latencies are shown in Table 2.

Overall system performance

The participants generated a total of 484 activations. Performance of the entire system was estimated by measuring the performance of each individual subsystem; a trial was considered unsuccessful if the CV system failed to recognize the object placed in front of the participant or if activation of the BCI required 15 seconds or longer. There were 78 failed trials (18%) of which 41 (53%) were the result of the CV system not recognizing the object correctly. In addition, 37 (47%) trials failed due to activations requiring 15 seconds or more. Only two trials (2%) out of the 78 failed trials displayed a simultaneous failure of both the CV and BCI (ie, activation >15 seconds) systems.

Discussion

The system presented here was developed to demonstrate how technological solutions can be combined to deliver novel upper limb therapy after SCI. The CV system allowed for a quick and transparent configuration of the FES device (normally requiring a lengthy reconfiguration for each movement to produce). The CV accuracies achieved are comparable those reported by Štrbac et al,³⁰ who have demonstrated the use of CV to configure the stimulation synergy to grasp an object. Closer inspection of our results revealed that most misclassifications occurred in trials using a ball or an eraser, likely due to shadows resulting in poor segmentation.

With respect to the latency of BCI activation, the average value (5.9 seconds) appears to be acceptable for its integration with FES therapy, in which it is not uncommon to ask patients to attempt a movement for up to 15 seconds.²⁰ However, it is important to mention that in some unique cases, individual latencies were as high as 21 seconds. It would be unlikely that this delay would be suitable for clinical use. To overcome this, we have used a button that allows triggering of the FES manually, at the discretion of the therapist.²³ It is important to acknowledge that many other BCI systems have been reported with activation latencies smaller than the ones reported here.^31,32 However, the purpose of those systems was to use the BCI as an access method to facilitate communication and/or control, for which a short response time would be important. The BCI system presented here serves as an enhancement to the therapeutic use of FES. As such, and in addition to the lack of a “time-out” period, the system has other features to support its use during therapy including the acquisition of EEG using a single electrode (reducing the complexity and setup time of the system), the fact that the users are only focused on a motor task (eg, grasping an object) and not on the actual operation of the BCI, and complete lack of training prior to operation of the system.

The proposed system was designed with a focus on clinical applicability. It is encouraging to note that the same electrode (C3) was found to be the most useful in all participants. These observations point to the possibility of a standardized, simple BCI setup that could be implemented in a clinical setting. Likewise, the CV system used readily available equipment (a webcam and desktop PC) and a small and easily collected training set. While the performance of both systems can be further improved, this study provides a strong proof of principle that these technologies can be integrated together to enhance FEST protocols in a practical and feasible manner.

Early results suggest that a BCI+FEST system can produce important recovery of function,^23,33–36 even in severe and chronic cases that would typically not be expected to undergo changes in function. As mentioned above, it is believed that the BCI+FEST combination can ensure that patients are actively attempting to perform the movements practiced during therapy, despite the BCI being limited to identifying the moment in which a patient is attempting to move, but not the exact movement that the person is attempting. Indeed, identifying specific movements with the same limb using low-density EEG recordings is one of the most difficult problems in the BCI field. Today, the therapeutic effects of FEST triggered by a BCI capable of identifying specific movement is unknown. The proposed system may serve as a testbed to gather initial data toward the understanding of this new, motor attempt–specific BCI+FEST system.

One important limitation of the work is the fact that the FES system only facilitated grasping. This limits the population that could potentially benefit from the presented system to individuals who retain reaching function. From a technical perspective, a restriction of the CV system, common among most automatic vision systems, is the fact that it can only recognize objects that were used during its training (configuration) process. However, it is not uncommon to have a limited and fixed set of objects used during FEST in a clinical environment. Another potential limitation of the study is the fact that it included only one participant with SCI. However, the work presented here was focused on the feasibility of the BCI-CV-FEST system exclusively. We expect that performance variations of the system, if any, would be specific to the population using it and not to the system itself.

Our future work will focus on the refinement of the technology with input from therapists experienced on the delivery of FEST and an eventual interventional study to determine the efficacy of the new BCI-CV-FEST presented here for upper limb rehabilitation after SCI.

Conclusion

We presented a system comprised of an FEST system controlled by CV and BCI modules. The CV subsystem is capable of recognizing the target object and its associated grip. The BCI is used to trigger the FES systems. Combined, the study demonstrates an upper extremity FES therapy system for persons with SCI, which is intended to promote voluntary movement attempts and higher numbers of varied, functional grasping tasks per session. The elements of the proposed system were designed to avoid the need for costly resources or lengthy setup, and therefore minimize barriers to translation into clinical practice.