EEG and EMG-based human-machine interface for navigation of mobility-related assistive wheelchair (MRA-W)

Abstract

The control of human-machine interfaces (HMIs), such as motorized wheelchairs, has been widely investigated using biopotentials produced by electrochemical processes in the human body. However, many studies in this field sometimes overlook crucial factors like special users' needs, who often have inadequate muscle mass and strength, and paresis needed to operate a wheelchair. This study proposes a novel solution: an economical, universally compatible, and user-centric manual-to-powered wheelchair conversion kit. The powered wheelchair is operated using a hybrid control system integrating electroencephalogram (EEG) and electromyography (EMG), utilizing an LSTM network. It uses a low-cost electroencephalogram (EEG) headset and a wearable electromyography (EMG) electrode armband to solve these constraints. The proposed system comprised three crucial objectives: the development of an EEG-based user attentive detection system, an EMG-based navigation system, and a transform conventional wheelchair into a powered wheelchair. Human test subjects were utilized to evaluate the proposed system, and the study complied with accepted ethical guidelines. We selected four EEG features (p < 0.023) for the attentive detection system and six EMG features (p < 0.037) to detect navigation intentions. User attentive detection was achieved at 83.33 (±0.34) %, while the navigation intention system produced 86.67 (±0.52) % accuracy. The overall system was successful in reaching an accuracy rate of 85.0 (±0.19) % and a weighted average precision of 0.89. After the dataset was trained using an LSTM network, the overall accuracy produced was 97.3 (±0.5) %, higher than the accuracy produced by the Quadratic SVM classifier. By giving older and disabled people a more convenient way to use powered wheelchairs, this research helps to build ergonomic and cost-effective biopotential-based HMIs, enhancing their quality of life.

1. Introduction

With increased independence and an improved quality of life, cutting-edge technology in assistive devices [1] has completely transformed the lives of people with mobility limitations [2]. Mobility-related assistive wheelchairs, one of these assistive devices, are crucial in enhancing the mobility of individuals with impairments. Researchers, engineers, and medical experts always work to improve these wheelchairs' usability and usefulness as technology develops [3]. Human-machine interfaces (HMI) refers to the point of interaction between a human and a machine. HMIs can take various forms, including physical interfaces like buttons and levers [4], graphical user interfaces (GUIs) on screens [5], voice recognition systems [6], gesture-based interfaces [7], and more [8]. The creation of biopotential-based HMIs [9] that enables users to operate mobility-related assistive wheelchairs using their physiological signals, such as electroencephalograms (EEG), electromyograms (EMG), and other biopotential measurements, is an impressive route for advancement in this sector [10].

By creating an automated wheelchair that is simple to maneuver, this research intends to help paresis sufferers with their daily routines. Paresis, commonly called partial paralysis, is when the muscles become weak due to sickness or nerve injury. Such people lack the strength or motor skills to operate a typical manual wheelchair [11]. Additionally, each patient may experience muscle weaknesses differently. We intend to develop a general remedy for most paresis sufferers. To qualify for the suggested wheelchair, the patient must have good eyesight and a positive outlook. According to Callejas et al. [12], roughly 10% of the world population or around 650 million individuals suffer from physical disabilities. 10% of the population that live with impairments, only a tiny fraction has access to wheelchairs, and even fewer have access to wheelchairs that are suitable for their needs. Moreover, the Department of Census and Statistics (DCS) 2023 survey revealed that 3.6 % of Sri Lankans had a walking impairment [13].

Wheelchairs are mechanical devices designed to give persons with disabilities relatively easy mobility. A wheelchair that uses an electric motor rather than manual power is called an electric-powered wheelchair (EPW) [14]. For people unable to push a manual wheelchair, an EPW is helpful. People are free to travel around their neighborhood and residence with an EPW. Without exerting a lot of effort or energy, it may maximize independence. The most common driving configuration for EPWs is rear-wheel drive, which offers superior stability for indoor and outdoor driving [15]. As per Leaman et al. [15], approximately 40 % of individuals within the disabled community report finding it challenging or impossible to operate a standard EPW. Since the 1980s, there has been research towards intelligent wheelchairs [16,17]. However, despite over 40 years of work, this technology hasn't seen a significant market uptake. Even though some self-driving wheelchairs are in use today, they are made to transport persons who don't typically utilize motorized wheelchairs. People alter their wheelchairs to add personal touches and equip them with the tools required for medical reasons. The chairs of the majority of long-term users are adorned with backpacks, catheter bags, laptops, respirators, and other equipment. These objects are challenging to simulate and may block the chair's sensors. A crucial area for navigation is blocked by the user's feet, which are usually in front of the chair. Installing the detectors above or outside potential obstructions is the only course of action left. It can also result in strange-looking "sensor masts," which, anecdotally, many wheelchair users we have spoken with find visually unpleasant [18]. This can increase the footprint of the chair.

A research gap exists in designing and evaluating low-cost solutions that seamlessly integrate these technologies to improve the user's control and navigation experience [19], even though EEG and EMG-based systems for mobility-related assistive wheelchairs have made significant progress. Studies that have already been conducted sometimes use expensive and complicated hardware setups or only use one biopotential modality (usually EEG or EMG). To close this gap, integrated systems that address attentive detection and navigation intention in a unified and accessible way must be created. This will make them more useable and widely applicable in assistive technology contexts. This research aims to design a modular kit using EEG and EMG technologies, transforming a conventional wheelchair into an automated, user-friendly system accessible to individuals with various levels and types of muscular weakness. The objectives of this research include the development of an EEG-based user attentiveness detection system, the design of an EMG-based navigation system, and the implementation of a powered wheelchair control system.

The remaining sections of the paper are organized as follows: Section 2 discusses the literature review and relevant works in the field. Section 3 discusses the methodology of the MRA-W. Results and discussion were presented in Section 4. Finally, the research is concluded in Section 5.

2. Literature review

This research aims to produce an affordable powered wheelchair that is EEG-assisted and EMG-controlled. The authors reviewed and discussed prior studies and techniques on powered wheelchairs that use EEG and EMG control techniques to accomplish research objectives. When developing a solution to the current issue, we evaluated the benefits and downsides of each technology and the cost factor. This section examines existing wheelchair control approaches based on EEG-EMG and briefly reviews earlier works in the field.

2.1. Biopotential signals for assistive device controlling

In the field of human-machine interfaces (HMI), electromyography (EMG) and electroencephalography (EEG) are essential biopotential signals with enormous potential for people with mobility issues [20]. EEG shows promise for interpreting user intents and translating them into commands for mobility-related assistive wheelchairs (MRA-Ws) due to its capacity to detect brain electrical activity [21]. Conversely, EMG captures the electrical impulses produced during muscle contractions and offers a simple yet accurate method of steering a wheelchair [22]. These biopotential signals serve as the foundation for biopotential-based HMIs, which give users a direct and intuitive means of interacting with their assistive technology. Exploration and improvement of these biopotential signals are at the core of research projects focused on enhancing the freedom and movement of people with limited mobility.

The expense of EEG devices, which tends to rise with the number of electrodes used, is one obvious obstacle in this field [23]. Increasing the quantity of EEG data channels also increases the requirement for more processing capacity, raising the hardware price [23]. According to Zhang et al. [24], EEG-based brain-computer interfaces (BCIs) have poor resilience and reliability and frequently extensive calibration and training periods, which pose usability problems. These difficulties are a result of the complexity of independent EEG control systems. The low robustness and considerable system complexity of EEG-based wheelchairs are also highlighted by Lalitharatne et al. [25]. In BCIs/BMIs using only EEG signals as the primary input in bio-robotic applications, many problems still exist, including low dependability, precision, user adaptability, and data transfer rates. The findings of Leerskov et al. [20] revealed that the accuracies achieved with EEG alone (51.2 %) were surpassed by EMG (95.5 %) and the combination of EMG and EEG (96.2 %). The development of assistive technology based on EEG and EMG remains a key focus, aiming to address usability and financial challenges associated with EEG-only approaches.

It is evident that as the number of electrodes increases, the price of the headset also increases. Additionally, as the number of EEG data channels rises, more processing power will be needed, increasing the processing unit's hardware cost. According to Zhang et al. [24], the main reason preventing EEG-based BCIs from being widely used is arguably their poor usability, which is particularly due to their low robustness and reliability, as well as their frequently lengthy calibration and training times when considering the complexity of standalone EEG control systems.

Multi-channel EEG devices provide a practical issue when these systems cover the entire user's head, according to Lalitharatne et al. [25] in their 2013 study. This difficulty results from the user's possible discomfort in such circumstances. In other words, using EEG systems that require numerous electrodes to be implanted all over the head may not provide the best user experience. The EEG helmet or headgear's weight and bulkiness, the requirement for tight or restrained headbands to secure the electrodes, and prolonged equipment wear are a few causes of discomfort. In addition to affecting the user's physical comfort, this discomfort may also hurt usability and user acceptance, which are important considerations in designing and implementing EEG-based systems for various applications, including assistive technologies and brain-computer interfaces.

Several recent studies have compared the performance of different classifiers to classify EMG signals. Dhindsa et al. [26] examined the performance of classifiers-LDA, NB, KNN, and SVM with various kernels-using EMG signals to forecast five levels of knee angles. Among the classifiers assessed, the quadratic kernel of SVM achieved the highest performance, with an accuracy of 93.07 ± 3.84 %. Another study by Purushothaman and Vikas [27] compared the classifiers SVM, LDA, and NB in classifying 15 different finger movements from 15 subjects. Their findings indicated that the SVM surpassed the other two classifiers with an accuracy of 95 %. In the research of Kehri and Awale et al. [28], the SVM classifier achieved an accuracy of 95 % compared to an ANN classifier.

2.2. Applications and user-centric developments

Developing biopotential-based human-machine interfaces (HMI) for mobility-related assistive wheelchairs must consider applications and user-centered concerns [29,30]. These interfaces are made with a strong emphasis on enhancing the freedom and quality of life of people with mobility disabilities. Making mobility and accessibility more feasible for a broad user community, including people with disorders like paresis, is a crucial application [31]. Researchers are trying to develop interfaces that adapt to each user's requirements and skills to ensure inclusiveness and usability. As a result, HMI systems are combined with advanced algorithms to identify impediments and increase user confidence during indoor and outdoor navigation [32,33]. Safety and collision avoidance are further priorities. The development of biopotential-based HMIs that empower users and seamlessly adapt to their unique needs and preferences is fundamentally driven by applications and user-centric considerations, and this has the potential to fundamentally alter how people with mobility impairments interact with their surroundings.

The primary goal of developing a hybrid EEG/EMG-based system rather than a standalone EEG-based or EMG-based system is to solve potential safety issues brought on by the shortcomings of standalone systems. Researchers have developed wheelchairs using EMG controls that have outstanding accuracy rates, including Jang et al. [34], Gondal et al. [35], and Manero et al. [36]. However, with isolated EMG systems, there is always a chance that the user will move while not paying attention, which could result in tragic accidents.

The intricacy of the technology and the cost are the problems with totally EEG-based wheelchair control systems, as the one developed by Abboudi et al. [37]. To develop an entirely EEG-based wheelchair control system, highly sensitive hardware is needed to accurately record EEG waveforms and categorize them as motion commands. The Emotiv Epoc Headset (799 USD), as contrasted to the Neurosky headset (99 USD), is the headset utilized in the research by Abboudi et al. [37]. Table 1 depicts studies in which researchers employed high-accuracy medical-grade EEG headsets for controlling wheelchairs and other assistive devices. The table illustrates the price differences across each headset corresponding to the increase in the number of channels.

Table 1.

Comparison of different EEG headsets used in related studies.

Author(s)	EEG Device	No of Channels	Headset Price (USD)
Almeida et al. [38]	OpenBCI Ultracortex Mark IV	8 and 16	399–499
Abid et al. [39]	Emotiv EPOC X	14	999
Mai et al. [40]	Biosemi Active 2	280	12,900
Achanccaray et al. [41]	G-tec Nautilus	8, 16, 32, and 64	1000–25,000
Proposed System	NeuroSky Mindwave Mobile 2	2	130

Two main classes of EMG-based control techniques can be distinguished depending on how a pattern classifier is used. When using direct EMG input signals for device control, control methods without a pattern classifier use characteristics like amplitude thresholds. However, in non-pattern-based control approaches, it becomes difficult to distinguish between different motions coming from the same muscle groups, restricting the system to extracting only one motion command per electrode. The classification of raw EMG signals produced through feature extraction, on the other hand, enables the generation of various control signals from the same electrodes in pattern-based EMG control systems. Features including Mean Absolute Value, Root Mean Square, Zero Crossing, and Standard Deviation can be retrieved from raw EMG data, according to Abbaspour et al. [42]. As demonstrated by Shahzaib et al. [43] and Toledo-Pérez et al. [44], the classification may be performed on these extracted features using techniques like Artificial Neural Networks (ANN) or Support Vector Machine (SVM). Pattern-based EMG gesture identification was preferred in the context of a wheelchair control system because it can recognize a variety of control signals while requiring fewer electrodes for the user's convenience.

Geethanjali et al. [45] conducted a study to recognize six distinct hand movements for controlling robotic hands through EMG signals. They evaluated SVM, ANN, LDA, and SLR classifiers using time domain features including MAV, ZC, SSC, WL, MAVS, VAR, RMS, WAMP, and fourth-order AR coefficients. The most favorable outcome, achieving 92.8 % accuracy, was obtained using a linear kernel of SVM.

3. Methodology

This research aims to develop a low-cost system for navigation mobility-related assistive wheelchairs using EEG (for user attentive detection) and EMG (navigation). This section discusses the step-by-step approach of the proposed development.

3.1. Overall system architecture

Fig. 1 illustrates a block diagram detailing the acquisition of EEG/EMG data through sensor electrodes. The proposed system comprised three crucial parts: an EEG-based user attentive detection system, an EMG-based navigation system, and a wheelchair control system. Hand gestures of grasp and flexion were used to steer the wheelchair. EMG detected each gesture motion, and EEG was used to determine whether the user was attentive. Using gesture commands, the wheelchair was driven forward, turned left, turned right, and stopped.

Fig. 1.

Architecture of the proposed wheelchair system.

The procedure started with collecting the user's EEG and EMG biosignals. EEG signals recorded attention-related brain activities, whereas EMG data recorded muscle activities related to navigational intents. Pre-amplification was used for the obtained EEG and EMG signals to improve their quality and ensure that weak signals were amplified sufficiently for future processing. The signals were processed via an amplification step after pre-amplification. The signals were further strengthened, making them suitable for additional signal processing. Several methods exist to extract relevant details from the EEG and EMG data. These characteristics consist of amplitude changes, time-domain characteristics, and spectral analysis. The generated decision instructions for user attention and navigation intents were completely based on the retrieved characteristics. For instance, some EEG patterns can signify attentiveness to move, whereas EMG signals indicate a user's intention to navigate a wheelchair (gesture patterns). A Raspberry Pi microprocessor was incorporated as the system's primary controller. The decision commands produced by the signal processing stage were processed by it, which acts as the wheelchair system's brain.

3.2. EEG and EMG integrated MRA-W navigation system

3.2.1. EMG acquisition and processing

Getting an EMG signal from the human forearm was necessary for the next steps. The pure output signal from the human muscle must be appropriately treated to these processes to get an adequate signal. These factors made a sensor unit that could meet the basic needs of signal processing necessary. The EMG signal were going through a pre-processing step, which included amplification, filtering, and rectification operations, before being supplied to the processing stage. This approach was adopted to achieve the best results. The EMG's signal condition was improved as a primary design goal of the sensor unit. The sensor circuit must add filters since the resulting signal contains artifacts from external noises, such as 50.0 Hz interferences from power lines and cable movements. Fig. 2 shows the hardware filter circuit used for the EMG acquisition system. As shown in the image, the input signal was amplified using several amplification stages.

Fig. 2.

EMG signal acquisition and processing.

As shown in Fig. 2, the inverting input (IN-), the non-inverting input (IN+), and the reference input (Ref) were used as the three input signals, which were fed into an instrumentation amplifier with a gain of 11 × to amplify the bio-potential signal obtained from the motor units.

The reference input is the amplifier reference, which was set the idle output at the mid-point (half-level) of the supply voltage. The output (OUT_1) was fed back into a differential gain amplifier.

This differential amplifier with a gain of 220 × used as a bandpass filter ranging between 72 and 720 Hz. A first-order bandpass filter was used in the system and for further processing, a Bandpass Butterworth digital IIR (Infinite Impulse Response) filter using machine code was used to identify the frequencies between 75 and 150 Hz. The differential gain amplifier accepted the input from the instrumentation amplifier and the Reference electrode. An input filter (R5 and C6) blocks the DC components, and R6 and C7 set the gain and frequency range. Thus, two filters allow only the passband (72–720 Hz) to go through, filtering out other noises.

A power supply circuit was utilized as an emitter-follower or voltage-follower circuit, which derives a mid-point voltage for a total output swing. V_Ref is a 2.5 V came from a simple voltage divider. Bio-signal amplifiers measured millivolt or microvolt ranges of voltages and thus were sensitive to every minor external interference. The human body can act as an antenna, picking up signals and causing a voltage to develop. When measuring between two points on the body, they will record a voltage relative to the earth/ground. The mains supply was one such source of interference, inducing 50/60 Hz noise indoors via electromagnetic fields.

Then, the amplified and filtered EMG signals were digitalized using an analog-to-digital converter (ADC). For this, the 10-bit ADC of the MRA-W control system was used. For gesture recognition and attentiveness prediction, two programming methods were employed for the wheelchair: (1) Online method and (2) Offline method.

For the wheelchair to recognize various gestures and to decide whether the user is in an attentive state, pre-obtained data must undergo the process of signal processing, feature extraction, and classification. They should undergo training to create a model file. When a new user operates the wheelchair, the extracted features would be compared to the model's data and cross-referenced for similarities. Depending on the similarities, the system correctly recognizes the gestures and predicts attentiveness.

In the online method, the newly obtained data of the user was used to retrain the model and to save it within the system. Overtraining the dataset could cause the system to fail; hence, the number of epochs must be controlled when the online method was used. In the offline method, only a pre-trained model was used from a selected set of users. The newly obtained data would not be used to retrain the model but only for gesture recognition and attentiveness prediction. After gesture recognition and attentiveness prediction, the decisions were sent to the MRA-W control system for operation.

The filter circuit conditions EMG signal to pass through a crucial transformation phase to produce relevant information for further categorization into gesture commands. The critical process of this transition was feature extraction, which refined conditioned signal data by highlighting important information. Three basic techniques were commonly used for feature analysis in EMG-based control systems: time domain analysis, frequency domain analysis, and time-frequency domain analysis. Each method helped convert the EMG signal into valuable directives for regulating various movements and activities.

The most used approach of analysis for feature extraction was time domain features. The primary explanation is that the time domain analysis computations were significantly more straightforward and faster. The input signal's amplitude was a critical factor in the time domain analysis. There are several techniques for extracting hand gesture features from EMG data. Equations (1), (2), (3), (4), (5), (6), (7)) represents the extended mean absolute value (f_MAV), root mean square (f_RMS), standard deviation (f_SD), zero crossing (f_ZC), variance (f_VAR), integrated EMG (f_IEMG), and maximum fractal length (f_MFL) which were the characteristics that were retrieved for the classification model in this research. The effectiveness of different feature extraction methods was varied due to disparities in hardware configurations, electrode characteristics, and other configuration parameters. The feature extraction methods were selected from various alternatives based on their exceptional performance with the collected dataset, leading to an elevated prediction rate. Support Vector Machines (SVMs) are effective tools for classification tasks where a distinct margin of separation between classes was required.

$$f_{MAV}=\frac{1}{N}\sum _{i=1}^N\left|x_i\right|$$ (1)

$$f_{RMS}=\sqrt{\frac{1}{N}\sum _{i=1}^N{\left|x_i\right|}^2}$$ (2)

$$f_{SD}=\sqrt{\frac{1}{N-1}\sum _{i=1}^N{\left(x_i-\bar{x}\right)}^2}$$ (3)

$$f_{ZC}=\sum _{i=1}^N\text{sgn}\left(-x_i{x}_{i-1}\right),sgn\left(x\right)=\{\begin{array}{c}1,ifx>0\\ 0,otherwise\end{array}$$ (4)

$$f_{VAR}=\frac{1}{N-1}\sum _{i=1}^N{x_i}^2$$ (5)

$$f_{IEMG}=\sum _{i=1}^N\left|x_i\right|$$ (6)

$$f_{MFL}={\log }_{10}\left(\sqrt{\sum _{i=1}^N{\left(x_{i+1}-x_i\right)}^2}\right)$$ (7)

The importance of the hand gesture features is shown in Table 2. Results from the Support Vector Machine (SVM) classifier's feature significance analysis for EMG data were convincing. Each classifier feature was evaluated based on its p-value, demonstrating its relevance in categorizing EMG data. Notably, several variables showed statistically significant relevance with p-values substantially below the threshold of 0.05, indicating that they significantly improved the classifier's performance. In addition to f_MAV (p = 0.023), f_RMS (p = 0.032), f_STD (p = 0.041), f_ZC (p = 0.012), and f_VAR (p = 0.031), these traits are also statistically significant. Values f_IEMG (p = 0.083) and f_MFL (p = 0.092) did not show a significant relationship between the two classes. Therefore, f_IEMG and f_MFL were not included in the classifier. These results highlighted the crucial contribution of these characteristics to the efficient categorization of EMG data by the SVM classifier, highlighting the significance of these features in precise signal analysis and classification processes.

Table 2.

EMG-feature importance of the SVM classifier.

Classifier Feature	p-Value
fMAV	0.023 (<0.05)
fRMS	0.032 (<0.05)
fSD	0.041 (<0.05)
fZC	0.012 (<0.05)
fVAR	0.031(<0.05)
fIEMG	0.083 (>0.05)
fMFL	0.092 (>0.05)

The primary objective of an SVM was to choose the optimum hyperplane to divide classes from one another in the feature space. A straight line in a two-dimensional space corresponds to a hyperplane because it is a subspace with one less dimension than the input space. The SVM classifier, among the examined classifiers, was determined to perform the best at consistently discriminating between EMG gestures, as mentioned in the literature review. Therefore, it was intended to classify the feature-extracted EMG data using the SVM classifier. After comparing several SVM approaches, the Quadratic SVM classifier was chosen, offering an 86.67 (±0.5) % accuracy.

As an alternative method, we developed an RNN model for extracting features. An enlarged version of the Feed Forward Neural Network with loops connecting back to itself inside the hidden layers is called a Recurrent Neural Network, or RNN. This feature allowed the network to maintain a hidden state that can encode data from previous time steps inside a data sequence. RNNs are a crucial part of artificial neural networks, primarily processing sequential data. However, RNNs have some limitations, most notably the difficulty in capturing long-term dependencies because of the vanishing gradient problem. Thus, sophisticated RNN architectures such as GRU (Gated Recurrent Unit) and LSTM (Long Short-Term Memory) were created to overcome these constraints. These architectures can efficiently capture long-term dependencies, improving the RNN's processing speed for sequential data. A specified architecture needs to be followed to use an LSTM Neural Network, as depicted in Fig. 3. This architecture was necessary to classify the dataset obtained from EMG and EEG sensors.

Fig. 3.

Proposed LSTM architecture.

An LSTM network's important layers are the sequence input and LSTM layers. A fully connected layer, a SOFTMAX layer, and a classification output layer were needed to predict the class labels of the dataset.

Three gating mechanisms were used for the LSTM layer to function: the input gate ($i_t=\sigma (W_i.\left[h_{t-1},x_t\right]+b_i$), forget gate ($f_t=\sigma (W_f.\left[h_{t-1},x_t\right]+b_f$), and output gate ($o_t=\sigma (W_o.\left[h_{t-1},x_t\right]+b_o$). Together, these mechanisms controlled the network's information flow. To be more precise, the input gate determined which relevant information needs to be integrated, and the forget gate determined which information from the previous cell should be ignored. In the meantime, the output gate selected what data should be sent to the output. Combining these processes, the LSTM unit calculated a candidate cell state at each time step. It then incorporated the appropriate gates to update the cell state and determine the hidden state.

3.2.2. EEG acquisition and processing

The wheelchair was primarily intended to be operated by EMG signals. Still, since paresis patients and other patients with limited mobility use it frequently, every conceivable scenario had to be considered while constructing it. Therefore, the hybrid method was considered rather than using a wheelchair that an EMG exclusively powers to exclude even the remotest risk of an accident caused by inattention. Therefore, the system ensured the patient was in a proper state of awareness to operate the wheelchair safely by combining EEG (electroencephalography) signals to check the degree of consciousness.

The Neurosky Mindwave 2 commercial low-cost EEG headset was used in this research. It is a single-channel device with a non-invasive electrode. Apart from the cost, this device also enhances user comfort because it is lightweight, portable, wireless, and non-invasive. Fig. 4 shows the proposed EEG acquisition and processing architecture. The Neurosky Mindwave 2 allows for the recording of raw EEG data. It also processes the captured raw EEG data internally using its proprietary algorithms to provide data about separate frequency bands of the EEG signal. The collected data underwent a series of intricate algorithms and rescaling operations from the initial voltage, resulting in frequency band data without specified units. The raw EEG data were pre-processed by the TGAM brainwave sensor ASIC module within the Neurosky Mindwave EEG headset, notably the world's first EEG sensor designed for consumer use. The signals recorded using the Neurosky headset were sampled at 512 Hz. By accompanying the EEGID App EEG Raw Value (V_raw) and EEG Raw Value Volts (V_acutal) were measured. Moreover, The V_acutal was computed using Equation (9).

$$V_{actual}=\frac{1.8\times V_{raw}}{4096\times 2000}$$ (9)

Where;

Fig. 4.

EEG acquisition and processing architecture.

V_actual = EEG Raw Value Volts.

V_raw = EEG Raw Value.

Moreover, as per the Neurosky documentation, the values 1.8 V, 4096, and 2000 correspond to the reference voltage, ADC resolution (12-bits), and amplifier gain, respectively. Rather than solely acquiring amplitudes, we obtained specific frequency band data for a more precise and accurate analysis. These data underwent intricate transformations and rescaling operations within the Neurosky Mindwave headset. Given that this device already provided a dataset containing the EEG power spectrum, this research used the data to create a classification model to control the wheelchair.

When someone crosses the threshold from alert to sleepy, alpha waves (8–13 Hz) can be considered the signal. Additionally, the activity of the brain's visual cortex is shown by alpha waves. When strong alpha waves are presented in an EEG signal, the person is likely not paying attention and has closed their eyes. Because of this, even if EMG motions were recognized in the wheelchair's control system, the wheelchair was not moved if alpha waves were detected beyond a particular level, assuring the user's safety.

The Fast Fourier Transform (FFT) method (see Equation (10)) was used to analyze EEG data, emphasizing the alpha frequency band to determine the user's degree of awareness and attentiveness. The kernel scale was automatic for the SVM classifier, where the optimum value was chosen while training. The box constraint level was set to one (1). The multiclass method was selected to be “One vs. One”. This approach was used in multiclass classification problems where multiple classes are predicted, and, in this approach, a binary classifier was trained for each pair of classes.

$$X_k\sum _{n=0}^{N-1}{x}_n{e}^{-i2\pi kn/N};Wherek=0,\dots ,N-1$$ (10)

The danger of accidents brought on by inattention can be lowered by creating a safety mechanism inside the wheelchair's control system by setting a threshold for alpha wave power. This stops operation while the user is in a state of diminished awareness. EMG and EEG signals were used as a hybrid method that improves wheelchair user's safety.

3.3. Development of assistive wheelchair and control system

The development of a customized electric wheelchair (refer to Fig. 5), featuring a sprocket system, two motors, and an advanced control unit, marks a notable advancement in traditional wheelchair mobility technology. This innovative design, which harnesses the power of modern engineering, increases the user's flexibility and comfort. A sprocket system enhanced traction on various surfaces and permits efficient power transmission, enabling a smooth and stable ride. The twin motors' precise control and agility was let users easily move and navigate limited spaces. The controller device allowed for smooth operation and incorporates safety precautions, speed control, and direction control. It also offered easy controls and innovative features.

Fig. 5.

Physical view of the assistive wheelchair device.

The development of a mathematical model for the wheelchair system, which consists of the motors, wheels, and sensors, was the initial stage. This model, which laid the groundwork for later design stages, considered the wheelchair's dynamic behavior. The motor transfer functions were derived using analytical methods. Due to its simplicity and ease of use, a proportional (P) controller was chosen for this simple model. The control goals and system dynamics were used to develop the controller's transfer function ($C\left(s\right)$). Simulator studies and repeated tweaking were used to select suitable control increases.

Signals from the EMG and EEG were used as control inputs. While EEG electrodes were used to understand higher-level navigation directives, EMG electrodes were implanted in the necessary muscle groups to record the degree of muscle activity. Fig. 6 depicts the prototype of a wearable EMG armband equipped with dry electrodes. The design prioritizes user-friendliness, allowed users to comfortably wear the armband for extended periods and easily remove it when not in use.

Fig. 6.

Prototyped wearable EMG armband with dry electrodes.

The complete cost of the powered wheelchair conversion kit, encompassing EEG/EMG sensor hardware and additional accessories, amounted to approximately 400 USD, aligning with the study's objective of delivering an affordable solution.

Fig. 7 illustrates the comprehensive schematic diagram of the wheelchair control system. This system comprises distinct sub-units, including the motor controller unit, EEG/EMG data acquisition unit, microcontroller (MCU) unit, safety unit, and various power stages for operation. To facilitate EMG gesture prediction and EEG attentiveness prediction, a Raspberry Pi Model 3B+ was chosen over a standard microcontroller unit for its enhanced computing capabilities. The ATmega328p microcontroller unit was employed for EMG/EEG data acquisition, safety sensors, and motor control. The Raspberry Pi and the ATmega328p established a serial communication link for seamless data transfer. Various power stages were implemented to generate 5.0 V and 3.3 V power per specific requirements.

Fig. 7.

Schematic diagram of the MRA-W control system.

For the closed-loop control system, a unity feedback mechanism ($H\left(s\right)$) was presented by contrasting the planned state (the reference) with the output of this feedback loop, allowing for mistake correction. Feedback was included to guarantee system stability. Assembling the controller, plant, and feedback transfer functions yielded the closed-loop transfer function ($T\left(s\right)$). In the generic form, Equation (11), $G\left(s\right)$ denotes the derived plant transfer function for the left and right motors.

$$T\left(s\right)=\frac{G\left(s\right)\bullet C\left(s\right)}{1+G\left(s\right)\bullet C\left(s\right)\bullet H\left(s\right)}$$ (11)

In summary, Table 3 lists the equivalent instructions for the EMG and EEG states in the proposed control system. The technology smoothly converted the grasping motions of the left and right arms into a "Forward" command, allowing for fluid and forward movement when the EMG state registers as "Grasp" while the EEG state is "Attentive". The system provided a "Left Turn" instruction in response to detecting a grasp in the left arm while the right arm is flexing, enabling quick directional adjustments. The "Right Turn" instruction, on the other hand, was activated by flexing the left arm while grasping the right arm, increasing maneuverability. The system activates the "Emergency Stop" instruction when both palms were flexed, prioritizing human safety by immediately stopping all system functions.

Table 3.

Control logic of the MRA-W system.

EMG State		EEG State	Command
EMG State Left Arm	EMG State Right Arm
Grasp	Grasp	Attentive	Forward
Grasp	Flexion		Turn Left
Flexion	Grasp		Turn Right
Flexion	Flexion		Stop

Fig. 8 depicts the wheelchair navigation control's grasp and flexion. To create a standard, we provided angle criteria for grasp and flexion of the wrist. As shown in the illustration, the grasp gesture was preserved at θg = 0° while flexion gesture maintained at θf <60°.

Fig. 8.

Flexion and grasp gesture movements during the experiment.

3.4. Experimentation and data collection procedure

Under reference number DPRI/EC/MT/01/01/23, the Sri Lanka Technological Campus (SLTC) Ethical Committee has approved the proposed wheelchair kit and data collection procedure. This approval enabled the collection of test data from voluntary paresis patients who are cognizant and of sound mind to assess the wheelchair's performance. Seventy-eight individuals were chosen as test participants, representing the age ranges 16–30 Yrs., 31–65 Yrs., and above 65 Yrs. 45 males and 33 females were included in these age ranges. Before collecting the test results, confirming that the selected individuals had no non-communicable diseases, such as cardiovascular diseases, heart diseases, strokes, etc., was crucial. This preventive procedure ensured the test data was fair to all test subjects.

Several procedures were taken during the wheelchair testing process to gather EMG and EEG data. The participant must first be seated in the wheelchair for a few minutes to become comfortable with the seating and for a short rest. It is mandatory for the participant to be at rest. The subjects' forearms were then fitted with EMG test electrodes. Then, they were told to do two distinct motions: a flexion and a grasp. Each participant provided eight data sets, with four data sets for each motion, after holding each gesture for 4 s and repeating the process four times. Participants used the Neurosky Mindwave 2 headset to collect EEG data. Participants were given a 10 min interval to calm down, much like during the EMG experiment. To simulate the effect of a concentrated, attentive state of mind when driving the wheelchair, each participant was provided the Snellen Scale Test at different distances. The Snellen Scale Test is suitable to determine the user's eyesight, and adequate eyesight is also mandatory when driving a wheelchair. To simulate the effect of a relaxed mind in a non-attentive state, each participant was placed in a quiet room and advised to close their eyes while listening to a slow music track. The EEG data was recorded from all 78 participants for 2 experimental scenarios.

The test data was recorded using the Saleae Logic Pro 16 Logic Analyzer. The direct connection of this portable logic analyzer to a laptop via USB made it simple to collect participant data. The signal acquisition pins of the EMG module were linked to the logic analyzer, which was programmed to record data at a sample rate of 62.5 kSPS. The process required showing the subject how to rest their forearm and demonstrate the movements while visually validating that they were at rest by examining the waveform. The test data were gathered as a consequence of repeating this procedure with each participant.

4. Results and discussion

This section discusses the result and analysis of the proposed system. The signal still contains noise after being as thoroughly pre-processed as feasible. The signal should be cleaned before extracting features after applying hardware filters. The DC offset was first eliminated before the signal was further filtered, as illustrated in Fig. 9. After that, the signal was corrected to eliminate its negative component. To eliminate the frequencies at 10 Hz, a low pass filter was afterward applied. Here, a sampling rate of 1000 Hz was applied to a fifth-order Butterworth filter. The sampling rate and filter order were established through experimentation with various sampling rates and filter orders in MATLAB. This process aimed to achieve the desired frequency response characteristics while mitigating excessive filter complexity.

Fig. 9.

EMG signal processing of (A): Raw EMG signal, (B): EMG signal without DC offset, (C): Rectified EMG signal, (D): Linear envelope of the EMG signal.

Using time domain analysis itself won't be sufficient to study the obtained data further via the EMG filter circuit. Hence, a frequency domain analysis was required. The Fourier transformation was required to convert a function of time x(t) into a function of frequency, X(ω). Generally, a frequency domain plot includes the x-axis as the frequency and the y-axis as the magnitude. However, obtaining a spectrogram was much more detailed and accurate in this case since the signal was recorded for four instances. The most important frequency ranges of EMG signals fall in the range of 50–150 Hz. In a frequency domain analysis, if signals were found in this frequency range, it can be concluded that the recorded signals were EMG signals. As shown in Fig. 10, the spectrogram plotted against the raw EMG signal contains four regions of interest in yellow bands. These bands represent instances of frequencies of the range 50–150 Hz identified across the time domain. In many cases, the gestures are prominent except for a bit of noise at a very low intensity.

Fig. 10.

EMG spectrogram of (A): Raw EMG signal, (B) Spectrogram plotted against raw EMG signal.

The SVM classifier performed the best at reliably differentiating between grasp and flexion among the evaluated classifiers. The accuracy results for various SVM classifier types, when trained using feature-extracted data, are shown in Table 4.

Table 4.

Comparison of SVM feature extraction model accuracy.

SVM Feature Extraction Models	Accuracy (ME)
Linear	59 (±0.7) %
Quadratic	87 (±0.5) %
Cubic	81 (±1.2) %
Fine Gaussian	68 (±1.7) %
Medium Gaussian	61 (±0.5) %
Coarse Gaussian	60 (±1.1) %

Table 5 contains SVM result sample data for five individuals who participated in a series of evaluations that focused on their age, gender, and two distinct hand motions (the flexion gesture and grasp gesture). These measures are the quantitative outcomes of each person's performance of the required gestures, offering information about their movement patterns and muscle activity during the exam.

Table 5.

Sample data of SVM feature results.

Test Case	Age (Yrs.)	Gender	Flexion Gesture					Grasp Gesture
			fMAV	fRMS	fSD	fZC	fVAR	fMAV	fRMS	fSD	fZC	fVAR
Spec 01	90	F	0.050	0.095	0.095	76	0.009	0.060	0.175	0.085	216	0.015
			0.057	0.116	0.116	63	0.013	0.044	0.081	0.171	354	0.006
			0.064	0.112	0.112	57	0.012	0.056	0.068	0.054	527	0.004
			0.060	0.125	0.125	131	0.015	0.056	0.165	0.068	320	0.005
Spec 02	65	M	0.026	0.042	0.042	14	0.001	0.039	0.068	0.068	18	0.004
			0.028	0.046	0.046	19	0.002	0.039	0.063	0.063	22	0.003
			0.026	0.041	0.041	15	0.001	0.044	0.077	0.076	22	0.005
			0.031	0.050	0.050	21	0.002	0.042	0.068	0.068	8	0.004
Spec 03	64	F	0.054	0.093	0.093	30	0.008	0.040	0.063	0.063	65	0.004
			0.060	0.105	0.105	69	0.011	0.037	0.057	0.057	52	0.003
			0.064	0.105	0.105	31	0.011	0.041	0.068	0.068	52	0.004
			0.060	0.108	0.108	67	0.011	0.035	0.058	0.058	42	0.003
Spec 04	83	M	0.045	0.069	0.069	431	0.004	0.050	0.075	0.075	316	0.005
			0.044	0.067	0.067	402	0.004	0.047	0.071	0.071	394	0.005
			0.045	0.070	0.070	555	0.005	0.046	0.069	0.069	590	0.004
			0.041	0.061	0.061	747	0.003	0.046	0.068	0.068	347	0.004
Spec 05	75	F	0.092	0.121	0.121	402	0.014	0.099	0.131	0.131	513	0.017
			0.094	0.120	0.120	384	0.014	0.100	0.136	0.132	390	0.017
			0.087	0.112	0.112	429	0.012	0.103	0.140	0.140	289	0.019
			0.088	0.114	0.114	408	0.013	0.107	0.145	0.145	291	0.021

Plotting separate plots for each gesture against a feature extraction technique to show how well the SVM classifier detected the gestures was unrealistic since seeing high-dimensional numerical data would be challenging. Due to the ability to see numerous lines for each feature extraction technique versus the standard deviation, the parallel coordinates plot shown in Fig. 11 is far more helpful. To evaluate the effectiveness of the SVM classifier, it can now compare each gesture to each feature extraction technique. The SVM classifier retained an overall accuracy of 88.6 % even if the movements of grasp and flexion in Fig. 11 overlapped one another.

Fig. 11.

Parallel coordinates plot of gesture results.

A plot of the true positive rate (TPR) against the false positive rate (FPR) at various thresholds makes up a receiver operating curve or ROC curve. The ROC curve was generated to achieve a clear graphical representation since the flexion and grip movements should be distinguished. The Area Under the Curve (AUC) for the Flexion gesture was 0.93, as seen in Fig. 12. The classifier was judged to have the greatest accuracy if the AUC value approaches trends more towards one (1). Since it is significantly closer to one (1) now, at 0.93, the classifier may be said to be accurate.

Fig. 12.

(A): ROC curve for flexion gesture, (B): ROC curve for grasp gesture.

Fig. 13, Fig. 14 depict the spectrogram plotted alongside the raw EEG signals. To illustrate the non-attentive state, the participant was instructed to close their eyes for a brief period. In Fig. 13(B), the spectrogram reveals the presence of alpha waves (8–13 Hz), highlighted by prominent yellow regions.

Fig. 13.

Non-attentive state EEG data (A): Raw EEG signal, (B): Spectrogram plotted against raw EEG signal.

Fig. 14.

Attentive state EEG data (A): Raw EEG signal, (B): Spectrogram plotted against raw EEG signal.

Fig. 14(B) illustrates the lack of alpha waves (8–13 Hz), represented by mild yellowish regions. The absence of Alpha waves indicates that the participant is attentive and has open eyes.

Fig. 15 displays the eight EEG powers delta (0.5–2.75 Hz), theta (3.5–6.75 Hz), low-alpha (7.5–9.25 Hz), high-alpha (10–11.75 Hz), low-beta (13–16.75 Hz), high-beta (18–29.75 Hz), low-gamma (31–39.75 Hz), and mid-gamma (41–49.75 Hz), obtained from a sample EEG data acquired via the Neurosky headset. Table 6 displays the numerical values of the above power bands concerning a specific time. These values are the results of calculations done by proprietary algorithms, and thus, they are not cross-comparable with magnitudes obtained from other EEG acquisition devices. Each of these values were calculated from 0 to 60 Hz every 0.5 s.

Fig. 15.

EEG wave behavior during the experiment.

Table 6.

Sample data of EEG features obtained from Neurosky Headset (f_A-attention level, f_M-meditation level).

	Time	δ	θ	α1	α2	β1	β2	γ1	γ2	fA	fM
1	8.5	102542	39384	6008	74790	6321	4703	1015	718	54	8.5
2	9.6	100372	14878	16892	76699	6558	2949	2179	471	54	9.6
3	10.5	1694970	97833	109312	25012	19604	11905	4618	5513	78	10.5
4	11.5	1628265	443417	64995	104332	26289	16805	7152	4209	80	11.5
5	12.5	22419	75563	27966	199732	15816	10715	1747	3690	74	12.5

An evaluation of a machine learning model's performance on a test dataset is frequently done using a confusion matrix (see Fig. 16). The confusion matrix in this situation thoroughly explains how successfully the Quadratic SVM classifier classified features collected using various feature extraction methods.

Fig. 16.

Confusion matrix of the SVM model, (A): EMG prediction, (B): EEG prediction.

A confusion matrix portrays a comprehensive summary of a model's prediction and its correspondence to the outcomes. Key components of a confusion matrix are.

• ●True Positive (TP): Instances where the model correctly predicts the positive class.
• ●True Negative (TN): Instances where the model correctly predicts the negative class.
• ●False Positive (FP): Instances where the model incorrectly predicts the positive class.
• ●False Negative (FN): Instances where the model incorrectly predicts the negative class.

Hence, the accuracy for different model predictions can be calculated using Equation (12).

$$Accuracy=\frac{\left(TP+TN\right)}{\left(TP+TN+FP+FN\right)}$$ (12)

Notably, the classifier achieved an impressive accuracy rate of 90 % when recognizing flexion motions, accurately identifying 54 out of 60 occurrences. The classifier performed well when categorizing grip gestures, correctly classifying 53 out of 60 cases for an accuracy rating of 88.3 %. Additionally, the classifier had excellent performance for rest signals, correctly identifying 26 out of 30 occurrences with an accuracy of 86.67 %.

In these trials, the Quadratic SVM classifier scored a remarkable total accuracy rate of 88.67 %, demonstrating its effectiveness. It's important to note that this experiment considered both low- and high-alpha regions to determine user attention levels. After removing irrelevant data, the classifier was used to analyze the dataset, detecting attentive signals with an accuracy of 87.2 (±0.34) % and demonstrating strong performance when identifying non-attentive signals with an accuracy of 86.67 %. An appropriate option for the task at hand, the Quadratic SVM classifier displayed consistent and reliable performance across various dimensions, earning it a remarkable overall accuracy rating of 86.7 (±0.52) %.

Fig. 16, Fig. 17 show the confusion matrix of the SVM and LSTM models, respectively. The LSTM neural network performed exceptionally well in the previous trials using EMG data, achieving a 96.67 % recognition accuracy in flexion motion detection, correctly identifying 58 out of 60 occurrences. It could locate grip motions with perfect 95.28 % accuracy, correctly identifying all 60 instances, and rest with impressive 93.33 % accuracy, correctly identifying 28 out of 30 instances. As a result, the LSTM neural network outperformed the SVM classifier with an exceptionally high overall accuracy of 97.33 (±0.5) %. Concerning EEG data, the LSTM neural network successfully identified 13 out of 15 instances of the attentive state with an accuracy of 86.67 %. Additionally, it correctly identified each of the 15 instances of the non-attentive state with a perfect 97.12 % accuracy.

Fig. 17.

Confusion matrix of the LSTM model, (A): EMG prediction, (B): EEG prediction.

Ultimately, the experiments using EMG and EEG data demonstrated the remarkable efficacy of the LSTM neural network, surpassing the original SVM classifier. The differences in accuracy rates highlight the LSTM neural network's superior capacity to recognize particular gestures and the degree of cognitive attentiveness.

The test subjects were given instructions on how to use the wheelchair after receiving a briefing on the wheelchair control algorithm. The test subjects were directed to move ahead for 3 s, make a left turn, a right turn, and then move forward again for 3 s. For the wheelchair to operate and move, it must meet the specified requirements, as illustrated in Fig. 18. These logic levels were captured at the microcontroller unit during operation. The diagram demonstrates how the left and right motors receive operation commands based on the MRA-W system's control logic. For instance, three conditions must be met for the wheelchair to move.

Fig. 18.

Logic level diagram of the wheelchair control unit.

5. Conclusion

Biopotentials, which are electrical impulses produced within the human body as a result of electrochemical processes, have been used extensively in research on human-machine interfaces (HMIs), particularly in the context of motorized wheelchairs. However, some critical factors, such as the unique requirements of people with low muscular mass and strength necessary for conventional wheelchair operation, are occasionally missed in current researches. This research aims to integrate an inexpensive electroencephalogram (EEG) headset and a wearable electromyography (EMG) electrode armband to propose a new approach that addresses current limitations. The main goal of this technology was to create an accessible and user-friendly way to operate modular powered wheelchairs. An EEG-based user attentive detection system, an EMG-based navigation system, and a powered wheelchair control system comprise the proposed system's three main objectives.

By ethical standards, human test subjects were used to assess the system's performance. Six EMG features (p < 0.037) were found for detecting navigation intention, while four EEG features (p < 0.023) were chosen for attentive detection. The findings showed 86.67 (±0.52) % accuracy in the navigation intention system and 83.3 (±0.34) % accuracy in user attentive detection. The system's overall performance was impressive, with a weighted average precision of 0.89 and an accuracy rate of 85.0 (±0.19) %. After the dataset was trained using an LSTM network, the overall accuracy produced was 97.3 (±0.5) %, higher than the accuracy produced by the Quadratic SVM classifier. This research improves the quality of life for older and disabled people by making it easier for them to use powered wheelchairs and advancing the development of ergonomic and affordable biopotential-based HMIs. As the future direction, authors believe incorporating other biometric sensors, such as speech recognition or eye tracking, might offer a more complete and reliable control system. This could expand the number of commands and control options available to users by enabling them to operate their wheelchairs using a mix of EEG, EMG, and other biometric signals.

Ethical clearance

This research approved ethical clearance from the Sri Lanka Technological Campus (SLTC) ethical committee under the ethical clearance number DPRI/EC/MT/01/01/23.

Data availability

The data supporting the findings of this study are available upon request by emailing kasunkh@sltc.ac.lk.

CRediT authorship contribution statement

D.V.D.S. Welihinda: Writing – original draft, Visualization, Validation, Software, Methodology, Data curation, Conceptualization. L.K.P. Gunarathne: Writing – original draft, Visualization, Software, Methodology, Data curation, Conceptualization. H.M.K.K.M.B. Herath: Writing – review & editing, Software, Methodology, Investigation, Data curation. S.L.P. Yasakethu: Writing – original draft, Validation, Formal analysis. Nuwan Madusanka: Supervision, Resources, Project administration, Investigation. Byeong-Il Lee: Writing – review & editing, Supervision, Resources, Project administration, Investigation, Funding acquisition.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.