Mental fatigue level detection based on event related and visual evoked potentials features fusion in virtual indoor environment

Abstract

The purpose of this work is to set up a model that can estimate the mental fatigue of users based on the fusion of relevant features extracted from Positive 300 (P300) and steady state visual evoked potentials (SSVEP) measured by electroencephalogram. To this end, an experimental protocol describes the induction of P300, SSVEP and mental workload (which leads to mental fatigue by varying time-on-task) in different scenarios where environmental artifacts are controlled (obstacles number, obstacles velocities, ambient luminosity). Ten subjects took part in the experiment (with two suffering from cerebral palsy). Their mission is to navigate along a corridor from a starting point A to a goal point B where specific flickering stimuli are introduced to perform the P300 task. On the other hand, SSVEP task is elicited thanks to 10 Hz flickering lights. Correlated features are considered as inputs to fusion block which estimates mental workload. In order to deal with uncertainties and heterogeneity of P300 and SSVEP features, Dempster–Shafer (D–S) evidential reasoning is introduced. As the goal is to assess the reliability for the estimation of mental fatigue levels, D–S is compared to multi layer perception and linear discriminant analysis. The results show that D–S globally outperforms the other classifiers (although its performance significantly decreases between healthy and palsied groups). Finally we discuss the feasibility of such a fusion proposal in real life situation.

Introduction

In recent years, brain computer interfaces (BCI) have become very trendy and embedded in different frameworks. The current field of applications of BCI systems is very wide and ranges from wheelchair navigation, evaluation of Brain-Computer Interface to categorize human emotional response (Crowley et al. 2010), assessment of cognitive loads (Haapalainen et al. 2010), shapes evaluation (Chew et al. 2015) to neurofeedback training for children with attention deficit disorders (Cortese et al. 2015). BCI allow users to communicate with the external environment without relying on muscular or nervous activity. A typical scheme is started by monitoring the user’s brain activity [measured by Electroencephalography (EEG), functional magnetic resonance imaging (fMRI), positron emission tomography (PET), and functional near-infrared brain monitoring (fNIRS)] which is conveyed into brain signals and processed to get feature vector. The mapping of the latter results into commands executed by the system and can be presented in two different ways :

• Active BCI allow the user to perform “direct” orders obtained from distinctive patterns reported by mental activities and recognized by BCI. These patterns are then mapped into actions depending on the application (e.g. wheelchair directions control). The correlation between the type of mental activity and EEG are known as electro-physiological source of control. They are categorized into internally generated signals (such as sensorimotor rhythms) or triggered by external stimuli [such as Event Related Potentials (ERP), Visual Evoked Potentials (VEP)]. ERP and VEP-BCI systems have shown better results compared to internally generated signals in terms of training and transfer rates (Wang et al. 2006). In this manuscript, we focus on Positive 300 wave (P300) and Steady State Visual Evoked Potentials (SSVEP) elicitation.

• Passive, implicit or non command BCI are set up by recognizing specific mental states of the user such as emotions (wakefulness, relaxation, nervousness… ) or mental fatigue levels to provide enhanced systems adapted to the user needs (e.g. switch to autopilot if high level of stress is detected …).

BCI is very flexible and can be associated with many other modalities. Consequently, new terminologies are introduced such as “hybrid-BCI” which is the acquisition of EEG data combined with physiological signals such as Electromyography (EMG), Electrocardiography (ECG)…(Liberati et al. 2015) to overcome the shortages of individual acquisitions and build more robust classifiers. Another field associates the user’s affective states with BCI schemes. This is termed as “emotional-BCI” where affective adaptation is suggested either in the input level to interpret and adapt the user’s current state or in higher levels to predict future actions (Molina et al. 2009).

The goal of the current manuscript is to estimate the user’s mental fatigue levels in the context of virtual navigation in indoor environment based on the fusion of P300 and SSVEP features. It is motivated by the complementarity between P300 and SSVEP as, each of which, presents some advantages and shortages. To the best of our knowledge, the fusion between both modalities is not investigated yet. In this manuscript we evaluate different fusion methods and investigate its efficiency compared to other conventional techniques. The conclusions will be exploited as a basis for future schemes in order to ensure the migration of those findings from simulation to real life situations. Speaking of which, in our former studies, P300 and SSVEP were used as a source of control for wheelchair commands (more details could be found in Lamti et al. 2016, 2018) (as part of active BCI systems). In an inverse scheme, under the framework of passive BCI, those same modalities are assessed in order to detect, estimate and predict the user’s state (mental fatigue and emotions).

To this end, different terminologies will be exposed such as P300, SSVEP and mental fatigue. Then, the experimental setup and methodologies will be described [for more details please refer to our former studies (Lamti et al. 2014a, 2016)].

P300, SSVEP and mental fatigue

Mental fatigue can be defined as a state that occurs after performing a cognitively demanding task for an extended period (Myers and Downs 2009). From clinical perspective, this state can engender different symptoms such as subjective feelings of tiredness, and the accompanying unwillingness for further mental effort (Meng et al. 2006). Nonetheless, from scientific point of view, mental fatigue is a complex multidimensional phenomenon: It includes changes in motivational, mood, and cognitive processes (Zhang et al. 2007). It has been found to result in a reduced goal-directed attention (Crippa et al. 2011), a decreased effectiveness in selective attention (Schlesinger et al. 2007) and an increased difficulty in dividing attention (Csatho et al. 2013). As far as mental fatigue impacts cerebral activity changes, EEG patterns show specific perturbations. We report the correlated features from P300 and SSVEP.

P300

Event Related Potentials (ERP) are consequent to the use of infrequent visual, auditory or somatosensory stimuli which evoke a positive peak over the parietal cortex after the stimulus presentation (Luck 2005). When the subject is unable to predict whether the next stimulus would be, the peak is generated around 300 ms post-stimulus. This is referred as P300. In a visual-based experiment, P300 is elicited by presenting a matrix of stimuli and request from the user to choose one of them. A P300 appears when the highlighted choice appears. Knowing the user’s choice is very important in BCI in order to execute corresponding action. One of the well known experiments that elicits P300 response is the “oddball” paradigm (Luck 2005), in which a sequence of 80% non-target stimuli of “X” and 20% of target stimuli “O” that rarely occurs among the more common non-target stimuli are presented to the user. Each time the target stimulus appears, the P300 response follows in the EEG signals. P300 was used in many projects related to wheelchair command such as Puanhvuan et al. (2017). They aimed to propose a prototype BCW to allow severe motor disabled patients to practically control a wheelchair for use in their home environment. The users were able to select from 9 possible destination commands in the automatic mode and from 4 directional commands (forward, backward, turn left and right) in the shared-control mode. These commands were selected via the designed P300 processing system. Recently, many auditory BCIs are using beeps as auditory stimuli, while beeps sound unnatural and unpleasant for some people. It is proved that natural sounds make people feel comfortable, decrease fatigue, and improve the performance of auditory BCI systems. Drip drop is a kind of natural sounds that makes humans feel relaxed and comfortable. In Huang et al. (2017) work, three kinds of drip drops were used as stimuli in an auditory-based BCI system to improve the user-friendness of the system. This study explored whether drip drops could be used as stimuli in the auditory BCI system. The auditory BCI paradigm with drip-drop stimuli, which was called the drip-drop paradigm (DP), was compared with the auditory paradigm with beep stimuli, also known as the beep paradigm (BP), in items of event-related potential amplitudes, online accuracies and scores on the likability and difficulty to demonstrate the advantages of DP. The results showed that the drip drops were reliable acoustic materials as stimuli in an auditory BCI system. A typical P300 is characterized by the following parameters (see Fig. 1):

• Maximum amplitude: the maximum magnitude of the generated peak it varies depending on the sensor and the region where the P300 occurred.

• Minimum amplitude: the minimum magnitude reached before signal deflection.

• Latency: the time that separates the onset time of the stimulus and the appearance of the P300. It is known that this value is approximately 300 ms. However, we try to investigate if this parameter changes jointly with mental fatigue levels.

• Period: the needed time for the EEG signal to reach its steady state.

Fig. 1.

P300 parameters (maximum, minimum amplitudes, latency and period)

Different types of stimuli can be used to elicit P300 (Girelli and Luck 1997). From those, features pop-outs can differ by color, orientation or motion. The results of different experiments confirm that all these targets showed successful P300 waveform. In our case, the stimuli feature pop-outs are of color type.

SSVEP

Steady-state Visual Evoked Potential (SSVEP) is a brain response to visual sinusoidally modulated stimuli (Vialatte et al. 2010). When the retina is excited by a visual stimulus presented at frequencies ranging from 3.5 Hz to 75 Hz, a continuous or oscillatory response is produced. According to Regan (1966), discrete frequency components in the steady state potentials are amplitude and phase constant over a long period of time. While these features are not visible in time domain, their spectral distribution is still constant. Hence, SSVEP peaks remain stable over time. Many applications are dervied from SSVEP-based BCI. These systems can be used for remotely controlled devices such as wheelchairs which can be useful for severely disabled people (Gao et al. 2003). We study SSVEP because of its excellent signal-to-noise ratio and relative immunity to artifacts produced by blinks and eye movements (Perlstein et al. 2003), SSVEP also provide a means to characterize preferred frequencies of neocortical dynamic processes. Multivariate Synchronization Index (MSI) has been proved to be an efficient method for frequency recognition in SSVEP-BCI systems. It measures the correlation according to the entropy of the normalized eigenvalues of the covariance matrix of multichannel signals. In the MSI method, the estimation of covariance matrix omits the temporally local structure of samples. Zhang et al. (2016) studied, a spatio-temporal method, termed temporally local MSI (TMSI), was presented. This method explicitly exploits temporally local information in modelling the covariance matrix. In order to evaluate the performance of the TMSI, they performed a comparison between the two methods on the real SSVEP datasets from eleven subjects. The results show that the TMSI outperforms the standard MSI. TMSI benefits from exploiting the temporally local structure of EEG signals and could be a potential method for the performance of SSVEP-based BCI. In our former study (Lamti et al. 2018), an active gaze/SSVEP hybrid wheelchair command system was set up in order to enhance navigation performance. This project deals mainly with the case of distracted users where gaze and cerebral distractions were added. In the current study, the passive aspect of the SSVEP based wheelchair navigation will be addressed.

In SSVEP-BCI paradigms, many considerations have to be addressed (Vialatte et al. 2010). In the following subsections, we will detail the choice of paradigm and stimuli.

Type of paradigm

There are two types of stimuli that can be used for SSVEP-BCI: either simple or complex. The simple stimuli ranges from blinking diodes, light-emitting diodes or flickering sources on an LCD computer screen while complex flickering includes reversing checkerboard. The latter produces more accentuated SSVEP patterns than simple stimuli at the same frequency (Lalor et al. 2005). It is recommended that the choice of the type of stimulus depends on the number of commands to generate (Dornhege et al. 2003): if it does not exceed four commands, complex stimuli is recommended because of its high detection rate while simple ones provide lower detection rate and faster responses with more commands. In this experiment, the fastness of response is privileged hence, simple stimuli are adopted with blinking lights.

Stimuli choice

In order to make the right choice of stimulus properties, different issues have to be addressed. First the stimulus generator depends on the complexity of the BCI (or the number of commands to be detected) (Wu et al. 2008). We choose liquid crystal display (LCD) computer screen as stimuli generator because it induces less eye fatigue [in contrast to cathode-ray tube (CRT) or Light-emitting diodes (LEDs) which tend to be more efficient for more complex BCI]. Second, the optimal stimulus frequency is very important for the experiment setup. Many criteria can be helpful to determine frequency but the most cited ones are (Wang et al. 2006): stability and signal-to-noise ratio (SNR). In our case, we choose 10 Hz because of its high SSVEP amplitude although its poor SNR due to spontaneous EEG.

Mental fatigue and its impact on P300 and SSVEP

In our living, we often have a sense of being tired due to a mental or physical work, plus a feeling of performance degradation even in the accomplishment of simple tasks. However, these mental states are often not consciously felt or are ignored, an attitude that may result in human failures, errors and even in the occurrence of health problems or on a decrease in the quality of life. States of fatigue may be detected with a close monitoring of some indicators, such as productivity, performance or even the health states. A Pimenta et al. (2013) proposed a model and a prototype to detect and monitor fatigue based on some of these items. They focused specifically on mental fatigue, a key factor in an individuals performance. With this approach they aimed to develop leisure and work context-aware environments that may improve the quality of life and the individual performance of any human being.

Mental fatigue and mental workloads are two overlapping terms but do not refer to the same concept. The mental workload is presented as the part of mental capacity allocated for a given task (O’Donnell and Eggemeier 1986) while mental fatigue is usually considered as a gradual process that affects negatively the subject performance and efficiency (Grandjean et al. 1971). In order to evaluate mental fatigue, varying the time-on-task is usually undertaken (Roy et al. 2013).

Many projects investigated the influence of mental fatigue on P300 and SSVEP. In Kthner et al. (2014), dichotic listening was used to increase mental fatigue. They found that P300 amplitude as well as alpha band power showed significant correlations with higher mental fatigue levels. While in Murata et al. (2005), mental fatigue was evaluated in VDT context using features extracted from P300. For this purpose, Principal Component Analysis was deployed to estimate the best correlated parameters. They concluded that evaluation of mental fatigue based on ERP must be conducted from other perspectives than P300 amplitude and latency as those measures were not powerful enough to characterize by themselves mental fatigue. In our pilot study (Lamti et al. 2014a), alpha and beta band ranges showed the best correlated parameters in SSVEP induction experiment during virtual wheelchair navigation.

Ryu and Myung (2005) developed a combined measure based on signals issued from EEG, ECG and HRV during dual task. Ten subjects took part in the experiment and performed different versions of dual tasks composed of tracking and mental arithmetic. They found that the extracted features (alpha rhythm, eye blink interval and heart rate variability) showed alternate correlations with respectively tracking and arithmetic tasks. By combining all those measures into a single one, the latter showed significant increase proportionally to the difficulty and the version of each task.

Dey and Mann (2010) performed a task analysis to assess mental fatigue coupled with operating an agricultural sprayer equipped with navigation device. The gathered observations consisted of eye-glance behavior and heart rate variability recording during a field spraying task. Based on eye-glance behavior external cues were suggested to be more important than lightbar in order to provide necessary navigation information. However, heart rate variability proved that operators using light-bar navigation experienced more mental fatigue.

Jo et al. (2012) proposed a cognitive architecture based on ACT-R. They suggested a quantitative methodology to predict mental fatigue based on mathematical representation of mental workload over time with respect to the activated time of the ACT-R modules. For this purpose, a series of experiments were set up based on three different tasks: memorization, visual-manual and menu selection tasks. They found that this method can reliably predict the mental fatigue as long as it can represent human performance in a given task properly.

Grandjean et al. (1971) present a study which examined the effect of differing levels of visual taskload on critical flicker frequency (CFF) change during performance of a complex monitoring task. The task employed was designed to functionally simulate the general task characteristics of future, highly automated air traffic control systems in which passive monitoring is likely to be a principal job requirement.

There have been few reports that investigated the effects of the degree and pattern of a spectral smearing of stimuli due to deteriorated hearing ability on the performance of auditory brain-computer interface (BCI) systems. In Ho Hwang et al. (2017) they assumed that such spectral smearing of stimuli may affect the performance of an auditory steady-state response (ASSR)-based BCI system and performed subjective experiments using 10 normal-hearing subjects to verify this assumption. they constructed smearing-reflected stimuli using an 8-channel vocoder with moderate and severe hearing loss setups and, using these stimuli, performed subjective concentration tests with three symmetric and six asymmetric smearing patterns while recording electroencephalogram signals. Then, 56 ratio features were calculated from the recorded signals, and the accuracies of the BCI selections were calculated and compared. These results imply that by fine-tuning the feature settings of the BCI algorithm according to the degree and pattern of hearing ability deterioration of the recipient, the clinical benefits of a BCI system can be improved.

Roy et al. (2013) proposed a novel electro-encephalography (EEG) signal processing chain designed to classify two levels of mental fatigue that appears after having spent a long time on a tedious task. The decrease in vigilance associated with mental fatigue makes it a dangerous state for operators in charge of complex systems. The processing chain, inspired from active brain computer interface computing, is implemented as follows: the EEG signal is initially filtered in a given frequency band and 15 electrodes out of 32 are then selected using a method based on Riemannian geometry. Next, a spatial filtering step is carried out using 6 common spatial pattern (CSP) filters.

To summarize, the influence of mental fatigue on P300 and SSVEP was investigated in different scenarios (Perez and Cruz 2007), however in the context of wheelchair navigation, environmental artifacts can influence mental workload such as: facing obstacles, going through hallways, lighting conditions… which in turn can induce a long term cognitive fatigue. While our main goal is to provide physiological indicators that characterize mental fatigue on EEG signals, its implementation in real world is at risk especially for security purposes. Consequently, a virtual indoor navigation implementation is needed in order to control efficiently experimental parameters and identify precisely P300 and SSVEP. Besides, the presented projects didn’t present any model to fuse between P300 and SSVEP features simultaneously in the same model.

The goal of this study is to set up a mental fatigue estimation block based on the fusion between SSVEP and P300 relevant features by the mean of evidential reasoning that will be discussed later.

In the following, in “Introduction” section, we will detail experimental environments setup for mental workload (and mental fatigue by varying time-on-task), P300 and SSVEP. In “P300, SSVEP and mental fatigue” section, we report succinctly, the most correlated features from P300 and SSVEP based BCI with mental fatigue levels. In “Materials and methods” section, Dempster–Shafer fusion technique will be proposed to assess its efficiency in comparison with multi layer perception (MLP) and linear discriminant analysis (LDA). Finally, the conclusion will present the shortages and the next steps to take in order to enhance the system.

Materials and methods

The mental fatigue detection system framework is illustrated in Fig. 2:

• Inputs are issued from the recordings of cerebral activity during experimental tests.

• A very important layer is the features extraction. In fact, one challenging problem is to consider the time zero to start recording the signal of the P300 accurately and localize the features (maximum, minimum, latency and period). This is also the case for the SSVEP where band wave signals $(\delta [0.5\text{Hz}-4\text{Hz}],\theta [4\text{Hz}-8\text{Hz}],\alpha [8\text{Hz}-13\text{Hz}],\beta [13\text{Hz}-30\text{Hz}],\gamma [30\text{Hz}-64\text{Hz}])$ must be identified synchronously with the occurrence of the SSVEP.

• All collected data will be treated through a correlation layer. The aim of this step is to select relevant features from P300 and SSVEP with mental fatigue using a Fisher test. For 14 sensors and 4 features (maximum, minimum, latency and period) per sensor, the overall crossings will result in 56 features for P300 and 70 features for SSVEP (5 bandwaves in the frequency domain per sensor).

• A fusion layer will be deployed in order to merge between previously selected features based on evidential reasoning and the Dempster–Shafer theory.

• The output of the fusion layer will result in four states: ‘NF’: ‘No Fatigue’ or awaken. ‘LF’: ‘Low Fatigue’, in this state the subject starts to feel tired. ‘MF’: ‘Medium Fatigue’, where the fatigue level is slightly alarming. The last is ‘HF’: ‘High Fatigue’ where the state is alarming and high fatigue level is reached.

Fig. 2.

The general fusion scheme based on D–S theory

In this section, we will present the experimental environments in order to induce mental workload and mental fatigue, P300 and SSVEP responses. Further details of these experiments are explained in Lamti et al. (2014a, b).

Materials

Hardware framework An Invacare Storm 3G Ranger X branded wheelchair is used with joystick to control a virtual world projected on a 180 degrees panoramic screen to help the immersion of the user in the world (Fig. 3).

Fig. 3.

Experimental platform: the main goal of this platform is to extract physiological indices that can measure motor, cognitive, ocular performance of the user through virtual navigation scenarios. In this manuscript we focus mainly on mental workload and fatigue

Virtual world The virtual world was developed using Reality Factory engine (Queteschiner 2012). The use of simulators allows to set up new cases that can induce physiological phenomena such as P300, SSVEP, mental workload by inserting special artifacts such as lights, stimuli and varying parameters like obstacles amount, type and velocity and mental fatigue by varying time-on-task. Those can help us to assess correlations between different parameters before proceeding to real world situations.

EEG sensors An Emotiv (Epoc model)1 with 16 sensors and 128Hz sampling frequency headgear is used to record brainwave activity. Sensors are placed according to the 10-20 system (Ernst Niedermeyer 2004) over frontal ($A{F}_3,A{F}_4,F_3,F_4,F_7,F_8$), fronto-central ($F{C}_5,F{C}_6$) parietal ($P_7,P_8$), occipital ($O_1,O_2$) and temporal ($T_7,T_8$) regions with a sampling frequency of 128 Hz.

NASA Task Load indeX scale In order to rate cognitive workload among a range of load situations, the NASA Task Load indeX scale (NASA-TLX) measures the workload in six different scales, each of which is associated with a different source (performance, effort, time pressure, mental demand, physical demand and frustration) (Hart and Staveland 1988). The overall weighed score reflects simultaneously physical and mental workload. Subjects have to rate the workload scales at the beginning, after each trial and at the end of the experiment. In line with (Kota et al. 2016) we assume that NASA-TLX has a strong correlation with mental fatigue.

Methods

The virtual world is composed of a corridor where, subjects are asked to navigate from starting area A to target area B placed respectively at the start and at the end of the hall. Environmental artifacts addressed are ambient luminosity (Low (50%), Normal (100%), High (150%) in local scale unit), number of obstacles (Low (1), Medium(4), High(7)) and obstacles velocity (No velocity (0), Low (2), Medium (4), High (7) in ${\text{ms}}^{-1}$). In each trial, one of the environmental parameters was manipulated (for example the trial (L,L,L) corresponds to (50% luminosity, 1 obstacle, 2 ${\text{ms}}^{-1}$)) (Fig. 4). The overall crossings result in a first block of 36 trials.

Fig. 4.

Experimental platform for P300 induction. The circles correspond to vertical pillars. Ellipses correspond to obstacles laying on the ground

The subject has to fulfill two missions; the first one is to be able to navigate from start to target area. In the second part, and in order to induce P300, each time the subject reaches the area B, a set of flickering circles (with 80% red and 20% green) are displayed. Inspired from crossing lights, the goal is to reach the exit when green stimulus appears. This constitutes the time zero of the P300 waveform recording. The onset of the target stimulus marks the beginning of the measurement of P300. The different combinations of environmental parameters are chosen randomly. This will help to avoid bias with respect to the difficulty of the task.

For SSVEP experiment (Fig. 5), only two parameters are modified in each navigation scenario: number of obstacles and obstacles velocity while keeping ambient luminosity to normal level. To induce SSVEP, flashing lights were placed in the corridor with a frequency of 10 Hz. This makes the second block that contains 12 trials.

Fig. 5.

Experimental platform for SSVEP induction. The circles correspond to vertical pillars. Ellipses correspond to obstacles laying on the ground. Yellow circles correspond to flashing lights

Ten subjects (with two suffering from cerebral palsy) took part in the experiment. They signed a consent form that explains the experiment goals and steps. After sitting comfortably in the wheelchair, they were given a set of instructions informing them of the experiment protocol and the meaning of the different scales used for self-assessment. An experimenter was also present there to answer any questions. After sensors placement and checking, the participants performed a practice trial to familiarize themselves with the system. Next, the experimenter started the physiological signals recording. The user is asked to navigate from the starting point A to the ending point B where a specific visual stimuli is displayed. At the end of each trial, the subject rates the NASA-TLX scale and takes a one minute break. The duration of each trial depends on the performance of the subject. Yet, we assume that the mental workload increases with the increase of the environmental modification difficulty where the highest combination corresponds to (H (150% luminosity), H (7 obstacles), H(7 ${\text{ms}}^{-1})$). At the end of the first block, the subject takes a break of one minute to enchain with the second block. Subjects repeated the same set of blocks three times with the same procedure as detailed. We assume that the repetition of P300 and SSVEP tasks as well as the time needed to finish all trials (an average of one hour and a half was recorded) impact the time-on-task and induce mental fatigue.

Theory/calculation

Features extraction

P300 features The EEG data were aggregated in windows of length 32 samples with an overlapping of 25%. They were common average and referenced. Eyes artifacts were removed with Independent Component Analysis (ICA) (Li et al. 2003). The signal recorded from the first five seconds of each trial constitute the baseline, from which, amplitudes (maximum and minimum) were averaged separately yielding to minimum and maximum reference amplitudes for each trial. Those were then automatically subtracted from the trial amplitudes, conceding the change of amplitudes. Latency and period are compared to the reference mentioned by literature (300ms and 600ms).

SSVEP features The activated stimulus, has a flickering frequency f (in Hz). Its corresponding SSVEP response is estimated as follows (Valbuena et al. 2007):

$$\begin{array}{c}\hfill y_i(t)=\sum _{h=1}^{h=H}(a_{s,h}sin(2\pi hft)+{\psi }_{s,h})+n(t)\end{array}$$ 1

where H is the number of harmonics, $a_{s,h}$ and ${\psi }_{s,h}$ are respectively the amplitude and the phase of the sinusoid in each electrode. n represents the noise of the signal. It can be caused by muscular disturbance or electrode noise. The goal is to minimize the noise in order to improve detection process. As it was stated in several studies (Mandel et al. 2009), a channel c can be considered as a linear combination of signals measured by electrodes. This means that at a time t, the channel c is calculated as follows :

$$\begin{array}{c}\hfill c(t)=\sum _{i=1}^{i=C}{w}_i{y}_i(t)\end{array}$$ 2

where C is the number of channels, $w_i$ is the optimal set that ensures minimum energy combination and minimum noise occurrence (Friman et al. 2007a). Thanks to its good performance which was validated in several studies, the minimum energy combination technique is implemented in this work.

The PSD for an harmonic h and an SSVEP model S can be obtained as follows (Friman et al. 2007b):

$$\begin{array}{c}\hfill PS{D}_h=||S_h{c}_i{||}^2\end{array}$$ 3

In order to extract features and classify frequency, the power spectral density (PSD) is calculated using Discrete Fourier Transform (DFT). The detection is ensured by thresholding i.e. SSVEP is detected if its frequency around the preset target (10Hz) is above a certain threshold which is calculated from the distribution of spectral EEG.

Results

Mental fatigue levels, subjective ratings and navigation performance

The ratings and navigation performance (i.e. obstacles collision and navigation time) were collected from experiment and averaged on chunks of 30 minutes length (Fig. 6). An overall increase is noticed in the three parameters: subjective ratings changed from 1.91(0.5) in the first chunk to 9.38(0.34) in the last one. This is accompanied by an increase in obstacles collisions: 4.8(3) to 13.76(5.1) and in navigation time: 56.2(8.2) to 132.35 (25.3). The reported ratings are correlated with environmental performance ($p<0.001$). Even though the repeated trials are with the same difficulty and assuming that subjects can develop more skills as they spend more time with the simulator, they weren’t able to hold the same performance during the three phases of the experiment. Finer investigations showed that the two palsied users were even worse in term of performance and feelings. An average of twelve minutes was enough to notice the transition from low to high rating scores. However, we cannot emit conclusions due to the lack of sufficient number of palsied subjects. This also corroborates the assumption that mental fatigue was successfully induced and impacts navigation performance and subjective ratings for healthy and pathological subjects.

Fig. 6.

The average of navigation performance (obstacles collisions, navigation time) and subjective ratings for each 30 min chunk

Mental fatigue levels, P300 and SSVEP features

Figure 7 summarizes the reported results which are discussed in more details in Lamti et al. (2014a, 2016). Due to its visual nature, occipital regions over the visual cortex showed a predominant correlation scores from P300 features especially maximum peak and latency and SSVEP where $\alpha$ and $\beta$ band waves have significant correlations. Yet, other regions were sensitive to the experiment manipulations especially frontal, fronto-central and temporal regions. To recall, 8 sensors are selected: $O_1$, $O_2$ (maximum and period from P300 features and $\alpha$ and $\beta$ waves from SSVEP), $T_8$ (maximum peak from P300), $F_3$, $F_4$, $F{C}_6$ (period from P300) and $P_7$, and $P_8$ ($\alpha$ waves from SSVEP) which concludes the overall 14 features.

Fig. 7.

EEG correlation illustrated by feature, band-wave,sensor and region

Fusion of data using evidential theory

Dempster’s work sets the origins of the theory of belief functions (Dempster 2008). Dempster–Shafer theory introduces the notion of beliefs and plausibility assigned to possible measurements hypothesis which extends several mathematical models such as (Khajanchi 2017; Banerjee et al. 2015; Ghosh et al. 2017) and (Khajanchi et al. 2018). Most of these models consider probabilistic as the evolution of mental fatigue level depends on the geometry correlation coefficients. However, they don’t account for the imprecision and uncertainties of the incoming information from each feature. Formally, the evidence theory concerns the following notations:

• Frame of discernment: let $\theta$ be a finite set of elements; an element can be a hypothesis, an object, or in our case a level of fatigue. A subset of $\theta$ can be denoted by $\omega (\theta )$. Suppose that the subject can be one of the three levels of fatigue LF, MF and HF where LF is Low Fatigue, MF is Medium Fatigue and HF is High Fatigue. In this case, our frame of discernment can be set as:

  $$\begin{array}{cc}\hfill \theta & =\{LF,MF,HF\}and\hfill \\ \hfill & \omega (\theta )=\{\mathrm{\varnothing },\{LF\},\{MF\},\{HF\},\{LF,MF\},\{LF,HF\},\hfill \\ \hfill & \{MF,HF\},\{LF,MF,HF\}\}\hfill \end{array}$$

  where $\mathrm{\varnothing }$ signifies “No Fatigue” condition. If $F=\{LF,MF\}$ is a subset of $\theta$, this implies that fatigue is either LF or MF.
• Mass functions: the mass function m is the mapping of the power set $\omega (\theta )$ to the number $t\in [0,1]$. The mass function can be expressed as follows:

  $$\begin{array}{c}\hfill \begin{array}{c}\hfill m:\omega (\theta )\to [0,1]\\ \hfill m(\mathrm{\varnothing })=0,\sum _{F\in \omega (\theta )}m(F)=1\end{array}\end{array}$$ 4

  The mass function m is a basic probability assignment. m(F) expresses the proportion of relevant evidence which supports the assumption that an element of $\theta$ belongs to the set F but to no subset of F. In our case, m(F) can be considered as the belief degree regarding a certain level of fatigue. In general, any subset F of $\theta$ that verifies $m(F)>0$ is called a focal element. In the same way, $C={\bigcup }_{m(F)\ne 0}$, F is the kernel element of mass m in $\theta$.
• Belief and plausibility functions: the belief function Bel is defined as follows:

  $$\begin{array}{c}\hfill Bel:\omega (\theta )\to [0,1]and\end{array}$$ 5

  $$\begin{array}{c}\hfill Bel(F)=\sum _{A\in F}m(A)\end{array}$$ 6

  While the plausibility function is defined as:

  $$\begin{array}{c}\hfill \begin{array}{c}\hfill Pls:\omega (\theta )\to [0,1]and\\ \hfill Pls(F)=1-Bel(\overline{F})=\sum _{A\cap F\ne \mathrm{\varnothing }}m(A)\end{array}\end{array}$$ 7

  The belief function Bel(F) defines the total amount of probability that must be distributed among elements of F and is a lower limit function on the probability of F. Plausibility function Pls(F) measures the maximum amount of probability that can be distributed among the elements in F.

• Belief interval [Bel(F), Pls(F)] is the belief interval that reflects uncertainty. Consequently, the interval span $[Pls(F)-Bel(F)]$ describes the unknown with respect to F.

• Properties of the belief function and plausibility are formulated as follows:

  $$\begin{array}{cc}\hfill Bel(\mathrm{\varnothing })& =Pls(\mathrm{\varnothing })=0,Pls(F)\ge Bel(F)\hfill \\ \hfill & Bel(F)+Bel(\overline{F})\le 1,Pls(F)+Pls(\overline{F})\ge 1\hfill \\ \hfill & ifA<F,Bel(A)<Bel(F)andPls(A)<Pls(F)\hfill \end{array}$$
• Rules of evidence combination: if we suppose that $m_1$ and $m_2$ are two mass functions from two different sources, we have:

  $$\begin{array}{c}\hfill \begin{array}{c}\hfill m(\mathrm{\varnothing })=0,m(F)=\frac{1}{1-K}\sum _{A\cap B=F}{m}_1(A)m_2(B)\\ \hfill whereK=\sum _{A\cap B=\mathrm{\varnothing }}{m}_1(A)m_2(B)>0\end{array}\end{array}$$ 8

  K is interpreted as the measure of the conflict between sources. Large K corresponds to more conflicting sources. The combination of $m_1$ and $m_2$ results in a mass function m which carries the joint information of the two sources:

  $$\begin{array}{c}\hfill \begin{array}{c}\hfill m=m_1\oplus m_2=m_2\oplus m_1\end{array}\end{array}$$

  In general, for n mass functions $m_1,m_2,\dots ,m_n$ in $\theta$, the measure of the conflict K is given by:

  $$\begin{array}{c}\hfill \begin{array}{c}\hfill K=\sum _{{\bigcap }_{i=1}^n{E}_i=F}{m}_1(E_1)m_2(E_2)\dots m_n(E_n)>0\end{array}\end{array}$$ 9

One of the most important steps in evidential reasoning is the calculation of the mass function based on information provided by sensors. Let :

$$\begin{array}{c}\hfill f=[x_1,x_2,\dots ,x_{14}]\end{array}$$ 10

f represents the fatigue level. $x_i$ is the ith feature given according to the corresponding fatigue level. For all fatigue levels F, the matrix could be expressed as follows:

$$\begin{array}{c}\hfill F=\left(\begin{array}{c}{f}_1\hfill \\ ⋮\hfill \\ f_4\hfill \end{array}\right)=\left(\begin{array}{cccc}{x}_{1,1}& x_{1,2}& \dots & x_{1,14}\\ ⋮& & \ddots & ⋮\\ x_{4,1}& x_{4,2}& \dots & x_{4,14}\end{array}\right)\end{array}$$ 11

$f_i$ is the feature vector describing the ith fatigue level. $x_{i,j}$ is the jth feature of the ith fatigue level. Let $S_k$ represent the measurement vector obtained from kth sensor:

$$\begin{array}{c}\hfill S_k=[S_{k1},S_{k2},\dots ,S_{k{e}_k}]withk=1,2,\dots ,8\end{array}$$ 12

$S_{\mathit{ki}}$ is the ith element of sensor $S_k$. $i=1,2,\dots ,e_k$ where $e_k$ is the number of elements provided by the kth sensor. For example, for the sensor $O_1$, $e_k=4$ as 4 parameters are extracted from this sensor: maximum amplitude, period, $\alpha$ and $\beta$. The next step consists of calculating the distance between the measured features and the feature that describes each fatigue level. One of the best known distance measure is Minkowski one (Lazar et al. 2002). This could be defined as:

$$\begin{array}{c}\hfill d_{\mathit{ki}}=\left\{\begin{array}{cc}{\left[{\sum }_{j=1}^{e_k},{(S_{\mathit{kj}}-x_{\mathit{ij}})}^{\alpha }\right]}^{\frac{1}{\alpha }}\hfill & \text{if}k=1\hfill \\ \hfill & \text{i}=1,2,3,4;\text{k}=1,2,\dots ,8\hfill \\ {\left[{\sum }_{j=1}^{e_k},{(S_{\mathit{kj}}-x_{i(j+e_{k-1})})}^{\alpha }\right]}^{\frac{1}{\alpha }}\hfill & \text{if}k>1\hfill \end{array}\right)\end{array}$$ 13

$d_{\mathit{ki}}$ is the distance between $S_k$ and $f_i$.$\alpha$ is the parameter that defines the type of the distance. If the latter is equal to 2, than the distance converges to an Euclidian one. On the other hand, if $\alpha =1$ this means that the distance converges to a corner distance. Consequently, a distance matrix could be established in the following form:

$$\begin{array}{c}\hfill D=\left(\begin{array}{cccc}{d}_{1,1}& d_{1,2}& \dots & d_{1,4}\\ ⋮& & \ddots & ⋮\\ d_{8,1}& d_{8,2}& \dots & d_{8,4}\end{array}\right)\end{array}$$ 14

where each row in the matrix D, is the distance between the measurements issued from one sensor and all fatigue levels. While each column is the distance of one fatigue level to all sensors measurements. $d_{\mathit{ki}}$ is inversely proportional to the ith fatigue level. In order to normalize data, we define $p_{\mathit{ki}}=e^{-\frac{d_{\mathit{ki}}^2}{2}}$. This results in the following matrix:

$$\begin{array}{c}\hfill P=\left(\begin{array}{cccc}{p}_{1,1}& p_{1,2}& \dots & p_{1,4}\\ ⋮& & \ddots & ⋮\\ p_{8,1}& p_{8,2}& \dots & p_{8,4}\end{array}\right)=\left(\begin{array}{c}{p}_1\hfill \\ ⋮\hfill \\ p_8\hfill \end{array}\right)\end{array}$$ 15

$p_k=[p_{k1},p_{k2},\dots ,p_{k4}]$ with $k=1,2,\dots ,8$ is termed as the mass function assigned by the kth sensor to the 4 levels of fatigue. Once the basic probability or the mass function is obtained, the final mass function can be formed as follows:

$$\begin{array}{c}\hfill m=p_1\oplus p_2\oplus \dots \oplus p_8=(m_1{m}_2\dots m_4)\end{array}$$

where $m_i$ is calculated as follows:

$$\begin{array}{c}\hfill m_i=\frac{{\prod }_{k=1}^8{p}_{\mathit{ki}}}{1-{\sum }_{i=1}^4{\prod }_{k=1}^8{p}_{\mathit{ki}}}\end{array}$$ 16

In this method, two conditions are to be considered carefully:

• Mutual exclusion: mental fatigue levels should be independent from one to another. Only one fatigue level could be expected to occur at a given instance. This means that for example $m(\{LF,MF\})$ or $m(\{LF,HF\})$ or $m(\{MF,HF\})$ is zero.

• Sensor weighting: the information detected from each sensor could vary depending on the sensitivity of the latter due to its placement. For example sensitivity of $T_8$ and $P_8$ differs in a way that the first is more sensitive in the case of P300 detection while the second is more sensitive in SSVEP one. This aspect could be accounted for by adding a weight factor to the $p_{\mathit{ki}}$:

  $$\begin{array}{c}\hfill p_{\mathit{ki}}^{\prime }=w_{\mathit{ki}}{p}_{\mathit{ki}}\end{array}$$ 17

  where: $w_{\mathit{ki}}$ is the weight associated with each sensor capability.

Finally the rules are very important as it can inform about the decision we have to carry. In this context, many rules were presented. But we give here the most commonly used:

• Maximum support rule; hypothesis with maximum belief function is selected

• Maximum plausibility rule; hypothesis with maximum plausibility function is selected

• Absolute supporting rule; hypothesis with maximum belief function is selected; it will not give a decision if the width of evidence interval is larger than the difference between the two largest supports.

• Maximum support and plausibility rule; hypothesis with maximum belief and plausibility functions is selected.

Tested with different rules, the maximum support and plausibility rule gave the best performance for mental fatigue estimation.

Each used sensor for classification can be considered as a piece of the whole information. Thus, each sensor can represent a belief structure. In the Table 1, the features extracted from sensors $O_1,O_2,T_8,F_3,F_4,F{C}_6,P_7,and{P}_8$ are presented for each fatigue level.

Table 1.

Features extracted from sensors $O_1,O_2,T_8,F_3,F_4,F{C}_6,P_7,and{P}_8$ under NF, LF, MF and HF levels

Level	$O_1$				$O_2$				$T_8$	$F_3$	$F_4$	$F{C}_6$	$P_7$	$P_8$
	Max	Per	$\alpha$	$\beta$	Max	Per	$\alpha$	$\beta$	Max	Per	Per	Per	$\alpha$	$\alpha$
NF	4655.4	495.1	19.2	11.1	164.7	305.7	8.9	5.0	119.7	392.7	505.8	562.6	6	5.6
LF	4709.5	239.751	20.3	14	166.2	233.59	12	8.66	79.6995	549.82	350.058	323.097	6.89	7.51
MF	4623.8	321.725	15.1	8.85	164.83	198.54	9.1	4.97	45.2407	313.76	194.4581	174.2632	8.65	8.11
HF	4572.7	133.36	11.6	8.94	169.5	157.88	8	5	39.9812	192.15	161.3171	166.9076	8.5	4.5

The matrix F (18) depicts fatigue levels according to the calculated features. Applying Eqs. 13 and 14 results in the distances matrix denoted by D and expressed using Minkowski distance with ($\alpha =2$). Once F is determined, the evidence theory could be carried out. To be noticed, the weights used for each sensor is 1/3 as they are considered sensitive with the same capacity to the fatigue level. Performance and decision classification are made using distance measures, combination of all mentioned information of all sensors and applying the rationality rules. We report the combined F-score (in percentage) based on the precision and recall measures (Table 2). This measure takes into consideration the class balance and is commonly employed in information retrieval (Koelstra et al. 2012).

$$\begin{array}{c}\hfill F=\left(\begin{array}{c}{f}_1\hfill \\ f_2\hfill \\ f_3\hfill \\ f_4\hfill \end{array}\right)=\left(\begin{array}{cccccccccccccc}4655.4& 495.1& 19.2& 11.1& 164.7& 305.7& 8.9& 5& 119.7& 392.7& 505.8& 562.6& 6& 5.6\\ 4709.5& 239.7& 20.3& 14& 166.2& 233.6& 12& 8.6& 79.7& 549.8& 350& 323.1& 6.9& 7.5\\ 4623.8& 321.7& 15.1& 8.85& 164.8& 198.5& 9.1& 5& 45.2& 313.8& 194.5& 174.3& 8.6& 8.1\\ 4572.7& 133.3& 11.6& 8.94& 169.5& 157.9& 8& 5& 40& 192.2& 161.3& 166.9& 8.5& 4.5\end{array}\right)\end{array}$$ 18

Table 2.

Comparison of the different techniques used for classification

Technique	Features used	Classification rate
		Healthy		Pathological		All
		Before (%)	After (%)	Before (%)	After (%)	Before (%)	After (%)
MLP	$O_{1max}$ $O_{2max}$ $O_{1per}$ $O_{2per}$ $T_{8max}$ $F_{3per}$ $F_{4per}$ $F{C}_{6per}$	65	80	60	71	63	77
D–S		60	86	59	75	60	80
	$O_{1\alpha }$ $O_{2\alpha }$ $O_{1\beta }$ $O_{2\beta }$ $P_{7\alpha }$ $P_{8\alpha }$
LDA		65	85	60	67	63	74

In order to make finer interpretation of the classification results, subjects were divided into two groups: healthy (8 subjects) and pathological (2 subjects). The reported results consider overall and individual differences between groups. The results showed that the joint information of SSVEP and P300 performed better than the individual trials of each modality. This could be explained by the fact that in our former studies (Lamti et al. 2014a, 2016), the correlation methods applied in feature selection helped to enhance features discriminability and consequently reach better results than individual modalities. The fusion of data using D–S theory is very efficient as it improves the classification rate among the other techniques. Although the classification rate for pathological group is not so good as healthy, D–S theory outperforms LDA and MLP. It is very well understood that the fusion of data could be very important; actually, information issued from different sensors are highly conflicting as for the same sensor we can extract different features (in some cases from temporal and frequency domains). Example: for sensor $O_1$ or $O_2$, features extracted are: $O_{1max}$, $O_{2max}$, $O_{1per}$ and $O_{2per}$ when assessing P300 and at the same time $O_{1\alpha }$, $O_{2\alpha }$, $O_{1\beta }$, and $O_{2\beta }$ are extracted as to assess SSVEP. This could lead to a critical problem if this aspect is not very well addressed and defined. The use of fusion technique could handle this issue. The definition of belief intervals for each sensor and feature proves to be very helpful to improve the classification of fatigue levels.

Conclusion and perspectives

In this paper, a pilot study was proposed in order to assess the influence of mental fatigue on P300 and SSVEP. The best correlated features were then used as inputs for a fusion technique which was proposed based on evidential reasoning. It showed better classification performance compared to MLP and LDA. The limited number of palsy group make it difficult to give conclusions about the efficiency of such an approach for severely disabled. However, it was persistent for healthy users and can be more enhanced if we consider enlarging the sample database. Another point which can be discussed, is the reliability of EEG in this context: in fact, the study of mental fatigue is very complex and can hold some effects on many scales: temporal, spatial and frequency. This emphasizes that more EEG sensors are needed to depict better information. Yet, this would increase the bulkiness of the headgear and rise the issue of its applicability in a real wheelchair system. Last, P300 and SSVEP were studied according to the scale given by the subject for fatigue level which tends to be subjective and EEG should be compared with other physiological sensors such as Electromyographic (EMG), Electrocardiographic (ECG) that would give richer information to assess fatigue levels.

Compliance with ethical standards

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.