Mental stress assessment using simultaneous measurement of EEG and fNIRS

Abstract

Previous studies reported mental stress as one of the major contributing factors leading to various diseases such as heart attack, depression and stroke. An accurate stress assessment method may thus be of importance to clinical intervention and disease prevention. We propose a joint independent component analysis (jICA) based approach to fuse simultaneous measurement of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) on the prefrontal cortex (PFC) as a means of stress assessment. For the purpose of this study, stress was induced by using an established mental arithmetic task under time pressure with negative feedback. The induction of mental stress was confirmed by salivary alpha amylase test. Experiment results showed that the proposed fusion of EEG and fNIRS measurements improves the classification accuracy of mental stress by +3.4% compared to EEG alone and +11% compared to fNIRS alone. Similar improvements were also observed in sensitivity and specificity of proposed approach over unimodal EEG/fNIRS. Our study suggests that combination of EEG (frontal alpha rhythm) and fNIRS (concentration change of oxygenated hemoglobin) could be a potential means to assess mental stress objectively.

1. Introduction

Stress is defined as the non-specific response of the body/mind to any demand of change [1]. Stress can affect the responsiveness of our central-peripheral regulatory system leading to a poorer health [2]. Stress causes the activation of hypothalamus-pituitary-adrenocortical axis (HPA axis) and sympathetic nervous system (SNS) leading to an increase in the stress hormone (cortisol) in adrenal cortex. The continuous release of cortisol has a direct impact on our body, function and structure of the brain. It increases blood pressure, weakens immune system, and stops generating new neurons in the hippocampus [3, 4]. Besides, it increases the size and the activity of amygdala which involved in storing memories associated with emotional events [5]. Stress has been recognized as one of the major factors contributing to chronic disorders and productivity loss. Long-term exposure to stress has been linked to a variety of health problems such as heart disease, obesity, diabetes, stroke and depression [6–10]. Therefore, stress assessment at early stage is important before our health deteriorates.

Stress can be measured and evaluated based on perceptual, behavioural and physiological responses. Physicians traditionally evaluate stress using questionnaires [11]. However, evaluating stress using questionnaires is a subjective method [12, 13]. A more objective measure is cortisol [14, 15] and alpha amylase [16, 17]. Alternative measures with faster response time are bio-signals, namely: heart rate, blood pressure (BP) and skin conductivity (SC) [18–22]. Direct measurements on cortical response to stress can be obtained non-invasively through modern neuroimaging modalities. Electroencephalography (EEG) is one of the most commonly used neuroimaging modalities to study brain functions and conditions. EEG measures the fluctuations of electrical fields due to en-masse neuronal activity at millisecond resolution [23, 24]. EEG signals are often assessed in several distinct frequency bands, such as: Delta (1-4 Hz), Theta (4-8 Hz), Alpha (8-12.5 Hz) and Beta (12.5-30 Hz) to examine their relationship with the emotional states. Alpha and beta frequency power are linked to negative mood, stress and depression [25–28]. Separately, Marshall and Lopez-Duran reported about negative correlation between EEG alpha power rhythm and stressful events in the prefrontal cortex where alpha rhythm reduced with stress [29, 30]. Choi et al found a positive correlation between EEG beta power rhythms with stress in the temporal lobe [31]. On the other hand, frontal theta power has been linked to task difficulty (theta rhythm decreases with increasing task difficulty) [32]. Several studies have showed that EEG signals could be used to classify mental stress from resting state [33–37]. However, EEG is susceptive to noise and has a poor spatial resolution [38]. To overcome this, combining EEG with another neuroimaging modality that has a complementary nature may be an option.

Functional Near-Infrared Spectroscopy (fNIRS) is a new neuroimaging technique which measure the cerebral hemodynamics associated with neural activity. The technique sends near-infrared light directly into the head [39]. Based on the absorptivity, the change in concentrations of oxygenated (O₂Hb) and deoxygenated (HHb) hemoglobin can be estimated using modified Beer-Lambert law [40]. Compared to EEG, fNIRS offers a better spatial resolution of cortical activation. The fNIRS has found its applications in cognitive and behavioural studies. Commonly used tasks to activate the prefrontal cortex (PFC) include mental arithmetic, word generation, colour-word matching, Stroop task, mental rotation, working memory task and inhibition [41–43]. The PFC is related to working memory which enables us to hold in mind and mentally manipulate information over a short period of time. These aforementioned tasks are used in brain-computer interface (BCI) and attention studies [44–48]. More recently, fNIRS has been accepted as an assistive tool to differentiate depression, bipolar and schizophrenia [49, 50].

In this work, we hypothesize that fusion of EEG and fNIRS can assess mental stress more accurately than using either technique alone. The EEG and fNIRS have several advantages over other neuroimaging modalities (e.g. Magnetic Resonance Imaging MRI, Positron Emission Tomography PET) as they are non-invasive, portable, less expensive, safe for long-term monitoring, and reported to be a good complementary [51, 52]. We propose joint Independent Component Analysis (jICA) to fuse EEG and fNIRS measurements [53–55]. The jICA technique has been previously developed for integrating EEG and fMRI signals, to improve spatio-temporal resolution [56, 57]. Whilst the fusion may be achieved by several approaches, we used the fNIRS response to identify key EEG measurement nodes, and then introduced jICA fusion technique to integrate features from both modalities at feature level.

2. Materials and methods

2.1 Participants

Twenty two healthy male, right-handed adults (aged 26 ± 4 with head size of 56 ± 2 cm) participated in the simultaneous EEG and fNIRS measurement study. To avoid influences of circadian rhythm on alpha amylase measurement, the experiment was conducted between 3.00 and 4.30 p.m [58]. All participants were asked not to eat or chew gum at least two hours before the experiment. They were informed about the experiment and gave written consent prior to the experiment. The experiment procedures were in accordance with the declaration of Helsinki and approval granted by ethics review committee of Universiti Teknologi PETRONAS. None of these participants has a history of psychiatric, neurological disorders or psychotropic drug use. The participants were seated in a comfortable room with good air condition to avoid any environmental stress and they were also asked to minimize head movement throughout the entire experiment.

2.2 Experiment setup and task sequence

The experiment was designed based on Montreal Imaging Stress Task (MIST) [59]. The experiment protocol was performed in four steps. Step 1: a brief introduction was given to the participants to familiarize them with the proposed tasks. Step 2: the participants were trained for five minutes in a mental arithmetic (MA) task in order to estimate time taken to answer each question by individual. The task involved 3 one-digit integers (ranging from 0 to 9) and the operators were limited to + or – (for example 7-3 + 1). The answer for each question was displayed on a computer monitor in the sequence of ‘0’ to ‘9’ (as shown in Fig. 1) and participant has to select the right answer by single left-click on the mouse. Step 3 (i.e. the control phase): simultaneous measurements of EEG and fNIRS were performed for a total of 5 minutes without time limit per question. Participants were instructed to answer the questions as quickly and as accurately as possible, and they received no feedback if their answer was correct. Step 4 (i.e. stress phase): the average time recorded during the training phase was reduced by 10% and was set as a time limit. Answering wrongly or failing to answer each question within the time limit, the participants would receive a negative feedback, i.e. a message of “Incorrect” or “Time’s up” being displayed on the monitor. Furthermore, the average peer performance set to 90% was displayed on the screen to further increase the stress on the participants. In actual fact, the participants were expected to score 40-50% when the time given to answer each question was reduced by 10%. The entire recording (control phase and stress phase) took a total duration of nearly 25 minutes, each phase consisted of five blocks of EEG + fNIRS recording. Prior to each recording phase (control and stress), baseline was measured for a total duration of 20 s. During the baseline recording participants were instructed to look at a fixation cross on the computer monitor and to get ready for the next task. Figure 1 gives an overview of the block design of the task. In each block, mental arithmetic task was introduced for 30 s followed by 20 s rest.

Fig. 1.

Experiment block design. A total of five active blocks existed for each of the (a) control and (b) stress condition. In each block, arithmetic tasks produced for thirty second followed by twenty second rest. During the 30 s task, several arithmetic questions would be posted depend on how fast the respond of the participant in answering each question. If the respond rate is 2 s per question, the total number of questions per block would be 15 questions and the total number of questions would be 75 questions, for example. During the 20 s rest, the computer screen would display with a fixation cross with black background and participants were instructed to look at the fixation cross as a visual cue for trial onset. The red dashed-line marks the start of the task and the green dashed-line marks the end of the task (the marker was presented in every block). The stressors were based on time pressure and negative feedback of individual performance as demonstrated in (b). Five samples (S1-S5) of alpha amylase were collected; 5-minutes before the control condition as baseline for control, immediately after control condition, 5-minutes before stress condition, immediately after stress condition and 5-minutes after stress condition as marked in the figure with yellow rectangular.

During the experiment, all participants were instructed to answer each question correctly and not to guess the answer. To evaluate if the participants paid attention to the task, their accuracies in answering the questions were calculated. The average score was 90% accuracy in the control phase, and 40% in the stress phase, as expected in the original MIST article [59].

2.3 Salivary alpha amylase sample collection

We used a hand-held monitor COCORO meter (Nipro, Osaka, Japan) to measure salivary alpha amylase activity. Five samples of salivary alpha amylase were collected for each participant during the experiment. The first sample was collected five minutes before the beginning of the experiment as a baseline sample (S1). Second sample was collected immediately at the end of control task (S2). Third sample was collected five minutes after the control phase as recovery (S3). Fourth sample was collected immediately at the end of the stress phase (S4). Fifth sample was collected five minutes after the stress phase (S5). Note that, in the analysis section, the third sample (S3) was also used as the baseline for the stress phase. The entire sequence for sample collection is as demonstrated in Fig. 1.

2.4 Data acquisition

Simultaneous measurements of EEG and fNIRS were performed during both phases (control and stress). Brain activities were recorded at the PFC area using a custom designed probe holder for seven EEG electrodes [FP1, F7, F3, Fz, FP2, F8, and F4] plus one reference electrode [A1] attached to the earlobe and 16 fNIRS optodes (equivalent of 23 fNIRS channels). The layout of the probe holder is shown in Fig. 2. The sampling frequency for EEG (BrainMaster 24E system) was set to 256 Hz and the impedance was minimized by using small amount of gel directly to the scalp (maximum impedance was kept to be less than 5kΩ in this experiment). The fNIRS system OT-R40 (Hitachi Medical Corp, Japan) was equipped with light sources at wavelength of 695 nm and 830 nm, and a sampling rate of 10 Hz. The distance between each of the source and detector of fNIRS probes layout was set to 3 cm. The control of simultaneous measurement was implemented in MATLAB and triggers were sent to BrainMaster 24E and OT-R40 systems through parallel and serial ports respectively to mark the start and the end of each task.

Fig. 2.

EEG + fNIRS probe holder, (a) fNIRS channels and electrodes marking in the probe holder, (b) inner-view of the probe holder, (c) outer-view of the probe holder. The holder consisted of eight sources/emission probes and eight detection probes. Total of twenty three channels and seven active EEG electrodes involved in the probe holder.

2.5 Pre-processing

EEG data were pre-processed with EEGLAB 2013a toolbox [60]. Firstly, the EEG data were bandpass filtered in the range of 0.5 Hz to 30 Hz using 3rd order Butterworth filter. Secondly, eye blink artifacts were removed using Independent Component Analysis (ICA) technique. Then, EEG data were further processed using wavelet transform (WT) to divide the EEG data into different frequency bands with a resolution matched to its frequency scale. In this work, we used WT to decompose the EEG data into four frequency bands; Delta (1-4 Hz), Theta (4-8 Hz), Alpha (8-12.5 Hz) and Beta (12.5-30 Hz). From the wavelet coefficients of Alpha and Beta frequency bands signals we extracted our features. These features were then used for statistical analysis and classification.

The fNIRS signals were transformed to the change in products of mean pathlengths and concentration changes of O₂Hb, HHb and total Hb using modified Beer-Lambert law. In order to reduce noise and artifacts, fNIRS signals were passed through several pre-processing steps using plug-in analysis software Platform for Optical Topography Analysis Tool [61]. The process included filtering the signal in the range of 0.012 to 0.8 Hz using 5th order Butterworth filter to reduce the physiological noise of low and high frequency such as respiration and cardiac related fluctuations in oxygen supply. Additionally, baseline correction, epoch extraction and moving average were applied to the fNIRS signals. In baseline correction, we defined a period starting from the onset of the task condition to the end of each task condition as one analysis block. Then, we applied linear regression by least mean squares to determine the linear trend of its baseline [62, 63]. Finally, data were averaged in all the analysis blocks [64]. As responses were more pronounced in O₂Hb, we limited our analysis to O₂Hb signals [65].

2.6 Feature extraction and classification

For each analysis block, features were extracted by calculating the mean powers of EEG in Alpha and Beta frequency bands signals and the mean concentration of O₂Hb over the analysis block using a moving time-window of 500 ms, according to the following equations:

$$P_j=\frac{1}{N}{\displaystyle {\sum }_{n=1}^N|x_j(n){|}^2,j=1,2}.$$ (1)

where P_j is the EEG power, x_j(n) represents the segmented EEG signal in Alpha band at j = 1 and Beta band at j = 2 and N is the length of the signal.

$$O_{2}Hb=\frac{1}{N}{\displaystyle {\sum }_{n=1}^N{(\Delta O_{2}Hb)}_n}$$ (2)

where ΔO₂Hb represents the segmented O₂Hb signal and N is the length of O₂Hb signal.

In this study, we used support vector machine (SVM) as a common classifier for the performance assessment of individual modality and the fusion of EEG and fNIRS modality [66]. SVM was selected for its ability to model linear boundaries as well as more complex decision planes. The kernel function used in this paper was based on Radial Basis Function (RBF) defined as:

$$K(x,y)=\exp (\frac{-\left|\right|x-y|{|}^2}{2{\sigma }^2})$$ (3)

where x and y are the two data points and $\sigma$is the width of RBF.

The EEG and fNIRS signals were computed at 0.5 Hz in feature extraction stage. This gave 60 samples of EEG/fNIRS per analysis block. These samples were then fed as input features for the SVM. In this study, we adopted leave-one-out approach for cross validation among the 22 subjects.

2.7 EEG and fNIRS fusion

In this study, EEG and fNIRS data were fused in two consecutive steps. Firstly, we used the advantages of spatial resolution of fNIRS to identify spatial locations of interest (regions of interest). The selection of EEG electrodes was based on the significance of their neighbouring fNIRS channel response to the mental stress task. Secondly, we explored the spatial and temporal advantages of each modality. EEG sources were transformed via temporal ICA into tICA components and fNIRS sources on the other hand were transformed via spatial ICA into sICA components. We then applied joint independent component analysis (jICA) to compute the mixing matrix A, between tICA components of EEG and sICA components of fNIRS using the following generative model:

$$X^{EEG}=A{S}^{EEG},X^{fNIRS}=A{S}^{fNIRS}$$ (4)

Assuming we have two sources per modality (EEG and fNIRS) and two subjects, then the mixed data for EEG is $X^{EEG}={[X_1^{EEG},X_2^{EEG}]}^T$ and the mixed tICA components/sources of EEG is:

$$S^{EEG}={[{S_1}^{EEG},{S_2}^{EEG}]}^T,$$ (5)

Similarly, the mixed data for fNIRS modality of the two subjects is presented by $X^{fNIRS}={[X_1^{fNIRS},X_2^{fNIRS}]}^T$ and the fNIRS sICA components is:

$$S^{fNIRS}={[{S_1}^{fNIRS},{S_2}^{fNIRS}]}^T,$$ (6)

And the shared linear mixing matrix A is given by:

$$A=\left[\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right],$$ (7)

To form a data and source vector for each subject, the two data and sources set then concatenated and the given equations can be written in a single matrix equation,

$$\left[\begin{array}{cc}{X_1}^{EEG}& {X_1}^{fNIRS}\\ {X_2}^{EEG}& {X_2}^{fNIRS}\end{array}\right]=\left[\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right]\left[\begin{array}{cc}{S_1}^{EEG}& {S_1}^{fNIRS}\\ {S_2}^{EEG}& {S_2}^{fNIRS}\end{array}\right],$$ (8)

To estimate the mixing matrix A, we employed the infomax algorithm [67]. The infomax algorithm uses a natural gradient ascent technique to maximize the output entropy of a neural network. In this context, entropy refers to the independence between the ICA components. The weight matrix of the neural network, W refers to the inverse matrix of the shared mixing matrix, A. The optimization of the weight matrix is achieved by weight updating rule:

$$\Delta W=\eta \{I-2y^{EEG}{(S^{\text{'}EEG})}^T-2y^{fNIRS}{(S^{\text{'}fNIRS})}^T\}W,$$ (9)

$$S^{\text{'}EEG}=W{X}^{EEG},$$ (10)

$$S^{\text{'}fNIRS}=W{X}^{fNIRS},$$ (11)

$$y^{EEG}=g(S^{\text{'}EEG})$$ (12)

$$y^{fNIRS}=g(S^{\text{'}fNIRS}),$$ (13)

$$g(x)=\frac{1}{1+e^{-x}},$$ (14)

where I is the identity matrix, $S^{\text{'}EEG}$, $S^{\text{'}fNIRS}$are the estimated independent sources of EEG and fNIRS, $y^{EEG}$and $y^{fNIRS}$ are the regenerated EEG and fNIRS data respectively. From Eq. (14), $g(x)$is the nonlinear transfer function in the neural network [68]. The initial value for W, W(0) is a matrix composed of random vectors [69]. In our fusion approach, we assumed that the sources associated with EEG and fNIRS data modulated the same way across all subjects. This assumption of common linear covariation for both modalities presents a parsimonious way to link multiple data types and has led to improved results in fMRI studies [70, 71]. Furthermore, unlike Calhoun model [72], our fusion is performed at feature level.

3. Result and analysis

3.1 Salivary alpha amylase

The increase in salivary alpha amylase level in response to stress task was noticeable in all the participants (refer to Fig. 3). We further analyzed the salivary alpha amylase responses using two-sample t-test. Assuming the two distributions have the same variance, the t-test was calculated using the following equation:

$$\hspace{0.17em}t=\frac{{\overline{X}}_1-{\overline{X}}_2}{s_{X_1{X}_2}·\sqrt{\frac{1}{n}}}$$ (15)

where,$\hspace{0.17em}{s}_{X_1{X}_2}=\sqrt{(s_{X_1}^2+s_{X_2}^2)}$. Here $\hspace{0.17em}{s}_{X_1{X}_2}$ is the grand standard deviation, the order of $X_1{X}_2$ represents the group in control and stress condition respectively, $s_{X_1}^2+s_{X_2}^2$ are the unbiased estimators of the variance. Note that, the denominator of t is the standard error of the differences between mean of control and mean of stress group. The degree of freedom was set to 2n-2, where n is the number of participants in each group. We found the increase in alpha amylase level during stress condition to be significant, as compared to control condition, with mean p-value of <0.001. This confirmed that the proposed task induced stress successfully.

Fig. 3.

Salivary alpha amylase responses under control and stress condition. Blue color shows the salivary alpha amylase response under control condition at three measurement instances (5 min before (baseline), at the end of control condition (Task), 5 min after the task (recovery). Red colour shows the salivary alpha amylase response under stress condition with three measurement times (5 min before (baseline), at the end of stress condition (Task), 5 min after the stress task (recovery).The marks “***” indicate that, the task is significant with p<0.001.

3.2 EEG analysis

The cortical activation of the brain during the stress task revealed an increase in the beta rhythm power and a significant decrease in the alpha rhythm power on the PFC, respectively. Figure 4 shows the boxplot representing the normalized mean power values of alpha and beta under control and stress condition in all the 22 subjects. The result also demonstrated that, alpha rhythm responded more significantly to mental stress, compared to beta rhythm. This suggests alpha rhythm may be a better indicator of mental stress.

Fig. 4.

Normalized Alpha and Beta rhythm power values in two mental state: control and stress for average of 22-subjects. The Alpha and Beta rhythm power values were calculated from all the measured electrodes on the PFC.

3.3 fNIRS analysis

The fNIRS result demonstrated significant increase in O₂Hb concentration change during control condition in most PFC regions. In contrast, less cortical activation and, in some cases, significant decline in O₂Hb concentration change from baseline was observed during stress phase. The mean time-course of O₂Hb concentration change on selected channels is shown in Fig. 5. Figure 5(a) shows the time-course at Ch14 in which the O₂Hb concentration was significantly increased from baseline during the control condition (red line), but did not increase when given similar stimuli during stress condition (blue line). Figure 5(b) shows the time-course at Ch17 in which the O₂Hb concentration significantly increased during the control condition (red line) and increased at much reduced amplitude during stress condition (blue line). Under both control and stress conditions, the concentration change in O₂Hb returns to their baseline at the end of rest condition (last 20 s). The overall behavioural of PFC regions under control and stress conditions is presented by their topographical maps in Fig. 6.

Fig. 5.

Mean time-courses of oxygenated hemoglobin concentration changes, (a) control and stress conditions at Ch14 and (b) control and stress conditions at Ch17. The vertical red dash-line marks the start of the task and the vertical green dash-line marks the end of the task condition.

Fig. 6.

Topographical map of oxygenated hemoglobin activation for average of 22 subjects, (a) under control condition and (b) under stress condition. Red colour indicates higher activation and blue colour indicates less activation. Under stress condition, reduced hemodynamic response around the right PFC region.

The topographical maps of O₂Hb response measured and averaged from all the subjects are shown in Fig. 6(a) (control case) and Fig. 6(b) (stress case), with channel number labelling. Examining closer, we found, on average, the decreased O₂Hb response was highly localized to the right PFC.

4.1 Statistical analysis

Statistical analysis on all EEG electrodes and all fNIRS channels are summarized in Table 1. We identified the EEG electrodes and fNIRS channels of interest to be used for performance evaluation/classification. For EEG modality, six electrodes with the highest t values were selected; F4, FP2, F8, Fz, F3 and F7 (see Table 1). Since alpha rhythm responds more significantly to stress than beta rhythm, we only considered alpha rhythm features for further analysis. For fNIRS modality, since most of the channels respond significantly to the task, we set a threshold value of t ≥ 5 for channel selection. Six channels respond above the threshold, with four channels located on the right PFC and two channels located on the left PFC, namely Ch4, Ch9, Ch14, Ch15, Ch18 and Ch23 (see Table 1). These channels were then selected for fNIRS performance evaluation/classification.

Table 1. Statistical analysis of EEG alpha and beta and O2Hb of fNIRS measurements based on two-sample t-test. In terms of electrode naming, F4A represents EEG electrode F4 in Alpha band, F4B represents EEG electrode F4 in Beta band, and so on.

Channel No	t-value	p-value	Channel No	t-value	p-value	Electrode	t-value	p-value
1	4.3	0.0016	14	5.5	0.0000	Fz Alpha	2.9	0.0073
2	2.2	0.0370	15	5.3	0.0000	Fz Beta	3.1	0.0058
3	3.8	0.0022	16	2.1	0.0496	F3 A	5.3	0.0000
4	5.6	0.0000	17	3.0	0.0070	F3 B	2.5	0.0200
5	3.35	0.0033	18	5.2	0.0000	F8 A	5.4	0.0000
6	4.8	0.0002	19	2.2	0.0207	F8B	1.7	0.0700
7	0.6	0.5081	20	2.8	0.0011	FP2 A	3.3	0.0030
8	3.3	0.0038	21	3.2	0.0045	FP2 B	4.1	0.0013
9	5.5	0.0000	22	0.7	0.4193	FP1 A	2.89	0.0080
10	3.1	0.0059	23	5.1	0.0001	FP1B	1.8	0.0670
11	1.1	0.2904	F4 A	6.4	0.0000	F7 A	2.9	0.0010
12	2.3	0.0355	F4 B	3.8	0.0021	F7 B	3.0	0.0051
13	2.14	0.0396

For EEG and fNIRS fusion, we maintained the same number of features for performance evaluation. Three fNIRS channels with the highest t values were selected, and the EEG electrodes located in the closest proximity to these three fNIRS channels were also selected. The three fNIRS + EEG pairs are: Ch-4 with F4, Ch-15 with FP2, and Ch-18 with F7 (refer to Fig. 2(a) and Table 1), and their signals were used as inputs to the proposed jICA model. The fusion model was then evaluated using SVM, in the same manner as unimodal EEG and fNIRS.

The classification accuracy obtained from SVM classifier is shown in Fig. 7. It was calculated as the summation of all true data points (correctly labelled as stress or control) over the total data points of both conditions. The classification results are illustrated as boxplot for all the three modalities, namely EEG, fNIRS and EEG + fNIRS. The classification accuracy was 91.7 ± 5.3% for EEG, 84.1 ± 6.8% for fNIRS, and 95.1 ± 3.9% for fusion of EEG + fNIRS for averaged of 22-subjects. The fusion increased the accuracy by an average of + 3.4% compared to sole EEG and + 11.0% to fNIRS alone. The improvements was found to be significant, p<0.001 using t-test.

Fig. 7.

Boxplots representing the classification accuracy measured by SVM for 22 subjects. The results demonstrate significant improvements in the mean classification accuracy when combining both modalities, p<0.001. High improvement in the classification due to combining both modalities with + 3.4% compared to EEG alone and + 11% compared to fNIRS alone.

The classification results on sensitivity and specificity are shown in Fig. 8 and Fig. 9. The sensitivity is defined as the probability of obtaining a positive test result (i.e. classifier prediction) given that stress is present. Similarly, the specificity is the probability of obtaining a negative test result given that stress is absent. The classification sensitivities of EEG, fNIRS and EEG + fNIRS were 90.4 ± 5.7%, 82.4 ± 6.3% and 94.2 ± 4.3%, respectively. There were significant improvements (p<0.001) in the sensitivity of fusion approach (EEG + fNIRS) compared to unimodals with an average improvement of + 3.8% for EEG and + 11.8% for fNIRS alone, respectively. The classification specificities of EEG, fNIRS and EEG + fNIRS, on the other hand, were 93.4 ± 4.4%, 86 ± 7.2% and 96.6 ± 2.8%, respectively. Significant improvements due to the fusion over individual modality were observed, as expected, with p<0.001.

Fig. 8.

Boxplots representing the classification sensitivity calculated for 22 subjects. High improvement in the sensitivity occurred when combining both modalities with + 3.8% compared to EEG alone and + 11.8% compared to fNIRS alone, p<0.001.

Fig. 9.

Boxplots representing the classification specificity calculated for 22 subjects. High improvements in the specificity obtained when combining both modalities with + 3.2% compared to EEG alone and + 10.6% compared to fNIRS alone, p<0.001.

4. Discussion

This study investigated if the fusion of EEG and fNIRS signals could help improve the detection of mental stress. The mental states of twenty two healthy subjects performing mental arithmetic task under neutral-control and stressful condition were studied. Brain hemodynamic responses and electrical brainwaves were acquired simultaneously using fNIRS and EEG. The key features were identified using statistical analysis and were fused based on jICA technique. The detection rate of mental stress by the proposed fusion approach was then compared and analyzed statistically with the detection rate by individual modality (EEG/fNIRS).

Using alpha amylase as a reference, we confirmed that the applied time- and peer-pressure did increase significantly (p<0.001) the stress level of all participants, as compared to baseline and control condition. The EEG results showed significant decrease in alpha rhythm power (p<0.01) and increase in beta rhythm power (p<0.02) on the PFC under stress condition. The decrease in alpha rhythm power in this study is consistent with previous emotional and anxiety studies [28, 73–75]. Additionally, previous EEG studies, reported an increase in alpha and decrease in beta rhythm in the frontal cortex under fatigue (fatigue task with 90-150 minutes time) which is not the case in this study [76, 77]. In this study, we intentionally kept each recording phase limited to five minutes. This is to ensure the increment in alpha amylase level in this study is due to stress but not to other factors [78].

The fNIRS results showed significant reduction in hemodynamic response (O₂Hb concentration change) on the PFC during stress condition compared to control condition, with p<0.01. This observation of reduction in cortical activities on the PFC is consistent with the results from previous fMRI human and animal stress studies [79–84]. Our statistical analysis results further demonstrate that stress response was highly localized to the right PFC, (p<0.00001). As we know, gender difference affects brain functions such as memories, emotions, solving certain problems, decision making [85–87]. Thus, we aim to extend this to female subjects in our future study.

Using support vector machine (SVM) as classifier, we were able to classify brain activities under stress from that of neutral-control state significantly. The performance of SVM classifier was evaluated based on accuracy, sensitivity and specificity. Using features from EEG only gives a mean classification of 91.7 ± 5.3%, 90.4 ± 5.7% and 93.4 ± 4.4% accuracy, sensitivity and specificity respectively. In contrast, features from fNIRS were classified with 84.15 ± 6.8, 82.4% ± 6.3 and 86% ± 7.2 accuracy, sensitivity and specificity respectively. These fNIRS results outperformed previous studies [47, 88–98]. It however needs to be cautious as the data sets were different. Direct comparison is thus not possible. Fusion of EEG and fNIRS signals in the feature level demonstrated the highest classification performance. Using the same classifier as in individual modality (SVM), the proposed fusion technique demonstrated mean classification of 95.1% ± 3.9, 94.2% ± 4.3 and 96.6% ± 2.8 accuracy, sensitivity and specificity respectively.

The experiment results revealed that significant improvements were achieved by the proposed fusion approach in all three performance metrics, with p<0.001. This confirmed our hypothesis that fusion of EEG and fNIRS could improve the mental stress detection. The proposed feature-fusion model enabled us to take the advantages of the strengths of both modalities in unified analytic. The improvements in classification accuracy achieved in this study is also consistent with previous EEG and fNIRS fusion studies but not in all cases [99–101]. The inconsistency in fusion results may due to the level of fusion being adopted. Both Fazli and Putze applied fusion at decision level, i.e. using a meta classifier to integrate the outputs from one EEG classifier and one fNIRS classifier. It is likely that the outputs from the EEG classifier and the fNIRS classifier were highly correlated with little complementary information. In contrast, Yin was able to improve consistently the decoding of motor imagery tasks when considered feature level fusion of bimodal EEG and fNIRS [102]. Our experiment results further support the use of feature level fusion.

In this work, we used simple preprocessing techniques to extract fNIRS features. Admittedly, the processed fNIRS data involved the systemic signal that might affect the results. Future work will explore advanced signal processing methods such as depth-resolved techniques and general linear model (GLM) with systemic regression techniques as suggested by Tachtsidis [103] and Katura [104] to separate the fNIRS signals. With a better definition of the source signals, we expect the detection rate of mental stress to be improved.

5. Conclusion

Simultaneous recording of EEG and fNIRS signals was exploited for mental stress assessment. We found that the right PFC region was sensitive to mental stress (induced by time- and peer-pressure) during arithmetic tasks. The results of joint EEG and fNIRS features using the proposed fusion approach demonstrated + 3.4% and 11.0% improvement in the classification accuracy of mental stress, as compared to unimodal EEG and fNIRS, respectively. This suggests that fusion of EEG and fNIRS signals using proposed jICA approach can help improve the diagnosis of mental stress.

Funding

This research is funded by the Ministry of Education, Malaysia under Higher Institution Centre of Excellence (HiCOE) scheme.