Assessment of mental stress effects on prefrontal cortical activities using canonical correlation analysis: an fNIRS-EEG study

Abstract

This paper presents an investigation about the effects of mental stress on prefrontal cortex (PFC) subregions using simultaneous measurement of functional Near-Infrared Spectroscopy (fNIRS) and Electroencephalography (EEG) signals. The aim is to explore canonical correlation analysis (CCA) technique to study the relationship among the bi-modality signals in mental stress assessment, and how we could fuse the signals for better accuracy in stress detection. Twenty-five male healthy subjects participated in the study while performing mental arithmetic task under control and stress (under time pressure with negative feedback) conditions. The fusion of brain signals acquired by fNIRS-EEG was performed at feature-level using CCA by maximizing the inter-subject covariance across modalities. The CCA result discovered the associations across the modalities and estimated the components responsible for these associations. The experiment results showed that mental stress experienced by this cohort of subjects is subregion specific and localized to the right ventrolateral PFC subregion. These suggest the right ventrolateral PFC as a suitable candidate region to extract biomarkers as performance indicators of neurofeedback training in stress coping.

1. Introduction

Mental stress is one of the risk factors for neuropsychiatric disorders such as bipolar disorders, schizophrenia, anxiety and depression [1–3]. Stress disrupts creativity, problem solving, decision making, working memory and other prefrontal cortex (PFC)-dependent activities [4–6]. Animal and human studies have demonstrated detrimental effects of glucocorticoids (stress hormone) on PFC functioning [7] and identified it as the brain region susceptive to mental stress. A variety of external stress treatments [8,9] have shown the potential to remedy the PFC functioning in animals and human subjects, e.g. in post-traumatic stress disorders (PTSD) neurofeedback has been used as a potential means of treatments and assessments in the clinical practice. Neurofeedback (NF) is a type of biofeedback technique that uses real-time recordings of brain activity to enhance self-regulation of specific brain function in connection with a related behavior. The underlying assumption is that one can entrain, change and regulate neuronal activity through brain training with such feedback. Recent findings suggest that NF could also be used as an alternative treatment for young children with attention-deficit disorders. However, most of these studies were hardly translated from cognitive neuroscience lab into clinical practices due to lack of effectiveness in modulating the brain training activities. For full review about suggestions for more effective clinical applications refer to [10, 11]. One suggested approach is to have more specific target area for brain modulation [12]. This study aims to explore this approach and investigate whether mental stress affects specific sub-region or the entire region of PFC.

To study the development of stress, continuous measurements are required to study the mental state between consecutive tasks over a long period of time [13, 14]. These studies used neuroimaging technologies (e.g. positron emission topography [PET] and functional magnetic resonance imaging [fMRI]) which have good spatial resolution but have limitations in term of temporal resolution and susceptibility to movement artefacts. Furthermore, these neuroimaging techniques constrain subjects to a fixed position and in a laboratory setting. Alternative techniques such as functional Near Infrared Spectroscopy (fNIRS) and electroencephalography (EEG) offer portability for continuous monitoring, hence allowing a more naturalistic work setting. The fNIRS offers sub-second temporal resolution and spatial resolution in cm [15], and has shown its potential in localizing task activations similarly to fMRI [16–18]; meanwhile, EEG can measure cortical activity with temporal resolution in milliseconds, and has been used in the studies about emotional states, workload and fatigue [19–22]. Due to their complementary nature, fusion of fNIRS and EEG may yield richer information about the cortical activities in our study.

In this study, we propose canonical correlation analysis (CCA) method as a vehicle to assess the effects of mental stress on prefrontal cortical activity. The CCA is a statistical method to identify a linear relationship between two sets of variables by determining the inter-subject co-variances [23]. The CCA works as a linear mixing model which maximizes the correlation between pairs of canonical variates, in this case, the features of brain response recorded by each modality (EEG/fNIRS) for individual subject. In other words, the CCA is used to find a transformed coordinate system that maximizes the inter-subject covariance between the two modality data sets. Based on the covariance, we assess the association between EEG and fNIRS data to study the effects of mental stress on our working memory. The CCA has found applications in image recognition, remote sensing and neuroimaging studies [24–26]. We propose this method due to its potential not only in finding discriminative set of features but also in the elimination of redundant information within the features [24].

2. Materials and methods

2.1 Participants

A total of twenty-five males, right-handed adults (aged 22 ± 3, head size 54 ± 2 cm) participated in the simultaneous EEG and fNIRS measurement. To avoid cortisol changes due to confounders, participants were asked to refrain from exercise one day before the experiment and not to drink black tea, caffeine and carbonated beverages on the day of experiment, and from eating and drinking two hours prior to the experiment. Additionally, the experiment was conducted between 3.00 and 4.30 p.m to minimize the influences of circadian rhythm [27]. All participants were informed prior to the experiment and gave written consent. None of these participants had a history of psychiatric or neurological disorders. They were seated in comfortable chair in a room with good air condition to avoid the influence of environmental stress and were asked to minimize their head movements and to keep calm throughout the entire experiment. The experiment protocol was approved by local ethical committee and performed in accordance with the declaration of Helsinki.

2.2 Experiment protocol

The experiment task was designed based on Montreal Imaging Stress Task (MIST) [28] and was present to participants using a graphical user interface (GUI) designed with MATLAB (Mathworks, Natick, MA). The task involved 3-one digit integers (ranging from 0 to 9) and the operands were limited to + or – (example 2-3 + 9). The answer for each question was displayed in the GUI in the sequence of ‘0’ to ‘9’, and participants could select the answer by single left-click of the mouse. The task was performed in three consecutive phases. In the first phase (i.e. practice phase), participants practiced the task for 5 minutes during which the average time taken to answer questions by each individual was recorded as t_ans. In the second phase (i.e. control condition), an integrated cap which held a set of EEG electrodes and fNIRS optodes was placed on the PFC area of individual participant, and simultaneous measurement was performed while the participant was completing the arithmetic task for a duration of 5 minutes. In the third phase (i.e. stress condition), time pressure was imposed where the participants had to answer within a time limit (derived based on 10% reduction from the time t_ans recorded during the practice phase). Besides, participants were exposed to a mock user performance indicator that implied a poor performance by the participants in comparison with their peers, as shown in Fig. 1(b) (in green and red bars). In addition, answering wrongly or failing to answer each question within the time limit, the participants would receive a negative feedback, i.e. a message of “Incorrect” or “Time’s up” being displayed on the monitor. Such feedbacks add more stress on the participants. The total duration for this phase is also 5 minutes. The inducement of mental stress was confirmed by measuring salivary alpha amylase level with a COCORO meter (Nipro, Osaka, Japan).

Fig. 1.

Experiment block design. There is a total of five active blocks for each of the conditions: (a) control and (b) stress. In each block, arithmetic tasks are presented for 30 s followed by 20 s rest. The red dashed-line marks the start of the task and the green dashed-line marks the end of the task (the marking is done at every block). The stressors are based on time pressure and negative feedback of individual performance as demonstrated in (b).

The entire recording (control and stress) took a total duration of nearly 10 minutes, and each record consisted of 5-blocks. In each block, arithmetic task was introduced for 30 s followed by 20 s rest. Baseline was introduced in the first 20 s of each recording phase (control and stress). During the baseline participants were instructed to look at a fixation cross displayed on the computer monitor. During the 30 s task, participants were shown a set of mental arithmetic problems on the computer monitor, and had to respond either as quickly as possible during the control phase, or under time pressure with negative feedback during the stress phase. During the 20 s rest, the GUI displayed a white fixation cross with black background. All participants were instructed to answer the questions correctly and not to guess the answer. Their answers in all sessions were recorded as a reference. According to the record of answers, all participants were attentive to the task and their accuracies in answering the questions were >90% in the control phase and <40% in the stress phase, as expected in the original MIST article [28]. Figure 1 gives an overview of the block design and task sequence used in this study.

2.3 Measurement setup

EEG and fNIRS data were simultaneously recorded using Discovery 24E system (BrainMaster Technologies Inc, Bedford, OH) and OT-R40 fNIRS system (Hitachi Medical Co., Japan). The EEG system was equipped with seven electrodes placed on the positions of FP1, F7, F3, Fz, FP2, F8, F4 and from the mastoids (A1 + A2) as reference, according to the international 10-20 system [29]. The number of EEG electrodes was kept relatively low after consideration about space constraints and setup time suitable for daily mental health monitoring. Furthermore, it was decided to prioritize the placement of optodes over electrodes. The approximate locations of these electrodes on the PFC area are marked as shown in Fig. 2 as triangular dark-blue symbols (the exact locations were based on head measurements in the 10-20 system). The sampling frequency for EEG was set to 256 Hz and the impedance was reduced to at least 2K ohms by applying small amount of conductive gel directly to the scalp. The fNIRS system was equipped with 16 optodes (eight sources and eight detectors). The distance between each pair of source and detector optodes was set to be 3 cm. The measurement area between each pair was defined as a channel (Ch). A total of 23 channels were recorded in this study. Channels were co-registered into three PFC scalp quadrants (Frontopolar area (FPA): Ch- [9–11, 15, 16, 20–22], Ventrolateral prefrontal area (VLPFC): Ch- [8, 13, 14, 19, 12, 17, 18, 23] and Dorsolateral prefrontal area (DLPFC): Ch- [1–7]), and sub-divided into left and right hemispheres. The sampling frequency for the fNIRS system was 10 Hz. The placement of EEG electrodes and fNIRS channels is shown in Fig. 2. The control of simultaneous measurement was implemented in MATLAB and triggers were sent to both Discovery 24E system and OT-R40 system through parallel and serial ports to mark the start and the end of the task in each block.

Fig. 2.

EEG-fNIRS channel placement based on international 10-20 system. EEG electrodes and fNIRS channels were registered on three lateral PFC subregion namely: Dorsolateral PFC (yellow circles), Ventrolateral PFC (green circles) and Frontopolar area (grey circles). There were a total of seven EEG electrodes and 23 fNIRS channels.

2.4 Data analysis

EEG data were analysis was carried out in MATLAB (version 2013b) using custom script as well as the EEGLAB version 9.0.4 [30]. EEG data were bandpass-filtered between 0.5 Hz and 30 Hz using third order Butterworth filter to eliminate high frequency physiological noise. Independent component analysis (ICA) technique was applied to remove eye-movement and eye-blink artefacts. Each of the EEG channels was decomposed into seven independent components, and the one which described prefrontal eye blink artifacts was manually rejected similarly as the case of Putz [31]. Then we defined an interval of 300ms before the task as the baseline, and its average was subtracted from the signals at all subsequent data points. The EEG data in all channels were then re-referenced to linked-earlobes (A1 + A2) offline. The signals were decomposed using wavelet transform of Daubechies-8 (db8) [32] into four frequency bands, namely delta, theta, alpha and beta. Due to its relatively higher sensitivity to stress exposure [33], this study is limited to alpha rhythm at a frequency interval of 8-13 Hz. In this particular case, we were not using Evoked Response Potentials (ERPs) such as P300 but rather we considered the entire EEG signal as the ERPs might not provide a full view of neural activity [34]. To permit fusion of data from the two modalities, we used a common sliding time window of 1 s to segment EEG and fNIRS data. From the wavelet coefficients of EEG alpha frequency-band signals, the average power values within the 1-s sliding time window were extracted according to Eq. (1).

$$P=\frac{1}{N}{\displaystyle {\sum }_{n=1}^N|x(n){|}^2},$$ (1)

where P represents the extracted power, x(n) is the segmented EEG signal and N is the length of the EEG signal.

Similarly, the raw fNIRS data was transformed to the product of optical path length and concentration of oxygenated and de-oxygenated hemoglobin using modified Beer-Lambert approach, which could be represented as ΔO₂Hb and ΔHHb, respectively. The ΔO₂Hb and ΔHHb were preprocessed to remove low-frequency drift and high-frequency system noise using plug-in analysis software Platform for Optical Topography Analysis Tool [35]. The signals were band-pass filtered with third order Butterworth filter between 0.01 and 0.8 Hz. We then defined a period from the onset to the end period of the task condition (30 s) as one single analysis block. Each block of the arithmetic task was baseline corrected by subtracting the average value of the pre-task period (i.e. during the presentation of fixation cross). After baseline correction, the five blocks were averaged into a single block of 30 s. The mean ΔO₂Hb under control and stress conditions were calculated with a moving-time window of 1 s, as in the case of EEG signals. Whilst the deoxygenated hemoglobin concentration changes were also processed, it was merely for validation of actual oxygenation purpose [36]. In summary, each electrode/channel produced a total of 30-features, (one-feature for every second). These features were then used for statistical analysis and classification evaluation.

2.5 Canonical correlation analysis

EEG data set comprised seven signal components (each corresponds to signal from one of the seven EEG electrodes in the alpha frequency band) and fNIRS data set had twenty-three signal components (each corresponds to single fNIRS channel). Both data sets were preprocessed to extract features using a sliding window of 1 s, as described in Section 2.4. The canonical correlation analysis (CCA) of EEG and fNIRS data was performed at feature level. Suppose that $X\in {ℝ}^{n\times p}$and$Y\in {ℝ}^{n\times q}$ are two matrices, each contains n observations with p and q feature-dimensions from the two modalities, respectively. Let $S_{xx}\in {ℝ}^{p\times p}$ and $S_{yy}\in {ℝ}^{q\times q}$ represent the within-sets covariance matrices, and $S_{xy}\in {ℝ}^{p\times q}$ represents the between-set covariance matrix in which$S_{xy}=S_{yx}^T$. We proposed CCA method to derive a linear combination of canonical variates $X^{*}=W_x^{T}X$and $Y^{*}=W_y^{T}Y$that maximizes the pair-wise correlation across the two feature sets according to:

$$\rho (X^{*},Y^{*})=\rho (W_x^{T}X,W_y^{T}Y)=\frac{W_x^T{S}_{xy}{W}_y}{\sqrt{(W_x^T{S}_{xx}{W}_x)(W_y^T{S}_{yy}{W}_y)}},$$ (2)

where the canonical coefficients $W_x\in {ℝ}^p$ and$W_y\in {ℝ}^q$are two arbitrary non-zero vectors and the solution involves constraining the two terms in the denominator to be equal to 1:

$$W_x^T{S}_{xx}{W}_x=W_y^T{S}_{yy}{W}_y=1,$$ (3)

Note that, the canonical variates are uncorrelated within each data set and have zero mean and unit variance. Additionally, these variates have nonzero correlation only in their corresponding indices. The maximization was performed using Lagrange multipliers that solve the following optimization model [37]:

$$Model\{\begin{array}{c}\max \rho (X^{*},Y^{*}),\\ W_x^T{S}_{xx}{W}_x=W_y^T{S}_{yy}{W}_y=1,\\ W_x\in {ℝ}^p,W_y\in {ℝ}^q\end{array}$$ (4)

Applying Lagrange multiplier to Eq. (2), the transformation can be obtained as:

$$L(X^{*},Y^{*})=L(W_x^{T}X,W_y^{T}Y)=W_x^T{S}_{xy}{W}_y-\frac{{\lambda }_1}{2}(W_x^T{S}_{xx}{W}_x-1)-\frac{{\lambda }_2}{2}(W_y^T{S}_{yy}{W}_y-1),$$ (5)

where ${\lambda }_1$ and ${\lambda }_2$ are the Lagrange multipliers. Setting the partial derivatives of L($X^{*},Y^{*}$) with respect to $W_x$ and $W_y$ equal to zero gives:

$$\frac{\partial L}{\partial W_x}=S_{xy}{W}_y-{\lambda }_1{S}_{xx}{W}_x=0,$$ (6)

$$\frac{\partial L}{\partial W_y}=S_{yx}{W}_x-{\lambda }_2{S}_{yy}{W}_y=0,$$ (7)

Multiplying both sides of Eqs. (6) and (7) with $W_x^T$and $W_y^T$ respectively, and taking consideration of the constrains in Eq. (3), the equations can be simplified into:

$$W_x^T{S}_{xy}{W}_y={\lambda }_1{W}_x^T{S}_{xx}{W}_x={\lambda }_1,$$ (8)

$$W_y^T{S}_{yx}{W}_x={\lambda }_2{W}_y^T{S}_{yy}{W}_y={\lambda }_2,$$ (9)

Let${\lambda }_1={\lambda }_2=\lambda$ then

$$\rho (X^{*},Y^{*})=W_x^T{S}_{xy}{W}_y=W_y^T{S}_{yx}{W}_x=\lambda ,$$ (10)

This shows that, the Lagrange multipliers ${\lambda }_1$ and ${\lambda }_2$ are equal to the correlation coefficient of $W_x^T$ and$W_y^T$. Substitute into Eqs. (6) and (7), the transformation matrices $W_x$ and $W_y$ can be found using the eigenvalue equations [38]:

$$S_{xx}^{-1}{S}_{xy}{S}_{yy}^{-1}{S}_{yx}{W}_x={\lambda }^2{W}_x,$$ (11)

$$S_{yy}^{-1}{S}_{yx}{S}_{xx}^{-1}{S}_{xy}{W}_y={\lambda }^2{W}_y,$$ (12)

where $W_x$ and $W_y$ are the eigenvectors and${\lambda }^2$ is a vector of eigenvalues or squared of canonical correlations. The number of non-zero eigenvalues in each equation is$d=rank(S_{xy})\le \text{min}(n,p,q),$sorted in decreasing order,${\lambda }_1\ge {\lambda }_2\ge \mathrm{...}\ge {\lambda }_d$. Eventually, fusion was performed by concatenation of the transformed feature vectors within the associated components according to [39], using the following equation:

$$F=\left(\begin{array}{c}{X}^{*}\\ Y^{*}\end{array}\right)=\left(\begin{array}{c}{W}_x^{T}X\\ W_y^{T}Y\end{array}\right)={\left(\begin{array}{cc}{W}_x& 0\\ 0& W_y\end{array}\right)}^T\left(\begin{array}{c}X\\ Y\end{array}\right),$$ (13)

where F is the canonical correlation discriminant features.

2.6 Definition of statistical significance

To better understand the statistical significance of stress effects on PFC activities, we used two sample t-test to measure the differences in task response (EEG and fNIRS signal features) between control and stress conditions. The differences in EEG responses (control versus stress) were considered statistically significant if p value < 0.01. On the other hand, for fNIRS responses, topographical maps were used to study the effects of mental stress on different PFC subregions. As two or more neighboring channels may respond to the task equally, we defined empirically a higher threshold value of t > 3 (equivalent to p<0.004) to find the focus of O₂Hb change. Only those data above the aforementioned statistical threshold values were used for subsequent classification and fusion evaluation.

2.7 Classification for quantitative analysis

In addition to study the correlation, we further evaluated the feature sets from the two modalities as well as the fusion of them to (a) quantitatively assess if fusion of the bimodality provide a better description of mental stress, and (b) identify if the effects of mental stress is localized to PFC subregion or as a whole. We used support vector machine (SVM) for classifying stress from control state in sole EEG, sole fNIRS and fusion of EEG-fNIRS. The SVM is a supervised machine learning technique widely used for classification, regression and density estimation [40]. We selected SVM for its ability to model linear as well as more complex decision boundaries. The decision boundary hyperplane in SVM is estimated based on the training data set by maximizing the distance between the hyperplane to the nearest data point. LIBSVM software was used to build the SVM classifier and employed radial basis function (RBF) kernel to nonlinearly map data onto a higher dimension space [41].

A 10-fold cross-validation scheme with randomization was applied to each feature vector. In the 10-fold cross validation, each of the EEG, fNIRS and EEG-fNIRS feature sets was split into ten subsets. Nine subsets were used to train the SVM classifier, and the remaining one subset was used for estimation of classification accuracy, sensitivity and specificity. This procedure was repeated ten times with each subset having an equal chance of being the testing data, and the average classification accuracy, sensitivity and specificity were then evaluated. The classification accuracy was defined as the ratio of correctly classified number of samples and the total number of samples. The sensitivity of the classifier was evaluated as the probability of obtaining a positive test result (i.e. classifier prediction) given that stress was presented. Similarly, the specificity was the probability of obtaining a negative test result given that stress was absent. Then we analyzed the classifier performance by using receiver operating characteristic (ROC) curves, which plot the sensitivity versus (1 minus the specificity). We also evaluated the area under the ROC curve (AUC) as a measure of a classifier’s discriminatory power.

3. Result and analysis

3.1 Results of individual modality

The arithmetic task under time pressure with negative feedback gave a significant increase in the salivary alpha amylase level, as compared to baseline and control condition. The mean values and standard deviation values of measured alpha amylase were 10 ± 4.6 (KIU/L), 50 ± 6.6 (KIU/L), and 100 ± 4.2 (KIU/L), for baseline, control and stress conditions respectively. The increase in alpha amylase level, from baseline to control condition, was significant with mean p-value < 0.001, and the increase in alpha amylase level from control condition to stress condition was significant with mean p-value < 0.0001. Similarly, the increase in alpha amylase level from baseline to stress condition was significant with mean p-value < 0.00001, respectively. Overall, the significant increase in alpha amylase level from control condition to stress condition, across all subjects, confirms the repeatability of stress induction by using time pressure and negative feedback. EEG result shows decrease in alpha rhythm from control condition to stress condition in all seven electrodes for all the participants. Figure 3(a) shows the averaged normalized alpha rhythm under control and stress conditions of all subjects at all EEG electrodes. The statistical analysis demonstrated significant differences in alpha rhythms between control and stress conditions in all the electrodes with mean p-values < 0.01 as the case in ‘F7’; p-value < 0.001 as the case in ‘FP1’, ‘FP2’, ‘F3’ and ‘Fz’; and p-value < 0.0001 as the case in ‘F4’, and ‘F8’, respectively. The changes in alpha suppression indicate that the PFC responded differently to situations when the arithmetic task was presented with time pressure and negative feedback. Similarly, results from fNIRS show decrease in the concentration change of oxygenated hemoglobin from control condition to stress condition, and the decrease was found consistently across all subjects, with mean p-values < 0.01. Figure 3(b) shows the topographical maps of four subjects where the first subject has higher cortical activation at the left DLPFC region whilst the other three has on the right FPA, under the control condition. On the other hand, Fig. 3(c) shows the topographical maps of the same subjects under stress condition. The right VLPFC is consistently the region with the least oxygenation under stress condition among the participants. Figure 3(d) shows the topographical maps for average of 25-subjects under control and stress condition with their corresponding t-map respectively. The calculated t-values were reconstructed to generate T-map using bicubic interpolation function developed by Katura et. al., and was embedded in the Platform for Optical Topography Analysis Tool [33]. From the overall topographical maps of all the subjects, it is clearly seen that the reduction in brain activation under stress is localized to specific PFC subregion – right VLPFC, instead of being distributed across the entire PFC.

Fig. 3.

Results. (a) Normalized alpha rhythm obtained from all EEG electrodes. The marks ‘**’, ‘***’ and ‘****’ indicate that, the task is significant with p<0.01, p<0.001 and p<0.0001, respectively. (b) Mean change in oxygenated hemoglobin concentrations of four example subjects under control condition. (c) Mean change in oxygenated hemoglobin concentrations of the same subjects under stress condition. (d) Mean change in oxygenated hemoglobin concentrations of all 25 subjects, (1) under control condition, (2) under stress condition and (3) average T-map of between control and stress conditions. The numbers 1 to 23 in each topographical image indicate the number of particular channel at that location.

We analyzed the covariance matrices of the feature sets from the two modalities on the entire PFC region with six subregions being of interest. Each of the six subregion was being represented by single EEG/fNIRS channel (VLPFC [right: F8 and Ch19; left: F7 and Ch23], DLPFC [right: F4 and Ch1; left: F3 and Ch3], and FPA [right: FP2 and Ch20; left: FP1 and Ch22]). The criteria of selection was based on the correlation level of components from the transformed feature vectors, discarding those with small canonical correlations. Figure. 4 shows the canonical correlations of alpha rhythm and oxygenated hemoglobin of stress features obtained by applying CCA to the entire data sets arranged in descending rank order. This canonical correlations were computed from the estimated joint covariance matrix. As observed from Fig. 4, the correlation varies among subregions with highest correlation coefficient of 0.95 at the right VLPFC and the 6th highest correlation coefficient of 0.48 located at the left FPA respectively. The higher the correlation value, the more focal/localize the stress is. It is also observed that, the precision in localizing stress to the right VLPFC across all subjects is high as demonstrated by small standard deviation at component 1. The pair of components showing the strongest correlation (r = 0.95) across the two data sets demonstrates the highest significant difference (p<0.00001) between control and stress subjects. The cross-subject source correlation matrix (map) is displayed in Fig. 5. As expected, the matrix shows the consistency in inter-subject correlation between the modalities.

Fig. 4.

Left: Correlation coefficients resulting from EEG-fNIRS CCA (sorted in decreasing order). Right: CCA of EEG alpha rhythm and O2Hb of fNIRS.

Fig. 5.

Cross-subjects source correlation matrix using CCA technique.

3.2 Classification evaluation

We selected six-bilateral electrodes (FP1, FP2, F3, F4, F7 and F8, each electrode corresponding to one subregion of the PFC area), and six-channels (Ch22, Ch20, Ch19, Ch23, Ch1 and Ch3) to study local and global EEG/fNIRS response. Same number of electrodes/ channels was used to ensure a fair comparison in terms of the number of features as inputs to classifier because feature size in general may affect classification results. This however does not suggest the two modalities have the same spatial resolution. To evaluate the fusion, we fused each of the FPC subregion separately (e.g. VLPFC [F8, F7, Ch19 and Ch23], DLPFC [F4, F3, Ch1 and Ch3] and FPA [FP2, FP1, Ch20 and Ch22]) to study their sensitivity to stress. Finally, we compared the fusion of each PFC subregion, with those of the entire PFC area evaluated by individual EEG/fNIRS modality. The higher the performance of the subregion in comparison to the performance of whole PFC region, the more dominant that PFC subregion is. The classification results could further reveal if the fusion provided a better result than single modality (EEG/fNIRS).

The classification results for individual modality and after fusion are presented by their ROC curves in Fig. 6. Figure 6(a) shows the classification evaluation of sole EEG modality at all the PFC subregions (based on combination of FP1, FP2, F3, F4, F7 and F8), VLPFC subregion (based on combination of F8 and F7), DLPFC subregion (based on combination of F4 and F3) and FPA subregion (based on combination of FP1 and FP2), respectively. As expected, combining multiple electrodes provides a better result. Similar results are obtained for fNIRS, as illustrated in Fig. 6(b).

Fig. 6.

ROC curves (a) EEG modality with different combinations (using six-bilateral electrodes red-line, VLPFC blue-line, DLPFC green-line and FPA black-line), (b) fNIRS modality with different combinations (using six-channels red-line, VLPFC blue-line, DLPFC green-line and FPA black-line), and (c) fusion of EEG and fNIRS with different combinations (two-electrodes and two-channels within: the VLPFC red-line, DLPFC blue-line, FPA green-line and six-bilateral electrodes black-line and six-channels cyan line).

On the other hand, Fig. 6(c) shows the classification evaluation under fusion of EEG-fNIRS signal modality within the VLPFC subregion, DLPFC subregion and FPA subregion in comparison with the optimum results obtained by individual modality from the entire PFC area. The average classification accuracy, sensitivity, specificity and AUC values in the format of entire PFC area (at six-bilateral electrode/channel), VLPFC, DLPFC and FPA are calculated, as shown in Table 1. Additionally, the overall classification performance and improvement of fusing each subregion over the entire PFC area for each modality and over each subregion by individual modality is presented in Table 1.

Table 1. Overall classification performance and fusion improvement.

Region		EEG	fNIRS	EEG-fNIRS	+ + REEG	+ + RfNIRS	+ + EEG	+ + fNIRS
Six-Bilateral	Accuracy %	89.8	85.6
	Sensitivity %	87.5	82.3
	Specificity %	92.0	88.0
	AUC %	95.7	92.7
VLPFC	Accuracy %	85.8	82.9	97.7	11.9***	14.8***	7.9***	12.1***
	Sensitivity %	87.0	79.5	96.6	9.6***	17.1***	9.1***	14.3***
	Specificity %	84.5	86.2	98.7	14.2***	12.5***	6.7***	10.7***
	AUC %	90.3	89.7	99.5	9.2***	9.8***	3.8**	6.8***
DLPFC	Accuracy %	83.7	80.8	92.5	9.0***	11.7***	2.7*	6.9***
	Sensitivity %	86.6	83.3	92.9	6.3***	9.6***	5.4**	10.6***
	Specificity %	80.8	78.3	92.0	11.1***	13.7***	0	4.0**
	AUC %	90.0	86.3	97.6	7.6***	11.3***	1.9	4.9**
FPA	Accuracy %	83.1	79.3	92.5	9.4***	13.2***	2.7*	6.9***
	Sensitivity %	86.0	82.9	92.0	6.0***	9.1***	4.5**	9.7***
	Specificity %	80.1	75.8	92.9	12.8***	17.1***	0.9	4.9**
	AUC %	92.5	86.3	97.0	4.5***	10.7***	1.3	4.3**

+ + REEG represents the improvement of fusing EEG and fNIRS in each subregion over EEG modality in that subregion; + + RfNIRS represents the improvement of fusing EEG and fNIRS in each subregion over fNIRS modality in that subregion; + + EEG represents the improvement of fusing EEG and fNIRS in each subregion over optimum EEG modality; + + fNIRS represents the improvement of fusing EEG and fNIRS in each subregion over optimum fNIRS modality, respectively. The marks ‘*’, ‘**’ and ‘***’ indicate that, the improvement of fusion was significant with p<0.05, p<0.01 and p<0.001, respectively.

As summarized in Table 1, the classification result of sole EEG signals demonstrated high performance in discriminating stress condition from control condition with 89.8% accuracy, 87.5% sensitivity, 92% specificity and 95.7% area under the curve using 6-bilateral electrodes over the entire PFC area. Similarly, the classification performance of sole fNIRS signals demonstrated 85.6% accuracy, 82.3% sensitivity, 88.0% specificity and 92.7% area under the curve using six-bilateral channels over the entire PFC. Due to its superior temporal resolution, the EEG outperforms fNIRS modality by + 4.2% in accuracy, 5.2% in sensitivity, 4% in specificity and + 3.0% in AUC over the entire PFC area. On the other hand, fNIRS outperforms EEG modality by + 1.7% in term of specificity over the VLPFC area.

The classification performance of CCA fusion model at each PFC subregion demonstrated significant improvement, compared to sole EEG and sole fNIRS modality. At the VLPFC subregion, it achieved 97.7% accuracy, 96.6% sensitivity, 98.7% specificity and 99.5% AUC respectively. Compare to sole EEG modality, the fusion of the dual-modality at the VLPFC subregion demonstrated an improvement of + 7.9% in the accuracy, + 9.1% in the sensitivity, + 6.7% in specificity and + 3.8% in the area under the curve, respectively. Compare to sole fNIRS modality, the fusion at the VLPFC subregion demonstrated improvement of + 12.1% in the accuracy, + 14.3% in the sensitivity, + 10.7% in specificity and + 6.8% in the area under the curve respectively. Similar levels of improvement by fusing data from the two modalities were found at the DLPFC and FPA subregion. Using two sample t-test, we found that the improvement introduced by the proposed fusion to be significant in terms of the classification accuracy, sensitivity, specificity and area AUC, as in Table 1 indicated with ‘*’ symbols.

In addition, we compared the performance between subregions in terms of classification accuracy, sensitivity, specificity and AUC using different modalities. We found that the VLPFC subregion outperforms, in terms of accuracy, other PFC subregions even using individual EEG/fNIRS modality. The significance of the performance difference at the VLPFC over other PFC subregions is marked with ‘*’ symbols, as shown in Table 2. Here, we define “mean overall performance” as the average of all performance metrics (accuracy, sensitivity, specificity and AUC). Using the proposed fusion method, the VLPFC subregion outperforms all others subregions in mean overall performance with significant p-values, as shown in Table 2.

Table 2. Performance evaluation between different pairs of PFC subregions.

Regions of comparison	Modality	Accuracy %	Sensitivity %	Specificity %	AUC %	Mean Overall Performance %
VLPFC - DLPFC	EEG	+ 2.1*	+ 0.4	+ 3.7**	+ 0.3	+ 1.6
	fNIRS	+ 2.1*	−3.8	+ 7.9***	+ 3.4**	+ 2.4*
	EEG-fNIRS	+ 5.2**	+ 3.7**	+ 6.7***	+ 1.9	+ 4.4**
VLPFC - PFA	EEG	+ 2.7*	+ 1.0	+ 4.4**	−2.2	+ 1.5
	fNIRS	+ 3.6**	−3.4	+ 10.4***	+ 3.4**	+ 3.5**
	EEG-fNIRS	+ 5.2**	+ 4.6**	+ 5.8***	+ 2.5*	+ 4.5**
DLPFC - PFA	EEG	+ 0.6	+ 0.6	+ 0.7	+ 2.5*	+ 1.4
	fNIRS	+ 1.5	+ 0.4	+ 2.5*	0	+ 1.1
	EEG-fNIRS	0	+ 0.9	−0.9	+ 0.6	+ 0.2

The mean overall performance is based on the average of accuracy, sensitivity, specificity and AUC values. The marks ‘*’, ‘**’ and ‘***’ indicate different levels of significance in performance between pairs of subregions, with p<0.05, p<0.01 and p<0.001, respectively.

4. Discussion

In this study we aimed to investigate if the effects on PFC functions under mental stress are focal to specific subregion or all subregions as a whole. We also investigated if fusion of bimodality (EEG-fNIRS) at each PFC subregion could improve stress detection. To evaluate, we performed simultaneous measurement of EEG and fNIRS on volunteer subjects while solving mental arithmetic task under two different conditions: control and stress. We performed statistical analysis on the EEG/fNIRS response of each subject individually as well as on the average of all the subjects, respectively. Region of interests within each PFC subregion were selected for individual modality (EEG and fNIRS) evaluation. Using CCA method, the regions highly correlated to stress were used in the fusion of the bimodality. The results of the CCA method showed that the cortical activities under maintained psychological stress were localized to the right VLPFC and the correlation variates significantly improve the detection rate of mental stress. To our knowledge, this is the first study using CCA fusion in mental stress study.

In this study, the stress inducement procedures were confirmed with alpha amylase level, as reported in our previous study [33]. All participants showed significantly increase in their alpha amylase level during stress as compared to control condition, with p<0.001.The results from EEG signals demonstrated significant decrease in alpha rhythm in all the PFC areas. Specifically, electrodes from the right PFC subregions: F4 and F8 shown in Fig. 3(a) were highly sensitive to stress, p<0.0001. On the other hand, one electrode ‘F7’ from the left PFC subregion responded differently with less significant difference in cortical activities, p<0.01 as compared to other electrodes. This difference in hemispheric activities is thought to be associated with frontal-alpha asymmetry. Thus, the decrease in the right frontal alpha rhythm is an evidence of negative response due to the stressors. The decrease in alpha rhythm on the PFC obtained in this study is consistent with previous emotional and anxiety studies and with studies that injected cortisol to human subjects [42–45]. Additionally, the difference in activities of the right and left PFC in this study is in line with previous EEG study that showed hemispheric difference under stress condition [46]. This result suggests that cortical activities shifted from diffused to focal under stress condition.

Likewise, fNIRS signals showed higher activation/increase in oxygenated hemoglobin concentration on the entire PFC area while solving the arithmetic task under control condition. Specifically, higher activation was found within the left PFC area as stimulated by the arithmetic task. The higher activation of oxygenated hemoglobin was observed consistently in all subjects. Significant decreases in the oxygenated hemoglobin concentrations at specific PFC subregion (the right VLPFC) were found when under stress condition, p<0.0001. It’s clearly seen that, the oxygenated hemoglobin shifted from diffused to focal under maintained psychological stress condition. Interestingly, the reduction in the oxygenated hemoglobin concentration at the PFC subregions in our study is consistent with previous fMRI human and animals stress studies [4, 47–49]. Furthermore, similar reduction in the oxygenated hemoglobin at the DLPFC was observed in subjects with post-traumatic stress disorder [50, 51]. The reduction in cortical activities has also been reported at the ventromedial PFC while inhibiting fear response [52]. Obviously, it is important to consider the methodological and neuroimaging modality differences between studies. Our approach may however have a greater sensitivity to detect stress-induced patterns on the PFC subregions under naturalistic settings.

The overall changes in suppression of alpha rhythm and oxygenated hemoglobin concentration in our study were found across all the subjects, unlike workload studies where two people had similar number and type of tasks, but they might perceive the workload differently [53]. High workload is regarded as an important but not a critical factor in the development of stress symptoms. It is possible to work hard in difficult and complex tasks, even under unfavorable conditions, without cognitive strain, psychosomatic complains, or adverse physiological effects [54]. In contrast, stress is regarded as a mental state in which the equilibrium between cognitive and energetic process is distributed by ineffective energy mobilization and negative emotions. The consistency and localization in reduction of the brain activities were observed as a result of the stress induced through time pressure and negative feedback of peer performance. The small standard deviation of ± 4.2 KIU/L in the alpha amylase level between all subjects under stress condition confirms that the reduced brain activities were due to the stressful situations and not to high workload.

Fusion of EEG-fNIRS features using proposed CCA method discovered the associations across the two modalities and estimates the components responsible for these associations. It jointly analyses the two modalities to fuse information without giving preference to either modality. The method identifies the relationships based on the natural inter-subject covariances between the modalities. Six pair of components were estimated based on their degree of correlation across the modalities as showing in Fig. 4. The different in correlation values across the modalities indicates the level of subregion activities (dysfunctioning) and the dominant of specific subregion to mental stress. The first pair of components having a correlation of 0.95 showing the most significant reduced in the oxygenated hemoglobin on the right VLPFC associated with a decrease on alpha rhythm at the time of stress. This association between modalities (given by the highest correlation at the right VLPFC) confirmed the dominant of the subregion to mental stress. The performance evaluation of the CCA also supports the dominant of right VLPFC to stress as measured by the classifier accuracy, sensitivity, specificity and AUC of each subregion compare with the entire PFC measured by individual EEG/fNIRS modality.

A few limitations of the current study should be noted. Firstly, this study was performed in healthy male subjects. Cortical activities on PFC are expected to behave differently with healthy subjects than in more elderly populations. Secondly, for the analysis of task-related changes and topographical distribution an averaging approach was used. More sophisticated analytical methods may be used such as the generalized linear model [55]. Additionally, in order to translate to clinical practice, more extensive research with a larger sample size involving both genders must be performed.

5. Conclusion

This study investigated the effects of mental stress on PFC subregions based on simultaneous measurement of EEG and fNIRS signals. Our results discovered that the cortical activities under maintained psychological stress shifted from diffuse to focal point on the right VLPFC subregion. The results of CCA confirmed the focal of cortical activities under stress to the right VLPFC subregion and demonstrated + 7.9% and + 12.1% improvement in the detection rate of mental stress as compared to sole EEG and sole fNIRS modality respectively. The current results clearly demonstrate the potential of the fused modalities as a monitoring tool for the early diagnosis and localization of mental stress in human adults, although its actual use in clinical diagnosis will be realized after further validation involving larger populations.

Funding

Ministry of Education, Malaysia (Higher Institution Centre of Excellence (HiCOE) scheme).