Electrodermal Activity for Measuring Cognitive and Emotional Stress Level

Abstract

Stress can lead to harmful conditions in the body, such as anxiety disorders and depression. One of the promising noninvasive methods, which has been widely used in detecting stress and emotion, is electrodermal activity (EDA). EDA has a tonic and phasic component called skin conductance level and skin conductance response (SCR). However, the components of the EDA cannot be directly extracted and need to be deconvolved to obtain it. The EDA signals were collected from 18 healthy subjects that underwent three sessions – Stroop test with increasing stress levels. The EDA signals were then deconvoluted by using continuous deconvolution analysis (CDA) and convex optimization approach to electrodermal activity (cvxEDA). Four features from the result of the deconvolution process were collected, namely sample average, standard deviation, first absolute difference, and normalized first absolute difference. Those features were used as the input of the classification process using the extreme learning machine (ELM). The output of classification was the stress level; mild, moderate, and severe. The visual of the phasic component using cvxEDA is more precise or smoother than the CDA's result. However, both methods could separate SCR from the original skin conductivity raw and indicate the small peaks from the SCR. The classification process results showed that both CDA and cvxEDA methods with 50 hidden layers in ELM had a high accuracy in classifying the stress level, which was 95.56% and 94.45%, respectively. This study developed a stress level classification method using ELM and the statistical features of SCR. The result showed that EDA could classify the stress level with over 94% accuracy. This system could help people monitor their mental health during overworking, leading to anxiety and depression because of untreated stress.

Introduction

Governmental “lockdown” to reduce COVID-19 transmission obtains social impact, including rises of stress. Even after the restriction got loosen, the stress issue still arises because people need to work harder to restore economic stability. Stress itself is divided into cognitive stress and neurological stress. Fear, anxiety, and constant worrying and racing thoughts are concluded as cognitive stress.[1] In comparison, neurological stresses are caused by neurological disease. One of the most significant factors that can affect mental health is stress. When someone receives too much external pressure, they will become restless, easily anxious, irritable, and difficult to concentrate on reducing their daily activities. If a person experiences so much stress, it can harm his physical and mental condition. The cause of stress plays a significant role in determining how much pressure you will possibly receive. The situations that may trigger stress are workload, heat, cold, noise, a room that smells strong, too bright light, a dirty environment, and inadequate ventilation.[2] Stress can lead to harmful conditions in the body, such as anxiety disorders and depression.

According to Indonesian mental medicine specialist Teddy Hidayat, there are 80%–90% of suicides due to mental-emotional disorders, especially depression.[3] Besides, Maramis and Maramis,[4] in their book Mental Medicine, state that stress contributes 50%–70% to the onset of cardiovascular disease, hypertension, cancer, and metabolic and hormonal disorders. Seizures in epilepsy can also be triggered by stress. Epilepsy is known to be the most common neurological disease. The WHO data for 2018 show that around 50 million people in the world experience this disorder. If epileptic seizures are not handled, patients can experience sudden unexpected death in epilepsy.[5] SUDEP is the sudden unexpected death in epilepsy patients. Moreover, the “work from home” order used to control the spread of a pandemic also led to cognitive stress and lowered the immune system.[6] Thus, monitoring mental health strategies to assist in managing stress and boosting the immune system has become a priority.

Birjandtalab et al.[7] monitored neurological statuses based on the type of stress (relaxation, physical stress, emotional stress, and cognitive stress) using biosignals, such as electrodermal activity (EDA), temperature, accelerometer, heart rate, and arterial oxygen level (SpO2). They used an unsupervised Gaussian mixture model to separate the type of stress with over 84% accuracy. Cho et al.[8] also combined physiological signals, such as photoplethysmogram (PPG), EDA, and skin temperature, to classify stress levels (mild, moderate, and severe) using Kernel-based extreme learning machine (ELM) with 24 statistical features and reached an accuracy over 95%.

Based on the previous research, EDA becomes one of the promising noninvasive methods widely used in detecting stress and emotion. EDA is a powerful method for real-time measurement and linearly related to arousal.[9] EDA measures the skin's electrical conductance changes, which depend on the quantity of sweat secreted by eccrine sweat glands in the hypodermis of the palmar and plantar regions. Sweat secreted in the palmar and plantar regions is caused mainly by central nervous activity related to affective and cognitive states, including mental or emotional sweating. Therefore, EDA could be used as an index of emotional or cognitive stimulation related to stress or mental health.[10,11,12]

EDA has a tonic and phasic component, where the tonic component is related to the slow-changing signals and background characteristics of the signal. The tonic component is also known as skin conductance level (SCL). In contrast, the phasic components are the fast-changing elements of the signal, also known as skin conductance response (SCR) or spontaneous fluctuation of skin response because it only exists in a certain period.[13,14] SCR arises when the sympathetic nervous system reacts to certain stimuli. Therefore, SCR can be associated with a stimulus and could help measure the cognitive stress level. However, the components of the EDA cannot be directly extracted. The results of the measurement signal from EDA sensors in the form of raw skin conductivity (SC) signals are usually measured in units of microsiemens. The SC signals need to be deconvolved to obtain the SCL and SCR component. If the original SC signals are not separated, overlapping SCRs will cause less precise information when feature extracted.[15] Hence, it is essential to separate the SCR and SCL signals by deconvoluting the SC signals.[8]

The main contribution of this paper is to explore the level of stress related to the biomedical signal, which has not been investigated previously. Most of the previous researches only concentrated between stress and nonstress conditions. Further, this paper investigates the use of EDA only to monitor stress without combining it with other biomedical signals, such as plethysmograph, electrocardiograph, and temperature. In addition, a comparison with the most popular deconvolution methods for deconvoluting SC signals and classifying the signal based on stress levels using few features is also provided. In practical, less sensor with less features is crucial for the real-time operation.

In this study, the deconvolution was obtained using the continuous deconvolution analysis (CDA)[16] method compared with the convex optimization approach, called a convex optimization approach to EDA processing (cvxEDA).[13] These methods were chosen because they are faster and more efficient than discrete nonnegative deconvolution analysis.[16] This deconvolution process aims to obtain a clear SCR with no overlapping and get the zero baseline SCR driver. The clear SCR peaks are essential for identifying the responses related to the stimulus (stressor). The following step is extracting the feature of the SCR for classifying the cognitive stress level based on the provided stimulus. This study used statistical features such as sample average, standard deviation (SD), first absolute difference, and normalized first absolute difference, which are simple and commonly used to analyze emotion and short-term anxiety recognition.[17,18,19] The ELM was chosen as the classifier in this study because of its ability to obtain high accuracy with fast and efficient training.[20] This system could help overcome the harmful effect of untreated stress because of the ignorance of stress symptoms. Moreover, it could also help people monitor their mental health during overworking, reducing suicides because of depression and anxiety.

Materials and Methods

Experiment protocol

Generally, measuring the emotional and cognitive stress system is designed by connecting a surface electrode Ag-AgCl to the EDA sensor from BITalino (fs = 100Hz)[21] and then transmitted to the laptop using a Bluetooth connection. This study collected data from 18 healthy subjects (18–22 years old; ten females and eight males) who have filled in the informed consent. Each subject was in good health (no mental disorder or depression), enough sleep, not taking drugs, and relaxed conditions before measurement started. The SC signals were recorded using Open Signals software,[22] and the EDA sensors were placed on the subject's palm, as shown in Figure 1.

Figure 1.

Electrode placement of electrodermal activity sensor

Before collecting the data, subjects were asked to wash their hands and then sat comfortably in front of the laptop while they wore the provided headphones. The headphones became the source of the audio stimulus, while the laptop became the visual stimulus. The relaxing songs were played for 12 min before the test began to make the subject relaxed. After that, the subjects were asked to answer the Stroop test in three sessions which its difficulty increased in each session by varying the given stimulus. The number of questions in session 2 and session 3 also increased, from 75 questions in session 1 to 100 in sessions 2 and 3. The time for answering the Stroop test was also shortened. If the subject chooses the wrong color, there would be a buzzer sound, and a red cross appears to cause more tension. The stimulus is given from the beginning of the recording, which starts with the Stroop test for about 3 min, and then continued with session 2 and session 3 for about 4 min.

The Stroop test is a cognitive test provided in Bahasa with six color variants: red, blue, green, yellow, purple, and brown. During the test, the subjects were asked to choose the color based on the font color, not based on its writing. The first session had three color variants without any interference, while session 2 had five color variants interrupted with noisy or shocking sound, as shown in Table 1. The Stroop test would continue to session 3 by giving six color variants interrupted with frightening and terrifying sounds or videos, as shown in Table 2. The stimuli in this experiment are considered the emotional stressors combined with cognitive stress because the stimulus was given to disturb the concentration on answering the test with terrified and frightened subjects. The cognitive and emotional stressors were chosen because it represents human fear that indicates stress, and it could be measured well by EDA.[23] Because the given stressors were multiplied, it is assumed that session 1 is the lowest level of stress while session 3 is the highest level of stress. After the recording ended, the participants were asked to fill in the postquestionnaire to determine how they feel during the test. This questionnaire was used for analyzing the results and comparing the work with the actual condition of the subjects.

Table 1.

Audio stimuli given in session 2

Time (seconds)	Given stressor	Duration (second)
14	Cats fight	20
36	Squall	27
80	Horror night atmosphere	7
s100	Lightning sound	22

Table 2.

Audio and Visual stimuli are given in session 3

Time (seconds)	Given stressor	Duration (second)
25	Horror night atmosphere	7
33	Lightning sound	22
52	Horror night atmosphere	7
54	Lightning sound	22
70	Pocong (Indonesian ghost)	2
80	Grinder	30
95	Maggot	13
120	Hollow hand	15

Electrodermal activity deconvolution method

Continuous deconvolution analysis

Before doing the deconvolution process, the SC signal was preprocessed at first using downsampling and smoothing to eliminate the ripples that appeared in the signal. CDA method is initially inspired by the pharmacokinetic model of sweat glands, which has two compartments called sweat gland compartment and stratum corneum compartment depicted in Figure 2. The kinetic model showed the change in sweat concentration because of diffusion. The stratum corneum has a larger area than the sweat gland and called restricted diffusion.[11] The relationship of those two compartments was shown in the first-order differential equation in Equation 1 (for the sweat gland) and Equation 2 (for the stratum corneum).

Figure 2.

Mathematical model of sweating process for continuous deconvolution activity

The solution of Equations 1 and 2 is called impulse response function (IRF). It is also called Bateman function, which can be seen in Equation 3. It is assumed that sweat concentration changes only due to diffusion. Based on Equation 3, the Bateman function showed that the onset would rise steeply and fall slowly when the τ₁ and τ₂ are constant. Criterion (C) is the gain or Bateman area.[24] The initialization value of τ₁ and τ₂ was 2 and 40, respectively. The gain was 1/sampling rate (SR = 100 Hz).

The SC is constructed as a model of a convolution process in sudomotor nerve activity (SMNA), which is a part of the sympathetic nervous system. SMNA is the summation of SCR and SCL drivers.[16] The SMNA will be obtained from the deconvolution of SC with IRF, and it could be found in Equation 4. This value is the main item to get the SCL and SCR being separated. The τ value in Equation 3 will be optimized by changing the C value until it reached the minimum value. Besides that, the criterion will be calculated by using Equation 5. The indist in Equation 5 is the indicator of differentiation among hills formed in the SCR, neg is the negative proportion of SCR driver shown in Equation 6, and α is a constant that is equal to 6s² / (s μS).

SC = SMNA * IRF (4)

c = indist + neg. α (5)

The SMNA collected from the deconvolution process was then undergone a smoothing process with a 200 ms-wide Gaussian window to eliminate the ripples in the deconvolution process. The peak in this signal was detected by using a threshold of 0.2 μS. The peaks with a value more than the threshold are considered as SCR, and the other peaks will be flattened by using interpolation to arrange SCL drivers. The SCL drivers were calculated by using the interpolation of the point lower than the threshold with a cubic-spline fitting method.

A convex optimization approach to electrodermal activity processing (cvxEDA)

The cvxEDA method separated the phasic component of the SC signal. This method was proposed by Greco et al.,[13] which split the SC signal by decomposing the signals into three different components, called phasic (r), tonic (t), and a noise component (ϵ) that can be found in Equation 7. Before the data were processed, it was normalized by using Z-score ( N = the sample size).

y = r + t + ∈     (7)

The phasic component (r) is constructed by using the Bateman function in Equation 3. The phasic response then was utilized the ARMA model to make the ARMA cascade shown in Equations 8 and 9, in which p is the SMNA and M A are tridiagonal matrix.

q = A^-1 p, r = Mq     (8)

r = MA^-1 p     (9)

The tonic component (t) is represented as the cubic B-spline basis function shown in Equation 10, in which B is a cubic B-spline basis function matrix, λ is the vector of spline coefficients, C is an Nx2 matrix with C_i,1 = 1 and C_i,2 = i / N, and d is a 2x1 vector with the offset and slope coefficients for the linear trend.

t = Bλ + C     (10)

The final model was completed by substituting Equation 9 and 10 into Equation 7. The final aim is to look for the maximum posterior spike of P and t for measuring the EDA signals.

By substituting Equation 11 into the final model, the convex optimization approach is defined as shown in Equation 12.

Subj. to Aq(x) ≥ 0

The fast time constant 𝜏₁ = 0.7 s was used for all subjects. The optimal slow time constant 𝜏₀ was determined on a per-subject basis as the value 𝜏₀ ∈ [2.0, 4.0] s that minimized the λ²-residual norm after fitting the cvxEDA model. Fixed values α =0.0008 and γ =0.01 were chosen during previous studies and implemented throughout this analysis.

Feature extraction and classification

Sample average (μ_x)

It represents the average of all raw signals calculated by using Equation 13.

Standard deviation (σ _x)

It represents the deviation of the data around the average and can be calculated by using Equation 14.

First absolute difference (ᵟx)

It represented the average of the absolute value of raw signal first difference and was calculated using Equation 15.

Normalized first absolute difference ( δ̄x )

Before calculating the normalized first absolute difference, the signal was normalized using Equation 16 and then continued to calculate its first absolute difference shown in Equation 17.

All four features were then normalized to get a range of 0 and 1 shown in Equation 18. The result was saved as an HDF5 file as the input of the classification.

Extreme learning machine

The nonlinear classifier was used in this study, called the ELM. It could distinguish the stress level based on the features used in this study. ELM is a neural network that has a single hidden layer of feedforward with L hidden neuron, activation function G(x) as the following Equation 19:[20]

where a_i is an input weight that connected j-input nodes (x_j) with i-hidden nodes, b_i is an impact factor from the i-hidden node, while β_i is an output weight that connected the i-hidden node with output node. The t_j is a threshold or the input target from the j-input node.

ELM was implemented to minimize the error training using Moore–Penrose pseudo inverse by multiplying the inverse matrix H (Hǂ) and target (T). The training was required to obtain weight values that were suitable for testing by using the following algorithm:

1. Start by defining the input weight (a_i) and impact factor (b_i) randomly based on the suitable activation function G(x). The sigmoid function was used as the activation function

   

2. Calculate all the outputs at the hidden neuron using the activation function and substitute them into matrix H

   

   

3. Calculate the output weight (β) from the hidden layer.

β = H^†T (23)

While the testing was performed by using the following algorithm:

1. Inserting the parameters obtained from the last training

2. Calculate all the output at the hidden neuron using the activation function and substitute them into matrix H (Equations 21 and 22)

3. Calculate the output node by multiplying the matrix H with output weight (β) from the training process.

The result of the output node O_j was then compared to the input target (T) to obtain accuracy.

The classification, comprising training and testing, was conducted using K-fold cross-validation (K = 6). The number of data in each subset is 9 data. Thus, the total training data was 45 data (5 subsets), and the testing data was 9 data (1 subset). The accuracy of the method was calculated from the average of each accuracy.

Results and Discussion

EDA signals were recorded from eighteen subjects with eight males and ten females in the age range 18-22 years with the criteria mentioned previously. Each subject was recorded from session 1 to session 3, and the result, as shown in Figures 3-5. This signal shows the SC signal as the raw data and is then deconvolved using CDA methods compared with the cvxEDA method to obtain the SCR or phasic component. Both methods are indicated by the fluctuated signal and the peaks that change rapidly over time caused by the stimulus. As shown in Figures 2-3, the left side (a) resulted from the CDA method, while the right side (b) resulted from the cvxEDA method. The amplitude differences on the raw signal of the cvxEDA method compared with the raw signal of the CDA method were caused by Z-score normalization done in the preprocessing. However, it did not change the characteristics of the raw signal that contains the SC signals. The results of the deconvolution process and its driver signals were seen on the second row and the third row of the figures, respectively.

Figure 3.

One of the results of the electrodermal activity deconvolution using (a) the continuous deconvolution activity method and (b) the convex optimization approach to electrodermal activity during recording on session 1

Figure 5.

One of the results of the electrodermal activity deconvolution using (a) the continuous deconvolution activity method and (b) the convex optimization approach to electrodermal activity during recording on session 3

Figure 4.

One of the results of the electrodermal activity deconvolution using (a) the continuous deconvolution activity method and (b) the convex optimization approach to electrodermal activity during recording on session 2

Based on the SCR signals from both methods, the visual of the phasic component using cvxEDA is more precise or more smooth compared to the CDA's result. The peaks were more visible to see on the cvxEDA method than on the CDA method because it depends on the noise power line. Therefore, the filtering process is essential on the CDA method because the phasic CDA signal's sparsity is less remarkable than the cvxEDA, which shows a robust behavior.[11]

However, both methods could separate SCR from the original SC raw and indicate the small peaks from the SCR, as pointed with the square box on the figures. The SCR's peaks were analyzed from subjects while doing the Stroop test with various emotional stressors is shown in Tables 1 and 2. The peaks occur as the response of the body is caused by the given stimulus or stressors. Some stressors produce significant peaks, while others not. The small SCR peak occurred because the subject already made a mistake before, or the subject experiences the repeating stressor, so the subject was not afraid of the sound anymore. This phenomenon is called habituation.[10] This phenomenon was followed by the postexperimental questionnaire, which also stated so. Hence, stress quickly falls back during the adapted situation. The very high peak occurred because of the stress made by the fear and surprise feelings at something new that had never been felt before.

The subject responds by covering her eyes or looks away from the monitors. The results of the postexperimental questionnaire also showed that the subject did feel scared when they saw a ghost, maggot, and hollow hands. The habituation phenomenon also leads to the condition of the EDA lability.[10] People who are considered EDA labile may be described as calm, deliberative, restrained, cooperative, and responsible, while EDA-stable people tend to be active, emotionally expressive, animated assertive, more irritable, more antagonistic, more impulsive, and more irresponsible.[10,23] Both depressed and suicidal patients indicated electrodermal hypoactivity and concluded as EDA stable.[25]

Besides the habituation phenomenon or EDA lability, Figures 3-5 pointed out that the number of peaks also increases during the rises of the session. The more stimuli received correlated to the cognitive and emotional stress, the greater the number of peaks generated. Therefore, SCR could also classify the level of stress, primarily cognitive and emotional stress. This characteristic underlies the feature extraction for classifying the level of cognitive and emotional stress level based on the SCR or phasic component.

The next following step was the feature extraction process, calculating the average SCR value, SD, first absolute difference, and normalized first absolute. All calculations are then normalized so that the values of each feature are in the range 0–1 using the min–max normalization, as shown in Figure 6. After being normalized, the resulting array was stored in the HDF5 file to become input in the classification process using ELM. ELM would classify the level of cognitive and emotional stress into mild, moderate, and severe. The result of the error and accuracy in the classification process is shown in Table 3.

Figure 6.

The resulting array of features after being normalized

Table 3.

Error calculation in classifying the level of cognitive and emotional stress from both methods

Fold	CDA				cvxEDA
	Hidden Neuron=30		Hidden Neuron=50		Hidden Neuron=25		Hidden Neuron=50
	Error Training	Error Testing	Error Training	Error Testing	Error Training	Error Testing	Error Training	Error Testing
1	0.04	0.22	0.00	0.00	0.00	0.55	0.00	0.00
2	0.04	0.33	0.00	0.00	0.00	0.44	0.00	0.00
3	0.04	0.22	0.00	0.00	0.00	0.44	0.00	0.11
4	0.04	0.22	0.00	0.00	0.04	0.55	0.00	0.11
5	0.04	0.22	0.00	0.00	0.04	0.55	0.00	0.11
6	0.04	0.33	0.00	0.22	0.00	0.44	0.00	0.00
Average Error	0.04	0.26	0.00	0.04	0.01	0.50	0.00	0.05
Accuracy		73.36%		95.56%		50.00%		94.45%

Table 3 shows that the higher the number of hidden neurons, the higher the accuracy will be obtained. In this case, 50 hidden neurons were the best choice because it produced a training error of 0% with an average accuracy for the CDA method of 95.56%, which is much better than 30 hidden neurons which still have errors in the training of 0.04% and an average accuracy of 73.36%. The average accuracy for the convex optimization method with 50 hidden neurons also obtained a high accuracy value, around 94.45%. This result is much better than 25 hidden neurons, which produces a training error of 0.0148% and an average accuracy of 50%.

Using a one-sample t-test, a nonsignificant difference was found between the CDA method error (mean = 0.0367; SD = 0.0898) and cvxEDA method error (mean = 0.0367; SD = 0.0898) while deconvoluting the SCR and SCL signal (t (5)=2.23; P > 0.05). However, even the deconvolution result shows that cvxEDA is clearer and smoother than CDA; both deconvolution methods could give high accuracy. Thus, the smoothness or clearness peak is not the only variable in classifying the level of stress.

The results showed that EDA has the potential for classifying the stress level. Stress-level classification using EDA gave the accuracy as high as the accuracy of the EDA if it combined with other physiological signals, such as skin temperature, accelerometer, heart rate, and arterial oxygen level, and PPG.[7,8,17] Moreover, based on the deconvolution and classification process, it showed that the CDA or cvxEDA for separating the SCR could obtain over 94% accuracy, even with four statistical features. However, the study about EDA for stress-level identification, such as stress levels due to physical, cognitive, or neurological stress that could trigger seizures in epilepsy, still needs to be discovered. It is essential to expand the study to monitor their mental health during overworking, leading to anxiety and depression because of untreated stress.

Conclusions

This study developed a stress-level classification method using ELM and the statistical features of SCR. The CDA and cvxEDA methods could be used as the deconvolution method to separate SCR from the original SC. The SCRs are a potential component to identify EDA lability related to cognitive and emotional stress, including specifying stress levels, which could help people monitor their mental health during overworking, leading to anxiety and depression because of untreated stress. Furthermore, the study about the EDA needs to continue to give insight into neurological stress that could lead to neurological diseases, such as trigger seizures in epilepsy.

Financial support and sponsorship

None.

Conflicts of interest

There are no conflicts of interest.