Emotion classification using flexible analytic wavelet transform for electroencephalogram signals

Abstract

Emotion based brain computer system finds applications for impaired people to communicate with surroundings. In this paper, electroencephalogram (EEG) database of four emotions (happy, fear, sad, and relax) is recorded and flexible analytic wavelet transform (FAWT) is proposed for the emotion classification. FAWT analyzes the EEG signal into sub-bands and statistical measures are computed from the sub-bands for extraction of emotion specific information. The emotion classification performance of sub-band wise extracted features is examined over the variants of k-nearest-neighbor (KNN) classifier. The weighted-KNN provides the best emotion classification performance 86.1% as compared to other KNN variants. The proposed method shows better emotion classification performance as compared to other existing four emotions classification methods.

Introduction

Brain computer interface (BCI) is the medium of establishing the connection between brain and external peripherals. The BCI system can provides support for the physically disabled and impaired people. The BCI system can be develop by analysis, transforming, monitoring, and evaluation of electrical activity of the brain known as electroencephalogram (EEG) signals, which is extracted by placing electrodes over the scalp of the brain. The emotion recognition based BCI system can be develop by classification of EEG signals.

Most of the features extracted from EEG signals are in time domain, frequency domain and time-frequency domain for classification of emotions. The Kolmogorov entropy and the principal Lyapunov exponent parameters [1]. Fractal dimension [2], event related potential and event related oscillation based features [3], higher order crossing based features [4], the time domain and frequency domain based features [5], time-frequency based features [6], short time Fourier transform based features [7], the spectrogram, Zhao–Atlas–Marks and Hilbert–Huang spectrum based features [8], combination of surface Laplacian (SL) filtering, wavelet transforms (WT) [9], multiwavelet based features [10, 11] are explored as input to classifier for classification of emotion EEG signals.

Wavelet transform with surface Laplacian filter have been fed into k-nearest neighbor (KNN) classifier [12]. Higher order crossing (HOC) features used different classifiers for emotion classification [13]. Fractal dimension used as feature for emotion classification [14]. Features extracted from Discrete wavelet transform have been fed into artificial neural network (ANN) classifier for emotion classification [15]. Entropy of bands of EEG signals is used as features to least squares support vector machine (LS-SVM) with radial basis function (RBF) kernel for classification of emotions [16].The power spectral density (PSD) based features are classified the emotions using SVM classifier [17]. Features based on magnitude squared coherence (MSCE) are fed as input to KNN classifier for emotions classification [18]. Wavelet based features are proposed to classify emotions [19].The late positive potential (LPP) based selected frequency domain features are tested on KNN and SVM classifier for emotion classification [20].

Recently, recurrence quantification analysis (RQA) with learning method has been explored for emotion recognition [21]. Power spectral density (PSD) of EEG signals and statistical features of ECG used as input to SVM classifier for emotion recognition [22]. Correlation-based feature selection (CFS) and a KNN have been explored for three class emotion classification [23]. Wavelet packet decomposition based features has been explored for emotion recognition [24]. Evolutionary computation (EC) algorithms used for optimal features selection method for emotion recognition [25]. Correlation-based subset selection technique with higher order statistics has been used for emotion classification [26].

In this paper, an audio–video stimuli based EEG dataset is recorded and flexible analytic wavelet transform is proposed to classified emotions with single EEG channel. FAWT explores the complexity of EEG signal into sub-bands. The statistical measures extract the different emotions information from FAWT provided sub-bands. The emotion classification ability of statistical measures is evaluated over the variants of KNN classifier.

Methodology

The proposed methodology is described under the following subsections: EEG signals (data-set), feature extraction, and classification.

Data-set

In this paper, audio–video stimuli based experiment is conducted for recording of happy, fear, sad, and relax emotions EEG dataset. The subjects who are participated in the experiment has age group of 20–25 years. All subjects are the undergraduate students of Indian Institute of Information Technology Design and Manufacturing, Jabalpur. Before beginning the experiment subjects are introduced with the experiment procedure and rating scale. The 12-video clips, 3 for each emotion are selected based on the questioner session of 30 volunteers, who do not participate in the experiment. The first neutral or relax emotion clip is shown to subjects prior to any clip for generating the baseline signal and avoid the daily dependencies on the recorded signals [27, 28]. The subjects evoked emotional states is assessed by Self-Assessment Manikin (SAM) standard [29].

The EEG signals are acquired using a 24-channel EEG traveler system. The bipolar montage is considered for acquiring the emotion EEG signals with 10–20 international electrode placement system. The EEG signals are recorded at 256 Hz sampling frequency using different bipolar channels. In this paper, audio–visual stimulation method is used for elicitation of emotion EEG signals, because this stimulus is more effective for evoking emotions as compared to alone audio or visual stimulus [30]. For a stimulus the frontal region of brain is responded more actively as compared to other regions, so electrode positions FP1, FP2, F3, F4, F7, and F8 recorded EEG signals are used for emotion classification [31]. An example of happy, fear, sad, and relax emotions EEG signals is shown in Fig. 1.

Fig. 1.

EEG signals of happy, fear, sad, and relax emotions EEG signals

Flexible analytic wavelet transform

In this work, a recently proposed flexible analytic wavelet transform (FAWT) is used for emotion classification. FAWT has several advantages as compared to traditional dyadic wavelet transform. FAWT low-pass and high-pass channels are allowed arbitrary sampling rates, which provide flexible TF covering [32]. FAWT uses complex pair of atoms in high-pass channel that gained the flexibility in selection of transform parameters. Because of these merits, FAWT finds applications for analysis of complex oscillatory signals like vibrations and ECG [33, 34].

FAWT uses iterative filter bank structure for decomposition of EEG signals. The filter bank stages of FAWT consist of one LP-channel and two HP-channels. The HP-channels consist of Hilbert transform pair of atoms for analyzing the positive and negative structure. Such complex structure provides the flexible selection of FAWT parameters. The parameters of FAWT are: LP-channel up sampling factor k and down sampling factors l; HP-channel up sampling factor m and down sampling factors n; and quality factor (QF) controlling parameter $\beta$. These parameters are controlled the frequency response of FAWT filter banks. The HP filter $H(\omega )$ and LP filter $G(\omega )$ used in FAWT filter bank are can be defined as, respectively [32]:

$$\begin{array}{c}\hfill H(\omega )=\left\{\begin{array}{cc}{(kl)}^{1/2},\hfill & |\omega |<{\omega }_p\hfill \\ {(kl)}^{1/2}\theta (\frac{\omega -{\omega }_p}{{\omega }_s-{\omega }_p}),\hfill & {\omega }_p\le |\omega |\le {\omega }_s\hfill \\ {(kl)}^{1/2}\theta (\frac{\pi -\omega +{\omega }_p}{{\omega }_s-{\omega }_p}),\hfill & -{\omega }_s\le |\omega |\le -{\omega }_p\hfill \\ 0,\hfill & |\omega |\ge {\omega }_s\hfill \end{array}\right.\end{array}$$ 1

$$\begin{array}{c}\hfill G(\omega )=\left\{\begin{array}{cc}{(2mn)}^{1/2}\theta (\frac{\pi -\omega -{\omega }_0}{{\omega }_1-{\omega }_0}),\hfill & {\omega }_0\le \omega <{\omega }_1\hfill \\ {(2mn)}^{1/2},\hfill & {\omega }_1\le \omega <{\omega }_2\hfill \\ {(2mn)}^{1/2}\theta (\frac{\omega -{\omega }_2}{{\omega }_3-{\omega }_2}),\hfill & {\omega }_2\le \omega \le {\omega }_3\hfill \\ 0,\hfill & \omega \in [0,{\omega }_0)\bigcup ({\omega }_3,2\pi )\hfill \end{array}\right.\end{array}$$ 2

where

$$\begin{array}{c}\hfill {\omega }_p=\frac{(1-\beta )\pi +ϵ}{k},{\omega }_s=\frac{\pi }{l},{\omega }_0=\frac{(1-\beta )\pi +ϵ}{l},\\ \hfill {\omega }_1=\frac{k}{\mathit{lm}}\pi ,{\omega }_2=\frac{(\pi -ϵ)}{m},{\omega }_3=\frac{(\pi +ϵ)}{m},ϵ\le \left(\frac{k-l+\beta l}{k+l}\right)\end{array}$$ 3

the terms $\theta (\omega )$ can be defined as [31],

$$\begin{array}{c}\hfill \theta (\omega )=\frac{1}{2}(1+cos\omega )\sqrt{2-cos\omega }\text{for}\omega \in [0,\pi ].\end{array}$$ 4

To satisfied perfect reconstruction condition of FAWT filter bank, the QF controlling parameter $\beta$ is defined as [31],

$$\begin{array}{c}\hfill \left(1-\frac{k}{l}\right)\le \beta \le \left(\frac{m}{n}\right)\end{array}$$ 5

With these constraints, the FAWT is tuned with following values of parameters: FAWT stages $J=15$, $k=3$, $l=4$, $m=1$, and $n=2$ for decomposition of EEG signals [31]. An example of FAWT decomposed EEG signal into SBs is shown in Fig. 2.

Fig. 2.

FAWT provided sub-bands of an emotion EEG signals

Feature extraction

In this work, statistical measures of FAWT provided sub-bands are used as features for emotion classification. The statistical measures are minimum value (MIN), maximum value (MAX), interquartile range (IQR), mean absolute deviation (MAD), activity ($A_c$), and entropy (H(S)). The minimum and maximum values are the part of five point summary in a database, which can assessed the variation of data points. The IQR and MAD are assessed the variability of sub-bands. The activity is a measures of power of the sub-band and entropy is assessed the complexity of sub-band. The equations used for computation of these features are defined as [35–37],

$$\begin{array}{c}\hfill \text{MIN}=min(S)\end{array}$$ 6

$$\begin{array}{c}\hfill \text{MAX}=max(S)\end{array}$$ 7

$$\begin{array}{c}\hfill \text{MAD}=\frac{1}{N}{\displaystyle \sum _{i=1}^N|S_i-\mu |}\end{array}$$ 8

$$\begin{array}{c}\hfill \text{IQR}=Q_3-Q_1\end{array}$$ 9

$$\begin{array}{c}\hfill A_c=\frac{1}{N-1}{\displaystyle \sum _{i=1}^N{(S_i-\mu )}^2}\end{array}$$ 10

$$\begin{array}{c}\hfill {\displaystyle H[S]=-\sum _{i=1}^{N}p(S_i)log[p(S_i)]}\end{array}$$ 11

where S represents sub-band (SB), N number of samples in SB, $\mu$ mean of SB, $Q_1$ first quartiles of SB, $Q_3$ third quartiles of SB, and $p(S_i)$ represents the probability of SB samples lying nearby of particular value. The computed features are normalized by unit length normalization method to avoid the biased classification [38].

Classification

KNN is a one of the simplest non-parametric classification algorithm, which has good predictive power for instances and requires less computation time. KNN classified the testing instance according to the k-nearest training instances by computing each possible category probabilities. The largest category probability defines the class of testing instance. The probability density function $P(F,c_k)$ for a feature vector F of class $c_k$ can be defined as [39],

$$\begin{array}{c}\hfill P(F,c_k)=\left[\sum _{d_i\in K_n}Si(F,d_i).y(d_i,c_k)\right]\end{array}$$ 12

where $K_n$ represents number of nearest neighbors, $d_i$ represents one of the neighbors of training set $y(d_i,c_k)$, and $Si(F,d_i)$ represents similarity function for F and $d_i$. In this paper, six variants of KNN fine, medium, coarse, cosine, cubic, and weighted are tested for emotion classification [40].

Results and discussion

FAWT decomposes EEG signals into sub-bands and statistical measures are computed from each sub-band for classification of emotions. The dimension of sub-band wise features set for each emotion is $490\times 6$, where rows represent the number of features and columns represent the number of instances. The emotion discrimination capability of proposed features set is assessed by their histogram distribution. Figure 3 shows an example of histogram of extracted features set for happy, fear, sad, and relax emotions. From this figure, it can be observed that proposed features set is shown different occurrences of feature values for different emotions. The different histogram distributions for emotions signified that the proposed features set has ability to discriminate emotions properly.

Fig. 3.

Histograms of a sub-band features for different emotions

In this paper, frontal bipolar channels (FP1-F3, FP1-F7, FP2-F4, and FP2-F8) recorded EEG signals are compared for emotion classification. The classification performance of each bipolar channel features is evaluated with variants of KNN classifier. The fine, medium, and coarse KNNs are defined in terms number on neighbors, which are 1, 10, and 100, respectively, while their distance metric is euclidean. The cosine and cubic KNNs are defined by their used distance metrics are cosine and minikowski, respectively and number of nearest neighbors 10 is common in both variants. The weighted-KNN contains number of nearest neighbors 10 and distance metric is euclidean, but the squared inverse distance weight is considered. The classification performance of each KNN variant is evaluated with 10-folds cross validation method. Tables 1, 2, 3 and 4 present the SB-wise classification performance with KNN variants for all frontal channels. In all KNN-variants the fine and weighted KNNs are provided better classification accuracy for all sub-bands as compared to other variants. Although, highest emotion classification accuracy 86.1% is obtained by weighted-KNN for SB7 of FP2-F8 bipolar channel. The confusion matrix of SB7 of FP2-F8 channel is shown in Table 5, where the diagonal elements show the exactly classified instances and off diagonal elements show the misclassified instances of different classes. In this table, the emotion-wise classification accuracy is also computed from the diagonal elements. The classification accuracy of happy, fear, sad, and relax emotions are 96.12%, 78.37%, 72.24%, and 97.76%, respectively.

Table 1.

Sub-band wise classification accuracy of FP1-F3 channel for different KNN classifiers

KNN variants	SB1	SB2	SB3	SB4	SB5	SB6	SB7	SB8
Fine KNN	81.9	73.9	84.4	74.3	40.9	60	51.4	47.8
Medium KNN	71.2	67.7	79.3	72.5	40.9	59	47.1	45.7
Coarse KNN	61.9	58.4	63.2	59.4	38.1	47.9	42.8	42
Cosine KNN	74.5	71.4	70.3	68.7	42.9	57.2	45.8	45.6
Cubic KNN	65.7	63.9	77.4	72.3	41.7	57.9	47.5	44.5
Weighted KNN	80.4	73.9	84.3	75.9	44.1	62.4	50.9	48.5

Table 2.

Sub-band wise classification accuracy of FP1-F7 channel for different KNN classifiers

KNN variants	SB1	SB2	SB3	SB4	SB5	SB6	SB7	SB8
Fine KNN	71.2	80.9	78.6	71.4	45.2	62.2	53.1	51.4
Medium KNN	64	75.3	73.2	67.7	42.1	62.6	50.9	49.4
Coarse KNN	55.1	59.7	52.2	54.7	36.7	51.5	44.9	44.6
Cosine KNN	63.3	73.7	65.6	58.8	43.5	57.3	52.5	47.4
Cubic KNN	60	71.8	71.4	67	42.2	61.5	52.8	50.3
Weighted KNN	69.4	79.6	78.9	70.9	44.9	64.1	54.1	52.4

Table 3.

Sub-band wise classification accuracy of FP2-F4 channel for different KNN classifiers

KNN variants	SB1	SB2	SB3	SB4	SB5	SB6	SB7	SB8
Fine KNN	72.4	70.9	70.3	73.7	73.4	78.6	85.2	79.7
Medium KNN	65.3	67.6	66.6	70.6	70.3	79.4	84	78.3
Coarse KNN	55.7	59.1	56.3	58.3	57.9	68.8	78	69.4
Cosine KNN	68.7	60.7	61.4	62.1	61.3	78.2	84.8	81.4
Cubic KNN	60.6	65	64.9	70.2	69.6	78.6	83.2	76.6
Weighted KNN	71.3	72	71.8	74.1	73.5	79.9	86.1	80.7

Table 4.

Sub-band wise classification accuracy of FP2-F8 channel for different KNN classifiers

KNN variants	SB1	SB2	SB3	SB4	SB5	SB6	SB7	SB8
Fine KNN	70.5	78.1	66.4	65.8	68.3	67.7	57.4	55.7
Medium KNN	64.2	73	63.5	65.8	67.5	67.1	57	56.5
Coarse KNN	56.9	60.1	50.5	54.3	57.4	58.9	55.7	57.6
Cosine KNN	64.1	66.6	60.5	59.7	65.3	65.5	56.5	54.6
Cubic KNN	58.7	69.5	61.9	64.9	66.3	64.8	57.1	56.9
Weighted KNN	70.6	78.4	68.3	68.1	69.7	69.7	58.1	58.1

Table 5.

The confusion matrix of weighted KNN classifier for SB7 of FP2-F4 channel

Emotions	Happy	Fear	Sad	Relax
Happy	471	3	13	3
Fear	1	384	103	2
Sad	1	126	354	9
Relax	0	9	2	479
ACC (%)	96.12	78.37	72.24	97.76

Table 6 presents the emotion classification performance comparison report of proposed method with other existing methods. The all methods in this table are worked for classification of four emotions. These methods are compared in terms of used stimuli for evoking the emotions, proposed feature extraction approach, and used classifier. In this table it can be seen that in all existing methods audio–video stimuli based method proposed by Bajaj [28] is provided highest classification accuracy 84.79%. The proposed method is also used audio–video stimuli and obtained better classification accuracy 86.1% as compared to Bajaj [28]. The classification performance of proposed method signified its utility for emotion classification, which can be used in emotion recognition based BCI system.

Table 6.

Emotion classification performance comparison report

Authors	Stimulus, classes	Feature extraction method	Classifier	Accuracy (%)
Mikhail [17]	Perform movements, 4	PSD	SVM Linear	55.75
Majid Mehmood [20]	IAPS, 4	Power-spectrum, statistical and LPP	SVM	58
Wang et al. [5]	Video, 4	Minimum redundancy, maximum relevance	SVM	66.5
Bhatti [19]	Music, 4	Statistical, PSD, FFT, and WT	MLP	78.11
Lin [7]	Music, 4	FFT	SVM RBF	80.86
Khosrowabadi et al. [18]	IAPS with music, 4	MSCE	KNN	84.5
Bajaj [10]	Audio–video, 4	Multi wavelet	MC-LS-SVM	84.79
Proposed method	Audio–video, 4	FAWT	KNN	86.1

FFT fast fourier transform, MLP multi-layer perceptron

Conclusion

In this work, audio–video stimuli based EEG dataset of four basic emotions is recorded and FAWT-based features are proposed for classification of emotions. FAWT decomposed recorded EEG signal into sub-bands and statistical features are extracted from sub-bands for each frontal channel. The sub-band wise features of each channel are tested on the variants of KNN classifier. The different frontal channels features are compared for emotion classification and utility of single channel is explored for emotion classification. The SB7 of channel FP2–FP4 with weighted-KKN provides highest emotion classification accuracy 86.1% as compared to other channels SBs. The obtained emotion classification performance of proposed method is better as compared to other existing methods. The proposed single EEG channel based emotion classification method can be used in BCI system for the impaired people to assessed their emotions.