Operator functional state estimation based on EEG-data-driven fuzzy model

Abstract

This paper proposed a max–min-entropy-based fuzzy partition method for fuzzy model based estimation of human operator functional state (OFS). The optimal number of fuzzy partitions for each I/O variable of fuzzy model is determined by using the entropy criterion. The fuzzy models were constructed by using Wang–Mendel method. The OFS estimation results showed the practical usefulness of the proposed fuzzy modeling approach.

Introduction

Fuzzy system, as a universal approximator, has been widely applied to real-world modeling, control and decision-making problems since fuzzy sets theory was first proposed by Zadeh (1965). However, the design of an efficient fuzzy system for a practical problem is not easy and many aspects of system design, such as fuzzy partition of the system I/O variables and the proper selection of membership functions (MFs), still depend largely on experiences, expertise and know-how of the system designer. While many fuzzy system design methods focus on the elicitation and tuning/optimization of fuzzy rule-base, the fundamental issue of ‘what is the optimal number of fuzzy partition for each I/O variable’ has not been well addressed yet. Wang and Mendel proposed the Wang–Mendel (WM) method to design fuzzy systems with the number of fuzzy partition pre-specified empirically by the designer (Wang and Mendel 1992). Wang further improved the WM method, in which fuzzy partition can be pruned based on performance but the number of fuzzy partition still needs to be predetermined (Wang 2003). Marsala (2001) investigated fuzzy partition methods using a merging paradigm. Sanyal (2011) proposed a novel fuzzy interval selection method by integration of bacterial foraging algorithm.

The number of fuzzy partition was traditionally determined by experienced designer of fuzzy system. Thus, an objective criterion is required to determine how many fuzzy partitions are needed for a variable. For conventional grid partitioning method, too many partitions of each input variable would result in not only an exponentially increased number of fuzzy rules, but overfitting phenomenon. On the contrary, a sparse partition may not well capture/describe the inherent data structure. Hence, it is necessary to find a way to determine a reasonable number of partitions (Liu 2007; Wei and Liang 2013).

The concept of fuzzy entropy was originally introduced by Zadeh (1968). Since then the entropy paradigm has been widely used in fuzzy systems. Entropy provides a measure to characterize the fuzziness of a fuzzy system (Karayiannis 1994). So far various definitions of fuzzy entropy have been proposed (Pal 1991; Liu 2006). However, the entropy ideas have mostly been applied to the area of fuzzy image segmentation (Yan 2010). Among these ideas, maximum and minimum entropy criteria are most frequently used. Under the maximum entropy criterion, a fuzzy system has the highest fuzziness (or fuzzy uncertainty), which is characterized by as few as possible partition. On the contrary, under the minimum entropy criterion, a fuzzy system has the least fuzziness (and thus highest crispness) in terms of as many as possible partitions.

The operator functional state (OFS) analysis is an effective paradigm to realize adaptive automation (AA) of human–machine integrated systems in safety–critical scenarios. In recent years, researchers have worked on regulating OFS by using adaptive aiding/automation (AA) strategy so as to enhance the overall human–machine system performance. AA usually employs an intelligent controller to allocate dynamically the type and/or level of tasks between the operator and machine in order to achieve a best fit (or match) between operator’s momentary cognitive capability and status and the current task demand. It was shown in several work, e.g., (Parasuraman et al. 1996; Prinzel et al. 2000), that adaptively automated systems can outperform fully automated systems. Nevertheless, it should be noted that the AA controller should only trigger adaptive task (or functional) allocation when absolutely necessary, as the frequent manual intervention may disturb or undermine the normal operation of human–machine system (Haarmann et al. 2009). For this reason, accurate and reliable estimation or prediction of the OFS is a crucial prerequisite for designing any adaptively automated (or human-centric) systems. From recent literature (e.g., Sharma and Gedeon 2012; Chen and Epps 2013), it can be observed clearly that the neuroimaging techniques and psychophysiological measures indicative of inherent brain function and state are very promising to provide continuous and unobtrusive OFS monitoring. In AA, the cognitive task is adaptively allocated between human operators and machine, based on his/her estimated OFS, in order to prevent operator performance breakdown. Thus accurate OFS estimation is crucial basis of any AA implementation (Wilson and Russell 2003, 2007). It was shown that the mental workload aspect of OFS can be reliably assessed by using physiological features (Zhang et al. 2015). Fuzzy systems were also shown to be a useful tool to construct OFS estimation model (Yang and Zhang 2013). In this work, a new entropy-based fuzzy partition method using the so-called MMEP criterion is proposed for OFS estimation. The remainder of this paper is organized as follows. “Max–min-entropy-based fuzzy modeling approach” section introduces the MMEP-based fuzzy partition method. In “Fuzzy model based OFS estimation” section, the method is applied to fuzzy modelling of operator functional state data. The results are presented and discussed. “Conclusions and future work” section concludes the paper and points a few research problems to be addressed in future work.

Max–min-entropy-based fuzzy modeling approach

Definition of fuzzy entropy

The entropy of a fuzzy set can be considered as a measure of its fuzziness. The entropy concept is defined as follows:

Definition 1

Areal-valued mapping E: E(a) → [0, 1] is called a fuzzy entropy of a if it satisfies the following conditions:

• C1: E(a) = 0, if a = 1 or a = 0;

• C2: E(a) = 1, if a = 0.5;

• C3: If d(a, 0.5) ≥ d(b, 0.5), E(a) ≤ E(b), where d(X, Y) is the Euclidean distance between two fuzzy sets X and Y; and

• C4: E(a) = E(~a), where ~a is the complement of a.

The fuzzy entropy of a dataset X, expressed as follows, is adopted in this paper.

$$E(X)=4·\sum _{i=1}^N{\mathit{\mu }}_{A_i}(X(i))(1-{\mathit{\mu }}_{A_i}(X(i)))$$ 1

It is easy to check that it satisfies conditions in Definition 1.

Relationship between entropy and number of fuzzy partitions

From a data-driven modeling point of view, the crux of a fuzzy system is its fuzzy rule-base which can be learned from the available training dataset. For that purpose, fuzzy partition of each variable must be determined first. Usually, fuzzy partition of a variable is represented by several MFs whose center and width parameters are obtained by clustering methods. As a result, different number of fuzzy partition can generate quite different fuzzy systems, as schematically illustrated in Fig. 1. Thus the number of fuzzy partition of each variable is an essential parameter for fuzzy system design.

Fig. 1.

Different number of fuzzy partition: 3 (left), and 6 (right)

The effect of partition number can be reflected by the variation in fuzzy entropy. In Fig. 2, the value x = 0.8 has different entropy with different number of partitions. This suggests that the entropy measure can be used to identify the optimal number of fuzzy partition for a fuzzy system.

Fig. 2.

Same data (i.e., x = 0.8) has different entropy under two different partitions of the variable x

As shown in Fig. 3, the piecewise (i.e., left–right parameterized) Gaussian function is used as the MF. In a piecewise Gaussian function, the widths of the left and right Gaussian functions are different (see the MF in red in Fig. 3). However, the widths of two adjacent Gaussian functions are equal. For instance, the two adjacent MFs A1 and A2 have the same width d1.

Fig. 3.

The L–R parameterized MF, e.g., MF A2 comprising left and right Gaussian functions with the same center 0.4 but different widths d1 and d2

The piecewise MF is expressed by:

$$\mathit{\mu }(x)=e^{-\frac{{(x-c)}^2}{{\mathit{\sigma }}^2}}$$ 2

$$\left\{\begin{array}{c}\hfill {\mathit{\sigma }}_1=\frac{c2-c1}{2\sqrt{-lnb}}\\ \hfill {\mathit{\sigma }}_2=\frac{c3-c2}{2\sqrt{-lnb}}\\ \hfill \end{array}\right.$$ 3

where c and σ are the center and spread of the Gaussian function, σ₁ and σ₂ are the left and right width parameters, c1, c2 and c3 are centers of MF A1, A2 and A3, and b is the membership value of the cross point and set to be 0.5 in this paper.

Obviously the crossover point has the same degree (i.e., 0.5) of membership in the neighboring two fuzzy subsets. To make Eq. (3) valid for each cluster, two default MFs, cdl and cdr, are placed at two ends of the domain, as shown in Fig. 4.

Fig. 4.

Fuzzy partition with two default MFs at both ends of the universe of discourse of the variable

Entropy-based fuzzy partition

From Eq. (1), the entropy would get its maximum when all μ_A(X) = 0.5. This means that the variable would undergo the fuzziest partition. This is called the maximum entropy (MaxEP) criterion/principle. In other words, the MaxEP makes the number of partition as few as possible and thus makes the partition fuzziest. However, too few partitions may not well describe the inherent structure in the dataset.

On the other hand, when a fuzzy model is learned from a set of sample data, it is often expected that each data point has a bigger degree of membership (i.e., close to 1) in at least one fuzzy subset under any fuzzy partition scheme. This usually requires a large number of fuzzy partitions. This is called the minimum entropy (MinEP) principle. While the MinEP makes the number of partitions as large as possible, the overfitting issue, which severely undermines the generalization performance of fuzzy models, may unavoidably arise.

To overcome the disadvantages of the above two criteria, we proposed a Max–Min Entropy (MMEP) based fuzzy partition method. Under the MMEP criterion, for each I/O variable, the number of fuzzy partition is increased from 2 to M. For each number of partitions, the total entropy of training dataset is calculated. Suppose the number of partitions to be P₁ when the entropy gets its maximum value and P₂ when the entropy gets its minimum value, then under MMEP criterion the final number of partitions P is expressed by:

$$\left\{\begin{array}{cc}& P=⌈\frac{P_1+P_2}{2}⌉\hfill \\ \hfill & P_1=arg\left[\underset{i\in \left\{2,3,\dots ,M\right\}}{\text{max}}{E}_i\right]\hfill \\ \hfill & P_2=arg\left[\underset{i\in \left\{2,3,\dots ,M\right\}}{\text{min}}{E}_i\right]\hfill \\ \hfill \end{array}\right.$$ 4

where E_i denotes the entropy of training data and i is the number of partitions.

Wang–Mendel (WM) fuzzy modeling method

The universal function approximation property of fuzzy systems was rigorously shown by Wang and Mendel (1992). Based on this solid mathematical foundation, fuzzy inference system (FIS) has become a mainstream paradigm for modeling and identification of nonlinear systems because of its tolerance of uncertainty and fuzziness, its power in modeling human perception, reasoning and decision making, transparency and interpretability, and modeling accuracy. The Takagi–Sugeno–Kang (TSK) and Mamdani types are two major classes of FIS. In this work, the Mamdani-type fuzzy model is employed. A typical fuzzy rule used by a MISO Mamdani-type FIS is in the form of:

$$R^i:If{x}_1\text{is}{A}_{i1},x_2\text{is}{A}_{i2},\dots ,\text{and}{x}_m\text{is}{A}_{im},\text{Then}y\text{is}{B}_i,i=1,2,\dots ,M$$

where x = [x₁, x₂, …, x_m]^T∈ R^m is the input vector and y is the output, A_i1, A_i2, …, A_im and B_i are fuzzy linguistic labels, and M denotes the number of fuzzy rules. The output of the FIS is explicitly given by:

$$y=\frac{\sum _{i=1}^M\left[\left(\prod _{j=1}^m({\mathit{\mu }}_{A_{ij}}(x_j))\right)·C(B_i)\right]}{\sum _{i=1}^M\left(\prod _{j=1}^m({\mathit{\mu }}_{A_{ij}}(x_j))\right)}$$ 5

where C(B) represents the central value of Gaussian MF B.

Wang–Mendel (WM) method is a straightforward way to design FIS by generating fuzzy rule-base from sample data (Wang and Mendel 1992; Wang 2003). In the WM method, a general procedure for eliciting fuzzy rule-base from training dataset consists of the following 4 steps:

• Step 1. Each input or output domain is partitioned into certain fuzzy regions. Each region is assigned with a fuzzy subset with linguistic label, which is characterized by its MF;

• Step 2. Fuzzy rules are generated from given I/O sample data pairs. According to the fuzzy regions and MFs, each I/O dimension of a data pair is represented by one linguistic label. The linguistic label is the one with the maximum membership value among all possible labels. The antecedent of a rule is formed by connecting the linguistic labels of all input variables with logical connector ‘AND’, while the consequent of the rule is the linguistic label of the output. Each sample data pair generates one rule;

• Step 3. For same or conflicting rules, only one is extracted. Meanwhile, expert rules are added to the rule base; and

• Step 4. A weight is assigned to each rule in the final rule base.

In this work, the optimal number of fuzzy partitions for each I/O variable is determined based on entropy-based criterion. For simplicity, the weight of each rule is assigned to 1, no expert rule is used, and Gaussian MFs are adopted.

Fuzzy model based OFS estimation

Data acquisition experiments

In this section, OFS data are used to further test the MMEP-based fuzzy modeling approach. OFS can be defined as the temporally variable ability of a human operator for completing required (usually cognition-demanding) tasks.

The OFS data (i.e., physiological and performance data) were collected experimentally while the subject was involved in simulated safety–critical process control tasks under AutoCAMS software environment developed by a group at Technical University of Berlin, Germany. All six subjects selected were male healthy graduate students (21–24 years old) at East China University of Science and Technology, Shanghai.

The aim of the experiments is to assess quantitatively the OFS on an AutoCAMS human–machine task simulation platform for monitoring and controlling the air quality of a space capsule. A simplified version of AutoCAMS was used. The operational task is to keep four variables (and thus four technical subsystems), i.e., O₂ concentration, CO₂ concentration, air pressure and temperature, within their respective target ranges (or set-points). Initially, all subsystems were controlled in automatic operating mode based on the AutoCAMS software itself. When the programmed failure of particular subsystem occurs, the operator has to take over the control authority from the computer (i.e., AutoCAMS software) by assuming manual control until the system failure is fixed and that subsystem returns to normal automatic mode of operation. The operator task of manually controlling several subsystems simultaneously is thus rather demanding for operator’s cognitive capacity and inevitably leads to OFS variations over time with the change of level of task difficulty or complexity. The AutoCAMS software displayed all subsystems’ time evolution in real time on a 15” PC monitor placed 50 cm ahead of the subject. Whenever a subsystem drifts out of its target range due to the failure/faults in it, the operator was required to intervene manually to monitor and control the AutoCAMS tasks by inspecting the control panel shown on the monitor. The OFS is continuously quantified by the primary AutoCAMS task performance of the operator.

Each subject participated in 3 sessions of experiment and each session consisted of 6 task-load conditions. In different conditions, the subject was asked to carry out different numbers of manual control tasks. The number of tasks to be manually controlled by the subject in 6 conditions is 1, 3, 4, 4, 3 and 1, respectively. Each session lasted 90 min (6 conditions * 15 min per condition). In each experimental session, EEG and ocular electric signal of the subject were recorded and sampled at a rate of 500 Hz using Nihon Kohden^® EEG equipment. A band-pass (0.5–40 Hz) filter was applied to separate higher-frequency myoelectric signal on from the measured psychophysiological signals. Only EEG features were used to characterize and quantify the OFS, while the ocular signal was used as a reference (or template) to remove the artifacts. The EEG signals were measured by using 11 electrodes, namely F3, F4, Fz, C3, C4, Cz, P3, P4, Pz, O1, and O2, with reference to earlobes (A1 and A2) according to the 10–20 international EEG electrode placement system. The ocular artifacts in EEG were eliminated by using independent component analysis (ICA). Specifically, the filtered EEG signals were first decomposed into an equal number of independent components (ICs) by using FastICA algorithm. Then, the IC accounting for the ocular artifact was determined by comparison with the ocular signal, measured with two electrodes placed above and below the left eye of the subject. After the artifact ICs were all set to zero, the artifact-free EEG can be reconstructed by using inverse ICA transformation. Subsequently, the power spectrum of each 20-s. EEG epoch was obtained by using fast Fourier transform (FFT). Here, two thirds of the data were randomly selected for training the FIS and the rest for testing. Simultaneously the operator performance data were recorded. Here, the operator performance was quantified by the time in range (TIR). TIR is the percentage of time when the required number of subsystems is regulated in the target range, thus it lies in the interval [0, 100 %]. Figure 5 shows the I/O data measured from subject M. The input data were normalized into the unit interval [0, 1].

Fig. 5.

The model I/O data (subject M): Training data (left); testing data (right); the first three rows indicate the model inputs while the last row the model output

OFS estimation problem statement

The task is to construct a fuzzy model which correlates the OFS to time-variant physiological features (or parameters). According to Wilson and Russell (2003) and Wilson and Russell (2007), three EEG variables, Fzθ, Pzα and TLI1, were used as the model inputs and performance parameter TIR as the output. Fzθ, Pzα, and TLI1 are the theta band power at from EEG measurement channel (electrode) Fz, alpha band power from channel Fz, and a Task Load Index related to EEG.

The basic assumptions relating to this problem are:

1. The OFS can be quantified by the task performance variable TIR, which reflects actually the overall performance of the human–machine system as a whole. If we assume that the machine performance is stationary, then TIR can be used solely to characterize quantitatively the OFS temporal variations.

2. EEG features are correlated with human performance under human–machine cooperative task environment.

Existing work (Chuang et al. 2012) showed that the OFS can be estimated based on temporal variations in the EEG power features within particular frequency bands. The corresponding neuroergonomics mechanism was also established in Parasuraman and Caggiano (2005) that OFS is closely correlated with lower frequency cerebral activity at certain cortical regions responsible for working memory, such as prefrontal cortex (PFC), anterior cingulate cortex, and posterior parietal cortex. In particular, the frontal cortex and parietal cortex are of importance for OFS characterization and recognition. Moreover, in Gundel and Wilson (1992), the authors showed that with the increasing task demand to elicit vulnerable or high-risk OFS, the parietal and occipital alpha (8–13 Hz) power of EEG decreases, whereas the frontal theta (4–7 Hz) power increases.

Therefore, the EEG Task Load Index (TLI) is defined as:

$$TL{I}_1=\frac{P_{Fz\mathit{\theta }}}{P_{Pz\mathit{\alpha }}}$$ 6

where $P_{Fz\mathit{\theta }}$ and $P_{Pz\mathit{\alpha }}$ represent the averaged EEG power in Fz theta band and Fz alpha band, respectively.

We also computed the correlation coefficient between $P_{\mathit{\theta }}$ and $P_{\mathit{\alpha }}$ with TIR for other EEG channels. The results showed that among all 11 EEG electrodes, there is the highest (with a correlation coefficient of 0.8) negative correlation between $P_{\mathit{\theta }}$ for Fz channel and TIR and the highest (with a correlation coefficient of 1.0) positive correlation occurs between $P_{\mathit{\alpha }}$ for Pz channel and TIR. Therefore, TLI₁ is defined as a function of theta and alpha powers from channel Fz and Pz.

OFS estimation results

Prior to building fuzzy models for OFS estimation, the optimal number of fuzzy partition of each input ($P_{Fz\mathit{\theta }}$, $P_{Pz\mathit{\alpha }}$ and TLI₁) and output (TIR) domain must be determined. Figure 6 illustrates how to use the MMEP as an objective function to determine fuzzy partition numbers for the model of subject M. Taking the time evolution of TIR as an example, the maximum value of entropy corresponds to 6 fuzzy partitions, while the minimum entropy results in 20 partitions. In this case, the optimal fuzzy partition number is thus the rounded average of 6 and 20 following Eq. (4). Hence, the best fuzzy partition number for the model output TIR is determined to be 13. With the same computing procedure, the optimal number of fuzzy partitions of inputs $P_{Fz\mathit{\theta }}$, $P_{Pz\mathit{\alpha }}$ and TLI₁ is computed to be 12, 10, and 12, respectively. It can be also seen that the entropy of output TIR is much smaller than that of input data. This is reasonable as the variability of TIR data is smaller than that of input data. It is also noticeable that the entropies of the three inputs are quite close. This implies that the three input variables have similar data distribution, which is consistent with in Fig. 5. It is noted that smaller number of fuzzy partitions leads to a less accurate fuzzy model, while too large value gives rise to model overfitting problem as the resulting fuzzy model is too complex in terms of both its structure and parameters. Thus, the proposed MMEP method is employed to achieve a good balance between modeling accuracy and the model complexity.

Fig. 6.

The entropy vs. the number of fuzzy partitions for the OFS data (subject M)

The number of fuzzy partition for the domain of each variable is determined under three different entropy criteria (i.e., MaxEP, MinEP and MMEP). In Fig. 7, the target (desired) TIR (blue) and estimated TIR (red) are compared for subject M. It is shown that the Mean Squared Error (MSE) in the case of MinEP and MMEP is much lower than that of MaxEP case and the results of MinEP and MMEP are comparable. This indicated that the constructed fuzzy model using optimal fuzzy partitions with the MMEP criterion achieved acceptable OFS estimation precision even with a simpler model structure.

Fig. 7.

The fuzzy model testing result (subject M) based on different entropy criteria: a MaxEP; b MinEP; and c MMEP

The results of fuzzy partition and modeling accuracy are presented in Tables 1 and 2. It can be seen that the best MSEs were obtained based on MinEP and MMEP criteria. Among the 12 best training and testing MSEs (for six subjects), MinEP achieved 6 (5 for training and 1 for testing) and MMEP 4 (all for testing). The results showed the effectiveness of the proposed approach to model-based OFS estimation.

Table 1.

Model training and testing results for each of six subjects based on three entropy criteria (Average of 50 random runs)

	MaxEP		MinEP		MMEP
	Train.	Test	Train.	Test	Train.	Test
M	140.7	159.3	74.2	129.6	91.2	130.0
N	104.5	113.6	67.3	109.1	70.2	99.1
R	105.7	146.7	65.1	147.9	76.1	140.7
S	118.3	157.7	81.3	168.8	87.7	153.0
U	179.0	204.6	82.3	175.3	109.3	149.7
V	153.4	222.6	79.5	200.2	90.5	190.1

Table 2.

Optimal number of fuzzy partitions for each of six subjects based on three entropy criteria (Average of 50 random runs)

		M	N	R	S	U	V
MaxEP	Fz-θ	6	3	9	8	3	10
	Pz-α	5	11	9	10	10	11
	TLI1	4	7	8	6	4	6
	TIR	7	6	7	8	7	7
MinEP	Fz-θ	18	15	12	11	15	15
	Pz-α	17	4	14	12	11	10
	TLI1	18	16	13	16	17	14
	TIR	19	19	19	20	20	20
MMEP	Fz-θ	12	9	11	10	9	14
	Pz-α	11	7	10	10	10	11
	TLI1	10	11	11	12	11	10
	TIR	12	12	13	14	13	13

Discussion

Although the training error of MMEP-based fuzzy model is larger than that of MinEP-based model for each subject, all its testing error is smaller except for subject M. From Table 2, it can be seen that fuzzy partitions obtained under MMEP criterion are fewer than MinEP criterion, but more than MaxEP criterion. The number of fuzzy partitions under MaxEP criterion is smaller than other two criteria with the expense of reduced training and testing accuracy. From the comparative results, we may conclude that the use of MMEP criterion not only reduces the number of fuzzy partitions to a reasonable one, but also results in comparable or even higher modeling accuracy.

The MMEP method is aimed at achieving a good compromise between the model complexity and its data-fitting accuracy. Thus it may be particularly suitable for classification or regression of complex (e.g., nonlinear and nonstationary) dataset, including the EEG dataset measured in OFS experiments under consideration. Another advantage of MMEP is that subject-specific OFS models can be constructed based on different fuzzy partitions, resulting in improved OFS estimation accuracy.

Conclusion and future work

In this paper, a MMEP-based fuzzy modeling was proposed to balance the number of fuzzy partitions and modeling performance. It was applied to OFS estimation problem. Compared with the minimum- (MinEP) and maximum-entropy (MaxEP) criteria, the MMEP criterion leads to comparable or even higher modeling accuracy with a reduced number of fuzzy partitions. Therefore, the proposed fuzzy modeling approach is promising for real-world data modeling and analysis problems.

Along this line of research, the following research issues should be further addressed in future work: (1) To validate more comprehensively and statistically the generalizability and replicability of the presented method and results, more subjects, e.g., female ones and a subgroup of participants in a different age range, must be used in the experiments; (2) It can be expected that the proposed MMEP-based fuzzy modeling approach has potential to be reliably applied to EEG-based cognitive state recognition problem as well as other complex data mining and analysis problems. However, the practical effectiveness of the proposed fuzzy modeling approach needs to be further validated on other similar data-driven problems; (3) The entropy definition other than that used in this paper should be considered; and (4) The MMEP is essentially a simple average of the maximum and minimum entropy, thus other forms of entropy combination (e.g., weighted averaging or more sophisticated nonlinear transformation) can be examined and compared with arithmetic averaging approach adopted here.