GRU-powered sleep stage classification with permutation-based EEG channel selection

Abstract

We present a new approach to classifying the sleep stage that incorporates a computationally inexpensive method based on permutations for channel selection and takes advantage of deep learning power, specifically the gated recurrent unit (GRU) model, along with other deep learning methods. By systematically permuting the electroencephalographic (EEG) channels, different combinations of EEG channels are evaluated to identify the most informative subset for the classification of the 5-class sleep stage. For analysis, we used an EEG dataset that was collected at the International Institute for Integrative Sleep Medicine (WPI-IIIS) at the University of Tsukuba in Japan. The results of these explorations provide many new insights such as the (1) drastic decrease in performance when channels are fewer than 3, (2) 3-random channels selected by permutation provide the same or better prediction than the 3 channels recommended by the American Academy of Sleep Medicine (AASM), (3) N1 class suffers the most in prediction accuracy as the channels drop from 128 to 3 random or 3 AASM, and (4) no single channel provides acceptable levels of accuracy in the prediction of 5 classes. The results obtained show the GRU’s ability to retain essential temporal information from EEG data, which allows capturing the underlying patterns associated with each sleep stage effectively. Using permutation-based channel selection, we enhance or at least maintain as high model efficiency as when using high-density EEG, incorporating only the most informative EEG channels.

Introduction

Sleep is a natural and essential state to regulate our mood and emotions, maintaining good health and well-being. We also sleep to help our brain process, consolidate, and organize new information and memories^1,2. This is important because, considering 6–8 h of sleep per day, humans spend approximately 20–25 years sleeping throughout their entire lives^3,4. During sleep, the human brain undergoes different patterns of activity, transitioning through wakefulness (W), the rapid eye movement (REM) sleep stage, and three non-rapid eye movement (NREM) sleep stages: N1, N2, and N3^5,6.

Sleep staging is a fundamental process used in sleep medicine to assess and understand the different stages of sleep a subject experiences during the night. For this, sleep experts manually determine sleep stages based on polysomnography (PSG), including the use of electroencephalograms (EEG), electrooculograms (EOG) and electromyograms (EMG)⁷. All of which mainly uses well-known information from each sleep stage, such as: W is characterized by alpha rhythms, muscle activity is present, and eye movements typically include eye blinks, rapid eye movement, and reading eye activity⁸. N1 is a stage of light sleep that lasts for a brief period, and brain activity begins to shift from alpha waves (8–13 Hz) to low-amplitude mixed frequency (LAMF, predominantly 4–7 Hz). EOG frequently shows slow eye movement during N1⁸. N2 is characterized by sleep spindles (oscillations (11–16 Hz) appear in the thalamus and cortex) and K-complexes (brief, high-amplitude waves)⁹. Eye movements are minimal and heart rate and body temperature decrease. N3 is dominated by delta waves^10,11. REM is characterized by rapid eye movements with the lowest level of chin muscle tone; Brain activity shows fast and desynchronized brain waves, similar to LAMF in N1⁸.

Sleep stage studies can help unravel the mysteries surrounding sleep, dreams, dream emotions, memory consolidation, and other essential functions of the brain and body during rest^4,12–14. Sleep disorders, such as insomnia, sleep apnea, narcolepsy, and restless leg syndrome, among others, can be diagnosed and treated with the help of sleep staging^14–16. Interrupted sleep periods can cause sleep disorders that affect approximately 60% of adults and sleep apnea in 2–4% of adults and 1–3% of children^5,16. Therefore, it is highly desirable to measure sleep quality through sleep monitoring and sleep stage classification.

Manual sleep staging, which is still prevalent in sleep clinics and sleep research, is an expensive and time-consuming process. To address these challenges, several research papers have suggested the use of artificial intelligence techniques such as machine learning (ML) or deep learning (DL) models^17–26.

ML methods require certain specialized expertise from the data engineer to perform feature extraction (such as fractal dimensions, energy distribution, power spectral density, or wavelet coefficients), with manually crafted features to develop a classification model^27–29. ML methods have been shown to achieve high performance, even when using only a single bipolar EEG channel¹⁹.

DL has demonstrated good capabilities in domains such as natural language processing (NLP), computer vision, and time-series analysis³⁰. Recurrent Neural Networks (RNNs) are particularly well suited for sequential data, making them ideal for EEG signal classification. Among RNNs, the Gated Recurrent Unit (GRU) has shown superior performance in modeling long-range dependencies and mitigating the problem of vanishing gradients^30,31.

CNN has been used successfully to extract sleep-related waveform characteristics, but, as with most DL methods, it requires a task-specific architecture, large models, and is data-intensive^{21,22,25–27,32–36}. A CNN approach called EEGNet has been applied to EEG-related applications, including sleep staging, and has shown high performance^37,38. It has been reported in combination with bidirectional long- and short-term memory (BiLSTM) and the mixing of randomly selected delta, theta, and beta frequencies for data augmentation to overcome the problem of class imbalance. In this way,³⁹ reported accuracies of up to 0.87, using 5-class classification models.

The authors in⁴⁰ combined two bipolar EEG channels (Fpz-Cz and Pz-Oz) and one horizontal EOG channel from Sleep-EDF Expanded dataset (Sleep-EDFX)⁴¹ with EEGNet-BiLSTM. The authors reported an accuracy of 0.90 and a confusion matrix in which the most misclassified classes are N1, N2 and N3.Another recent approach based on 5-class models, called SalientSleepNet, was proposed by Ziyu Jia et al.⁴² and tested on the SleepEDF-39 and Sleep-EDF-153 datasets using Fpz-Cz EEG and horizontal EOG channels^41,43, reporting an accuracy of 0.85 on average, but only 0.55 on average to detect the N1 sleep stage.

Other work used Sleep-EDF-20 (Fpz-Cz channel), Sleep-EDF-78 (Fpz-Cz channel)^41,43 and Sleep Heart Health Study (SHHS, C4-A1 channel)⁴⁴ datasets. In the best case, they obtained an accuracy of 0.856, 0.829 and 0.866, respectively for each dataset using an attention-based deep learning architecture (AttnSleep)⁴⁵. Another approach used a CNN model in the SHHS dataset, obtained an accuracy of 0.87, 0.87 Fscore, and 0.81 for Kappa³². A recent work proposed the use of cascaded CNN + LSTM and the use of only one bipolar channel from the Sleep-EDF dataset, showing that it is possible to obtain an accuracy of 0.827 using only Fpz-Cz, and 0.797 using Pz-Oz⁴⁶.

The mentioned methods are mainly tested in different versions of the public Sleep-EDF dataset or SHHS dataset, in all the cases using bipolar EEG^41,43,44. There are many factors that confound the comparisons in the state of the art, since different approaches are tested under different conditions, such as using a subset of subjects, splitting into different percentages, and using a different number of folds in the cross-validation. In general, the state-of-the-art covers a range of methods for classification of sleep stages from different perspectives and using features hand-crafted with ML or DL models. Furthermore, many works have presented experimental results creating different combinations, such as considering sleep versus W, N1 and W as a single class, N1N2 versus N3, W versus NREM versus REM, etcetera^{21,33,38,45–54}.

Here, we present a method for the classification of 5-class sleep stages using different DL architectures, comparing performance in training and validation sets, and for the test set, we measure performance using accuracy, Fscore, precision, recall, and area under the receiver operating characteristic (AUROC). For the analysis, we used a dataset collected at the International Institute for Integrative Sleep Medicine (WPI-IIIS) at the University of Tsukuba in Japan.

One of the challenges in any EEG-related task, including sleep staging, is the selection of channels, as some of them may be more informative than others for certain stages of sleep^{25,26,36,38,55}. Although properly selecting EEG channels that contribute more to classification tasks will drastically reduce the computational cost to train and predict new data, the computational cost of the channel selection process itself can be high^28,29.

For example, methods are proposed for EEG-related tasks that use Non-dominated Sorting Genetic Algorithm (NSGA), which have been shown to be effective in selecting the most informative channels for classification tasks; however, the number of channels in the datasets is less than 64^28,29. For this, the process creates up to 20 models (20 chromosomes) per population and up to 200 generations to minimize a multiobjective optimization (MOO) function that consists of reducing the number of channels while increasing or at least maintaining the performance.

We present a new approach for sleep stage classification that incorporates a computationally inexpensive method based on permutations for channel selection and leverages the power of DL, specifically the GRU model, along with other deep learning methods. By systematically permuting the channels, different combinations of channels are evaluated to identify the most informative subset for the classification of sleep stages. Together, our results can provide indices that could improve understanding of sleep disorders and pave the way for more personalized and effective treatments in sleep medicine.

Results

Comparison of different algorithms for the classification of 5-class sleep stages using 128 EEG channels

Here, we compare the performance of various DL algorithms to create a 5-class classifier (W, N1, N2, N3, and REM). With this comparison, our aim is to analyze the performance of the different DL algorithms and thus select the one with the highest performance for further analysis. We used 128 channels and created a model per subject, which was validated using 5-fold cross-validation. The average results for all subjects are presented in Table 1.

Table 1.

Average 5-class sleep stage classification performance from all the subjects in the dataset with various DL algorithms.

Model	M-acc	V-Acc	Acc	FScore	Precision	Recall	AUROC	Kappa
GRU	0.879 ± 0.03	0.867 ± 0.03	0.885 ± 0.03	0.860 ± 0.02	0.889 ± 0.02	0.843 ± 0.03	0.983 ± 0.01	0.838 ± 0.04
EEGNeX	0.885 ± 0.03	0.878 ± 0.03	0.882 ± 0.03	0.859 ± 0.02	0.874 ± 0.03	0.849 ± 0.02	0.981 ± 0.01	0.837 ± 0.04
1D-CNN	0.817 ± 0.05	0.791 ± 0.05	0.809 ± 0.05	0.763 ± 0.05	0.808 ± 0.04	0.746 ± 0.06	0.948 ± 0.02	0.731 ± 0.07
EEGNet	0.785 ± 0.05	0.759 ± 0.07	0.811 ± 0.05	0.747 ± 0.06	0.801 ± 0.05	0.734 ± 0.06	0.947 ± 0.03	0.730 ± 0.08
Single 2D-CNN-LSTM	0.766 ± 0.09	0.764 ± 0.09	0.797 ± 0.06	0.711 ± 0.10	0.775 ± 0.07	0.702 ± 0.10	0.930 ± 0.04	0.707 ± 0.10
Bi-LSTM	0.615 ± 0.12	0.624 ± 0.12	0.677 ± 0.11	0.504 ± 0.14	0.520 ± 0.16	0.522 ± 0.13	0.820 ± 0.08	0.512 ± 0.18
Single LSTM	0.749 ± 0.19	0.747 ± 0.19	0.825 ± 0.09	0.777 ± 0.11	0.821 ± 0.10	0.765 ± 0.09	0.954 ± 0.04	0.753 ± 0.12

As stated above, we have tested all algorithms presented in a previous benchmark; however, we report only the models with the highest performance, without including those that gave low or random performance for the sleep staging task⁵⁶.

Table 1 shows that the performance is similar when using GRU or EEGNeX, in both cases with a low standard deviation. It is clear that when using any other model, performance decreases $>=8\%$ in most metrics. An important point to highlight is that the standard deviation is lower than 0.05 in both GRU and EEGNeX, but not for the other algorithms.

The 2-way ANOVA test found no statistical significance in performance between using GRU and EEGNeX (p-value=0.53, see Table S1 from the supplementary information), but for both it is significant when compared with 1D-CNN (p-value= 0.000), which, as shown in Table 1, are the top three methods for obtaining the highest performance.

5-class sleep stage classification with GRU and EEGNeX using 3 recommended channels of AASM or 3 random channels

The American Academy of Sleep Medicine (AASM) has recommended the use of 3 channels and many studies have used them to compare their reported results. In this experiment, we performed the sleep stage classification process using the 3 channels recommended by AASM, and the results were obtained with the models created using 3 random channels. The 3 random channels are recalculated separately for each subject and for each of the 5 folds. We present this comparison to understand if the 3 channels proposed by AASM contain more useful information for the classifiers than any other channels.

For this analysis, we use the top 3 algorithms according to the results presented in Table 1, and the average results for the 5-class sleep staging of all subjects are presented in Fig. 1.

Figure 1.

Comparing the 5-class sleep stage classification performance with 3 AASM recommended channels and 3 random channels. The presented results are the average of all subjects and the 5-fold, which were tested with the top 3 methods according to the results from Table 1.

The results in Fig. 1 show that we can obtain the highest performance for sleep staging using the GRU model, in both cases, using 3-AASM or 3-random channels (p-value<0.002 for comparisons between GRU and EEGNeX/1D-CNN across methods). Additionally, the performance appears to be similar with both sets of channels, indicating that the set of 3 channels recommended by AASM may not necessarily provide more information than other sets of channels for automatic classification of sleep stages (p-value>0.53 for GRU across channel subsets (3-AASM and 3-random channels). See Table S2 from supplementary information for 2-way ANOVA results). However, further analysis in different datasets and with a higher number of subjects is needed to obtain a more solid conclusion.

The 2-way ANOVA test shows no statistical significance for the results between GRU and EEGNeX using the 128 EEG channels (see Table 1), however, when using 3-AASM or 3-random channels, GRU shows higher performance with statistical significance, highlighting the correct decision to choose GRU as the best method for further analysis in the experiments.

Figure 2 presents the confusion matrix of the average results for all subjects. To compare which stages are most affected by channel reduction, the figure includes the confusion matrix using GRU with the 128 channels (results from Fig. 1), 3 recommended channels of AASM, and 3 random channels.

Figure 2.

Confusion matrix of the average results (in percentage) from all the subjects and models created with GRU models, presented in Table 1 and Fig. 1. (A) 128-channel model, (B) 3 recommended channels of AASM, and (C) 3-random channel model.

The confusion matrices clearly show that the main affected sleep stages are W and N1, but the percentages of misclassification also increase for the other stages. When using 3 channels, 38–44% of the instances of N1 are misclassified with N2 and 13–15% of W are misclassified with N2. Based on the results of Table 1 and Fig. 1, we selected GRU for further analysis.

Single-channel for 5-class sleep staging using GRU

As presented in the Introduction, there are many current studies that have used single bipolar channels for sleep staging. To evaluate the effectiveness of using bipolar electrodes, as in the state-of-the-art, we created models considering the Pz-Oz or Fpz-Cz channels to create single-channel models.

According to the biosemi cap with 128 channels, Pz corresponds to A19, Oz to A23, Fpz to C17 and Cz to A1. This makes it easier to compare the models with the methods proposed in the state-of-the-art using a bipolar single channel. The experiments using single channel EEG were also repeated using each channel individually, that is, creating a model with channel A19, A23, C17, or A1.

The average results for all subjects and 5-fold are presented in Table 2. The highest performance is obtained using A1 (Cz) or C17-A1 (Fpz-Cz), indicating the importance of the A1 channel for the staging of sleep, at least from the set of channels tested (A1, A19, A23, C17). However, the performance during training and validation is lower than 0.60, and for the test set it is random.

Table 2.

Average performance with single-channel models for 5-class sleep stage classification.

Channel	M-acc	V-Acc	Acc	FScore	Precision	Recall	AUROC	Kappa
Cz (A1)	0.568 ± 0.06	0.567 ± 0.06	0.586 ± 0.06	0.467 ± 0.07	0.559 ± 0.12	0.466 ± 0.06	0.802 ± 0.04	0.388 ± 0.08
Pz (A19)	0.493 ± 0.06	0.497 ± 0.06	0.503 ± 0.06	0.353 ± 0.11	0.404 ± 0.15	0.371 ± 0.08	0.72 ± 0.08	0.249 ± 0.10
Oz (A23)	0.482 ± 0.07	0.488 ± 0.07	0.485 ± 0.08	0.300 ± 0.09	0.343 ± 0.12	0.325 ± 0.06	0.712 ± 0.07	0.205 ± 0.11
Fpz (C17)	0.482 ± 0.05	0.488 ± 0.05	0.492 ± 0.06	0.307 ± 0.08	0.363 ± 0.12	0.331 ± 0.06	0.716 ± 0.05	0.216 ± 0.08
Pz-Oz (A19, A23)	0.514 ± 0.08	0.523 ± 0.08	0.530 ± 0.08	0.392 ± 0.11	0.437 ± 0.14	0.407 ± 0.09	0.751 ± 0.09	0.298 ± 0.11
Fpz-Cz (C17, A1)	0.591 ± 0.07	0.600 ± 0.08	0.604 ± 0.07	0.483 ± 0.08	0.573 ± 0.10	0.488 ± 0.08	0.817 ± 0.06	0.415 ± 0.12

The channel names are indicated in the 10-20 international system and in parentesis as the biosemi system.

Permutation-based channel selection for 5-class sleep staging using GRU, classification performance comparison selecting the k best channels versus the k worst channels.

Here, our objective is to perform 5-class sleep staging with channels selected by permuting channels and evaluating the value of the performance delta (that is, performance with all channels minus performance after changing information from a given channel). We have followed the process explained in Fig. 10.

Figure 10.

Flowchart of the permutation-based channel selection process.

For this, we created a model per subject 5 times (5-fold), and the average performance delta is considered to understand how important a channel is. To illustrate this, Fig. 3 presents an example of the performance delta in accuracy, Fscore, and kappa metrics, where each line represents the importance of the channel in one-fold. From this, we consider that a channel is more important if the performance delta value is higher, since it means that by removing that channel, the performance decreases more. When the performance delta value is close to 0, it means that, using the channel or not, the performance is not affected at all; therefore, we could use fewer channels and ensure the same or very similar performance than using a high number of EEG channels.

Figure 3.

Example of performance delta values in accuracy, Fscore and kappa for channel importance from the 5 folds with one of the subjects.

An important point presented in Fig. 3 is that in the five different folds, the important channels are similar or the same. However, the importance of the channel may be different for other subjects in the dataset; therefore, to better understand it, Fig. 4 presents the topographic map with the biosemi model for all subjects. The values used to plot the topographic maps were calculated as the average performance delta between the different metrics used, but the unreported results show the same behavior as when using the performance delta change in common metrics for sleep staging: accuracy, Fscore, or kappa.

Figure 4.

Topographic maps from all the subjects in the dataset showing the permutation-based channel importance for 5-class sleep staging. Each topographic map represents one of the 14 subjects in the dataset.

Figure 7 presents the average topographic map of all subjects that evaluates the importance of the channel with the 128-channel model, which is the same for both A) and B) because there is no change in the calculated importance at that point; the changes occur from models with 64 EEG channels.

Figure 7.

Topographic map of the average channel importance from all the subjects. Importance of the channels of the models created by recursively removing half of the most important (A) or the less important (B) channels for 5-class sleep staging.

As shown in Fig. 4, the importance of the channel of some subjects is different from that of others, but there are only a few cases where it occurs. The most important channels in subjects tend to be in the central to frontal lobe.

Based on the importance of the channel presented in Figs. 7 and 4, we created 5-class models for sleep staging using GRU. To see the evolution of the importance of the channel, we created a model considering only the first most important channel, that is, the channel with which the highest performance delta value on average of all metrics was obtained based on the 128-channel model. Then, we created a model using 2 channels, the first and second most important channels, and then created 5-class models for sleep staging using the 3–10 most important channels for each subject.

To compare the classification performance if we select the incorrect channels, we also repeated the process but considering the k less important channels from the 128-channel models, i.e. the channels that when including them or not, the performance does not change or does not decrease too much (less than 0.01 on average). All processes were carried out for one subject at a time, that is, considering the importance of the channel for each subject, and the average performance is reported in Fig. 5.

Figure 5.

Average performance obtained from all the subjects using A) 1–10 most important channels or B) 1–10 less important channels for 5-class sleep staging. p-value$=$0.000 for 2-way ANOVA using the k best or k less important channels (see Table S3 from supplementary information).

Figure 5 shows that by incorrectly selecting channels, the performance can decrease approximately 5%, especially with fewer than 8 channels. It also shows that performance in all metrics is random, creating a model with the best or worst channel. However, by adding information from one more channel (that is, using at least information from two EEG channels), performance increases significantly.

Permutation-based channel selection for sleep staging with recursive removal of half of the best/worst channels

Selecting the channels that are more important when using all the 128 available channels may not ensure that those channels are globally the most relevant for sleep staging, i.e., the channels that are going to be the most relevant if we calculate the channel importance with a model created with fewer channels. To analyze it further, we performed the same process for sleep staging, but iteratively removed half of the best or worst channels to then recalculated the channel importance. The process was repeated using models with 128, 128/2, 64/2, 32/2, 16/2, 8/2 and 4/2.

In short, we created a GRU model using 128 channels, computed the performance and the channel importance, then selected the 64 most relevant channels to create another model and computed the performance and the channel importance. From the model with 64 EEG channels, we selected the 32 most relevant channels and repeated the process until a model with two channels was created.

In this way, the important channels are confirmed with a smaller model, which provides more information about which channels are more/less relevant for the classification process since with the smaller model, the performance is different from only validating the channel importance with the 128-channel models.

The whole process was repeated using 5-fold cross-validation and for each subject individually. The average performance of all subjects is presented in Fig. 6, and the average importance of the channel of all subjects is presented in a topographic map in Fig. 7.

Figure 6.

Average performance obtained from all the subjects and models created by recursively removing half of the most important (A) or the less important (B) channels for 5-class sleep staging. p-value$=$0.000 for 2-way ANOVA using the k best or k less important channels, except when using 32 and 64 channels, where no statistical significance was found (p-value >0.064, see Table S4 from supplementary information).

When we consider the evolution of removing half of the channels, keeping only the K most important channels to create the models, and seeing the importance based on permutation, the results of all subjects may vary. As shown in Figs. 8 and 9, if we plot the information from a single subject but the average from the 5-fold, the evolution is clear, and it is clear how the important channels remain important when the models are created using each time half of the channels.

Figure 8.

Topographic map of the average channel importance from one subject at a time (subjects 1–7). Importance of the channel of models created by recursively removing half of the most important channels for 5-class sleep staging.

Figure 9.

Topographic map of the average channel importance from one subject at a time (subjects 8–14). Importance of the channel of models created by recursively removing half of the most important channels for 5-class sleep staging.

In B) from Figure 6, we iteratively removed the most important channels, and the difference in performance compared to A) is clear. For example, using only two channels from A) we can obtain a Kappa value of 0.667, but only 0.416 with the two channels selected in B).

Discussion

This paper investigated DL methods for 5-class sleep stage classification from EEG signals, showing that GRU-based models can outperform methods such as EEGNet, which have been used previously and show high performance. The proposed method involves an approach to channel selection based on permutations, which is computationally cheap, since it requires training a single model (or 5 if using 5-fold cross-validation) to obtain an overview of EEG channels that contribute more to sleep stage classification.

The results obtained using GRU-based models have effectively classified sleep stages using less than 6 EEG channels and obtained a performance as high as using all the 128 channels.

By comparing the performance with the best channels found and the less informative ones, we have shown the importance of selecting them correctly. The results show a decrease of approximately 10% using the best or worst channels, especially with fewer than 6 EEG channels.

In our experiments, we compared the use of the 3 channels recommended by AASM with random sets of 3 channels, and the results show the same or similar performance. This suggests that using expert knowledge for channel selection may not be optimal for automatic classification using ML/DL. As the characteristic waveforms of sleep (K complex, spindle and slow wave activity) or wakefulness (alpha wave) are distributed in the brain in a topographic manner^57–59, it was possible to extract the characteristic waveforms required for the classification of the sleep stage even if 3 channels were extracted at random. Furthermore, even if the channels were randomly extracted, the electrodes around the electrodes used to label the AASM sleep stages could have been selected and automatic sleep scoring could have been performed without significant differences in the EEG waveform.

We believe that the proposed method is a promising approach to determining the stages of sleep. The channel selection method is simple to implement and easy to understand, and since the channel importance is calculated with the trained model, it can be used with any ML or DL algorithm.

Importantly, and contrary to the state-of-the-art using a single bipolar channel, in the WPI-IIIS dataset, we found that performance decreases approximately 10% in all metrics when using fewer than 3 EEG channels.

The results have shown that we can select a set of the most relevant channels for sleep staging, but the set selected may be different between subjects. Therefore, we have to (1) obtain a set of more relevant channels from a larger population or (2) select a set of channels for each subject. The essential channels for sleep scoring in subjects in this study tended to be in the frontal to central lobes. This is consistent with the fact that characteristic EEG waveforms, such as the K complex and sleep slow waves, are frontally dominant and the spindles appear centrally dominant¹⁸, which may also be one reason for the contribution to autosleep scoring performance. In addition, possibly due to EOG artifacts such as blinks during wakefulness, rapid eye movements characteristic of stage W and REM, and slow eye movements (SEM) joints in N1 may also be included in the frontal EEG, which may help determine stage W, N1, and REM. However, performance delta presented in Figures 4 and 9 tend to be higher in the central and left temporal regions. This may first be explained by the fact that autosleep scoring picks up delta wave changes in the central and left temporal regions rather than in the frontal region, which is essential for visual scoring. The central and temporal regions have lower delta wave amplitudes than the frontal regions, so amplitude and power value changes may be easier to track than the high-amplitude frontal regions. Another possible reason is that artifacts resemble sleep-slow waves, such as sweat with slow frequency drift, or rapid eye movements from EOG, making frontal slow waves less reliable when these artifacts are present. Information from the central and temporal regions, where artifacts are less likely to glide, may have been necessary to automatically determine sleep stage N3.

AASM induction and random extraction of the 3-EEG electrode increased the number of misclassifications from N1 to N2; however, 128 channels increased the detection performance of N1, which was higher than the N1 agreement rate (63%) for the reliability of human interrater⁶⁰. The following reasons for the disagreement with N1 can be considered; (1) The detection performance of the N1 feature waveforms, such as sharp vertex waves, and the N2 feature waveforms, such as the K complex and the spindle, may be higher with 128 channels than with 3 channels. (2) In the AASM manual sleep stage scoring rule, cortical arousal during stage N2 results in the termination of N2 and the change in sleep stage to N1⁸. A total of 128 channels may have detected cortical arousals that decrease the sleep stage from N2 to N1 and more accurately detected changes in EEG frequency. In addition, 3-channel models may not be able to detect cortical arousals. (3) N1 increases SEM more than N2; N2 may produce SEM but flatter eye movements; 128 channels may have captured SEM artifacts in areas closer to the eyeballs⁸. The definition of sleep stage N1 is relatively poor and has the lowest inter-scorer reliability^60,61; Therefore, compared to human scorers, the GRU model performs relatively well.

A weak point of permutation-based channel selection is that it may provide a general indication of channels containing less/more information, since it does not consider complex channel combinations that could increase the classification performance. Complex channel combinations can be explored with approaches such as recursive channel elimination, forward addition, or NSGA^28,29. However, in all cases the computational cost is higher than using permutation-based channel selection, since all mentioned approaches need to create as many models as channel combinations are tested, which in the case of our dataset, the channel combinations are high using the 128 channels.

Future work will focus on solving the problem associated with the N1 sleep stage, since, as shown in the confusion matrices in Fig. 2, even using all available channels, N1 is misclassified mainly with N2, but other works have also shown that it is misclassified with REM and to a lesser extent with W epochs³⁸.

For this, we would like to explore the use of one-class classifiers, methods for data augmentation, use a large dataset from many subjects, or transfer learning. When using data augmentation methods, the authors have reported improvements of approximately 1–3% in N1 and up to 12% for the rest of the sleep stages, but the problem with N1 remains unsolved⁶².

In our work, we focused on the sleep staging using only EEG signals, since EOG/EMG channels could introduce noise (movements, blinking, muscle contractions) and potentially affect the classification performance without a preprocessing step, as well as adding signals in different amplitudes/frequencies. However, in our future work, we would like to explore the use of EOG signals, since other works had shown that combining EEG and EOG can obtain performance up to 0.85⁴².

Our future work will include testing our approach for sleep staging using public datasets such as SHHS, Sleep-EDF^41,43,44, and compare the performance with state-of-the-art methods. Our experimental configuration (skipping a preprocessing step and using 2-s segments) were defined based in our previous findings, however, we will further explore whether a preprocessing step or the use of a different segment size helps to increase the classification performance, with both, our proposed method and state-of-the art methods. We plan to test the permutation-based channel selection in high-density EEG datasets (higher than 128 channels), and compare if different methods results in similar performance and identifying common relevant brain areas for sleep staging.

Materials and methods

EEG recording

The dataset was collected at the International Institute for Integrative Sleep Medicine - WPI-IIIS at the University of Tsukuba, Ibaraki, Japan. The data collection process was approved by the Clinical Research Ethics Review Committee of the University of Tsukuba Hospital (R02-213). This study was also conducted in accordance with the Declaration of Helsinki, and all subjects gave their informed consent. The dataset consists of EEG recordings of 14 subjects (5 females and 9 males, 22.5±0.9 years), who were sleeping for approximately 8 h and 15 min ($\pm 2$ min for females and $\pm 4$ min for males. Sleep time was controlled by turning the lights off and on.) while their data were collected using the BioSemi headcap with 128 EEG channels, which use a radial montage and a sample rate of 1024 Hz, three EOG channels, three EMG channels, and two mastoid channels. It should be noted that we did not include EOG/EMG data for any of the experiments in this work.

The sleep stages were manually labeled every 30 seconds by a registered polysomnographic technologist. Table 3 presents the average percentage distribution of the five stages of all subjects: W, N1, N2, N3 and REM.

Table 3.

Average percentage distribution of the awake and the four sleep stages from all the subjects in the WPI-IIIS dataset.

W	N1	N2	N3	REM
9.3 ± 5.8%	11.6 ± 6.1%	37.4 ± 9.2%	20.9 ± 6.7%	20.8 ± 8.5%

In the preprocessing step, the EEG channels were re-referenced by subtracting $\frac{(M1+M2)}{2}$, which corresponds to the average between the right and left mastoids. We downsampled the dataset to 128 Hz and, to increase the number of instances in the dataset, each 30-s epoch was divided into 2-s segments, since using that segment has shown as high performance as using a larger segment (previous experiments considered 2, 5, 10, 15 and 30-s segments), it increases the number of instances, while it would be possible to detect the onset of the sleep stage faster^38,63,64.

Additional preprocessing, such as bandpass filter or notch filtering, was skipped since the experimental results have shown that the tested algorithms perform better with the raw data.

Deep learning models

For the interest of this work, here we describe relevant details of GRU; however, we have tested many other architectures proposed for EEG-related tasks, including brain-computer interfaces (BCIs)⁵⁶.

As mentioned before, EEGNet has shown good performance, and there are many current versions, and this work tested all of them, but reported only the one with the highest performance. In the case of CNN, we have tested different padding configurations, such as causal, dilated, and causal-dilated, but we report only the configurations with the highest performance.

Gated recurrent unit—GRU

GRUs are a type of RNN architecture that are designed to model sequential data and capture long-term dependencies. This makes them suitable for analyzing sleep EEG signals, which are time series data that contain information about the brain’s electrical activity during sleep. GRUs work by using a set of gating mechanisms to control the flow of information within the network. These gating mechanisms include an update gate and a reset gate^31,65.

The update gate determines how much of the previous hidden state should be remembered, while the reset gate determines how much of the previous hidden state should be forgotten^31,65. The candidate hidden state is then computed by combining the reset gate and the current input, and passing them through a nonlinear activation function, such as the hyperbolic tangent (tanh) function⁶⁶. This candidate hidden state is then combined with the output of the update gate to produce the new hidden state⁶⁵.

The GRU’s gating mechanisms allow it to selectively update and remember information over time, making it particularly effective in handling long sequences and capturing dependencies across time steps^31,65. This is important for analyzing sleep EEG signals, as these signals can be long and complex and contain information on brain activity over time.

The number of GRU layers in a model can affect its performance. Adding more GRU layers can allow the model to capture more complex patterns and dependencies in the data, but it also increases the model’s capacity, which may lead to overfitting³¹.

Table 4 presents the architecture of the 3-layer GRU model that we use, which was tested experimentally and inspired by the results obtained in a previous benchmark showing that 3-layer GRU is a good balance between performance and complexity and also has a relatively low risk of overfitting⁵⁶. The model has an input layer with 128 neurons, followed by 3 GRU layers with 100 neurons each. The output layer has 5 neurons, one for each sleep stage. We use the L2 regularization penalty with a regularization factor of 0.0001, the exponential linear unit (ELU) activation function and softmax. These parameters have been shown to improve the performance of GRU models for the classification of EEG tasks^25,26,56.

Table 4.

Followed GRU architecture for 5-class sleep staging.

Layer	# units	Parameters
Input		Input = (None, 128, 256)
GRU	100	Input$\_$shape = (no$\_$chs, samples), return$\_$sequences = True, kernel$\_$regularizer = l2(0.0001)
GRU	100	Return$\_$sequences = True, kernel$\_$regularizer = l2(0.0001)
GRU	100	Kernel$\_$regularizer = l2(0.0001)
Dropout		Rate = 0.5
Dense	100	Activation = elu
Dense		Activation = softmax, output = (None, 5)

Classification configuration

For all DL algorithms tested, we use the categorical cross-entropy loss function, and the networks are optimized with the Adam algorithm⁶⁷. In all cases, we consider a maximum of 300 epochs to train the models, applying an Early Stopping Callback to interrupt training when the validation accuracy stops improving.

The dataset was divided into 50% as training data, 25% as validation data, and 25% as test data. The performance of the model was validated using the model accuracy (referred to in the text as M-Acc), the validation accuracy (referred to in the text as V-acc), and for the test set: Accuracy, Fscore, precision, recall, AUROC (for this, we have used the one-versus-rest approach) and kappa. We tested the models using 5-fold cross-validation^68,69 , which was done by repeating the experiments, and then reported the average performance.

Since the dataset is unbalanced, we experimented with various weight values of the classes; in this way, we observed how performance increased or decreased by adjusting the weights of the classes⁷⁰. However, the experimental results show the highest performance when we use the same weight for all classes.

EEG channel selection

EEG signals are commonly used in sleep research, and there has been a desire to use a few channels to achieve real-time sleep staging. For this, methods have used expert knowledge for selecting the channels, as well as automatic methods such as greedy algorithms and MOO algorithms. The methods have been tested not only for sleep staging, but also for other tasks related to EEG, showing promising results^28,29,63.

The American Academy of Sleep Medicine - AASM recommends the use of EEG channels F4, C4 and O2, which correspond, respectively, to C4, B22 and A28 of the biosemi cap with 128 channels¹⁸. For the experiments presented, we tested three main configurations: (1) considering a set of 3 channels recommended by AASM, (2) using a set of 3 random channels (randomly selected in each fold/subject) and (3) selecting the most important channels using a method based on permutation. Additionally, we present a set of experiments using a single channel, as is done in the state-of-the-art.

Permutation-based channel selection

Traditional approaches to classifying sleep stages typically use all available EEG channels. However, the use of irrelevant or redundant information from EEG channels could lead to poor classification performance, as well as the need to use high-density EEG devices.

To address this issue, we propose the use of a computationally cheap permutation-based channel selection method. By permuting the EEG channels and evaluating the performance of the classification model, we can identify the most informative channels for the classification of sleep stages. This process guarantees the use of only the most relevant EEG channels, resulting in improved classification performance comparable to, if not exceeding, the use of high-density EEG.

Figure 10 presents an illustrative example of the process followed to identify the most relevant channels.

The process starts by creating a model with all the 128 EEG channels, for which it uses 50% of the data for training and 25% for validation.

Once the model is created, the test set instances, consisting of 25% of the dataset, are used to predict sleep stages and measure performance. The performance obtained is considered the baseline performance, from which the other performances will be compared.

We selected data from the first EEG channel of all instances in the test set and then randomly assigned them to the instances. This means that instances will have different information at that channel position than when they were collected, and since the model learned that some specific information must be at that channel position, we can measure whether the information from the channel is important to classify the sleep stages.

When computing the performance delta, we can use any of the metrics, but in this paper we use the average difference between the metrics (accuracy, Fscore, precision, recall, AUROC, and kappa). Throughout the document, we refer to Performance delta as the difference between the performance obtained at the baseline (using all channels) and the model that permutes the information from certain channels.

This process is repeated for all EEG channels, and we can then analyze which channels are decreasing or increasing performance, reflected in the importance of the channel for the classification of stages of sleep. Furthermore, this process is repeated using a 5-fold cross-validation and for all subjects in the dataset individually, thus ensuring that the important channels are the same for different test sets.

In the example presented in Fig. 10, the baseline accuracy is 90%, and the accuracy decreases to 85% when the information from channel 2 (ch 2 in the figure) is changed. The accuracy decreases to 89% when removing the information from channel 3 (ch 3 in the figure). After the process is completed, we can see the graph located at the bottom right, which illustrates that the performance decreases more when we remove the information from channel 2, which means that channel 2 is more important than the other channels for high performance.

From this analysis, we can also obtain more information, for example, if some channels are not relevant or even if they contribute to reducing performance, as in the example illustrated with the last channel in the graph.

Two-way ANOVA

A two-way ANOVA was performed to evaluate the effects of the methods tested, the performance metrics. For this, we used the python library Statsmodels⁷¹. The ANOVA results were calculated using ordinary least squares (OLS) regression and the ANOVA tables were generated, showing the main effects and interaction effects of the factors.

The analysis was carried out using two factors: Method (two levels, representing the two different methods being compared. Example 1: GRU and EEGNeX, example 2: GRU-3AASM and EEGNeX-3AASM), and Metric (six levels, corresponding to the performance metrics used to validate the classification models). For more information on comparison settings, see the y information. Multiple comparison corrections were not performed, since the performance metrics used for ANOVA already represent the averages from the 5-fold cross-validation⁷².

Author contributions

L.A.M devised the methods, performed the experiments, analyzed the results, and wrote the manuscript. Y.S, M.M, and T.A discussed the results and partially wrote, read and revised the manuscript. J.F collected and labeled the dataset.

Data and code availability

The dataset collected at the University of Tsukuba is available from the corresponding author upon reasonable request. The classifier code, including an example with a random subset of the dataset, is available at https://github.com/wavesresearch/sleep-staging_IIIS.

Competing interests

The authors declare that the investigation was carried out in the absence of commercial or financial relationships that could be construed as a potential conflict of interest.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-024-68978-4.