Validation of a wearable forehead sleep recorder against polysomnography in sleep staging and desaturation events in a clinical sample

Abstract

Study Objectives:

Wearable sleep recording devices may be a helpful alternative method for polysomnography (PSG) due to their higher accessibility and comfort as well as lower cost, but their validities need to be examined. The aim of this study was to evaluate the accuracy of a novel single-channel, electroencephalography-based wearable forehead sleep recorder (UMindSleep) to assess sleep staging and oxygen desaturation.

Methods:

Two hundred and three Chinese adults recruited from a sleep medicine center underwent an overnight study wearing UMindSleep and PSG simultaneously. Sleep parameters including sleep staging and oxygen desaturation index were compared between UMindSleep and PSG.

Results:

A total of 195,349 valid epochs from 197 participants (171 with obstructive sleep apnea, 86.8%) were included in analyses of sleep staging. Sensitivities of UMindSleep compared to PSG were 79.7% for wake, 85.8% for light sleep, 79.4% for deep sleep, and 82.7% for rapid eye movement sleep. Specificities were 95.3% for wake, 83.4% for light sleep, 97.0% for deep sleep, and 96.8% for rapid eye movement sleep. Furthermore, the kappa agreements of 0.69–0.79 were indicative of a substantial agreement for sleep staging between UMindSleep and PSG. Sensitivity and specificity regarding oxygen desaturation index were 93.4% and 88.9%, yielding a kappa coefficient of 0.82.

Conclusions:

Our findings suggest that UMindSleep may serve as a feasible, accurate, and dependable device for screening of sleep disorders (eg, obstructive sleep apnea) and assessing sleep structure.

Citation:

Chen X, Jin X, Zhang J, Ho KW, Wei Y, Cheng H. Validation of a wearable forehead sleep recorder against polysomnography in sleep staging and desaturation events in a clinical sample. J Clin Sleep Med. 2023;19(4):711–718.

BRIEF SUMMARY

Current Knowledge/Study Rationale: Electroencephalography-based sleep monitoring techniques have considerable potential to measure sleep more precisely; however, few wearable devices adopting electroencephalography-based techniques have been validated. The current study validated an electroencephalography-based wearable forehead sleep recorder, UMindSleep, against polysomnography on sleep staging and oxygen desaturation in a clinical sample.

Study Impact: This validation study found that this electroencephalography-based device showed an overall good capacity in automatic sleep staging, with a clear advance in detecting wakefulness. It also showed a satisfying capability and dependability to detect sleep-related desaturation events among patients with obstructive sleep apnea.

INTRODUCTION

Sleep is of vital importance for health and quality of life. A large portion of the population experiences sleep disorders such as obstructive sleep apnea (OSA) and insomnia and various sleep-related daytime symptoms such as fatigue, excessive sleepiness, morning headaches, and impaired concentration and memory.¹ Individuals with sleep disorders also face a significantly higher risk of motor vehicle accidents,² cardiovascular morbidity, hypertension, stroke, type 2 diabetes, and diminished quality of life.^3,4

There are various methods for the diagnosis and assessment of sleep disorders, among which laboratory-based polysomnography (PSG) is considered to be the gold standard.^5,6 PSG captures and records multiple physiological parameters during sleep, allowing the precise quantification of sleep staging, airflow and respiratory effort, oxygen saturation, heart rate and rhythm, leg movement, and body position,⁷ from which the key features of sleep disorders can be detected. However, PSG is unnatural, potentially intrusive, money- and time-consuming, laboratory-based, labor-intensive, and not always available;^3,8 all of these factors limit its clinical application to a large portion of the population.

A wearable sleep recording device may have a role as a valuable alternative to PSG, allowing for longitudinal and convenient habitual sleep monitoring. Various kinds of wearable sleep monitors have been tested, and an overall acceptable accuracy convinced the researchers that wearable monitors are potentially suitable for screening of sleep disorders at home.^6,9 However, considering the fact that wearable devices include fewer sensors and different algorithms compared to standard PSG and need to be applied to different populations, it is necessary to examine the validity of each device carefully.

The present study aimed to examine the validity of a wearable forehead sleep recorder, UMindSleep, for sleep assessment in a clinical sample of Chinese adults. UMindSleep is a newly developed forehead-worn sleep monitor embedded with a single-channel electroencephalograph (EEG) and an oximeter. In this study, UMindSleep was compared to simultaneous standard PSG monitoring during a whole-night sleep in order to determine its accuracy in sleep staging and measurement of oxygen desaturation index (ODI), both of which are critical parameters in assessing and screening sleep disorders.

METHODS

Study design and setting

This was an observational study with a within-subject design. Data were collected at the Sleep Medicine Center of Shenzhen People’s Hospital in China from October 14, 2020 to May 12, 2021. All procedures were approved by the Ethics Committee of Drug Clinical Trial of Shenzhen People’s Hospital (SYL-202010-03).

Participants

The sample size was calculated using PASS version 15.0 (PASS, Inc., NCSS Company, Kaysville, Utah) based on Cohen’s kappa, alpha = 0.05, power = 90%. Results suggested a minimum sample size of 154 participants to be recruited in order to determine an absolute difference between PSG and UMindSleep. We further anticipated a 20% attrition rate, resulting in an anticipated sample size of 193 participants for the current study. As Figure 1 shows, a total of 203 participants were recruited in this study. Data of 197 participants between 20 and 63 years of age (37 ± 8.7 years; 148 were male) were included in analysis. Among them, 171 (86.8%) were diagnosed with OSA by standard PSG. See Table 1 for more details. All participants provided written informed consent.

Figure 1. Flow chart of the study participants.

PSG = polysomnography, UMS = UMindSleep forehead sleep recorder.

Table 1.

Demographic information, BMI, and AHI of the participants.

	OSA (n = 171)	Non-OSA (n = 26)	t/χ2	P
Sex (male), n (%)	145 (84.8%)	3 (11.5%)	65.242	<.001
Age (years), mean ± SD	37.40 ± 8.45	34.73 ± 10.25	1.452	.148
BMI (kg/m2), mean ± SD	25.48 ± 3.20	21.56 ± 3.60	5.750	<.001
AHI (events/h), mean ± SD	31.18 ± 23.24	1.83 ± 1.29	16.373	<.001

AHI = apnea-hypopnea index, BMI = body mass index, OSA = obstructive sleep apnea.

Participants meeting the following criteria were enrolled: (1) male or female, 18–65 years of age and (2) ability to understand the study procedures and assessment.

Individuals were excluded for (1) less than 18 years of age or more than 65 years of age; (2) patients with narcolepsy or rapid eye movement sleep behavior disorder whose abnormal sleep structure or increasing muscle tone and myoclonic twitching during sleep could interfere with the validation; (3) pregnancy; (4) serious primary heart, liver, kidney, or blood system diseases or other life-threatening diseases; (5) taking sleep medications or antipsychotics regularly; (6) serious mental or neurological diseases (bipolar disorder, schizophrenia, epilepsy, etc); (7) participation in any clinical trials in the past 3 months; (8) basic skin problems such as acne, breakage, allergic rash, etc, of the forehead; and (9) confirmed skin sensitivity.

UMindSleep

UMindSleep (EEGSmart Co., Ltd., Shenzhen, Guangdong, China) is a wearable forehead sleep recorder designed for at-laboratory and at-home sleep monitoring. Embedded with a single-channel EEG, it makes continuous automatic identification of 4 sleep stages, wake, light sleep, deep sleep, and rapid eye movement (REM) sleep, from the captured EEG signals using built-in algorithms every 30 seconds during sleep. It also integrates a pulse oximeter to calculate blood-oxygen saturation level (SpO₂) at a 30-second rate.

Laboratory PSG system and recordings

All participants underwent a full‐night at-laboratory monitoring using the Philips Alice 6 PSG system (Philips, Respironics, Murrysville, Pennsylvania). Recorded data included all electrophysiological signals for sleep evaluation as well as airflow by thermistor and nasal pressure and pulse oximetry.

Sleep scoring and oxygen decrement events calculation

Sleep and respiratory events were scored visually, according to American Academy of Sleep Medicine 2012 guidelines,¹⁰ by qualified technicians. Certified polysomnographic technologists conducted visual scoring of PSG data in 30-second epochs to identify the sleep stage (wake, N1, N2, N3, REM, or unknown) and any arousal. Epochs with poor‐quality signals were excluded from the analysis.

An oxygen decrement event is determined if the monitored oxygen value during sleep is 3% below the oxygen baseline and lasts no less than 10 seconds. ODI refers to the average number of oxygen desaturation events that occur per hour throughout the sleep period.

Statistical analyses

We examined the accuracy and validity of UMindSleep with 2-part data analyses: part 1 for sleep staging and part 2 for desaturation events. In both parts we compared the outcomes of UMindSleep with those of the standard PSG.

Data with normal distribution are expressed as mean ± SD, and classification variables are defined as percentage (%). Between‐group comparisons were performed using the chi‐squared test for categorical variables and t test for continuous variables.

Cohen’s kappa was used to assess agreements for the pairwise comparisons, providing a method for quantifying the agreement between UMindSleep and PSG. As a supplement, statistical measures of agreements, ie, sensitivity and specificity, as well as positive predictive value (PPV), negative predictive value (NPV), and accuracy were calculated to assess the performance of UMindSleep at sleep staging and desaturation events.

To further demonstrate the agreement between UMindSleep and PSG, a Bland-Altman plot analysis was conducted. It provided a visually informative way of finding systematic biases and outliers. A Deming linear regression was applied to evaluate the correlations between the UMindSleep and PSG outcomes.

Statistical analyses were performed using IBM SPSS version 26.0 software (SPSS, Inc., IBM Company, Armonk, New York), GraphPad Prism version 9.0.0 software (GraphPad Software, Inc., La Jolla, California), and MATLAB version R2018b software (The MathWorks, Inc., Natick, Massachusetts). For all statistical tests, P < .05 was considered statistically significant.

RESULTS

Part 1: Sleep staging

Data preprocessing

A total of 196,444 epochs of data were generated by the forehead sleep recorder, among which 1,095 epochs were determined as invalid (0.56%) after quality control and were deleted from further analysis. The remaining 195,349 epochs of data were used to compare with sleep stages derived from PSG data.

Epoch-by-epoch comparison

The 2-state epoch-by-epoch agreement was defined as the percentage of epochs that were assigned the same state. Agreement on sleep staging between UMindSleep and PSG were compared at 4 levels: wake, light sleep (combining PSG stages N1 and N2), deep sleep (PSG stage N3), and REM sleep. Table 2 shows the confusion matrix for UMindSleep vs PSG scoring consensus. According to the matrices, wake is most often misclassified as light sleep (16.8% of epochs), deep sleep is most often misclassified as light sleep (16.8% of epochs), and REM sleep is most often misclassified as light sleep (14.0% of epochs).

Table 2.

The confusion matrix for UMS vs PSG scoring consensus.

	UMS
PSG	Overall	Wake	Light sleep	Deep sleep	REM sleep
	Wake	21,138 (79.7%)	4452 (16.8%)	102 (0.4%)	815 (3.1%)
	Light sleep	6,805 (5.9%)	98,618 (85.8%)	4,991 (4.3%)	4,537 (4.0%)
	Deep sleep	237 (1.0%)	4,737 (19.4%)	19,402 (79.4%)	47 (0.2%)
	REM sleep	950 (3.3%)	4010 (14.0%)	5 (0.0%)	23,740 (82.7%)

Values represent the number of epochs in each condition, and percentages (%) are normalized by row. PSG = polysomnography, REM = rapid eye movement, UMS = UMindSleep forehead sleep recorder.

Kappa values

Kappa values for sleep staging described in Table 3 ranged from 0.7 to 0.8, which indicated substantial agreements (0.6 ≤ kappa < 0.8) between UMindSleep and PSG according to the classification of Viera and Garrett.¹¹

Table 3.

The sensitivity, specificity, PPV, NPV, and kappa coefficients of UMS for wake, light sleep, deep sleep, and REM sleep.

	Wake		Light Sleep		Deep Sleep		REM Sleep
	Value	95% CI	Value	95% CI	Value	95% CI	Value	95% CI
Sensitivity (%)	79.74	79.26–80.23	85.79	85.59–85.99	79.44	78.93–79.95	82.70	82.26–83.14
Specificity (%)	95.25	95.14–95.35	83.43	83.17–83.68	97.00	96.92–97.08	96.75	96.66–96.83
PPV (%)	72.56	72.12–73.00	88.20	88.03–88.36	79.19	78.73–79.65	81.47	81.06–81.87
NPV (%)	96.76	96.68–96.83	80.27	80.04–80.50	97.05	96.98–97.12	97.00	96.92–97.07
Accuracy (%)	93.13	93.02–93.25	84.82	84.66–84.98	94.80	94.70–94.90	94.67	94.57–94.77
Cohen’s kappa	0.72	0.715–0.724	0.69	0.685–0.691	0.76	0.759–0.768	0.79	0.785–0.793

CI = confidence interval, NPV = negative predictive value, PPV = positive predictive value, REM = rapid eye movement, UMS = UMindSleep forehead sleep recorder.

Sensitivity, specificity, PPV, and NPV

Table 3 shows the statistics of sensitivity, specificity, PPV, NPV, and accuracy for each state. Taking distinguishing wake/sleep as an example, the overall sensitivity of UMindSleep on the entire sample of 197 participants was 79.74% for detecting wake, specificity was 95.25%, and the PPV and the NPV were 72.56% and 96.76%, respectively (Table 3, row 2). Similarly, the sensitivity of UMindSleep to identify light sleep/non-light sleep, deep sleep/non-deep sleep, and REM sleep/non-REM sleep ranged from 79.44–85.79% and the NPV ranged from 80.27–97.05%.

Bland-Altman plots

The level of agreement between UMindSleep and PSG on sleep staging was further assessed by Bland-Altman plots¹² (Figure 2). Results showed no significant biases for UMindSleep except that it tended to overestimate the wake after sleep onset (bias: −9.85 ± 25.83, P < .001) and underestimate the percentage of light sleep (bias: 1.90 ± 9.51, P < .001). The paired t test was also used to compare the mean differences of the sleep parameters between UMindSleep and PSG (see Table S1 ^{(351.4KB, pdf)} in the supplemental material for details).

Figure 2. Bland-Altman plot comparisons between the PSG and UMS for detection of the indexes of sleep architecture.

The horizontal coordinate of the Bland-Altman plot is the mean of the sleep parameters recorded by the UMS and the PSG, and the vertical coordinate is the difference between them. Each dot represents a participant, the solid black line represents the mean difference of PSG minus the mean difference of UMS, and the upper and lower red dashed lines represent the 95% confidence interval for the mean difference, which indicates very good agreement between the two detection methods when the difference lies within the 95% confidence interval. PSG = polysomnography, REM = rapid eye movement, UMS = UMindSleep forehead sleep recorder.

Correlation analysis

As depicted in Figure S1 ^{(351.4KB, pdf)} , a significant positive correlation between UMindSleep and PSG was found for sleep staging.

Part 2: Desaturation events

Cohen’s kappa, sensitivity, and PPV

Using a cutoff of ODI > 10, UMindSleep was found to have a moderate level of agreement in identifying OSA with PSG (kappa value = 0.82, P < .001), with a high level of sensitivity (93.4%) and specificity (88.9%). The PPV and NPV were 93.4% and 88.9% respectively.

Bland-Altman plots

Figure 3 shows that the mean difference between the two methods was 1.88 and the 95% confidence interval for the difference was −11.13 to 14.89 (P < .001), with most of the measurements falling within the 95% confidence interval. It seemed that the frontal-post sleep recorder tended to underestimate the severity of the ODI compared with PSG mainly among those with severe OSA. A subgroup analysis between participants with and without OSA suggested that UMindSleep specifically underestimated ODI for the OSA subgroup (bias: 2.51 ± 6.85, P < .001). More detailed results of the subgroup analysis are shown in Figure S2 ^{(351.4KB, pdf)} .

Figure 3. Bland-Altman plot comparisons between PSG and UMS for detection of oxygen desaturation index.

The horizontal coordinate of the Bland-Altman plot is the mean of the ODI recorded by the UMS and the PSG, and the vertical coordinate is the difference between them. Each dot represents a participant, the solid black line represents the mean difference of PSG minus the mean difference of UMS, and the upper and lower red dashed lines represent the 95% confidence interval for the mean difference, which indicates very good agreement between the two detection methods when the difference lies within the 95% confidence interval. ODI = oxygen desaturation index, PSG = polysomnography, REM = rapid eye movement, UMS = UMindSleep forehead sleep recorder.

Correlation analysis

As shown in Figure S3 ^{(351.4KB, pdf)} , a significant positive correlation between UMindSleep and PSG was found for oxygen desaturation.

DISCUSSION

The current study examined the validity of UMindSleep, a wearable forehead sleep recorder, with standard whole-night PSG for sleep assessment in a clinical sample. Results suggested an overall satisfying validity of UMindSleep: a competitive agreement with PSG for the staging of all 4 sleep stages—wake, light sleep, deep sleep, and REM sleep. No bias in the estimation of object sleep parameters relative to PSG was found, except that it slightly overestimated wake after sleep onset and underestimated the percentage of light sleep. UMindSleep also agreed well with PSG on ODI among patients with OSA overall but tended to underestimate ODI among those with more severe OSA. These results indicated promising applicability of UMindSleep to clinical practices as well as at-home use but also recognized some potential limitations.

For sleep staging, the agreement between PSG and UMindSleep was substantial. Among different sleep stages, light sleep generally had the lowest level of agreement, and its proportion was underestimated by around 2%, which was the only sleep stage whose detection bias reached a statistical significance. Poor detection of light sleep has been found repeatedly among different sleep-monitoring techniques.^13,14 This is probably due to the nature of PSG-defined N1 sleep, which has repeatedly been found to consist of a vaguely understood mixture of wakefulness and sleep rather than being clearly defined.^15,16 Nevertheless, the agreement for light sleep was still favorable compared to that of most of the existing wearable devices and single-channel EEG-based algorithms.^13,14,17,18

Being EEG-based, UMindSleep is expected to be superior to actigraphy-based devices, especially in detecting wakefulness.¹³ The current findings supported this hypothesis by showing a specificity of 95.25% for wake detection, which is comparable to other wearable sleep monitoring devices (93–98%), and a sensitivity of 79.74% for wake detection that surpassed most of the existing devices (∼20% to ∼80%, averaging ∼50%).^19,20 This straightforwardly revealed the inbuilt significance of EEG-based devices in sleep staging. In PSG, sleep stages are derived from EEG channels. Therefore, even single-channel EEG signals intuitively contain richer information about sleep staging and are therefore capable of carrying out more precise sleep staging.^18,21,22 In the current study, UMindSleep showed a significant bias to overestimate wake after sleep onset by 9.8 minutes, while the majority of existing wearable devices tended to underestimate this sleep metric.¹⁹ This pointed out a need and direction for future improvement of the built-in algorithms of UMindSleep.

ODI plays a vital role in the definition of a hypopnea event by the American Academy of Sleep Medicine²³ and is highly accessible with low cost compared to other OSA diagnostic indicators. Agreement on ODI between UMindSleep and PSG reached a satisfying level in the current study. UMindSleep tended to underestimate ODI among patients with more severe OSA, but their ODIs were already far above the diagnostic criteria. Results revealed insensitivity of UMindSleep to ODI among the non-OSA participants, but this interpretation was limited by the small sample size. Future studies on this group of people may reveal the utility of UMindSleep in predicting OSA onset.

The current findings validated the use of UMindSleep in sleep assessment among Chinese populations, especially among the clinical sample of which the majority were diagnosed with OSA. Currently, OSA diagnosis in China depends almost exclusively on in-laboratory PSG, which is expensive, inconvenient, and largely inadequate to meet the needs of a large number of patients. Wearable sleep recording devices have the potential to completely overcome these deficiencies. However, the majority of existing wearable devices are actigraphy-based and are largely limited in their agreement with PSG. Together with some existing evidence,¹³ the current finding suggests that a single-channel, EEG-based sleep monitoring technique is capable of more precise sleep assessment while being wearable, affordable, user-friendly, and mass-producible, making it an exciting alternative to PSG.

Some potential limitations concerning applying UMindSleep to sleep assessment need to be recognized and discussed. First, the device is designed for both in-laboratory and at-home use. Only in-laboratory validation was included in the current study, so the at-home validity needs further investigation. Second, participants were recruited from a clinical OSA sample, which may make the conclusions in the current study somewhat less applicable to more generalized populations other than patients with OSA. Third, 1.5% of the UMindSleep monitoring failed due to lead-fall or disconnection. Lead-fall is somewhat inevitable considering that UMindSleep is stuck to the forehead during sleep. In this case, sticking the device to the forehead again and rerunning the assessment will solve the problem.

CONCLUSIONS

Overall, the current results reveal that UMindSleep is of good validity in sleep assessment in a Chinese adult clinical sample. Although UMindSleep does have limitations compared to standard in-laboratory PSG, considering that it is designed to be highly user-friendly and affordable it may have potential as an alternative tool for both clinical and research purposes.

ABBREVIATIONS

EEG

electroencephalograph

NPV

negative predictive value

ODI

oxygen desaturation index

OSA

obstructive sleep apnea

PPV

positive predictive value

PSG

polysomnography

REM

rapid eye movement

DISCLOSURE STATEMENT

All authors have seen and approved the manuscript. This study was funded by EEGSmart Co., Ltd., which also provided the UMindSleep devices used in the study. The funding source played no role in the study design, data collection and analyses, finding interpretation, manuscript preparation, review and approval, or decision to submit the manuscript for publication. Dr. Jihui Zhang collaborated with EEGSmart Co., Ltd. for a grant funded by the National key Research and Development Program of China (2021YFC2501500), which also played no role in any part of the current study.