Abstract
BACKGROUND:
Manual polysomnography is considered as the gold-standard for diagnosis of sleep disorders. However, it is a labor intensive and time-consuming task and is being replaced by automated sleep scoring systems.
METHODS:
This cross-sectional study was conducted at Sleep Medicine and Research Center, King Abdulaziz University Hospital, Jeddah, Saudi Arabia, between January 2022 and July 2024, the correlation and agreement between various parameters recorded through automated and manual sleep scoring.
RESULTS:
Of a total of 442 participants, 234 (53.4%) were male and 206 (46.6%) were female. The mean age of participants was 48 ± 18.5 years. A moderate degree of agreement was found between apnea–hypopnea index (AHI) (κ = 0.510) and respiratory disturbance index supine (κ = 0.474) recorded through automated and manual scoring (P < 0.001). A fair degree of agreement (κ = 0.308) was recorded for AHI-rapid eye movement (P < 0.001) (P < 0.001). Intra-class correlation showed a good agreement between total sleep time (TST) recorded through both systems (ICC = 0.727, P < 0.001). AHI events recorded through both systems showed good agreement (ICC = 0.753, P < 0.001), whereas an excellent agreement was observed for TST supine (ICC = 0.902, P < 0.001).
CONCLUSION:
There appears to be a marked variation in the accuracy of various sleep parameters by automated sleep recording. Further studies that utilize advance machine learning algorithms can help better understanding of the role of automated sleep scoring in screening and diagnosis of sleep disorders.
KEYWORDS: Obstructive sleep apnea, polysomnography, sleep disorders, sleep stages
Background
Sleep is a vital function of human body which is essential for maintenance of physical and mental health. On an average, humans spend one-third of their life sleeping.[1] The sleep cycles involves two phases; the rapid eye movement (REM) phase and the non-REM (NREM) phase. The NREM phase further comprises of three stages from N1 to N3. Each of these phases or stages shows variations in muscle tone, eye movements, and unique brain wave patterns. A human body cycles through these stages on an average of four to six times each night with each cycle spanning over 90 min.[2] Various disorders of sleep can affect the balance between these stages and lead to poor quality of sleep. Obstructive sleep apnea (OSA), which is one of the most common sleep disorders, causes collapse of airways during deep sleep, which results in reduced time in N3 stage and the REM phase of sleep. This affects the quality of sleep and results in symptoms such as excessive day time somnolence and feeling tired throughout the day.[3]
The screening and diagnosis of sleep disorders is conducted through overnight polysomnography (PSG), a process which involves continuous monitoring and recording of physiological activity during sleep. PSG combines electroencephalography (EEG) with electrocardiography (ECG), respiratory channels, and body position sensors. Recently, PSG has been transformed to incorporate electromyography (EMG), electrooculography (EOG), and end-tidal capnography.[4] Multiple 30 s epochs taken from a PSG can be studied to assign a sleep stage, a process known as sleep scoring or sleep staging.[5] Various parameters of sleep macrostructure can be studies to diagnose sleep disorders. These parameters include sleep cycle, total sleep time (TST), relative time spent in each stage of sleep, REM latency, and apnea–hypopnea index (AHI).[6] and are, in fact, a source of diagnostic workup of sleep-related disorders including common conditions such as OSA, insomnia, parasomnia, and restless leg syndrome[7] to much rarer varieties of sleep disorders such as sleep paralysis and narcolepsy.[8]
Manual PSG has long been used as the gold-standard for diagnosis of sleep disorders. It is based on well-established guidelines, such as those published by the American Academy of Sleep Medicine (AASM).[9,10] Nevertheless, it is a time consuming and labor intensive process and is generally prone to inter-rater variations.[4] As the awareness of sleep disorders has increased in the recent years, handling large amount of data sets and screening of millions of suspected cases of sleep disorders is becoming increasingly difficult with manual PSG. These limitations have effectively been addressed by the development of automated sleep scoring systems.[11,12]
Automated sleep scoring systems can incorporate objective decision making process into sleep monitoring through remote ecosystem. This has an added benefit that data acquisition can be done through the comfort of home environment instead of sleep clinics.[13] Automated sleep scoring systems utilize artificial intelligence (AI)-based algorithms to interpret physiological signals and aid in the diagnosis of sleep disorders.[14] Various studies have shown automated systems to be reasonably effective in screening for sleep disorders.[15,16,17] However, there is a variation in the results of these studies and large scale implementation of automated scoring systems still faces certain challenges that need to be addressed. Various automated scoring systems are now being routinely practiced in Saudi Arabia but there are limited data to support the widespread use of automated scoring systems in local population. This study was therefore designed to find the correlation and agreement between various parameters recorded through automated and manual sleep scoring systems.
Materials and Methods
Study design and participants
The present study is a prospective cross-sectional study conducted at Sleep Medicine and Research Center, King Abdulaziz University Hospital, Jeddah, Saudi Arabia, between January 2022 and July 2024. Ethical approval for the study was taken from the hospital Ethical Review Committee (Reference No 357-24). A nonprobability consecutive sampling techniques were applied and all the patients referred to the center for PSG to screen for OSA were included in the study. Informed consent was taken from all the participants. All the participants presenting during the study and who provided consent regardless of age, gender, and body mass index (BMI) were included in the study. Patients not providing informed consent, having degenerative neurological diseases or history of alcohol or any drug addiction were excluded from the study. A complete medical history was taken and patients with a history of any psychiatric comorbidities as per the Diagnostic and Statistical Manual of Mental Disorders, Fifth edition were excluded from the study. Height and weight of all the study participants were recorded and BMI was calculated. The BMI was categorized as per the World Health Organization classification.[18] Participants with BMI <18.5 Kg/m2 were labeled as “underweight,” 18.5–24.9 Kg/m2 as “normal weight,” 25–29.9 Kg/m2 as “overweight,” and >30 Kg/m2 as “obese.”
Evaluation protocol
Sleep scoring was done by certified sleep technologists with at least 5 years of experience. Both manual and automated systems were applied to assess various sleep parameters such as TST, REM latency, REM duration, N1, N2, and N3 time, the AHI, the AHI in REM sleep (AHI-REM), sleep time supine, respiratory disturbance index (RDI) supine, and minimum SpO2 concentration.
Manual sleep scoring
Manual PSG was conducted in-lab in the sleep testing center where the participants were required to stay over-night. Manual PSG recordings were obtained following the AASM scoring protocol. Six-electrode scalp-encephalography including Central [C] 3–Auricular [A] 2, C4–A1, Occipital O1–A2, O2–A1, Frontal F3–A2, and F4–A1 electrodes was performed.[19,20,21] Two-electrode EOG was done by placing electrodes near the outer canthus of both eyes. ECG recording was done and EMG was conducted by placing sensors oven chin and anterior tibialis muscle, bilaterally. Sensors for detection of airflow through oral/nasal orifices and for detection of chest/abdominal wall movements were placed. Pulse oximetry was used for oxygen saturation (SpO2) monitoring. The default sampling rate of EEG and filter settings were set at 200 Hz and 0.3–35 Hz, respectively. Data recorded from EEG were analyzed to classify sleep into five stages (wake stage, REM, and N1, N2 and N3 stages). Scoring was done for each 30 s epoch by PSG experts following the guidelines of AASM scoring manual.[9,10]
To eliminate bias, three experts with more than 5 years of experience, who were blinded to study participants’ conditions evaluated the data separately. The AHI was estimated by counting the number of apnea episodes per hour. An episode of apnea was labeled by cessation of breathing for at least 10 s and hypopnea was labeled as a reduction in respiratory effort of amplitude of airflow by 30% or more along with more than 3% decrease in SpO2 for at least 10 s or the presence of arousal from sleep.[22]
Automated sleep scoring
Automated sleep scoring system by “SOMNOmedics” based on “DOMINO” software version 3.0.0.8 was used.
Statistical analysis
The sleep data were first entered into Microsoft Excel version 2016. The worksheets from Microsoft Excel were later imported in to Statistical Package for Social Sciences (SPSS) (IBM SPSS Inc., Armonk, NY, USA) version 26.0 for further analysis through its statistical package. Demographic details of the participants such as height, weight, and BMI were recorded and presented as mean ± standard deviation (SD). The categorical variables such as BMI categories were presented as frequency and percentages. Sleep parameters such as TST, REM latency, REM duration, N1, N2 and N3 time, AHI, AHI-REM, sleep time supine, RDI supine, and minimum SpO2 (%) were also presented as mean ± SD for both automated and manual scoring systems along with 95% confidence intervals for the means. Paired sample T-Tests were applied to assess significant difference between the values recorded by both the scoring systems.
The agreement between sleep parameters recorded through both scoring systems was assessed through applying Cohen’s kappa statistics. A κ = 0.00–0.20 was labeled as “poor agreement,” 0.21–0.40 as “fair agreement,” 0.41–0.60 as “moderate agreement,” 0.61–0.80 as “good agreement,” and 0.85–1.00 as an “excellent agreement.”[23] The intra-class correlation (ICC) between the sleep parameters recorded through both systems was also assessed. ICC estimated values <0.40 were regarded as “poor agreement,” 0.40–0.54 as “weak agreement,” 0.55–0.69 as “moderate agreement,” 0.70–0.84 as “good agreement,” and 0.85–1.00 as an “excellent agreement.”[24] Bland–Altman plots were constructed to find the agreement between various sleep parameters recorded through both systems. The mean difference along with the limits of agreement (LoA) was calculated. A P < 0.05 was considered as statistically significant.
Results
A total of 442 participants were included in the study. Out of these cases, 234 (53.4%) were male and 206 (46.6%) were female. The mean age of all the participants was 48 ± 18.5 years. The mean height of all the participants was 162.5 ± 14.0 cm and the mean weight was 86.9 ± 27.6 kg. The mean BMI was 32.5 ± 9.3 kg/m2 and 248 (56.1%) participants were categorized as obese (BMI > 30 kg/m2). The baseline characteristics of the study participants are presented in Table 1.
Table 1.
Baseline characteristics of study participants (n=442).
| Mean±SD | Minimum | Maximum | |
|---|---|---|---|
| Age (years) | 48.0±18.5 | 3 | 95 |
| Height (cm) | 162.5±14.0 | 66.0 | 199.0 |
| Weight (kg) | 86.9±27.6 | 5.6 | 205.0 |
| BMI (kg/m2) | 32.5±9.3 | 10.0 | 70.3 |
|
| |||
| Frequency, n (%) | |||
|
| |||
| Gender | |||
| Male | 236 (53.4) | ||
| Female | 206 (46.6) | ||
| BMI categories (kg/m2) | |||
| Underweight - <18.5 | 16 (3.6) | ||
| Normal weight - 18.5–24.9 | 70 (15.8) | ||
| Overweight - 25–29.9 | 108 (24.4) | ||
| Obese - >30 | 248 (56.1) | ||
SD=Standard deviation, BMI=Body mass index.
Difference in sleep parameters recorded through automated and manual scoring systems
Various sleep parameters were recorded through automated and manual scoring systems. The mean TST recorded through automated scoring system was 267.0 ± 117.9 min and through manual scoring system was 294.5 ± 103.8 min, with a mean difference of −27.5 min (P < 0.001). REM latency in the automated scoring system was 76.7 ± 91.7 min and in the manual scoring system was 128.2 ± 84.7 min, with a mean difference of-52.6 min (P < 0.001). Total duration of REM sleep was 46.9 ± 48.4 min in the automated scoring system and 46.5 ± 32.2 min in the manual scoring system with a mean difference of 0.4 min (P = 0.861). The number of AHI events per hour was 16.3 ± 22.9 in automated scoring system and 18.5 ± 21.2 in manual scoring system showing a mean difference of −2.2 events (P = 0.003). The minimum SpO2 concentration recorded through automated scoring system was 83.7 ± 14.0% and through manual scoring system was 84.4 ± 10.9% and a mean difference of −0.7% was observed (P = 0.296). The difference between various sleep parameters recorded through automated and manual scoring systems is given in Table 2.
Table 2.
Sleep parameters recorded through automated and manual scoring (n=442).
| Sleep parameter | Automated scoring |
Manual scoring |
Mean difference | P | ||
|---|---|---|---|---|---|---|
| Mean±SD | 95% CI for mean | Mean±SD | 95% CI for mean | |||
| TST (min) | 267.0±117.9 | 256.0–278.0 | 294.5±103.8 | 284.8–304.2 | −27.5 | <0.001* |
| REM latency (min) | 76.7±91.7 | 65.8–83.6 | 128.2±84.7 | 121.7–138.7 | −52.6 | <0.001* |
| REM duration (min) | 46.9±48.4 | 42.4–51.4 | 46.5±32.2 | 43.5–49.5 | 0.4 | 0.861 |
| N1 time (min) | 18.3±16.4 | 16.7–19.8 | 37.9±27.8 | 35.3–40.5 | −19.6 | <0.001* |
| N2 time (min) | 98.0±67.9 | 91.6–104.3 | 142.3±63.7 | 136.4–148.3 | −44.4 | <0.001* |
| N3 time (min) | 104.3±77.0 | 97.1–111.5 | 69.1±47.9 | 64.6–73.5 | 35.3 | <0.001* |
| %TST REM | 17.4±17.8 | 15.7–19.1 | 38.3±28.1 | 35.3–40.5 | −20.9 | <0.001* |
| %TST N1 | 8.6±12.6 | 7.4–9.8 | 14.9±13.3 | 13.6–16.1 | −6.3 | <0.001* |
| %TST N2 | 35.0±18.8 | 33.2–36.8 | 47.4±13.9 | 46.2–48.8 | −12.4 | <0.001* |
| %TST N3 | 37.0±21.8 | 34.9–39.0 | 22.9±13.9 | 21.5–24.2 | 14.1 | <0.001* |
| AHI (events/h) | 16.3±22.9 | 14.2–18.5 | 18.5±21.2 | 16.5–20.5 | −2.2 | 0.003* |
| AHI REM (events/h) | 17.0±21.2 | 15.1–19.0 | 21.6±23.3 | 19.4–23.8 | −4.6 | <0.001* |
| Sleep time supine (min) | 51.8±32.6 | 48.8–54.9 | 51.8±31.7 | 48.8–54.8 | 0.0 | 0.961 |
| RDI supine (events/h) | 20.4±32.8 | 17.3–23.5 | 21.5±24.5 | 19.2–23.8 | −1.1 | 0.424 |
| Minimum SpO2 (%) | 83.7±14.0 | 82.4–85.1 | 84.4±10.9 | 83.4–85.4 | −0.7 | 0.296 |
Significance is P<0.005. Paired samples t-tests were applied and statistically significant P are presented in bold and marked with an asterisk. RDI=Respiratory disturbance index, AHI=Apnea–hypopnea index, REM=Rapid eye movement, TST=Total sleep time, SD=Standard deviation, CI=Confidence interval.
Agreement between apnea-hypopnea index, apnea-hypopnea index-rapid eye movement, and respiratory disturbance index supine recorded through automated and manual scoring systems
The agreement between AHI, AHI-REM, and RDI supine events recorded through automated and manual scoring systems was studied by applying kappa statistics. A moderate degree of agreement (κ = 0.510) was found between AHI events recorded through automated and manual scoring systems (P < 0.001). A fair degree of agreement (κ = 0.308) was recorded for AHI-REM (P < 0.001) and a moderate degree of agreement (κ = 0.474) for RDI supine events was recorded (P < 0.001). Table 3 represents agreement between various sleep parameters recorded through automated and manual scoring systems.
Table 3.
Agreement between sleep parameters recorded through automated and manual scoring (n=442).
| Sleep parameter | Automated scoring, n (%) | Manual scoring, n (%) | kappa (k) | P |
|---|---|---|---|---|
| AHI (events/h) | ||||
| <5 | 141 (32.4) | 138 (31.5) | 0.510 | <0.001* |
| 5–15 | 153 (35.2) | 135 (30.8) | ||
| 16–30 | 71 (16.3) | 76 (17.4) | ||
| >30 | 70 (16.1) | 89 (20.3) | ||
| AHI REM (events/h) | ||||
| <5 | 154 (35.5) | 135 (30.8) | 0.308 | <0.001* |
| 5–15 | 119 (27.4) | 97 (22.1) | ||
| 16–30 | 77 (17.7) | 82 (18.7) | ||
| >30 | 84 (19.4) | 124 (28.3) | ||
| RDI supine (events/h) | ||||
| <5 | 136 (31.5) | 141 (32.3) | 0.474 | <0.001* |
| 5–15 | 132 (30.6) | 111 (25.5) | ||
| 16–30 | 70 (16.2) | 71 (16.3) | ||
| >30 | 94 (21.8) | 113 (25.9) |
Significance is P<0.005. Kappa statistics for agreement were applied and statistically significant P are presented in bold and marked with an asterisk. RDI=Respiratory disturbance index, AHI=Apnea–hypopnea index, REM=Rapid eye movement.
Intra-class correlation between sleep parameters recorded through automated and manual scoring systems
ICC was applied to assess the relationship between sleep parameters recorded through automated and manual scoring systems. A good agreement was observed between TST recorded through automated and manual systems (ICC = 0.727, P < 0.001). A poor or weak agreement was observed for REM latency (ICC = 0.204, P < 0.001), total REM duration (ICC = 0.313, P < 0.001), N1 time (ICC = 0.274, P < 0.001), N2 time (ICC = 0.464, P < 0.001), and N3 time (ICC = 0.428, P < 0.001). AHI events recorded through both systems showed good degree of agreement (ICC = 0.753, P < 0.001), whereas an excellent agreement was observed for TST supine (ICC = 0.902, P < 0.001). Table 4 depicts ICC of various sleep parameters recorded through automated and manual scoring systems.
Table 4.
Intra-class correlation between sleep parameters recorded through automated and manual scoring systems (n=442).
| Parameter | Automated versus manual | 95% CI for ICC | P |
|---|---|---|---|
| TST (min) | 0.727 | 0.641–0.789 | <0.001* |
| REM latency (min) | 0.204 | 0.081–0.316 | <0.001* |
| REM duration (min) | 0.313 | 0.226–0.394 | <0.001* |
| N1 time (min) | 0.274 | 0.035–0.459 | <0.001* |
| N2 time (min) | 0.464 | 0.150–0.652 | <0.001* |
| N3 time (min) | 0.428 | 0.231–0.572 | <0.001* |
| %TST REM | - 0.060 | −0.141–0.024 | 0.959 |
| %TST N1 | 0.088 | 0.000–0.176 | 0.021 |
| %TST N2 | 0.184 | 0.045–0.309 | <0.001* |
| %TST N3 | 0.309 | 0.075–0.486 | <0.001* |
| AHI (events/hour) | 0.753 | 0.709–0.791 | <0.001* |
| AHI REM (events/hour) | 0.510 | 0.432 – 0.759 | <0.001* |
| Sleep time supine (min) | 0.902 | 0.884–0.918 | <0.001 * |
| RDI supine (events/h) | 0.544 | 0.475–0.606 | <0.001* |
| Minimum SpO2 (%) | 0.441 | 0.362–0.513 | <0.001* |
Significanse is P<0.005. Statistically significant P are presented in bold and marked with an asterisk. RDI=Respiratory disturbance index, AHI=Apnea–hypopnea index, REM=Rapid eye movement, TST=Total sleep time, CI=Confidence interval, ICC=Intra-class correlation.
Bland–Altman analysis of sleep parameters recorded through automated and manual scoring systems
The agreement between various sleep parameters recorded through automated and manual scoring systems was also assessed using Bland–Altman plots. The mean ± SD bias in TST recorded through both systems was 27.5 ± 78.8 (LoA ranging from −181.88 to 126.83). The total REM duration showed a mean ± SD bias of 0.4 ± 4.2 (LoA ranging from −94.1 to 94.9). The mean ± bias of AHI was 2.2 ± 15.4 (LoA ranging from −32.4 to 28.0) and of AHI-REM was 4.55 ± 21.8 (LoA ranging from −47.3 to 38.2). Figure 1a-f shows Bland–Altman plots for the assessment of agreement between sleep parameters recorded through automated and manual scoring systems.
Figure 1.
Bland–Altman plots comparing (a) total sleep time, (b) duration of rapid eye movement (REM), (c) apnea–hypopnea index (AHI) events, (d) AHI-REM events, (e) RDI supine, and (f) minimum SpO2 concentration between manual and automated scoring system.
Discussion
The main findings from the present study are discussed here. The results show that the mean BMI was on the higher side, i.e., 32.5 ± 9.3 kg/m2 and more than half of the participants (56.1%) were categorized as obese (BMI > 30 kg/m2). Such a high level of obesity observed in this study does not necessarily represent a general trend of the population; in fact, it is linked with the inclusion criteria of the study where patients suspected of OSA who were referred for PSG were included. It is a well-known fact that obesity is a major risk factor for OSA along with diabetes mellitus, old age, tobacco and alcohol use, nasal congestion, and asthma.[25] Almost half of the patients of OSA are obese and up to 90% of individuals with severe obesity suffer with OSA.[26]
The TST was compared between automated and manual systems. A mean difference of 27.5 ± 78.8 min was observed. A good agreement was observed between TST recorded by both systems (ICC = 0.727, P < 0.001). Similar results were observed in a study by Zhang et al.[27] in which an automated system WatchPAT 200 was used. A mean difference of 15.8 min and a strong correlation of 0.70 was observed between automated and manual scoring system. Thus, the authors concluded that TST can be reliably assessed using automated systems in a time efficient and cost effective manner.[27] The mean ± SD bias of total REM duration in the present study was 0.4 ± 4.2 min and a poor agreement between automated and manual scoring was observed (ICC = 0.313, P < 0.001). A very high accuracy of automated scoring system in detecting REM sleep was observed in a study by Coorey et al.[28] in which automated systems were found to be 96% accurate. The authors concluded that automated systems can effectively be used as an alternative to manual scoring for detection of REM sleep behavior disorders.[28] To the contrary, a study conducted on Study of Infants’ Emergent Sleep TrAjectories (SIESTA) dataset showed that automated system had a suboptimal performance (κ = 0.47, accuracy = 70.5%) in patients with REM parasomnias. However, certain other sleep parameters such as wake time, N1, N2, and N3 times could reliably be assessed using automated system.[29]
REM latency showed a mean difference of 52.6 min between the two scoring systems and a poor degree of agreement was observed (ICC = 0.204, P < 0.001). In a study on the effectiveness of convolutional neural network based automated sleep analyzer, a moderate correlation (r = 0.461) was observed between automated and manual scoring.[30] However, these results differ from the results of the study by Choo et al.[17] where a strong agreement (κ = 0.863) was observed between automated and manual scoring systems.
In the present study, a poor or weak degree of agreement of automated scoring system with manual scoring was observed for N1 time (ICC = 0.274, P < 0.001), N2 time (ICC = 0.464, P < 0.001), and N3 time (ICC = 0.428, P < 0.001). Literature has shown mixed results with varying accuracies of automated scoring systems in detecting sleep stages. In a study conducted at Montreal Neurological Institute and Hospital, Montreal, Canada, a moderate degree of agreement (κ = 0.710) in detecting various stages of sleep was observed. When assessed separately, a strong sensitivity and specificity was observed for detection of N2 (85% and 76%) and N3 stages (87% and 94%) but a poor sensitivity and specificity in detection of N1 stage (21% and 30%) was observed.[31] Wu et al.[32] found that automated scoring system could detect N1, N2 and N3 stages with a low accuracy of 57% and showed a poor degree of agreement (κ = 0.39) with manual scoring. Contrary to these studies, SimpleSleepNet, a sleep staging model, was shown to outperform trained human scorers in detection of sleep stages.[33] Thus, there appears to be a variation in the usefulness of various automated systems in detecting sleep stages depending upon the AI models and other technical specifications of the automated system being used.
The agreement between AHI, AHI-REM, and RDI supine recordings of automated and manual scoring systems was studied. A moderate degree of agreement (κ = 0.510) was observed for AHI events recorded through both systems. ICC analysis showed good agreement (ICC = 0.753) between AHI events recorded through automated and manual systems. Studies have shown that automated systems can reliably detect AHI events, which is an important indicator for detection of OSA. A study by Nikkonen et al.[34] on the diagnostic utility of a recurrent neural network architecture based automated system showed that the system could reliably detect OSA (κ = 0.728). The AHI calculated through the automated system showed a very strong correlation (r = 0.985) with that of the manual system.[34] Kristiansen et al.[35] found an automated sleep scoring system, the Nox T3 portable sleep monitor, to be more than 90% accurate is detecting OSA. These studies point toward the usefulness of automated sleep scoring systems in detecting OSA.
A fair degree of agreement (κ = 0.308) was observed for AHI-REM and a moderate degree of agreement (κ = 0.474) was observed for RDI supine recorded through both systems. A Korean study on the utility of an automated scoring system based on Noxturnal software showed that although the system could reliably detect certain sleep parameter; nevertheless, it showed a significant difference in AHI and RDI recordings as compared to the manual scoring system.[16] The authors concluded that the accuracy of automated system in recording AHI and RDI events can be improved if a slight modification to the AI algorithms is made. Martinot et al. used Sunrise automated system to detect sleep disturbance through mandibular movements.[36] The system showed an area under curve of 0.75 in detection of RDIs, hence signifying the usefulness of automated systems in detection of sleep disturbances. It was proposed that automated systems can be used at home for reliable detection of OSA, particularly in symptomatic children with moderate-to-severe OSA.[36]
The results from the present study and similar other studies published in literature point out that there is a marked variation in the accuracy of various automated systems in detection of different sleep parameters. These variations exist across systems utilizing different AI algorithms as well as among different study cohorts and datasets utilized for the analysis.[37,38] Signal complexities, parameter definitions, and sensor limitations are some of the reasons as to why differences exist in accuracy of various parameters detection.[39] The present study utilized automated sleep scoring system developed by SOMNOmedics based on DOMINO software. A study by Mikkelsen et al.[40] on a similar automated system showed that automated system outperformed manual system in terms of sleep onset and sleep time assessment. Studies have also evaluated consumer-targeted wrist devices for automated sleep scoring to be similar in efficacy, if not superior, to the manual devices.[41,42] Thus, it is evident that automated sleep scoring systems can be used for sleep staging in a time efficient and cost effective manner. However, there are certain challenges regarding standardization protocols, unavailability of large scale datasets, data mismatch due to heterogeneity of datasets, device limitations linked to quality of sensors used for signal detection, and data privacy issues that are needed to be catered before wide scale implementation of automated sleep scoring systems.[43,44]
The authors of the present study have strived to limit the biases of the study by strictly following the methodology and keeping an adequate sample size. The adequacy of sample size is also reflected by the statistical significance achieved in most of the results of the study.[45] In addition, many of the previous studies were conducted on available datasets, which in contrast to the present study, may not be a reflection of real-world application of automated systems. Nevertheless, the authors acknowledge that the study is not without limitations. This is a single-center study and utilized a single automated sleep recording device, therefore, it may not be possible to generalize the results as the clinical expertise and the technical sophistication and specification of the sleep scoring systems may vary from center to center and from region to region. The quality of sleep signal capture of the automated system can also significantly affect the outcome of the study.[46,47] Therefore, a comparison of different devices may provide better insight into the utility of automated systems. As majority of the individuals undergoing sleep studies are cases of OSA, therefore, the study included only those participants who were referred for the workup of OSA. This perhaps implies that the results from this study cannot be generalized to patients with other conditions. Finally, the authors recommend further multi-centric studies involving different automated sleep scoring devices with more advanced and up-to-date AI algorithms to provide a better understanding of the utility of automated systems in the detection of sleep disorders.
Conclusion
Based on the results of this study, it may be concluded that there is a strong correlation between automated and manual scoring systems for detection of TST and a weak correlation exists for total REM duration, REM latency, and the duration of N1, N2, and N3 stages. A high degree of agreement is observed between number of AHI events recorded through automated and manual systems. A fair degree agreement is observed for number of AHI-REM events and a moderate degree of agreement for RDI supine. Although the utility of automated sleep scoring in detection of various sleep disorders is evident, yet there appears to be a marked variation in the accuracy of automated system in the assessment of various sleep parameters. Therefore, multi-centric studies involving different automated sleep scoring devices based on advance machine learning algorithms are recommended to provide deeper insights into the role of automated sleep scoring systems in the detection of various sleep disorders.
Ethical statement
Ethical approval for the study was taken from the Ethical Review Committee of King Abdulaziz University Hospital, Jeddah, Saudi Arabia (Reference No 357-24).
Data availability statement
Request for data may be directed to the corresponding author.
Patient consent
The written informed consent was obtained from all patients for this study.
Conflicts of interest
There are no conflicts of interest.
Acknowledgements
The authors have no acknowledgements to declare.
Funding Statement
The Deanship of Scientific Research at King Abdulaziz University, Jeddah, Saudi Arabia has funded this project, under grant no. (RG-19-140-42).
References
- 1.Robbins R, Quan S. Sleep health. NEJM Evid. 2024;3:EVIDra2300269. doi: 10.1056/EVIDra2300269. [DOI] [PubMed] [Google Scholar]
- 2.Burman DE, Muzumdar HI. Management of Sleep Disorders in Psychiatry. New York: 2020. Sleep architecture and physiology; pp. 12–22. [Google Scholar]
- 3.Madaeva I, Berdina O, Titova E, Kolesnikova L. Pattern of “sleep spindles” in obstructive sleep apnea patients. Eur Phys J Spec Top. 2024;233:699–703. [Google Scholar]
- 4.Markun LC, Sampat A. Clinician-focused overview and developments in polysomnography. Curr Sleep Med Rep. 2020;6:309–21. doi: 10.1007/s40675-020-00197-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Reis MJ, Vago EL, Frange C, Coelho FM. Objective assessment of sleep. In: Frange C, Coelho FM, editors. Sleep Medicine and Physical Therapy: A Comprehensive Guide for Practitioners. Switzerland: Springer, Cham; 2022. pp. 401–10. [Google Scholar]
- 6.Patel AK, Reddy V, Shumway KR, Araujo JF. StatPearls [Internet] StatPearls Publishing; 2024. Physiology, sleep stages. [Google Scholar]
- 7.Bukhari MA, Alghtani MA, Aljohani ZS, Qasem AA, Alhazmi IH. Diagnosis and treatment of sleep disorders: A brief review. IJMDC. 2021;5:364–9. [Google Scholar]
- 8.Zhang Y, Ren R, Yang L, Zhang H, Shi Y, Sanford LD, et al. Polysomnographic nighttime features of narcolepsy: A systematic review and meta-analysis. Sleep Med Rev. 2021;58:101488. doi: 10.1016/j.smrv.2021.101488. [DOI] [PubMed] [Google Scholar]
- 9.Krahn LE, Arand DL, Avidan AY, Davila DG, DeBassio WA, Ruoff CM, et al. Recommended protocols for the multiple sleep latency test and maintenance of wakefulness test in adults: Guidance from the American Academy of Sleep Medicine. J Clin Sleep Med. 2021;17:2489–98. doi: 10.5664/jcsm.9620. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Kapur VK, Auckley DH, Chowdhuri S, Kuhlmann DC, Mehra R, Ramar K, et al. Clinical practice guideline for diagnostic testing for adult obstructive sleep apnea: An American Academy of Sleep Medicine clinical practice guideline. J Clin Sleep Med. 2017;13:479–504. doi: 10.5664/jcsm.6506. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Alsolai H, Qureshi S, Iqbal SM, Vanichayobon S, Henesey LE, Lindley C, et al. A systematic review of literature on automated sleep scoring. IEEE Access. 2022;10:79419–43. [Google Scholar]
- 12.Pépin JL, Bailly S, Tamisier R. Big data in sleep apnoea: Opportunities and challenges. Respirology. 2020;25:486–94. doi: 10.1111/resp.13669. [DOI] [PubMed] [Google Scholar]
- 13.Korkalainen H, Nikkonen S, Kainulainen S, Dwivedi AK, Myllymaa S, Leppänen T, et al. Self-applied home sleep recordings: The future of sleep medicine. Sleep Med Clin. 2021;16:545–56. doi: 10.1016/j.jsmc.2021.07.003. [DOI] [PubMed] [Google Scholar]
- 14.Xu S, Faust O, Seoni S, Chakraborty S, Barua PD, Loh HW, et al. A review of automated sleep disorder detection. Comput Biol Med. 2022;150:106100. doi: 10.1016/j.compbiomed.2022.106100. [DOI] [PubMed] [Google Scholar]
- 15.Magalang UJ, Johns JN, Wood KA, Mindel JW, Lim DC, Bittencourt LR, et al. Home sleep apnea testing: Comparison of manual and automated scoring across international sleep centers. Sleep Breath. 2019;23:25–31. doi: 10.1007/s11325-018-1715-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Jeong SJ, Seong JY, Kim J, Park JM, Sung CM, Yang HC, et al. Performance of an automated polysomnography scoring using noxturnal program versus manual scoring. Korean J Otorhinolaryngol Head Neck Surg. 2020;64:169–75. [Google Scholar]
- 17.Choo BP, Mok Y, Oh HC, Patanaik A, Kishan K, Awasthi A, et al. Benchmarking performance of an automatic polysomnography scoring system in a population with suspected sleep disorders. Frontiers in Neurology. 2023;14:1123935. doi: 10.3389/fneur.2023.1123935. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Madhok B. Diagnosis and Classification of Obesity. Handbook of Metabolic and Bariatric Surgery. 2022:16–9. [Google Scholar]
- 19.Benevides AB, da Paz Floriano AS, Sarcinelli-Filho M, Bastos-Filho TF. Introduction to Non-Invasive EEG-Based Brain-Computer Interfaces for Assistive Technologies. CRC Press; 2020. Review of the human brain and EEG signals; pp. 1–49. [Google Scholar]
- 20.Marcuse LV, Fields MC, Yoo JJ. Elsevier Health Sciences; 2024. Rowan’s Primer of EEG E-Book. [Google Scholar]
- 21.Lüders HO, Noachtar S, Rémi J. Oxford University Press; 2024. Electroencephalography: Textbook and Atlas. [Google Scholar]
- 22.Malhotra A, Ayappa I, Ayas N, Collop N, Kirsch D, Mcardle N, et al. Metrics of sleep apnea severity: Beyond the apnea-hypopnea index. Sleep. 2021;44:zsab030. doi: 10.1093/sleep/zsab030. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Vanacore A, Pellegrino MS. Robustness of κ-type coefficients for clinical agreement. Stat Med. 2022;41:1986–2004. doi: 10.1002/sim.9341. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Schober P, Mascha EJ, Vetter TR. Statistics from A (agreement) to Z (z score): A guide to interpreting common measures of association, agreement, diagnostic accuracy, effect size, heterogeneity, and reliability in medical research. Anesth Analg. 2021;133:1633–41. doi: 10.1213/ANE.0000000000005773. [DOI] [PubMed] [Google Scholar]
- 25.Mitra AK, Bhuiyan AR, Jones EA. Association and risk factors for obstructive sleep apnea and cardiovascular diseases: A systematic review. Diseases. 2021;9:88. doi: 10.3390/diseases9040088. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Kuvat N, Tanriverdi H, Armutcu F. The relationship between obstructive sleep apnea syndrome and obesity: A new perspective on the pathogenesis in terms of organ crosstalk. Clin Respir J. 2020;14:595–604. doi: 10.1111/crj.13175. [DOI] [PubMed] [Google Scholar]
- 27.Zhang Z, Sowho M, Otvos T, Sperandio LS, East J, Sgambati F, et al. A comparison of automated and manual sleep staging and respiratory event recognition in a portable sleep diagnostic device with in-lab sleep study. J Clin Sleep Med. 2020;16:563–73. doi: 10.5664/jcsm.8278. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Cooray N, Andreotti F, Lo C, Symmonds M, Hu MT, De Vos M. Detection of REM sleep behaviour disorder by automated polysomnography analysis. Clin Neurophysiol. 2019;130:505–14. doi: 10.1016/j.clinph.2019.01.011. [DOI] [PubMed] [Google Scholar]
- 29.Fonseca P, van Gilst MM, Radha M, Ross M, Moreau A, Cerny A, et al. Automatic sleep staging using heart rate variability, body movements, and recurrent neural networks in a sleep disordered population. Sleep. 2020;43:zsaa048. doi: 10.1093/sleep/zsaa048. [DOI] [PubMed] [Google Scholar]
- 30.Cheng H, Yang Y, Shi J, Li Z, Feng Y, Wang X. Comparison of automated deep neural network against manual sleep stage scoring in clinical data. Comput Biol Med. 2024;179:108855. doi: 10.1016/j.compbiomed.2024.108855. [DOI] [PubMed] [Google Scholar]
- 31.von Ellenrieder N, Peter-Derex L, Gotman J, Frauscher B. SleepSEEG: automatic sleep scoring using intracranial EEG recordings only. J Neural Engineering. 2022;19:026057. doi: 10.1088/1741-2552/ac6829. [DOI] [PubMed] [Google Scholar]
- 32.Wu X, Yang J, Pan Y, Zhang X, Luo Y. Automatic sleep-stage scoring based on photoplethysmographic signals. Physiol Meas. 2020;41:065008. doi: 10.1088/1361-6579/ab921d. [DOI] [PubMed] [Google Scholar]
- 33.Guillot A, Sauvet F, During EH, Thorey V. Dreem open datasets: Multi-scored sleep datasets to compare human and automated sleep staging. IEEE Trans Neural Syst Rehabil Eng. 2020;28:1955–65. doi: 10.1109/TNSRE.2020.3011181. [DOI] [PubMed] [Google Scholar]
- 34.Nikkonen S, Korkalainen H, Leino A, Myllymaa S, Duce B, Leppanen T, et al. Automatic respiratory event scoring in obstructive sleep apnea using a long short-term memory neural network. IEEE J Biomed Health Inform. 2021;25:2917–27. doi: 10.1109/JBHI.2021.3064694. [DOI] [PubMed] [Google Scholar]
- 35.Kristiansen S, Traaen GM, Øverland B, Plagemann T, Gullestad L, Akre H, et al. Comparing manual and automatic scoring of sleep monitoring data from portable polygraphy. J Sleep Res. 2021;30:e13036. doi: 10.1111/jsr.13036. [DOI] [PubMed] [Google Scholar]
- 36.Martinot JB, Cuthbert V, Le-Dong NN, Coumans N, De Marneffe D, Letesson C, et al. Clinical validation of a mandibular movement signal based system for the diagnosis of pediatric sleep apnea. Pediatr Pulmonol. 2022;57:1904–13. doi: 10.1002/ppul.25320. [DOI] [PubMed] [Google Scholar]
- 37.Patterson MR, Nunes AA, Gerstel D, Pilkar R, Guthrie T, Neishabouri A, et al. 40 years of actigraphy in sleep medicine and current state of the art algorithms. NPJ Digit Med. 2023;6:51. doi: 10.1038/s41746-023-00802-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Alattar M, Govind A, Mainali S. Artificial intelligence models for the automation of standard diagnostics in sleep medicine-a systematic review. Bioengineering (Basel) 2024;11:206. doi: 10.3390/bioengineering11030206. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Yazdi M, Samaee M, Massicotte D. A review on automated sleep study. Ann Biomed Eng. 2024;52:1463–91. doi: 10.1007/s10439-024-03486-0. [DOI] [PubMed] [Google Scholar]
- 40.Mikkelsen KB, Ebajemito JK, Bonmati-Carrion MA, Santhi N, Revell VL, Atzori G, et al. Machine-learning-derived sleep-wake staging from around-the-ear electroencephalogram outperforms manual scoring and actigraphy. J Sleep Res. 2019;28:e12786. doi: 10.1111/jsr.12786. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Pesonen AK, Kuula L. The validity of a new consumer-targeted wrist device in sleep measurement: An overnight comparison against polysomnography in children and adolescents. J Clin Sleep Med. 2018;14:585–91. doi: 10.5664/jcsm.7050. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Haghayegh S, Khoshnevis S, Smolensky MH, Diller KR. Application of deep learning to improve sleep scoring of wrist actigraphy. Sleep Med. 2020;74:235–41. doi: 10.1016/j.sleep.2020.05.008. [DOI] [PubMed] [Google Scholar]
- 43.Chriskos P, Frantzidis CA, Nday CM, Gkivogkli PT, Bamidis PD, Kourtidou-Papadeli C. A review on current trends in automatic sleep staging through bio-signal recordings and future challenges. Sleep Med Rev. 2021;55:101377. doi: 10.1016/j.smrv.2020.101377. [DOI] [PubMed] [Google Scholar]
- 44.Phan H, Mikkelsen K. Automatic sleep staging of EEG signals: recent development, challenges, and future directions. Physiological Measurement. 2022;43:04TR01. doi: 10.1088/1361-6579/ac6049. [DOI] [PubMed] [Google Scholar]
- 45.Serdar CC, Cihan M, Yücel D, Serdar MA. Sample size, power and effect size revisited: simplified and practical approaches in pre-clinical, clinical and laboratory studies. Biochemia medica. 2021;31:27–53. doi: 10.11613/BM.2021.010502. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Imtiaz SA. A systematic review of sensing technologies for wearable sleep staging. Sensors (Basel) 2021;21:1562. doi: 10.3390/s21051562. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Kwon S, Kim H, Yeo WH. Recent advances in wearable sensors and portable electronics for sleep monitoring. iScience. 2021;24:102461. doi: 10.1016/j.isci.2021.102461. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
Request for data may be directed to the corresponding author.