Validation of the Withings Sleep Analyzer, an under-the-mattress device for the detection of moderate-severe sleep apnea syndrome

Abstract

Study Objectives:

To assess the diagnostic performance of a nonintrusive device placed under the mattress to detect sleep apnea syndrome.

Methods:

One hundred eighteen patients suspected to have obstructive sleep apnea syndrome completed a night at a sleep clinic with a simultaneous polysomnography (PSG) and recording with the Withings Sleep Analyzers. PSG nights were scored twice: first as simple polygraphy, then as PSG.

Results:

Average (standard deviation) apnea-hypopnea index from PSG was 31.2 events/h (25.0) and 32.8 events/h (29.9) according to the Withings Sleep Analyzers. The mean absolute error was 9.5 events/h. The sensitivity, specificity, and area under the receiver operating characteristic curve at thresholds of apnea-hypopnea index ≥ 15 events/h were, respectively, sensitivity (Se)₁₅ = 88.0%, specificity (Sp)₁₅ = 88.6%, and area under the receiver operating characteristic curve (AUROC) ₁₅ = 0.926. At the threshold of apnea-hypopnea index ≥ 30 events/h, results included Se₃₀ = 86.0%, Sp₃₀ = 91.2%, AUROC₃₀ = 0.954. The average total sleep time from PSG and the Withings Sleep Analyzers was 366.6 (61.2) and 392.4 (67.2) minutes, sleep efficiency was 82.5% (11.6) and 82.6% (11.6), and wake after sleep onset was 62.7 (48.0) and 45.2 (37.3) minutes, respectively.

Conclusions:

Withings Sleep Analyzers accurately detect moderate-severe sleep apnea syndrome in patients suspected of sleep apnea syndrome. This simple and automated approach could be of great clinical value given the high prevalence of sleep apnea syndrome in the general population.

Clinical Trial Registration:

Registry: ClinicalTrials.gov; Name: Validation of Withings Sleep for the Detection of Sleep Apnea Syndrome; URL: https://clinicaltrials.gov/ct2/show/NCT04234828; Identifier: NCT04234828.

Citation:

Edouard P, Campo D, Bartet P, et al. Validation of the Withings Sleep Analyzer, an under-the-mattress device for the detection of moderate-severe sleep apnea syndrome. J Clin Sleep Med. 2021;17(6):1217–1227.

BRIEF SUMMARY

Current Knowledge/Study Rationale: New connected devices may help to reduce barriers in diagnosing obstructive sleep apnea syndrome. Therefore, we evaluated the Withings Sleep Analyzer, a nonintrusive unattended mattress device.

Study Impact: Compared to polysomnography, the automated analysis of the Withings Sleep Analyzer showed good accuracy for apnea-hypopnea index measurement and diagnostic performances similar to type III home sleep apnea testing at a lower cost. We think that the Withings Sleep Analyzer may play a role in the home diagnosis of moderate-severe obstructive sleep apnea syndrome.

INTRODUCTION

Obstructive sleep apnea syndrome (OSAS) is characterized by a cessation (apnea) or a decrease (hypopnea) in breathing during sleep. Episodes of apnea and hypopnea result in intermittent hypoxia and/or sleep fragmentation. Untreated OSAS is associated with a significant decrease in quality of life and cognitive performance, as well as an increased risk of accidents caused by daytime sleepiness. OSAS also leads to cardiovascular and metabolic complications.^1,2

Obstructive sleep apnea (OSA) is the most prevalent mechanism of sleep apnea syndrome (SAS). OSAS affects a large and growing number of people, particularly in developed countries, because obesity and age are the main risk factors. It is estimated that between 4% and 8% of men and 2% and 6% of women have OSAS, although estimates vary by an order of magnitude depending on the definitions retained and equipment used.^1,3–9 It is also estimated that around 80% of patients are undiagnosed,^5,10–12 which is in part explained by expensive and time-consuming diagnostic means.

Diagnosis of the syndrome is made by overnight polygraphy (PG) or polysomnography (PSG). Testing can be performed in a sleep laboratory or at home and consists in recording several physiological signals such as electrocardiogram, respiratory movements, airflow, electroencephalogram, electromyogram, and electrooculogram. However, the presence of these numerous sensors may deteriorate the quality of sleep.¹³

Wearables and other consumer devices offer new perspectives for disease management, from mass screening to fine-grain phenotyping, from prevention and early detection to monitoring of treated patients, and for the knowledge of the natural history of a disease.¹⁴ But sufficiently powered clinical studies are needed to assess their sensitivity and specificity against the gold-standard PSG.¹⁵

The Withings Sleep Analyzer (WSA) is a new concept of nonwearable devices that can be placed under mattresses in a nonintrusive manner and unattended by a technician. The WSA is powered by a deep-learning algorithm that uses body movement, breathing patterns, cardiac activity, and snoring to estimate an apnea-hypopnea index (AHI). In this study, we evaluated the diagnostic performance of the WSA compared to the gold-standard PSG manually scored by certified specialists in a population of patients with suspected OSAS referred to a PSG test in a sleep laboratory.

METHODS

Study design

This prospective multicentric study with blind analysis was conducted at Antoine-Béclère Hospital, Clamart, France, and Saint-Pierre University Hospital, Brussels, Belgium. The study was approved by the French review board Sud Méditerranée I and by the Saint-Pierre University Hospital Review Board. Withings, the sponsor, provided the sleep analyzers (NCT04234828).

Study participants

Men and women between ages 18 and 70 years who had been referred for a PSG test were consecutively offered the opportunity to participate. Patients were included if they were suspected to have OSAS based on regular snoring, possibly witnessed apnea, and nonrefreshing sleep with various degrees of daytime sleepiness. Patients screened preoperatively for bariatric surgery were also enrolled at the French site. Noninclusion criteria were patients under positive airway pressure therapy, children younger than age 18 years, and persons with linguistic or psychological incapacity to sign a written informed consent form. Stable and mild respiratory, cardiovascular, or metabolic comorbidities were not exclusion criteria. The eligible patients were informed and freely gave their signed consent prior to the study. Patients filled in a STOP-BANG questionnaire.¹⁶

The 4 primary endpoints of this study were the sensitivity and specificity in detecting an AHI ≥ 15 events/h and an AHI ≥ 30 events/h. With an expected specificity and sensitivity of 90%, application of the binomial law required a sample size of at least 132 patients to obtain a 10% precision on all 4 primary endpoints with a 1-sided type I error of 5%, assuming an equal distribution of patients in the 3 categories of suspected OSAS (none or mild, moderate, and severe) and a 20% dropout rate.

Data collection

Participants spent 1 night at the clinic. The recordings of PSG and the WSA were simultaneous. The PSG monitors (French center: Smart PSG, CIDELEC, Sainte Gemmes sur Loire, France; Belgian center: Brain Net, MEDATEC, 1440 Braine-le-Château, Belgium) were installed according to the manufacturer’s specifications. The WSA was placed under the mattress, at the level of the torso according to the manufacturer’s instructions. The PSG was monitored by a sleep technician during the entire night. The WSA required no attendance during the night.

Raw data from the WSA and the predicted AHI were automatically uploaded at the end of the night through Wi-Fi and stored on Withings’s secured servers. PSG raw data were stored on the hospital’s secured servers before proceeding to manual scoring. Failure to upload PSG or WSA data led to exclusion from the full analysis set population.

Data processing

Recordings with the PSG and the WSA were simultaneous, but because each device had its own internal clock, a synchronization method was necessary. First, a common time origin was defined by setting the starting time of the PSG recording equal to the time where presence in bed was detected by the WSA (see WSA section). Then the differential of clock speed was compensated by superposing common remarkable features on the PSG and the WSA data (such as motion and pronounced respiration effort). This permitted the correction of the relative drift and offset of the clocks. Failure to synchronize PSG and WSA data led to exclusion from the full analysis set population.

PSG acquisition and scoring

Signals recorded by PSG included 3 electroencephalogram channels, 2 electrooculograms, 1 chin electromyogram, bilateral anterior tibial electromyograms, 1 lead electrocardiogram, pulse oximetry, airflow (nasal cannula), tracheal sounds and suprasternal pressure (from the French Center), thermistance (from the Belgian center), snoring (microphone), respiratory effort (thoracic and abdominal inductance plethysmography signals), and body position sensor.

PSG recordings were manually scored twice, first as PG (no electroencephalogram scoring) and then as PSG, by certified sleep physicians who were blinded to the WSA signals and its AHI prediction, and according to American Academy of Sleep Medicine criteria, including a hypopnea definition with a reduction of flow above 30% of baseline and a 3% desaturation or an arousal.¹⁷ To reduce the interindividual and intercenter scoring bias, the scorers were trained together during 1 session (intraclass correlation coefficient was 0.980 [95% confidence interval (CI), 0.89–1.0] for the scoring of AHI).

WSA

The WSA is a medical device consisting of a hardware piece, the Withings Sleep, and software that estimates AHI.

Withings Sleep (Figure 1A) consists of a thermoplastic polyurethane pad inflated with air and is connected to a pressure sensor. The unit is covered with a protective sleeve and is powered by a 5V 1A power supply. A USB power adapter connected to the main socket is included with the unit. Inside the sleeve (Figure 1B) is the air bladder, connected to a case protecting the electronic components of the device.

Figure 1. WSA hardware.

Dimensions (A) and components (B). WSA = Withings Sleep Analyzer.

The device, previously described, is positioned under the mattress, beneath the patient’s torso.¹⁸ In brief, the device uses a sensor that measures pressure in the air bladder relative to the atmospheric pressure. The pressure signal is filtered and amplified to isolate 3 separate mechanical sources: body movements, displacement of the chest (breathing), and vibrations due to cardiac ejection (ballistocardiography). They are transmitted by the mattress to the air bladder and are recorded as pressure variations. The device also includes a microphone, also placed under the mattress (Figure 1B).

The pressure and sound signals are analyzed by WSA embedded software. Filtered in different frequency bands, the pressure signal provides data on the following physiological functions: breathing, heart rate, and movement. The audio signal is down-sampled and filtered to extract a signature of snores.

The automatic detection of the sleep periods is performed in 2 steps. An algorithm first detects the presence or absence of a user on the bed, followed by a second algorithm that tags every minute spent in bed as either awake or asleep. Both algorithms use a random forest classifier, which takes as input features extracted from the body movement, respiration, and heart activity channels and takes as outputs the state of the user (in bed/out of bed and awake/asleep, respectively). Total sleep time (TST) provided by the WSA is defined as the total number of minutes when a user is detected in the bed and also tagged as asleep. Sleep efficiency (SE) is defined as TST divided by the total number of minutes when a user is detected in bed.

The algorithm that counts the number of apnea and hypopnea events during sleep uses features built from the physiological phenomena observed during an apneic or hypopneic event.^19,20 From the pressure signal, breathing effort, actigraphy, amplitude of the respiratory sinus arrhythmia, and heart rate are extracted. From the down-sampled sound signal, the amplitude of snoring that varies periodically during repeated respiratory events is extracted, as well as snorting sounds, which sometimes happen at the end of respiratory events.

A classic deep learning architecture with a 1-dimensional convolutional neural network is used to output the density of apnea and hypopnea events in a 5-minute time window based on the extracted features.²¹ The recordings of 104 patients (41 men and 63 women, aged 47 [11.7] years, body mass index 30.6 [4.7] kg/m²] from a preliminary study in the French laboratory were used to train the algorithm. An 8-fold cross-validation was used for hyperparameter tuning. Running the neural network continuously during sleep yields the density distribution of apnea and hypopnea events, which can be summed to obtain the count of apnea and hypopnea events during sleep.

AHI is obtained by dividing the count of apnea and hypopnea events during sleep by the estimated TST. This AHI is finally transmitted by Wi-Fi to Withings’ secured servers when the user leaves the bed.

The AHI estimate is provided only if sleeping time is more than 5 hours or if it is less than 5 hours but with more than 5 hours of presence in bed and an AHI ≥ 30 events/h, in which case the AHI is estimated with the time in bed (TIB) instead of time asleep. This is to account for the often poor sleep quality and/or duration of people who are severely apneic. Failure to meet this criterion led to exclusion from the per-protocol population.

According to the Sleep Cardiovascular Oximetry Position Effort and Respiratory Classification (SCOPER) categorization, the WSA classifies as S3 (sleep surrogate) C5 (other cardiac measure) O0 (no oximetrer) P0 (no position) E4 (other effort measure) R5 (other respiratory measure) as it has no measurement of oximetry or body position.²² The sleep apnea detection feature of the WSA has received the CE (European Conformity) marking according to Medical Device Directive 93/42/EEC as amended by the 2007/47/EC in March 2020. The EC certificate number is ECM19MDD008.

Statistical analysis

The statistical analysis was conducted with the open source software Python (Python Software Foundation, Delaware, United States) and the library Scikit-learn (https://scikit-learn.org/stable/), on frozen databases, after review of the data to identify deviations from the protocol.

All statistical tests were 2-sided, with a statistical significance threshold at 0.05. Baseline characteristics are described with the mean and standard deviation for continuous variables and the numbers and percentages for categorical variables. A Shapiro-Wilk test and examination of QQ-plots were used to test for normality. The statistical significance of the differences between the investigation centers was determined with the following tests: t test for sample means when the distribution was normal, Mann-Whitney U test for sample medians otherwise, and Fisher exact test for proportions. To test differences between diagnostic methods on the pooled dataset, we used a t test for sample means when normality could be assumed and a Wilcoxon signed-rank test for medians when normality was not verified.

The performance of SAS detection by WSA compared to PSG was evaluated using the sensitivity and the specificity at the AHI threshold values AHI ≥ 5 events/h, AHI ≥ 15 events/h, and AHI ≥ 30 events/h; the positive and negative likelihood ratios L_+/−; the area under the receiver operating characteristic curve; a Bland-Altman analysis²³; the mean absolute error (MAE); and the root mean squared error (RMSE). The 95% CIs were calculated with the following methods: the Clopper-Pearson exact intervals for sensitivity and specificity (which were assumed to follow binomial distributions), the bootstrap method for the area under the curve, and the log method for the likelihood ratios. The TST, SE, and wake after sleep onset (WASO) estimated by the WSA were compared to the PSG using the bias, the MAE, and RMSE.

We performed several subgroup analyses to assess how the precision (MAE) and accuracy (bias) of the AHI estimate were affected by the following factors: the position (decubitus dorsalis or latero-ventral), the apnea mechanism (central or obstructive), the prevalence of hypopnea events, and sex. Because of the small size of the subgroups, sensitivity and specificity could not be estimated with sufficient precision. In addition, the distribution of AHI values was not normal and a transformed variable with a normal distribution was used to perform linear regression. Hence, only the biases were compared in these subgroups.

RESULTS

Population

One hundred sixty-two participants were included in the study and provided signed informed consent, 137 participants were included in the full analysis set population (population for whom the PSG and WSA data were seamlessly obtained), and 118 were in the per-protocol population (population for whom the PSG and WSA data were obtained and the WSA provided an AHI).

Exclusions from the full analysis set population were caused by issues with the synchronization within the PSG signals (10 participants), corrupted PSG files (4 participants), missing PSG report (1 participant), the WSA falling from the bed during the night (1 participant), and data loss during the wireless transmission of WSA (9 participants). This data loss occurred because raw data at 250 specificity were uploaded to be able to synchronize the clocks of the WSA and PSG in postprocessing (see “Methods” section). This situation does not reflect real-use conditions where raw sensor data are not transmitted but processed in real-time by the embedded algorithm.

The WSA provides an AHI only for participants with more than 5 hours of sleep detected by WSA or with an AHI ≥ 30 events/h and more than 5 hours of presence in bed (see “Methods” section). The 19 exclusions from the per-protocol population were nights that did not meet those requirements.

The baseline characteristics of the per-protocol population are given in Table 1. Only patients with mild and stable comorbidities were included. As is usual in OSAS populations, a significant proportion of patients had treated hypertension, diabetes, dyslipidemia, and/or depression. The 3 patients with neurological concerns were patients with epilepsy.

Table 1.

Baseline characteristics of the PP population.

Characteristic	Pooled Centers (n = 118)	French Center (n = 85	Belgian Center (n = 33)	French vs Belgian P
Female	67 (56.8%)	51 (60.0%)	16 (48.5%)	.303
Age (y)	49.3 (±12.1)	49.4 (±12.2)	49.2 (±12.1)	.939
BMI (kg/m2)	33.0 (±8.2)	34.1 (±8.4)	30.4 (±7.0)	.017
SBP (mm Hg)	129.1 (±16.1)	129.9 (±16.1)	127.3 (±16.0)	.434
DBP (mm Hg)	77.0 (±11.0)	76.5 (±10.7)	78.2 (±11.9)	.460
Neck size (cm)	39.5 (±4.2)	39.5 (±4.0)	39.6 (±4.7)	.888
AHI by PSG (events/h)	31.2 (±25.0)	32.8 (±23.6)	27.2 (±27.8)	.323
AHI < 5	12 (10.1%)	6 (7.1%)	6 (18.2%)	.092
5 ≤ AHI < 15	23 (19.4%)	13 (15.2%)	10 (30.3%)	.075
15 ≤ AHI < 30	33 (28.0%)	28 (32.9%)	5 (15.2%)	.068
AHI ≥ 30	50 (42.4%)	38 (44.7%)	12 (36.4%)	.534
Central*	22 (18.6%)	17 (20.6%)	5 (15.2%)	.610
AI > HI	16 (13.6%)	12 (14.1%)	4 (12.1%)	> .99
SE (%)	82.5 (±11.6)	85.8 (±9.8)	74.0 (±11.7)	< .001
TST (min)	366.6 (±61.4)	364.9 (±61.3)	370.8 (±62.7)	.643
WASO (min)	62.7 (±48.2)	59.4 (±45.8)	70.9 (±53.8)	.282
STOP-BANG	3.47 (±1.67)	3.58 (±1.73)	3.18 (±1.47)	.218
0–2	12 (36.4%)	22 (25.9%)	34 (28.8%)	.2669
3–4	14 (42.4%)	39 (45.9%)	53 (44.9%)	.837
5–6	6 (18.2%)	19 (22.4%)	25 (21.2%)	.802
7–8	1 (3.0%)	5 (5.9%)	6 (5.1%)	> .99
Comorbidities
Hypertension	30 (25.4%)	20 (23.5%)	10 (30.3%)	.66
Epilepsy	3 (2.5%)	3 (3.5%)	0 (0.0%)	> .99
Digestive disease	18 (15.3%)	17 (20.0%)	1 (3.0%)	> .99
Metabolic disease	29 (24.6%)	20 (23.5%)	9 (27.2%)	.083
Psychiatric disease	32 (27.1%)	19 (22.4%)	13 (39.4%)	.018
Other	12 (10.2%)	6 (7.1%)	6 (18.2%)	.363

Data presented as mean (SD) or n (%). *Central AHI > 5 events/h and more than 50% of the events are central. AHI = apnea-hypopnea index, AI = apnea index, BMI = body mass index, CVD = cardiovascular diseases, DBP = diastolic blood pressure, HI = hypopnea index, PP = per protocol, PSG = polysomnography, SBP = systolic blood pressure, SD = standard deviation, SE = sleep efficiency, STOP-BANG = screening test,¹⁴ TST = total sleep time, WASO = wake after sleep onset.

WSA vs PSG: respiratory events

Since the AHI was not distributed normally, a Wilcoxon rank sum test was performed on the median values of the AHI from PSG and from the WSA, with a nonstatistically significant result (P = .273) (Table 2). The Bland-Altman analysis and correlation plots are shown on Figure 2, and the receiver operating characteristic curves are in Figure 3. There was a residual tendency: the Spearman correlation coefficient was r = .40 (P < .001).

Table 2.

AHI by PSG, PG, and WSA.

	Mean	Median	STD	Bias With PSG	MAE	RMSE
AHI PSG	31.2	24.4	25.0	—	—	—
AHI WSA	32.8	21.3	29.9	1.6	9.5	12.5
AHI PG	21.4	13.2	23.6	–9.9	11.3	16.2

AHI = apnea-hypopnea index, MAE = mean absolute error, PG = polygraphy, PSG = polysomnography, RMSE = root mean squared error, STD = standard deviation, WSA = Withings Sleep Analyzer.

Figure 2. AHI from PSG and WSA.

Correlation (A) and Bland-Altman analysis (B) of AHI from PSG and WSA. AHI = apnea-hypopnea index, LoA = limits of agreement, PSG = polysomnography, WSA = Withings Sleep Analyzer.

Figure 3. ROC curves of WSA compared to PSG at an AHI threshold of 15 and 30 events/h.

AHI = apnea-hypopnea index, PSG = polysomnography, ROC = receiver operating characteristic, WSA = Withings Sleep Analyzer.

The confusion matrix (Table 3) shows that the vast majority of misclassifications were off by 1 class (90%), and the rest were off by 2 classes (10%). There was no misclassification between the no-apnea class and the severe apnea class. Metrics describing the performances of the WSA against PSG are in Table 4.

Table 3.

Confusion matrix for 4 classes.

		Reference PSG
		None	Mild	Moderate	Severe	Total
WSA	None	9	9	3	0	21
	Mild	2	11	7	0	20
	Moderate	1	3	17	7	28
	Severe	0	0	6	43	49
	Total	12	23	33	50	118
PG	None	12	13	2	2	29
	Mild	0	9	23	3	35
	Moderate	0	1	8	16	25
	Severe	0	0	0	29	29
	Total	12	23	33	50	118

None: AHI < 5 events/h; mild: 5 events/h ≤ AHI < 15 events/h; moderate: 15 events/h ≤ AHI < 30 events/h; severe: AHI ≥ 30 events/h. AHI = apnea-hypopnea index, PG = polygraphy, PSG = polysomnography, WSA = Withings Sleep Analyzer.

Table 4.

Summary of the diagnostic performance of WSA and PG compared to PSG.

	WSA	PG
Bias (h−1)	1.6 (–0.7 to 4.1)	–9.9 (–12.1 to –7.5)
MAE (h−1)	9.5	11.3
RMSE (h−1)	12.5	16.2
Se5 (%)	88.7 (81.1–94.0)	84.0 (75.6–90.4)
Sp5 (%)	75.0 (42.8–94.5)	100.0 (73.5–100.0)
Se15 (%)	88.0 (79.0–94.1)	63.9 (52.6–74.1)
Sp15 (%)	88.6 (73.3–96.8)	97.1 (85.1–99.9)
L+15	7.7 (5.0–11.9)	22.4 (3.3–152.0)
L–15	0.136 (0.11–0.16)	0.37 (0.36–0.39)
AUC15	0.926 (0.873–0.968)	0.958 (0.920–0.985)
Se30 (%)	86.0 (73.3–94.2)	58.0 (43.2–71.8)
Sp30 (%)	91.2 (81.8–96.7)	100 (94.7–100)
L+30	9.75 (7.2–13.2)	$+\infty$ (1.57 to +∞)
L–30	0.154 (0.12–0.20)	0.42 (0.40–0.44)
AUC30	0.954 (0.916–0.982)	0.950 (0.904–0.985)

Items in parentheses are the 95% confidence intervals. AUC = area under the curve, L+ = positive likelihood ratio, L– = negative likelihood ratio, MAE = mean absolute error, PG = polygraphy, PSG = polysomnography, RMSE = root mean squared error, Se = sensitivity, Sp = specificity, WSA = Withings Sleep Analyzer.

PG vs PSG: respiratory events

A Wilcoxon rank sum test for the difference between median AHI of the PSG and of the PG (Table 2) gave a statistically significant result (P < 10⁻⁶). Indeed, with a bias of –9.9 hours⁻¹, PG significantly underestimated AHI. The Bland-Altman and correlation plots are in Figure 4, and the receiver operating characteristic curves are in Figure 5. There was no statistically significant residual tendency: the Spearman correlation coefficient was r = .11 (P = .22).

Figure 4. AHI from PSG and PG.

Correlation (A) and Bland-Altman analysis (B) of AHI from PSG and PG. AHI = apnea-hypopnea index, LOA = limits of Aareement, PG = polygraphy, PSG = polysomnography.

Figure 5. ROC curves of PG compared to PSG at an AHI threshold of 15 and 30 events/h.

AHI = apnea-hypopnea index, PG = polygraphy, PSG = polysomnography, ROC = receiver operating characteristic.

Metrics describing the performances of PG against PSG are in Table 4, and the confusion matrix is in Table 3. The vast majority of misclassifications (88.3%) were off by 1 class, 5 were off by 2 classes (8.3%), and 2 were misclassifications of patients with severe apnea in the no-apnea class (3.3%).

Sleep analysis

Table 5 summarizes the mean values and standard deviations of the TST, SE, and WASO from PSG and WSA. SE and WASO could not be calculated from PG, and TIB was used in place of TST to calculate the AHI. SE was defined for both PSG and the WSA as the TST divided by the TIB.

Table 5.

Statistics of sleep quality indices of PSG and WSA.

	TST (min), Mean (SD)	SE (%), Mean (SD)	WASO (min), Mean (SD)
PSG	366.6 (61.2)*	82.5 (11.6)	62.7 (48.0)***
WSA	392.3 (67.2)*	82.6 (11.6)	45.2 (37.3)***
PG	TIB = 447.4 (62.4)	—	—

*P < .05, ***P < .001 (comparison PSG vs WSA). PG = polygraphy, PSG = polysomnography, SD = standard deviation, SE = sleep efficiency, TIB = time in bed, TST = total sleep time, WASO = wake time after sleep onset, WSA = Withings Sleep Analyzer.

The Wilcoxon rank sum test between median values from PSG and the WSA was statistically significant for the 3 indices. The WSA tended to overestimate TST by 25.8 minutes, which represented 7.0% of the mean duration on the dataset. The precision was acceptable, with an MAE = 53.0 minutes (14.4%) and an RMSE = 74.2 minutes (20.2%). In contrast, TIB given by PG exceeded TST by 80 minutes on average, with a large dispersion of errors (more than 1 hour). This accounted for a large part of the bias on the AHI. The WSA estimate of SE was identical to the PSG value (82.6% vs 82.5%, respectively). Finally, the WSA underestimated WASO by 17.5 minutes, which represented 27% of the mean WASO. Accordingly, the precision, given by an MAE = 36.2 minutes and an RMSE = 53.2 minutes, was poor.

Influence of position, hypopneas, apnea mechanism, and sex

First, we analyzed the influence of the position determined by the PSG by grouping the patients with ≥ 50% of the TST in supine position (“supine” group) vs patients with < 50% of the TST in supine position (“latero-ventral” group)—see Table 6. As expected, the mean AHI was significantly higher in the supine group (38.3 vs 27.2 events/h), but this had no influence on bias (0.9 and 2.0 events/h, respectively). The MAE and RMSE were smaller in the supine group (7.6 and 9.8 events/h) than in the latero-ventral group (10.6 and 13.8 events/h).

Table 6.

Summary of the 4 subgroup analyses of WSA.

Parameter	Groups	n	Mean AHI from PSG	WSA-PSG (bias)	MAE	RMSE
Position (> 50% of TST)	Supine	43	38.3	0.9	7.6	9.8
	Latero-ventral	75	27.2	2.0	10.6	13.8
Predominant type of events (> 50%)	Hypopnea	102	26.8	–0.1	8.9	11.7
	Apnea	16	59.6	12.2	13.7	17.0
Dominant apnea mechanism (> 50% of apneic events)	Obstructive	96	34.6	1.8	10.5	13.5
	Central	22	16.6	0.5	5.2	6.7
Sex	Male	51	32.7	4.8	9.8	12.7
	Female	67	30.1	–0.8	9.3	12.4

AHI = apnea-hypopnea index, MAE = mean absolute error, PSG = polysomnography, RMSE = root mean squared error, TST = total sleep time, WSA = Withings Sleep Analyzer.

Second, apnea and hypopnea events were not detected with the same accuracy (Table 6). With a bias of 12.2 events/h, the WSA significantly overestimated the AHI in case of an apnea predominance, ie, > 50% of apneic events compared to a predominance of hypopnea events (bias −0.1 events/h). This, however, had little consequence on the misclassification rate because most of these patients with predominant apneas (13/16) had a severe SAS, with the mean AHI = 59.6 events/h from PSG in this group. The MAE and RMSE were also larger in the apnea group (13.7 events/h and 17.0 events/h) than in the hypopnea group (8.9 events/h and 11.7 events/h).

The third analysis considered the influence of the apnea mechanism, labeled as central if central apnea events occurred more frequently (> 50% of events) than obstructive apnea events (Table 6). Central hypopnea events were not taken into account. Patients with central and mixed apneas were pooled into a single group called “central.” Biases of the subgroups were not clinically significantly different (0.5 vs 1.8 events/h). The precision was much higher in the central group with MAE and RMSE halved (5.2 and 6.7 events/h) compared to the obstructive group (10.5 and 13.5 events/h).

Finally, we analyzed the influence of the sex of patients (Table 6). The mean AHI from PSG was higher in the male group (32.7 vs 30.1 events/h), and the WSA overestimated AHI for men (bias 4.8 events/h) but not for women (−0.8 events/h). However, MAE and RMSE were similar for men (9.8 and 12.7 events/h) and for women (9.3 and 12.4 events/h).

DISCUSSION

This prospective multicentric study was aimed at evaluating the diagnostic performance of a new consumer device without oximetry, the WSA, for sleep apnea against simultaneously recorded single-night PSG in 2 sleep laboratories, with manual blinded scoring. Our findings show that the WSA has a close agreement with the PSG for the estimation of the AHI. Its performances (sensitivity (Se)₁₅ = 0.88 and Se₃₀ = 0.86) met the criteria of a sensitivity above 0.825 set for out-of-center testing to diagnose at least two-thirds of the population with sleep apnea.²² No patient with severe apnea was classified as normal or mild by the WSA, whereas 2 who were normal were classified as mild and 1 as moderate (Table 3). These results support the feasibility of a nonintrusive device placed under the mattress for detecting moderate-severe SAS without oximetry measurement.

Comparison of performances with type III home sleep apnea test

This device favorably compares for an AHI threshold of AHI ≥ 15 events/h with type III devices reviewed for the American Academy of Sleep Medicine clinical guidelines for sleep apnea diagnosis, where for the 6 devices studied on 457 participants, the range of sensitivities was 0.62–0.94 and the range of specificities was 0.25–0.97.²⁴

For an AHI ≥ 30 events/h the pooled sensitivity in 5 studies on 545 participants was 0.87 (0.77, 0.93), pooled specificity was 0.88 (0.59, 0.97), the positive likelihood ratio was 7.06 (1.88, 26.6), and the negative likelihood ratio was 0.14 (0.08, 0.25). Despite having access to an oximeter, the averaged performances of this type of device were similar or even lower to those obtained with the WSA. A possible explanation for this result could be the usage of deep learning methods in WSA, which are sometimes more efficient than the human eye at recognizing patterns in time series.

Additionally, in its present form, the human factor is removed from the interpretation of the WSA signals. This grants the device a truly unattended attribute and may overcome the AHI scoring variability among experienced scorers.²⁵ However, making the raw data of the WSA available would not suffice to manually score a night since specific training would be required to interpret these signals as physiologic events.

Effect of position and type and mechanism of events

Subgroup analyses provided some indications about the effect of the sleeping position, the type, and the mechanism of events on the device performance.

The first factor is the sleeping position, which had no influence on the bias, but errors were more dispersed in the latero-ventral positions then in the supine position. This could be due to less-frequent snoring in latero-ventral positions, causing a reduced relevance of the snoring detection channel. Another explanation could be a modification of the ballistocardiographic signals and/or respiratory patterns that may cause more errors in the AHI counts in the latero-ventral positions, since this process relies in part on pattern recognition in these signals.

The second effect tested is the type of the events. WSA has more difficulty identifying hypopneic events than apneic events because they are less pronounced on the respiration signal. The significant difference of mean error between the groups with a majority of apneas or a majority of hypopneas is the result of the training of the algorithm. Indeed, patients with a majority of apneic events were also a minority in the training set, and no compensating weight was applied to correct for this imbalance. This explains 2 observations of this study: (1) an overestimation of apneic events and (2) the predicted AHI of patients presenting with many short hypopneic events are poorly diagnosed. As a result, some patients with mild or moderate apneas had a predicted AHI less than 5 events/h.

The third factor evaluated is the apnea mechanism. The results confirm the intuition that central apnea is more easily detected than obstructive apnea, because the absence of movement of the thorax in central apnea is more clearly seen on the respiration channel of the WSA than persistent respiratory effort during obstructive events.

Evaluation of sleep with the WSA

The WSA measured SE and TST with good accuracy and reasonable precision. WSA overestimated TST and underestimated WASO, because a quiet awakening is difficult to differentiate from sleep, as observed with research-grade actimeters.²⁶ Indeed, the WSA discriminates wake from sleep through an analysis of the movement, breathing rate, and cardiac rate. Nevertheless, this estimation of TST allows a better measurement of AHI than the TIB provided by the PG.

Comparison to PG

Using a 2-step scoring process, we were able to indirectly compare WSA to PG. Their area under the curve at AHI ≥ 15 events/h and AHI ≥ 30 events/h thresholds were comparable, but their operating points differed. PG is usually a highly specific technique but significantly underestimates the AHI, whereas the WSA balances sensitivity and specificity with a negligible bias. Numerous publications have shown that misclassification of disease severity is considerably higher in PG with potential adverse consequences on therapeutic management.^27,28

Comparison to other connected devices

The WSA is not the first bed/mattress device.²⁹ A recent meta-analysis found 6 studies conducted on bed/mattress devices either embedded in a mattress (SD-101 and Sonomat) or used under the bed (Emfit).³⁰ They share similarities with our device in terms of comparator, recruitment strategy, and demographic characteristics. All but 1 were monocentric studies. The pooled pretest probability was 0.61 at cutoff AHI ≥ 15 events/h (0.604 in our study) and 0.44 at cutoff AHI ≥ 30 events/h (0.424 in our study). Four of them showed a low risk of bias according to QUADAS-2 evaluation by the reviewers. The meta-analysis (n = 515) showed the following results: at cutoff AHI ≥ 15 events/h, sensitivity was 0.944 (95% CI, 0.886–0.973) and specificity was 0.845 (95% CI, 0.634–0.945), and at cutoff AHI ≥ 30 events/h, sensitivity was 0.917 (95% CI, 0.833–0.961) and specificity was 0.887 (95% CI, 0.909–0.935). For mild OSA screening, sensitivity varied from 0.77–1, at the expense of the specificity, which varied from 0.06–0.81.³¹ In conclusion, these devices are slightly more sensitive and slightly less specific than WSA.

In addition, Davidovich et al³² tested an algorithm on an EarlySense piezoelectric sensor and showed a sensitivity and specificity at a cutoff AHI ≥ 15 events/h of 0.88 and 0.89, respectively, but the authors did not provide details about the protocol. Finally, Huysmans et al³³ evaluated Emfit and found a sensitivity of 0.70 and a specificity of 0.72 at cutoff AHI ≥ 30 events/h (n = 83).

Potential uses of WSA

The WSA is a certified consumer device in North America and in Europe, and the sleep apnea detection feature is certified as medical in Europe, Australia, and New Zealand. Because the WSA is low cost and unattended, the device overcomes some limitations of PSG and PG.

One opportunity pertains to the simplicity with which WSA provides an accurate diagnostic test. Indeed, the intrinsic night-to-night variability of the AHI due to sleep posture, sleep quality, quantity of rapid eye movement sleep, and the effect of dietary and environmental factors is rarely monitored over more than 2 nights. Yet classifying a patient’s sleep based on a single night of data leads to large misclassification rates, even in symptomatic patients.^34,35 Because the WSA is contactless and fully automatic, the device could be a simpler way to access AHI, TST, and SE without disrupting sleep habits and thus provide a better selection of patients eligible for the diagnostic tests.

Another opportunity concerns care management. Under-the-mattress devices could expand access to the diagnosis for those who cannot readily take advantage of sleep laboratory services. Indeed, continuous positive airway pressure and alternative treatments for OSAS (positional therapy or mandibular advancement device) do not currently benefit from long-term monitoring of apnea (particularly continuous positive airway pressure when adherence is poor) and sleep disturbances. In this way, the WSA could be useful in providing objective data on the long-term efficacy of the treatments of sleep apnea. Indeed, the WSA could be useful in providing objective data in subsequent studies on the potentially remaining apneas when adherence is low and therefore the patient is off continuous positive airway pressure during part of the night.

Study and device limitations

First, the WSA does not measure blood oxygen saturation. Therefore the desaturation burden is not measurable. Because the actual respiratory flow is not measured, it is challenging to distinguish apnea from hypopnea events, which may limit this specific patient phenotyping. Furthermore, the WSA does not provide a detailed analysis of sleep stages nor can the device differentiate leg movements from overall body movements in an awakened state. Therefore the results of this study may not be generalizable to patients with fragmented sleep, insomnia, or other sleep-disturbing conditions. However, in the present work, despite the fact that patients exhibited important sleep disturbance (reflected by WASO), the measurements remained accurate. The results apply to a sleep laboratory population suspected to have sleep apnea with a pretest probability of 60%. Also, this is a European patient population and results may not be the same in different populations.

Second, this algorithm was neither trained nor tested on patients whose condition might modify the pattern of the cardiac or respiratory signals recognized by the algorithm. The population studied included patients suspected to have OSAS with some comorbidities, but no acute or severe chronic condition such as potential respiratory muscle weakness due to neuromuscular condition, awake hypoventilation, or suspicion of sleep-related hypoventilation, chronic opioid medication use, or a history of stroke that precluded the use of home sleep apnea testing per the American Academy of Sleep Medicine.³⁶ Specifically, atrial fibrillation and severe chronic heart failure cause beat-to-beat heart rate patterns that are not present in our training dataset.^37,38

Ten patients of our population were on beta-blockers for hypertension, but this sample size was too small for a meaningful subanalysis of the effect of heart rate variations on the diagnostic performance. Pulmonary diseases such as nocturnal asthma have a distinctive breathing pattern with prolonged expirations,³⁹ which the device may confuse with apnea or hypopnea breathing patterns, incorrectly increasing the AHI. Chronic hypoventilation observed in chronic obstructive pulmonary disease or neuromuscular diseases (Charcot’s disease) has a different breathing pattern than apneas, with prolonged shallow amplitude of breathing. Other conditions may also negatively affect the performance of the algorithm. These include disorders such as periodic limb movements in sleep and restless legs syndrome, which could create interfering signals, or nonrespiratory sleep disorders that result in fragmented sleep. Therefore the results of this study may not be generalizable to these patient populations.

Third, WSA uses a machine-learning algorithm, which replicates the scoring habits of the certified physicians who scored the calibration dataset. Scrupulous efforts were made to comply with American Academy of Sleep Medicine scoring rules, but a bias could exist toward the scoring habits of this particular group of scorers.

Finally, the study was conducted in a sleep laboratory, but the product is intended for home use.

Each of these limitations outlines a future research question.

CONCLUSIONS

The WSA is an under-the-mattress device that can accurately and automatically measure AHI. Compared to PSG and PG, the WSA has several advantages: it is nonintrusive and no technician is required for sensor placement and analysis. Furthermore, the WSA has the advantage over PG to accurately measure TST and SE, which should be of great value in the care management of patients with moderate-severe SAS.

DISCLOSURE STATEMENT

All authors have seen and approved this manuscript. Work for this study was performed at the Sleep Medicine Department, Antoine-Béclère Hospital, Clamart, France and Chest Service, Saint-Pierre University Hospital, Brussels, Belgium. The sleep analyzers used in this study were provided by Withings, Issy-les-Moulineaux, France. Paul Edouard, David Campo, Pierre Bartet, and Rui-Yi Yang are employees of Withings. Pierre Escourrou is a consultant for Withings. The remaining authors report no conflicts of interest.

ABBREVIATIONS

AHI

apnea-hypopnea index

CI

confidence interval

LoA

limits of agreement

MAE

mean absolute error

OSA

obstructive sleep apnea

OSAS

obstructive sleep apnea syndrome

PG

polygraphy

PSG

polysomnography

RMSE

root mean squared error

SAS

sleep apnea syndrome

SE

sleep efficiency

TIB

time in bed

TST

total sleep time

WASO

wake after sleep onset

WSA

Withings Sleep Analyzer