A longitudinal study of the accuracy of positive airway pressure therapy machine-detected apnea-hypopnea events

Abstract

Study Objectives:

During positive airway pressure (PAP) therapy for sleep apnea syndromes, the machine-detected respiratory event index (REI_FLOW) is an important method for clinicians to evaluate the beneficial effects of PAP. There are concerns about the accuracy of this detection, which also confounds a related question, How common and severe are residual events on PAP?

Methods:

Patients with obstructive sleep apnea who underwent a split-night polysomnography were recruited prospectively. Those treated with PAP and tracked by the EncoreAnywhere system (Philips Respironics, Murrysville, PA) were analyzed. Those who stopped PAP within 1 month were excluded from this analysis. Compliance, therapy data, and waveform data were analyzed. Machine-detected vs manually scored events were compared at the first, third, sixth, and 12th month from PAP initiation. Logistic regression was used to determine factors associated with a high REI_FLOW difference.

Results:

One hundred and seventy-nine patients with a mean age 59.06 ± 13.97 years, median body mass index of 33.60 (29.75–38.75) kg/m², and median baseline apnea-hypopnea index of 46.30 (31.50–65.90) events/h were included. The difference between the machine-detected REI_FLOW and manually scored REI_FLOW was 10.72 ± 8.43 events/h in the first month and remained stable for up to 12 months. Male sex and large leak ≥ 1.5% were more frequent in patients who had an REI_FLOW difference of ≥ 5 events/h of use. A titration arousal index ≥ 15 events/h of sleep, and higher ratio of unstable to stable breathing were also associated with an REI_FLOW difference ≥ 5 events/h of use.

Conclusions:

There is a substantial and sustained difference between manual and automated event estimates during PAP therapy, and some associated factors were identified.

Citation:

Ni Y-N, Thomas RJ. A longitudinal study of the accuracy of positive airway pressure therapy machine-detected apnea-hypopnea events. J Clin Sleep Med. 2022;18(4):1121–1134.

BRIEF SUMMARY

Current Knowledge/Study Rationale: The machine-detected respiratory event index (REI_FLOW) is an important method for clinicians to evaluate the beneficial effects of positive airway pressure. It is known that machine-detected REI_FLOW can underestimate the true residual REI_FLOW in obstructive sleep apnea patients on positive airway pressure. Different manufacturers have varying detection criteria, while the raw waveforms are not easily accessible in some devices, which adds to the uncertainty of the meaning of intermediate-level elevations of device-estimated residual respiratory abnormality.

Study Impact: The present study shows that differences between auto-detection and manual estimation of residual REI_FLOW can be substantial, and persist for up to 12 months, and possibly indefinitely. The factors associated with this difference (of ≥ 5 events/h of use) were male sex, large leak ≥ 1.5%, a titration arousal index ≥ 15 events/h of sleep, and higher ratio of unstable to stable breathing. The study also demonstrates that persistent respiratory instability during positive airway pressure therapy is common and deserves further study.

INTRODUCTION

The issue of residual apnea during use of positive airway pressure (PAP) therapy of obstructive sleep apnea (OSA) has generated controversy and uncertainty regarding prevalence, persistence, predictors, and clinical impact.^1–5 Residual events may occur due to high leak, unresponsive anatomical factors, inadequate pressure, or a nonanatomical factor such as high loop gain, low arousal threshold, etc.^6,7 PAP is regarded as the optimal therapy to maintain upper airway patency.⁸ However, up to 25% of the patients have a residual apnea-hypopnea index (AHI) of more than 10 events/h on PAP.³ The high loop gain endotype/phenotype is regarded as an important apnea contributor^9,10 and may also increase the risk of residual apnea.¹¹ A low arousal threshold is another endotype⁷ which can be associated with poor outcomes, including reduced adherence.¹² Accurate detection of such events is central to understanding the clinical implications of various endotypes/phenotypes, residual disease, and the efficacy of PAP treatment alone in general.

The evolution of PAP device technology has given us a range of useful information beyond adherence, including leak, residual apnea in general, and detection of central apnea and periodic breathing. The American Thoracic Society (ATS) has a formal statement on continuous positive airway pressure adherence tracking systems and possible clinical use.⁵ The usefulness of these guidelines has been evaluated.¹³ Patients from the prospective InterfaceVent study (NCT03013283, conducted in an adult cohort undergoing at least 3 months of continuous positive airway pressure [CPAP]) and eligible for the ATS algorithm usage were analyzed. The residual device apnea-hypopnea index (AHI_flow) and high large leak thresholds proposed in the ATS algorithm were evaluated for predicting adherence (ie, AHI_flow > 10 events/h, and manufacturer-specific leak thresholds). Adherence was defined according to the generally accepted algorithm (ie, CPAP [continuous PAP] use ≥ 4 h/night for at least 70% of days). With a CPAP treatment duration of 5.1 (2.2–7.8) years, a logistic regression analysis demonstrated no significant relationship between the ATS-proposed AHI_flow or high large leak thresholds and nonadherence. However, event detection inaccuracies if present can confound results of such assessments.

Event detection and therapy algorithms vary based on the manufacturer. Nevertheless, several studies have explored agreement between machine-detected respiratory events index (REI) and polysomnography detected AHI in the sleep lab.^1,14–19 However, these were all in-lab studies with manual titration, and it is hard to mimic real PAP usage in the home setting. In a previous cross-sectional retrospective report from our group, we noted limited agreement between machine-detected results and manual-scoring results at a single time point.²⁰ The factors leading to this inaccuracy have not been fully explored.²¹ The aims of this study were to prospectively assess accuracy of auto-detection during PAP therapy at multiple time points, determine the factors which could explain any notable discrepancies, and assess respiratory stability using the novel approach of characterizing breathing on PAP as stable or unstable. We hypothesized that nondetection would be common and in a clinically significant range, that it would be persistent during use, and that risk factors could be identified.

METHODS

Participants and recruitment

This is a prospective cohort study that included adult patients (> 18 years) who underwent split-night polysomnography (PSG) between January 2017 and March 2020 at the Beth Israel Deaconess Medical Center, Boston, MA, affiliated sleep laboratories. All split-night PSGs were performed at an American Academy of Sleep Medicine (AASM)–accredited sleep center, and standard scoring criteria for sleep stages and respiratory events was used. Institutional Review Board approval was obtained for review of clinical, device, and PSG data. Informed consent was waived.

Polysomnography

The following criteria were used to enroll participants: 1) Combined diagnostic and therapeutic tests (“split” nights). This approach was taken as the majority of straightforward patients with clinical OSA undergo home sleep studies and no laboratory titration. Thus, titration PSG metrics would be unavailable. 2) Rapid eye movement (REM) sleep of at least 5 minutes was a requirement on both sides of the data to enable assessment of REM vs non-REM dominance. 3) After an initial more open enrollment, difficulty in tracking down therapy data resulted in restriction to only patients seen in the Beth Israel Deaconess Medical Center sleep clinic. The montage included all the standard signals recorded according to AASM guidelines.²² Apneas, hypopneas (3% and/or arousal), and respiratory effort–related arousals were scored.

Sleep apnea treatment tracking

The patients who were treated with PAP after the split-night PSG and tracked in the EncoreAnywhere system were included in the database. This program is an online tracking system that provides waveforms uploaded by Philips Respironics devices (including the DreamStation Auto CPAP and DreamStation Auto Bilevel; Murrysville, PA). A key advantage of this system is the availability for viewing and download (as portable document format, pdf, only) the respiratory flow waveforms. Our center uses auto-CPAP or bilevel devices in all patients treated, regardless of having had a diagnostic polysomnogram only or a split-night study. The lack of readily available waveform data from other manufacturers precluded evaluation of such devices.

Data collection

Demographic data including age, sex, body mass index, self-reported sleepiness (Epworth Sleepiness Scale), sleep and medical history, and medications at the time of the study were prospectively collected. The respiratory disturbance index including respiratory effort–related arousals (RERAs), AHI 3% (AHI when hypopneas were associated with 3% oxygen desaturation and/or an arousal), central apnea-hypopnea index, and arousal index during the baseline part and titration components of the split-night were tabulated.

Machine data extraction

Machine-detected parameters were extracted from the online software interface, while manual scoring was performed after download of a pdf containing detailed high-resolution flow data (“waveforms”).²⁰ In this waveform data, each horizontal line is 6 minutes of data. Events are tagged automatically by the system, including closed and open airway apnea, hypopnea, vibratory snoring, RERAs, large leak, and periodic breathing. The system detects the respiratory events and calculates the “AHI” automatically based on fixed thresholds and then shows it in the system; this output cannot be revised. The auto-algorithm establishes a baseline of patient flow based on a moving flow signal window and detects an hypopnea as a 40% reduction in flow lasting at least 10 seconds, followed by a recovery breath. Apneas are scored when no flow is detected for 10 seconds.

The periodic breathing/Cheyne-Stokes respiration detection algorithm in the Philips PAP therapy devices provide a means to identify waxing and waning breathing cycles that repeat regularly in a range between 40 and 90 seconds. There are several processing steps that convert the patient flow into a collection of breath amplitudes. These amplitudes are pattern-matched against a typical Cheyne-Stokes respiration pattern that is stored in the device memory. Determination of fit is computed as a coherence function, and a second determination regarding the degree of amplitude modulation is computed over breath windows sufficient to cover 2 or 3 periodic cycles. The incoming breaths are labeled as Cheyne-Stokes respiration whenever the coherence measurement and amplitude sufficiently satisfy pre-established thresholds.

The machine-detected parameters were extracted from the first week’s data of every month from the start of use of the PAP device and averaged. This was a pragmatic decision to minimize data loss from episodic nights with nonuse or low use, and to allow selection of an alternate week from a given month. If patients did not use the machine during a certain week due to various reasons such as traveling, the data from the adjacent week was extracted. We required an average of 4 hours’ use, and for this analysis excluded those who were nonadherent by this more stringent standard (we did not use ≥ 4 hours 70% of nights to minimize data gaps). An auto–machine-detected respiratory events index (aREI_FLOW) was used to present the machine auto-detected respiratory events per hour.

Manual data extraction (visual scoring)

The authors (Y.N.N., who is a registered polysomnographic technologist) scored and counted the manually scored respiratory events index (sREIfow) using modifications of standard criteria (see below) from the last waveform graph (usage ≥ 4 hours) during every month. Events were scored when there was a clear reduction of signal amplitude (≥ 30%) or clear flow-limitation abruptly terminated with 2–3 larger recovery breaths. Apneas and hypopneas were not counted separately. RERAs were not specifically scored (ie, differentiating from hypopneas); as with the 6-minute/line pdf documents, there was only so much resolution to work with and no arousal surrogates were available (eg, photoplethysmographic signal amplitude or heart rate changes). In the compressed pdf, this invariably included some amplitude reduction. This approach is very similar to scoring hypopneas and some RERAs on polysomnograms. When scoring a respiratory event index, the scorer was blinded to treatment details of the patients.

Auto-PAP event–detection algorithms typically do not attempt to differentiate wake from sleep events. Wake respiratory event detection or scoring could over- or underestimate the abnormality, especially if there is prolonged sleep-wake transitional instability. However, in most instances it is likely an underestimate. The machine measures both the wakefulness time and the respiratory events (which should be less in quiet wakefulness than during sleep).

Scoring of stable and unstable breathing

To capture efficacy of therapy beyond discrete scorable events, we generated a unique measure of breathing stability. On visual review of waveforms, stable and unstable breathing is readily recognizable, with relatively clean boundaries (Figure 1). Stable breathing periods are characterized by a sequence of breaths for longer than 120 seconds without reduction in signal amplitude or discrete scorable events; there may or may not be flow-limitation. Such periods of stable breathing are well described during formal polysomnography.^23,24 Unstable breathing periods are characterized by rising and falling flow signal amplitudes that usually but not always meet the criteria for a discrete scorable event. Periodic breathing is always unstable, but not all unstable breathing is periodic, with self-similar waxing-waning events. If unstable breathing occupied ≥ 50% of a 6-minute line, that line was scored as unstable, otherwise scored as stable. The problem of “wakefulness” breathing requires additional attention: The pattern of transition of wakefulness to unstable pathological breathing is visually discernable in most instances—wake breathing looks “ataxic” unless opiates are being used, while unstable breathing shows the typical 30–40 second rhythmic rise and fall (cyclic variation in tidal volumes/flow). The presence of fluctuating flow-limitation is also typical of sleep-related rather than wakefulness breathing. We calculated the percentage of unstable periods during the entire period of use for the individual nights.

Figure 1. Examples of stable breathing and unstable breathing.

This patient had short-cycle (≤ 30 seconds) respiratory events. Arrows mark some of the events missed by the auto-detection algorithm. Each horizontal line is a 6-minute time span. The blue horizon line labeled H is the hypopnea detected by the machine. The green horizon line labeled OA is the obstructive sleep apnea detected by the machine. The orange horizon line labeled CA is the central sleep apnea detected by the machine. The yellow horizon line labeled RE is the respiratory effort–related event detected by the machine.

Scoring accuracy

To establish visual scoring accuracy for both event detection and stable/unstable breathing detection, 12 nights of data, 1 from each of 12 participants, were independently scored by both the authors, blinded, and then the Bland-Altman coefficient calculated. The values of sREI_FLOW and stable/unstable breathing scored by 2 authors were highly consistent. Coefficient of repeatability was 2.3236 for sREI_FLOW, P = .816. The Coefficient of repeatability for percentage of unstable breathing was 4.1383, P = .207. (Table S1 ^{(22.4KB, pdf)} in the supplemental material).

Statistical analysis

Statistical analyses included summary measures (mean ± standard deviation) when data were normally distributed and median (interquartile range [IQR]) when data was not normally distributed. Pearson correlation analysis was done to test the correlations between machine auto-detected index aREI_FLOW and manually scored results (sREI_FLOW). One-way analysis of variance was used to compare the value of REI_FLOW difference between the 4 time points.

To explore the factors associated with detection accuracy, a logistic regression model was used and adjusted by variables independently associated with the difference between aREI_FLOW and sREI_FLOW, using a 5-events/h difference to dichotomize variables with a P value less than .10. In order to reduce the risk of overfitting, we used the Pearson correlation test for the relevant variables (such as PSG data before and after split), and the variable with a smaller P value and/or greater clinical meaning was included in the multivariable logistic regression model when variables were highly correlated.

The analysis was performed using SPSS 19.0 (IBM, Armonk, NY). Bland-Altman plots were performed by MedCalc (version 15.2.2; MedCalc Software Ltd, Ostend, Belgium). A P value of less than .05 was taken as the level of statistical significance.

RESULTS

Subjects and demographics

A total of 336 patients who had split-night PSGs were included in the database. Individual patients were dropped from analysis for the following reasons: 1) no follow-up data available for 14 participants; 2) no data in an online system in 44 despite a PAP prescription on file (treatment refusal, treated in other sleep centers); 3) non-PAP therapies in 49; 4) treatment by a Resmed, sydney, Australia device with data in AirView where waveforms were not available in 34. From the remaining 195 patients, 16 did not complete 12 months of post-treatment use of PAP at the time of analysis. Thus, for this accuracy analysis we started with 179 patients who averaged 4 hours or more of use.

The mean age was 59.06 ± 13.97 years, 113 (63.1%) were male (Table 1). The median body mass index was 33.60 (IQR, 29.75–38.75) kg/m². Among the 140 patients for whom the Epworth sleepiness scale score was available, the median value was 8 (IQR 4.25–13). The comorbidities and medications used are listed in Table 1. Seven patients were not included in the first month analysis due to non-availability of waveforms (usage ≥ 4h) in the first several months. There were 172 patients available for comparison of manual and automated detection in the first month, 166 at the third month, 137 at the sixth, and 101 at the 12th month, respectively. Drop out was in part due to the stringency of criteria used (4 hours average/night).

Table 1.

Clinical characteristics.

Measure	Overall (n = 179)
Age, years	59.06 ± 13.97
Sex	113 (63.1%)
BMI, kg/m2	33.60 (29.75–38.75)
ESS	8 (4.25–13)
Stimulants	9 (5.0%)
Antidepressants	48 (26.8%)
Opiates	12 (6.7%)
Hypnotics	24 (13.4%)
Antihypertensive dugs	107 (58.9%)
Diuretic	44 (24.6%)
Lipid lowering drugs	93 (52%)
Antidiabetic drugs	46 (25.7%)
Acetazolamide	12 (6.7%)
RLS	2 (1.1%)
Delayed circadian rhythm	5 (2.8%)
Congestive heart failure	22 (12.3%)
Hypertension	110 (61.5%)
Diabetes	49 (27.4%)
CAD	23 (12.8%)
AF	19 (10.6%)
Depression	57 (31.8%)
GERD	44 (24.6%)
Renal failure	10 (5.6%)
Baseline RDI, events/h	47.30 (33.10–67.96)
Baseline AHI, events/h	46.30 (31.50–65.90)
Baseline CAHI, events/h	0.50 (0.0–3.30)
Baseline AI, events/h	37.20 (23.25–53.88)
Titration RDI, events/h	10.80 (5.43–20.05)
Titration AHI, events/h	10.60 (5.50–19.10)
Titration CAHI, events/h	0.9 (0.0–3.6)
Titration AI, events/h	17.30 (10.90–25.50)
Machine mode
Auto-CPAP	161 (89.9%)
Auto-Bilevel	18 (10.1%)
Mask type
Vented	141 (78.8%)
Nonvented	38 (21.2%)

Values are reported as mean ± SD, median (interquartile range) or n (%). ESS was reported in 140 patients. Titration events were across all pressures, not the optimal pressure. AF = arterial fibrillation, AHI = apnea-hypopnea index, AI = arousal index, BMI = body mass index, CAD = coronary artery disease, CAHI = central apnea-hypopnea index, CPAP = continuous positive airway pressure, ESS = Epworth Sleepiness Scale, GERD = gastroesophageal reflux disease, PAP = positive airway pressure, RDI = respiratory disturbance index, RLS = restless legs syndrome.

PSG data

The median baseline indices per hour of sleep were as follows: respiratory disturbance index 47.30 (IQR 33.10–67.96), AHI 3% 46.30 (IQR 31.50–65.90), central apnea-hypopnea index 0.50 (IQR 0.0–3.30), and arousal index 37.20 (IQR 23.25–53.88). The median titration indices/h of sleep, for the entire titration component (all pressures), were: respiratory disturbance index 10.80 (IQR 5.43–20.05), AHI 10.60 (IQR 5.50–19.10), central apnea-hypopnea index 0.9 (IQR 0.0–3.6), and arousal index 17.30 (IQR 10.90–25.50). Auto-CPAP was being used by 89.9% of the included patients, and the remainder, auto-bilevel mode. A vented mask was being used by 78.8% of patients, and 21.2% used a nonvented mask to stabilize carbon dioxide levels as treatment of high loop gain sleep apnea.²⁵

Residual events: machine vs manual

The median aREI_FLOW and sREI_FLOW during the 4 time points of evaluation are listed in the Table 2. The aREI_FLOW and sREI_FLOW were significantly correlated in the first month (r = .445, P < .001), the third month (r = .526, P < .001), the sixth month (r = .525, P < .001), and the 12th month (r = .560, P < .001). However, the Bland-Altman plots also demonstrated limitations in agreement between the 2 measurements with a trend for aREI_FLOW to highly underestimate relative to the sREI_FLOW from the first month to 12th month. The 95% confidence interval of the differences was 9.46–11.99, 10.29–12.93, 10.28–13.31, and 10.66–14.37, respectively, in the 4 time points. The agreement was reasonable, under 5 events/h, but progressively diverged with increasing severity of residual disease (Figure 2). The aREI_FLOW and REI_FLOW difference in 4 time points are shown in Figure 3, which represents the “burden” of residual apnea in the patient cohort. There was no significant difference in the value of REI_FLOW gap between the 4 time points (Table S2 ^{(22.4KB, pdf)} in the supplemental material). The REI_FLOW differences also remained stable in these patients. (Table S3 ^{(22.4KB, pdf)} and Table S4 ^{(22.4KB, pdf)} in the supplemental material).

Table 2.

Comparison of REI_flow between machine-detected and manually scored results.

	aREIflow	sREIflow	r	P	Mean REIflow Difference	95% CI of the REIflow Difference	P
First month (n = 172)	3.70 (2.00–6.78)	15.01 (8.20–22.02)	.445	< .001	10.72 ± 8.43	9.46–11.99	< .001
Third month (n = 166)	2.65 (1.50–5.20)	13.48 (8.84–21.83)	.526	< .001	11.61 ± 8.63	10.29–12.93	< .001
Sixth month (n = 137)	2.80 (1.45–6.10)	13.20 (9.04–22.17)	.525	< .001	11.80 ± 8.97	10.28–13.31	< .001
12th month (n = 101)	2.70 (1.30–5.35)	13.26 (9.07–20.23)	.560	< .001	12.12 ± 9.37	10.66–14.37	< .001

CI = confidence interval, aREI_flow = auto-machine-detected respiratory event index, REI_flow = respiratory event index, sREI_flow = manually scored respiratory event index.

Figure 2. Bland-Altman plots of aREI_flow and sREI_flow.

(A) The Bland–Altman plots of aREI_flow and sREI_flow in the first month. (B) The Bland–Altman plots of aREI_flow and sREI_flow in the third month. (C) The Bland–Altman plots of aREI_flow and sREI_flow in the sixth month. (D) The Bland–Altman plots of aREI_flow and sREI_flow in the 12th month.

Figure 3. The aREI_flow, sREI_flow, and REI_flow difference in individual patients at 4 time points (n = 86).

(A) The sREI_flow in each patient at 4 time points. The Y-axis indicates the value of sREI_flow for each patient. The X-axis indicates duration after PAP initiation. (B) The REI_flow difference in each patient at 4 time points. The Y-axis indicates the value of REI_flow difference for each patient. The X-axis indicates duration after PAP initiation (C) The mean sREI_flow and REI_flow difference in each patient at 4 time points. The Y-axis indicates the mean value of sREI_flow and REI_flow difference, respectively. The X-axis indicates duration after PAP initiation.

Stable and unstable breathing

The median percentage of unstable breathing at the 4 time points were 10.23% (IQR, 3.00–22.71), 8.20% (IQR, 2.50–19.49), 7.69% (IQR, 2.04–21.73), and 9.21% (IQR, 3.08–21.56), respectively (Table 3). The percentages of machine-estimated periodic breathing at the 4 time points were 0.50 (IQR, 0.20–1.55), 0.40 (IQR, 0.10–1.33), 0.40 (IQR, 0.10–1.35), and 0.30 (IQR, 0.05–1.45). Manual unstable breathing percentage was significantly correlated with machine-detected periodic breathing; the r values were .431, .301, .154, and .566, all P values were less than .05.

Table 3.

The relationship between the percentage of unstable breathing and machine-detected periodic breathing.

	Percentage of Unstable Breathing	Machine-Detected Percentage of Periodic Breathing	r	P
First month (n = 172)	10.23 (3.00–22.71)	0.50 (0.20–1.55)	.431	< .001
Third month (n = 166)	8.20 (2.50–19.49)	0.40 (0.10–1.33)	.301	< .001
Sixth month (n = 137)	7.69 (2.04–21.73)	0.40 (0.10–1.35)	.154	.036
12th month (n = 101)	9.21 (3.08–21.56)	0.30 (0.05–1.45)	.298	.003

Factors associated with detection accuracy

The comparison of characteristics in patients who have REI_FLOW difference < 5 or not are shown in Table 4. The error/difference was greater in males (68.6% vs 51%, P = .029), those with higher rate of large leak > 1.5% (31.4% vs 50.4%, P = .022), and those with a higher percentage of unstable breathing (2.80 [0.00–8.06]% vs 15.00 [7.48–27.35]%, P < .001). More patients in the REI_FLOW difference ≥ 5 group had a baseline arousal index > 45 events/h of sleep (40.5% vs 24%, P = .034) and a titration arousal index > 15 events/h of sleep (65.3% vs 41.2%, P = .003). No difference in mask type or mode type was found between the 2 groups. After adjustment, the percentage of unstable breathing (odds ratio 1.126, 95% confidence interval 1.067–1.188, P < .001) and the arousal index during titration (odds ratio 2.274, 95% confidence interval 1.049–4.931, P < .001) were the independent predictors for the detection accuracy (REI_FLOW difference ≥ 5) (Table 5).

Table 4.

Comparison of the characteristics between patients divided by REI_flow difference.

Measure	REIflow Difference < 5 (n = 51)	REIflow Difference ≥ 5 (n = 121)	P
Age, years	57.55 ± 14.08	58.96 ± 3.74	.543
Sex	26 (51%)	83 (68.6%)	.029
BMI, kg/m2	32.4 (28.95–38.80)	34 (29.90–38.75)	.705
ESS	8 (5–12.75)	9 (4–13)	.242
Stimulants	2 (3.9%)	7 (5.8%)	.999
Antidepressants	14 (27.5%)	32 (26.4%)	.892
Opiates	6 (11.8%)	6 (5%)	.186
Hypnotics	6 (11.8%)	17 (14%)	.688
Antihypertensive dugs	27 (52.9%)	73 (60.3%)	.370
Diuretic	11 (21.6%)	30 (24.8%)	.650
Lipid lowering drugs	25 (49%)	61 (50.4%)	.867
Antidiabetic drugs	13 (25.5%)	30 (24.8%)	.923
Acetazolamide	3 (5.9%)	7 (5.8%)	.999
RLS	0 (0%)	2 (1.7%)	N/A
Delayed circadian rhythm	1 (2%)	3 (2.5%)	.999
Congestive heart failure	8 (15.7%)	12 (9.9%)	.281
Hypertension	26 (51%)	77 (63.6%)	.129
Diabetes	15 (29.4%)	31 (25.6%)	.706
CAD	6 (11.8%)	15 (12.4%)	.999
AF	4 (7.8%)	14 (11.6%)	.591
Depression	17 (33.3%)	37 (30.6%)	.723
GERD	16 (31.4%)	27 (22.3%)	.248
Renal failure	3 (5.9%)	5 (4.1%)	.696
Baseline RDI, events/h	44.5 (31.5–59.6)	51.1 (34.1–67.78)	.267
Baseline AHI, events/h	43.8 (30.3–58.4)	47.3 (32.35–67.78)	.285
Baseline CAHI, events/h	0 (0–3.2)	0.5 (0–3.3)	.171
Baseline AI, events/h	31.95 (17.55–43.45)	38.9 (25.3–57.7)	.030
Baseline AI ≥ 45 events/h	12 (24.0%)	49 (40.5%)	.034
Titration RDI, events/h	4.95 (11.9–17.5)	5.5 (10.6–20.68)	.598
Titration AHI, events/h	11.9 (5.2–17.5)	10.21 (5.3–20.45)	.750
Titration CAHI, events/h	0.8 (0–3.6)	0.9 (0–3.4)	.507
Titration AI, events/h	13.5 (9.89–23.00)	18.30 (11.25–25.85)	.048
Titration AI ≥ 15 events/h	21 (41.2%)	79 (65.3%)	.003
Machine mode
Auto-CPAP	47 (30.5%)	107 (69.5%)	.591
Auto-bilevel	4 (22.2%)	14 (77.8%)
Mask type
Vented	43 (84.3%)	93 (76.9%)	.272
Nonvented	8 (15.7%)	28 (23.1%)
Large leak > 1.5%	16 (31.4%)	61 (50.4%)	.022
Percentage of unstable breathing	2.80 (0.00–8.06)	15.00 (7.48–27.35)	< .001
Duration of PAP daily use, min	284.47 ± 130.05	272.74 ± 129.24	.588

Values are reported as mean ± SD, median (interquartile range), or n (%). ESS was reported in 136 patients. Titration events were across all pressures, not the optimal pressure. AHI = apnea-hypopnea index, AF = arterial fibrillation, AI = arousal index, BMI = body mass index, CAD = coronary artery disease, CAHI = central apnea-hypopnea index, CPAP = continuous positive airway pressure, ESS = Epworth sleepiness scale, GERD = gastroesophageal reflux disease, PAP = positive airway pressure, RDI = respiratory disturbance index, REI_flow = respiratory event index, RLS = restless leg syndrome, SD = standard deviation.

Table 5.

Univariable and multivariable logistic regression for assessing predictors for detection accuracy.

	Univariable Logistic Regression		Multivariable Logistic Regression
	OR (95% CI)	P	OR (95% CI)	P
Sex	0.476 (0.244–0.930)	.030	0.646 (0.291–1.434)	.283
Baseline arousal index ≥ 45	2.155 (1.025–4.533)	.043	N/A	N/A
Titration arousal index ≥ 15	2.687 (1.373–5.260)	.004	2.274 (1.049–4.931)	.037
Large leak > 1.5%	2.224 (1.115–4.436)	.023	1.515 (0.677–3.392)	.312
Percentage of unstable breathing	1.144 (1.084–1.207)	< .001	1.126 (1.067–1.188)	< .001

Baseline arousal index and titration arousal index was highly correlated (r = .446, P < .001), and the measure with a stronger statistical association on univariable analysis was used for the multivariable analysis. CI = confidence interval, N/A = not applicable, OR = odds ratio.

DISCUSSION

The key results of our analysis are as follows: 1) machine-detected REI_FLOW of a commonly used current generation PAP substantially underestimates residual respiratory abnormality during PAP treatment; 2) the discordance between automated detection and manually scored events does not diminish with the duration of use; 3) stable and unstable breathing are readily quantified visually and may be clinically useful if available in automated outputs; and 4) some factors associated with larger discrepancies were identified: percentage of unstable breathing and arousal index during titration.

All flow generators used for treatment of sleep apnea employ a single tube for airflow delivery to the patient’s interface. Typically, expiration occurs through the interface’s intentional leak or through an intentional leak from a device placed between the interface and the delivery tubing (eg, Whisper Swivel [Respironics]). Moreover, there is unintentional leak from around the interface, and thus there is no method to directly measure patient airflow. Instead, airflow is estimated by measuring pressure and flow at the delivery end of the flow generator. A calculation of mask leak is made and subtracted from proximal total airflow to derive estimated patient airflow. Apneas and hypopneas are determined as deviations from baseline flow, which is not constant. Ideally, simultaneous polysomnography should be performed with CPAP estimation,²⁶ but recording for multiple nights becomes prohibitively expensive and burdensome and is not done. Moreover, each manufacturer is free to change the manner in which these calculations are made with little oversight by the US Food and Drug Administration, since all positive airway pressure devices have been designated as belonging to class II, which requires less review before marketing after any changes are made to algorithms.

The hard thresholding of event detection by device algorithms, using criteria motivated by conventional polysomnogram scoring, may not work as well during therapy, as respiratory waveforms are modified by dynamics of positive airway pressure output patterns. Thus, an event may be too short, too long, or have a rate of change that is outside the algorithm’s criteria or ataxic patterns that are not recognized. One complementary approach is to quantify respiratory stability, as stable breathing periods are readily recognizable. We found that the ratio of unstable to stable breathing was significantly associated with the level of REI_FLOW difference, in both short cycle or long cycle events, as shown in Figure 1 and Figure 4. Cyclic unstable breathing often presents as central hypopneas,²⁷ while auto-detection of hypopnea in general is difficult.²⁸ The unstable breathing patterns shown in Figure 1 and Figure 4 have the characteristics of self-similar breathing oscillations, as are often seen in heart failure,²⁹ and high loop gain apnea of diverse etiologies and clinical associations.³⁰ Previous studies excluded patients with heart failure and those who experienced central events during the titration.^14–16,31 Thus, this may be a reason why the REI_FLOW difference found by us was greater than in prior studies. Our study may be a better reflection of the clinical practice challenges of auto-detection accuracy. However, our sleep service is also a referral center for central/complex sleep apnea, and we use a nonvented mask with or without additional dead space for stabilization of carbon dioxide.²⁵ Appropriate for this approach are those patients with overt central sleep apnea or periodic breathing, non–rapid eye movement sleep-dominant obstructive sleep apnea, and treatment-emergent central sleep apnea. This referral bias may also have contributed to the high residual apnea in our population.

Figure 4. Periodic breathing but underdetection of discrete respiratory events in the presence of long cycle respiratory events.

Snapshot of the waveform of 1 patient with long cycle respiratory events. The black arrows mark examples of respiratory events missed by the machine. In this example, periodic breathing detection is accurate. Each horizontal line is a 6-minute time span. The blue horizon line labeled H is the hypopnea detected by the machine.

Our study found that the discordance between the aREI_FLOW and sREI_FLOW slightly increased over time but did not meet statistical significance. There are 2 possible explanations if this is a true effect: 1) patients with late onset treatment-emergent/complex sleep apnea³² can contribute to increasing disagreement; 2) at a given pressure or pressure range, manifest effects of high loop gain or low arousal threshold may decrease over time, resulting in “shallower” events that escape algorithmic detection, yet are readily discernable to the human eye.

It may be better to estimate detection accuracy of devices by manual vs computational assessment of event detection, and more than 1 approach to auto-detection could be considered as complementary. A study including 20 patients and using an older auto-CPAP device found that although all apneas were scored by both systems, 41% more hypopneas were scored on PSG and these were clinically significant, with 78% ending in cortical arousal.³³ The AutoSet (ResMed, San Diego, CA) detected hypopneas during wakefulness also and could not distinguish wakefulness and sleep periods, and thus also underestimated the value of REI.³³ The detection of large breaths might help the detection of hypopneas ending with cortical arousals. A study of the Philips Respironics REMstar Auto M-Series showed that using an automatic event detection REI ≥ 10 events/h cut-off, the positive predictive value was 0.67 and negative predictive value was 0.92 for a PSG AHI ≥ 10 events/h.³⁴ A study using the WatchPAT (Itamar Medical, Caesarea, Israel) arterial tonometry-based event detection technique demonstrated substantially more events than the device in those with CPAP-detected REI ≤ 5 events/h and clinical suspicion of residual events.² Nearly half of those with acceptable CPAP-estimated REI had elevated WatchPAT events. The patient population was an admixture of device types and manufacturers. A study evaluating the Philips Respironics System One REMstar Auto A-Flex service showed good accuracy, but patients with difficult CPAP titrations were excluded, the very group for whom detection accuracy is critical.³⁵ Algorithms have changed and likely improved over the years, with improved detection, but our results suggest that further improvements are needed to capture the entire spectrum of residual events. Our data used the same signal that the device used to score events and evaluated a current device and associated algorithms.

Leak effects require consideration in explaining some of the differences in detection and persistence of events. Prior studies comparing polysomnography and machine estimation were done in a setting of manual titration, where large leaks would be corrected in time during the study.^1,14–18 Our previous study excluded the patients on PAP who had large leaks.²⁰ In the current analysis, it did not take a very large leak for an event to become associated with inaccurate detection. Large leaks can cause respiratory instability including central apneas and hypopneas,³⁶ which are largely ignored by the machine, especially central hypopnea³³ (as shown in the Figure 5). Leak can cause and amplify pressure fluctuations that may induce arousals, which by increasing tidal volume can feed back to the device algorithms.^37,38 A large leak can be caused by high pressures.³⁹ In split-night studies, the time for titration is relative short and we cannot exclude the possibility that the pressure was too high for some patients. However, the majority were on standard auto-CPAP devices at typical clinical settings. Large leaks can wash out carbon dioxide from the anatomical dead space in the upper airway and lower the carbon dioxide reverse,⁴⁰ and be a contributor to unstable respiratory drive and periodic breathing.⁴¹ With a large leak, the ventilator might overestimate the pressure output and thus lower the effect of positive airway pressure therapy.⁴²

Figure 5. Respiratory events undetected by machine related to large leak.

Snapshot of the waveform of 1 patient with large leak (gray bands). The black arrows mark examples of respiratory events missed by the machine. Each horizontal line is a 6-minute time span. The blue horizon line labeled H is the hypopnea detected by the machine. The green horizon line labeled OA is the obstructive sleep apnea detected by the machine. The yellow horizon line labeled RE is the respiratory effort–related event detected by the machine.

One sleep apnea driver trait is a low arousal threshold, which may contribute to fragility of sleep even during CPAP. The degree of sleep fragmentation associated with sleep apnea of similar degrees also varies widely and is common in conditions such as heart failure and atrial fibrillation. Fragmentation and low arousal threshold add a further dynamic into the sleep system that may impact residual events and their accurate detection. Our study found that if the arousal index during titration was more than 15 events/h of sleep, the risk of underestimating increased significantly. One explanation is that the machine detects arousal-linked tidal volume fluctuations as RERA and does not report them in the AHI. Another possibility is that respiratory cortical arousal is followed by a large breath, causing respiratory drive instability and leading to central hypopneas,^24,43 which may be missed by the machine.³³ Finally, easy arousability and sleep fragmentation results in increased wake during the sleep period, and sleep-wake transitional events may have characteristics visually recognizable but not within algorithmic range.

Our study noted a sex effect in the univariable analysis: Males were more likely to have a machine-/manual-scoring REI_FLOW difference ≥ 5 events/h of recording; this was not significant in the multivariable analysis. Males have more discrete events and females may have flow-limitation–dominant abnormality, but that does not readily explain why detection may be more accurate in females. Males have higher loop gain, and male sex dominates such conditions across high altitude, complex apnea, and even periodic breathing associated with heart failure.^44–47 In our study, the median percentage of unstable breathing was higher (11.76 [5.31–23.61]% vs 6.82 [1.59–15.91]%, P = .016) in males compared to females. Speculatively, there may be greater instability on CPAP and reduced differentiation between normal and abnormal breathing patterns. Males also have a greater ventilatory response to arousal than females,⁴⁸ which may again change the background on which the automated algorithms compute respiratory event occurrence.

Although underestimation was the most common feature, overestimation can occur, as shown in Figure 6. We are not able to query the algorithm decision-making process in order to understand why these occurred. Very slow respiratory rates, as induced by opioids, for example, may result in excessive central apnea “detection” as the long expiratory phase durations may reach algorithmic thresholds.

Figure 6. Overestimation of respiratory events.

Snapshot of the waveform of 1 patient. The black arrows marks examples of tagging of respiratory events without a clear hypopnea or apnea. Each horizontal line is a 6-minute time span. The blue horizon line labeled H is the hypopnea detected by the machine. The green horizon line labeled OA is the obstructive sleep apnea detected by the machine. The yellow horizon line labeled RE is the respiratory effort–related event detected by the machine.

The issue of the clinical importance—residual apnea during CPAP use—remains unresolved. Clearly, improvement in overt central sleep apnea can occur with time on PAP.⁴⁹ However, the underestimation of events we show and the difficulty with central hypopnea detection can result in undetected residual high loop gain–driven respiratory instability. High loop gain can be improved by PAP,⁵⁰ but central events may also persist long-term, be associated with reduced treatment adherence, and improve when central sleep apnea is targeted by adaptive ventilation.⁵¹ We show that the underestimation of residual sleep apnea can persist for up to 12 months in adherent patients, and state effects (eg, hypoxia effects) should have resolved by then. This means that residual sleep apnea may not be a transitional problem, especially given the substantial underestimation by the auto-algorithms. The pattern of abnormality may evolve with time, converting from more typical central events to unstable breathing with some features of high loop gain such as self-similarity but subthreshold to the typical long cycle periodic breathing tagged by the current generation of auto-CPAPs.

Our study has several strengths. Careful hand-scoring of events was performed. We included patients with large leak and were generally inclusive. We restricted analysis to adherent patients so that trait vs state effects could be better differentiated. Prior studies used 2 signals, 1 from the PSG and 1 as detected by the machine, but we scored the respiratory events based on the same signal. Our stable/unstable scoring method can be applied to research studies and, conceptually, used clinically when raw waveforms are reviewed.

There are some limitations in our study. A pragmatic approach to manual-scoring burden limited the estimation of night-to-night variability of manually scored events, as we averaged a week of machine-estimated event data. The patients were split-night PSG instead of full-night PSG. The reason for this was that it proved difficult to obtain separate laboratory diagnostic and titration studies for the average patient (insurance coverage limitation). There are different physiological features of first vs second half of the sleep that could influence some of the results. Our analysis was limited to the Philips system as the waveforms were readily available and may not fully generalize to other manufacturers. Although the respiratory events detecting accuracy can vary by various manufacturers, a study showed that the mean difference was only 0.68 events/h.⁵² The value of the REI_FLOW difference between the machine-detected and manual-scoring results in our study was much higher than differences between the manufacturers. We could not blind the scoring—there was no method to obtain unmarked waveforms data from the online system. However, this nonblinding should not have directly influenced scoring and the conclusions since the aREI_FLOW and sREI_FLOW were almost entirely from different nights/periods. All of our included patients had moderate to severe OSA (meeting the criteria for a split night), and the results may not generalize to milder apnea. We did not manually score periodic breathing, as mild flow-limitation often is seen in residual events that have a periodic appearance, and no effort signals were available; this also did not allow us to accurately independently hand-score central events.

CONCLUSIONS

Residual respiratory events including apneas and hypopneas on PAP are common, underestimated, and persistent. Sex, large leak, and especially the amount of unstable breathing during use, and arousal frequency during the titration were factors associated with inaccuracy. This report does not address the clinical implications and predictors of residual apnea. Our study included patients with moderate to severe OSA, and with a high proportion of mixed physiology (obstruction and high loop gain), which limits the generalizability of our conclusions.

DISCLOSURE STATEMENT

All authors read and approved the final manuscript. Work for this study was funded by the American Academy of Sleep Medicine Foundation, category-I award to R.J.T. Dr. Thomas is co-inventor and patent holder of the ECG-derived sleep spectrogram, which may be used to phenotype sleep quality and central/complex sleep apnea. The technology is licensed by Beth Israel Deaconess Medical Center to MyCardio, LLC. He is also co-inventor and patent holder of the Positive Airway Pressure Gas Modulator being developed for treatment of central/complex sleep apnea. He has consulted for Jazz Pharmaceuticals and consults for Guidepoint Global and GLG Councils. He is co-inventor of an auto-CPAP software licensed to Drive DeVilbiss Healthcare. Ms. Ni reports no conflicts of interest.

ABBREVIATIONS

AHI

apnea-hypopnea index

aREI_FLOW

auto–machine-detected respiratory events index

CPAP

continuous positive airway pressure

IQR

interquartile range

OSA

obstructive sleep apnea

PAP

positive airway pressure

PSG

polysomnography

REI

respiratory events index

RERA

respiratory effort–related arousal

sREI_FLOW

manually scored respiratory events index