Correlations between home sleep apnea tests and polysomnography outcomes do not fully reflect the diagnostic accuracy of these tests

Abstract

Study Objectives:

The clinical performance of home sleep apnea tests (HSATs) can be described by their (diagnostic) accuracy, defined as the percentage agreement with the obstructive sleep apnea severity category (normal, mild, moderate, and severe) based on polysomnography. Rather than reporting on accuracy, there has been a strong reliance in the literature to report correlation coefficients between the apnea-hypopnea index of HSATs and polysomnography to support claims of diagnostic performance. This is surprising, as it has been well described that correlation coefficients are inadequate to evaluate equivalence between 2 parameters. The aim of this study was to systematically investigate the magnitude of the discrepancies between correlation coefficients and diagnostic accuracy reported in or retrievable from HSAT validation studies.

Methods:

We compared the discrepancy between accuracy and apnea-hypopnea index correlation coefficients of all validation papers that met the inclusion criteria. A total of 20 papers were retained, representing a participant pool of 1,652.

Results:

The weighted average apnea-hypopnea index correlation across all 20 papers was 0.82 and the weighted average accuracy was 0.61, highlighting a discrepancy of 0.21 and an overall misdiagnosis rate of 39%.

Conclusions:

The results of our study confirm the need for increased scientific rigor in selecting primary performance endpoints to support clinical performance claims of HSATs.

Citation:

Massie F, Van Pee B, Bergmann J. Correlations between home sleep apnea tests and polysomnography outcomes do not fully reflect the diagnostic accuracy of these tests. J Clin Sleep Med. 2022;18(3):871–876.

BRIEF SUMMARY

Current Knowledge/Study Rationale: Rather than reporting on diagnostic accuracy, there has been a strong reliance in the home sleep apnea testing literature on correlation coefficients between the apnea-hypopnea index of home sleep apnea tests and polysomnography as primary endpoint parameters to support claims of diagnostic performance. Nevertheless, it is known that correlation coefficients are inadequate to evaluate equivalence between 2 parameters. It was the aim of our study to systematically investigate the magnitude of the discrepancies between correlation coefficients and diagnostic accuracy reported in or retrievable from home sleep apnea test validation studies.

Study Impact: Our meta-analysis revealed a discrepancy of 21% between the apnea-hypopnea index correlation and diagnostic accuracy. This highlights the need for increased scientific rigor in selecting performance endpoints to describe the performance of home sleep apnea tests.

INTRODUCTION

The American Academy of Sleep Medicine (AASM) clinical guideline for the evaluation, management, and long-term care of obstructive sleep apnea (OSA) in adults¹ defines the severity of OSA by whether the apnea-hypopnea index (AHI) is lower than 5 (normal), between 5 and 15 (mild OSA), between 15 and 30 (moderate OSA), or above 30 events per hour of sleep (severe OSA).

The performance of a home sleep apnea test (HSAT) can be evaluated by determining the percentage agreement of the OSA severity (normal, mild, moderate, or severe OSA) estimated by the HSAT with that determined by concurrently administered gold-standard polysomnography (PSG). This percentage agreement is commonly referred to as the (diagnostic) accuracy or the (diagnostic) concordance.

Yalamanchali et al² performed a comprehensive meta-analysis of the clinical performance of the most frequently deployed HSAT by systematically reviewing all papers that reported on the AHI correlation between the HSAT and PSG. From the apparently very strong correlation coefficients reported in the included papers, the authors concluded that the HSAT represents a viable alternative to PSG for confirmation of clinically suspected sleep apnea.

However, it has been well described in domains outside of sleep medicine that correlation coefficients such as Pearson’s and Spearman’s correlations are inadequate in describing the degree of one-on-one correspondence of 2 diagnostic endpoint parameters.³ Almost 2 decades ago, Flemons et al^4,5 pointed out, based on an example comparing the respiratory disturbance index (RDI) of HSAT and the AHI of PSG, how (Pearson) correlation coefficients are only able to indicate whether 2 measurements are (linearly) related, and not whether they have a similar magnitude.

Indeed, the Pearson correlation coefficient attains the maximum value of 1 upon a perfect linear relationship between 2 endpoint parameters, but it does not penalize a constant offset or scaling factor between the parameters. Worse in this context is the Spearman correlation coefficient, as it attains the maximum value of 1 when there is a perfect monotonously increasing relationship between the 2 parameters, without penalizing for the nonlinearity of such a relationship. Correlation coefficients are also heavily influenced by extreme datapoints, such as very high AHIs.

These issues can be illustrated by the examples presented in Figure 1. All 4 panels show how AHI correlation coefficients can be arbitrarily high for devices with little to no clinical utility (as illustrated by diagnostic accuracies of < 0.6). The top left and right panels, respectively, show how the Pearson correlation does not take scaling mismatches and constant offsets between the AHIs of HSAT and PSG into account. The bottom left panel shows how low AHIs (< 5 events/h) and extreme AHIs (> 30 events/h) skew Pearson correlations to misleadingly large values, while the intrinsic diagnostic accuracy is very low. The bottom left panel has the same diagnostic accuracy of 0.5 as the bottom right panel but has fewer extreme AHI values in the sample pool, highlighting the collapse of the Pearson correlation in the absence of extremity-introduced bias. Importantly, extremely low and high AHIs, while excessively influencing correlations, have the least relevance in the performance evaluation of an HSAT. Indeed, it is comparatively straightforward to detect a near-complete absence of respiratory events (eg, by a completely flat oxygen saturation [SpO₂] trace) or an extreme prevalence of respiratory events (eg, by the presence of extremely frequent oxygen desaturations). The patients most challenging to assess with any HSAT device are those with AHIs between 5 and 30 events/h, a range of particular importance for therapeutic decisions.

Figure 1. Sample scatterplots highlighting issues of relying on correlation coefficients to describe equivalence between 2 parameters.

The green-shaded squares mark the areas in which PSG and HSAT agree on the AHI severity category. Top left: AHIs differ by a constant scaling factor (AHI PSG = 2 × AHI HSAT). Top right: AHIs differ by a constant offset (AHI PSG = AHI HSAT + 12). The differences between AHI PSG and AHI HSAT do not impact the correlation coefficient but strongly impact the accuracy. Bottom: Extreme values disproportionally influence correlation coefficients. Moving extreme values (dots within red circles) more to the center of the spectrum, without altering the accuracy, has a strong effect on the correlation coefficient. This is observed by comparing the bottom left (correlation = 0.9) and bottom right (correlation = 0.6) panels. AHI = apnea-hypopnea index, HSAT = home sleep apnea test, PSG = polysomnography.

A recent study by Ioachimescu et al,⁶ in which the HSAT device described by Yalamanchali et al² was evaluated against PSG on 500 participants, reported a very apparent discrepancy between the study’s AHI Pearson correlation of 0.80 and diagnostic accuracy of 0.53. These findings drove a starkly different conclusion than the one by Yalamanchali et al.² Indeed, the authors stated that because of the large diagnostic misclassification rate, patients without OSA or mild disease assessed by the HSAT should undergo repeat in-laboratory PSG as such, limiting the device’s utility to inclusion screening. These conflicting conclusions highlight the peril of reliance on correlation coefficients as opposed to diagnostic accuracy as a primary clinical performance endpoint of HSAT devices.

Prompted by the scientific community’s surprise⁷ about these findings, we systematically compared the discrepancy between diagnostic accuracy and AHI correlation coefficients of all published validation studies of the HSAT devices that met the search criteria.

METHODS

In order to allow for cross-trial, weighted-average aggregation of accuracy and correlation parameters, we limited the inclusion of HSAT types to the single most frequently validated HSAT device. The WatchPAT device (Itamar Medical, Caesarea, Israel) was found to be the most frequently included in clinical validation studies (∼ 20% of all HSAT validation studies report on this device).

As such, we searched PubMed on April 3, 2021, for the search term “‘watchpat’ or ‘watch-pat’ or ‘watch pat.’”

Any paper that met at least 1 of the exclusion criteria was discarded from further analysis. The following exclusion criteria were used:

1. The paper was not available in the English language.

2. The paper did not report on the performance validation of the HSAT device.

3. The HSAT was not concurrently administered with the PSG.

4. No diagnostic accuracy could be retrieved from the paper.

5. No AHI correlation coefficient could be retrieved from the paper.

For all included papers, the correlation between the (automated) AHI estimate of HSAT and PSG was retrieved. We additionally registered the type of the reported correlation coefficient (Spearman, Pearson, or intraclass correlation coefficient). For all included papers, we retrieved the accuracy of the HSAT. For papers for which the accuracy was not explicitly reported, or for which it could not be precisely calculated from the confusion matrix, we estimated the accuracy based on the datapoints of the scatter plots or the Bland-Altman plots, or inverse-calculated the accuracy from the disease prevalence, sensitivity, and specificity values. Such scatterplots are equivalent to those presented in Figure 1. We counted the number of dots falling in the green rectangles and divided this number by the total number of dots to arrive at the accuracy value. Any discrepancy between the estimated participant size from the scatter plots or Bland-Altman plots and the reported participant size was also identified.

As accuracy estimates could be impacted by incomplete or incorrect visual retrieval of the datapoints, we performed a subgroup analysis in which papers that did not allow exact diagnostic accuracy retrieval were excluded.

For papers in which only an RDI correlation and accuracy could be retrieved, we used the RDI instead of the AHI. Finally, the scoring rules used to score the AHI by PSG were noted for each paper. We performed a subgroup analysis on papers that reported on the current (2012) AASM respiratory event scoring rules.⁸

We charted out the diagnostic accuracy and the AHI correlation coefficient for each included paper, ranked by decreasing study participant number. Finally, we calculated the overall (sample-size weighted average) accuracy and the AHI correlation coefficient of the total participant pool.

RESULTS

As illustrated in Figure 2, the search term resulted in the identification of 86 papers, of which 20 were retained after application of the exclusion criteria. Fifty out of 86 identified papers were not considered HSAT validation studies, rendering this the main reason for exclusion. The results of the systematic analysis are summarized in Figure 3 and Table 1.

Figure 2. Overview of identified and retained papers and the reasons for any exclusions.

PSG = polysomnography.

Figure 3. Comparison of accuracy and correlations for each included paper.

Comparison of accuracy and AHI correlations (horizontal axis) as retrieved from each included paper (vertical axis). Papers are ranked by increasing sample size (largest at bottom). Overall accuracy and correlation were calculated from a weighted average of all individual trials, with the weights equal to the trial population size. AHI = apnea-hypopnea index, appr. = accuracy was approximated from the graphs or inverse-calculated, ex. = accuracy was exactly inferred.

Table 1.

Tabulation of accuracy and correlations for each included paper.

Study	AHI or RDI	Correlation (type)	Accuracy (appr. or exp.)	Difference	n (reported)	Scoring Rules
Holmedahl et al, 20199	AHI	0.96 (ICC)	0.81 (appr.)	0.15	16	O
Penzel et al, 200410	RDI	0.89 (Pearson)	0.41 (appr.)	0.48	17	O
Gan et al, 201611	AHI	0.94 (Pearson)	0.80 (appr.)	0.14	20	A
Choi et al, 201012	AHI	0.94 (Pearson)	0.80 (ex.)	0.14	25	O
Boyd et al, 201613	AHI	0.87 (Pearson)	0.48 (appr.)	0.39	27 (28)	O
Weimin et al, 201314	AHI	0.92 (Pearson)	0.61 (ex.)	0.31	28	O
Pinto et al, 201515	AHI	0.76 (Spearman)	0.35 (appr.)	0.42	29 (30)	O
Körkuyu et al, 201416	AHI	0.80 (Spearman)	0.76 (appr.)	0.04	29 (30)	O
Ayas et al, 200317	AHI	0.87 (Pearson)	0.60 (ex.)	0.27	30	O
O’Brien et al, 201218	AHI	0.73 (Pearson)	0.74 (ex.)	−0.01	31	O
Jen et al, 202019	AHI	0.85 (Pearson)	0.61 (appr.)	0.24	33	O
Choi et al, 201820	AHI	0.95 (Spearman)	0.95 (ex.)	0.00	38	A
Pittman et al, 200621	AHI	0.74 (ICC)	0.59 (appr.)	0.15	68 (70)	O
Pillar et al, 201922	AHI	0.87 (Pearson)	0.62 (appr.)	0.25	77 (84)	A
Yuceege et al, 201323	AHI	0.96 (Pearson)	0.74 (appr.)	0.22	85	O
Zou et al, 200624	AHI	0.90 (Pearson)	0.67 (appr.)	0.23	92	O
Tauman et al, 202025	AHI	0.80 (Pearson)	0.59 (appr.)	0.21	101	A
Zhang et al, 202026	AHI	0.65 (Spearman)	0.61 (appr.)	0.04	170	A
Hedner et al, 201127	RDI	0.87 (ICC)	0.60 (appr.)	0.27	236	O
Ioachimescu et al, 20206	AHI	0.80 (Pearson)	0.53 (ex.)	0.27	500	A and B
Overall (A or B)	AHI	0.79	0.59	0.20	906	A or B
Overall (O)	AHI or RDI	0.87	0.63	0.24	746	O
Overall (appr.)	AHI or RDI	0.83	0.62 (appr.)	0.21	1,000	A, B, or O
Overall (ex.)	AHI or RDI	0.82	0.58 (ex.)	0.24	652	A, B, or O
Overall	AHI or RDI	0.82	0.61	0.21	1,652	A, B, or O

Papers are ranked by increasing sample size. Papers exclusively reporting on the RDI were highlighted as such in column 2. For “appr.” papers, the participant number (n) is displayed based on the paper’s graphs, with the reported number between parentheses when it was different to the sample size determined from the graphs or metrics used to estimate the accuracy. “Overall (A or B)” and “Overall (O),” respectively, contain the sample-size weighted average of all studies scored by “A or B” rules or “O” rules. “Overall (appr.)” and “Overall (ex.),” respectively, contain the sample-size weighted average of all studies with an “appr.” or “ex.” accuracy. A = AASM 2020 scoring rules with 1A rule for hypopnea, AHI = apnea-hypopnea index, appr. = accuracy was approximated from the graphs or inverse-calculated, B = AASM 2020 scoring rules with 1B rule for hypopnea, ex. = accuracy was exactly inferred, ICC = intraclass correlation coefficient, O = scoring rule other than any AASM 2012 rule, RDI = respiratory disturbance index.

The weighted average AHI correlation across all 20 papers was 0.82 and the weighted average accuracy was 0.61, highlighting a discrepancy of 0.21. The lowest observed accuracy was 0.35 (correlation 0.76) and the highest observed accuracy was 0.95 (correlation 0.95), which illustrates a large degree of performance variation between studies. The lowest observed AHI correlation was 0.65 (accuracy 0.61) and the highest observed AHI correlation was 0.96 (accuracy 0.74).

The weighted average AHI correlation and accuracy across all papers for which the accuracy could exactly be retrieved were, respectively, 0.82 and 0.58. For 6 out of 20 papers, the accuracy was exactly retrievable. Six out of 20 studies reported on the AASM 2020 scoring rules using the 1A rule for hypopnea requiring 3% desaturations.⁸ Only 1 study reported (partially) on the AASM 2020 scoring rules using the 1B rule for hypopnea requiring 4% desaturations. Comparing the studies that scored using any of the AASM 2012 rules with the papers that applied an older scoring rule revealed a difference of 0.04 (from 0.59 to 0.63) for accuracy and 0.08 (0.79 to 0.87), for correlation, where the highest number was found for the older rules.

DISCUSSION

Similar to the results reported by Ioachimescu et al,⁶ our meta-analysis determined a discrepancy of 21% between the diagnostic accuracy of 61% and correlation coefficient of 82%, derived from a total participant pool size of 1,652. When only considering papers reporting on the latest AASM scoring rules, a misdiagnosis rate of 41% was found. For only 6 out of the 20 included papers, the diagnostic accuracy could directly be retrieved or exactly calculated, revealing a large underreporting of this important endpoint parameter.

These observations confirm the urgent need for researchers and device manufacturers to increase scientific rigor and transparency in the presentation of HSAT devices’ clinical performance by complementing their reporting with diagnostic accuracy, in a move away from the overreliance on correlations. This study can be viewed as a precursor to much-needed future work on the determination of a statistically robust set of parameters to characterize the performance of HSATs in general, and the agreement between HSATs and PSG in particular. This problem is embedded within the broader observation that hypothesis testing is typically being deployed for the determination of significant differences between 2 measurements, and not for the determination of their equivalence.

This meta-analysis had several limitations. First, the accuracy had to be estimated from alternative data such as scatter plots and Bland-Altman plots for 14 out of 20 papers, due to lack of explicit reporting of the parameter. Second, only 6 papers reported on the latest 2012 AASM scoring rules and only 1 paper reported on the latest 1B rules for the scoring of hypopnea. Finally, there was heterogeneity in participant population characteristics. Most papers (11/20) reported on a participant population with a high suspicion of OSA, while other papers discussed specific participant populations, such as those with cardiovascular or pulmonary comorbidities. Nevertheless, the totality of participants represents a patient group that closely aligns with the HSAT’s utilization in the market.

DISCLOSURE STATEMENT

All authors have seen and approved the manuscript. Institution where work was performed: University of Oxford. Frederik Massie and Bart Van Pee are the founders of Ectosense, the manufacturer of the NightOwl HSAT device. At the time of acceptance of the article and at the time of this research, Frederik Massie and Bart Van Pee were the majority shareholders of Ectosense and received salary from Ectosense. As of October 1, 2021 ResMed has acquired Ectosense. With the acquisition, Ectosense is now wholly owned by ResMed and continues to develop and market diagnostics. Frederik Massie and Bart Van Pee continue to earn salary for their roles but report no current shareholdings of Ectosense or ResMed. Jeroen Bergmann reports no conflicts of interest.

ABBREVIATIONS

AASM

American Academy of Sleep Medicine

AHI

apnea-hypopnea index

HSAT

home sleep apnea test

OSA

obstructive sleep apnea

PSG

polysomnography

RDI

respiratory disturbance index