The presence of obstructive sleep apnea (OSA) has been associated with many negative health outcomes, increased risk for motor vehicle collisions, reduced quality of life and high medical costs. OSA is likely undiagnosed in a significant proportion of the population, but is more prevalent in men than in women. However, no prediction rules to detect clinically significant (ie, symptomatic or severe) OSA exist. Accordingly, the goal of this study was to develop and validate a diagnostic algorithm for such a purpose, which could prove useful in several aspects of triage, management and research.
Keywords: Clinical prediction, Decision rule, Diagnostic algorithm, Obstructive sleep apnea
Abstract
OBJECTIVES:
To assess the ability of electronic health data and existing screening tools to identify clinically significant obstructive sleep apnea (OSA), as defined by symptomatic or severe OSA.
METHODS:
The present retrospective cohort study of 1041 patients referred for sleep diagnostic testing was undertaken at a tertiary sleep centre in Calgary, Alberta. A diagnosis of clinically significant OSA or an alternative sleep diagnosis was assigned to each patient through blinded independent chart review by two sleep physicians. Predictive variables were identified from online questionnaire data, and diagnostic algorithms were developed. The performance of electronically derived algorithms for identifying patients with clinically significant OSA was determined. Diagnostic performance of these algorithms was compared with versions of the STOP-Bang questionnaire and adjusted neck circumference score (ANC) derived from electronic data.
RESULTS:
Electronic questionnaire data were highly sensitive (>95%) at identifying clinically significant OSA, but not specific. Sleep diagnostic testing-determined respiratory disturbance index was very specific (specificity ≥95%) for clinically relevant disease, but not sensitive (<35%). Derived algorithms had similar accuracy to the STOP-Bang or ANC, but required fewer questions and calculations.
CONCLUSIONS:
These data suggest that a two-step process using a small number of clinical variables (maximizing sensitivity) and objective diagnostic testing (maximizing specificity) is required to identify clinically significant OSA. When used in an online setting, simple algorithms can identify clinically relevant OSA with similar performance to existing decision rules such as the STOP-Bang or ANC.
Abstract
OBJECTIFS :
Évaluer si les données de santé électroniques et les outils de dépistage en place permettent de diagnostiquer une apnée obstructive du sommeil (AOS) importante sur le plan clinique, conformément à la définition d’AOS symptomatique ou grave.
MÉTHODOLOGIE :
Les chercheurs ont mené la présente étude de cohorte rétrospective auprès de 1 041 patients orientés vers un centre tertiaire d’étude du sommeil de Calgary, en Alberta, pour subir un test diagnostique de troubles du sommeil. Chaque patient a reçu un diagnostic d’AOS importante sur le plan clinique ou un autre diagnostic de trouble du sommeil après l’examen indépendant en insu des dossiers, effectué par deux médecins du sommeil. Les chercheurs ont déterminé les variables prédictives à partir des données d’un cyberquestionnaire et mis au point des algorithmes diagnostiques. Ils ont déterminé le rendement d’algorithmes dérivés des données électroniques pour dépister les patients atteints d’une AOS importante sur le plan clinique. Ils ont comparé le rendement diagnostique de ces algorithmes aux versions du questionnaire STOP-Bang et au score de circonférence cervicale (CCA) dérivé des données électroniques.
RÉSULTATS :
Les données du questionnaire électronique étaient extrêmement sensibles (plus de 95 %) pour dépister une AOS importante sur le plan clinique, mais n’étaient pas spécifiques. L’indice de perturbation respiratoire déterminé par le test diagnostique de trouble du sommeil était très spécifique (au moins 95 %) pour les maladies pertinentes sur le plan clinique, mais n’était pas sensible (moins de 35 %). Les algorithmes dérivés avaient une précision similaire au questionnaire STOP-Bang ou au score de CCA, mais nécessitaient moins de questions et de calculs.
CONCLUSIONS :
D’après ces données, il faut utiliser un processus en deux étapes faisant appel à un petit nombre de variables cliniques (afin de maximiser la sensibilité) et à un test diagnostique objectif (afin de maximiser la spécificité) pour diagnostiquer une AOS importante sur le plan clinique. Utilisés dans un contexte électronique, des algorithmes simples peuvent dépister des AOS pertinents sur le plan clinique selon un rendement similaire aux règles décisionnelles comme le questionnaire STOP-Bang ou le score CCA.
Obstructive sleep apnea (OSA) affects at least 24% of men and 9% of women, and is associated with negative cardiovascular and metabolic health outcomes, increased risk of motor vehicle collisions, poor quality of life and increased medical costs (1–8). OSA is normally defined by the respiratory disturbance index (RDI) as determined by polysomnography or ambulatory monitoring. However, the RDI does not correlate well with sleep apnea symptoms or treatment outcomes (9,10). In the Wisconsin Sleep Cohort Study, approximately one-fifth of patients with OSA reported sleepiness (1). Moreover, adverse health outcomes related to OSA, such as cardiovascular disease or diabetes, may occur independently of symptoms, particularly in those with severe OSA (3,6,7). Current clinical guidelines recommend treatment of OSA to improve symptoms or to mitigate the risk of adverse health outcomes in patients with severe disease (11,12).
A number of clinical prediction rules for OSA have been described, including the adjusted neck circumference score (ANC), cricomental space, elbow sign, Berlin Questionnaire, and the STOP and STOP-Bang questionnaires (13–18). These tools were validated for identifying OSA as defined by RDI. However, no prediction rules exist that identify patients with clinically significant OSA; that is, symptomatic or severe OSA.
Moreover, many of the clinical prediction rules have not been evaluated as part of an online screening process. An automated online process for identifying patients with clinically significant OSA (ie, those who might benefit from treatment) would be useful for triaging patients to the appropriate health care provider, improving clinical management and identifying patients for research purposes.
The primary objective of the present study was to develop diagnostic algorithms for identifying patients with symptomatic or severe OSA. The secondary objective was to validate these algorithms, as well as adaptations of existing rules, when collected in an electronic format.
METHODS
Patients
The present cohort was used in a previous study describing the use of online algorithms for identifying patients with insomnia (19). All patients who completed an online questionnaire and underwent clinical assessment and/or sleep diagnostic testing at the Foothills Medical Centre Sleep Centre (Calgary, Alberta) between January 1, 2009 and January 1, 2011 were identified. The Foothills Medical Centre Sleep Centre is the only tertiary referral centre for Calgary (a catchment area of approximately 1.3 million people). All referred patients are required to fill out an online questionnaire at the time of referral.
Determination of the primary diagnosis
A primary diagnosis of clinically significant OSA superseded other secondary nonrespiratory diagnoses. OSA was defined as an RDI ≥5/h. OSA severity was defined as mild (RDI 5/h to 14.9/h), moderate (RDI 15/h to 29.9/h) or severe (RDI ≥30/h).
Two American Board of Sleep Medicine- or American Board of Internal Medicine-certified sleep physicians independently reviewed all patient charts and assigned a primary sleep diagnosis to each patient in the cohort, as well as determining whether OSA was clinically significant. If a diagnosis was not agreed on independently, the disagreement was noted and the patient was excluded from the analysis. The sleep physicians performing the chart review were blinded to the results from the online questionnaire.
Clinically significant OSA was defined as symptomatic or severe OSA. Symptomatic OSA was based on the treating physician’s impression as documented in the patient’s chart. This impression took into account the Epworth Sleepiness Scale, adherence to treatment and the patient’s reported perception of benefit. Patients with a severe OSA (RDI ≥30/h) were considered to have clinically significant OSA (regardless of symptoms or comorbid conditions), with the rationale being that severe OSA is associated with comorbid disease for which treatment is recommended by clinical guidelines (11,12). The decision to pursue further diagnostic testing and treatment was at the discretion of the treating physicians.
Ambulatory monitoring
The RDI was determined from level III sleep diagnostic testing (20). The Remmers Sleep Recorder (SagaTech Electronics Ltd, Canada) is an ambulatory monitor that measures snoring, oxygen saturation, respiratory airflow (by monitoring nasal pressure) and body position. The RDI is derived from automated shape analysis based on falls and recovery of digitally recorded oxygen saturation using a 4% desaturation threshold. The flow signal was used for quality assurance on manual review by a sleep physician.
Electronic data elements
The online questionnaire is comprised of 108 questions, which provide a comprehensive overview of a patient’s demographics, anthropometrics, snoring history, daytime function and medical history, as well as sleep schedule, behaviour and complaints.
Several questions pertain to heart disease, hypertension and diabetes. These diseases were identified based on positive responses to binary yes/no questions, or self-report in the free-text medical history. Heart disease was defined as a self-reported history of angina, heart failure, heart attack or coronary artery bypass surgery. Hypertension was defined according to self-reported history of hypertension or high blood pressure. Diabetes was defined as self-reported diabetes.
The STOP and STOP-Bang questionnaires consist of four and eight yes/no questions, respectively. The questions in the present study were adapted from the STOP-Bang questions, but wording was not identical (Appendix 1) (18).
Statistical analysis
The distribution of the primary sleep diagnoses among patients referred for clinical assessment/sleep diagnostic testing within the study time-frame was calculated. The agreement between physician-assigned diagnoses was assessed by the kappa statistic. Descriptive statistics were used to compare the clinical and demographic characteristics of patients stratified according to the presence or absence of clinically significant OSA. Univariate logistic regression was then used to identify predictive variables from the online questionnaire, using the presence of clinically significant OSA as the dependent variable. A full model was constructed from the univariate predictors and reduced by stepwise regression. ROC curves and box plots were constructed. Cut-points were selected by visual inspection, taking into account ROC area under the curve.
Using identified predictive variables, several diagnostic algorithms were constructed. The sensitivity, specificity, positive predictive value and negative predictive value were calculated to assess the performance of the various diagnostic algorithms in predicting clinically significant OSA. A sensitivity analysis was performed by repeating the modelling process to predict mild, moderate and severe OSA as defined by the RDI. The performance of the STOP and STOP-Bang questionnaires, and the ANC score were also assessed to identify clinically significant OSA (13,18). Last, the ability of the univariate predictors to identify OSA (as defined by ambulatory monitoring cut-offs) was examined.
All analyses were performed using Stata 9.0 statistical software (Stata Corporation, USA). Data are presented as mean (± SD) unless otherwise indicated. For all statistical tests, P<0.05 was considered to be statistically significant.
Ethics
The University of Calgary Conjoint Health Research Ethics Board approved the study.
RESULTS
Patient characteristics and diagnostic agreement
Between January 1, 2009 and January 1, 2011, a total of 1426 patients completed an online sleep questionnaire. Of these, one patient did not complete the questionnaire correctly and 202 patients refused consent to participate. An additional 16 patients were excluded due to disagreement between reviewers on their primary diagnosis, leaving 1207 patients in the baseline cohort. However, a total of 166 patients did not have an ambulatory monitoring test, leaving a final cohort size of 1041 patients for analysis (Figure 1).
Figure 1).
Patient flow. FMC Foothills Medical Centre (Calgary, Alberta)
Clinically significant OSA was the most common primary diagnosis in the cohort (46%), followed by insomnia (28%). Table 1 shows the distribution of primary sleep diagnoses. Patient characteristics are summarized in Table 2. The mean age of all patients was 45±2 years, 57% of all patients were men, and the mean body mass index (BMI) was 30.6±7.6 kg/m2. Patients with clinically significant OSA were significantly older, had higher BMI and larger neck circumferences than those without. Within the entire cohort, 21.4%, 31.3% and 6.3% of patients self-reported heart disease, hypertension and diabetes, respectively.
TABLE 1.
Distribution of primary diagnosis
| Obstructive sleep apnea syndrome | 554 (45.9) |
| Insomnia | 339 (28.1) |
| Central nervous system hypersomnolence | 58 (4.8) |
| Normal | 50 (4.1) |
| Other | 48 (4.0) |
| Primary snoring | 41 (3.4) |
| Restless leg syndrome | 28 (2.3) |
| Obstructive sleep apnea/hypoventilation | 27 (2.2) |
| Upper airway resistance syndrome | 22 (1.8) |
| Depression | 13 (1.1) |
| Fatigue | 9 (0.8) |
| Parasomnia | 8 (0.7) |
| Fibromyalgia | 6 (0.5) |
| Central sleep apnea | 4 (0.3) |
| Total | 1207 (100) |
Data presented as n (%)
TABLE 2.
Patient characteristics
| Characteristic | All patients (n=1207) | Clinically significant OSA | |
|---|---|---|---|
|
| |||
| Yes (n=554) | No (n=653) | ||
| Age, years, mean ± SD | 45.4±12.1 | 47.3±11.5 | 43.8±12.4* |
| Sex | 685 (56.8) | 396 (71.5) | 289 (44.3)** |
| Weight, kg, mean ± SD | 90.3±23.9 | 99.4±23.8 | 82.5±21.1* |
| BMI, kg/m2, mean ± SD | 30.6±7.6 | 33.0±7.7 | 28.5±7.0* |
| Neck size, cm, mean ± SD | 40.1±7.1 | 42.4±6.9 | 38.4±7.1* |
| Self-reported heart disease | 258 (21.4) | 153 (27.6) | 105 (16.1)** |
| Self-reported hypertension | 378 (31.3) | 219 (39.5) | 159 (24.3)** |
| Self-reported diabetes | 76 (6.3) | 42 (7.6) | 34 (5.21) |
Data presented as n (%) unless otherwise indicated.
P<0.05 between groups with and without clinically significant obstructive sleep apnea (OSA);
P<0.05 (X2) between groups with and without clinically significant OSA. BMI Body mass index
The reviewing physicians agreed on 98.7% of diagnoses (1207 of 1223). The kappa statistic indicated that the agreement between the two reviewing physicians was high (0.98±0.016).
Predictors of clinically significant OSA
A self-reported history of snoring, witnessed apneas, history of heart disease, nocturnal choking sensation, total sleep time >4 h, ANC ≥48 cm and BMI were significant predictors of a diagnosis of symptomatic or severe OSA (Table 3). Self-reported hypertension, diabetes and an Epworth Sleepiness Scale score >10 did not significantly contribute to the model (P>0.05) but were included in the algorithm analysis based on previous associations with OSA in the literature.
TABLE 3.
Univariate predictors of clinically significant obstructive sleep apnea identified from electronic data
| Question | OR (95% CI) | P |
|---|---|---|
| Snoring history (yes/no) | 5.01 (2.71 to 9.28) | <0.05 |
| Stop breathing (yes/no) | 2.84 (2.12 to 3.79) | <0.05 |
| Heart disease (yes/no) | 1.65 (1.14 to 2.40) | <0.05 |
| Choking (yes/no) | 1.41 (1.06 to 1.89) | <0.05 |
| Total sleep time, h | 1.16 (1.06 to 1.28) | <0.05 |
| Body mass index, kg/m2 | 1.07 (1.04 to 1.09) | <0.05 |
| Sleep aid use (yes/no) | 0.57 (0.42 to 0.78) | <0.05 |
| Sleep latency, h | 0.48 (0.34 to 0.69) | <0.05 |
| Diabetes | 1.26 (0.91 to 1.75) | 0.16 |
| Epworth Sleepiness Scale score >10 | 1.03 (0.78 to 1.37) | 0.84 |
| Hypertension | 0.74 (0.42 to 1.31) | 0.30 |
Diagnostic performance of decision rules
The ability of diagnostic algorithms to identify symptomatic or severe OSA is summarized in Table 4. A self-reported history of snoring, total sleep time >4 h and a BMI >25 kg/m2 were associated with sensitivities >90%, although specificity was poor. No single univariate predictor had simultaneously high sensitivity and specificity.
TABLE 4.
Diagnostic performance of predictive algorithms derived from an online questionnaire at identifying clinically significant obstructive sleep apnea
| Sens | Spec | PPV | NPV | |
|---|---|---|---|---|
| Univariate predictors identified within electronic data | ||||
| Snoring history (yes/no) | 97.5 | 25.3 | 52.5 | 92.2 |
| Stop breathing (yes/no) | 74.2 | 64.0 | 63.6 | 74.5 |
| Heart disease (yes/no) | 27.6 | 83.9 | 59.3 | 57.7 |
| Choking (yes/no) | 50.9 | 68.6 | 57.9 | 62.2 |
| Total sleep time >4 h | 98.0 | 9.04 | 47.8 | 84.3 |
| BMI >25 kg/m2 | 91.5 | 33.5 | 53.9 | 82.3 |
| BMI >30 kg/m2 | 59.9 | 67.4 | 60.9 | 66.5 |
| Sleep aid use (yes/no) | 22.6 | 52.7 | 28.8 | 44.5 |
| Sleep latency >1 h | 11.0 | 74.9 | 27.1 | 49.8 |
| Diabetes | 7.58 | 94.8 | 55.3 | 54.7 |
| Epworth Sleepiness Scale score >10 | 61.2 | 52.5 | 52.2 | 61.5 |
| Hypertension | 39.5 | 75.7 | 57.9 | 59.6 |
| Combinations of univariate predictors identified within electronic data | ||||
| Snoring history or witnessed apneas | 98.4 | 23.1 | 52.1 | 94.4 |
| Snoring history or choking | 98.0 | 20.7 | 51.2 | 92.5 |
| Snoring history or BMI >25 kg/m2 | 99.6 | 12.6 | 49.2 | 97.6 |
| Choking or BMI >25 kg/m2 | 94.9 | 24.3 | 51.6 | 85 |
| Heart disease and BMI >30 kg/m2 | 17.1 | 92.3 | 65.5 | 56.8 |
| Level III sleep diagnostic testing | ||||
| RDI ≥5/h and <15/h | 34.1 | 63.7 | 47.1 | 50.4 |
| RDI ≥15/h and <30/h | 28.2 | 96.3 | 87.7 | 58.5 |
| RDI ≥30/h | 32.7 | 94.6 | 86.0 | 59.8 |
| Univariate predictors in combination with level III sleep diagnostic testing | ||||
| Snoring history and RDI >30/h | 32.3 | 95.7 | 87.7 | 59.8 |
| BMI >30 kg/m2 and RDI >15/h | 39.4 | 95.3 | 88.9 | 62.4 |
| Performance of existing clinical prediction rules adapted from electronic questionnaire | ||||
| STOP | 85.6 | 51.9 | 60.2 | 80.9 |
| STOP-Bang | 91.0 | 45.8 | 58.7 | 85.7 |
| AdMusted neck circumference score ≥48 | 39.2 | 85.6 | 69.8 | 62.4 |
Data presented as %. BMI Body mass index; NPV Negative predictive value; PPV Positive predictive value; RDI Respiratory disturbance index; Sens Sensitivity; Spec Specificity
Multivariate prediction algorithms increased either sensitivity or specificity (as compared with univariate predictors), but at the expense of reducing specificity or sensitivity, respectively. For example, snoring or a BMI >25 kg/m2 was very sensitive but not specific at identifying clinically significant OSA (sensitivity 99.6%, specificity 12.6%), as was a history of snoring or choking (sensitivity 98.0%, specificity 20.7). Conversely, the combination of self-reported heart disease and a BMI >30 kg/m2 had low sensitivity but was highly specific at identifying clinically significant OSA (sensitivity 17.1%, specificity 92.3%). No combination of univariate or multivariate predictors from the questionnaire was both highly specific or highly sensitive (Table 4). Moreover, diagnostic performance was not increased by increasing the number of predictive variables.
In general, the RDI was very specific at identifying patients with clinically significant disease particularly for moderate or severe OSA (specificity ≥95%). However, while objective testing alone could identify patients with OSA, it was relatively insensitive with respect to clinically relevant disease (sensitivity <35%). Combining a clinical predictor with level III sleep diagnostic testing failed to improve the diagnostic performance with respect to clinically significant disease.
In contrast, existing prediction rules such as the STOP and STOP-Bang, as derived from our modified clinical questionnaire, provided high sensitivities (85% and 91%, respectively) and intermediate specificities (52% and 46%, respectively) when identifying clinically significant OSA. The STOP-Bang had lower sensitivity than simpler multivariate predictors, but had slightly higher specificity. The performance of the ANC was similar to level III sleep diagnostic testing in that it had high specificity (86%) and lower sensitivity (39%).
In keeping with the previous literature, clinical predictors including the STOP-Bang were sensitive (>90%) at identifying patients with OSA (as defined by RDI on diagnostic testing); however, specificity was poor (<40%) (Table 5). Moreover, when administered as an online questionnaire, the derived STOP-Bang questionnaire exhibited similar diagnostic performance to that described in the original paper (Appendix 2) (17).
TABLE 5.
Diagnostic performance of predictive algorithms at identifying obstructive sleep apnea (as defined by respiratory disturbance index [RDI])
| Sens | Spec | PPV | NPV | |
|---|---|---|---|---|
| Snoring history | ||||
| RDI ≥5/h and <15/h | 90.7 | 13.4 | 36.3 | 72.6 |
| RDI ≥15/h and <30/h | 97.5 | 13.7 | 17.3 | 96.8 |
| RDI ≥30/h | 96.9 | 13.9 | 20.4 | 95.2 |
|
| ||||
| BMI >25 kg/m2 | ||||
| RDI ≥5/h and <15/h | 81.7 | 20.9 | 36.0 | 67.8 |
| RDI ≥15/h and <30/h | 92.6 | 22.3 | 18.1 | 94.2 |
| RDI ≥30/h | 92.2 | 22.8 | 21.4 | 92.8 |
|
| ||||
| Snoring history or witnessed apneas | ||||
| RDI ≥5/h and <15/h | 92.1 | 11.3 | 36.1 | 72.4 |
| RDI ≥15/h and <30/h | 99.4 | 11.8 | 21.8 | 17.3 |
| RDI ≥30/h | 97.4 | 11.8 | 20.1 | 95.2 |
|
| ||||
| Snoring history or choking | ||||
| RDI ≥5/h and <15/h | 91.8 | 10.2 | 35.8 | 69.7 |
| RDI ≥5/h and <30/h | 98.8 | 11.0 | 17.1 | 98.0 |
| RDI ≥30/h | 97.4 | 11.1 | 20.0 | 94.9 |
|
| ||||
| Snoring history or BMI >25 kg/m2 | ||||
| RDI ≥5/h and <15/h | 96.7 | 5.9 | 35.9 | 76.9 |
| RDI ≥15/h and <30/h | 100.0 | 5.9 | 16.5 | 100 |
| RDI ≥30/h | 99.5 | 6.0 | 19.4 | 98.1 |
|
| ||||
| STOP | ||||
| RDI ≥5/h and <15/h | 67.8 | 30.7 | 34.8 | 63.7 |
| RDI ≥15/h and <30/h | 84.7 | 34.2 | 19.3 | 92.3 |
| RDI ≥30/h | 89.1 | 35.8 | 24.0 | 93.5 |
|
| ||||
| STOP-Bang | ||||
| RDI ≥5/h and <15/h | 78.5 | 27.3 | 37.0 | 70.0 |
| RDI ≥15/h and <30/h | 94.5 | 28.9 | 19.8 | 96.6 |
| RDI ≥30/h | 94.3 | 29.7 | 23.4 | 95.8 |
Data presented as %. BMI Body mass index; NPV Negative predictive value; PPV Positive predictive value; Sens Sensitivity; Spec Specificity
DISCUSSION
Diagnostic algorithms derived from an online questionnaire can be highly sensitive (>95%) or moderately specific (<75%) at identifying patients with clinically significant OSA, but not both. In contrast, the objectively determined RDI is highly specific (specificity ≥95%) when identifying clinically relevant disease, but suffers from low sensitivity (<35%). Our results suggest that when screening for clinically significant OSA, a combination of both clinical variables and objective testing is required. An automated process involving an online questionnaire and level III sleep diagnostic testing has similar diagnostic performance to previously reported screening strategies.
The present is the first study to evaluate electronically acquired data including existing screening tools (modified for online use) to identify clinically relevant OSA, defined as symptomatic or severe disease and for which treatment is recommended. Most studies have focused on identifying OSA as defined by the RDI. In our study, both the STOP-Bang and ANC also identify OSA (as defined by RDI) with high sensitivity or high specificity, but not both simultaneously. When used for determining clinically relevant OSA, clinical variables such as a history of snoring, witnessed apneas or a BMI >25 kg/m2 favour high sensitivity rather than specificity. Given that treatment is recommended in patients with symptomatic or severe OSA, the ability to identify this subgroup of patients with OSA would provide more timely care.
Using online and electronic data to screen for OSA has the potential to improve clinical efficiency, namely by providing an automated screening strategy. In a previous article (19), we outlined a similar approach for identifying patients with insomnia. In combination, a screening strategy that identifies patients with OSA or insomnia would help to direct patients to the most appropriate provider for assessment and management. Furthermore, if there is reasonable confidence in a diagnosis of OSA or insomnia based on this screening strategy, patients could be directed to non-sleep physician providers for initiation of continuous positive airway pressure (CPAP) or cognitive behavioural therapy, respectively.
Regardless of the decision rules employed, modification for online use did not impact diagnostic performance as compared with the original self-administered (or administered) questionnaires. The specific questions varied slightly among screening tools; however, diagnostic performance is similar. Recently, Chung et al (21) demonstrated that the diagnostic performance of STOP-Bang could be improved by alternating the scoring model. However, simultaneously high sensitivities and specificities could not be achieved (21).
As expected, the RDI alone was an imperfect predictor of clinically relevant disease. In a previous study, Whitelaw et al (22) demonstrated that when clinicians are provided with an RDI or apnea-hypopnea index, the ability to predict improvement in quality of life or CPAP compliance is poor (60%). In contrast, we have demonstrated that the RDI (at moderate or greater severity) is very good at identifying patients with clinically relevant disease (specificity ≥95%); however, sensitivity is poor (<35%). Despite slightly different outcomes of interest, both studies suggest that an objective measure alone (RDI) is insufficient for predicting clinically relevant outcomes.
Recently, Pereira et al (23) found that adding clinical data from questionnaires did not increase the discriminant ability of level III testing alone to identify OSA (as defined by the apnea-hypopnea index). Similarly, we found that combining clinical predictors with the RDI did not improve diagnostic accuracy for predicting clinically significant OSA. While the specificity remained high whether RDI and clinical predictors were used alone or in combination, sensitivity was decreased compared with using clinical variables alone. These results suggest that when screening for clinically relevant disease, a two-step process incorporating clinical variables (to maximize sensitivity) and objective testing (to maximize specificity) may be beneficial and warrants further investigation. Our findings should be interpreted within the strengths and limitations of the study. Diagnosis of clinically significant OSA was based on chart review, and heavily dependent on documentation by the treating physician. While follow-up Epworth Sleepiness Scale scores and CPAP adherence were taken into consideration, these were not systematically collected. As such, the treating physician would have been reliant on subjective patient perception. To mitigate this, we required that two board-certified sleep physicians reach consensus on diagnosis for each patient after independent chart review. Diagnosing physicians agreed on 98.7% of diagnoses (kappa score of 0.98). Furthermore, although a two-step process involving an online questionnaire and level III sleep diagnostic is highly specific, it is not 100%, as would be expected by our definition of clinically significant OSA. This speaks to the limitations of level III sleep diagnostic testing in determining CPAP response as well as night to night variability in RDI. Polysomnography and a clinical assessment are still required in some patients.
We also assumed a single primary sleep diagnosis for all patients in this analysis. While secondary sleep diagnoses were coded, treatment decisions were typically based on the primary diagnosis. It is not uncommon for patients to have overlapping sleep disorders and comorbidities. However, we assumed a single primary diagnosis because patients with multiple sleep diagnoses would likely require assessment by a sleep physician rather than benefiting from an automated screening strategy.
Furthermore, the use of ambulatory monitoring rather than polysomnography to determine RDI may have also been perceived as a limitation. However, the Remmers Sleep Recorder has been validated against polysomnography both in terms of bias and clinical management of OSA (24,25). Given a sensitivity of 98% (using a case definition of an RDI >15/h), it is unlikely that the Remmers Sleep Recorder would have missed many patients with OSA (25). Moreover, polysomnography was performed in 30% (n=75) of patients without OSA on ambulatory monitoring (n=276). However, polysomnographic evaluation was at the discretion of the treating physician based on their post-ambulatory monitoring suspicion of OSA.
Finally, the size of our existing questionnaire is too cumbersome for real-world use. It consisted of 108 questions, including data elements derived from STOP-Bang, ANC, insomnia severity index and the Patient Health Questionnare. However, the present study indicates that the number of questions can be markedly reduced. With respect to identifying clinically significant OSA, use of existing decision rules or simple two-variable decision algorithms offer similar diagnostic performance. A similar approach can be used for diagnosing insomnia (19).
Historically, screening tools have focused on prediction of OSA (as defined by RDI), or in terms of predicting treatment compliance. Moreover, these have frequently been paper questionnaires administered at the time of assessment. We propose an automated two-step diagnostic process for identifying clinically relevant OSA, using an online clinical questionnaire to identify patients who should then go on to further objective testing.
Given limited access to sleep clinicians, as well as challenges associated with improving CPAP compliance, an automated process for identifying patients with clinically relevant disease could be beneficial in terms of patient access and triage, resource allocation and timeliness of care.
CONCLUSIONS
When used in an online setting, simple two-variable algorithms or existing decision rules such as STOP-Bang or ANC (adapted for the online setting) can identify symptomatic or severe OSA (ie, clinically significant OSA) with high sensitivity or specificity, but not both. An automated two-step process combining online collection of clinical variables (maximizing sensitivity) and objective testing (maximizing specificity) may improve diagnostic accuracy.
Acknowledgments
This work was supported through funding from the O’Brien Centre for the Health Sciences and the Division of Respirology, University of Calgary, Calgary, Alberta.
APPENDIX 1. STOP-Bang questionnaire and analogous questions in sleep centre questionnaire
| STOP-Bang questions | Our analogous questions |
|---|---|
| Do you snore loudly (louder than talking or loud enough to be heard through closed doors)? | Do you snore?* OR Does your snoring cause your bed partner to sleep in another room?* |
| Do you often feel tired, fatigued, or sleepy during daytime? | From Epworth Sleepiness Scale:
|
| Has anyone observed you stop breathing during your sleep? | Has anyone ever told you that you stop breathing when you sleep? |
| Do you have or are you being treated for high blood pressure? | Do you, or have you ever, suffered from high blood pressure (hypertension)? |
| BMI more than 35 kg/m2? | Calculated from height and weight. |
| Age over 50 yr old? | Calculated from date of birth and questionnaire input date. |
| Neck circumference > 40cm. | Calculated from self reported neck size in inches. |
| Gender male? | What is your gender? |
No difference was found when the analysis was repeated using either of these analogous questions.
APPENDIX 2. STOP and STOP-Bang comparison between the present Calgary cohort and the original Chung et al (21) cohort
| Calgary cohort | Original cohort | |||||||
|---|---|---|---|---|---|---|---|---|
|
|
|
|||||||
| Sens | Spec | PPV | NPV | Sens | Spec | PPV | NPV | |
| STOP | ||||||||
| RDI >5/h | 77.4 | 50.6 | 78.0 | 49.7 | 65.6 | 60.0 | 78.4 | 44.0 |
| RDI >15/h | 87.3 | 40.7 | 43.2 | 86.1 | 74.3 | 53.3 | 51.0 | 76.0 |
| RDI >30/h | 89.5 | 35.8 | 23.9 | 93.8 | 79.5 | 48.6 | 30.4 | 89.3 |
|
| ||||||||
| STOP-Bang | ||||||||
| RDI >5/h | 86.4 | 51.6 | 80.2 | 62.6 | 83.6 | 56.4 | 81.0 | 60.8 |
| RDI >15/h | 94.6 | 35.5 | 43.1 | 92.7 | 92.9 | 43.0 | 51.6 | 90.2 |
| RDI >30/h | 94.8 | 29.7 | 23.3 | 96.2 | 100 | 37.0 | 31.0 | 100 |
Data presented as %. NPV Negative predictive value; PPV Positive predictive value; RDI Respiratory disturbance index; Sens Sensitivity; Spec Specificity
Footnotes
AUTHOR CONTRIBUTIONS: All authors contributed equally to data collection, analysis and interpretation, as well as manuscript drafting and revision. All authors reviewed and approved the final version. Drs Tsai and Pendharkar were responsible for the majority of study design.
DISCLOSURES: The authors have no financial disclosures or conflicts of interest to declare.
REFERENCES
- 1.Young T, Palta M, Dempsey J, Skatrud J, Weber S, Badr S. The occurrence of sleep-disordered breathing among middle-aged adults. N Eng J Med. 1993;328:1230–5. doi: 10.1056/NEJM199304293281704. [DOI] [PubMed] [Google Scholar]
- 2.George CF, Nickerson PW, Hanly PJ, Millar TW, Kryger MH. Sleep apnoea patients have more automobile accidents. Lancet. 1987;330:447. doi: 10.1016/s0140-6736(87)90974-3. [DOI] [PubMed] [Google Scholar]
- 3.Marin JM, Carrizo SJ, Vicente E, Agusti AG. Long-term cardiovascular outcomes in men with obstructive sleep apnoeahypopnoea with or without treatment with continuous positive airway pressure: an observational study. Lancet. 2005;365:1046–53. doi: 10.1016/S0140-6736(05)71141-7. [DOI] [PubMed] [Google Scholar]
- 4.Peppard PE, Young T, Palta M, Skatrud J. Prospective study of the association between sleep-disordered breathing and hypertension. N Engl J Med. 2000;342:1378–84. doi: 10.1056/NEJM200005113421901. [DOI] [PubMed] [Google Scholar]
- 5.Redline S, Yenokyan G, Gottlieb DJ, et al. Obstructive sleep apneahypopnea and incident stroke: The Sleep Heart Health Study. Am J Respir Crit Care Med. 2010;182:269–77. doi: 10.1164/rccm.200911-1746OC. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Ronksley PE, Hemmelgarn BR, Heitman SJ, et al. Excessive daytime sleepiness is associated with increased health care utilization among patients referred for assessment of OSA. Sleep. 2011;34:363–70. doi: 10.1093/sleep/34.3.363. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Yaggi HK, Concato J, Kernan WN, Litchman JH, Brass LM, Mohsenin V. Obstructive sleep apnea as a risk factor for stroke and death. N Engl J Med. 2005;353:2034–41. doi: 10.1056/NEJMoa043104. [DOI] [PubMed] [Google Scholar]
- 8.Kapur V, Blough DK, Sandblom RE, et al. The medical cost of undiagnosed sleep apnea. Sleep. 1999;22:749–55. doi: 10.1093/sleep/22.6.749. [DOI] [PubMed] [Google Scholar]
- 9.Flemons WW, Reimer MA. Measurement properties of the Calgary sleep apnea quality of life index. Am J Respir Crit Care Med. 2002;165:159–64. doi: 10.1164/ajrccm.165.2.2010008. [DOI] [PubMed] [Google Scholar]
- 10.Bennett LS, Barbour C, Langford B, Stradling JR, Davies RJ. Health status in obstructive sleep apnea. Am J Respir Crit Care Med. 1999;159:1884–90. doi: 10.1164/ajrccm.159.6.9808107. [DOI] [PubMed] [Google Scholar]
- 11.Fleetham J, Ayas N, Bradley D, et al. Canadian Thoracic Society 2011 guideline update: Diagnosis and treatment of sleep-disordered breathing. Can Respir J. 2011;18:25–47. doi: 10.1155/2011/506189. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Epstein LJ, Kristo D, Strollo PJ, the Adult Obstructive Sleep Apnea Task Force Clinical guideline for the evaluation, management and long-term care of obstructive sleep apnea in adults. J Clin Sleep Med. 2009;5:263–76. [PMC free article] [PubMed] [Google Scholar]
- 13.Flemons WW, Whitelaw WA, Brant R, Remmers JE. Likelihood ratios for a sleep apnea clinical prediction rule. Am J Respir Crit Care Med. 1994;150:1279–85. doi: 10.1164/ajrccm.150.5.7952553. [DOI] [PubMed] [Google Scholar]
- 14.Skjodt NM. Approach to outpatient management of adult sleep apnea. Can Fam Physician. 2008;54:1408–12. [PMC free article] [PubMed] [Google Scholar]
- 15.Tsai WH, Remmers JE, Brant R, Flemons WW, Davies J, Macarthur C. A decision rule for diagnostic testing in obstructive sleep apnea. Am J Respir Crit Care Med. 2003;10:1427–32. doi: 10.1164/rccm.200112-110OC. [DOI] [PubMed] [Google Scholar]
- 16.Fenton ME, Heathcote K, Bryce R, et al. The utility of the elbow sign in the diagnosis of OSA. Chest. 2014;145:518–24. doi: 10.1378/chest.13-1046. [DOI] [PubMed] [Google Scholar]
- 17.Netzer NC, Stoohs RA, Netzer CM, Clark K, Strohl KP. Using the Berlin Questionnaire to identify patients at risk for the sleep apnea syndrome. Ann Intern Med. 1999;131:485–91. doi: 10.7326/0003-4819-131-7-199910050-00002. [DOI] [PubMed] [Google Scholar]
- 18.Chung F, Yegneswaran B, Liao P, et al. STOP questionnaire: A tool to screen patients for obstructive sleep apnea. Anesthesiology. 2008;108:812–21. doi: 10.1097/ALN.0b013e31816d83e4. [DOI] [PubMed] [Google Scholar]
- 19.Severson CA, Tsai WH, Ronksley PE, Pendharkar SR. Identification of insomnia in a sleep center population using electronic health data sources and the insomnia severity index. J Clin Sleep Med. 2013;9:655–60. doi: 10.5664/jcsm.2830. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Vazquez JC, Tsai WH, Flemons WW, et al. Automated analysis of digital oximetry in the diagnosis of obstructive sleep apnea. Thorax. 2000;55:302–7. doi: 10.1136/thorax.55.4.302. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Chung F, Yang Y, Brown R, Liao P. Alternative scoring models of STOP-Bang questionnaire improve specificity to detect undiagnosed obstructive sleep apnea. J Clin Sleep Med. 2014;10:951–8. doi: 10.5664/jcsm.4022. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Whitelaw WA, Brant RF, Flemons WW. Clinical usefulness of home oximetry compared with polysomnography for assessment of sleep apnea. Am J Respir Crit Care Med. 2005;171:188–93. doi: 10.1164/rccm.200310-1360OC. [DOI] [PubMed] [Google Scholar]
- 23.Pereira EJ, Driver HS, Stewart SC, Fitzpatrick MF. Comparing a combination of validated questionnaires and level III portable monitor with polysomnography to diagnose and exclude sleep apnea. J Clin Sleep Med. 2013;9:1259–66. doi: 10.5664/jcsm.3264. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Váquez J-C, Tsai WH, Flemons WW, et al. Automated analysis of digital oximetry in the diagnosis of obstructive sleep apnoea. Thorax. 2000;55:302–7. doi: 10.1136/thorax.55.4.302. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Mulgrew AT, Fox N, Ayas NT, Ryan CF. Diagnosis and initial management of obstructive sleep apnea without polysomnography: A randomized validation study. Ann Intern Med. 2007;146:157–66. doi: 10.7326/0003-4819-146-3-200702060-00004. [DOI] [PubMed] [Google Scholar]