DOISNORE50: a perioperative sleep questionnaire predictive of obstructive sleep apnea and postoperative medical emergency team activation. A learning health system approach to sleep questionnaire development and screening

Abstract

Study Objectives:

Patients with obstructive sleep apnea (OSA) have a disproportionate increase in postoperative complications and medical emergency team activation (META). We previously introduced DOISNORE50 (Diseases, Observed apnea, Insomnia, Snoring, Neck circumference > 18 inches, Obesity with BMI > 32, R = are you male, Excessive daytime sleepiness, 50 = age ≥ 50) from sleep questionnaire ISNORED using features associated with increased odds of META in perioperative patients. Performance of DOISNORE50 (DOISNORE) had yet to be tested.

Methods:

The performance of DOISNORE was tested along with questionnaire ISNORED and STOP-BANG questionnaires among 300 out of 392 participants without known OSA referred to the sleep lab. In study 2, the performance of DOISNORE was tested among 64,949 lives screened in perioperative assessment clinic from 2016 to 2020.

Results:

Receiver operating characteristic curve demonstrated that best performance was achieved with responses, with area under curve of 0.801. DOISNORE’s predictability of OSA risk remained stable from 2018 to 2020 with area under curve of 0.78 and a Cronbach alpha of 0.65. Patients at high risk for OSA (DOISNORE ≥ 6) were associated with an increase of META (odds ratio 1.30, 95% confidence interval 1.12–1.45). Higher relative risk was noted among patients with congestive heart failure and hypercapnia.

Conclusions:

DOISNORE is predictive of OSA and postoperative META. Perioperative strategies against META should consider DOISNORE questionnaire and focused screening among patients with heart failure and hypercapnia.

Citation:

Namen AM, Forest D, Saha AK, et al. DOISNORE50: a perioperative sleep questionnaire predictive of obstructive sleep apnea and postoperative medical emergency team activation. A learning health system approach to sleep questionnaire development and screening. J Clin Sleep Med. 2022;18(8):1909–1919.

BRIEF SUMMARY

Current Knowledge/Study Rationale: In 1994, sleep questionnaire ISNORED was introduced as a sleep review of systems and was validated as predictive of obstructive sleep apnea. We previously derived DOISNORE50 from ISNORED using features associated with increased odds of medical emergency team activation in perioperative patients, but the performance of DOISNORE50 had yet to be tested.

Study Impact: This study demonstrated that DOISNORE50 is predictive of obstructive sleep apnea and validated its use among a robust group of perioperative patients as an addition to the existing number of obstructive sleep apnea screeners available. Patients who scored higher on DOISNORE50 were at increased risk for medical emergency team activation, with higher relative risk noted among patients with congestive heart failure and hypercapnia.

INTRODUCTION

Obstructive sleep apnea (OSA) is a common condition associated with inflammation, sympathetic nervous system activation, and cardiac and cerebrovascular morbidity.^1–6 With an overall estimated US prevalence of 4%–26%,^7–12 OSA has a greater prevalence among certain populations with specific comorbidities, including but not limited to patients afflicted with diabetes, congestive heart failure, stroke, Alzheimer’s disease, and chronic opioid use.^13–17 It is estimated that approximately 83% of men and 92% of women with OSA are undiagnosed.¹⁸ In addition to a higher prevalence in the general population, more than 20% of patients in preoperative clinics are at risk for OSA.^19–21 These patients may have increased days of mechanical ventilation, intensive care unit days, and postoperative length of stay.^22–27 Despite these risks, providers under-recognize OSA in preoperative evaluations.²⁰ The need for improved recognition of patients at risk has been identified as a priority.²⁸

Sleep questionnaires have been developed to aid in preoperative identification of patients at risk for OSA. The Berlin and STOP-BANG (SB), among others, have been tested and validated in preoperative and other populations to identify patients at risk for OSA.^{19,21,29–32} These and other questionnaires incorporate a combination of sleep symptoms and anthropometric measures including maxillofacial measures (Mallampati score), neck circumference, body mass index (BMI), and medical conditions. Thresholds and combinations for BMI, neck circumference, and age have been helpful and have shown predictability, but the predictability varies depending on clinical population and ethnic group studied.^33–36 Utilizing these tools has in some populations led to a reduction in hospital events including rapid response systems (among general hospital admissions).³⁷

Some of these questionnaires have gone through multiple iterations to better identify patients at risk of OSA. Challenged by these factors, a learning health system approach can contribute to the development of an OSA questionnaire beyond its initial iteration. When “science, informatics, incentives, and culture are aligned for continuous improvement and innovation”,³⁸ these questionnaires can become a major clinical outcome measure that may also be predictive of medical emergency team activation (META).

In 1994, ISNORED (Insomnia, Snoring, Not breathing, Obesity, R nonrefreshing sleep, Excessive daytime sleepiness, Diseases and drugs) was introduced by Haponik and colleagues to the American Academy of Sleep Medicine as a sleep review of systems to identify patients with sleep disorders.³⁹ Since that time, it has been used in several clinical venues, including house officers’ objective simulated clinical examinations, general medicine clinics, general medicine ward services, geriatric ward services, and perioperative clinics. ISNORED (IS) assisted providers in recognizing the risk of OSA and enhanced recognition of sleep disorder breathing within these cohorts.^40–42 In a 2001 American Thoracic Society conference poster presentation and abstract publication, we introduced the concept that when all questions were equally weighted and affirmed by the patient, IS was predictive of OSA.⁴²

In 2015, we conducted a prospective study with a cohort of 898 perioperative patients referred for polysomnogram. The self-administered IS questionnaire combined with anthropometric cutoffs, including neck circumference of 18 inches, body mass index of 32, and male sex, demonstrated 100% positive predicative value for identifying OSA. Among the study patients, 3% sustained META. Multivariate analysis was used to identify comorbidities predicative of META.⁴³ IS was then modified to identify patients not only at risk of OSA but also of META in perioperative settings. Questions of insomnia, snoring, observed apnea, and excessive daytime sleepiness were predictive and remained in the questionnaire. Disease states associated with increased odds of META such as stroke, atrial fibrillation, and hypertension were included. Chronic pain, sedative use, history of chronic obstructive pulmonary disease, or coronary artery disease did not bear significant predictability. Anthropometric cutoffs of neck circumference of 18 inches for males/17 inches for females, body mass index of 32, and male sex were added to complete the questionnaire. ISNORED was refashioned to DOISNORE50 (Diseases including stroke/atrial fibrillation/hypertension, Observed apnea, Insomnia, Snoring, Neck circumference over 18 inches, Obesity, R are you male, Excessive daytime sleepiness, 50 age ≥ 50).

Previous study result: derivation of DOISNORE from prospective cohort

As previously reported, of the 898 patients who received a sealed envelope containing IS and SB, 575 completed and returned the questionnaire during their initial preoperative evaluation within the Preoperative Assessment Clinic.⁴² Univariate analysis, multivariate analysis, and receiver operating characteristics (ROC) predicted best threshold for diagnosis of OSA and META. IS was then refashioned to DOISNORE. Following completion of the questionnaires, 392 patients without prior diagnosis of OSA were referred to our sleep laboratory.

Study 1: performance of DOISNORE in sleep lab referral

In a similar fashion in 2015, prior to attending an in-lab polysomnogram, 392 consecutive patients without prior diagnosis of OSA or sleep-disordered breathing were provided a sealed envelope containing DOISNORE, IS, and SB. Consent was waived as approved by the Institutional Review Board. Patients were allowed time to complete the questionnaire during their visit and to ask questions for clarification. Independent of sleep screening questionnaire responses, demographic data including age, sex, BMI as recorded in entry to the clinic, and comorbidities as documented by the admitting and consulting providers’ medical record were collected. Neck circumference was measured in person by clinic staff. Screened comorbidities included any history of the following: myocardial infarction, congestive heart failure, stroke, diabetes mellitus, atrial fibrillation, dysrhythmia, chronic pain, chronic obstructive pulmonary disease (COPD), asthma. Prior to entry into the study, patients were screened and excluded from study if they admitted to a diagnosis of OSA and/or were on positive airway pressure or alternative sleep apnea treatment.

Analysis of the electronic medical record was conducted to confirm no prior diagnosis of OSA. We used logistic regression to determine the association between the sleep screen responses (SB, IS, DOISNORE) and OSA. Receiver operating characteristic curves and area under the curve statistics were computed to assess the model’s predictability with regard to the performance of SB, IS, and DOISNORE questionnaires. Sensitivity, specificity, positive predictive values, and negative predictive values of self-administered questionnaires were calculated based on apnea-hypopnea index (AHI) ≥ 5 events/h or respiratory event index ≥ 5 events/h, per Centers for Medicare & Medicaid Services/American Academy of Sleep Medicine guidelines.^44,45 Of the 392 participants, 92 were excluded from analysis due to presence of central sleep apnea during sleep study or completion of fewer than 3 questions on any of the 3 questionnaires.

Study 2: DOISNORE implementation and analysis

In August 2015, as part of a safety and quality initiative to provide risk stratification of OSA and to identify perioperative patients at risk for postoperative META, DOISNORE sleep screening was initiated among all patients referred to perioperative assessment clinic and surgical nurse navigation center (PAC/SNC). Completion of the questionnaire was required for each patient encounter either by telephone or face-to-face. Certain discrete elements—age, BMI within last 30 days, sex—were extracted from existing data in the electronic medical record (EMR) and to complete the questionnaire responses. Neck circumference measurements were completed in all face-to-face visits (98.5%). A majority of visits were telephone-screened, and 24% of those respondents did not know their neck circumference. If a condition or anthropometric measure was unknown, including BMI or neck circumference during telehealth/phone consultation, a value of “no” was placed on the questionnaire.

Then in late 2016, implementation of additional patient safety measures in response to sleep questionnaire thresholds was introduced. From 2015 through 2020, 70,974 lives were evaluated, of which 64,949 (91.5%) completed the DOISNORE questionnaire. Polysomnograms (baseline, split night, and home sleep testing) of 7,433 patients were included in the analysis of the total 16,101 sleep studies (including positive airway pressure titration or repeated baseline studies) (Andrew M. Namen, MD, unpublished data, 2022). Inclusion and exclusion criteria are outlined in Figure 1. Demographic data including age, sex, BMI as recorded upon entry to the clinic, and comorbidities as documented by the admitting and consulting providers’ medical records were collected. Screened comorbidities included any history of the following: myocardial infarction, congestive heart failure, stroke, diabetes mellitus, atrial fibrillation, dysrhythmia, chronic pain, pulmonary disorder, COPD, malignancy, liver disorder, hypercapnia, and AIDS. American Society of Anesthesiologists Physical Status Classification System class, and general categories of surgery (including vascular, cardiovascular, neurosurgical, gastrointestinal, urologic, solid organ transplantation [such as kidney, heart, lung], orthopedic, and obstetrical/gynecologic) were also collected from a chart review.

Figure 1. Inclusion and exclusion criteria for study 2.

Low Risk, At Risk, and High Risk for OSA is defined by DOISNORE50 < 4, ≥ 4 and < 6, and ≥ 6, respectively. OSA = obstructive sleep apnea, SNC/PAC = surgical navigation center and perioperative assessment clinic, WFBMC = Wake Forest Baptist Medical Center.

A hard stop function within the EMR was inserted to ensure all patients referred would be screened either by phone interview with a nurse or face-to-face by providers. Only a nurse or clinician could enter a value in EMR for the information requested in the sleep questionnaire before the questionnaire could be closed and the patient advanced to other clinical activities. In order to avoid provider/nurse interpretation of patients’ sleep symptoms, staff was strongly encouraged to select yes or no responses as stated by the patient. Those with preexisting OSA and those naive to diagnosis were identified based on either responses to 3 prescreen OSA questions (Do you have OSA?; Are you on continuous positive airway pressure?; Are you using CPAP currently?) at time of screening or EMR evidence of OSA at the time of screening, including attended or unattended polysomnogram evidence of AHI or respiratory event index ≥ 5 events/h, previous positive airway pressure titration prescription, or evidence of positive airway therapy. Central sleep apnea–noted patients naive to OSA diagnosis underwent further sleep screening with the DOISNORE questionnaire. Based on operational characteristics in Study 1, patients who were “Low Risk”, “At Risk”, or “High Risk” for OSA were defined by DOISNORE < 4, ≥ 4 and < 6, and ≥ 6, respectively.

Following screening, electronic medical record reminders and patient identifiers of risk or known OSA were created and implemented. Performance characteristics of DOISNORE’s identification of patients with OSA were ascertained. Overall and yearly analysis of questionnaire responses included sensitivity, specificity, and positive predictive and negative predictive values. Receiver operating characteristic curves were generated.

META

Prospectively collected META consisted of number of events associated with rapid response team (RRT) activation, code blue activation, and code stroke activation: patients with respiratory distress prompting an RRT, reintubation (RI), code blue (CB), or code stroke (CS). RRT was determined as documented by a rapid response team member (nurses and for respiratory events a respiratory therapist) responding to the alert of critical change in patient respiratory status. CB alerts were for primary cardiovascular or respiratory collapse (as documented by a physician, nurse, and respiratory therapist). CS was identified as sudden neurologic deterioration, a concerning cerebrovascular event that triggered the attendance of a neurologist or other support staff at the patients’ bedside. RI was identified as an event following extubation after completion of surgery and within the postanesthesia care unit or during that admission. Although formal designation of META was made by the specific team addressing the CB, CS, or reintubation, for the purpose of our analysis, the term META incorporated all events as outlined above that prompted an RRT.⁴⁶ In addition, if an RRT was called and the patient’s condition declined, the event was counted once, from RRT to CB, CS, or RI.

Statistical analysis

All data elements were extracted from the electronic health record system at Wake Forest Baptist Medical Center. Patient information extracted via our electronic medical record included all demographics, sleep labs results, preoperative assessment clinic visit, comorbidity, and diagnosis information along with postoperative outcomes.

Descriptive statistics associating risk factors with sleep apnea and META were performed. Analysis of variance and Chi-square tests were used for continuous and categorical predictors to determine which variables were associated with OSA (defined as an AHI measure greater than or equal to 5 events/h) and META (RRT, RI, CB, and CS), respectively. We used logistic regression to determine the association between the sleep screen responses (DOISNORE) and OSA, having removed patients with previous history of OSA. Receiver operating characteristic curves and area under the curve statistics were computed to assess the models’ predictive ability with regard to the performance of DOISNORE questionnaires. Sensitivity, specificity, positive predictive values, and negative predictive values were calculated based on OSA. The variation of DOISNORE with AHI groups (Mild with AHI ≥ 5 events/h, Moderate with AHI ≥ 15 events/h, and Severe with AHI ≥ 30 events/h) for patients who underwent sleep study with sensitivity, specificity, positive predictive value, and negative predictive values is shown in Table S1 ^{(636.4KB, pdf)} in the supplemental material. Additionally, multivariate logistic regression analysis was performed to determine the risk of comorbidities and DOISNORE assessment on META. All data were analyzed using SAS 9.3 (SAS Institute Inc., Cary, NC). Statistical models were conducted to identify best predictive thresholds for OSA in each questionnaire.

Multivariate propensity analysis was conducted between matched populations of 61,036 patients with length of stay more than 2 days not screened with DOISNORE and 18,698 DOISNORE-screened patients with inpatient length of stay of 2 days or more. Nonscreened patients included as a control group did not go through the perioperative assessment clinic and were propensity-matched randomly in 1:1 ratio based on surgical categories and demographic characteristics: sex, ethnicity, race, age, hospital length of stay, and comorbidity, including hypercapnia, human immunodeficiency virus infection (AIDS), cerebrovascular disease, pulmonary disorder, atrial fibrillation, COPD, congestive heart failure, diabetes mellitus, tumor malignancy, liver disorder, and mitochondrial infection (per Charlson comorbidity index). We used a caliper width of 0.2 for propensity score matching as recommended by Austin.^47,48 We analyzed the equalized balance of covariates between propensity score–matched groups using absolute standardized differences. We considered an absolute standardized difference score > 0.1, after matching, as indicative of potential residual confounding, and we adjusted for such factors directly in the analyses by comparing the groups in the outcome.

RESULTS

Study 1: demographic data and analysis

Demographics and comorbidities of 392 consecutive study patients naive to diagnosis of OSA are displayed in Table 1. Of 392 screened patients 300 were included in the predictive analytics (92 excluded for diagnosis of central sleep apnea, other non-OSA diagnosis including Cheyne-Stokes respirations, and those who answered fewer than 3 questions). Analysis of DOISNORE questionnaire prediction of OSA for different DOISNORE scores is shown in Table 2A. Except for the score of 9 or above, each point increase in DOISNORE was associated with higher positive predicative value and each point decrease with higher negative predictive value. Odds ratio of mild, moderate, and severe OSA as defined by AHI cutoffs with increasing DOISNORE scoring is shown in Table 2B. Receiver operating characteristics highlight the best performance of DOISNORE compared to that of SB and IS as shown in Figure 2. At score ≥ 6, DOISNORE has higher area under the curve compared to IS ≥ 4, SB ≥ 6, and SB or IS ≥ 5. For every point increase above 4 on the DOISNORE50, there is an associated increase in AHI by 4.5 events per hour.

Table 1.

Demographics and comorbidities in patients without previous OSA diagnosis who were referred to the sleep lab (n = 392).

Demographics	All (n = 392)	AHI ≥ 5 events/h (n = 299)	AHI < 5 events/h (n = 93)
Age, mean (SD)	51 (16)	52 (15)	45 (17)
Male, n (%)	196 (50%)	164 (55%)	32 (34%)
BMI, mean (SD)	34 (8)	35 (8)	32 (7)
Comorbidities	All (n = 392)	AHI ≥ 5 events/h (n = 299)	AHI < 5 events/h (n = 93)
HTN	215 (55%)	173 (58%)	42 (45%)
CHF	25 (6%)	21 (7%)	4 (4%)
COPD	41 (10%)	29 (10%)	12 (13%)
Chronic pain	27 (7%)	23 (8%)	4 (4%)
DM	90 (23%)	81 (27%)	9 (10%)
Asthma	50 (13%)	37 (12%)	13 (14%)
CVA	19 (5%)	16 (5%)	3 (3%)
Afib	25 (6%)	23 (8%)	2 (2%)

Significant number of positive obstructive sleep apnea patients are associated with male predominance, diabetes mellitus, atrial fibrillation, chronic pain, and hypertension. Afib = atrial fibrillation, AHI = apnea-hypopnea index, BMI = body mass index, CHF = congestive heart failure, COPD = chronic obstructive pulmonary disease, CVA = cerebrovascular accident, DM = type 2 diabetes mellitus, HTN = hypertension, OSA = obstructive sleep apnea, SD = standard deviation.

Table 2.

Predicative values of DOISNORE50 (A) and DOSISNORE50 scoring vs odds of OSA severity (B).

(A) Positive predicative value and negative predictive value of DOISNORE at different scores identify best thresholds for identifying patients’ probability of obstructive sleep apnea.
DOISNORE50 Score		Sensitivity	Specificity		PPV		NPV
Score ≥ 3		98%	8%		78%		62%
Score ≥ 4		93%	26%		80%		50%
Score ≥ 5		82%	42%		82%		41%
Score ≥ 6		58%	81%		84%		34%
Score ≥ 7		41%	85%		91%		30%
Score ≥ 8		18%	96%		96%		27%

(B) Increasing DOISNORE scoring is also associated with increasing odds ratio of OSA.
	OR (95% CI)
DOISNORE50 Score Reference = Score 3	Mild OSA AHI ≥ 5 events/h	Moderate OSA AHI ≥ 15 events/h	Severe OSA AHI ≥ 30 events/h
Score 4	2.45 (1.58, 3.80)	3.32 (2.02, 5.45)	1.39 (0.84, 2.32)
Score 5	3.34 (2.21, 5.04)	2.47 (1.50, 4.06)	1.73 (1.08, 2.77)
Score 6	5.21 (3.26, 8.33)	4.09 (2.50, 6.70)	1.94 (1.20, 3.20)
Score 7	6.7 (3.80, 11.70)	4.7 (2.80, 8.00)	5.2 (3.30, 8.10)

AHI = apnea-hypopnea index, CI = confidence interval, NPV = negative predicative value, OR = odds ratio, OSA = obstructive sleep apnea, PPV = positive predicative value.

Figure 2. Comparison of ROC curves between DIS, SB, and IS.

ROC curves highlight sensitivity over 1 minus specificity and establish the probability of best performance of DOISNORE ≥ 6 (AUC 0.801), in comparison to SB ≥ 5 (AUC 0.772), or IS ≥ 4 (AUC 0.701). AUC = area under curve, DIS = DOISNORE50, IS = ISNORED, ROC = receiver operating characteristic, SB = STOP-BANG.

Study 2: demographic data and analysis

We propensity-matched the demographics and comorbidities of the 61,036 patients who were not screened and not evaluated at SNC/PAC and the 18,698 DOISNORE-screened patients who had inpatient stay of 2 days or more without any previous history of OSA or an active or inactive use of positive airway pressure therapy. Table 3 characterizes the groups. A trend of younger patients and female predominance is noted in the DOISNORE-screened group. A significant number of patients with pulmonary disorders, AIDS, and myocardial infarction are noted in the nonscreened cohort, and a slightly greater percentage of patients with atrial fibrillation in the DOISNORE group.

Table 3.

Demographic and comorbidities of matched cohort, n = 79,734.

Demographics	Not Screened (n = 61,036)	DOISNORE50 (n = 18,698)	P
Age, mean (SD)	55.81 (20.9)	61.2 (15.8)	.292
Male, n (%)	28,883 (47.3%)	8,281 (44.3%)	< .001
Race, n (%)			< .001
White	44,237 (72.5%)	14,913 (79.8%)
Black	11,780 (19.3%)	2,907 (15.5%)
Other	5,019 (8.2%)	878 (4.7%)
BMI > 32, n (%)	11,265 (23.4%)	4,512 (24.1%)	.284
OSA, AHI ≥ 5 events/h, n (%)	301 (0.5%)	112 (0.6%)	.361
Comorbidity	n (%)		P
Hypercapnia	2,540 (4.2%)	353 (1.9%)	.133
AIDS	306 (0.5%)	62 (0.3%)	.026
CVA	5,395 (8.8%)	886 (4.7%)	.164
CHF	8,560 (14%)	1,916 (10.2%)	.071
Pulmonary disorder	12,816 (21%)	3,676 (19.7%)	.033
Afib	6,317 (10.3%)	2,201 (11.8%)	.045
COPD	9,976 (16.3%)	3,965 (21.2%)	.125
DM	10,757 (17.6%)	2,858 (15.3%)	.063
Tumor Malignancy	2,390 (3.9%)	1,656 (8.9%)	.202
Liver Disorder	1,946 (3.2%)	357 (1.49%)	.081
MI	5,605 (9.2%)	1,267 (6.8%)	.011

Demographics and comorbidities of 61,036 patients not screened and 18,698 patients screened with DOISNORE50. Afib = atrial fibrillation, AHI = apnea-hypopnea index, AIDS = acquired immunodeficiency syndrome, BMI = body mass index, CHF = congestive heart failure, COPD = chronic obstructive pulmonary disease, CVA = cerebrovascular accident, DM = type 2 diabetes mellitus, MI = myocardial infarction, OSA = obstructive sleep apnea, SD = standard deviation.

Study 2: DOISNORE implementation and evaluation (retrospective field study analysis)

Table 4 demonstrates relative risk of META comparing the DOISNORE group with non- DOISNORE50 group. Highest relative risk of META was noted among patients with hypercapnia, malignancy, and congestive heart failure (Table 4).

Table 4.

Relative risk for META comparing DOISNORE50 group with nonscreened group using propensity score matching.

Comorbidity	Relative Risk (95% CI)
Hypercapnia	3.69 (3.38–4.02)
AIDS	1.45 (0.91–2.18)
CVA	1.22 (1.08–1.36)
CHF	1.64 (1.52–1.76)
Pulmonary disorder	1.32 (1.23–1.41)
Afib	1.49 (1.19–1.84)
COPD	1.15 (1.05–1.26)
DM	0.96 (0.88–1.03)
Malignancy	1.90 (1.74–2.08)
Liver Disorder	1.38 (1.16–1.63)
MI	1.20 (1.09–1.33)

Highest relative risk of META was noted among patients with hypercapnia, malignancy, and congestive heart failure. Afib = atrial fibrillation, AIDS = acquired immunodeficiency syndrome, CHF = congestive heart failure, CI = confidence interval, COPD = chronic obstructive pulmonary disease, CVA = cerebrovascular accident, DM = type 2 diabetes mellitus, META = medical emergency team activation, MI = myocardial infarction, RR = risk ratio.

Figure 3 demonstrates overall predictability of OSA across years from 2016 to 2020.

Figure 3. Overall ROC curves of DOISNORE50 from 2018, 2019, and 2020.

ROC curves demonstrate stable predictability across years of preoperative screening regardless of other initiatives that were conducted within the PAC/SNC environment. In a mixed model of sleep questionnaire delivery, by phone or face-to-face contact, and during the COVID-19 pandemic, performance of the questionnaire did not wane. ROC = receiver operating characteristic, SNC/PAC = surgical navigation center and perioperative assessment clinic.

Figure 4 shows relative risk of META and RRT using multivariate logistic regression on propensity-matched cohort between DOISNORE and non-DOISNORE patients with length of stay more than 2 days from 2016 to 2020. We observed significant risk of META among patients “At Risk” (DIS ≥ 4 and < 6) and “High Risk” (DOISNORE ≥ 6) for OSA vs surgical patients not screened. Figure 4 shows a statistically significant increase in RRT noted for patients “At Risk” and “High Risk” for OSA. This finding supports that DOISNORE is an independent predictor for META and RRT.

Figure 4. Box plot showing varying risks of META (top) and RRT (bottom) for different DOISNORE groups in comparison to propensity-matched non-DOISNORE controls after adjustment for comorbidities.

META = medical emergency team activation, OSA = obstructive sleep apnea, RRT = rapid response team.

DISCUSSION

We describe a learning health system approach to remodeling IS (a sleep review of symptoms) into a questionnaire predictive of both OSA and META. The sleep questionnaire in conjunction with sex and other anthropometric indices was highly specific for OSA. As an acronym, it had a demonstrated usefulness in assisting residents and interns to recall and obtain a sleep history. From our analysis of perioperative sleep screening and analysis of postoperative META, we refashioned the IS to DOISNORE. Its predictive characteristics for OSA were validated among a prospective cohort of patients referred to an academic sleep lab. With predictive performance comparable to known sleep questionnaires, DOISNORE was subsequently implemented in a progressive perioperative sleep screening process. Follow-up analysis of results after implementation of this program confirmed its predictability and reliability of diagnosing OSA over years. Propensity-matched multivariate analysis demonstrates that DOISNORE is also predictive of META when applied preoperatively. This study not only establishes DOISNORE as a validated tool with year-over-year stable operational characteristics to predict OSA but by virtue of the learning health systems approach demonstrates predictability of META among matched cohorts.

Threshold selection

A cutoff of 4 or more on the DOISNORE questionnaire was established in Study 1 as predictive for OSA based on sensitivity and specificity analysis. This score identified sleep lab–referred patients as “At Risk” for OSA in 73% of patients.

A score of 4 or more (“At Risk” and “High Risk”) was associated with statistical risk of META vs no additional risk seen in the “Low Risk” population (DOISNORE < 4). Among propensity-matched populations, “Low Risk” by DOISNORE had no significant risk for META compared to nonscreened patients. This data supports the threshold ≥ 4 as a clinical threshold for significant risk of META. Further, less than 4 on the DOISNORE was associated with a lower risk of reintubation, supportive of a meaningful threshold.

Although the test characteristics may warrant a lower threshold of 3, the predictive threshold for clinical outcome of META would not support this given a lack of statistical significance. The number of patients that would be included as “At Risk” would likely create a population of false positive “At Risk” for OSA patients and lead to unnecessary testing. In using DOISNORE a threshold of 4 obligates at least 1 sleep symptom. Not all patients with sleep apnea or sleep-disordered breathing demonstrate symptoms of snoring.⁴⁹ Older patients and women tend to report snoring less often than their younger and male counterparts.^50–52 Therefore, other sleep symptoms of nocturnal choking, witnessed apnea, nonrefreshing sleep, multiple nocturnal arousals, and self-perceived excessive daytime sleepiness are helpful and contribute to the predictability of DOISNORE.

The merit of using sleep questionnaires in the preoperative setting is to identify patients at risk for OSA; however, the ability to identify those at risk for major postoperative hospital events, including medical emergency team activation, has been supported in the literature.^22–27,53 In our initial prospective cohort of preoperative patients in 2015, 3.1% required META, represented by RRT, RI, CB, and CS.^54,55 Because those who had a history of hypertension, atrial fibrillation, and stroke had increased odds of META, using an OSA questionnaire with these comorbid conditions did appear to favorably affect the questionnaire’s predictability of META, particularly following a propensity-matched analysis encompassing 18,698 patients screened with DOISNORE. In our present study, the prevalence of META persisted at 3%, higher among “At Risk” and “High Risk” OSA. Among 64,949 unique patients prior to surgery, the burden of sleep apnea based on sleep screening is common and consistent with other published data, suggesting up to 66% of preoperative patients were “At Risk” for OSA or known to have OSA. In our cohort, 28% of patients presenting to the PAC had a known diagnosis of OSA. An additional 38% were classified “At Risk” or “High Risk” for OSA by DOISNORE. This quantifies the enormity of this proposed condition, and therefore a practical and disciplined approach to screening is required.

This study suggests that a sleep questionnaire crafted from sleep review systems applied in a perioperative patient cohort and redesigned to predict META successfully identifies patients when applied as a quality and safety initiative for OSA and has a positive predicative value of > 90%. DOISNORE has demonstrated its predictability and reliability over years. Its internal accuracy by Cronbach alpha of 0.65 complements and is comparable to other large patient screening tools such as prostate screener and others.

The perioperative experience is especially challenging; it imposes demands on perioperative staff efficiency and the ability to identify other disease states that contribute to a patient’s postoperative complications. Given their proven ability to identify OSA and patients at risk for META, the use of OSA screeners tested and validated in a patient self-administered approach is a practical long-term process for perioperative clinics, especially in the COVID-19 pandemic. The use of DOISNORE is predictive of OSA, and this study validated its use among a robust group of perioperative patients and adds to the existing number of OSA screening tools available.^19,29,30 It also assesses the risk of postoperative META.

Future research to validate self-administered OSA questionnaires could utilize a dynamic “At Risk” stratification and a screener with a robust sleep-specific review of systems. A sleep review of symptoms specific only for OSA limits a potentially robust program by neglecting questions necessary for prediction in other populations, including inpatient, general medicine services, and the geriatric ward.

Limitations

Although predictive and reliable, not all questions in DOISNORE performed well. Given the limitations of patient’s knowledge in regards to their neck circumference, these important metrics were left unanswered by many patients when self-administered or during phone call screening given estimates of those at risk for OSA in a preoperative clinic were similar to other published reports.^19,20 Specifically, neck circumference was not answered in a significant number of phone interview patients. This is consistent with our previous publication using IS.⁴² This raises the need of an online and alternative approach to measuring neck circumference and other anthropometric measures to avoid the need for a provider visit, especially in the COVID-19 pandemic era. It should also be noted that the presence of neck circumference was not associated with increased predictability of META in our present or previous analysis. Alternative anthropometric measurements that do not require face-to-face contact, like computer-based, image-interface models, may be more suitable and predictive.⁵⁶

Studies have outlined that Cronbach values > 0.6 are supportive of a questionnaire’s reliability.⁵⁷ The higher the value, the greater the confidence in the questionnaire’s performance for a given population. Our analysis demonstrated that the DOISNORE questionnaire demonstrated a Cronbach alpha of 0.65. Similarly, Cronbach alpha among other sleep questionnaires, such as Berlin and STOP-BANG, ranged from 0.62 to 0.8 depending on the population examined.^58,59 Although the Cronbach alpha value for DOISNORE in this large population provides confidence in its application, further improvements and studies are needed as part of the learning health system approach to improve performance.

In our application of DIS, all questions were weighted equally. Based on previous multivariate analyses and present observations, placing additional weight on questions related to snoring, choking, and disease, which have greater predictability, may enhance DOISNORE’s performance.

Age cutoff and male sex may be associated with previously formulated biases regarding the prevalence of OSA. Previous publications have identified that Asians and postmenopausal women tend to be underdiagnosed as result of these biases.⁵⁰ More sophisticated modeling with a dynamic adjustment of risk factors may also enhance predictability.

Retrospective analysis of a sleep registry has inherent biases that do not account for all influences affecting a sleep screen’s predictability. However, the robust size of the population observed and the rigorous approach utilizing propensity-matched multivariate analysis supports our findings. Application of self-administered questionnaires is feasible and supported by patient performance: 91% of 6 questions were completed using the DOISNORE questionnaire. The ability to identify patients with OSA appeared preserved based on a population of known diagnosis or polysomnogram-proven OSA.⁴² This supports the probability that provider administration of questionnaires may not be necessary to identify OSA.

Design differences between Study 1 using paper documents and Study 2 using an EMR approach were associated with additional challenges. In Study 1, patients were able to skip questions if they preferred or could not answer. The benefit of the EMR with a built-in hard stop prevented this error. However, regardless of the model applied (paper or electronic), the performance of DOISNORE remained consistent and demonstrated its general application in either environment.

When selecting sleep questionnaires, institutions must decide which outcome is best and which questionnaire enhances the EMR experience for the patient and the provider. Sleep-related symptoms, including choking, snoring, and daytime fatigue, have been used in combination in multiple questionnaires and are helpful to the diagnosis of sleep apnea. After application of the Plan-Do-Study-Act cycle, the predictive questions assisted in validating DOISNORE, and include other self-perception questions regarding nocturnal arousals and nonrefreshing sleep. With an expanded number of sleep-related symptoms, databases can reconfigure sleep symptoms combined with differing comorbidities applied to multiple populations with the potential to improve risk assessment for sleep apnea and META. Since we were unable to compare DOISNORE with other popular questionnaires in the large cohort Study 2, we are uncertain of the benefit or weakness. We believe the sleep-related symptoms within DOISNORE benefits the EMR database’s ability to identify META. However, comparison studies may be helpful.

CONCLUSIONS

Our acronym ISNORED developed to promote sleep histories has been refined and now incorporates major comorbidities associated with OSA. Successive iterations informed by laboratory and perioperative clinic experience have culminated in DOISNORE, a tool highly predictive of META. This extends the application of the learning health system to sleep disorders and perioperative medicine, promoting and enhancing a patient-centered focus on safety. Over time, despite new initiatives in the PAC and changes to home sleep testing (whether flow vs pulse tonometry device), performance of the questionnaire has remained stable. The percentage of our undiagnosed population identified “At Risk” and at “High Risk” (combined with known disease) has improved greatly the prospective recognition of those patients at increased risk of META and enhanced a culture of safety.

ABBREVIATIONS

AHI

apnea-hypopnea index

BMI

body mass index

CB

code blue

COPD

chronic obstructive pulmonary disease

CS

code stroke

DOISNORE

DOISNORE50

EMR

electronic medical record

IS

ISNORED

META

medical emergency team activation

OSA

obstructive sleep apnea

PAC

perioperative assessment clinic

RI

reintubation

RRT

rapid response team

SB

STOP-BANG

SNC

surgical nurse navigation center

DISCLOSURE STATEMENT

All authors have seen and approved the manuscript. The authors declare that they have no known competing financial interests, personal relationships, or connections that may influence the work reported in this paper.