Polysomnographic Endotyping to Select Patients with Obstructive Sleep Apnea for Oral Appliances

Abstract

Rationale: Oral appliance therapy is efficacious in many patients with obstructive sleep apnea (OSA), but prediction of treatment outcome is challenging. Small, detailed physiological studies have identified key OSA endotypic traits (pharyngeal collapsibility and loop gain) as determinants of greater oral appliance efficacy.

Objectives: We used a clinically applicable method to estimate OSA traits from routine polysomnography and identify an endotype-based subgroup of patients expected to show superior efficacy.

Methods: In 93 patients (baseline apnea–hypopnea index [AHI], ≥20 events/h), we examined whether polysomnography-estimated OSA traits (pharyngeal: collapsibility and muscle compensation; nonpharyngeal: loop gain, arousal threshold, and ventilatory response to arousal) were associated with oral appliance efficacy (percentage reduction in AHI from baseline) and could predict responses to treatment. Multivariable regression (with interactions) defined endotype-based subgroups of “predicted” responders and nonresponders (based on 50% reduction in AHI). Treatment efficacy was compared between the predicted subgroups (with cross-validation).

Results: Greater oral appliance efficacy was associated with favorable nonpharyngeal traits (lower loop gain, higher arousal threshold, and lower response to arousal), moderate (nonmild, nonsevere) pharyngeal collapsibility, and weaker muscle compensation (overall R² = 0.30; adjusted R² = 0.19; P = 0.003). Predicted responders (n = 54), compared with predicted nonresponders (n = 39), exhibited a greater reduction in AHI from baseline (mean [95% confidence interval], 73% [66–79] vs. 51% [38–61]; P < 0.0001) and a lower treatment AHI (8 [6–11] vs. 16 [12–20] events/h; P = 0.002). Differences persisted after adjusting for clinical covariates (including baseline AHI, body mass index, and neck circumference).

Conclusions: Quantifying OSA traits using clinical polysomnography can identify an endotype-based subgroup of patients that is highly responsive to oral appliance therapy. Prospective validation is warranted.

Oral appliances, intraoral devices worn during sleep to protrude the mandible, are increasingly used as a treatment alternative for obstructive sleep apnea (OSA) (1). The literature indicates that oral appliance therapy reduces OSA severity (indicated by the apnea–hypopnea index [AHI]) by an average of 50–70% (2–6). Although not as efficacious as continuous positive airway pressure (CPAP) at ameliorating OSA, oral appliance therapy has a proven positive impact on sleepiness, blood pressure, and quality of life (7–11). In addition, studies suggest that treatment outcomes of oral appliances and CPAP are similar (12), reflecting superior adherence to oral appliance therapy. Efficacy of oral appliance therapy is variable across patients with OSA. Without experimental testing in each patient (13, 14), there is currently no clinically applicable means to predict the likelihood of oral appliance therapy success before treatment prescription (15).

The variability in oral appliance efficacy across patients with OSA may be attributed to the extent to which OSA endotypic traits (pharyngeal: collapsibility and muscle compensation; nonpharyngeal: loop gain, arousal threshold, and ventilatory response to arousal) contribute to the pathogenesis of the condition (16–25). Small, detailed physiological studies have revealed two key traits associated with reduced oral appliance efficacy: namely, greater pharyngeal collapsibility, whereby the severity is beyond the scope of treatment (26), and higher loop gain (i.e., ventilatory control instability) that cannot be resolved with anatomical interventions (20). Notably, these detailed studies were performed in specialized laboratories using invasive instrumentation, which is out of reach for clinical sleep laboratories.

Recently, our team has developed a method for estimating the key endotypic traits causing OSA from routine diagnostic polysomnography (27–29). The method is based on automated analysis of a surrogate uncalibrated ventilation signal (derived from nasal pressure) from which ventilatory drive is estimated and OSA endotypic traits are characterized. In the present study, we applied this method to diagnostic polysomnography of patients treated with oral appliance therapy. We aimed to 1) determine whether greater oral appliance efficacy is associated with favorable nonpharyngeal OSA endotypes (i.e., lower loop gain, higher arousal threshold, and lower ventilatory response to arousal) with the ultimate goal of 2) defining an endotype-based subgroup of patients with OSA (prediction model) who are most likely to benefit from oral appliance therapy (predicted responders).

Methods

Subjects

In this study, we performed a secondary analysis of polysomnographic data from previous oral appliance research studies (which included newly diagnosed patients with OSA with baseline AHI >10 events/h) (12, 30, 31). Patients were included in our analysis if they had a baseline polysomnography-derived AHI greater than or equal to 20 events per hour (prespecified), which was selected to minimize the influence of night-to-night variability (noise) on the percentage reduction in AHI with treatment (efficacy) (32). For example, a 75% reduction in AHI from 20 to 5 events per hour was considered more reliable than the same from 10 to 2.5 events per hour. (The latter is within the expected night-to-night variability; i.e., ∼10 events/h.)

Polysomnographic data (n = 94) were taken from three parent studies: a three-center randomized crossover trial (12) comparing health outcomes of CPAP therapy with oral appliance therapy in patients who were recommended both treatments (n = 108; 80% males; mean ± standard deviation (SD) age, 50 ± 11 yr; body mass index [BMI], 30 ± 6 kg/m²; AHI, 26 ± 12 events/h), a single-center observational study (30) designed to examine awake-based predictors of oral appliance efficacy in patients to whom mandibular advancement splint therapy was recommended (n = 142; 59% males; mean ± SD age, 56 ± 11 yr; BMI, 30 ± 5 kg/m²; AHI, 29 ± 18 events/h), and an ongoing dual-center observational study (31) that is using magnetic resonance imaging–based genioglossus dynamics to explain heterogeneity in oral appliance efficacy (at assessment, n = 40; 72% males; mean ± SD age, 43 ± 11 yr; BMI, 30 ± 5 kg/m²; AHI, 27.7 ± 16.9 events/h). Although some of the parent studies were multicenter, all polysomnographic data included in our analysis were derived from a single sleep clinic. Key exclusion criteria for the original research studies were previous oral appliance use, oral appliance contraindications (including insufficient number of teeth, periodontal disease, and severe daytime sleepiness requiring urgent intervention), and predominance of central sleep apnea at baseline. Oral appliance efficacy was determined using in-laboratory polysomnography with the oral appliance in situ. Because all data were deidentified, the present analysis was deemed to be exempt from the need for consent by the Human Research Ethics Committee at North Sydney Local Health District, New South Wales, Australia, and the Partners Institutional Review Board, Boston, Massachusetts.

Study Protocol

Patients first attended in-laboratory diagnostic polysomnography (electroencephalography, electrocardiography, electrooculography, chin and leg electromyography, thoracoabdominal plethysmography, pulse oximetry, body position, nasal pressure airflow, and thermistor signals), which was scored according to the 2012 American Academy of Sleep Medicine (AASM) criteria (12, 33) (30% reduction in airflow with either 3% oxygen desaturation or cortical arousal) or 2007 AASM criteria (30, 31, 34) (30% reduction in airflow with 4% oxygen desaturation). Models were not adjusted for scoring type, because no effect was evident (see Results). All patients were treated with a custom-made oral appliance (SomnoDent; SomnoMed Ltd.) implemented for individual patients under the supervision of a treating dentist. Devices were initially set to 70% of the maximal mandibular protrusion from habitual bite. Patients were instructed to incrementally advance the protrusive level of the device until the maximum comfortable limit was reached (∼6–8 wk), which was then confirmed by the treating dentist. A second in-laboratory polysomnography was performed to determine response to therapy.

Oral Appliance Efficacy

Oral appliance efficacy was described by the percentage reduction in AHI with treatment relative to baseline (primary outcome, continuous variable). This measure was selected (over absolute reduction or treatment AHI) to maximize statistical power (typically largest mean/SD ratio [35], correlates least with baseline AHI [36]).

Endotypic Trait Analysis

Raw data

We identified 94 patients who met our prespecified eligibility criteria for analysis (n = 50 excluded for AHI < 20 events/h). For optimal endotyping analyses, a thorough manual check of cortical arousal onset and end times was performed for baseline polysomnography by three experienced scorers. Adjustment of arousal timing (if required) was performed blinded to treatment outcome. Raw data and accompanying annotations (staging, arousals, and respiratory events) were exported for each patient. Data from one participant were excluded because of poor nasal pressure signal (automated quality control algorithm, verified visually), leaving 93 patients for analysis. Analysis of the traits was restricted to non–rapid eye movement (non-REM) sleep for consistency with previous validation studies using our methods and to avoid the influence of night-to-night variability in REM duration on the measurements (27–29). Data from supine and lateral positions were pooled, given the interest in predicting changes in the total AHI with treatment, regardless of position.

Quantifying OSA endotypic traits

These methods have been described in detail previously (27–29, 37). Each diagnostic polysomnogram was automatically segmented into 7-minute overlapping windows containing non-REM sleep. The analyses were performed for each window separately, and median values across windows were used to represent each individual. First, nasal pressure (linearized, square root) provided an uncalibrated breath-to-breath ventilation signal (volume × rate), calibrated such that the mean eupneic ventilation for the window being analyzed was 100% (29). “Ventilatory drive” was defined as the intended ventilation that would be observed if the airway was completely open (i.e., immediately after a scored cortical arousal). Ventilatory drive was estimated by least squares fitting of a regression model that seeks to predict ventilation (i.e., overshoot between obstructive events) on the basis of previous values of ventilation. This chemoreflex model is physiologically constrained such that three key parameters are identified (gain, time constant, and delay [27]). These parameters were used to calculate the loop gain (ventilatory drive response to an oscillatory disturbance at the natural frequency, which captures the combined influence of chemoreflex sensitivity, plant gain, and circulatory delay; a value of 1 would predict central sleep apnea) (27, 37). The ventilatory response to arousal (additional ventilatory drive response that accompanies arousal, independent of the chemoreflex contribution) was found by including the presence of a scored electroencephalographic arousal on any breath as a covariate. The arousal threshold was calculated as the mean ventilatory drive on the breath immediately preceding scored arousals (29).

To calculate collapsibility and muscle compensation, an overnight endotype plot (28) was generated, whereby all breath-by-breath values of ventilation and ventilatory drive for the whole night (except breaths in awake, arousals, and REM) are tabulated and plotted against each other. The median value of ventilation at eupneic ventilatory drive was taken as a measure of passive collapsibility (V_PASSIVE). A lower V_PASSIVE value reflects greater collapsibility. The median value of ventilation at maximal ventilatory drive (at arousal threshold) was taken as a measure of active collapsibility (V_ACTIVE). The difference between V_ACTIVE and V_PASSIVE was used as a measure of pharyngeal muscle compensation. Analysis was fully automated using custom in-house software (MATLAB; MathWorks; interested users are directed to contact the authors) and visually verified.

Model Development and Statistical Analyses

The goals of the statistical analyses were twofold. First, we sought to describe the associations between oral appliance efficacy and OSA endotypic traits (in combination) at baseline to provide insight into mechanistic causes of variability in oral appliance efficacy. A multivariable regression model approach (37) was employed (outlined below). Second, we sought to use the same endotype-based regression model as the basis of a prediction model to examine the extent to which these endotypes could be employed to identify an endotype-based subgroup of patients with OSA who would exhibit greater oral appliance efficacy (“predicted responders”) than other patients (“predicted nonresponders”).

Statistical power

Ninety patients were estimated to provide 86% power to detect significant independent associations with R² > 0.1 (α = 0.05) and 92% power to detect differences in efficacy between endotypic subgroups of at least 20 ± 40%.

Data transformation

Several variables were not normally distributed (Shapiro-Wilk test) and were transformed accordingly: V_PASSIVE and arousal threshold were square root transformed via the equations y = 1 + (x − 1)^0.5 and y = 1 − (1 − x)^0.5, respectively, where x = 1 describes 100% (37). The percentage reduction in AHI with treatment versus baseline (primary outcome variable) was transformed to avoid left skewness via the equation y = x/(2 − x), which is equivalent to y = (baseline AHI − treatmentAHI)/(baseline AHI + treatment AHI), where x ranges between −1 and +1. This transformation was made so that, for instance, halving or doubling baseline AHI with oral appliance therapy would produce equal and opposite effects on the transformed outcome (y is noted as ΔAHI from now on). Thus, halving baseline AHI (x = 0.5) yields ΔAHI = 33%, and doubling baseline AHI (x = −1) yields ΔAHI = −33%. Baseline and treatment AHI were also leftward skewed and transformed using y = x^1/3. All variables were back transformed for presentation.

Bivariate analyses

Simple linear regression analyses were initially performed to evaluate the relationships between ΔAHI and each OSA endotypic trait individually.

Multivariable regression analysis

Initial visual inspection of the data (plots showing responders and nonresponders against combinations of collapsibility, loop gain, and arousal threshold; data not shown) suggested complex interactions between multiple traits and oral appliance efficacy that could not be captured appropriately using bivariate regression. We therefore employed a quadratic regression analysis in which the total possible variable terms (i.e., 20 terms) were included. These 20 terms were the 5 individual OSA endotypic traits, the 5 endotypic traits after square transformation (1 square transformation per trait), and 10 interaction terms. Significant square-transformed terms would indicate nonlinear relationships with efficacy (e.g., U-shaped curve), and interaction terms would imply that the relationship between efficacy and an endotypic trait varies depending on the level of another trait. An example quadratic model expression with just two traits would be β₀ + β₁(loop gain) + β₂(V_PASSIVE) + β₃(loop gain)² + β₄(V_PASSIVE)² + β₅(loop gain × V_PASSIVE). To determine which terms should be included, we employed a backward elimination method. We started with a model that included all 20 terms. Subsequently, backward stepwise elimination iteratively removed each term with the highest P value if P > 0.157 (Wald test, equivalent to Akaike information criterion, indicating that the relative quality of the model was not improved with the inclusion of the term [37–41]). This approach was used on the basis that removal of very weak predictors reduces uncertainty of the remaining model coefficients. Terms were accepted as significant at P < 0.05. For interpreting associations, we did not adjust the P-value threshold for multiple comparisons (e.g., five traits: conservative P threshold = 0.01).

Defining endotypic subgroups

Defining the endotypic subgroups of predicted responders and predicted nonresponders was based on the regression model above and the following steps. True responders and true nonresponders were defined by percentage reduction in AHI with treatment (true efficacy cutoff, 50%). Predicted responders and predicted nonresponders were defined by determining the optimal cutoff, derived from the multivariable regression model output, that maximized sensitivity plus specificity (a model-predicted efficacy cutoff of 60% was found; see Results); note that model-predicted efficacy and true efficacy are not equal. Predicted subgroups were allocated using a “leave one patient out” cross-validation procedure to avoid overestimating predictive performance. This procedure ensured that the outcome status of a given patient was predicted on the basis of a model that included all patients’ data except his/her own. Thus, cross-validated results are more conservative (more likely indicative of future retest performance). For example, allocation of patient 1 to a predicted responder subgroup or a predicted nonresponder subgroup was determined by building a modified model (via rerunning backward elimination regression described above) without the data of patient 1 and then using this modified model to predict patient 1’s response. This process was then repeated for all other patients from patient 2 to patient 93. The primary statistical comparison for the study was the difference in percentage reduction in AHI (primary outcome variable) between the predicted endotypic subgroups.

Adjusting for clinical covariates

Multivariable linear regression was used to determine whether predicted response status (being a predicted responder vs. a predicted nonresponder) could predict oral appliance efficacy (ΔAHI) independently of clinical covariates (i.e., baseline AHI, BMI, age, sex, and neck circumference; baseline REM/NREM AHI and change in REM sleep duration with treatment were also assessed). To perform this test, predicted response status (1 or 0, respectively) was included as an independent variable, and clinical covariates were sequentially included and then removed from the model.

Data presentation

Data are presented as mean ± SD for descriptive variables and mean ± SEM for comparisons. Back-transformed data were presented as mean (95% confidence interval [CI]). Data were described as median [25th–75th centile] for nonnormally distributed data as appropriate. Significance was accepted at P < 0.05. Figures were created using custom MATLAB software (MathWorks).

Results

Baseline Characteristics

Data from 93 participants (56% males) were analyzed. Baseline versus treatment characteristics for the overall group are presented in Table 1. On average, participants were middle-aged (56.2 ± 11.0 yr), obese (30.5 ± 5.3 kg/m²), and had moderate to severe OSA (30.6 [24.4–43.5] events/h).

Table 1.

Patient characteristics

Characteristic	Baseline	Oral Appliance	P Value
Sex, M:F	52:41
Age, yr	56.2 ± 11.0
BMI, kg/m2	30.5 ± 5.3
Neck circumference, cm	40.2 ± 4.1
Maximum possible advancement, mm	10.4 ± 3.4
Final advancement, mm	9.2 ± 3.0
Mandibular advancement, % maximum	88.5 ± 14.8
AHI, total events/h	30.6 [24.4–43.5]	11.3 [5.5–19.1]	<0.001
% reduction (ΔAHI)	67.4 [42.5–83.1]
AHI, non-REM, events/h	31.4 [21.5–48.6]	7.3 [3.1–16.9]	<0.001
AHI, REM, events/h	48.8 [29.1–66.1]	25.6 [6.1–45.3]	<0.001
AHI, supine*, events/h	52.9 [33.0–72.4]	16.7 [8.7–38.4]	<0.001
Arousal index, events/h	43.5 [35.7–54.6]	9.9 [4.4–16.5]	<0.001
Minimum oxygen saturation, %	80 [76–84]	87 [81–90]	<0.001
TST, min	356 ± 65	363 ± 65	0.43
REM sleep time, % TST	16.8 ± 6.2	17.6 ± 6.9	0.45
Supine sleep time*, % TST	40.1 [26.3–69.3]	37.0 [21.2–80.6]	0.83

Definition of abbreviations: AHI = apnea–hypopnea index; BMI = body mass index; F = female; M = male; REM = rapid eye movement; TST = total sleep time.

On average, participants were typical patients with obstructive sleep apnea, middle aged, predominantly obese with moderate to severe obstructive sleep apnea. Continuous variables are presented as mean ± standard deviation or median [25th–75th centile].

^*Data available in n = 62.

Oral Appliance Therapy

The final protrusion provided by the oral appliance was, on average, 89% (range, 44–100%) of the maximal mandibular protrusion. Overall, treatment lowered AHI by a median of 67% and had favorable effects on arousal frequency and oxygenation (Table 1). Sixty-three patients were responders (>50% reduction in AHI).

Bivariate Analyses

Using simple linear regression analyses, we observed no bivariate associations between oral appliance efficacy (percentage reduction in AHI transformed [ΔAHI]) and any of the individual endotypic traits at baseline (R² < 0.01 for all). There were also no associations between oral appliance efficacy and baseline AHI, BMI, age, sex, or neck circumference.

Multivariable Regression Analysis

When endotypic traits were considered in combination (multivariable regression), we found that greater oral appliance efficacy was associated with moderate V_PASSIVE (nonsevere and nonmild), lower pharyngeal compensation, and more favorable nonpharyngeal traits (i.e., lower loop gain, higher arousal threshold, and lower response to arousal) (see Table 2 and Figure 1). Several interaction variables were also associated with treatment efficacy (see Table 2 and Figure 1 for interpretation of each of the 12 terms included in the model).

Table 2.

Traits associated with oral appliance efficacy: multiple regression

Variable	β-Value	SEM	β-SD	P Value	Interpretation
Constant	47.1	5.1	1.4	<0.0001
Pharyngeal traits
VPASSIVE	−0.771	0.325	−0.47	0.02	Not severe and not mild collapsibility → success
VPASSIVE2	−0.0293	0.0095	−1.1	0.003
Compensation2	0.0215	0.0067	0.94	0.002	Higher compensation → failure
Compensation × arousal threshold	0.0486	0.0107	1.4	<0.0001	Particularly when arousal threshold is low or response to arousal is high
Compensation × response to arousal	−0.0171	0.0064	−0.41	0.009
Nonpharyngeal traits
Loop gain	−112	41	−0.37	0.008	Higher loop gain → failure
Loop gain × compensation	−6.95	2.23	−0.52	0.003	Particularly when compensation or response to arousal is high
Loop gain × response to arousal	−8.79	2.52	−0.70	0.0008
Arousal threshold	0.420	0.233	0.32	0.076	Lower arousal threshold → failure
Arousal threshold2	0.0151	0.0055	0.51	0.007
Response to arousal	−0.514	0.193	−0.34	0.009	Higher response to arousal → failure
Response to arousal × VPASSIVE	−0.0212	0.0115	−0.50	0.069

Definition of abbreviations: SD = standard deviation; SEM = standard error of the mean; V_PASSIVE = passive collapsibility.

Oral appliance efficacy is defined as the percentage reduction in apnea–hypopnea index with treatment compared with baseline (transformed; see Methods). The table describes final results (12 of 20 terms) after backward stepwise elimination (P-to-remove = 0.157), which began with five traits, their squares, and all interaction terms. Note that the significance levels were P < 0.05 in 10 of 20 terms and P < 0.01 in 9 of 20 terms. Traits were mean subtracted before terms were generated and applied to the model (see below). β-SD describes the number of SDs of change in treatment efficacy per SD increase in each term (1.3 SD is needed to move a typical nonresponder to a typical responder). Mean values of the endotypic traits before mean substraction: V_PASSIVE = 79.0 ± 20.8; loop gain = 0.43 ± 0.11; compensation = −9.5 ± 27.0%; arousal threshold = 141.8 ± 26.0%; response to arousal = 36.3 ± 22.6%. A regression model cutoff of 60% (predicted reduction in apnea–hypopnea index, untransformed) was used to define predicted responders and predicted nonresponders (maximized sensitivity plus specificity). Overall R² = 0.30; adjusted R² = 0.19; P = 0.003.

Figure 1.

Key aspects of the five-trait multivariable model (see Table 2) illustrating how combinations of traits may influence oral appliance efficacy. Each plot depicts a two-trait cross-section of the full model drawn at the mean values of the remaining three traits. Dots represent true response observations of individual patients: red for nonresponders (<50% reduction in apnea–hypopnea index [AHI] with treatment), orange and green for responders (50–70% reduction in AHI and >70% reduction in AHI, respectively). Background regions represent predicted response subgroups (light green for predicted responders and light red for predicted nonresponders). Top and left: A U-shaped relationship between collapsibility (V_PASSIVE) and efficacy is evident. For example, in the top figure, the light green shading indicating predicted responders is only seen in a midrange of moderate collapsibility and at lower loop gain. Note that nonresponders with high V_PASSIVE (mild collapsibility) tend to have high loop gain, low arousal threshold, and higher compensation (see dense regions of red dots). Top and right: A higher loop gain is associated with reduced treatment efficacy, particularly in milder collapsibility (high V_PASSIVE), but also in the presence of a lower arousal threshold and higher compensation. Open gray circles on each plot represent individual patients whose values for the three remaining traits were too far from the mean to be fairly represented in the simplified two-trait view (i.e., two-trait prediction differed from the full model prediction).

Defining Endotypic Subgroups

Use of the multivariable regression model above to define endotype subgroups of predicted responders and predicted nonresponders revealed the findings reported in the subsections below.

Before cross-validation

Predicted responders (n = 57), compared with predicted nonresponders (n = 36), exhibited a greater reduction in AHI from baseline (mean [95% CI], 76% [70–80] vs. 42% [28–55]; P < 0.0001) and had lower treatment AHI (8 [6–10] vs. 18[14–23] events/h; P < 0.0001). Positive and negative predictive values were 83% and 56%, respectively; accuracy was 72%.

After cross-validation (main results)

Differences in responses between subgroups remained clinically significant after cross-validation: Predicted responders (n = 54), compared with predicted nonresponders (n = 39), exhibited a greater reduction in AHI from baseline (mean [95% CI], 73% [66–79] vs. 51% [38–61]; P < 0.0001) and had lower treatment AHI (8 [6–11] vs. 16 [12–20] events/h; P = 0.002) (see Figure 2). Positive and negative predictive values were 78% (42:12) and 46% (18:21), respectively (P = 0.02, Fisher’s exact test); accuracy was 65%.

Figure 2.

(A–B) On the basis of combined endotype traits, predicted responders (black), compared with predicted nonresponders (gray), exhibited a greater oral appliance efficacy, indicated by a greater reduction in apnea–hypopnea index (AHI) (untransformed) from baseline (A) and a lower residual AHI on treatment (B). Error bars illustrate 95% confidence in the mean. Results are based on cross-validated analysis, whereby the endotypic subgroup allocation for each individual patient was based on a modified regression model using data from all other patients. Thus, group differences are not guaranteed by definition, based on the regression model results in Table 2.

Further analyses

Adjusting for covariates (baseline AHI, BMI, age, sex, neck circumference, baseline REM/NREM AHI, and change in REM sleep duration with treatment) did not attenuate the differences between groups. Notably, baseline AHI was similar between groups (predicted responders vs. predicted nonresponders, mean [95% CI], 34 [30–38] vs. 33 [29–37] events/h; P = 0.5). In addition, none of the clinical covariates above were significantly associated with oral appliance efficacy (ΔAHI) when considered individually (linear regression) or in combination (multivariable regression, total R² = 0.08).

Adjusting for scoring type had no impact (<1% change in model coefficient) on the association between endotypic subgroup and oral appliance efficacy and was not associated with efficacy (P = 0.9). Altering the cutoff of true responder from greater than 50% to greater than 70% reduction in AHI yielded similar results, with group differences in efficacy of 22% (cross-validated, P = 0.0006) becoming 20% (P = 0.0011). Positive and negative predictive values became 65% (30:16) and 72% (34:13), respectively (P = 0.0004, Fisher’s exact test); accuracy was 69%.

Discussion

The present study is, to our knowledge, the first to demonstrate that the endotypic traits causing OSA, estimated from routine diagnostic polysomnography, have utility in defining a subgroup of patients who are more likely to respond to oral appliance therapy. Our study shows that a greater treatment efficacy is associated with favorable nonpharyngeal traits (lower loop gain, higher arousal threshold, and lower ventilatory response to arousal), moderate collapsibility (not mild or severe, U-shaped), and weaker pharyngeal muscle compensation. Using measurements of the traits alone, predicted responders, on average, exhibited half the residual AHI (8 events/h, approximately one-fourth of baseline) compared with predicted nonresponders (16 events/h, approximately half of baseline), despite similar baseline AHI. Moreover, 78% of patients in the predicted responders group exhibited at least a 70% reduction in AHI. These results provide a basis for future identification of patients who could potentially be prioritized for personalized therapy with oral appliances based on the OSA endotypic traits estimated from diagnostic polysomnography.

Consistency with Available Literature and Novel Physiological Insights

Our findings confirm previous work in that OSA endotypes can be estimated from routine diagnostic polysomnography and provide insight into therapeutic outcomes (26–29, 42). In concordance with physiological principles and our recent small, detailed physiology study, we confirmed the finding in a larger dataset that lower loop gain contributes significantly to greater oral appliance efficacy (20). We emphasize, however, that in the present study, unlike our prior work, we did not find a strong bivariate relationship between loop gain and oral appliance efficacy. However, the requirement for multiple interacting endotypic predictors to be considered in combination is consistent with our previous study (37).

Previous studies have also found that severe collapsibility is associated with reduced oral appliance efficacy (15, 20, 26, 43–45). Oral appliance therapy typically reduces critical collapsing pressure by 3–5 cm H₂O (26, 46–48) and therefore is unlikely to resolve OSA in patients with severe collapsibility at baseline. Greater collapsibility (lower V_PASSIVE), higher BMI, nonpositional OSA (a marker of greater collapsibility), and higher CPAP requirement have each been shown to predict poor response to oral appliance therapy (15, 20, 26, 43–45), although these are not robust predictors individually. The present study found a U-shaped relationship between collapsibility and response to oral appliances. As expected, more severe collapsibility predicted reduced responses to oral appliance therapy. Milder collapsibility, unexpectedly, also predicted a reduced response to treatment. We consider that these individuals, rather than being easier to treat, have primarily nonpharyngeal mechanisms underpinning their sleep apnea. We emphasize that although we initially considered that the U-shaped relationship could be spurious, we noted that a large proportion of patients were nonresponders with mild collapsibility (and high loop gain or low arousal threshold; see Figure 1), such that this unexpected U-shaped effect at the mild end of the spectrum was unlikely to be attributable to low sample size.

We also found that elevated loop gain, lower arousal threshold, and greater ventilatory response to arousal also contributed to a reduced oral appliance efficacy. These nonpharyngeal factors contributing to breathing instability are unlikely to be corrected by mandibular advancement (20). Indeed, it was precisely this subgroup of patients that responded preferentially to supplemental oxygen in our recent study (37). Furthermore, we found that reduced pharyngeal compensation was associated with a higher oral appliance efficacy. According to physiological principles, a stronger pharyngeal dilator muscle compensation will act to mask a more severely collapsible airway. Therefore, attempts to improve collapsibility via oral appliance therapy will be partially counteracted by attenuation of the pharyngeal dilator muscle activity as airway obstruction is mitigated. Thus, our finding that poor compensation is associated with a higher oral appliance efficacy is consistent with physiological principles.

Our study shows no significant predictive value of routine clinical variables (such as baseline AHI, BMI, neck circumference, age, or sex), whether individually or in combination with OSA endotypic traits. These data confirm the difficulty in using routine clinical variables to predict outcomes of oral appliance therapy (44). Our study also supports the concept that baseline severity of OSA (AHI) is not a useful predictor of response to therapy.

Clinical Implications

The present study sought to advance knowledge for future precision sleep medicine. In the context of heterogeneous oral appliance efficacy in unselected patients, a major goal of our work was to enable the identification of a subgroup of patients with (moderate to severe) OSA with a superior treatment efficacy compared with that of other patients with OSA whose average efficacy is more modest. We used an automated clinical tool to estimate the key endotypic traits causing OSA from routine diagnostic polysomnography and combined these traits to define two endotype-based subgroups of patients. On average, the predicted responders subgroup exhibited good treatment efficacy (∼75% reduction in AHI), which, when coupled with the reported high adherence to therapy (12), appears sufficient to justify offering oral appliances as a first-line therapy in select patients with (moderate to severe) OSA (specifically those who have a preference for this intervention). Although our results were based on unseen holdout data (leave one patient out cross-validation), these findings require replication in a larger prospective study before this method can be adopted for routine clinical use. Notably, even the predicted nonresponders subgroup had, on average, 50% reduction in AHI (residual AHI, ∼16 events/h). Although this level of efficacy seems unlikely to show superior health benefit compared with CPAP, the considerable improvement in nonresponders is likely to confer benefit over no therapy, justifying prescription of oral appliance therapy as a second-line option even in this subgroup (e.g., in CPAP-intolerant patients).

Our automated method has several advantages as a clinical tool for predicting outcomes. It is based on OSA endotypes (e.g., rather than demographic factors) and therefore has a close connection with the underlying mechanisms. The approach used in the present study is inexpensive, is not dependent on specialized equipment or physiological interventions, and can produce results rapidly. The data used for analysis in the present study were also clinical in nature, supporting clinical generalizability and translatability of physiological endotypes. Data were extracted from standard clinical sleep studies (rather than research studies) acquired using a commercially available sleep recording system (Profusion PSG; Compumedics Ltd.). Because the analysis was retrospective, there was no opportunity to pay extraordinary attention to nasal pressure quality beyond AASM standards (unfiltered nasal pressure). Other challenges for widespread implementation of our tool in clinical practice include incorporation of endotyping methods into commercial systems and requirement for rescoring of arousal timing (not performed clinically). Neither obstacle is insurmountable.

Methodological Considerations

There are several limitations of our work. First, the endotypic traits described are not based on gold standard measurements but rather estimated from a nasal pressure surrogate of ventilatory airflow and a mathematically estimated ventilatory drive signal. However, it would be a highly challenging endeavor to perform gold standard measurements of physiology (via CPAP decreases or esophageal catheterization) in such large numbers of patients undergoing a specific treatment regimen. Thus, a strength and novelty of our work is obtaining such measures in a sample size of more than 90 oral appliance–treated patients with OSA. Second, we studied patients with baseline AHI greater than 20 events per hour (average AHI, 30 events/h), and thus our results are relevant to those with similar OSA severity and may not apply to many patients with a milder condition who seek oral appliance therapy. Indeed, a major goal of our work was to identify patients who might exhibit favorable outcomes of oral appliance therapy despite having more severe OSA. Further investigation is needed to identify those with milder OSA (AHI, <20 events/h) who might be suitable for oral appliance therapy regardless of their endotypic characteristics.

Third, the incomplete data nature of retrospective studies precluded full assessment of the impact of some other relevant variables. For example, we did not have systematic data at baseline and on therapy for supine sleep duration. Controlling for body position would likely reduce a source of undesirable variability. Nonetheless, the influence of endotypes on efficacy is unlikely to be confounded by differences in body position at baseline and on therapy (i.e., no plausible mechanism by which treatment-related changes in supine sleep duration could influence endotypic traits of OSA and thus oral appliance efficacy). We also did not have systematic measures of daytime sleepiness (e.g., Epworth Sleepiness Scale) and thus could not assess the role of daytime sleepiness in the context of the endotypic traits. However, we found a relationship between lower arousal threshold and reduced oral appliance efficacy, suggesting that a higher propensity for arousal from sleep might render oral appliance treatment less efficacious. Further investigation along these lines is warranted.

Fourth, we used the percentage reduction in AHI as a continuous outcome measure and a single cutoff (i.e., 50% reduction in AHI) to define the true response subgroups. However, changing the cutoff (e.g., to a 60% or 70% reduction in AHI) did not alter the findings substantially. We also note that proportions of patients defined as complete responders (≥50% reduction in AHI and residual AHI <10 events/h), partial responders (≥50% reduction in AHI and residual AHI ≥10 events/h), and nonresponders (<50% reduction in AHI) were 30:12:12 in predicted responders and 10:11:18 in predicted nonresponders (P = 0.01, Fisher’s exact test), respectively. Fifth, although subgroup differences in efficacy appear clinically relevant, the overall model accuracy is modest; as noted above, predicted nonresponders show an average of 50% reduction in AHI. Thus, currently, we are unable to identify a subgroup of patients with OSA who may exhibit negligible benefit. Incorporation of additional information on site/structure of pharyngeal obstruction (e.g., through coupling of our approach with other polysomnographic methods such as airflow shape [49]) may further improve the model precision and predictive performance.

Finally, we caution that the noninvasive measurements of endotypic traits were validated against gold standard values in relatively small samples (n = 28–41) and would benefit from further refinement and validation studies, including efforts to improve reliability (e.g., incorporating respiratory inductance plethysmography to handle mouth leak) and make the measurements independent of manual scoring (e.g., quantitative electroencephalographic analysis [50]).

Conclusions

In the largest study to date, we elucidated the relationships between the pathophysiological traits causing OSA and oral appliance treatment efficacy. Although bivariate linear associations between efficacy and endotypes were not evident, our multivariable analyses showed that greater oral appliance efficacy is associated with favorable nonpharyngeal endotypic traits of OSA at baseline (including lower loop gain, higher arousal threshold, and lower ventilatory response to arousal). Greater efficacy was also associated with moderate (nonmild or nonsevere) collapsibility and weaker dilator muscle compensation. Combining endotypic traits identified a predicted responders subgroup of patients who exhibited good treatment efficacy and could potentially be targeted judiciously for early oral appliance intervention compared with a predicted nonresponders subgroup. Further studies are needed to prospectively validate our predictive model for clinical use. We anticipate that identifying endotypes from routine diagnostic polysomnography will allow patient selection for oral appliance therapy in OSA.