Shooting STAR: Caution in Interpreting Long-Term Cost Effectiveness from a Short-Term Case-Series

Treatment of moderate to severe obstructive sleep apnea (OSA) in patients intolerant to continuous positive airway pressure (CPAP) remains a source of concern given the association with cardiovascular events and mortality.^1,2 Despite the efficacy of CPAP, its effectiveness is limited by patient use of the device. Recently, an upper airway stimulation (UAS) device was tested in the stimulation therapy for apnea reduction (STAR) Trial with promising results.³ In this issue of SLEEP, Pietzsch et al.⁴ report using these results to provide a long-term cost effectiveness analysis (CEA) of UAS. UAS is an exciting new therapy for OSA, but we highlight some key cautions in the interpretation of the long-term CEA at this stage of study.

A cost effectiveness analysis measures the cost per unit of effectiveness for a health intervention. A common measure of effectiveness, the quality-adjusted life year, synthesizes quality and quantity of life. While conceptually simple, it is highly complex to quantify quality-adjusted life years for any single intervention, as it requires statistical modeling of different clinical outcomes (good and bad), their impacts on quality and quantity of life, and their probabilities. Likewise, cost is conceptually simple but is also highly complex to quantify for each health state related to the disease and its intervention, and even just direct costs are prone to dramatic variances across health systems, payers, providers, and patient groups. One challenge of cost effectiveness analysis is that an inaccuracy or bias in any part of the analysis gets propagated or even amplified in the final result. To mitigate the risk of inaccuracy (and uncertainty) it is critical to use the best available evidence, have a broad sample of data for measuring cost and effectiveness, and to quantify the impacts of uncertainty.⁵ The devil is in the details for interpreting any CEA. Here, we wish to highlight some important details that warrant caution in the interpretation of the upper airway stimulation cost effectiveness analysis.

Limited Evidence Base

While the UAS CEA appears compelling, it is based on a single case series in a highly selected sample (STAR Trial). While the STAR Trial also included a post hoc randomized trial of withdrawal among UAS responders, those data provide no insight to prospective outcome and appropriately were not used in the CEA. Other studies of UAS devices showed less dramatic results in less selected samples of patients, and with other UAS devices.^6–8 The only prospective controlled study of UAS is a randomized trial of activated UAS compared to implanted but inactivated controls⁹; however, that trial was discontinued and the company shut down because the study was unlikely to meet the primary clinical endpoint.¹⁰ The results have not been published. Thus, while UAS is an exciting and promising therapy for selected OSA patients, it is premature to quantify the effectiveness of UAS with any confidence based on the single case series (STAR Trial) used for the CEA. Thus, the CEA interpretation warrants caution.

Extrapolation of Apnea-Hypopnea Index

The STAR trial did not measure long-term health impacts of UAS, such as death, cardiovascular events, stroke, motor vehicle crashes, and others. The CEA investigators⁴ used a statistical model to extrapolate from the mean reduction in 12-month apnea-hypopnea index (AHI) to improvements in these long-term OSA outcomes. While AHI is associated with these clinically important outcomes, this type of modeling should be interpreted with caution. The strength of association is variable between cohorts, definition of AHI,¹¹ and clinical outcome measure. This variability across epidemiological studies was not accounted in the CEA modeling, because the model was derived from only a limited number of studies for each clinical outcome. Modeling from AHI is even more concerning for treatments dependent on patient use of a device, such as CPAP and UAS. Unfortunately, methods for modeling adherence are limited (in the current study, it is assumed to be constant).^12,13 The important distinction between AHI efficacy in the lab and clinical effectiveness in the real world is illustrated in studies of surgery and mandibular advancement devices, where the AHI effect underestimates the demonstrated clinical effects on mortality, cardiovascular disease, motor vehicle crashes, and quality of life.^14–22 Furthermore, the investigators for the UAS CEA⁴ make a critical assumption that improvements in AHI at 12 months with UAS accurately predict lifelong reduction of death, cardiovascular events, strokes, motor vehicle crashes, and others. This surrogate modeling method and the assumptions it includes warrant serious caution in the interpretation.

Generalizability

The STAR trial cohort is dissimilar to the cohorts used to model the long-term health outcomes. They differed in age, sex, body mass index, comorbidity, health system, country, culture, OSA management strategies, and other known and unknown factors. Pietzsch and colleagues⁴ attempted to account for some differences using a sophisticated polynomial modeling method to match the cohorts on a small number of known variables. However, this matching did not account for most of the known variables (let alone the unknown factors); it did not account for important variable interactions (e.g., obesity, lung disease, and AHI)²³; and it was not validated by testing it with clinical outcomes. The limited generalizability and unvali-dated model matching of cohorts warrants further caution in such a model extrapolation of clinical effectiveness outcomes.

Time Horizon

The time horizon for the CEA modeling should be long enough to reflect the important differences between costs and outcomes, but it warrants caution when the time horizon extends far beyond the data.^24,25 The studies fueling the mortality and cardiovascular effects of CPAP had a follow-up period of five to ten years to define the difference in the groups.^26–28 The appropriate balance between this time period, the follow-up period for the UAS device (12 months),⁴ and the maximum time-horizon of a lifetime is unclear. Pietzsch et al. chose the maximum duration (arguably an extreme extrapolation) as their base case analysis, and their sensitivity analysis shows that their results and conclusions are highly dependent on this assumption (Appendix A.3).⁴ The lifetime time horizon is of particular concern in this analysis because the long-term effectiveness and risks of UAS are unknown.

Synthesis and Caution

Even with the probabilistic sensitivity analysis employed by Pietzsch et al., the limited range of uncertainty modeled and the extent of assumptions, surrogates, proxies, lack of direct data, and single data source for UAS, collectively warrant great caution in interpreting the UAS CEA. This long-term cost effectiveness analysis of upper airway stimulation for OSA should be considered very preliminary. If one were confi-dent in a CEA of such modeled assumptions based on a single case series, then CEA could be reported for every OSA treatment ever tested. For example, didgeridoo playing has been shown to improve OSA comparably to UAS,²⁹ but surely at much reduced cost and long-term risk. This CEA methodology might render it the cost-effective treatment of choice; however, the lack of replication studies for didgeridoo raises concerns about the validity, reliability, and generalizability of that treatment. Likewise, the use of just one UAS case series is premature to extrapolate its life-long cost-effectiveness. A UAS CEA will be more reliable after further testing of this new and exciting device in many different cohorts, with broader patient inclusion, with long-term clinical and adverse effect outcomes quantified, and with costs measured in established practice.

CITATION

Skirko JR, Weaver EM. Shooting STAR: caution in interpreting long-term cost effectiveness from a short-term case-series. SLEEP 2015;38(5):665–667.

DISCLOSURE STATEMENT

The authors have indicated no financial conflicts of interest. Work supported by resources from the Veterans Affairs Puget Sound Health Care System, Seattle, WA. The contents of this commentary do not represent the views of the U.S. Department of Veterans Affairs or the United States Government.