Abstract
Background:
Little is known about predicting the risks of asthma exacerbation after stopping asthma biologics.
Objective:
To develop and validate a predictive model for the risk of asthma exacerbations after stopping asthma biologics using machine learning models
Methods:
We identified 3,057 people with asthma who stopped asthma biologics in the Optum Labs Database Warehouse and considered a wide range of demographic and clinical risk factors to predict subsequent outcomes. The primary outcome used to assess success after stopping was having no exacerbations in the 6 months after stopping biologic. Elastic-net logistic regression (GLMnet), random forest (RF), and gradient boosting machine (GBM) models were used with 10-fold cross validation within a development (80%) cohort and validation cohort (20%).
Results:
Mean age of the total cohort was 47.1 (SD, 17.1) years, 1,859 (60.8%) were female, 2,261 (74.0%) were white, and 1,475 (48.3%) were in the Southern region of the US. The GLMnet model yielded an AUC 0.75 (95% CI, (0.71,0.78)) in the development and AUC 0.72 in the validation cohort. The RF model yielded an AUC of 0.75 (95% CI, (0.68,0.79)) in the development cohort and an AUC of 0.72 in the validation cohort. The GBM model yielded an AUC of 0.76 (95% CI, (0.72,0.80)) in the development cohort and an AUC of 0.74 in the validation cohort.
Conclusion:
Outcomes after stopping asthma biologics can be predicted with moderate accuracy using machine learning methods.
Keywords: asthma, biologics, monoclonal antibodies, step down treatment, artificial intelligence, explainable machine learning
Introduction
Medications are one part of a comprehensive asthma management strategy. When disease activity is interfering with a person’s life, or there is a high risk for future exacerbations, asthma medication may be increased. When disease activity has been stable for 3–12 months, asthma medications may be decreased or stopped.1 The ability to predict outcomes after decreasing or stopping asthma medications has been primarily considered for choices about inhaled corticosteroids (ICS), long acting β-agonists (LABA), and leukotriene antagonists (LTRA) and primarily in people with mild-moderate asthma.2–10 Less is known about predicting outcomes after decreasing medications in people with severe asthma or who are on asthma biologics.
Asthma biologics are a costly class of medications intended for people who cannot be controlled on asthma medications such as ICS,LABA, LTRA, long-acting muscarinic antagonists, and oral corticosteroids. There are no current guidelines for if, how, or when one should stop asthma biologics following a period of treatment. It would be helpful if outcomes after reducing or stopping asthma biologics could be predicted, as this would inform adjustment choices and reduce risks for over- and under-treatment with this class of medication. Previous studies have employed both traditional statistical and artificial intelligence (AI) methods to predict asthma exacerbations;11,12 however, we are unaware of any other reports that utilized AI methods to predict asthma exacerbations after stopping asthma biologics.
People who stop asthma biologics may have an increased risk for asthma exacerbations.13–15 The risk after stopping may be small, as we demonstrated using a propensity matching approach comparing risk of asthma exacerbation in people who stop (10.2%) versus those who continue (9.5%).15 Two other studies, randomized controlled trials with smaller sample sizes of omalizumab and mepolizumab, respectively, showed a slightly higher risk of total asthma exacerbations in people who stopped versus continued (55% versus 33% and 49% versus 32%, respectively), but no differences when considering severe exacerbations leading to emergency department visits or hospitalizations.13,14
We aim to develop a robust, accurate, and interpretable risk assessment tool for asthma exacerbations that would improve or complement existing methodologies. Our hypothesis is that we can predict outcomes after asthma biologics are stopped by using variables available in a claims-based dataset (prognosis hypothesis). To test this hypothesis, we leverage a nationally representative cohort of patients with asthma used in previous asthma outcome studies,15 and machine learning (ML) methods capable of identifying and associating high-dimensional patient characteristics with important outcomes. We focus on predicting no exacerbation in the 6 months after stopping biologic therapy and demonstrate through extensive experiments that the applied prediction models are accurate, interpretable, and can therefore inform clinical decision making for asthma patients.
Methods
Cohort selection
We used OptumLabs Data Warehouse (OLDW), a national dataset of over 100 million commercially insured and Medicare Advantage beneficiaries across the U.S., to identify a cohort of people with asthma from February 1, 2003 to June 30, 2020 using a modified version of the Healthcare Effectiveness Data and Information Set (HEDIS) definition of persistent asthma.15 The cohort includes people who had the following over a rolling 365-day period: a) at least one ED visit or inpatient encounter with a principal diagnosis of asthma, or b) at least 4 outpatient encounters on different days with an asthma diagnosis (any position) plus at least two asthma medication fills, or c) at least four asthma medication fills. For criterion c), if all asthma medication fills were for leukotriene modifiers or biologic therapies, at least 1 asthma diagnosis for any type of visit (ED, inpatient, or outpatient) was also required.
We refined the cohort by selecting those who had at least one administration or pharmacy fill of one of the five asthma biologic drugs. We created episodes of biologic use, starting with the first observed administration or pharmacy fill of a particular biologic drug, and continuing until the patient had a break of 120+ days in treatment or had filled or been administered a different asthma biologic drug. The end of the episode was the date the patient was last administered (or last filled) the biologic. To ensure we were capturing the beginning of the biologic use episode, we required 6 months of medical and pharmacy coverage with no asthma biologic claims prior to the start of the episode. We excluded biologic use episodes that lasted less than 6 months, which is generally considered an adequate trial period for people who can tolerate a biologic therapy.
Variable and outcome definitions
We estimated consistency of ICS/LABA controller medication use by calculating a medication possession ratio (MPR) over the six months prior to index asthma biologic use. The MPR was calculated as the sum of days supplied for use during the period divided by the number of days in the period; previously filled medications were carried over from prior fills, and medications available for use after the period ended were excluded. We assessed whether patients had accessed specialist care for asthma by looking for one or more visits to a pulmonologist or allergist in the 6 months before index biologic use. We defined tapering before stopping of the biologic if the last biologic was received > 2 weeks before the second-to-last biologic dose.
Treatment response in the initial 6 months of biologic use was defined as a reduction of at least 50% in the number of exacerbations, compared to the six months prior to index biologic use. People with no exacerbations during the pre-period could not achieve treatment response by this definition. Exacerbations were defined as a hospital or emergency department visit with asthma in first diagnostic position (or second if another respiratory diagnosis was in the first position), or a systemic corticosteroid fill associated with an outpatient visit. In secondary analyses, we looked only at severe exacerbations, defined as hospital or emergency department only (i.e., excluding those who met exacerbation criteria by outpatient systemic corticosteroid criterion).
The primary outcome of success was no asthma exacerbations in the 6 months after stopping asthma biologic therapy. We justify selecting the 6-month evaluation period, as opposed to a longer time period, to increase our overall sample size and because in previous studies most asthma exacerbations occur in the 6 months following discontinuation.15 The secondary outcome used to assess success of discontinuation was a decrease in the asthma exacerbation rate in the 6 months after discontinuing the biologic compared to the six -month period before biologic initiation.
Predictor variables
Predictor variables in the models included pre-index exacerbation count, age at index biologic use, sex, race/ethnicity, geographic region, insurance type (commercial vs. Medicare Advantage [MA]), household income, allergist or pulmonologist visit in the pre-index period, grouped Charlson comorbidity count, chronic idiopathic urticaria (included because omalizumab also used to treat this condition), atopic dermatitis, GERD, rhinitis, sinusitis, COPD, depression, the number of outpatient visits in the 6 months before stopping, biologic treatment length, season in which biologic was stopped, successful reduction in exacerbations in first 6 months of biologic use, and whether tapering occurred prior to stopping.15
Performance measures
Area under curve (AUC), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV)were all used to describe the performance of the models. SHapley Additive exPlanations (SHAP) values16 were used to describe the top 10 variables and levels of those variables that contributed most to the model prediction within the RF model. SHAP values measure the impact of variables in the model, while taking their interaction with other variables into account.
Analysis
We trained three machine algorithms: elastic-net logistic regression (GLMnet), random forest (RF), and gradient boosting machine (GBM) to predict the primary and secondary outcomes. These algorithms were chosen because they are commonly used to develop clinical risk prediction models, interpretable, and considerable evidence in healthcare, biomedical sciences and bioinformatics suggests that they are highly effective for the type of prediction problem investigated in study.17,18 Multiple machine learning algorithms were selected due to their intrinsic differences and the overall goal of the project to develop the most accurate prediction model. GLMnet is an extension of the least absolute shrinkage and selection operator (LASSO) and the ridge (Ridge) regularization methods for the logistic regression model.19 Unlike logistic regression, GLMnet will penalize (shrinks) large nonzero coefficients to prevent overfitting (inferior performance on new data). RF is modeling technique where multiple decision trees are trained on a random sample of subjects in the study data and the predictions combined using majority vote (or mean).20 GBM is equally a learning approach where a large number of relatively weak and simple models such as decision stumps (decision trees_with only one split) are iteratively trained and combined to produce a better performing model.21
The data was randomly divided into a development cohort (n=2,447, 80%) for training and validating the machine learning models and a validation cohort (n=610, 20%) for evaluating the performance of the models. Since each machine learning algorithm requires choosing tuning parameters (e.g. LASSO and Ridge for GLMnet) for best performance, we apply a grid search method to select the best combination of parameters through a 10-fold cross-validation using the development cohort. For each algorithm, a final model was obtained by retraining the algorithm on the entire development cohort (all 10-folds combined) using the best parameters and evaluated on validation cohort. We report the mean and 95% confidence intervals (CI) of the performance measures over the cross-validation and the point estimates of the final models on the validation cohort. Figure 1 illustrates the workflow of the training, validation and generation of the final model.
Figure 1. Workflow of the training, validation procedure and selection of final machine learning model.
The development cohort was randomly divided into 10 equal and independent parts (folds): models were trained on 9 folds and tested on the holdout fold. The best model tuning parameters was selected through grid search and the model evaluated on the holdout fold. This procedure was repeated 10 times until each fold has been used for training and testing. The performance results over the cross-validations were then averaged for reporting. The final model was obtained by training the model on the 10-folds combined using the overall best tuning parameters from the cross-validation and evaluated on the 20% holdout validation cohort. The best parameters were selected based on the highest AUC performance.
Predictor variables used for the analysis were complete with no missing values, except race/ethnicity and household income, where we treated the missing values in each variable as a separate category. All data analysis was performed using SAS v. 9.4 and R v. 3.6.3, with the packages glmnet,22 ranger,23 gbm,24 and fastshap.16
Results
Baseline characteristics
A total of 3,057 people met the pre-set criteria for the cohort (n=2,447 in development cohort and n=610 in validation cohort). In the development cohort, 552 people had an exacerbation in the 6 months after stopping (22.6%) and 296 had an increase in the asthma exacerbation rate in the six months after discontinuing the biologic compared to the six -month period before biologic initiation (12.1%).
The average age in the cohort was 47.1 years, 60.8% were female, 74.0% were white, and 48.3% were in the Southern region of the US. In the 6 months prior to index biologic use, 83.5% of people in the cohort had at least 1 specialist visit; 78.3% had either 0 or 1 Charlson comorbidity conditions (Table 1).
Table 1.
Baseline Characteristics
| Development (N=2,447) | Validation (N=610) | Total (N=3,057) | |
|---|---|---|---|
| Age, Mean (SD) | 47.1 (17.1) | 47.2 (17.0) | 47.1 (17.1) |
| 0–11 | ~ | ~ | 30 (0.98%) |
| 12–17 | ~ | ~ | 198 (6.48%) |
| 18–64 | 1,899 (77.61%) | 477 (78.20%) | 2,376 (77.72%) |
| 65+ | 371 (15.16%) | 82 (13.44%) | 453 (14.82%) |
| Gender: Female | 1,505 (61.50%) | 354 (58.03%) | 1,859 (60.81%) |
| Race | |||
| Unknown | 103 (4.21%) | 23 (3.77%) | 126 (4.12%) |
| Asian | 89 (3.64%) | 20 (3.28%) | 109 (3.57%) |
| Black | 227 (9.28%) | 64 (10.49%) | 291 (9.52%) |
| Hispanic | 214 (8.75%) | 56 (9.18%) | 270 (8.83%) |
| White | 1,814 (74.13%) | 447 (73.28%) | 2,261 (73.96%) |
| Region | |||
| Midwest | 562 (22.97%) | 132 (21.64%) | 694 (22.70%) |
| Northeast | 280 (11.44%) | 84 (13.77%) | 364 (11.91%) |
| South | 1,195 (48.84%) | 280 (45.90%) | 1,475 (48.25%) |
| West | 404 (16.51%) | 114 (18.69%) | 518 (16.94%) |
| Income | |||
| Unknown | 215 (8.79%) | 65 (10.66%) | 280 (9.16%) |
| <$40,000 | 377 (15.41%) | 88 (14.43%) | 465 (15.21%) |
| $40,000–$74,999 | 550 (22.48%) | 127 (20.82%) | 677 (22.15%) |
| $75,000–124,999 | 625 (25.54%) | 168 (27.54%) | 793 (25.94%) |
| $125,000–$199,999 | 363 (14.83%) | 93 (15.25%) | 456 (14.92%) |
| $200,000+ | 317 (12.95%) | 69 (11.31%) | 386 (12.63%) |
| Insurance | |||
| Commercial | 1,990 (81.32%) | 507 (83.11%) | 2,497 (81.68%) |
| Medicare Advantage | 457 (18.68%) | 103 (16.89%) | 560 (18.32%) |
| Specialist Access in 6 Months Before Index Biologic Use | 2,032 (83.04%) | 519 (85.08%) | 2,551 (83.45%) |
| Number of Outpatient Visits in 6 Months Before Stopping, Mean (SD) | 7.6 (5.4) | 7.2 (5.6) | 7.5 (5.5) |
| Charlson Condition Group | |||
| 0–1 | 1,917 (78.34%) | 477 (78.20%) | 2,394 (78.31%) |
| 2–3 | 389 (15.90%) | 106 (17.38%) | 495 (16.19%) |
| 4+ | 141 (5.76%) | 27 (4.43%) | 168 (5.50%) |
| Baseline Conditions | |||
| Chronic Uticaria | 256 (10.46%) | 63 (10.33%) | 319 (10.44%) |
| Atopic Dermatitis | 497 (20.31%) | 108 (17.70%) | 605 (19.79%) |
| GERD | 997 (40.74%) | 246 (40.33%) | 1,243 (40.66%) |
| Rhinitis | 1,841 (75.23%) | 461 (75.57%) | 2,302 (75.30%) |
| Sinusitis | 1,427 (58.32%) | 367 (60.16%) | 1,794 (58.68%) |
| COPD | 724 (29.59%) | 187 (30.66%) | 911 (29.80%) |
| Depression | 661 (27.01%) | 161 (26.39%) | 822 (26.89%) |
| Days on Treatment with Biologic Before Stopping, Mean (SD) | 556.6 (524.0) | 547.3 (461.6) | 554.8 (512.1) |
| Number of Exacerbations in the 6 Months Pre Index Biologic Use, Mean (SD) | 0.5 (0.8) | 0.6 (0.9) | 0.5 (0.8) |
| Number of Exacerbations in the 6 Months Before Stopping, Mean (SD) | 0.4 (0.8) | 0.4 (0.8) | 0.4 (0.8) |
| Number of Severe Exacerbations in the 6 Months Pre Index Biologic Use, Mean (SD) | 0.1 (0.4) | 0.2 (0.5) | 0.2 (0.5) |
| Number of Severe Exacerbations in the 6 Months Before Stopping, Mean (SD) | 0.1 (0.4) | 0.1 (0.4) | 0.1 (0.4) |
| Number of Reliever Fills in 6 Months Before Stopping, Mean (SD) | 1.7 (2.7) | 1.8 (2.8) | 1.7 (2.7) |
| Initial Response to Biologic (any exacerbations) | 621 (25.38%) | 182 (29.84%) | 803 (26.27%) |
| Initial Response to Biologic (severe exacerbations) | 259 (10.58%) | 63 (10.33%) | 322 (10.53%) |
| Season of Stopping Biologic | |||
| Q1 | 587 (23.99%) | 134 (21.97%) | 721 (23.59%) |
| Q2 | 577 (23.58%) | 150 (24.59%) | 727 (23.78%) |
| Q3 | 573 (23.42%) | 149 (24.43%) | 722 (23.62%) |
| Q4 | 710 (29.02%) | 177 (29.02%) | 887 (29.02%) |
| Biologic Drug | |||
| Benralizumab | * | * | 60 (1.96%) |
| Dupilumab | 288 (11.77%) | 65 (10.66%) | 353 (11.55%) |
| Mepolizumab | 216 (8.83%) | 47 (7.70%) | 263 (8.60%) |
| Omalizumab | 1,874 (76.58%) | 482 (79.02%) | 2,356 (77.07%) |
| Reslizumab | * | * | 25 (0.82%) |
| Last biologic received > 2 weeks before 2nd to last biologic (tapering) | 683 (27.91%) | 179 (29.34%) | 862 (28.20%) |
Value may be <11, which cannot be reported in OLDW to avoid data deidentification. The Unknown category corresponds to missing values.
Predicting no asthma exacerbation
In the development cohort assessing the primary endpoint, we found the three models to perform similarly (AUC= GLMnet [95% CI] : 0.75 [0.71,0.78], RF: 0.75 [0.69,0.79], GBM: 0.76 [0.72,0.80], Figure 2 and Table 2). The models equally demonstrated similar performances in the validation cohorts (AUC = GLMnet: 0.72, RF: 0.72, GBM: 0.74), thus demonstrating good generalization properties. The models did not perform as well when assessing the secondary outcome, with AUCs of 0.66, 0.68, and 0.66 in the GLMnet, RF, and GBM, respectively.
Figure 2A.
ROC Curve in Development Cohort
Table 2.
Model Performance Metrics
| Model | Data | Sensitivity | Specificity | PPV | NPV | |
|---|---|---|---|---|---|---|
| LR | Development | 0.75 (0.71,0.78) | 0.72 (0.63,0.81) | 0.67 (0.58,0.75) | 0.88 (0.87,0.91) | 0.42 (0.35,0.47) |
| LR | Validation | 0.72 | 0.68 | 0.64 | 0.87 | 0.37 |
| RF | Development | 0.75 (0.68,0.79) | 0.71 (0.63,0.78) | 0.70 (0.60,0.80) | 0.89 (0.85,0.92) | 0.42 (0.34,0.48) |
| RF | Validation | 0.72 | 0.81 | 0.45 | 0.84 | 0.41 |
| GBM | Development | 0.76 (0.72,0.80) | 0.71 (0.63,0.80) | 0.71 (0.63,0.76) | 0.89 (0.87,0.91) | 0.42 (0.36,0.50) |
| GBM | Validation | 0.74 | 0.73 | 0.58 | 0.86 | 0.38 |
Interpretation
Figure 3 presents SHAP values for the top 10 variables from the GBM model to help explain the models’ predictions. SHAP assigns each variable a weighting score that reflects the contribution of the variable in making individual predictions. Positive SHAP values indicate the variable contributes to make high-risk prediction (no exacerbations), while negative values indicate low risk (exacerbations). Variables on the y-axis are sorted by the sum of absolute SHAP values over all patients, and the x-axis shows the raw SHAP values (impact of variables on the model). Each plotted point represents an individual patient, and the color represents the variable value: red indicates high and green low. We see that fewer exacerbations in the 6 months prior to index, fewer outpatient visits, less days on treatment, having chronic spontaneous urticaria or atopic dermatitis, and not having sinusitis or COPD were associated with successful stopping. As shown by the coloring, the above relationships do not necessarily apply to every patient. While on average (over all patients), some variables indicate an overall greater risk of asthma exacerbation (the points are skewed toward negative values), for some individual patients (points on the plot), these variables may be contributing to greater or lower risk. Consequently, the model creates both population-level and individualized effects for each variable that can be explained and understood.
Figure 3. SHAP summary plot for the top 20 variables of the RF model contributing to predict risk of no asthma exacerbation.
Positive SHAP values (x-axis) indicate the variable contribute to push the model to make no asthma exacerbation, while negative values indicate the variable contribute to push the model to make asthma exacerbation predictions. Variables on the y-axis are sorted by the sum of the absolute SHAP values over all patients. The values adjacent to each variable are the corresponding mean absolute SHAP value for the variable. Each plotted point represents a patient, and the color represents the variable value: red indicates high and green low.
Discussion
We demonstrated that the occurrence of asthma exacerbations after stopping asthma biologics can be predicted with moderate accuracy, using 3 popular machine learning algorithms. We believe this report represents the first development and validation of a machine learning based predictive model for outcomes following cessation of asthma biologics.
Several other studies report on predicting outcomes after non-biologic asthma medication (primarily ICS) is decreased or stopped in people with mild-to-moderately severe asthma. DiMango et al found that a history of asthma exacerbations (HR=1.53, 95% CI 1.06–2.21) predicts asthma exacerbations after stepping down asthma treatments, and that every increase of 10% in FEV1 was associated with a 14% lower hazard ratio.2 In our model, the number of asthma exacerbations before starting treatment was the variable with the greatest contribution to our model. We were not able to assess pulmonary function in our cohort due to lack of availability for most participants in the cohort. Perez de Llano et al used 4 variables—2 lung functions variables, past asthma exacerbations, and asthma control—to derive and validate a model predicting successful stopping of biologic medications in people with moderate asthma with AUC=0.76, sensitivity 81%, specificity 53%, PPV 31%, and NPV 89%.3 The potential importance of including lung function variables is again demonstrated, and the use of a tool to assess asthma control is also raised as a potentially important input. Martinez-Moragon et al found that older people with more severe asthma and a shorter period of control had a higher risk of exacerbations after stepping down asthma medication.5 Similarly, we found that previous asthma exacerbations and length of treatment with biologic were important in our predictive model. Koskelo et al found that asthma control questionnaire responses and FEV1 were able to predict outcomes after stepping down with sensitivities (54%/61%) and specificities (69%/62%) as single predictive variables, respectively.7 These data highlight potential utility of asthma control and lung function inputs. Usmani et al found that a history of asthma exacerbation was associated with exacerbations after stepping down asthma medications (OR=11.67, 95% CI 2.16–91.20), although a low number of events limited the precision with this estimate.8 Bose et al found that fractional excretion of nitric oxide (FeNO) and blood markers of T2 inflammation did not predict outcomes after stepping down asthma medication, although a systematic review and meta-analysis of 384 participants found an OR=3.08, (95% CI 1.36–6.98) for predicting asthma exacerbation if FeNO≥ 50 parts per billion.9,10 Drummond et al used a scoring system based on variable coefficients to predict outcomes in people who stopped montelukast, and, using lung function, age of asthma onset, and history of exacerbations, could predict the outcomes with a sensitivity of 67% and specificity of 68%. Taken together, these studies suggest that variables not available in our dataset may help improve prediction of asthma exacerbation. Our study demonstrates that we can predict outcomes after asthma biologics are stopped using only claims data, which lacks some of these variables.25
There are important limitations to this study. First, there are additional variables which are likely relevant including measures of day-to-day asthma control, physiologic measures like lung function parameters, bioassays, social factors, psychological factors, and adherence to medications that were not captured in the model, but are likely to add to the predictive accuracy of models predicting outcomes after stopping biologics.2–10 These variables are not available in the claims dataset we used. Capturing these variables, improving the predictive accuracy, and building a program allowing for ease of prediction in a busy healthcare setting are all important and necessary steps that will allow better decisions about stopping (or continuing) asthma biologics. Second, most of the data in our cohort were for omalizumab; the 5 biologics in the dataset may be associated with different predictive characteristics and outcomes. Additional information on biologics with low representation in our cohort (e.g. reslizumab) will be needed. Third, the intended primary use of the biologics may not have been asthma for everyone in the cohort. All people in the cohort met our criteria for asthma but may have been prescribed omalizumab for chronic spontaneous urticaria (CSU) or dupilumab for atopic dermatitis (AD) or chronic rhinosinusitis with nasal polyposis. In fact, the SHAP value analysis demonstrates lower risk of exacerbation after stopping biologics in people with CSU and AD, likely because some people in these groups had milder asthma to start, and the biologics were being used primarily for the CSU and AD. Fourth, some people who started biologics may not have been severe or have had consistent use of ICS, which could influence how well the prediction models performed.”
Our study has several strengths. First, we employed a large and diverse cohort of patients with the expected burden of comorbidities who stopped asthma biologics seen across multiple healthcare settings and used 21 potentially relevant variables in three modern machine learning models to predict a clinically relevant outcome. Second, we followed a robust model development through 10-fold cross-validation to prevent model bias and overfitting and validation on a holdout set to assess model generalizability in testing our hypothesis that we could accurately predict asthma exacerbation outcomes after biologics are stopped using claims data. All models yielded consistent results and can be deployed into clinical practice to help support decision making for asthma patients about to stop asthma medications. Third, while the predictive models yielded moderate accuracy in predicting the outcome after asthma biologics are stopped, the accuracy is like other models aimed at predicting asthma prognosis following a change in medical treatment. Finally, a common concern about the use of modern machine learning models in clinical practice is that they are difficult to interpret or explain how they make predictions.26 As such, clinicians and patients may hesitate to use these “black box” tools for critical decision making. Our approach attempts to open the black box and explains in a natural way the reasons why the model is making individual predictions, which can be understood and trusted.27,28
Conclusions
We present the development and validation of a predictive model for asthma exacerbations in people who stop asthma biologics. Additional research using new input variables, simplified model inputs, and translating the statistical model into clinical practice are the next important steps to helping our patients make the best possible decisions about ongoing use of asthma biologics.
Figure 2B.
ROC Curve in Validation Cohort
Funding Source:
National Institutes of Health, National Heart, Lung, and Blood Institute (NIH R21 HL140287) and the Mayo Clinic Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery.
Abbreviations/Acronyms:
- AD
Atopic dermatitis
- CSU
Chronic spontaneous urticaria
- FeNO
Fractional excretion of nitric oxide
- HEDIS
Healthcare Effectiveness Data and Information Set
- ICS
Inhaled corticosteroid
- LABA
Long-acting β-agonist
- MA
Medicare Advantage
- MPR
medication possession ratio
- NPV
Negative predictive value
- OLDW
OptumLabs Data Warehouse
- PPV
Positive predictive value
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Conflict of Interest: None.
References
- (1).Gionfriddo MR, Hagan JB, Rank MA. Why and how to step down chronic asthma drugs. BMJ. 2017;359:j4438. [DOI] [PubMed] [Google Scholar]
- (2).DiMango E, Rogers L, Reibman J, Gerald LB, Brown M, Sugar EA, et al. Risk factors for asthma exacerbation and treatment failure in adults and adolescents with well-controlled asthma during continuation and step-down therapy. Ann Am Thorac Soc. 2018; 15(8):955–961. [DOI] [PubMed] [Google Scholar]
- (3).Perez de Llano L, Garcia-Rivero JL, Urrutia I, Martinez-Moragon E, Ramos J, Cebollero P, et al. A simple score for future risk prediction in patients with controlled asthma who undergo guidelines-based step-down strategy. J Allergy Clin Immunol Pract. 2019;7:1214–21. [DOI] [PubMed] [Google Scholar]
- (4).Saito N, Kamata A, Itoga M, Tamaki M, Kayaba H, Ritz T. Assessment of biological, psychological and adherence factors in the prediction of step-down treatment for patients with well-controlled asthma. Clin Exp Allergy. 2017;(47):467–478. [DOI] [PubMed] [Google Scholar]
- (5).Martinez-Moragon E, Delgado J, Mogrovejo S, Fernandez-Sanchez T, Jesus JL, Angel MOM, et al. Factors that determine the loss of control when reducing therapy by steps in the treatment of moderate-severe asthma in standard clinical practice: A multicentre Spanish study. Rev Clin Esp. 2020;220:86–93. [DOI] [PubMed] [Google Scholar]
- (6).Drummond MB, Peters SP, Castro M, Holbrook JT, Irvin CG, Smith LJ, et al. Risk factors for montelukast treatment failure in step-down therapy for controlled asthma. J Asthma. 2011;48(10):1051–1057. [DOI] [PMC free article] [PubMed] [Google Scholar]
- (7).Koskela HO, Pruokivi MK, Kokkarinen J. Stepping down from combination asthma therapy: The predictors of outcome. Resp Med. 2016;117:109–115. [DOI] [PubMed] [Google Scholar]
- (8).Usmani OS, Kemppinen A, Gardener E, Thomas V, Raju Konduru P, Callan C, et al. A randomized pragmatic trial of changing to and stepping down fluticasone/formoterol in asthma. J Allergy Clin Immunol Pract. 2017;5:1378–87. [DOI] [PubMed] [Google Scholar]
- (9).Wang K, Verbakel JY, Oke J, Fleming-Nouri A, Brewin J, Roberts N, et al. Using fractional exhaled nitric oxide to guide step-down treatment decisions in patients with asthma: a systematic and individual patient data meta-analysis. Eur Respir J. 2020;55:1902150. [DOI] [PubMed] [Google Scholar]
- (10).Bose S, Bime C, Henderson RG, Blake KV, Castro M, DiMango E, et al. Biomarkers of type 2 airway inflammation as predictors of loss of asthma control during step-down therapy for well-controlled disease: The long-acting beta-agonist step-down study (LASST). J Allergy Clin Immunol Pract. 2020;8:3474–81. [DOI] [PMC free article] [PubMed] [Google Scholar]
- (11).Finkelstein J, Wood J. Predicting asthma exacerbations using artificial intelligence. Stud Health Technol Inform. 2013;190:56–8. [PubMed] [Google Scholar]
- (12).Zein JG, Wu C-P, Attaway AH, Zhang P, Nazha A. Novel machine learning can predict acute asthma exacerbation. Chest. 2021;159:1747–1757. [DOI] [PMC free article] [PubMed] [Google Scholar]
- (13).Ledford D, Busse W, Trzaskoma B, Omachi TA, Rosén K, Chipps BE, et al. A randomized multicenter study evaluating Xolair persistence of response after long-term therapy. J Allergy Clin Immunol. 2017;140:162–9.e2. [DOI] [PubMed] [Google Scholar]
- (14).GlaxoSmithKline. Cessation Versus Continuation of Long-term Mepolizumab in Severe Eosinophilic Asthma Patients [Internet]. [cited 2020 Sep 22]. Available from: https://clinicaltrials.gov/ct2/show/NCT02555371
- (15).Jeffery MM, Inselman JW, Maddux JT, Lam RW, Shah ND, Rank MA. Asthma patients who stop asthma biologics have a similar risk of asthma exacerbations as those who continue asthma biologics. J Allergy Clin Immunol Pract. 2021;9(7):2742–2750. [DOI] [PMC free article] [PubMed] [Google Scholar]
- (16).fastshap: Fast Approximate Shapley Values, Greenwell Brandon, 2020. https://CRAN.R-project.org/package=fastshap
- (17).Chang W, Liu Y, Xiao Y, Yuan X, Xu X, Zhang S, et al. A machine-learning-based prediction method for hypertension outcomes based on medical data. Diagnostics. 2019;9(4), 178. [DOI] [PMC free article] [PubMed] [Google Scholar]
- (18).Osawa I, Goto T, Yamamoto Y, Tsugawa Y. Machine-learning-based prediction models for high-need high-cost patients using nationwide clinical and claims data. NPJ Digital Medicine. 2020;3(1), 1–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- (19).Zou H, Hastie T. Regularization and variable selection via the elastic net. J R Stat Soc. 2005;67:301–320. [Google Scholar]
- (20).Breiman L Random forests. Mach Learn. 2001;45:5–32. [Google Scholar]
- (21).Friedman JH. Greedy function approximation: a gradient boosting machine. Ann Stat. 2001:1189–1232. [Google Scholar]
- (22).Friedman J, Hastie T, Tibshirani R. Regularization paths for generalized linear models via coordinate descent. J Stat Softw. 2010;33:1. [PMC free article] [PubMed] [Google Scholar]
- (23).Wright MN, Ziegler A. ranger: A fast implementation of random forests for high dimensional data in C++ and R. arXiv preprint arXiv:150804409. 2015. [Google Scholar]
- (24).Gbm: Generalized Boosted Regression Models, Greenwell Brandon, Boehmke Bradley, Cunningham Jay, GBM Developers, 2020. https://CRAN.R-project.org/package=gbm
- (25).Tyree PT, Lind BK, Lafferty WE. Challenges of using medical insurance claims data for utilization analysis. Am J Med Qual. 2006;21(4): 269–275. [DOI] [PMC free article] [PubMed] [Google Scholar]
- (26).Price WN. Big data and black-box medical algorithms. Sci Transl Med. 2018;10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- (27).Watson DS, Krutzinna J, Bruce IN, Griffiths CE, McInnes IB, Barnes MR, et al. Clinical applications of machine learning algorithms: beyond the black box. BMJ. 2019;364. [DOI] [PubMed] [Google Scholar]
- (28).Lundberg SM, Lee S-I. A Unified Approach to Interpreting Model Predictions. Advances in Neural Information Processing Systems 30. 4765–4774. 2017. Curran Associates, Inc. http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf [Google Scholar]