Predictive machine learning algorithm for COPD exacerbations using a digital inhaler with integrated sensors

Abstract

Purpose

By using data obtained with digital inhalers, machine learning models have the potential to detect early signs of deterioration and predict impending exacerbations of chronic obstructive pulmonary disease (COPD) for individual patients. This analysis aimed to determine if a machine learning algorithm capable of predicting impending exacerbations could be developed using data from an integrated digital inhaler.

Patients and methods

A 12-week, open-label clinical study enrolled patients (≥40 years old) with COPD to use ProAir Digihaler, a digital dry powder inhaler with integrated sensors, to deliver their reliever medication (albuterol, 90 µg/dose; 1–2 inhalations every 4 hours, as needed). The Digihaler recorded inhaler use through timestamps, peak inspiratory flow (PIF), inhalation volume, inhalation duration, and time to PIF throughout the study. By applying machine learning methodology to data downloaded from the inhalers after study completion, along with clinical and demographic information, a model predictive of impending exacerbations was generated.

Results

The predictive analysis included 336 patients, 98 of whom experienced a total of 111 exacerbations. PIF and inhalation volume were observed to decline in the days preceding an exacerbation. Using gradient-boosting trees with data from the Digihaler and baseline patient characteristics, the machine learning model was able to predict an exacerbation over the following 5 days with a receiver operating characteristic area under curve of 0.77 (95% CI: 0.71–0.83). Features of the model with the highest weight were baseline inhalation parameters and changes in inhalation parameters before an exacerbation compared with baseline.

Conclusion

We demonstrated the development of a proof-of-concept machine learning model predictive of impending COPD exacerbations using data from the integrated digital reliever inhaler. This approach may potentially support patient monitoring, help improve disease management, and enable pre-emptive interventions to minimise exacerbations.

Clinical trial registration number

NCT03256695.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Data that can be captured and recorded by a digital inhaler may be able to be used to predict impending exacerbations of chronic respiratory disease, as has previously been demonstrated in asthma.

WHAT THIS STUDY ADDS

• Patients with chronic obstructive pulmonary disease (COPD) participated in a clinical study in which they used a digital inhaler with an electronic sensor that recorded usage and inhalation parameters. These data were used as inputs to a machine learning model, which demonstrated the ability to predict exacerbations within 5 days.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• This study demonstrates proof of principle that data from digital inhalers have predictive utility with respect to COPD exacerbations. The potential for such data to be used to prevent exacerbations in the future warrants further exploration.

Introduction

Chronic obstructive pulmonary disease (COPD) is a leading cause of death worldwide.¹ COPD exacerbations, especially when resulting in hospital admission, can impact health-related quality of life, healthcare resources and mortality.^{2 3} Exacerbations requiring hospitalisation in COPD are a key target for risk-stratified treatment.^{4 5} A challenge of COPD care is to reliably identify patients at risk of an impending exacerbation, which could result in hospitalisation or respiratory failure. Inhalation parameters, patterns of inhaler technique and short-acting beta₂ agonist (SABA) use prior to an impending exacerbation could potentially be used to detect early clinical deterioration. Early intervention, informed by reliable detection of impending exacerbation, could prevent deterioration, reduce hospitalisation and improve outcomes.

Dry powder inhalers (DPIs) require appropriate technique, which involves performing a full exhalation, followed by a deep and forceful inhalation, in order to adequately deaggregate drug particles for delivery to the lungs.6,8 Peak inspiratory flow (PIF) is a well-established measure of inhalation technique, and PIF and inhalation volume (InhV) are important drivers of particle deaggregation when using DPIs.^{8 9} As a patient’s clinical status deteriorates, SABA use may increase, but worsening COPD could decrease PIF and InhV due to hyperinflation.^{10 11} Although increased SABA use has been associated with poor clinical outcomes including exacerbations, access to patient-level inhaler data is required to gain a better understanding of this association.12,15

Previously, oscillometry parameters have shown promise as a potential source of day-to-day variability data with potential to support predictive insights with respect to worsening of COPD.¹⁶ However, in order for data on these parameters to be obtained, the patient is required to use a specialised self-measurement device on an ongoing basis. By contrast, PIF and InhV can be obtained through the ordinary use of a digitally enabled inhaler. ProAir Digihaler (Teva Pharmaceuticals, Israel) is a digital DPI approved by the US Food and Drug Administration.¹⁷ Integrated sensors timestamp device actuation (opening of the cap, which prepares a dose) and accurately measure inhalation parameters, including PIF, InhV, inhalation duration and time to PIF. In a study of 150 participants, the Digihaler was shown to provide accurate inhalation parameter measurements when used by patients.¹⁸ In a previous proof-of-concept analysis, data from asthma patients who used the ProAir Digihaler in a pragmatic clinical trial were combined with baseline demographic information and clinical data as inputs to a machine learning model. This model demonstrated predictive capabilities with respect to asthma exacerbations within a 5-day window.¹⁹ We therefore hypothesised that data obtained from usage of the ProAir Digihaler by patients with COPD in a similarly designed study might be suitable for use as inputs to a machine learning model designed to predict exacerbations of COPD.

Machine learning methods are increasingly used for a variety of clinical applications to allow for interrogation of high-dimensional data sets, such as data from multiple sources.20,24 In a machine learning approach, computer algorithms use data inputs and develop a model capable of making accurate predictions for use with future data inputs.²⁵ Supervised learning, in which the model is ‘trained’ to classify data using input data with corresponding labels (or outcome), can determine if any relationships exist between the input data and the outcome label. After the model has been trained, it attempts to classify new data inputs based on the previous training data.²⁶ One established approach to machine learning algorithms is the use of decision trees, which hierarchically classify data inputs according to prespecified characteristics.²⁷ Decision trees are used by gradient-boosting, a supervised machine learning technique, to combine and optimise diverse data inputs in an iterative process that seeks to maximise the accuracy of a probabilistic model.²⁸ Such a gradient-boosting algorithm was used in the model described in this article.

Materials and methods

Study design and participants

The 12-week, open-label, single-arm study from which data were obtained for this analysis enrolled adult patients (≥40 years old) with a physician diagnosis of COPD and at least one moderate or severe exacerbation in the 12 months prior to screening. The diagnosis included spirometry within the prior 3 months with forced expiratory volume in 1 s (FEV₁)/forced vital capacity <70% predicted and FEV₁<80% predicted. Enrolled patients were those receiving SABA plus at least one of the following: long-acting beta₂ agonist (LABA), combined inhaled corticosteroid/LABA, long-acting muscarinic antagonist (LAMA) or combination LABA/LAMA. Patients with congestive heart failure or any confounding underlying lung disorder other than COPD were excluded. The study was conducted across 40 study centres in the USA between September 2017 and April 2018 and consisted of a 2-week screening period and a 12-week intervention period (NCT03256695). A schematic overview of the study is shown in figure 1. Details of sample size selection are found in online supplemental methods. Prespecified primary outcomes recorded in the intent-to-treat population are reported in online supplemental table 1. Adverse event data are reported in online supplemental table 2).

Figure 1. A schematic overview of the study. COPD, chronic obstructive pulmonary disease.

Exacerbations were defined according to the 2009 American Thoracic Society/European Respiratory Society recommendation.²⁹ Moderate exacerbations were those involving worsening respiratory symptoms for at least two consecutive days requiring administration of systemic corticosteroid (SCS) above baseline and/or systemic antibiotics, or an unscheduled healthcare provider visit (eg, doctor’s office or emergency care) associated with an exacerbation. Severe exacerbations were those requiring both administration of SCS/antibiotics, as above, and a hospitalisation. Patients continued their current maintenance therapy but were required to replace all other reliever medication containing SABA or short-acting antimuscarinic agents with ProAir Digihaler (albuterol, 90 µg/dose) as their reliever medication (1–2 inhalations every 4 hours, as needed) for the duration of the study. Patients were trained on correct inhaler technique during the baseline visit (study day 1) and received seven Digihaler devices for use during the study: three were provided on day 1 and a subsequent four additional devices were provided by courier on day 21.

An electronic module integrated into the Digihaler contains sensors that recorded a timestamp with each use, along with PIF, time to PIF, InhV and inhalation duration. The data from the Digihaler were downloaded at the end of the study by appropriately designated and trained personnel using extraction software. A companion Digihaler mobile phone application (App) was not used during this study, as it was not available at the time.

Patients were aware of the recording of measurements by the Digihaler, as per informed consent. Patients were contacted by phone on a monthly basis for collection of information regarding exacerbations, maintenance medication and adverse events. During this monthly call, instructions for use were also discussed.

Throughout the study period, COPD maintenance medication was continued or altered as per the treating physician’s judgement. Although measurements were recorded in real-time by the Digihaler, data were downloaded after the patients’ treatment period had concluded; therefore, any treatment modifications during the study were based on patient report and not data from the Digihaler. The use of nebulised albuterol for treatment of acute exacerbations at home or at the hospital could not be prohibited for safety reasons and was therefore permitted if deemed necessary by the patient or their physician. Additionally, patients could continue use of other medications, with any modifications at the discretion of their treating physician.

Predictive model

Enrolled patients who made at least one valid inhalation (PIF ≤120 L/min with no usage errors) from the Digihaler and completed the study were eligible for inclusion in the predictive analysis dataset. Patients with exacerbations during the first 10 days of the study or those who made no inhalation during the 4 days preceding an exacerbation were excluded from the predictive analysis dataset.

The target of the predictive model was defined as the prediction of an exacerbation within the following 5 days. The basis for selecting 5 days for the predictive window was to balance the accuracy of the prediction against providing adequate time to allow for treatment intervention. A shorter 3-day model would have provided better accuracy, but the time to intervene before the predicted exacerbation would be restricted; a 7-day model would allow for additional time for intervention but would likely have been less accurate.

Qualitative and quantitative aspects of albuterol Digihaler use were the primary measures for the predictive analysis. The model was used each day to predict whether the patient would have an exacerbation within the following 5 days. Predictions where an exacerbation was anticipated to occur within the following 5 days were described as positive predictions, and negative predictions were those where no exacerbation was anticipated to occur within the following 5 days. Each day’s prediction was based on features input into the model, including comparisons of data on numbers of inhalations and inhalation parameters during the preceding days (‘days prior to prediction’) with the baseline features (figure 2). Respiratory symptoms were not collected after enrolment; therefore, they were not used in the predictive model. Data on actual lung function could have been important additional variables, but these were not collected during the study.

Figure 2. A patient example showing prediction periods and confirmed exacerbation on study day 67 in relation to albuterol use between study days 45 and 75, with data on albuterol use, peak inspiratory flow, and inhalation volume for the same patient over the full study period. The vertical dashed line represents a confirmed exacerbation. *Predictions were made on every study day. Predictions where an exacerbation was anticipated to occur within the following 5 days were described as positive predictions, and negative predictions were those where no exacerbation was anticipated to occur within the following 5 days.

The development of the predictive model involved applying machine learning techniques to a combination of case report form data taken on study day 1 (age, body mass index (BMI), systolic and diastolic blood pressure, previous exacerbations and the number of exacerbations and hospitalisations in the previous 12 months), data from the Digihaler prior to (and including) the day of the prediction, and patient baseline inhalation features from the Digihaler (timestamp of inhalation, inhalation status, PIF, time to PIF, inhalation volume and inhalation duration). The patient baseline inhalation features for the predictive model were the number of inhalations and mean (SD) of each inhalation parameter during the first 10 days of the study. A feature engineering process was conducted in order to determine the most relevant features for the model.

Patient results were randomly divided into three groups to train the model (‘training set’), test and optimise the model (‘test set’) and validate the chosen model (‘validation set’). The predictive model utilised a gradient-boosting trees²⁸ approach, which demonstrated satisfactory performance on the test set. XGBoost³⁰ implementation of gradient-boosting uses a tree learning algorithm optimised for the handling of sparse data to iteratively combine trees and optimise the predictive model.

The predictive performance metrics of the algorithms were compared using a fourfold cross-validation technique. Receiver operating characteristics (ROC) curve of sensitivity versus specificity was used to evaluate the predictive performance of the algorithms. The ROC area under the curve (AUC) value indicated the model’s capability to separate between classes. ROC AUC values range from 0 to 1, with 1 representing perfect performance of the model.³¹

Statistical analyses and algorithm development were performed using R Statistical Software (V.3.6.1; R Foundation for Statistical Computing, Vienna, Austria), on a Windows operating system. The R package xgboost_1.7.5.1 was utilised to implement the gradient-boosting machine learning algorithm. Descriptive statistics were used to report demographics and outcome measures, but no between-group statistical comparisons were made.

Ethics

The clinical study from which data were used in this analysis (NCT03256695) was conducted in full accordance with the Declaration of Helsinki and International Conference on Harmonisation Guidelines for Good Clinical Practice. All study documents and procedures were approved by the Duke University Health System Institutional Review Board (Pro00085456 and Pro00109774). Written informed consent was obtained from each patient before study participation.

Patient and public involvement

No patients or public were involved in the design, recruitment, conduct or interpretation of this study.

Results

Study populations

Overall, 423 patients with COPD were screened for enrolment and 405 patients were enrolled (intention-to-treat (ITT) population). Of these, 336 patients performed at least one valid inhalation using the Digihaler and completed the study; 98 of whom experienced a total of 111 exacerbations. Patients who experienced an exacerbation during the first 10 days of the study (n=10) or did not make an inhalation from the Digihaler during this period (n=32) were excluded from the predictive analysis population. Also excluded were patients who did not use the Digihaler in the 4 days preceding an exacerbation (n=11) and one patient who did not use the Digihaler after the first 10 days of the study (figure 3). Thus, the predictive analysis population comprised 282 patients, with a total of 80 moderate or severe exacerbations experienced by 74 (26%) of the patients during the 12-week study.

Figure 3. Patient disposition. ITT, intention-to-treat.

Baseline demographics and maintenance medication use of the predictive analysis population are shown in table 1; 57% of patients were female and the mean age was 65 years. Patients experienced a mean of 1.5 exacerbations in the 12 months prior to the study. The estimated annualised exacerbation rate for patients during the study was 1.26 exacerbations per patient per year. For patients with at least one valid inhalation during the study, daily inhalations ranged from 0 to 64. Baseline demographics and maintenance medication use of the ITT population are shown in online supplemental table 3.

Table 1. Baseline demographics and maintenance medication use: predictive model analysis population.

	Predictive model analysis population,n=282
Mean age, years (range)	65.4 (45–88)
Women, n (%)	161 (57.1)
Mean BMI, kg/m2 (SD)	28.8 (7.2)
Race, n (%)
White	250 (88.7)
Black or African American	31 (11.0)
Mean number of exacerbations in the previous 12 months (SD)	1.5 (0.9)
Maintenance medication, n (%)
ICS	2 (0.7)
ICS/LABA	104 (36.9)
ICS/LAMA	2 (0.7)
ICS/LABA/LAMA	117 (41.5)
LABA	2 (0.7)
LAMA	23 (8.2)
LABA/LAMA	32 (11.3)

BMI, body mass index; ICS, inhaled corticosteroid; ITT, intent-to-treat; LABA, long-acting beta₂-agonist; LAMA, long-acting muscarinic agent.;

Inhalation parameters: predictive analysis dataset

Over the full study period, the mean (SD) PIF for predictive analysis eligible patients who completed the study with at least one valid inhalation from the Digihaler was 66.3 (17.5) (table 2). An increase in daily albuterol inhalations was observed during the 30 days preceding the exacerbation peak (figure 4). Exacerbating patients also had a higher number of daily albuterol inhalations during the ±14-day window compared with baseline (mean (SD) 3.54 (4.56) vs 3.20 (4.03)) and compared with patients who did not have an exacerbation (2.61 (3.71)). For exacerbating patients, mean inhalation volume was similar during the ±14-day window from exacerbation peak compared with exacerbation-free periods (1.16 L (SD 0.56) vs 1.18 L (SD 0.52)) (figure 5). PIF was similar in exacerbating patients (during the ±14-day window) and non-exacerbating patients (figure 6); however, PIF and inhalation volume were observed to decline in the days prior to the exacerbation peak. A representative example of a patient who experienced an exacerbation is shown in figure 2. This patient had an exacerbation on day 67; this figure demonstrates the predictive periods leading up to this exacerbation and the changes in inhalation parameters before and after the exacerbation.

Table 2. Inhalation parameters and reliever albuterol use over the full study period captured by the Digihaler in patients who completed the study with≥1 valid inhalation.

	Predictive analysis eligible, N=336		With exacerbations,n=98		Without exacerbations, n=238
	Mean (SE)	95% CI	Mean (SE)	95% CI	Mean (SE)	95% CI
PIF, L/min	66.3 (17.5)	37.2 to 106.4	66.4 (15.9)	40.1 to 102.6	66.2 (18.2)	36.4 to 107.2
Inhalation volume, L	1.26 (0.59)	0.41 to 2.75	1.17 (0.54)	0.44 to 2.52	1.30 (0.61)	0.40 to 2.83
Inhalation duration, s	1.57 (0.71)	0.63 to 3.55	1.45 (0.60)	0.64 to 2.88	1.63 (0.74)	0.63 to 3.75
Time to PIF, s	0.42 (0.22)	0.16 to 0.96	0.39 (0.21)	0.15 to 0.91	0.43 (0.22)	0.17 to 0.97
Albuterol inhalations, n/day	2.82 (3.88)	0 to 13	3.32 (4.23)	0 to 14	2.61 (3.71)	0 to 12

PIF, peak inspiratory flow.;

Figure 4. Reliever albuterol use preceding a patient’s first COPD exacerbation during the study (mean). Data are shown for patients who completed the study and made ≥1 valid inhalation from the Digihaler. The vertical dotted line represents a confirmed exacerbation. Mean data presented in figure should be interpreted with caution, as number of inhalations is non-normally distributed. COPD, chronic obstructive pulmonary disease.

Figure 5. Inhalation volume for exacerbating patients during the ±14 days from exacerbation (mean (SD)). The vertical dotted line represents a confirmed exacerbation. Vertical grey bars represent the SD.

Figure 6. PIF for exacerbating patients during the ±14 days from exacerbation. The vertical dotted line represents a confirmed exacerbation. Vertical grey bars represent the SD. PIF, peak inspiratory flow.

Predictive factors and model validation

The predictive algorithm used a combination of baseline characteristics, inhalation parameters (including number of daily inhalations, night-time usage, PIF and inhalation volume) and trends in the days preceding an exacerbation. The training set comprised 160 patients, the test set comprised 59 patients and the validation set comprised 63 patients. Using the validation set, the gradient-boosting model predicted an exacerbation within the following 5 days with an ROC AUC value of 0.77 (95% CI 0.71 to 0.83; figure 7; online supplemental table 4). The strongest predictive factors for a future exacerbation were features based on baseline inhalation parameters (29%), comparison of inhalation parameters measured prior to an exacerbation with those at baseline (20%) and trends in inhalation parameters prior to the prediction being made (19%) (figure 8). Overall, 80% of features used to develop the model were informed by inhalation parameters; other important features included night-time usage and trends in usage over time.

Figure 7. Receiver operating characteristic curve of sensitivity and specificity of a model for predicting upcoming exacerbations within 5 days. Selected risk percentiles shown in table are based on model score, for example, the sensitivity and specificity values for risk percentile 1 are those for the 1% of data points with the highest predicted probability for an exacerbation within 5 days. Further details of selected risk percentiles are provided in online supplemental table 4.

Figure 8. Features making the greatest contribution to the model performance. The 4 days prior to prediction refers to the 4 days before the prediction was made and not the days prior to an exacerbation. For the predictive model, baseline features (number of inhalations and inhalation parameters) were those measured during the first 10 days of the study. Features shown are those that were found to contribute to the model performance. Other features besides these (not shown) were also input to the model and may have affected the probability of a patient experiencing an exacerbation.

Discussion

Data from the use of a digital reliever inhaler by participants in a pragmatic clinical study were used to develop a proof-of-concept predictive model for COPD exacerbation in patients with a history of prior exacerbations. Importantly, age, BMI, exacerbation history and medical history comprised only a small proportion of the features ultimately used in the prediction model. Specifics of the individual baseline inhalation parameters and changes prior to exacerbation peak contributed most of the data that informed the model. This model represents, to our knowledge, the first reported attempt to develop a COPD exacerbation predictive model from individual real-life use of a reliever inhaler device with integrated sensors.

Increased SABA inhaler use and worsening symptoms are associated with poor and potentially declining disease control;³² however, patients often have a poor recognition of their symptoms, worsening respiratory mechanics and changes in inhaler use.³³ Digital inhalers that inform patients and clinicians of increased SABA use (time/date recordings) can lead to better medication management.³⁴ The Digihaler device in the clinical study whose data were used in this analysis measures inhalation parameters and time-stamps reliever use. These inhalation measurements verify that an inhalation attempt was made as well as provide information on aspects of technique known to be important for drug deposition.^{35 36} One notable finding of the clinical study was that some patients habitually used very high levels of SABA medication. Besides the prediction algorithm, the reports available from the Digihaler showing high use of SABA, with associated time/date information, may alert the clinician to the need to re-evaluate COPD symptoms and levels of dyspnoea. Management can then be adjusted accordingly, including optimising maintenance therapy or improving management of comorbid conditions.

The explosion in the development of digital health technologies has enhanced interest in machine learning, whereby large volumes of data can be analysed and presented in a clinically meaningful way. Machine learning has the potential to promote a more personalised approach to medical care.

Changes in PIF and inhaled volume were incorporated into this model. Early hyperinflation potentially associated with an impending exacerbation may be reflected in decreased inspiratory reserve volume, PIF and inhaled volume. PIF is known to deteriorate with hyperinflation and exacerbations.⁷ Furthermore, these are the first reported findings indicating that inhaled volume measured through an integrated digital inhaler has the potential to provide important insights on COPD clinical status. Inhaled volume may even be a more reliable indicator of changes in lung function than PIF, as it is less influenced by inhalation effort or technique and could become a new biomarker of clinical deterioration. These inhalation parameters are, therefore, not only important measures of technique but may also provide insight into early pre-exacerbation physiologic changes. In a similar study employing the albuterol Digihaler in adult patients with uncontrolled asthma, both PIF and inhaled volume decreased prior to exacerbations.¹⁹ Increases in albuterol use prior to exacerbations were more evident among those patients than in the corresponding study of patients with COPD, a finding potentially suggestive of differing SABA usage patterns that further studies are needed to confirm. In both asthma and COPD studies, changes in inhaled volume around exacerbations were more substantial than changes in PIF.¹⁹

Many clinical, behavioural and social factors may be predictive of COPD exacerbation risk. A feature engineering process was conducted when developing this model to determine which features would be most relevant. Age and BMI added accuracy to the model but may not do so with a different data set, thus highlighting the importance of predictive modelling. Furthermore, the observation that night-time usage of albuterol was an important contributor to the model is a novel finding that is a direct outcome of digital monitoring. Thus, trends may become apparent through predictive modelling that are unexpected or counterintuitive. Lung function data and respiratory symptom data were not used in this model but may have been important additional variables had they been available. Further refinement of the model may enhance the predictive power and, ideally, would predict impending exacerbations sooner prior to the exacerbation, allowing the patient and clinician to act earlier on the data earlier to improve outcomes.³⁷ This could decrease the risk of severe COPD exacerbations, which are associated with increased mortality.³⁸

The Digihaler did not provide any feedback to the participant or clinician per protocol design; data were only downloaded and analysed at the end of the clinical study. The Digihaler companion mobile phone application and dashboard were not included in the clinical study. Future clinical studies could build on the approach used in this study to evaluate the use of digital systems to communicate data directly to clinicians and/or patients in real time and to assess whether notification of impending exacerbations would allow for earlier intervention to avert the exacerbation or decrease severity.

Several limitations of the analysis described in this article should be noted. First, there was a relatively small number of exacerbations. More events may have allowed for refinement of the model with improved specificity and a longer time prior to the exacerbation. For optimal clinical utility, the prediction of an exacerbation needs to occur sufficiently early to allow for effective intervention. Second, exacerbations were collected based on patient recall during the monthly phone calls. It is possible that dates were imprecise and/or exacerbations were missed, which may have affected the precision of the model. Unreported use of other short-acting bronchodilators or alternative aerosol devices, particularly nebulisers, in the patient’s home setting or during any emergency department visit or hospitalisation could not be ruled out. Therefore, use of reliever medication other than that delivered by the Digihaler was not accounted for in the model. Real-time data collection using the full Digihaler system, with phone app and dashboard, could have more precisely identified the time of an exacerbation by avoiding timestamp errors from data downloaded and analysed at the end of the clinical study. Combining these data with other information, such as subjective symptoms or other physiologic changes that could be detected with a phone app or wearable device, may enhance the diagnostic utility of the model.³⁹

While the ROC AUC provides an overall measure of the performance of the model, selected thresholds from the curve will be useful to determine its accuracy and clinical applicability. Within the structure of the study, for each individual patient, a positive prediction day (the day on which an exacerbation occurred) is a rare event relative to the full study duration of 90 days. In other words, only a small fraction of the predictions reflect exacerbation-positive days. Therefore, a high number of ‘false positives’ are reported. However, if the output of the model is a minimally impactful intervention such as an alert issued via an electronic interface to patients and HCPs indicating that an exacerbation may be imminent, even a threshold with a relatively high number of false positives has potential utility in clinical practice. For instance, in COPD patients who are at very high risk of exacerbations (eg, those at a risk percentile of ≤5 in figure 7) and are therefore associated with extremely high healthcare resource utilisation, a relatively low sensitivity (38%) and high false-positive rate (83%) might still be clinically useful and cost-effective, and therefore acceptable, if it predicts an exacerbation before it becomes severe and necessitates hospitalisation.

The analysis was not designed to compare the performance of a range of modelling techniques on the dataset, but rather to assess the performance of the gradient-boosting algorithm that was previously applied to a comparable asthma dataset.¹⁹ Previous reports have suggested that other approaches such as logistic regression models may achieve comparable predictive success to machine learning models such as that used in our study.⁴⁰

The findings reported highlight the potential opportunities offered by a predictive model driven by day-to-day patient data to provide useful clinical insights and inform target interventions. We do, however, recognise that this is only a first step in the journey and there is great potential for refinement and improvement of the model. For instance, we have demonstrated predictive capabilities within 5 days of the exacerbation peak; whereas in the clinical study, inhalation changes were noted as early as 30 days prior to the exacerbation peak. We, therefore, plan to continue to explore possible approaches to improving the sensitivity of prediction further in advance of exacerbations while maintaining an acceptable specificity.

By applying the model to data from distinct cohorts, for instance, from real-world use of a digital inhaler as opposed to its usage in a pragmatic clinical trial, we might in the future hope to demonstrate generalisability of this approach beyond the study setting. We might also hope to broaden the applicability of the model beyond the strictly defined COPD population from the source clinical study, for instance, by including patients with overlap syndromes with elements of asthma or bronchiectasis who were ineligible for the study.

Finally, we are continuing to investigate the optimal approach to communicating insights derived from the model to patients and healthcare professionals and to providing suitable education on how best to act on these insights to address symptomatic worsenings and, where possible, prevent exacerbations.

Conclusion

Our findings show that data from a digital reliever inhaler can be used to build a model predictive of near-term acute COPD exacerbations. This proof-of-concept model represents the first step in the development of an algorithm predictive of exacerbations and will inform future development and validation of predictive models from inhaler use data in respiratory care.

Data availability statement

Data are available upon reasonable request.