Primary Care Asthma Attack Prediction Models for Adults: A Systematic Review of Reported Methodologies and Outcomes

Abstract

Prognostic models hold great potential for predicting asthma exacerbations, providing opportunities for early intervention, and are a popular area of current research. However, it is unclear how models should be compared and contrasted, given their differences in both design and performance, particularly with a view to potential implementation in routine practice. This systematic review aimed to identify novel predictive models of asthma attacks in adults and compare differences in construction related to populations, outcome definitions, prediction time horizons, algorithms, validation, and performance estimation. Twenty-five studies were identified for comparison, with varying definitions of asthma attacks and prediction event time horizons ranging from 15 days to 30 months. The most commonly used algorithm was logistic regression (20/25 studies); however, none of the six which tested multiple algorithms identified it as highest performing algorithm. The effect of various study design characteristics on performance was evaluated in order to provide context to the limitations of highly performing models. Models used a variety of constructs, which affected both their performance and their viability for implementation in routine practice. Consultation with stakeholders is necessary to identify priorities for model refinement and to create a benchmark of acceptable performance for implementation in clinical practice.

Introduction

During an asthma attack, people with asthma experience a temporary exacerbation of their symptoms, including wheezing, coughing, breathlessness, and chest tightness, which can result in the need for emergency treatment to prevent fatality.¹ There are many possible triggers of asthma exacerbation, including viruses, allergies, irritants, adverse drug reactions, and air pollutants.^2–7 Asthma presents with high heterogeneity,^8–10 so early identification of worsening of symptoms or lung function is a challenge for clinicians and patients, but there is great hope that machine learning tools may be able to assist and create pathways for early intervention. A recent report by Asthma and Lung UK estimated that respiratory conditions, including asthma, cost the UK economy £188 billion in 2019, highlighting the value in investing in the development of efficient tools to improve clinical outcomes and implement timely interventions.¹¹

In recent years, two systematic reviews have explored asthma attack prediction models with slightly different characteristics and objectives.^12,13 The 2017 systematic review by Loymans et al¹² focused on investigating asthma attack predictors and assessing model performance. In contrast, Bridge et al¹³ (2020, but only including papers up to 2017) focused on comparing the methodology used in the development of prediction models for future asthma attacks. They primarily reported the impact of different model algorithms on predictive performance. The only major difference in their study inclusion and exclusion criteria, however, was that Bridge et al¹³ included studies of patients aged 12 years and over, whereas Loymans et al¹² included studies with a mean population age over 18.

Both reviews place a strong focus on how to obtain the highest model prediction performance. While strong predictive performance is clearly important for maximizing patient benefit and increasing user trust in the tool, which is required to promote integration into existing care pathways, there are elements to the model which may be even more influential on their impact. Some important considerations for the specification of such a model include the explainability of the results (either overall, or for specific patients), the target populations, and the outcome definition (including time horizon of prediction).

The aim of this study was to provide an updated review of the literature, including studies published since 2020 exploring more complex and intensive machine learning approaches. In addition, we aimed to reflect on the differences of these models with a view to their implementation in clinical practice and the balance between desirability and usability with predictive performance.

Methods

The methods and results of this systematic review were reported in line with the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) 2020 statement.¹⁴ The checklist, and the location of information pertaining to each item, is presented in Appendix A.

Search Strategy

Two bibliographic electronic databases were searched in May 2023: PubMed and Embase. The search strategy is given in Appendix B. Reference lists of all included papers were checked for potentially contributory papers, and any relevant papers not identified from our search that were included in the Loymans et al and Bridge et al reviews^12,13 were added. All papers had to be available in the English language.

Inclusion Criteria

Table 1 lists the inclusion and exclusion criteria by study design, population, outcome, setting, and metric. The research population of this systematic review is adults with asthma, however studies were included if the mean age of the population was over 18 years old and it did not include patients under the age of 12. Studies which reported on adults and children separately were included, but only the analyses on adult patients were described herein.

Table 1.

Systematic Review Inclusion and Exclusion Criteria

	Inclusion Criteria	Exclusion Criteria
Study types	Longitudinal study	Cross-sectional studies
Population	Adults with asthma, or an asthma population with a mean age of over 18 and a minimum age of 12.	Population with a mean age of 18 or under, or including patients under the age of 12.
Outcome	Asthma attack more than one week after the index date.	Other asthma event or status, such as uncontrolled asthma or severe asthma, or asthma attack recovery.
Settings	Data collected in a clinical setting, such as in general practice or from an asthma nurse.	Data collected in a non-clinical setting, such as using a home-monitoring device
Model metrics	Reported any of the following model performance measures: a confusion matrix, sensitivity, specificity, PPV, NPC, accuracy, balanced accuracy, F1 measure, AUC and AUPRC	Only reported measures of association, such as odds-ratios.

Abbreviations: PPV, Positive predictive value; NPV, negative predictive value; AUC, Area Under the Receiver-Operator Curve; AUPRC, Area Under the Precision-Recall Curve.

Studies not related to the development or validation of multivariate prediction models were excluded, such as mechanistic studies or those which only reported association measures (such as odds ratios) rather than performance measures (such as sensitivity and specificity).

This systematic review only included longitudinal studies as cross-sectional studies are unable to assess the ability of prognostic models to predict future asthma attacks. In addition, letters, pre-prints, conference abstracts, protocols, book chapters, and literature reviews were excluded.

The outcome of the prediction model was the onset of an asthma attack, and studies that only reported other asthma-related events (such as post-asthma attack hospital discharge) or statuses (such as uncontrolled asthma or asthma severity) were excluded. Additionally, we aimed to exclude studies whose aim was to detect, rather than predict, the clinically evident onset of an asthma attack. That is, the point at which there is contact between the patient and the healthcare provider. The distinction between detection and prediction was ascertained herein by proxy of whether the time between the index date and the asthma attack event was more than one week (if not, the study was excluded).

Finally, non-clinical predictive models were excluded, such as self-management and home-use prediction models.

Study Selection

The search results of the two databases were imported into Covidence and duplicates were automatically deleted. The study selection, including title/abstract screening and full-text review, was done independently by both authors, and consensus was conducted jointly for any discrepancies.

There were 9067 studies included in the title and abstract screening, of which 46 were passed to full-text screening. Twenty-five studies met the review criteria for the data extraction and quality assessment steps, including all of the twelve studies included by the review by Loymans et al¹² and the three of the nine studies included by the review by Bridge et al¹³ (all three of which were included in both reviews). The studies excluded at each stage and the reasons for their exclusion are provided in Appendix C.

Data Extraction

Data extraction was carried out independently by the authors. The data extraction form included elements pertaining to population characteristics, model characteristics and methodology, and model performance evaluation. The full list of data extracted is presented in Appendix D. Where studies reported sufficient information for additional (unreported) binary classification measures to be calculated, these values were highlighted as derived rather than reported.

Results

Study Characteristics

Twelve studies identified in the review by Loymans et al¹² were published between 1997 and 2016. Thirteen additional studies were added in our review, including one additional study in 2004 (which used a three-level risk stratification, but discussed outcomes in which the top risk-level is compared to the lowest and middle risk levels¹⁵) and twelve between 2017 and 2023 (Table 2). Henceforth, studies will be referred to by the surnames of the lead authors. There were two papers by Luo et al (denoted [A]¹⁶ and [B]¹⁷) and Schatz et al (denoted [A]¹⁵ and [B]¹⁸).

Table 2.

Summary of Included Studies

1st Author/ Year	Country	Outcome	Prediction Horizon	Outcome Incidence
Inselman19 (2023)	USA	Composite: asthma hospitalization, asthma emergency department visit, or dispensing of systemic corticosteroids for asthma	6 months	22.6%
Jiao20 (2022)	Canada	Composite: asthma hospitalization, asthma emergency department visit, or dispensing of systemic corticosteroids for asthma	1 year	0.3%
Lisspers21 (2021)	Sweden	Composite: asthma hospitalization, asthma emergency department visit, or dispensing of systemic corticosteroids for asthma	15 days	0.04%
Noble22 (2021)	UK	Hospitalization	1 year	1.7%
Tong23 (2021)	USA	Composite: asthma hospitalization or asthma emergency department visit	1 year	1.7%
Wu24 (2021)	China	Composite: asthma hospitalization, asthma emergency department visit, or dispensing of systemic corticosteroids for asthma	1 year	33.5%
Zein25 (2021)	USA	(A) Asthma hospitalization	Not specified: interquartile range 1 to 4 yearsa	1.5%
		(B) Asthma emergency department visit		2.9%
		(C) Dispensing of systemic corticosteroids for asthma		32.8%
Luo16 (2020) (A)	USA	Composite: asthma hospitalization or asthma emergency department visit	1 year	2.4%
Luo17 (2020) (B)	USA	Composite: asthma hospitalization or asthma emergency department visit	1 year	3.6%
Martin26 (2020)	USA	Composite: asthma hospitalization, asthma emergency department or immediate care clinic visit, or dispensing of systemic corticosteroids for asthma	1 year	54.8%
Schleich27(2020)	Belgium	Composite: asthma hospitalization, or dispensing of systemic corticosteroids for asthma	1 year	Not Reported
Xiang28 (2020)	USA	Composite: asthma hospitalization, asthma emergency department visit, or dispensing of systemic corticosteroids for asthma	1 year	7.2%
Loymans29 (2016)b	Netherlands	Composite: asthma hospitalization, asthma emergency department visit, or dispensing of systemic corticosteroids for asthma	1 year	13.1%
Bateman30 (2015)	International	Composite: asthma hospitalization, asthma emergency department visit, or dispensing of systemic corticosteroids for asthma	6 months	Not Reported
Eisner31 (2012)	USA	(A) Patient-reported exacerbations (recorded in primary care),	1 year	Not Reported
		(B) Unscheduled primary care visits,
		(C) Asthma hospitalization,
		(D) Asthma emergency department visit
		(E) Dispensing of systemic corticosteroids for asthma
Sato32 (2009)	Japan	Composite: 2+ consecutive days of a peak expiratory flow rate of ≤70% of baseline, asthma hospitalization, asthma emergency department visit, or dispensing of systemic corticosteroids for asthma	1 year	20.5%
Osborne33 (2007)	USA	Composite: asthma hospitalization, or asthma emergency department or immediate care clinic visit	30 months	18.2%
Miller34 (2006)	USA	Composite: self-reported asthma hospitalization or asthma emergency department visit	6 months	8.5%
Peters35 (2006)	USA	Composite: asthma hospitalization, or asthma emergency department or other acute care center visit	1 year	10.4%
Yurk36 (2004)c	USA	(A) Self-reported asthma hospitalization	1 year	9%
		(B) Self-reported asthma emergency department visit		35%
		(C) Missed work 5 or more days in the past month due to asthma (self-reported)		36%
		(D) 5 or more asthma attacks per week in the past month or having symptoms most of the time between attacks (self-reported).		53%
Schatz15 (2004) [A]	USA	Composite: asthma hospitalization, or asthma emergency department visit	1 year	4.9%
Schatz18 (2003) [B]	USA	Asthma hospitalization	1 year	1.2%
Lieu37 (1999)	USA	Composite: asthma hospitalization, or asthma emergency department visit	1 year	6.9%
Ellman38 (1997)	New Zealand	Composite: a marked decline in FEV1 (≥1.0L or ≥30% from baseline) or dispensing of systemic corticosteroids for asthma	20 weeks	27.5%
Grana39 (1997)	USA	Asthma hospitalization	1 year	1.8%

Notes: ^aThe study by Zein et al used the entire patient follow-up as the prediction horizon, which varied between patients. The mean was three years, and the interquartile range was 1 to 4 years. The range was not provided. ^bThis study (Loymans et al) has a correction, published in August 2018, which was used in place of the originally reported results. ^cOutcome incidence was only available as an integer for this study (Yurk et al).

Sixteen (64%) studies were conducted in the US. There was a single study from each of the United Kingdom,²² Sweden,²¹ New Zealand,³⁸ Netherlands,²⁹ Japan,³² China,²⁴ Canada,²⁰ and Belgium.²⁷ One study³⁰ was a secondary analysis of three multinational clinical studies, conducted across several different regions, including South Africa, Germany, Canada, France, Australia, the United Kingdom, Spain, and Sweden.

Study Population

The exact cohort inclusion and exclusion criteria differed between studies, but there was a clear distinction that some focused primarily on severe asthma. Most studies (n=20) included patients across all severity levels, with key variations including asthma ascertainment (including by diagnoses, prescriptions, and related healthcare encounters), requirements for stability (such as no exacerbations or infections in the last month), and exclusions for related respiratory diseases, such as COPD. For the remaining five studies, some additional filtering was applied to select only the patients with the highest risk of asthma attacks. Two studies specified the need for recent emergency care encounters (inpatient or emergency department): in the last year for Inselman¹⁹ and in the last three years for Peters.³⁵ Bateman³⁰ only included patients with moderate-to-severe asthma which was considered currently uncontrolled, whereas Eisner³¹ recruited currently stable patients with moderate-to-severe asthma with a history of positive allergy test. Finally, Miller³⁴ recruited patients considered “difficult-to-treat” by their physician. For Loymans,²⁹ the development dataset contained people with varying severity of asthma, but the external validation set specifically contained participants with poor symptom control and low lung function. Schatz [B]¹⁸ included all people with asthma, but tested the models in both the full population and in those with prior emergency department utilization.

Study Outcome

Outcome Ascertainment

Twenty-two studies only considered a single outcome, or composite outcome, while three studies investigated models for multiple different outcomes. These were Zein²⁵ (three outcomes), Eisner³¹ (five outcomes), and Yurk³⁶ (four outcomes). As such, there were 34 study-outcome combinations. While there were nuanced differences between the outcomes considered in each study, such as exact clinical code-lists, they can be broadly grouped into seven categories.

The first three categories relate to single source outcomes. The first category (n=3) was asthma attacks ascertained from primary care data – either through dispensing of systemic steroids (Zein²⁵ and Eisner³¹) or a composite of systemic steroid prescribing and/or marked decline in Forced Expiratory Volume in one second (FEV₁; Ellman³⁸). The second category was asthma-related emergency department visits, as investigated by Zein²⁵ and Eisner³¹ (n=2). The third outcome group was asthma-related inpatient admissions, as investigated by Noble,²² Zein,²⁵ Eisner,³¹ Schatz [B],¹⁸ and Grana³⁹ (n=5).

The next three categories relate to composites of these first three categories, in which any event is considered an attack. Firstly, and the most common outcome, nine studies looked at a composite of systemic steroids, emergency department presentation, and/or hospitalization (Inselman,¹⁹ Jiao,²⁰ Lisspers,²¹ Wu,²⁴ Martin,²⁶ Xiang,²⁸ Loymans,²⁹ Bateman,³⁰ and Sato³²). This definition is aligned with the American Thoracic Society and European Respiratory Society (ATS/ERS) joint task force definition of a severe exacerbation.⁴⁰ The fifth category was a composite of emergency department presentation and/or hospitalization, used by seven studies (Tong,²³ Luo [A],¹⁶ Luo [B],¹⁷ Osborne,³³ Schatz [A],¹⁵ Peters,³⁵ and Lieu³⁷). The sixth category was a composite of systemic steroids and/or hospitalization, used only by Schleich.²⁷

Finally, there were seven miscellaneous outcomes. Eisner³¹ considered both asthma attacks explicitly stated as patient-reported in primary care data, and unscheduled primary care visits. Miller³⁴ used self-reported asthma emergency department visits or hospitalisations. Yurk³⁶ used self-reported emergency department visits, self-reported hospitalisations, “having missed work five or more days in the past month due to asthma” (self-reported), or “five or more asthma attacks per week in the past month or having symptoms most of the time between attacks” (self-reported).

Outcome Prediction Horizon

The outcome prediction horizon is the maximum duration from the index date (the start of the observation period) and the outcome. Most studies use one year as the prediction horizon (n=18/25). Osborne³³ was the study which used a longer prediction horizon (30 months), while six studies used shorter horizons. Three studies used six months as the prediction horizon (Bateman,³⁰ Inselman,¹⁹ and Miller³⁴). Ellman³⁸ used 20 weeks as the prediction horizon (which they refer to as a “treatment period”) and Lisspers²¹ used only 15 days. The Zein²⁵ study was not able to explicitly report their prediction horizon, as it varied by participant.

Outcome Incidence

Only 2.9% of study participants had an emergency department visit in the Zein²⁵ study, and between 1.2 and 1.8% of participants had an inpatient admission in the studies by Noble,²² Zein,²⁵ Schatz [B],¹⁸ and Grana³⁹ (Table 2). In the composite of emergency department presentation and/or hospitalization, the five studies looking at the general asthma population over 12 months reported an incidence of between 1.7% and 6.9% (Tong,²³ Luo [A],¹⁶ Luo [B],¹⁷ Schatz [A],¹⁵ and Lieu³⁷), while the study looking at the severe asthma population in the same time frame reported an incidence risk of 8.5% (Miller³⁴). The Osborne³³ study, which looked at the general asthma population over 30 months, reported an incidence risk of 18.2%.

The incidence risk of asthma attacks ascertained from primary care data (not necessarily requiring emergency care) was estimated as 32.8% in the Zein²⁵ study, and 27.5% in the Ellman³⁸ study. In the studies using a composite outcome of systemic steroids, emergency department presentation, and/or hospitalization, the results were more mixed. The Lisspers²¹ study, which used an event horizon of only 15 days, found an incidence rate of only 0.04% - the lowest of any of the studies. There were six studies which looked at 12 months horizon in a general asthma population, but the incidence risk ranged from 0.31% to 54.8%, with a median of 16.8%.

Three studies did not provide any information about the outcome incidence in their publications: Schleich,²⁷ Bateman,³⁰ and Eisner³¹ (for all outcomes).

Modelling

Statistical Methods

Across the 25 studies, 16 developed and compared multiple-prediction models (Table 3). Multiple algorithms were tested in six studies (Inselman,¹⁹ Lisspers,²¹ Tong,²³ Luo [A],¹⁶ Zein,²⁵ and Xiang²⁸), feature sets in seven studies (Loymans,²⁹ Osborne,³³ Miller,³⁴ Peters,³⁵ Eisner,³¹ Lieu,³⁷ and Xiang²⁸), algorithm hyper-parameters in one (Sato,³² and possibly others which tested multiple algorithms but did not explicitly state this), populations in two (Schatz [B]¹⁸ and Lieu³⁷), and of course outcomes in three (Eisner,³¹ Zein,²⁵ and Yurk³⁶). Additionally, Schleich²⁷ compared models looking at the incidence risk of at least one and at least two asthma attacks, although only the former will be discussed herein. Similarly, in studies which investigated multiple age groups, only the adult (or non-paediatric) population models are discussed herein.

Table 3.

Methodology of Included Studies

1st Author/ Year	Algorithm	Outcome	Validation
Inselman19 (2023):  - Model A  - Model B  - Model C	A. Logistic regression with elastic net B. Random forest C. GBM	Composite: asthma hospitalization, asthma emergency department visit, or dispensing of systemic corticosteroids for asthma	Internal Validation: random split-sample
Jiao20 (2022)	Logistic regression with elastic net	Composite: asthma hospitalization, asthma emergency department visit, or dispensing of systemic corticosteroids for asthma	Internal Validation: random split-sample
Lisspers21 (2021)	XGBoosta	Composite: asthma hospitalization, asthma emergency department visit, or dispensing of systemic corticosteroids for asthma	Internal Validation: random split-sample
Noble22 (2021)	Logistic regression	Asthma hospitalization	External Validation: patients from Welsh national database
Tong23 (2021)	XGBoosta	Composite: asthma hospitalization or asthma emergency department visit	Internal Validation: temporal split
Wu24 (2021)	Logistic regression with LASSO	Composite: asthma hospitalization, asthma emergency department visit, or dispensing of systemic corticosteroids for asthma	Both Internal Validation: cross-validation, and External Validation: patients newly recruited after specific time-point
Zein25 (2021):  - Model A  - Model B  - Model C  - Model D  - Model E  - Model F  - Model G  - Model H  - Model I	A. Logistic regression B. Random forest C. GBM D. Logistic regression E. Random forest F. GBM G. Logistic regression H. Random forest I. GBM	A. Dispensing of systemic corticosteroids for asthma B. Dispensing of systemic corticosteroids for asthma C. Dispensing of systemic corticosteroids for asthma D. Asthma emergency department visit E. Asthma emergency department visit F. Asthma emergency department visit G. Asthma hospitalization H. Asthma hospitalization I. Asthma hospitalisation	External Validation: patients newly diagnosed after specific time-point
Luo16 (2020) (A)	XGBoosta	Composite: asthma hospitalization or asthma emergency department visit	Internal Validation: temporal split
Luo17 (2020) (B)	XGBoost	Composite: asthma hospitalization or asthma emergency department visit	Internal Validation: temporal split
Martin26 (2020)	Logistic regression	Composite: asthma hospitalization, asthma emergency department or immediate care clinic visit, or dispensing of systemic corticosteroids for asthma	Internal Validation: random split-sample
Schleich27 (2020)	Logistic regression	Composite: asthma hospitalization, or dispensing of systemic corticosteroids for asthma	External Validation: patients newly recruited after specific time-point
Xiang28 (2020)	TSANNa	Composite: asthma hospitalization, asthma emergency department visit, or dispensing of systemic corticosteroids for asthma	Internal Validation: random split-sample
Loymans29 (2016)	Logistic regression	Composite: asthma hospitalization, asthma emergency department visit, or dispensing of systemic corticosteroids for asthma	External Validation: U-BIOPRED41
Bateman30 (2015)	Cox regression	Composite: asthma hospitalization, asthma emergency department visit, or dispensing of systemic corticosteroids for asthma	Internal Validation: random split-sample
Eisner31 (2012):  - Model A  - Model B  - Model C  - Model D  - Model E	Logistic regression	A. Patient-reported exacerbations (recorded in primary care) B. Unscheduled primary care visits C. Asthma hospitalization D. Asthma emergency department visit E. Dispensing of systemic corticosteroids for asthma	Training data performance only
Sato32 (2009)	CART	Composite: 2+ consecutive days of a peak expiratory flow rate of ≤70% of baseline, asthma hospitalization, asthma emergency department visit, or dispensing of systemic corticosteroids for asthma	Internal Validation: cross-validation
Osborne33 (2007)b:  - Model A  - Model B  - Model C	Poisson regression	Composite: asthma hospitalization, or asthma emergency department or immediate care clinic visit	Internal Validation: random split-sample
Miller34 (2006)	Logistic regression	Composite: self-reported asthma hospitalization or asthma emergency department visit	Internal Validation: temporal split
Peters35 (2006)b:  - Model A  - Model B  - Model C  - Model D	CART	Composite: asthma hospitalization, or asthma emergency department or other acute care center visit	Training data performance only
Yurk36 (2004)c	Logistic regression	Composite: self-reported asthma hospitalization, self-reported asthma emergency department visit, or self-reporting missing 5 or more days of work in the past month due to asthma	Training data performance only
Schatz15 (2004) [A]	Logistic regression	Composite: asthma hospitalization, or asthma emergency department visit	External Validation: patients from Fontana, California
Schatz18 (2003) [B]:  - Model A  - Model B	Logistic regression	Asthma hospitalization	Internal Validation: bootstrap resampling
Lieu37 (1999):  - Model A  - Model B	CART	Composite: asthma hospitalization, or asthma emergency department visit	Internal Validation: random split-sample
Ellman38 (1997)  - Model A  - Model B	Logistic regression	Composite: a marked decline in FEV1 (≥1.0L or ≥30% from baseline) or dispensing of systemic corticosteroids for asthma	Training data performance only
Grana39 (1997)	Logistic regression	Asthma hospitalization	Internal Validation: temporal split

Notes: ^aIn these studies, multiple algorithms were tested, however only the final selected model(s) are reported. ^bThe studies by both Peters and Osborne had multiple models presented, distinguished by the set of predictive features used in the model. ^cA composite of three of the four outcomes presented in Table 2 was used in the final model for Yurk.

Abbreviations: GBM, gradient boosting machine; CART, classification and regression tree; TSANN, Time-Sensitive Attention Neural Network.

Of these studies, six used the AUC to rank their developed models (Miller,³⁴ Sato,³² Tong,²³ Luo [A],¹⁶ Eisner,³¹ and Xiang²⁸). Additionally, Lisspers²¹ used the Area Under the Precision-Recall Curve (AUPRC), an alternative to the AUC which may be preferred in cases with low outcome incidence. Unlike the other six studies, who simply selected the model with the highest AUC, Eisner³¹ compared two models for each outcome with different sets of features, and selected the model with the fewest features so long as it did not have a significantly lower AUC than the model built with the larger feature set. Lieu³⁷ selected two final models to present, based on “clinical face validity”, while the other eight studies did not rank their developed models.

In the 19 studies which only considered a single algorithm, logistic regression was the primary choice (n=14, including with LASSO or elastic net for feature selection in Jiao²⁰ and Wu²⁴). Bateman³⁰ used Cox regression, Osborne³³ used Poisson regression, Luo [B]¹⁷ used gradient boosting trees, and Sato,³² Lieu³⁷ and Peters³⁵ used CART.

There were six studies which considered multiple algorithms. Both Tong²³ and Luo [A]¹⁶ used gradient-boosting trees and the 39 algorithms native to the WEKA software.⁴² Lisspers²¹ used random forest, gradient boosting trees, recurrent neural network, and logistic regression (with multiple regularization methods). Inselman¹⁹ used elastic-net logistic regression, random forest, and gradient boosting trees. Zein²⁵ used logistic regression, random forest, and gradient boosting trees. Xiang²⁸ used logistic regression, multilayer perceptron, long short-term memory (LSTM), and Time-Sensitive Attention Neural Network (TSANN).

Zein²⁵ and Inselman¹⁹ presented the results (unranked) for each algorithm, but Tong,²³ Luo [A],¹⁶ and Lisspers²¹ ranked the gradient boosting trees (specifically XGBoost, according to AUC and AUPRC, respectively) highest, and Xiang²⁸ ranked the TSANN highest (according to AUC).

Model Validation

Five studies only carried out model development without validation, providing performance estimates in the same data that were used to train the model (Ellman,³⁸ Eisner,³¹ Yurk,³⁶ and Peters;³⁵ Table 3). Sixteen studies conducted internal validation. Eight used a random split sample (Inselman,¹⁹ Jiao,²⁰ Lisspers,²¹ Martin,²⁶ Xiang,²⁸ Bateman,³⁰ Osborne,³³ and Lieu³⁷), five used a temporal split – reserving datum from a later year for testing (Tong,²³ Luo [A],¹⁶ Luo [B],¹⁷ Miller,³⁴ and Grana³⁹), two used cross-validation (Wu²⁴ and Sato³²), and one used bootstrap resampling (Schatz [B]¹⁸). Six studies used external validation (Wu,²⁴ Schatz [A],¹⁵ Noble,²² Zein,²⁵ Schleich,²⁷ and Loymans²⁹).

Model Performance

Seventeen studies calculated the AUC of their model(s), of which 12 also reported binary classification performance measures, and 5 did not (Miller,³⁴ Loymans,²⁹ Einser,³¹ Schleich,²⁷ and Xiang²⁸). Lisspers²¹ reported the AUPRC instead of the AUC. Seven studies presented neither: Bateman,³⁰ Ellman,³⁸ Peters,³⁵ Grana,³⁹ Schatz [A],¹⁵ Lieu,³⁷ and Osborne.³³ The highest reported AUC was 0.93 by Schleich.²⁷

Of the twenty studies that reported at least one binary classification performance measure, fifteen presented both the sensitivity and specificity (for 3 studies, these were the only measures presented: Jiao,²⁰ Wu,²⁴ and Grana³⁹). Of the five that did not, three presented only the PPV (Bateman,³⁰ Ellman,³⁸ and Osborne³³), one the PPV and NPV (Peters³⁵), and one the PPV and sensitivity (Lisspers²¹). Sato³² presented the sensitivity, specificity, and positive and negative likelihood ratios. The other eleven studies all presented the sensitivity, specificity, and PPV. All but Lieu³⁷ (of these eleven) additionally presented the NPV, and three (Martin,²⁶ Tong,²³ and Luo [A])¹⁶ presented the accuracy additionally. Finally, of the twenty studies that reported at least one binary classification performance measure, seven presented a confusion matrix, a data table, or a figure, which would allow other binary classification measures to be calculated (Tong,²³ Luo [A],¹⁶ Luo [B],¹⁷ Ellman,³⁸ Peters,³⁵ Lieu,³⁷ and Sato³²).

In the prediction of rare outcomes, which is the case for many of these models, the majority of test cases will be predicted to not have the outcome, and be correct, resulting in high specificity. However, correctly identifying those cases in which an asthma attack will occur, especially when the data on these cases is outweighed by the negative cases, can be more of a challenge. As such, the sensitivity may be more pertinent to the real world “cost”. As shown in Table 4, the highest reported model sensitivity was 86% (Zein²⁵). The Lisspers²¹ study had the lowest reported sensitivity (7%). However, higher sensitivity often comes at the price of lower specificity, and as such we calculated the balanced accuracy for all studies which provided sufficient information. The results ranged from 50% to 81% for the Tong study.²³

Table 4.

Binary Classification Measure Performance of Included Studies

1st Author/ Year	Sensitivity	Specificity	PPV	Balanced Accuracy	F1 Measure
Inselman19 (2023):
- Model A	68	64	87	66	76
- Model B	81	45	84	63	82
- Model C	73	58	86	66	79
Jiao20 (2022)	72	57		65
Lisspers21 (2021)	7		3		4
Noble22 (2021)	29	93	6	61	10
Tong23 (2021)	70	91	11	81	19
Wu24 (2021)	63		72		67
Zein25 (2021):
- Model A	60	71		66
- Model B	60	67		64
- Model C	64	67		66
- Model D	67	77		72
- Model E	75	78		77
- Model F	84	76		80
- Model G	76	74		75
- Model H	59	86		73
- Model I	86	73		80
Luo16 (2020) (A)	54	92	23	73	32
Luo17 (2020) (B)	52	91	11	72	18
Martin26 (2020)	48	80	74	64	58
Bateman30 (2015)			40
Sato32 (2009)	44	92	58	68	50
Osborne33 (2007):
- Model A			30
- Model B			48
- Model C			48
Peters35 (2006):
- Model A	1	99	40	50	2
- Model B	17	94	25	56	20
- Model C	7	99	52	53	12
- Model D	11	99	48	55	18
Yurk36 (2004)	77	63	82	70	79
Schatz15 (2004) [A]	21	92	17	57	19
Schatz18 (2003) [B]:
- Model A	45	87	4	66	7
- Model B	47	89	14	68	22
Lieu37 (1999):
- Model A	49	84	19	67	27
- Model B	36	92	25	64	30
Ellman38 (1997)
- Model A	11	94	67	53	19
- Model B	27	72	48	50	35
Grana39 (1997)	70	71		71

Notes: ^aColour code: green = reported, orange = derived. ^bFor Yurk, the composite outcome model was reported. For Peters, the top one and two terminal nodes by risk were used to create the high-risk prediction group.

Perhaps even more crucially, high sensitivity typically has a trade-off for high PPV with rare outcome prediction, as the easiest way to increase the sensitivity may be to lower the classification threshold. The ratio of the misclassification costs between a false negative (missing an attack) and a false positive (flagging a low-risk patient) depends on the model setting, including the population and the event horizon (and the corresponding suggested intervention). As a simple investigation, however, we calculated the F1 measure, which is the harmonic mean of the sensitivity and PPV. The results ranged from 2% to 82% (Inselman¹⁹).

Discussion

Summary of Findings

This systematic review of 25 asthma attack prediction models aimed to review key differences in model design and investigate their impact on model predictive performance. The primary distinctions between the models related to the target populations, the outcomes (including the events and the future time horizons for prediction), and the statistical modelling approaches, but there were also fundamental variations in how predictive performance was evaluated, which affected the ability to directly compare model performance.

Most models (n=20/25) did not restrict the model to predicting in patients with severe asthma, and the most common outcome (n=9/25) was a composite including systemic steroid prescription, emergency department presentation, and/or hospitalization, in line with the American Thoracic Society and European Respiratory Society (ATS/ERS) joint task force definition of a severe exacerbation.⁴⁰ Most studies use one year as the prediction horizon (n=18/25), but the horizon ranged from 15 days to 30 months. Logistic regression was the most common algorithm, used in 20/25 studies, including six which tested multiple algorithms (however it was not the highest performing algorithm in any of these studies). Seventeen studies calculated the area under the curve (AUC), and 20/25 reported at least one binary classification performance measure.

Results in Context

The performance of a model should not be considered in isolation and can only be directly compared to models with the same study design: primarily the population, the outcome, the algorithm, and the model validation procedures.

Model performance may be affected by the outcome definition used in multiple ways. Firstly, the prediction event horizon which gives highest model performance will depend on the data used to make the prediction. For example, records from sporadic GP or secondary care contacts will likely not be sufficiently granular to detect change in risk from one week to the next, especially if there was no contact between those dates. Secondly, the outcome ascertainment may result in very heterogenous events being all labelled as an attack, despite great differences in severity and speed of onset. This is particularly likely to affect logistic regression models, which depend on low outcome heterogeneity,^43,44 whereas tree-based models may fare better.

Finally, the incidence of the outcome (itself a function of the ascertainment, event horizon, and study population) is likely to affect performance, particularly if there are limited data on asthma attacks, or if the model is not appropriately set up to overcome self-optimising to maximise the accuracy (which is heavily driven by rarely predicting the event⁴⁵). Generally, asthma attacks identified in both primary and secondary care, with a longer event horizon, and/or a population with higher severity asthma would be expected to have a higher incidence. Substantial variation was observed in the reported incidence of events, however, even in studies with comparable endpoints (by population and definition). This may be due to variable ascertainment of asthma attacks, affected by local clinical coding procedures or population demographics. The validity of administrative health data for purposes secondary to those of their initial collection must always be considered and evaluated where possible through manual review or linkage to other data sources.

Strengths and Weaknesses

This review has two main strengths. Firstly, we have provided an updated review highlighting papers since 2017 which have made use of improved computing capabilities to test a wider array of statistical methodologies.

Secondly, we have compared the performance of asthma attack prediction models by both the measures used in the studies’ evaluation, and the values themselves, contextualized with study design specifics. This allows the models to be contrasted with the nuance of the potential clinical use of the models.

However, the review has limitations. First, we made the decision to exclude studies which we considered to be asthma attack detection rather than prediction. Detection studies typically aimed to generate an automated system for asthma attack ascertainment and validate this against physician-diagnosed attacks. For example, a study by Kupczyk et al used electronic patient diary data to identify the start of an asthma attack.⁴⁶ While the data used were from at least 2 days prior to the attack “start” (defined by when the attack was reported), the clinical onset had clearly already commenced, as evident by the change in patient reported symptoms and measurements. The distinction between the two was ascertained on the basis of whether the time between the index date and the asthma attack event was more or less than one week.

Additionally, this review does not identify a “best” model to use as a benchmark, or indeed to identify the threshold for implementation in clinical practice. A clinically applicable model not only needs to have good predictive ability, but also be accepted by clinicians and patients. For example, a model which could be applied to anyone with asthma, had an outcome which was clinically meaningful and aligned well with a feasible intervention, was easily understandable and explainable, but had 1% lower accuracy than a model with none of these characteristics would almost certainly be preferable. More work is required to identify the most clinically important outcomes (relative to available interventions), populations, and model explainability mechanisms. This can only be conducted by consulting with clinicians, patients, and other stakeholders.

Finally, a risk of bias assessment was not included in this review, as our primary aim was to contrast the study design and modelling methods used. However, note the existence of several relevant publishing guidelines which highlight common pitfalls in study set up and reporting: PROBAST (A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies, by Wolff et al⁴⁷), TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis, by Collins et al⁴⁸), and RECORD (Reporting of studies Conducted using Observational Routinely collected health Data, by Benchimol et al⁴⁹). In particular, the RiGoR (Reporting Guidelines to address common sources of bias in Risk model development, by Kerr et al⁵⁰) guidelines detail the most common sources of bias in leakage between the training and testing data partitions.

Conclusion

The predictive performance is heavily influenced by the study design, including the population, the outcome definition, the algorithm, and the model validation procedures. Identifying the most clinically meaningful model characteristics is necessary to enable a “best” model to be identified and highlight routes for future development. This will boost likelihood of successful translation, adoption, and implementation at scale of clinical prediction models, and to bring benefits to patients.

Author Contributions

All authors made a significant contribution to the work reported, whether that is in the conception, study design, execution, acquisition of data, analysis and interpretation, or in all these areas; took part in drafting, revising or critically reviewing the article; gave final approval of the version to be published; have agreed on the journal to which the article has been submitted; and agree to be accountable for all aspects of the work.

Disclosure

The authors report no conflicts of interest in this work.