The Asthma Prediction Rule to decrease hospitalizations for children with asthma

Abstract

Purpose of review

The aim of the present review was to discuss the challenges around clinical decision-making for hospitalization of children with acute asthma exacerbations and the development, internal validation and future potential of the Asthma Prediction Rule (APR) to provide meaningful clinical decision-support that might decrease unnecessary hospitalizations.

Recent findings

The APR was developed and internally validated using predictor variables available before treatment in the ED, and performed well to predict need-for-hospitalization. Oxygen saturation on room air and expiratory phase prolongation were most strongly associated with need-for-hospitalization.

Summary

Research on prediction rules in pediatric asthma is rare. We developed and internally validated the APR using clinically intuitive predictor variables that are available at the bedside. Before incorporation in to electronic decision-support the APR must undergo external validation and an impact analysis to determine if use of this tool will change clinician behavior and improve patient outcomes.

INTRODUCTION

Asthma prevalence is among the most common and costly chronic diseases of U.S. children.[1, 2] Annual U.S. health care costs for asthma are $56 billion, with as much as 50% of directs costs attributable to hospitalizations.[3]

Acute asthma exacerbations are among the most frequent reasons for emergency department (ED) visits and are the most frequent reason for hospitalization of children in the U.S.[4–8] Acute exacerbations result in 640,000 ED visits yearly among children, and 24% of these visits result in hospitalization.[6] Of note, we have reported a similar rate of hospitalization (23%) in our population, but only 16.5% of those with exacerbations meet recognized need-for-hospitalization outcome criteria defined as either length-of-stay > 24 hours if admitted to hospital or relapse within 48 hours if discharged to home.[9–12]

Acute asthma exacerbations are variable in presentation and response to therapy, reflecting the heterogeneity of this complex genetic and environmental disease.[13] This heterogeneity may contribute to the difficulty ED clinicians face in deciding which children with exacerbations need to be hospitalized.[14–18*] Hospitalization decision-making may be challenging even for children with mild exacerbations; we have found that approximately 40% of asthma admissions from our children’s hospital ED do not meet the definition of need-for-hospitalization. Finally, we have demonstrated that most asthma hospitalization decisions can be made in the first 2 hours of ED care, yet this decision-making is prolonged, with median time to decision of 4.8 Hr. (SD 5.1 Hr.).[19]

The purpose of this review is to examine the literature pertaining to modeling and validating clinical prediction rules (CPRs). Further, we discuss how implementation of this decision-support instrument within computerized clinical decision-support (CDS) may decrease unnecessary asthma hospitalizations, thereby reducing the direct and indirect costs of hospitalization on patients, payers and providers.

DIAGNOSTIC UNCERTAINTY IN THE EMERGENCY DEPARTMENT

Commercial aviation, an industry at the forefront of safety, recognized that the majority of adverse incidents involve human error. Further, no matter how careful and well-trained health care providers are, individuals will make mistakes both because they are human and because of systems problems predisposing to error.[20, 21]

The ED practice environment is characterized by several risk factors predictive of error. ED clinicians encounter undifferentiated problems of varying acuity and must deal with high decision density and cognitive loading. ED clinicians face varying levels of experience in their care team, frequent hand-offs of care, and numerous distractions and interruptions.[22] In the context of these practice environment risk factors, ED clinicians also often lack standardized tools for translating ever-changing best medical evidence into bedside decision making for potential complex and high-risk diseases.[20]

Without standardized decision support tools tailored to the practice environment, ED clinicians’ decisions are prone to cognitive errors, many of which stem from clinicians’ use of use heuristics, a strategy of problem solving in which clinical experience leads to cognitive short-cuts that include mnemonics, acronyms and other ‘rules of thumb.”[23, 24] Two strategies have been suggested to avoid the cognitive errors that occur when heuristics fail. The first, metacognition, is “a reflective approach to problem solving that involves stepping back from the immediate problem to examine and reflect on the thinking process.”[25] Cognitive forcing, the second strategy, “implies a deliberate, conscious selection of a particular strategy in a specific situation to optimize decision making and avoid error.”[23] Cognitive forcing strategies include tools such as CPRs to facilitate evidence-based clinical judgment through increasing the degree of specification and providing a rapid and accurate response to individual clinical problems.

ACUTE ASTHMA EXACERBATION SEVERITY SCORES

Among the numerous acute asthma severity scores proposed or in use, only two have been internally validated using an objective and accurate measure of airway obstruction such as FEV₁ or airway resistance.[26,27,28] Inter-observer reliability has received little attention in the development of most scoring systems and scores have been largely comprised of subjective criteria.[29, 30] With these limitations in mind, a systematic review of asthma severity scores concluded that the predictive validity of existing bedside severity scores was inadequate to justify their use for the decision to admit or discharge a patient.[29]

In addition, severity scores have been validated using the hospitalization decision of the clinical team as a criterion measure. This may result in circular reasoning because the score components are used by clinicians to make hospitalization decisions, yet the hospitalization decision is used to ‘validate’ the score. [28, 31, 32] Biostatistical standards for prediction rule modeling and validation circumvent this limitation by masking individuals making clinical decisions from knowledge of the value of predictor variables used in prediction rule modeling. As a result, each patient’s outcome is determined without knowledge of the measured predictor variables.[33] These model building strategies are fundamental to CPRs but have not been incorporated into studies of bedside severity scores.[34*]

This principle is illustrated by the divergence of variables that remain in the reduced-form APR (discussed below) with those of the Acute Asthma Intensity Research Score (AAIRS, Table 1), a bedside severity score validated against %-predicted FEV₁. Whereas bedside severity scores may assist the clinical team in calibrating treatment intensity to exacerbation severity, they may not be assumed to predict outcomes with the validity of the APR.

Table 1.

Acute Asthma Intensity Research Score (AAIRS) and Asthma Prediction Rule (APR) components

Component	AAIRS Bedside severity score	APR Clinical prediction rule
Age	−	+
Gender
Need for albuterol > 2/week
Air entry	+	−
Accessory muscle use
Wheezing
SpO2 on room air	+	+
Expiratory phase prolongation

Abbreviations: AAIRS, Acute Asthma Intensity Research Score; APR, Asthma Prediction Rule; SpO₂, oxygen saturation by pulse oximetry on room air.

CLINICAL PREDICTION RULES

A CPR is “a decision-making tool, which is derived from original research and incorporates three or more variables from the history, physical examination, or simple bedside tests.”[35, 36] A CPR is developed by applying statistical techniques to determine weighted combinations of predictor variables that categorize heterogeneous groups of patients into risk groups.[33,36,37**,38**] As such, CPRs assist clinicians in dealing with uncertainty in clinical decision making, as well as in predicting prognosis and enhancing the efficiency of resource utilization.[33, 39**]

CPRs are intended to augment rather than supplant clinician decision-making. They do not replace clinicians’ qualitative reasoning or dictate an evaluation or management algorithm.[33, 40*] Indeed, during the modeling and validation phase, CPRs must be compared to clinicians’ quantitative suspicion of the outcome of interest.[40]

A CPR can provide clinical decision-support to risk-stratify patients. Additionally, a CPR may identify patients for whom more resource-intensive interventions and the potential for adverse effects of these interventions may reasonably be avoided. In addition, clinicians are generally concerned more about a rule’s sensitivity than specificity because greater value is appropriately assigned to true-positive decisions to provide correct care when patients need it.[41*] Nonetheless, avoiding unnecessary hospitalizations for children with acute asthma exacerbations is important to improve resource utilization, for which high specificity and true-negative decisions are also desirable.

Significant contributions to the refinement of CPRs included clinical and biostatistical methodological standards for their development. Clinical standards were first enumerated by Wasson and colleagues.[35, 42] These include standards for derivation and validation as well as a four-level hierarchy of evidence (Tables 2 and 3) based upon the manner and extent of clinical validation to which a CPR has been subjected.[43] Optimally, CPRs achieve level one hierarchy and “can be used in a wide variety of settings with confidence that they can change clinician behavior and improve patient outcomes.”

Table 2.

Process of CPR Development¹

Steps in Development of a CPR

Step 1. Derivation: Identify factors with predictive power Step 2. Validation:   a. Narrow validation: Apply the CPR in a clinical setting and population as in Step 1.   b. Broad validation: Apply the CPR in multiple clinical setting with varying prevalence and spectrum of disease. Step 3. Impact Analysis: Provide evidence that the rule changes physician behavior and improves outcome and/or reduces costs.

Standards for Derivation of a CPR

1. All important predictors are included in the derivation process. 2. All important predictors are present in a significant proportion of the study population. 3. All outcome events and predictors are clearly defined. 4. Those assessing the outcome event are blinded to the presence of the predictors and vice versa. 5. Sample size is adequate. 6. The CPR makes clinical sense (sensibility).

Standards for Validation of a CPR

1. Subjects are chosen in an unbiased manner and represent a wide spectrum of disease severity. 2. There is a blinded assessment of the criterion standard for all subjects. 3. There is an explicit and accurate interpretation of the predictor variables and the CPR without knowledge of the outcome. 4. There is 100% follow up of enrolled subjects.

Based on: McGinn TG et al.⁴³

Table 3.

Hierarchy of Evidence for Clinical Prediction Rules

Strength of Evidence	Evidence required
Level 1: Rules that can be used in a wide variety of settings with confidence that they can change clinician behavior and improve patient outcomes	At least 1 prospective validation in a different population and 1 impact analysis, demonstrating change in clinician behavior with beneficial consequences
Level 2: Rules that can be used in various settings with confidence in their accuracy	Demonstrated accuracy in either 1 large prospective study including a broad spectrum of patients and clinicians or validated in several smaller settings that differ from one another
Level 3: Rules that clinicians may consider using with caution and only if patients in the study are similar to those in the clinician’s clinical setting	Validated in only 1 narrow prospective sample
Level 4: Rules that need further evaluation before they can be applied clinically	Derived but not validated or validated only in split samples, large retrospective databases, or by statistical techniques

Based on: McGinn TG et al.⁴³

CPR DEVELOPMENT

Biostatistical standards for CPR development have been advanced by Harrell, Moons, and Steyerberg.[44, 45–51] Moons and colleagues recently summarized key principles and standards for CPR development, internal validation, implementation, impact analysis and external validation in a new setting.[37**, 39**, 52**, 53**, 54] Additionally, a recent advance is the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis Or Diagnosis (TRIPOD) standards.[34, 55, 56]

Predictive performance of CPRs is assessed using calibration and discrimination.[33, 44, 52**, 57] Calibration measures the accuracy of the predicted probability of the outcome provided by the model with the observed frequency of the outcome, assessed graphically using a plot of observed outcome frequencies against predicted outcome probabilities across a range of individual predicted risks. Discrimination is the ability of the model to distinguish between patients with different outcomes, quantified using the rank-order c-index that can assume values of 0 – 1, with a value of 1 indicating perfect discrimination.

Predictor variables for CPR modeling should be clearly defined, be clinically available, and enter the scoring system consistent with the manner in which each predictor becomes available in the clinical setting.[35, 46, 58] Stability and face validity are optimized by pre-selection of a limited set of candidate predictor variables based on clinical and biological plausibility.

The outcome to be predicted must be clinically relevant, important to stakeholders, and clearly defined.[35, 59] Identifying an outcome meeting these criteria may be challenging. For example, death from respiratory failure is clearly defined and of great concern but too infrequent to justify the expense of developing a CPR.[60]

With this in mind, an NIH expert panel convened to standardize asthma outcomes included hospitalizations on their list of recommended asthma-related healthcare outcome measures, with hospitalization for ≤ 23-hours be reported separately.[61] For stakeholders in broader clinical contexts (e.g., population health or preventive care) asthma-related hospitalizations have a universally negative value. However, for stakeholders relevant to the clinical context in which a CPR is implemented (i.e., patients presenting to the ED for an acute asthma exacerbation) hospitalizations may or may not indicate a worse outcome. Unnecessary hospital admission results in wasted resources, missed school/work and the risk of iatrogenic injury, whereas failure to admit when necessary results in delayed care and avoidable complications.

The other aforementioned criterion for a CPR outcome measure is that it is clearly defined. Most ED encounters result in one of two dispositions–discharge or hospitalization. In defining the correctness of ED discharge decisions, the most common measure is the absence of a return ED visit or other unscheduled visit (relapse) within 48 or 72 hours, although the construct validity of this measure is controversial.[20, 62–64] In defining the necessity of hospitalization, prior study has used the presence of medications or interventions that could not be provided in an outpatient context.[65] The medications or interventions available in an outpatient context vary widely, based on patient factors (e.g., access to care), hospital characteristics (e.g., asthma education provided) and community characteristics (e.g., air quality).[66–68]

Thus, we sought to define the necessity of an asthma hospitalization by length of stay (LOS), as it is easier to define clearly and uniformly than the presence of medications or interventions not available in an outpatient context. Asthma hospitalizations are typically brief, with mean length of stay (LOS) of 1.9 days in a national cohort of 44 children’s hospitals,[69] 3 days in New York State hospitals[12] and 2 days in 18 hospitals in King County, WA.[9] Moreover, 42% of the Washington children were hospitalized for only 1 day, and these children are likely those in whom the decision to admit to hospital is influenced by subjective and situational factors rather than medical indications for admission.[9] With these considerations in mind, LOS of greater than 24 hours or relapse within 48 hours was chosen as the outcome measure, need-for-hospitalization, for APR development.

CPR EXTERNAL VALIDATION AND IMPACT ANALYSIS

Before implementing a CPR in practice, it is necessary to externally validate the CPR in a new stream of patients in order to assess whether predictor variables perform as well in a new population as in the development population.[37**, 39**] Indeed, there has been a lack of external validation in published CPR studies.[54]

Additionally, an underlying assumption when a CPR is implemented in practice is that by accurately estimating the probability of the outcome of interest, clinicians’ decision-making and their patients’ outcomes will improve. This must be assessed by performing an impact analysis in which the effect of using the CPR on clinician behavior, clinical outcomes and/or cost of care is quantified. In contrast to external validation, this requires a comparative study, for example one in which clinicians are cluster randomized to receive CPR output or not.[38**] An additional feature of these studies is that they should incorporate clinicians’ input on the method of implementation.

Finally, a critical implementation feature is whether the CPR is directive or assistive for translating predictions into decisions.[41] A directive prediction rule explicitly recommends a specific decision, whereas an assistive prediction rule provides the probability of the outcome of interest without a specific recommendation. For some clinical encounters clinicians may prefer a directive rule that is based on a cut-point, for example the Ottawa Ankle Rule for obtaining x-rays.[70] However, hospitalization decision-making for pediatric acute asthma exacerbations is complex and includes consideration of the social situation, ability of the parent-child dyad to comply with outpatient treatment, and ability to return if necessary, in addition to the probability of need-for-hospitalization based on physiologic exacerbation severity. With this in mind, we believe that assistive APR implementation is most appropriate, in which the clinician will use the APR probability of need-for-hospitalization as one data point amongst others to be considered.

THE ROLE OF PREDICTION RULES IN CLINICAL DECISION-SUPPORT

The “Meaningful Use” incentive program as part of the Health Information Technology for Economic and Clinical Health Act (HITECH) is intended to spur “significant improvements in care” through implementation of electronic health records.[71] The breadth of computerized decision support efforts to date in the EHR is beyond scope for this review, but it is important to highlight efforts in the domain of pediatric asthma. Prior asthma-related CDS includes aids for diagnosis of clinical deterioration through patient electronic self-report,[72] providing evidence based guidelines on asthma action plans at point of care,[73, 74] and asthma self-management applications built for mobile devices.[75] A Cochrane review of the latter category in 2013 demonstrated insufficient evidence regarding integration of mobile device-enabled asthma self-management applications into the clinical workflow.[75] Tools such as “asthma kiosks” to collect disease-specific data from parents during ED visits have been assessed with small effects on quality measures.[76]

The literature on CDS to impact decision-making on clinical disposition is less well-studied. It includes a randomized controlled trial by Dexheimer et al in which a probabilistic model intended to detect the diagnosis of an asthma exacerbation was evaluated in a trial in the context of an asthma management system in the EHR; the primary process measure of time to disposition decision was not significantly different between intervention and control arms of the trial.[77*] A number of predictive risk scores examining “need for hospitalization” have been validated in the literature but have not been implemented as CDS in a trial design to our knowledge.

THE ASTHMA PREDICTION RULE (APR)

The features of pediatric acute asthma exacerbations fulfill Stiell’s criteria for need of a CPR:[58]

1. The high prevalence of and expense to our health care system of acute asthma exacerbations;[3–8]

2. The existence of expert-panel guidelines for asthma that advocate assessment of signs, symptoms and functional tests of lung function that contrast with the limited availability of spirometry and other tests of lung function available to clinicians in acute care settings;[78]

3. The variation in practice and limited adherence to expert-panel guidelines;[79, 80]

4. The perspective of some clinicians that recommended diagnostic tests such as spirometry are unnecessary;[81, 82] and

5. That clinicians’ accuracy of exacerbation assessment is highly variable.[14, 83, 84]

Computerized clinical decision support (CDS) has been effective in providing clinician alerts for primary preventive care.[85] For example, a personalized chronic asthma management system reduced the rate of exacerbations among patients with poorly controlled asthma.[86] To our knowledge there have not been CPRs or CDS directed toward hospitalization decision-making for acute asthma exacerbations in pediatric patients.

An initial step in developing the APR was to identify candidate predictor variables suitable for inclusion in APR modeling.[87–90] We then developed the multivariable APR in a population of children with acute asthma exacerbations in accordance with clinical and biostatistical standards for CPR development.[19, 44, 49, 51, 57, 91] The full-model APR included 15 variables available at the time of triage (before treatment in the ED), and displayed high prognostic performance to predict need-for-hospitalization, measured using the widely accepted criterion measures calibration and discrimination.[46, 92] Calibration of the APR was high and discrimination good with c-index measured 0.74.

An APR with fewer variables and decreased complexity that can be incorporated within normal ED workflow will increase the uptake and usability of the prediction rule for clinicians.[37**] An APR developed in this way will be of use only if it performs comparably to the full-model APR. With these principles in mind, we modeled a reduced-form APR using backward step-down variable selection in accordance with biostatistical principles for CPR modeling.[11*, 57] The reduced-form model comprises 5 variables (Table 4) and displayed high calibration and discrimination (c-index 0.73). Of note, oxygen saturation on room air and expiratory phase prolongation were most strongly associated with need-for-hospitalization. A nomogram (Figure 1) may facilitate use of the APR at the bedside, and the underlying algebraic formula may be incorporated in to electronic decision-making tools.

Table 4.

Reduced-model Asthma Prediction Rule modeled in 928 patients aged 5–17 years with acute asthma exacerbations.

Predictor variable	aOR (95%CI) for need-for-hospitalization
Age (change from 6.9 to 11 years)	1.5 (1.0 – 2.1)
Female gender	1.4 (1.0 – 2.1)
SpO2 (change from 98% to 94%)	2.8 (2.1 – 3.6)
Need for albuterol > 2/week	1.3 (0.9 – 1.9)
Inspiratory to Expiratory ratio (<=1:3 vs. 1:1)	4.4 (2.3 – 8.6)

Abbreviations: aOR (95%CI), adjusted odds ratio (95% confidence interval); SpO₂, oxygen saturation by pulse oximetry on room air.

Adapted from Arnold et al.¹¹

Figure 1.

Asthma Prediction Rule need-for-hospitalization reduced-form nomogram. For an individual patient, the points (top grid-line) for each predictor variable are assigned and the total points for all predictor variables are calculated. A vertical line from this value on the Total Points grid-line to the bottom-most grid provides probability of need-for-hospitalization.

Abbreviations: Exp, expiratory; Ins, inspiratory. Taken from (11).

FUTURE STEPS FOR THE APR: REDUCING ASTHMA HOSPITALIZATIONS

As noted, we have reported a rate of hospitalization (23%) in our investigation of children presenting to the ED with acute exacerbations, yet only 16.5% met recognized need-for-hospitalization outcome criteria defined as length-of-stay > 24 hours if admitted to hospital or relapse within 48 hours if discharged to home.[6, 9–12] We believe that the APR has potential to decrease unnecessary hospitalizations for asthma. Whether this can be accomplished effectively and safely will at least in part be determined by the manner of APR implementation.

As noted, a CPR may be implemented as an assistive or a directive decision-making tool. Although we believe an assistive approach is appropriate, we intend to obtain input from clinicians prior to a randomized trial to externally validate the APR and to measure impact on relevant outcomes. To assess whether APR implementation is safe will necessitate careful follow-up of parent-child dyads who are discharged to home after use of the APR.

A final consideration is whether to provide APR-calculated probability of need-for-hospitalization for all patients with exacerbations or to target the intervention to patients at greater risk of unnecessary hospitalization. With this in mind, patients who present to the ED with mild exacerbations are at greatest risk of unnecessary hospitalization and at least risk of an adverse outcome if discharged to home. Specifically, these patients are likely to return for care should the exacerbation not respond as expected or become more severe at home. Moreover, we have found that the rate of unnecessary hospitalizations is has high as 40% among patients with mild exacerbations.

CONCLUSION

Clinical prediction rules augment the judgment of clinicians, and this decision-support may facilitate improved care and resource utilization for common diseases for which there is unnecessary variability in assessment and treatment. The APR has been developed and internally validated using predictor variables that are clinically intuitive and available at the bedside before treatment. Before incorporation in to electronic decision-support the APR must undergo external validation and an impact analysis to determine if use of this decision-support will change clinician behavior and improve patient outcomes.

KEY POINTS.

• Acute asthma exacerbations are the most frequent reason for hospitalization of children in the U.S.

• Clinicians have limited decision-support tools to inform the decision whether to hospitalize a child with an exacerbation.

• There is great variability in decision-making for treatment and hospitalization of children with exacerbations.

• The Asthma Prediction Rule (APR) has the potential to provide clinical decision-support for asthma hospitalization and to improve resource utilization and decrease the burden of this disease on children and families.

• External validation and an impact analysis are needed before incorporation of the APR into clinical decision-support.

Abbreviations

APR

Asthma Prediction Rule

CDS

Computerized decision support

CPR

Clinical Prediction Rule

ED

Emergency department