Detecting Asthma Exacerbations in a Pediatric Emergency Department Using a Bayesian Network

Abstract

Objective

To develop and evaluate a Bayesian network to identify patients eligible for an asthma-care guideline using only data available electronically at the time of patient triage.

Population

Consecutive patients 2–18 years old who presented to a pediatric emergency department during a 2-month period.

Methods

A network was developed and evaluated using clinical data from patient visits. An independent reference standard for asthma guideline eligibility was established and verified for each patient through chart review. Outcome measures were area under the receiver operating characteristic curve, sensitivity, specificity, predictive values, and likelihood ratios.

Results

We enrolled 3,023 patient visits, including 385 who were eligible for guideline-based care. Area under the receiver operating curve for the network was 0.959 (95% CI = 0.933 – 0.977). At a fixed 90% sensitivity, specificity was 88.3%, positive predictive value was 44.7% and negative predictive value was 98.8%. The positive likelihood ratio was 7.69 and the negative likelihood ratio was 0.11.

Conclusion

The Bayesian network was able to detect patients eligible for an asthma guideline with high accuracy suggesting that this technique could be used to automatically initiate guideline use for eligible patients.

Introduction

Asthma is the most common pediatric chronic disease, affecting more than 6 million children in the US. Asthma exacerbations are frequent indications for Emergency Department (ED) visits, accounting for more than 2 million visits annually¹. National guidelines for the treatment of acute asthma have been developed and compliance with treatment recommendations have been shown to improve measures of clinical care^{2, 3}. Guidelines, however, remain underutilized and studies have demonstrated significant deviations from best care practices in the ED⁴. Requiring clinicians to actively seek out and initiate guideline use presents a significant behavioral bottleneck to guideline adoption and has been associated with low rates of adoption⁵. Automated identification of patients presenting to the ED who are eligible for an asthma guideline could allow electronic enrollment, increase compliance with care recommendations, and expedite initiation of appropriate therapy.

The purpose of this study was to develop and evaluate a Bayesian network (BN) for the detection of patients with asthma exacerbations in a pediatric ED. We sought to assess the feasibility of this technique in triggering guidelines to overcome the barrier of requiring active guideline initiation by providers.

Background

Computerized methods have been used previously to identify asthmatic patients, but the majority of these have been used to screen for prevalent cases of stable asthma⁶. Investigators have administered computerized questionnaires or performed searches of patient medical or billing records. Others have used signal analysis techniques to identify asthma cases from cough and breath sounds. Studies aimed to detect asthma exacerbations include electronically screening free-text ED presenting complaints and searching encounter notes. In these studies, however, detection was not integrated into the normal clinical workflow and they were not implemented or evaluated in a patient-care environment.

BNs have been implemented for disease detection and diagnosis^{7, 8}. They have also been used to screen patients for guideline eligibility for disorders such as pneumonia⁹ and pulmonary embolism¹⁰, but have not been described for the detection of asthma. A BN is a graphical representation of relationships between variables based on probability theory¹¹. It consists of a directed acyclic graph with nodes representing variables associated with a table of conditional probabilities. Nodes are connected by arcs which model the relationships between variables. The probability of a node can be calculated when the values of other variables are known. Advantages over rule-based diagnostic systems include the ability to train a BN using known cases and the tolerance of missing data.

Design Objectives

Our objective was to develop a system that could be implemented into a production clinical environment while remaining generalizable outside of our institution. Design considerations included:

1. Timing

Asthma care in the ED may need to be initiated even before physician evaluation, such as the measurement of respiratory peak flow or trial administration of beta-agonist medications. Early detection allows for prompt guideline enrollment and full support of care recommendations. Therefore, we chose to predict the presence of asthma early in the ED visit and to only include data available at the time of patient triage.

2. Data elements

We included only routinely collected patient data available in electronic form and queryable in real-time. Requiring additional data entry during the registration or triage process would increase staff workload and time demands and could prevent successful adoption. Developing a model using simple data elements which are likely to be widely available in clinical information systems increases the generalizability of the system to other applications or clinical environments.

3. Accuracy

Screening patients for asthma exacerbation requires high sensitivity so that few patients who are guideline-eligible will be missed. High specificity is also desirable so that few false positive predictions will be made. An ideal system will allow for adjustment of sensitivity to maximize desired operational characteristics.

4. Integration

The system needs to be compatible with the normal workflow of the ED and well integrated into the existing electronic infrastructure. Automated evaluation of each patient is necessary to avoid increased workload. Also, the result of the prediction needs to be available in real-time to allow for electronic triggering of computer-based guidelines or the use of a reminder system to support initiation of paper-based guidelines.

Methods

Setting

The study was performed at the Vanderbilt Children’s Hospital ED, an urban 29-bed pediatric facility with more than 40,000 visits annually. Computerized patient care applications include an electronic white board, computerized patient triage system¹², electronic medical record, and a computerized provider order-entry system.

Population

We screened for inclusion all patients aged 2–18 years, the age range eligible for a paper-based asthma guideline in use at our institution. Patients were included if they were triaged and assigned a coded chief complaint, chosen from a standard list of ICD-9 CM diagnosis codes. We excluded patients who were not assigned a coded chief complaint, were triaged but not treated in the ED (due to transfer to another clinical area or leaving before physician evaluation), or whose final diagnosis for the visit could not be determined through chart review. For patients with multiple ED encounters during the study period, each visit was included and evaluated separately. The study was approved by the local Institutional Review Board.

Reference standard diagnosis

We defined a reference standard for asthma exacerbation as a final assigned free-text ED visit diagnosis of “asthma exacerbation”, “status asthmaticus”, “wheezing”, or “reactive airway disease”. We also included patients in the diagnosis standard if asthma exacerbation was considered but was later ruled out by a trial of beta-agonist medication. For each visit, the presence or absence of the reference diagnosis and thus asthma guideline eligibility was evaluated. This was determined after the ED visit through review of written and electronic documentation including the attending physician’s note and a list of all orders performed in the ED.

Data sources

We used information routinely obtained during the triage process as well as data from the electronic medical record. All data were collected prospectively immediately after completion of the triage assessment. From triage, data included the coded chief-complaint, vital signs (temperature, respiratory rate, and oxygen saturation), and the coded presence of a past history of asthma. From the past medical record, data elements were obtained from the electronic problem list which contained free-text lists of past diagnoses, allergies, family history, medications, and social history. We used text-matching to search the past medical history and medication sections for concepts representing asthma and for medications commonly used to treat asthmatics. This technique has been shown to be accurate for identifying a history of asthma¹³. Concepts for asthma included: “asthma”, “reactive airway”, “RAD”, and “ashtma”, a common misspelling. Terms for medications included trade and generic names of inhaled and nebulized beta-agonists, oral and inhaled steroids, and other medications commonly used to treat asthma, such as leukotriene inhibitors. The number of past diagnosis codes for asthma (ICD-9 code of 493.*) was obtained from our electronic billing database.

Network development and outcome measures

We developed the BN with Netica^TM version 2.0, a software system that allows network construction and parameter learning from training data¹⁴. Multiple network structures were developed based on expert medical knowledge. The outcome node was the probability of a patient presenting to the ED having an asthma exacerbation and being eligible for treatment using asthma-care guidelines.

Patients included in the study were randomly divided into four subsets of equal size. Three subsets were used for network development while the fourth was kept as an independent validation set. Each candidate network structure was evaluated using 3-fold cross validation, where two of the development sets were used to train the network and the third was used to estimate classification performance.

Network performance was evaluated by calculating the area under the receiver operating characteristic (ROC) curve¹⁵. The ROC curve is a standard measure of overall test performance, and is obtained by plotting corresponding pairs of true positive classification rates (sensitivity), and false positive classification rates (1 – specificity).

Comparisons of candidate networks were made and the network with the highest performance was chosen. This network was trained with the development subsets and tested with the independent validation subset. The resulting area under the ROC curve (AUC) was calculated. We then determined standard operational characteristics of the network at set 80%, 85%, 90% and 95% sensitivity levels. Finally, an analysis of misclassified cases at the 90% level of sensitivity was performed. For each false positive case, the assigned ED visit diagnosis was examined.

Results

We included 4,023 patient encounters during the study period. We excluded 92 visits: 64 patients were triaged but transferred to another clinic for treatment, 21 patients left before evaluation by a physician, and 6 patients did not have sufficient documentation to determine a final diagnosis. There were 385 (9.6%) patient encounters with the reference standard diagnosis for asthma. Patient demographics for the development and verification data subsets are compared in Table 1. Univariate statistical analysis revealed no significant differences between the development and validation data sets.

Table 1.

Patient demographics for data subsets.

	Development (n=3017)	Validation (n=1006)
Asthma guideline eligibility rate (%)	9.6	9.5
Age (mean years)	7.7 ± 4.9	7.9 ± 5.1
Sex (% Female)	46.2	49.1
Mean Emergency	3.1 ± 0.9	3.1 ± 0.9
Severity Index (ESI)
Admission rate (%)	14.4	16.0

The final network structure with the corresponding value states for each variable is displayed in Figure 1. The average area under the ROC curve for the network across the 3 development subsets was 0.957. The AUC for the validation subset was 0.959 (95% CI = 0.933 – 0.977). ROC curves for the final network are displayed as Figure 2.

Figure 1.

On the left is displayed the Bayesian network structure. On the right are network variables and possible values.

Figure 2.

ROC curves for the final Bayesian network.

Table 2 displays operational characteristics of the final network at fixed sensitivities. A sensitivity of 90% was chosen for an optimal balance in detecting eligible patients and minimizing false positive predictions. Analysis of classifications at 90% sensitivity revealed 86 true positive predictions and 105 false positives resulting in a positive predictive value of 44.7%. There were 805 true negative predictions and 10 false negatives, yielding a negative predictive value of 98.8%

Table 2.

Operational characteristics at fixed sensitivity levels.

Sensitivity (%)	80	85	90	95
Specificity (%)	96.0	93.4	88.3	76.6
Positive PV (%)	67.9	57.4	44.7	29.9
Negative PV (%)	97.9	98.3	98.8	99.3
Positive LR	20.2	12.8	7.69	4.05
Negative LR	0.21	0.16	0.11	0.07

The most common discharge diagnoses for the false positive cases are given in Table 3. Pneumonia was the most common disorder misclassified as asthma exacerbation, followed by upper respiratory infection, viral syndrome, croup, cough, and febrile illness.

Table 3.

Most frequent false positive classifications at 90% sensitivity.

Diagnosis	Number
Pneumonia	15 (14.3%)
Upper respiratory infection	11 (10.5%)
Viral syndrome	8 (7.6%)
Croup	7 (6.7%)
Cough	4 (3.8%)
Febrile illness	4 (3.8%)

Discussion

This study demonstrated that a BN can be applied with high accuracy to detect asthma exacerbations using routinely available electronic data. The developed detection system was compatible with all of the stated design objectives.

Data quality was high, as we collected all information used in the network prospectively and immediately after patient triage. Additionally, we included all patients potentially eligible for guidelines and compared the ED diagnosis for each visit to a reference diagnosis for asthma exacerbation. Data collection required no additional effort by ED staff and did not alter the normal workflow in the triage area. These features suggest the feasibility of integrating such a system into routine clinical use. We used data elements that are becoming increasingly available in electronic format as institutions implement clinical information systems. Thus our system could be generalized to other locations where electronic problem lists and electronic capture of ED triage information are available.

The BN had very high predictive performance with an area under the ROC curve of 0.959. High sensitivity was desirable for our system since it was intended to be used as a screening tool for guideline eligibility. We chose 90% sensitivity as optimal, and at this level specificity remained high at 88.3%. High sensitivity comes with the tradeoff of accepting a lower positive predictive value and consequently more false positive predictions. At 44.7% positive predictive value, 4 of every 10 predictions of a guideline-eligible encounter should be correct. While this value is moderate compared to gold-standard diagnostic tests, it is considered high for an automated real-time screening test that does not require additional data entry.

During the study period a median of 6 patients per day were eligible for the asthma guidelines. This translates to an average of 13 daily asthma predictions by the detection system. Each provider would be expected to see up to 2 guideline-eligible patients per day and receive 1–4 alerts. Analysis of assigned diagnoses for false positive predictions revealed that the system was least able to distinguish asthma from other common respiratory illness such as pneumonia and upper respiratory infection, as well as nonspecific diagnoses such as cough and viral syndrome. These diagnoses are often difficult to distinguish from asthma early in an encounter before diagnostic testing and therapy is begun. For implementation and adoption, an elegant mechanism for opting out of asthma guideline use for false positive predictions would need to be devised. Approaches could be to report the degree of certainty (probability) that a patient has asthma exacerbation or to also provide treatment guidelines for common disorders in the differential diagnosis for asthma exacerbation.

Negative predictive value provides an estimation of how many guideline-eligible patients would be missed by the prediction system. At 90% sensitivity, the negative predictive value was very high at 98.8%. Thus, about 1 in every 100 patients (less than one per day on average) predicted to not be eligible for guidelines would actually have an asthma exacerbation. Alternative means to identify these patients would be necessary. Solutions could include triggers within a computerized order entry system based on common patterns for the treatment of asthma, such as an order for nebulized beta-agonist medications.

A major barrier to guideline use and adoption is the need for active initiation by clinicians. Our system overcomes this limitation by screening patients before physician evaluation and predicting eligibility for guideline-based care. Computerized guidelines could then be automatically delivered to a clinician or a reminder could be generated for the use of paper-based forms. Because our system screens all patients and does not require user interaction, it is scalable. If networks could be developed and deployed for the detection of other disorders, no increased clinical workload would be incurred.

One limitation of this study is that it did not prospectively evaluate the classification performance of the network. However, all data from the patient triage system and the electronic medical record were collected prospectively and only data available immediately after the triage period was used in the model. The BN has recently been integrated into the production environment of the pediatric ED for real-time prediction of asthma guideline eligibility and an evaluation of performance is ongoing.

In summary, our study demonstrates the application of a Bayesian network system for the detection of patients with asthma exacerbation who present to a pediatric emergency department. The system was able to identify guideline-eligible patients with high accuracy early in the clinical encounter using routinely available electronic data and could be used to automate the initiation of guideline-based asthma care.