CURB-65 Pneumonia Severity Assessment Adapted for Electronic Decision Support

Abstract

Background:

Accurate severity assessment is crucial to the initial management of community-acquired pneumonia (CAP). The CURB-65 (confusion, uremia, respiratory rate, BP, age ≥ 65 years) score contains data that are entered routinely in electronic medical records and are, thus, electronically calculable. The aim of this study was to determine whether an electronically generated severity estimate using CURB-65 elements as continuous and weighted variables better predicts 30-day mortality than the traditional CURB-65.

Methods:

In a retrospective cohort study at a US university-affiliated community teaching hospital, we identified 2,069 patients aged 18 years or older with CAP confirmed by radiographic findings in the ED. CURB-65 elements were extracted from the electronic medical record, and 30-day mortality was identified with the Utah Population Database. Performance of a severity prediction model using continuous and weighted CURB-65 variables was compared with the traditional CURB-65 in the US derivation population and validated in the original 1,048 patients from the CURB-65 international derivation study.

Results:

The traditional, binary CURB-65 score predicted mortality in the US cohort with an area under the curve (AUC) of 0.82. Our severity prediction model generated from continuous, weighted CURB-65 elements was superior to the traditional CURB-65, with an out-of-bag AUC of 0.86 (P < .001). This finding was validated in the international database, with an AUC of 0.85 for the electronic model compared with 0.80 for the traditional CURB-65 (P = .01).

Conclusions:

Using CURB-65 elements as continuous and weighted data improved prediction of 30-day mortality and could be used as a real-time, electronic decision support tool or to adjust outcomes by severity when comparing processes of care.

Community-acquired pneumonia (CAP), along with influenza, is the eighth leading cause of death in the United States.¹ About 6 million cases are reported annually, resulting in an estimated 500,000 to 1.1 million hospitalizations.^2,3 Treatment costs $8.4 billion to $10 billion annually, with the cost of inpatient care 25 times that of outpatient care.³ The outcome of CAP is extremely variable and depends upon the affected host’s response, the underlying pathogen, and the treatment delivered. Site of care is a major decision in the management of pneumonia and is guided by estimated severity of the patient’s disease.^4,5 However, variability in physician decision making regarding the need for hospital admission has been repeatedly demonstrated.^6-8 Therefore, accurate severity assessment during initial management is critical.

Two severity assessment tools have become widely used by clinicians to help distinguish high-risk patients who require inpatient management from those able to thrive with outpatient management. The pneumonia severity index (PSI) developed by Fine and colleagues⁹ is a prognostic model that calculates a severity-of-illness score based on 20 separate patient characteristics, including underlying comorbidities. The CURB-65 score is composed of five separate elements: confusion, uremia, respiratory rate, BP, and age ≥ 65 years (Table 1). This was initially developed by the British Thoracic Society in 1987 as the CRB criteria,¹⁰ later modified to the CURB-65 and validated by Lim et al¹¹ in an international, multicenter derivation/validation study from Nottingham, England; Christchurch and Waikato, New Zealand; and Alkmaar, The Netherlands.

Table 1.

—Elements of the CURB-65 Severity Score

Confusion

Abbreviated mental score ≤ 8 or orientation < 3 (time, place, self)

BUN ≥ 20 mg/dL

Respiratory rate ≥ 30 breaths/min

Systolic BP < 90 mm Hg or diastolic BP < 60 mm Hg

Age ≥ 65 years

Score of 0 or 1: mild to moderate CAP. Need additional assessment but may consider outpatient management.

Score of ≥ 2: severe CAP. Recommend hospitalization.

Data from Reference ². CAP = community-acquired pneumonia; CURB-65 = confusion, uremia, respiratory rate, BP, age > 65 years.

Both the PSI and the CURB-65 are accepted methods of assessing severity by the American Thoracic Society⁴; the British Thoracic Society recommends use of the CURB-65.⁵ Although the PSI has been shown to be slightly more accurate at predicting outcome,^12-16 CURB-65 is simpler to use. Additionally, all elements of the CURB-65 are routinely entered into the electronic medical record, making it possible to generate a mortality prediction automatically for each patient at the point of care.

CURB-65 attributes a point to each criterion in a dichotomous and equally weighted fashion. However, excluding confusion, CURB-65 elements are actually continuous variables that may not be of equal predictive value. CURB-65, therefore, may be more accurate if calculated with continuous and weighted variables. Instead of a severity score, a computer could generate an individualized mortality risk estimate using data elements from the electronic medical record. Generating an automated, accurate mortality estimate immediately available to providers could improve severity assessment and thus improve care.

Our aims were to:

• 1. Validate the CURB-65 severity assessment tool in patients with CAP presenting to a US ED using information from the electronic medical record.

• 2. Examine whether a computer-generated CURB-65 using continuous and weighted variables available electronically (e-Table 1) improves accuracy in mortality prediction over the traditional dichotomous CURB-65.

• 3. Validate our derived severity prediction model in an international multicenter database of patients with pneumonia.¹¹

Materials and Methods

Study Population

The study was performed at Latter-day Saints (LDS) Hospital, a university-affiliated community teaching hospital in Salt Lake City, Utah, with 520 beds. An electronic medical record was implemented in the ED in 1995, followed that same year by a paper-based, standardized pneumonia treatment protocol.¹⁷

In a retrospective analysis of data in the electronic medical record, we identified all patients > 18 years of age evaluated in the ED at LDS Hospital from 1996 to 2006 with International Statistical Classification of Diseases, 9th edition (ICD-9) codes specific for a primary diagnosis of pneumonia (480-487.1), or respiratory failure or sepsis (518.x, 038.x) as a primary diagnosis with pneumonia secondary. Patients diagnosed with aspiration pneumonia (507.0), immunocompromised conditions including AIDS or receipt of antiretroviral therapy (042), solid organ transplants (v420-7), hematologic malignancies (v106.x, 200.x, 208.x), and those meeting criteria for health-care-associated pneumonia¹⁸ were excluded. We included only the first episode of pneumonia in a given 12-month period; however, 45 individuals presented with more than one case of pneumonia outside a 12-month period. Chest radiograph and CT scan reports upon admission to the ED were manually reviewed for radiographic evidence of pneumonia by three physicians, with a three-way κ agreement score of 0.95 for a random sample of 100 reports. Patients lacking radiographic evidence for pneumonia were excluded. This study was approved by the LDS Hospital institutional review board and the Utah Population Database. Patient consent was not required.

Validation Cohort

Patient records contained in the database from the prior international CURB-65 derivation and validation study¹¹ were used to validate the model derived in our population. The study included patients with CAP in England, The Netherlands, and New Zealand. Exclusion criteria were similar to our study, with the exception that only patients admitted to the hospital as an emergency were included. Data for calculation of CURB-65 were recorded prospectively upon hospital admission, with 30-day all-cause mortality determined during follow-up.

Measurements

The first 6 hours of vital signs, orientation status (entered routinely during nursing assessment), and the first 12 hours of routine laboratory results were extracted from the electronic medical record. A manual review of 50 charts showed > 90% agreement between electronic and paper records; the majority of the disagreements were minor discrepancies in BP that were not significant enough to result in a change in CURB-65 classification. The methodology of an electronic, continuous-weighted CURB-65 requires a specific value for BUN. The PSI and CURB-65 conventionally assume normality for missing data. Additionally, CRB-65 has been validated as a predictive score for patients lacking a BUN.^11,15,19 Of the 304 patients missing BUN values, 268 were not admitted to the hospital. Therefore, for patients lacking a BUN determination, we imputed a median BUN matched by age quartile derived from nonadmitted patients in our cohort (e-Table 2). This same imputation was applied to the 40 patients with missing BUN values in the validation cohort. The outcome measure of 30-day all-cause mortality was determined by both the Intermountain Healthcare electronic database and the Utah Population Database.²⁰

Because many patients had several measurements of CURB elements during their ED evaluation, we created summary measures (first, last, minimum, and maximum) to determine which measure to carry forward. The summary measure for each element was selected in a forward stepwise selection procedure using out-of-bag area under the curve (AUC) for the receiver operating characteristic as a selection criterion. The first measurements of vital signs and orientation status performed as well or better than subsequent measurements and were thus used in the analysis. Additionally, systolic BP was used, as it was found to be nonsignificantly better than diastolic BP, and the latter did not add any additional predictive value.

Statistical Analysis

In the derivation (US) cohort, we compared 30-day mortality for the CURB-65 model vs a lasso penalized logistic regression model. Nonlinear effects were expected for CURB-65 elements, so natural cubic splines with three knots were applied to the BUN, respiratory rate, BP, and age elements. The lasso penalty (L1-5) was imposed on the logistic regression parameters to prevent overfitting, especially at extreme values and after using spline transforms.²¹

In order to avoid overestimating modeling performance during the derivation phase, the prediction modeling strategy was evaluated within the derivation cohort using a bootstrap sampling method.²² The method resulted in 63.2% of the original observations (“in-the-bag”) available to construct the logistic regression model and 36.8% of observations withheld (“out-of-bag”) for assessing predictive performance across each of 1,000 iterations. The final model sent to the international cohort for validation was based on the entire US derivation cohort. Receiver operating characteristic curves and P values for differences between the discrete CURB-65 score and the logistic regression model were then calculated and compared between the US and the international cohorts.^23,24 Analyses were performed using the R statistical package, version 2.9 (www.r-project.org).²⁵

Results

We identified 4,973 cases; 3,289 of those had radiographic evidence of pneumonia. We excluded 621 for risk factors for health-care-associated pneumonia and previous diagnosis of pneumonia within 1 year. Restricting the study population to patients evaluated in the ED reduced the number to 2,076 cases in 2,027 patients. Seven patients lacked an electronic record of orientation, respiratory rate, or systolic BP during the 6-h time window, and were, therefore, excluded from analysis. The resulting study population comprised 2,069 cases, of which 58% resulted in admission to the hospital (Fig 1). In the validation cohort, 20 patients lacked a vital sign and were thus excluded from the comparison.

Figure 1.

Study population. HCAP = health-care-acquired pneumonia.

Seventy percent of the patients in the LDS Hospital cohort presented with a CURB-65 score of 0 or 1. The mean age was 57 years, and 48% were women. Compared with the international CURB-65 validation study, our study population was younger (mean age 57 vs 64 years, P < .001 by Welch t test), had a lower mortality rate (4.1% vs 9.1%, P < .001 by Fisher exact test), and included patients discharged from the ED as well as those admitted to the hospital (Table 2). Of the patients from our study population who were admitted to the hospital, the age (64 years, SD 19 years) and mortality rate (6.8%) were not significantly different from the international study population (P = .931 and .059, respectively, by Fisher exact test).

Table 2.

—Patient Demographics

	US Cohort
Measure	Total	Inpatients	Outpatients	International Cohort
Study population, No. (%)	2,069	1,199 (58)	87 (42)	1,048
Mean age, y (SD)	57 (18)	63 (18)	46 (18)	64 (18)
% Women	48.0	48.8	47.2	48.5
30-d Mortality, % (CI)	4.1 (3.3-5.1)	6.80 (5.48-8.42)	0.34 (0.07-1.00)	9 (7.4-11.0)

A strong relationship between mortality and CURB-65 score was demonstrated, with a mortality rate of 2.1% for patients with a CURB-65 score of 1 vs 7.7% for those with a CURB-65 score of 2. This relationship was similar to that found in the international cohort (Fig 2).

Figure 2.

CURB-65 score vs 30-day mortality. CURB-65 = confusion, uremia, respiratory rate, BP, age > 65 years.

Effect plots²⁶ for the CURB-65 elements are shown in Figure 3. CURB-65 thresholds are represented by a vertical line. The effect plots for age, BUN, and respiratory rate suggest polytonic relationships with 30-day mortality; few patients had extreme values and the confidence intervals at these points are wide.

Figure 3.

Effect plots of CURB-65 variables vs event probability of 30-day mortality. Dashed lines represent CIs. Vertical lines represent threshold for CURB-65 severity score. BUN given in mg/dL, RR in breaths/min, BPs in mm Hg, age in years. DBP = diastolic BP; RR = respiratory rate; SBP = systolic BP. See Figure 2 legend for expansion of other abbreviation.

Figure 4A shows the performance of the dichotomous CURB-65 (AUC = 0.82) compared with the penalized logistic regression model using continuous, spline transformed values (AUC = 0.87) for the LDS Hospital data (P < .001) The AUC is known to be overestimated using the same data to train and test the model. The out-of-bag AUC estimate for the logistic regression model was 0.86 (95% CI, 0.80-0.91).

Figure 4.

A, Receiver operator curves for traditional vs electronic CURB-65 models in the US cohort. B, Receiver operator curves for traditional vs electronic CURB-65 models in the international cohort. AUC = area under the curve.

Figure 4B shows the performance of the CURB-65 score (AUC = 0.80) compared with the model derived from the LDS Hospital data and applied to the international data (AUC = 0.85). This shows a statistically significant improvement in predictive performance (P = .01). Our derived electronic CURB-65 model is available in e-Tables 1-4 for application.

Discussion

Our study demonstrates that the traditional CURB-65 pneumonia severity score reliably predicts 30-day all-cause mortality in a large population of ED patients with pneumonia in a North American community teaching hospital, comparable to results obtained abroad. This validation was achieved using data elements routinely entered in the electronic medical record. We were able to improve this predictive power by preserving the continuous nature of BUN, respiratory rate, BP, and age. We have validated this approach in an independent, prospectively identified, and more severely ill population in an international pneumonia database.¹¹

This more accurate pneumonia severity prediction model (hereafter referred to as eCURB) could be implemented as a real-time, electronic decision support tool. In contrast to the PSI, no predictors beyond those routinely present in the electronic record are required. The eCURB calculates a point estimate of 30-day mortality, instead of lumping mortality risk into dichotomous categories. Patients whose data are near the dichotomous cutpoints may be more accurately assessed with eCURB than CURB-65. For example, a 62-year-old man who is oriented × 3 with a BUN of 44 mg/dL, a BP of 95/65 mm Hg, and a respiratory rate of 28/min, although scoring 1 by the CURB-65 (mortality rate 2%), would generate a 30-day mortality estimate of 14% by the eCURB. The eCURB method is contained in e-Tables 1-4, which allows the user to enter values to estimate 30-day mortality. We believe that eCURB will be most useful as a method of severity assessment for a personal data assistant or automated as part of a real-time electronic medical record. The methodology is simple enough to be widely used in different electronic platforms, not confined to EDs with fully electronic medical records. A more accurate, fully automated severity assessment tool is likely to be more useful to clinicians than less accurate tools that require hand calculation.²⁷ The eCURB has the additional potential to be adapted to risk stratifications and admission thresholds specific to different populations.

Additional variables found to influence mortality, such as the ratio of oxygen saturation by pulse oximetry to Fio₂,⁴ could also be added to further improve mortality prediction. However, we chose to focus on the five CURB-65 variables because of the established nature of this score and because the international validation database does not consistently contain oxygenation data.

Electronic severity prediction could be integrated with other guideline elements in a clinical decision support tool available at the point of care. Although the pneumonia guideline implementation in our health-care system has been more successful than most,²⁸ provider use of severity assessment as a decision support tool has lagged behind use of other guideline elements. Automatic calculation of individualized risk prediction as part of a comprehensive decision support tool that can be adapted to local needs could be an important strategy for local implementation of national pneumonia treatment guidelines, as recommended by the British Thoracic Society, the Infectious Diseases Society of America, and the American Thoracic Society.^4,5 The ultimate clinical decision support tool would integrate severity assessment and triage recommendations with additional management guidelines, such as antibiotic selection and duration.

Beyond decision support, the eCURB could also be used to improve severity adjustment methods when comparing quality of care between hospitals. The US Center for Medicare Service’s Hospital Compare (www.hospitalcompare.hhs.gov) assesses outcomes and performance measures among hospitals, making results available to the public for common medical conditions, including CAP. However, analysis has shown only a modest relationship between adherence to performance measures and outcome, suggesting that attention should be focused on identifying measures more tightly linked to outcome.²⁹ Hospital Compare makes risk adjustments using past medical history and comorbidities, but it does not adjust for severity of illness upon presentation. A more accurate severity-adjusted mortality estimate might improve comparisons of mortality in the absence of reliable performance measures. Calculating eCURB would require few additional resources compared with the current method of performance measurement.

Our ED database is distinct from many large pneumonia study populations.^9,11 It includes not only hospital inpatients but also patients discharged home from the ED and managed as outpatients, thereby providing representation of patients with less severe disease. The cohort of patients used for validation of our prediction model contained only patients admitted to the hospital. Although both CURB-65 score and the eCURB had lower predictive performance in the international data, the relative superiority of the eCURB was similar in both cohorts.

Our study has limitations. It is retrospective and relies upon ICD-9 coding rather than prospective screening. In The Netherlands and Sweden, this methodology missed some pneumonias, particularly in patients with prolonged hospital admissions.^30,31 In the United States and particularly in our hospital, however, ICD-9 codes have demonstrated reasonable sensitivity for diagnosis of CAP.^32,33 The specificity of our technique was augmented by manual review of chest radiology reports. In addition, our use of real-time electronic charting for recording data significantly reduces the missing values generally associated with retrospective analyses. Validation of eCURB in the international database is also reassuring, as these patients were prospectively identified using different methodology. Our use of both Intermountain Healthcare and the Utah Population Databases to track statewide mortality further strengthens our analysis, although some out-of-state deaths may have escaped detection.

Calculation of CURB-65 using continuous and weighted variables is a logical extension of prior work, easily accomplished by a computer or personal digital assistant. Future research is directed toward further improving our model by integrating additional variables found to influence mortality, investigating performance of eCURB in predicting in-hospital mortality, length of stay, and the need for hospital admission. In addition, the improved severity estimate should be integrated into a real-time electronic pneumonia decision support tool, followed by implementation at the point of care. The ultimate test of electronic admission decision support will be to evaluate its influence on admission rate, mortality, length of stay, and cost.

Abbreviations

AUC

area under the curve

CAP

community-acquired pneumonia

CURB-65

confusion, uremia, respiratory rate, BP, age > 65 years

ICD-9

International Statistical Classification of Diseases, 9th edition

LDS

Latter-day Saints

PSI

pneumonia severity index