Derivation and Validation of a Prognostic Model for Pulmonary Embolism

Abstract

Rationale: An objective and simple prognostic model for patients with pulmonary embolism could be helpful in guiding initial intensity of treatment.

Objectives: To develop a clinical prediction rule that accurately classifies patients with pulmonary embolism into categories of increasing risk of mortality and other adverse medical outcomes.

Methods: We randomly allocated 15,531 inpatient discharges with pulmonary embolism from 186 Pennsylvania hospitals to derivation (67%) and internal validation (33%) samples. We derived our prediction rule using logistic regression with 30-day mortality as the primary outcome, and patient demographic and clinical data routinely available at presentation as potential predictor variables. We externally validated the rule in 221 inpatients with pulmonary embolism from Switzerland and France.

Measurements: We compared mortality and nonfatal adverse medical outcomes across the derivation and two validation samples.

Main Results: The prediction rule is based on 11 simple patient characteristics that were independently associated with mortality and stratifies patients with pulmonary embolism into five severity classes, with 30-day mortality rates of 0–1.6% in class I, 1.7–3.5% in class II, 3.2–7.1% in class III, 4.0–11.4% in class IV, and 10.0–24.5% in class V across the derivation and validation samples. Inpatient death and nonfatal complications were ⩽ 1.1% among patients in class I and ⩽ 1.9% among patients in class II.

Conclusions: Our rule accurately classifies patients with pulmonary embolism into classes of increasing risk of mortality and other adverse medical outcomes. Further validation of the rule is important before its implementation as a decision aid to guide the initial management of patients with pulmonary embolism.

Acute pulmonary embolism (PE) is a major health problem, with an estimated incidence of 23 to 69 cases per 100,000 persons annually (1, 2). Data from the National Hospital Discharge Survey show that in 2002, 101,000 patients with a primary diagnosis of PE were hospitalized in acute care hospitals in the United States, resulting in 676,700 inpatient days (3). The short-term mortality of this illness varies widely, ranging from less than 2% in many patients with nonmassive PE to more than 95% in patients who experience cardiorespiratory arrest (4–6).

Despite this broad variability in short-term mortality, models for the risk stratification during admission for PE are not well established. An accurate, objective, and simple clinical prediction rule may be helpful in guiding medical decision making. For example, patients estimated to be at low risk could be discharged early or managed entirely as outpatients using low-molecular-weight heparin (7), whereas patients estimated as high risk may benefit from a more intensive surveillance in an intensive care setting. Previous models of risk stratification for PE were limited by their reliance on arterial blood gas values at room air and leg vein ultrasound (8, 9) or by their ability to identify only low-risk patients with PE (10). We therefore sought to develop a practical clinical prediction rule for patients with PE that quantifies the risk of mortality and other adverse medical outcomes across the full spectrum of risk and that relies only on readily available clinical parameters.

METHODS

Patient Identification and Eligibility

We identified patients with PE from January 2000 to November 2002 using the Pennsylvania Health Care Cost Containment Council (PHC4) database (11). We included inpatients 18 years or older who were discharged with a primary International Classification of Diseases, 9th Clinical Modification (ICD-9-CM) diagnosis of PE or a secondary diagnosis for PE and one of the following primary diagnoses that represent complications or treatments of PE: respiratory failure, cardiogenic shock, cardiac arrest, secondary pulmonary hypertension, syncope, thrombolysis, and intubation/mechanical ventilation. A detailed description of the methods is available in the online supplement.

Baseline Predictor Variables and Outcome Measures

The baseline clinical variables used to derive our prediction rule were obtained by linking all eligible patients identified using PHC4 to the MediQual Atlas database (11). To derive our prediction rule, we used clinical variables routinely available at presentation that were previously shown to be associated with mortality in patients with PE or other acute diseases. These variables included demographics (12–15), comorbid conditions (8, 12–14, 16), physical examination findings (8, 14, 15, 17, 18), and laboratory and chest x-ray findings (8, 14, 16, 19–24).

The study outcome used to derive our prediction rule was death from all causes within 30 days of hospitalization based on mortality data from the National Death Index (25). We also assessed whether patients developed severe nonfatal outcomes (cardiogenic shock/cardiorespiratory arrest) during hospitalization.

Derivation and Internal Validation of the Prediction Rule

The study cohort comprised 15,531 discharged patients with PE treated at 186 hospitals. We randomly selected 10,354 discharged patients (67%) for the derivation and 5,177 (33%) for the internal validation sample. We derived our prediction rule using stepwise logistic regression, with 30-day mortality as the outcome, and the demographic and clinical variables previously described as predictors. In a first step, we constructed our logistic regression model excluding laboratory variables. To quantify the impact of including laboratory tests on model performance, we also estimated a model that included baseline laboratory tests. On the basis of the β-coefficients of the model, we generated a point score that divides patients into five risk classes for 30-day mortality (class I, very low risk; class II, low risk; class III, intermediate risk; class IV, high risk; and class V, very high risk).

External Validation of the Prediction Rule

We validated our rule in an independent patient population using data from 221 inpatients prospectively diagnosed with PE, using spiral computed tomography, at three emergency departments at two university hospitals in Switzerland and one in France (26). Patients who had a contraindication to computed tomography or who were severely ill were not eligible for this study. Follow-up information about mortality, recurrent venous thromboembolism, and major bleeding was obtained by phone interviews of patients, family members, and/or primary care physicians, and hospital chart review.

Statistical Analyses

We compared risk class–specific mortality and rates of nonfatal adverse medical outcomes in the derivation sample to each validation sample using logistic regression with a robust variance estimator or exact χ² tests. To assess the discriminatory power of our rule to predict 30-day mortality, we also compared the area under the receiver operating characteristic curves of the prediction rule (27).

RESULTS

Patients in the external validation sample had a lower prevalence of most comorbid illnesses and fewer abnormal findings on physical examination compared with those in the derivation and internal validation samples. This reflects the exclusion of more severely ill patients from the external validation sample (Table 1). Thirty-day mortality rates in the derivation, internal, and external validation samples were 9.2, 9.5, and 2.7%, respectively.

TABLE 1.

Comparison of baseline patient characteristics in the derivation and validation samples

Patient Characteristics*	Derivation Sample, % (n = 10,354)	Internal Validation Sample, % (n = 5,177)	External Validation Sample, % (n = 221)
Demographic factors
Age > 65 yr	52.8	53.1	59.3
Male sex	39.6	41.1	45.2
Comorbid illnesses
Cancer	19.9	19.0	15.8
Heart failure	16.1	15.3	11.8
Chronic lung disease	18.2	19.1	8.6
Chronic renal disease	4.4	4.2	4.5
Cerebrovascular disease†	8.9	9.9	4.5
Clinical findings
Temperature < 36°C	16.7	16.4	3.6
Pulse ⩾ 110/min	29.2	30.0	14.0
Systolic blood pressure < 100 mm Hg	10.6	10.2	1.8
Respiratory rate ⩾ 30/min	14.5	14.7	10.9
Altered mental status‡	6.9	8.1	0
Arterial oxygen saturation < 90%§	8.0	7.8	5.9
Laboratory findings
Hemoglobin < 12 g/dl	30.4	29.9	Not available
White blood cell count < 4,000 or > 12,000/mm3	27.6	28.3	Not available
Platelets < 100,000/mm3	3.0	2.5	Not available
Sodium < 130 or > 150 mmol/L	2.4	2.5	Not available
Blood urea nitrogen ⩾ 11 mmol/L (30 mg/dl)	13.5	13.0	Not available
Creatinine > 177 μmol/L (2.0 mg/dl)	4.7	4.6	0.9
Arterial pH < 7.25	1.1	1.2	0.5
PaCO2 < 25 or > 55 mm Hg	2.9	2.9	2.3

^*In the derivation and internal validation sample, 1.9% of patients had unknown values for temperature; 1.7% for pulse; 1.4% for systolic blood pressure; 1.9% for respiratory rate; 64.6% for arterial oxygen saturation; 8.1% for hemoglobin; 8.4% for white blood cell count; 9.2% for platelet count; 11.6% for sodium, blood urea nitrogen, and creatinine; 62.0% for arterial pH; and 62.5% for Pa_CO2. Comorbid conditions were coded as present versus unknown. In the external validation sample, 4.5% of patients had unknown values for temperature, 3.2% for respiratory rate, 45.2% for oxygen saturation, 0.5% for creatinine, 59.7% for pH, and 1.4% for Pa_CO2. Chronic renal disease and cerebrovascular disease were coded as present versus unknown. For calculating the frequency of baseline patient characteristics, unknown values were assumed to be normal and were included in the denominator.

^†Defined as transient ischemic attack or stroke.

^‡Defined as disorientation, lethargy, stupor, or coma. Information about mental status was not recorded in the external validation sample. Because patients with cognitive impairment were excluded from the study, mental status was assumed to be normal in all patients in this sample.

^§With and without administration of supplemental oxygen.

Derivation of the Prediction Rule

The 11 patient factors independently associated with 30-day mortality included two demographic characteristics (age, male sex), three comorbid illnesses (cancer, heart failure, chronic lung disease), and six clinical findings (pulse ⩾ 110/minute, systolic blood pressure < 100 mm Hg, respiratory rate ⩾ 30/minute, temperature < 36°C, altered mental status, and oxygen saturation < 90 mm Hg; Table 2). The scoring system shown in Table 2 was used to quantify the magnitude of the association of each of these 11 factors with mortality. These associations changed only minimally if the 5.5% of discharged patients with recurrent PE during the study period or the 1.8% who were identified using primary ICD-9-CM codes for PE complications or treatments were excluded from the analysis. In the derivation sample, risk class–specific 30-day mortality was 1.1% in risk class I, 3.1% in risk class II, 6.5% in risk class III, 10.4% in risk class IV, and 24.5% in risk class V.

TABLE 2.

Independent predictors of 30-DAY mortality in the derivation sample and points assigned to the risk score

Predictors	β-Coefficients (95% CI)	Points Assigned
Demographic characteristics
Age, per yr	0.03 (0.02–0.03)	Age, in yr
Male sex	0.17 (0.02–0.32)	+10
Comorbid illnesses
Cancer	0.87 (0.71–1.03)	+30
Heart failure	0.31 (0.14–0.49)	+10
Chronic lung disease	0.30 (0.12–0.47)	+10
Clinical findings
Pulse ⩾ 110/min	0.60 (0.44–0.76)	+20
Systolic blood pressure < 100 mm Hg	0.86 (0.67–1.04)	+30
Respiratory rate ⩾ 30/min	0.41 (0.23–0.58)	+20
Temperature < 36°C	0.42 (0.25–0.59)	+20
Altered mental status*	1.50 (1.30–1.69)	+60
Arterial oxygen saturation < 90%†	0.58 (0.37–0.79)	+20

Definition of abbreviation: CI = confidence interval.

A total point score for a given patient is obtained by summing the patient's age in years and the points for each applicable characteristic. Points assignments correspond with the following risk classes: ⩽ 65 class I, very low risk; 66–85 class II, low risk; 86–105 class III, intermediate risk; 106–125 class IV, high risk; > 125 class V, very high risk.

^*Defined as disorientation, lethargy, stupor, or coma.

^†With and without the administration of supplemental oxygen.

When laboratory variables also were assessed as potential predictors in the logistic regression model, all of the demographic and clinical variables in the simpler model except heart failure remained independently associated with 30-day mortality. In addition, seven laboratory variables (hemoglobin < 12 g/dl, white blood cell count < 4,000 or > 12,000/mm³, platelets < 100,000/mm³, sodium < 130 or > 150 mmol/L, blood urea nitrogen ⩾ 11 mmol/L [30 mg/dl], arterial pH < 7.25, and Pa_CO2 < 25 or > 55 mm Hg) were independently associated with mortality. Although this more complex, 17-variable model had a higher discriminatory power than the 11-variable model without laboratory variables (area under the receiver operating characteristic curve, 0.82 vs. 0.78; p < 0.001), the risk class–specific 30-day mortality rates for the more complex model (0.8% in risk class I, 2.5% in class II, 4.3% in class III, 9.9% in class IV, and 27.1% in class V) were similar to those for the simpler model.

Validation of the Prediction Rule

The prediction rule classified similar proportions of patients in each of the five risk classes across the derivation and internal validation samples (Table 3). In the external validation sample, a higher proportion of patients was classified in risk classes I–III and a lower proportion was classified in risk classes IV and V, reflecting the exclusion of severely ill patients from this sample. Thirty-day mortality rates in each risk class were not significantly different between the derivation and the validation samples; in the internal validation sample, these rates ranged from 1.6% in the very-low-risk (class I) patients to 23.9% in the very-high-risk (class V) patients (Table 4). The 30-day mortality was lower overall in the external validation sample and ranged from 0% among patients in risk class I to 10.0% among patients in class V. The rule's discriminatory power for 30-day mortality was nearly identical in the derivation and internal and external validation samples, with an area under the receiver operating characteristic curve of 0.78, 0.77, and 0.79, respectively (Figure 1).

TABLE 3.

Risk class distributions in the derivation and validation samples

Risk Class	Derivation Sample, % (95% CI) (n = 10,354)	Internal Validation Sample, % (95% CI) (n = 5,177)	External Validation Sample, % (95% CI) (n = 221)
Class I, very low risk	19.4 (18.7–20.2)	19.6 (18.5–20.7)	24.4 (18.9–30.7)
Class II, low risk	21.5 (20.7–22.3)	21.2 (20.1–22.4)	27.1 (21.4–33.5)
Class III, intermediate risk	21.7 (20.9–22.5)	22.2 (21.0–23.3)	28.1 (22.2–34.5)
Class IV, high risk	16.4 (15.7–17.1)	15.8 (14.8–16.8)	11.3 (7.5–16.2)
Class V, very high risk	21.0 (20.3–21.8)	21.3 (20.2–22.4)	9.0 (5.6–13.6)

For definition of abbreviation, see Table 2.

TABLE 4.

Risk class–specific medical outcomes in the derivation and validation samples

Medical Outcomes	Derivation Sample, % (95% CI) (n = 10,354)	Internal Validation Sample, % (95% CI) (n = 5,177)	External Validation Sample, % (95% CI) (n = 221)	p Value§	p Value‖
30-day mortality
Class I	1.1 (0.7–1.7)	1.6 (0.9–2.6)	0 (0–6.6)	0.32	0.66
Class II	3.1 (2.5–4.0)	3.5 (2.5–4.7)	1.7 (0–8.9)	0.63	0.72
Class III	6.5 (5.5–7.6)	7.1 (5.7–8.7)	3.2 (0.4–11.2)	0.51	0.43
Class IV	10.4 (9.0–11.9)	11.4 (9.3–13.8)	4.0 (0.1–20.4)	0.44	0.36
Class V	24.5 (22.7–26.4)	23.9 (21.4–26.5)	10.0 (1.2–31.7)	0.69	0.19
Inpatient mortality*
Class I	0.8 (0.5–1.3)	1.1 (0.5–1.9)	0 (0–6.6)	0.42	1.00
Class II	1.8 (1.3–2.4)	1.9 (1.2–2.9)	Not available‡	0.75	—
Class III	4.2 (3.4–5.1)	4.7 (3.6–6.1)	Not available‡	0.49	—
Class IV	5.9 (4.8–7.1)	7.0 (5.3–9.0)	Not available‡	0.31	—
Class V	15.8 (14.3–17.4)	17.2 (15.1–19.6)	Not available‡	0.29	—
Nonfatal cardiogenic shock or     cardiorespiratory arrest†
Class I	0.6 (0.3–1.0)	1.0 (0.5–1.8)	Not available‡	0.24	—
Class II	1.3 (0.9–1.9)	1.2 (0.6–2.0)	Not available‡	0.77	—
Class III	2.1 (1.6–2.8)	2.0 (1.3–3.0)	Not available‡	0.86	—
Class IV	1.9 (1.3–2.7)	2.1 (1.2–3.3)	Not available‡	0.82	—
Class V	4.6 (3.8–5.6)	5.3 (4.0–6.8)	Not available‡	0.40	—

For definition of abbreviation, see Table 2.

^*Patients who remained in the hospital for > 30 days were censored at 30 days.

^†During the initial hospital stay only.

^‡Inpatient complications such as death, cardiogenic shock, and cardiorespiratory arrest were not explicitly recorded in the external validation sample.

^§For comparisons of outcomes in the derivation and internal validation samples.

^‖For comparisons of outcomes in the derivation and external validation samples.

Figure 1.

Receiver operating characteristic curves for 30-day mortality in the derivation and validation samples. The area under the receiver operating characteristic curves were 0.78 (95% confidence interval [CI], 0.77–0.80) in the derivation sample, 0.77 (95% CI, 0.75–0.79) in the internal validation sample, and 0.79 (95% CI, 0.65–0.93) in the external validation sample (derivation vs. internal validation sample, p = 0.35; derivation vs. external validation sample, p = 0.93). Solid line = derivation; dashed line = internal validation; dotted line = external validation.

Inpatient mortality rates in risk class I (where comparable data were available) were similar across the three study samples (Table 4). The rates of nonfatal cardiogenic shock or cardiorespiratory arrest in each risk class in the derivation and internal validation samples also were similar, and ranged from 0.6 to 1.0% in class I to 4.6 to 5.3% in class V.

In the external validation sample, no patients in class I or II had nonfatal recurrent venous thromboembolism or major bleeding at 30 days after presentation. Overall, six patients died during follow-up in the external validation sample (three died of PE, one died of possible PE, and one from major bleeding). The rates of nonfatal recurrent venous thromboembolism were 1.6% in class III, 0% in class IV, and 0% in class V. The rates of nonfatal major bleeding were 1.6% in class III, 8.0% in class IV, and 0% in class V.

DISCUSSION

In this study to develop a clinical prediction rule for prognosis of PE, we identified 11 clinical findings from the history and physical examination that classify patients into five risk classes of increasing risk of death and other adverse medical outcomes. When validated in a retrospectively identified internal validation sample and a prospectively identified external validation sample, the performance of the rule was highly reliable. In all three study samples, mortality increased in a stepwise fashion with increasing risk class, and no significant differences in risk class specific mortality were observed across risk classes I–V.

Our rule accurately identifies patients who are at low risk of fatal and nonfatal medical outcomes: class I and class II patients had a 30-day mortality of 1.6% or less and 3.5% or less, respectively. Nonfatal cardiogenic shock or cardiorespiratory arrest occurred in 1.0% or less of patients in class I and 1.3% or less in class II, and no patient in these two risk classes had nonfatal major bleeding or recurrent venous thromboembolism. Recent evidence suggests that many patients with nonmassive PE can be safely treated entirely as outpatients using low-molecular-weight heparins or discharged early (28–32). On the basis of this evidence, the British Thoracic Society recommends outpatient treatment for clinically stable patients with PE (7). Thus, our rule provides clinicians with an explicit tool for identifying very-low-risk (class I) and low-risk (class II) patients with PE who may be potential candidates for outpatient treatment or early hospital discharge. If applied on a health system or national level, outpatient treatment or early discharge of only a small proportion of patients with PE is likely to result in substantial cost savings (33). However, it is important to note that our rule is intended to supplement, not replace, clinical judgment. The initial site of treatment decision for patients with PE must also consider psychosocial contraindications to outpatient care (e.g., frailty, lack of treatment adherence due to psychiatric or substance abuse problems) or the availability of outpatient systems of health care. Likewise, physicians would be unlikely to discharge a previously healthy 40-year-old woman who has severe hypoxemia and no additional pertinent prognostic factors, even if she was classified as very low risk (class I) by the rule.

Our rule also accurately identifies patients who are at higher risk of short-term death and other adverse medical outcomes. Patients in class V had 30-day mortality rates of up to 24.5% and rates of nonfatal cardiogenic shock or cardiorespiratory arrest of up to 5.3%. Whether these high-risk patients with PE could potentially benefit from more intensive forms of surveillance and care (e.g., in an intensive care unit setting) remains to be shown.

Our prediction rule has several distinctive strengths compared with a prior model of prognosis after PE (8, 9). First, it consists of clearly defined, routinely available predictors and does not require any laboratory tests or radiographic procedures not routinely performed in the management of PE. Second, the accuracy and generalizability of the rule are supported by its derivation (patients in the United States only) and validation in 15,752 patients from 189 hospitals in the United States, Switzerland, and France. Third, our study samples represent a broad disease spectrum, ranging from nonmassive PE to PE with cardiorespiratory arrest.

Our study also has potential limitations. First, patients in our derivation and internal validation samples were identified using ICD-9-CM codes for PE rather than standardized radiographic criteria, and therefore patient eligibility may be subject to study selection biases due to hospital coding procedures. However, two prior studies demonstrated that up to 96% of patients with specific ICD-9-CM codes for PE had objectively documented disease on the basis of chart review criteria (34–36). Furthermore, we cannot rule out the possibility that patients who were identified using a secondary ICD-9-CM code for PE with a primary diagnosis of possible PE complications (e.g., cardiogenic shock) actually developed PE as a consequence of one of these other conditions. Yet, the performance of our rule did not change when these patients were excluded from our analyses. Second, the study used to externally validate our rule was not originally designed for this task, and information about mental status was not explicitly recorded (26). Although it is very unlikely that more than a few patients had an altered mental status in the external validation sample, we cannot exclude the possibility that disease severity may have been underestimated in these patients. The external validation sample also excluded more severely ill patients (e.g., those who were hemodynamically unstable). As a result, there were fewer higher risk patients in this sample and the 95% confidence interval for risk class–specific mortality was relatively wide among patients in risk classes IV and V. Finally, we could not assess the occurrence of recurrent venous thromboembolism and major bleeding in the derivation and internal validation samples and cardiogenic shock or cardiorespiratory arrest in the external validation sample because these complications were not reliably documented in our databases.

In conclusion, we derived and validated a practical bedside tool for risk stratification that accurately classifies patients with PE at increasing risk of death and other adverse outcomes. Outpatient management or early hospital discharge of patients with PE identified as very low risk (class I) and low risk (class II) has the potential to result in large cost savings without added risk to patients. However, before this rule can be implemented into clinical practice, its clinical usefulness should be tested in a prospective study.