Abstract
Background
Treatments for non–ST-segment elevation myocardial infarction (NSTEMI) reduce ischemic events but increase bleeding. Baseline prediction of bleeding risk can complement ischemic risk prediction for optimizing NSTEMI care; however, existing models are not well suited for this purpose.
Methods and Results
We developed (n=71,277) and validated (n=17,857) a model that identifies 8 independent baseline predictors of in-hospital major bleeding among community-treated NSTEMI patients enrolled in the CRUSADE Quality Improvement Initiative. Model performance was tested by c statistics in the derivation and validation cohorts and according to post-admission treatment (i.e., invasive and antithrombotic therapy). The CRUSADE bleeding score (range 1–100 points) was created by assigning weighted integers corresponding to the coefficient of each variable. The rate of major bleeding increased by bleeding risk score quintiles: 3.1% very low risk (≤20); 5.5% low risk (21–30); 8.6% moderate risk (31–40); 11.9% high risk (41–50); and 19.5% very high risk (>50) (Ptrend<0.001). The c statistics for the major bleeding model (derivation=0.72 and validation=0.71) and risk score (derivation=0.71 and validation=0.70) were similar. The c statistics for the model among treatment subgroups were: ≥2 antithrombotics=0.72; <2 antithrombotics=0.73; invasive approach=0.73; conservative approach=0.68.
Conclusion
The CRUSADE bleeding score quantifies risk for in-hospital major bleeding across all post-admission treatments, enhancing baseline risk assessment for NSTEMI care.
Keywords: non-ST-segment elevation myocardial infarction, bleeding, risk assessment
Treatment of non–ST-segment elevation myocardial infarction (NSTEMI) has traditionally focused on preventing or minimizing ischemic complications with potent antithrombotic medications and catheter-based interventions.1–3 Yet these reductions in recurrent ischemic events have come at the cost of increased major bleeding,4–7 which is itself associated with worse clinical outcomes.7–13 Bleeding complications have received attention recently, in part because newer antithrombotic agents for NSTEMI have unique ischemia and bleeding profiles. Some agents demonstrate low rates of major bleeding with similar efficacy,5,14 while others demonstrate higher rates of major bleeding with superior efficacy.15 Given the importance of safety and efficacy,12 the recent American College of Cardiology (ACC)/American Heart Association (AHA) practice guidelines placed renewed emphasis on risk stratification to guide treatment for NSTEMI.3 While tools for ischemic risk stratification are well described (i.e., TIMI, PURSUIT, and GRACE risk scores),16–18 bleeding risk stratification is more limited. The few bleeding risk stratification models in existence include treatments known to influence bleeding or are derived from subgroups or trial populations not representative of those at greatest risk.10,13,19 Consequently, better estimation of baseline risk of bleeding in NSTEMI patients is needed to facilitate optimal treatment selection.
Using data from the Can Rapid risk stratification of Unstable angina patients Suppress ADverse outcomes with Early implementation of the ACC/AHA guidelines (CRUSADE) Quality Improvement Initiative, we developed and validated a scoring system to estimate baseline risk of in-hospital major bleeding in patients with NSTEMI. The CRUSADE bleeding score provides a tool that equips clinicians with the means to consider safety outcomes when making treatment decisions for patients with NSTEMI.
Methods
The CRUSADE Quality Improvement Initiative is a database of high-risk NSTE acute coronary syndromes (ACS) patients admitted to U.S. hospitals from November 2001 through December 2006.20 CRUSADE inclusion and exclusion criteria, data collection, and variables have been previously described.21 Data on baseline and nadir hematocrit (HCT) values were added to version 2 of the case report form, so the analysis in this study was limited to patients enrolled from February 15, 2003, through December 31, 2006. The institutional review board of each center approved participation in CRUSADE. As data were collected anonymously, informed consent was not required.
Population
The analysis population consisted of 89,134 patients enrolled across 485 U.S. sites. Starting from the CRUSADE population which had recorded HCT values (N=118,252), patients with unstable angina (N=7173) and those taking warfarin at home (N=7752) were excluded due to potential differences in treatment patterns influencing bleeding risk. Patients transferred out of the CRUSADE hospital (N=12,000) were also excluded, because treatments and outcomes after transfer could not be collected due to current U.S. privacy regulations. Patients with improperly recorded baseline HCT (N=739) or missing major bleeding data (N=143) were excluded. Additionally, patients who died within 48 hours of hospital arrival (N=1311) were excluded as they represent a censored population who have a truncated opportunity for both treatment and major bleeding events. Patients who underwent coronary artery bypass grafting (CABG) were included up until their procedures and were censored thereafter. The study population was then divided by using simple random sampling into a derivation cohort (80%, n=71,277) and a validation cohort (20%, n=17,857) for model development. Patients with missing variables for age, sex, and race were excluded from the model development process (derivation N=1545 and validation N=375).
Data Definitions
Baseline and nadir (lowest recorded) HCT were abstracted on the data collection form. Blood transfusion was defined as any nonautologous transfusion of whole or packed red blood cells (RBC). Witnessed bleeding was a variable on the case report form requiring evidence of a bleeding location. CRUSADE major bleeding was defined as intracranial hemorrhage, documented retroperitoneal bleed, HCT drop ≥12% (baseline to nadir), any RBC transfusion when baseline HCT ≥28%, or any RBC transfusion when baseline HCT <28% with witnessed bleed. The HCT cut-point of 28% was to eliminate transfusions given for baseline anemia from being considered as bleeding events. As the primary goal of our analysis was to identify baseline risk of bleeding, bleeding in CABG patients was included in the analysis only if it occurred prior to surgery. Bleeding during or after surgery was not considered. Creatinine clearance (mL/min) was estimated using the Cockcroft-Gault equation.22 Congestive heart failure (CHF) was defined as signs of CHF at presentation indicated by exertional dyspnea, orthopnea, shortness of breath, labored breathing, fatigue at either rest or with exertion, rales >1/3 of the lung fields, elevated jugular venous pressure, S3 gallop, or pulmonary congestion on x-ray believed to represent cardiac dysfunction. Prior vascular disease was defined as either prior stroke or peripheral arterial disease.
Statistical Analysis
The relationship between potential covariates and major bleeding was explored. Continuous variables (such as age, weight, baseline HCT, creatinine clearance, heart rate, and systolic blood pressure) were investigated for nonlinearity, and plots of each continuous variable versus rates of major bleeding were reviewed in order to create dichotomous cut-points when suitable. Systolic blood pressure cut-point values of <110 mm Hg or >180 mm Hg were chosen because the relationship between bleeding and systolic blood pressure increased linearly past these ranges but was flat in between. Similarly, a cut-point HCT of 36% was chosen because major bleeding only increased below this value. In addition, heart rate values ≤70 bpm were set to 70 bpm and creatinine clearance values ≥120 mL/min were set to 120 mL/min, because the relationship between heart rate and creatinine clearance with major bleeding was flat beyond those values.
Variables with clinically and statistically significant univariate relationships with major bleeding were included in the multivariate model. The degree of missing data was approximately 2% across covariates. Missing values were set to the lower risk group for discrete variables and replaced with gender-specific medians for continuous variables. To investigate the sensitivity of missing data imputation, two sensitivity analyses were performed where the first analysis excluded all missing data of the covariates in the model (e.g., complete case analysis, N=63,117) and the second analysis imputed missing data of the discrete variables to the higher risk group. Because the c statistics of the sensitivity analyses were not remarkably different from the main analysis in which missing values were set to the lower risk group for discrete variables, only the main analysis is presented. Logistic generalized estimating equations (GEE) method was used to account for within-hospital clustering. This method produces estimates similar to those from ordinary logistic regression, but the estimated variances of the estimates are adjusted for the correlation of outcomes within a hospital.23 The predictive performance of the model was assessed with c statistics and observed versus predicted probabilities plots.
The CRUSADE bleeding score was developed by assigning a weighted integer to each independent predictor based on its coefficient in the final model. A point score for each patient is calculated by summing the weighted integers (range 1–100 points). The predicted rate of major bleeding was plotted as a continuous function of the score. The bleeding score was also divided into quintiles: very low risk (≤20; n=19,486), low risk (21–30; n=12,545), moderate risk (31–40; 11,530), high risk (41–50; n=10,961), and very high risk (>50; n=15,210). The performance of the CRUSADE bleeding score was tested in derivation and validation cohorts, and in relevant post-admission treatment subgroups: patients treated with ≥2 antithrombotic medications (antiplatelet [aspirin or clopidogrel], anticoagulant, or glycoprotein [GP] IIb/IIIa inhibitors; n=50,969); patients receiving <2 antithrombotic medications (n=5931); and, among patients receiving ≥2 antithrombotic medications, those who did not undergo cardiac catheterization (conservative strategy, n=3200) and those who underwent cardiac catheterization invasive strategy, n=43,492). In-hospital mortality was also determined for those who did and did not experience a major bleeding event in each risk group. In determining association between in-hospital outcomes (major bleeding and mortality) and bleeding risk score groups, bleeding risk group was entered as an ordinal independent variable in the logistic GEE models to test for a linear trend. All comparisons were two-tailed, and a P value <0.05 was considered statistically significant. All analyses were performed using SAS software (version 9.1, SAS Institute, Cary, NC).
The authors had full access to and take full responsibility for the integrity of the data. All authors have read and agree to the manuscript as written.
Results
Baseline characteristics and outcomes of the derivation and validation cohorts were similar (Table 1). CRUSADE patients had a median age of 67 years, were 60% male, and had a high prevalence of cardiovascular risk factors. There was a high prevalence of prior cardiovascular disease. The rate of major bleeding was 9.4% in the derivation cohort and 9.6% in the validation cohort (P=NS for cross-cohort comparisons). Among the patients with major bleeding, the (non-exclusive) occurrence of the individual components of the CRUSADE major bleeding definition were as follows: intracranial hemorrhage (0.7%), documented retroperitoneal bleed (1.9%), HCT drop ≥12% (baseline to nadir) (44.4%), any RBC transfusion when baseline HCT ≥28% (68.8%), or any RBC transfusion when baseline HCT <28% with witnessed bleed (2.9%). Patients who experienced a CRUSADE major bleed (n=6701) had higher rates of in-hospital HF (15.9% vs. 6.5%), cardiogenic shock (7.7% vs. 1.5%), and mortality (8.5% vs. 2.1%) (all P<0.0001) compared with those who did not.
Table 1.
Baseline Characteristics of Derivation and Validation Cohorts
| Variable | Derivation cohort (N = 71,277) | Validation cohort (N = 17,857) |
|---|---|---|
| Demographics | ||
| Age (years) | 67.0 (56.0, 79.0) | 67.0 (56.0, 79.0) |
| Weight (kg) | 81.2 (68.0, 95.3) | 81.1 (68.1, 95.3) |
| Male sex | 60.2% | 60.3% |
| White | 80.1% | 79.6% |
| African-American | 10.8% | 10.8% |
| Asian | 1.1% | 1.1% |
| Hispanic | 3.9% | 4.1% |
| Medical history | ||
| Family history of CAD | 33.9% | 33.9% |
| History of hypertension | 70.5% | 70.6% |
| Diabetes mellitus | 32.7% | 32.5% |
| Prior vascular disease* | 18.4% | 18.1% |
| Current/recent smoker | 28.4% | 27.8% |
| Hyperlipidemia | 52.0% | 51.7% |
| Prior myocardial infarction | 28.1% | 27.9% |
| Prior PCI | 21.0% | 20.5% |
| Prior CABG | 18.2% | 18.5% |
| Prior congestive heart failure | 16.2% | 16.1% |
| Signs and symptoms at presentation | ||
| Signs of congestive heart failure | 22.9% | 23.2% |
| Heart rate (bpm) | 83 (70, 98) | 83.0 (70, 98) |
| Systolic blood pressure (mm Hg) | 144 (124, 165) | 144 (124, 165) |
| Baseline HCT (%) | 40.7 (36.5, 44.2) | 40.7 (36.6, 44.1) |
| Creatinine clearance (mL/min)† | 70.3 (43.8, 101.9) | 70.8 (44.0, 102.0) |
| ECG: ST depression | 27.4% | 27.6% |
| In-hospital events | ||
| Death | 2.7% | 2.6% |
| Major bleeding | 9.4% | 9.6% |
Data are presented as median (25th, 75th percentile) for continuous variables and as percentage for categorical data.
Prior vascular disease defined as peripheral artery disease or prior stroke.
Creatinine clearance estimated by the Cockcroft-Gault formula.
CABG, coronary artery bypass graft; CAD, coronary artery disease; ECG, electrocardiogram; HCT, hematocrit; PCI, percutaneous coronary intervention.
Univariate Associations with Major Bleeding
CRUSADE major bleeding was associated with older age (median 74 vs. 67 years), lower weight (median 74.8 vs. 81.6 kg), higher heart rate (median 90 vs. 82 bpm), and lower systolic blood pressure (median 142 vs. 144 mm Hg) (all P<0.0001). Major bleeding was also significantly associated with lower baseline hematocrit and lower creatinine clearance (Table 2A). Tables 2A and 2B describe the incidence of major bleeding according to the presence of continuous (Table 2A) and dichotomous (Table 2B) risk factors used in the development of the bleeding model.
Table 2A and 2B.
Univariable Relationship between Continuous and Dichotomous Baseline Characteristics and In-hospital Major Bleeding in the Derivation Cohort
| 2A. Continuous variables | Major bleeding | Median (25th, 75th) | P value* |
|---|---|---|---|
| Age (years) | Yes | 74 (63, 82) | <0.0001 |
| No | 67 (55, 78) | ||
| Weight (kg) | Yes | 75 (64, 89) | <0.0001 |
| No | 82 (69, 96) | ||
| Hematocrit (baseline %) | Yes | 37 (32, 43) | <0.0001 |
| No | 41 (37, 44) | ||
| Heart rate (bpm) | Yes | 90 (75, 107) | <0.0001 |
| No | 82 (70, 98) | ||
| Systolic blood pressure (mm Hg) | Yes | 142 (118, 167) | <0.0001 |
| No | 144 (124, 165) | ||
| CrCl (mL/min) | Yes | 38 (25, 56) | <0.0001 |
| No | 56 (37, 78) |
| 2B. Dichotomous variables | Present | Major bleeding(% ) | Pvalue* |
|---|---|---|---|
| Sex | Males | 7.2 | <0.0001 |
| Females | 12.7 | ||
| Hypertension | No | 7.1 | <0.0001 |
| Yes | 10.4 | ||
| Diabetes mellitus | No | 8.1 | <0.0001 |
| Yes | 12.1 | ||
| Current/recent smoker | No | 10.2 | <0.0001 |
| Yes | 7.4 | ||
| Hyperlipidemia | No | 9.5 | NS |
| Yes | 9.3 | ||
| Prior vascular disease† | No | 8.4 | <0.0001 |
| Yes | 14.0 | ||
| Prior MI | No | 9.2 | 0.016 |
| Yes | 10.0 | ||
| Prior PCI | No | 9.5 | NS |
| Yes | 8.9 | ||
| Prior CABG | No | 9.3 | 0.017 |
| Yes | 9.9 | ||
| Prior CHF | No | 8.4 | <0.0001 |
| Yes | 14.4 | ||
| Signs of CHF | No | 7.7 | <0.0001 |
| Yes | 15.1 |
P value test (Cochran-Mantel-Haenszel statistics stratified by center).
Prior vascular disease is defined as history of peripheral arterial disease or prior stroke.
CABG, coronary artery bypass graft surgery; CHF, congestive heart failure; CrCl, creatinine clearance; MI, myocardial infarction; PCI, percutaneous coronary intervention.
CRUSADE Bleeding Model and Risk Score
From multivariable analysis, the factors independently associated with major bleeding included baseline HCT, estimated creatinine clearance, baseline heart rate and systolic blood pressure, female sex, signs of CHF on presentation, prior vascular disease, and diabetes mellitus (Table 3). Although a univariate predictor, age did not remain an independent predictor of major bleeding after adjusting for other covariates. The final regression model (the CRUSADE major bleeding model) discriminated patients who did and did not have a major bleeding event in both the derivation (c statistic=0.72) and validation (c statistic=0.71) cohorts.
Table 3.
Multivariate Predictors of In-hospital Major Bleeding
| Variable | χ2 | Derivation cohort OR, 95% CI | Validation cohort OR, 95% CI | ||
|---|---|---|---|---|---|
| Baseline HCT <36% (vs. ≥ 36%) | 434.6 | 2.28 | 2.11–2.46 | 2.17 | 1.92–2.44 |
| CrCl (per 10 mL/min decrease)* | 433.2 | 1.12 | 1.10–1.13 | 1.11 | 1.09–1.13 |
| Heart rate (per 10 bpm increase) | 159.2 | 1.08 | 1.07–1.10 | 1.09 | 1.07–1.12 |
| Female sex | 77.8 | 1.31 | 1.23–1.39 | 1.33 | 1.19–1.50 |
| Signs of CHF at presentation | 37.7 | 1.23 | 1.15–1.31 | 1.13 | 1.01–1.28 |
| Prior vascular disease† | 30.4 | 1.19 | 1.12–1.27 | 1.10 | 0.98–1.24 |
| Diabetes mellitus | 26.6 | 1.16 | 1.10–1.23 | 1.25 | 1.12–1.40 |
| SBP ≤110 mm Hg (vs. 110–180 mm Hg) | 12.6 | 1.26 | 1.16–1.36 | 1.27 | 1.10–1.47 |
| SBP ≥180 mm Hg (vs. 110–180 mm Hg) | 1.24 | 1.14–1.35 | 1.18 | 1.02–1.37 | |
| c Statistic | 0.72 | 0.71 | |||
Creatinine clearance estimated by the Cockcroft-Gault formula.
Prior vascular disease defined as history of peripheral artery disease or prior stroke.
CHF, congestive heart failure; CrCl, creatinine clearance; HCT, hematocrit; SBP; systolic blood pressure.
The CRUSADE bleeding score (Table 4) was derived by assigning weighted integers to each independent predictor based on its coefficient in the regression model. The sum of the weighted integers (range 1–100 points) estimates the risk of in-hospital major bleeding (Figure 1). Figure 1 demonstrates the curvilinear relationship between CRUSADE bleeding score and predicted probabilities of major bleeding observed in the derivation cohort, where the rate of bleeding increases tenfold (<3% to >30%) from lowest to highest scores. Similar to the multivariable model, the CRUSADE bleeding score had good ability to discriminate between patients who did and did not have a major bleeding event in the derivation (c statistic=0.71) and validation cohorts (c statistic=0.70). The CRUSADE bleeding model was similarly able to predict rates of moderate to severe bleeding according to the GUSTO definition (c index 0.713, data not shown).
Table 4.
Algorithm Used To Determine the Risk Score of CRUSADE In-hospital Major Bleeding
| Predictor | Range | Score |
|---|---|---|
| Baseline HCT (%) | <31 | 9 |
| 31–33.9 | 7 | |
| 34–36.9 | 3 | |
| 37–39.9 | 2 | |
| ≥40 | 0 | |
| Creatinine clearance* (mL/min) | ≤15 | 39 |
| >15–30 | 35 | |
| >30–60 | 28 | |
| >60–90 | 17 | |
| >90–120 | 7 | |
| >120 | 0 | |
| Heart rate (bpm) | ≤70 | 0 |
| 71–80 | 1 | |
| 81–90 | 3 | |
| 91–100 | 6 | |
| 101–110 | 8 | |
| 111–120 | 10 | |
| ≥121 | 11 | |
| Sex | Male | 0 |
| Female | 8 | |
| Signs of CHF at presentation | No | 0 |
| Yes | 7 | |
| Prior vascular disease† | No | 0 |
| Yes | 6 | |
| Diabetes mellitus | No | 0 |
| Yes | 6 | |
| Systolic blood pressure (mm Hg) | ≤90 | 10 |
| 91–100 | 8 | |
| 101–120 | 5 | |
| 121–180 | 1 | |
| 181–200 | 3 | |
| ≥201 | 5 |
Creatinine clearance was estimated using the Cockcroft-Gault formula.
Prior vascular disease was defined as history of peripheral artery disease or prior stroke.
CHF, congestive heart failure; HCT, hematocrit.
Figure 1.
Predicted probability of in-hospital major bleeding across the spectrum of CRUSADE bleeding score in the derivation cohort.
Figure 2 compares the rates of in-hospital major bleeding across quintiles of risk according to CRUSADE bleeding score in the derivation and validation cohorts. In the derivation cohort, the rates of major in-hospital bleeding across the quintiles of risk groups were 3.1% (very low risk), 5.5% (low risk), 8.6% (moderate risk), 11.9% (high risk), and 19.5% (very high risk). The rate of major bleeding also increased across quintiles of risk groups in the validation cohort (Ptrend<0.001, Figure 2).
Figure 2.
Rate of major bleeding across CRUSADE bleeding score risk groups in the derivation and validation cohorts. Very low (bleeding score ≤20): derivation N=19,486 and validation N=4920; low (bleeding score 21–30): derivation N=12,545 and validation N=3141; moderate (bleeding score 31–40): derivation N=11,530 and validation N=2873; high (bleeding score 41–50): derivation and validation N=2787; and very high (bleeding score >50): derivation and validation N=3761. Ptrend <0.001.
CRUSADE Bleeding Score in Treatment Subgroups
CRUSADE includes patients who underwent an initial invasive strategy with cardiac catheterization (n=52,048) and subsequent revascularization (n=38,209), as well as those managed medically (without catheterization, n= 6407). Treatments (i.e., invasive care or antithrombotics) that increase the risk of bleeding were intentionally omitted from the CRUSADE bleeding score; however, the performance of the CRUSADE bleeding score across treatment subgroups was confirmed by formal testing.
The model had preserved discrimination in groups receiving ≥2 antithrombotic medications and those receiving <2 antithrombotic medications (c statistics 0.72 and 0.73, respectively). Using the derivation cohort, the incidence of major bleeding was 8.2% among those who received ≥2 antithrombotic medications (n=50,969) versus 6.9% among those who received <2 antithrombotic medications (n=5931). The rate of major in-hospital bleeding was higher if given ≥2 antithrombotic medications as compared with <2 antithrombotic in every risk quintile: 3.1 vs. 1.9% (very low risk), 5.5 vs. 2.6% (low risk), 8.4 vs. 5.3% (moderate risk), 12.0 vs. 6.7% (high risk), and 19.9 vs. 13.5% (very high risk) (Ptrend<0.001 within each of the 2 strata) (Figure 3). However, the absolute difference in bleeding was greater in the high and very high risk groups.
Figure 3.
Rate of major bleeding among patients treated with <2 antithrombotics versus ≥2 antithrombotics across CRUSADE bleeding score in the derivation cohort. Quintiles defined as: very low (≤20), N=18,406; low (21–30), N=11,368; moderate (31–40), N=9871; high (41–50), N=8290; and very high (>50), N=8965. Ptrend<0.001 within each of the two strata.
Among patients receiving ≥2 antithrombotics, the c statistic of the model in those treated with a conservative approach (no catheterization) was 0.68, while the c statistic of the model in those treated with an invasive approach (catheterization) was 0.73. The rate of major in-hospital bleeding was higher if undergoing invasive approach as compared to the conservative approach in every risk quintile: 3.1 vs. 2.5% (very low risk), 5.6 vs. 3.2% (low risk), 8.6 vs. 6.4% (moderate risk), 13.4 vs. 6.4% (high risk), and 22.6 vs. 13.9% (very high risk) (Figure 4). Similarly, the absolute difference in major bleeding was magnified in the high and very high risk groups. In-hospital mortality rates increased along with the CRUSADE bleeding risk quintiles. The rate of in-hospital mortality is also shown for patients who did and did not have a bleeding event within each CRUSADE bleeding risk group. In each bleeding risk quintile, patients who experienced a major bleed had higher mortality than those who did not (Figure 5).
Figure 4.
Rate of major bleeding among patients treated with ≥2 antithrombotics undergoing an invasive approach (catheterization) versus a conservative approach (no catheterization) across CRUSADE bleeding score in the derivation cohort. Quintiles defined as: very low (≤20), N=16,974; low (21–30), N=10,067; moderate (31–40), N=8142; high (41–50), N=6105; and very high (>50), N=5404. Ptrend<0.001 within each of the two strata.
Figure 5.
In-hospital mortality among patients having a major bleed versus those without a major bleed across CRUSADE bleeding score quintiles in the derivation cohort. Within each risk quintile, the P value for difference between patients who had a bleed versus those who did not was (chi-square adjusting for hospital clustering) <0.0001.
Discussion
The CRUSADE bleeding score, which predicts baseline risk of in-hospital major bleeding, was developed and validated in >89,000 community-treated NSTEMI patients. It is unique in that it only considers admission variables including baseline characteristics, clinical presentation, and key laboratory data. The 8 variables in the final model were female sex, history of diabetes, prior vascular disease, heart rate, systolic blood pressure, signs of CHF, baseline HCT <36%, and creatinine clearance. Although post-admission treatments were not included in the model, the CRUSADE bleeding score demonstrated preserved discrimination across treatment subgroups. Therefore, it complements ischemic risk prediction, enabling clinicians to consider net clinical outcomes in patients with NSTEMI.
Bleeding is a common problem complicating treatment of NSTEMI, with important immediate and late clinical consequences. Clinical trials involving almost 48,000 patients with NSTEMI demonstrate that major bleeding is associated with a 5-fold increase in 30-day mortality.8,9 Observations from a randomized trial comparing antithrombotic agents suggest that a reduction in bleeding events translates into improved survival.14 Prevention of major bleeding may represent an achievable step in improving outcomes by balancing safety and efficacy in the treatment of NSTEMI.
Several studies have examined predictors of major bleeding or developed predictive instruments for the estimation of bleeding risk in this population.9,10,13,19 Moscucci et al. determined independent predictors of bleeding among 24,045 STEMI and NSTEMI patients in the GRACE registry. Similar to our results, they observed that female sex, renal insufficiency, and blood pressure were independent predictors of major bleeding. More recently, Spencer et al. also found that female sex, peripheral artery disease, heart rate, and renal insufficiency were among the predictors of major bleeding in the first 30 days after admission in GRACE.13 Only one other study has developed a risk stratification tool or bleeding score. Nikolsky et al.19 used 6002 patients enrolled in the REPLACE-2 trial to derive and 1056 patients enrolled in REPLACE-1 to validate a risk score to predict major bleeding for patients undergoing elective or urgent percutaneous coronary intervention (PCI) via the femoral approach. Similar to the CRUSADE bleeding score, Nikolsky et al. found female sex, baseline anemia, and lower creatinine clearance were independent predictors of bleeding.19 However, REPLACE-2 enrolled a highly selected population, all of whom underwent PCI by the femoral approach, limiting its generalizability. Furthermore, the predictive model from GRACE10,13 and the risk score from REPLACE-219 included treatment variables (e.g., invasive procedures and antithrombotics), limiting their utility for assessing bleeding risk at presentation. These studies, therefore, do not address baseline risk in a community population.
The CRUSADE bleeding score builds upon these studies in several ways. It was developed in a diversely treated community population including those undergoing initial invasive strategy and revascularization, and those conservatively managed without catheterization. It includes only baseline factors, including creatinine clearance, a more precise estimate of renal function than creatinine or a history of renal insufficiency. Age was a significant univariate predictor of bleeding; however, it did not remain significant in multivariable testing due to other variables such as creatinine clearance which may account for age-associated risk.24–26 However, female sex, diabetes, and signs of CHF continue to contribute unique information regarding bleeding risk. Importantly, the CRUSADE bleeding score has preserved discrimination regardless of treatment (e.g., antithrombotic medications or invasive care), thereby increasing its utility in clinical decision-making.
The effect of treatment strategy on the incidence of bleeding in the study population is evident (Figures 3 and 4) because multiple antithrombotic agents or an invasive approach increased the risk of bleeding in every CRUSADE bleeding score quintile. Furthermore, the gradient of bleeding risk related to treatment appears magnified at the high end of CRUSADE bleeding score. These findings imply that those at high risk may have reduced bleeding rates with careful treatment selection, yet the effect of such adjustments in treatment strategy on outcomes will require confirmation by prospective testing.
The CRUSADE bleeding score identifies baseline factors associated with an increased propensity for bleeding. Moreover, those who experience a bleeding event have higher in-hospital mortality across all quintiles of baseline risk. The mortality among those who experience a bleeding event is also higher within each quintile. By identifying patients at higher propensity for bleeding, NSTEMI care may be improved by prompting clinicians to make judicious treatment selections, carefully dose antithrombotic medications, and select invasive strategies to optimize patient-centered care.27,28 With a growing number of antithrombotic agents,5,14,15,29,30 appreciation of baseline bleeding provides an objective starting point either for treatment selection or strategy comparison. The CRUSADE bleeding score provides a complement to existing risk stratification.
Limitations
Several limitations of this analysis should be considered. Given the dependence on registry data for this analysis, we chose to limit our population to those with NSTEMI to limit false positives. When unstable angina patients were included (n=5462), the model c statistic did not change (c statistic=0.72). We conclude our model will predict bleeding events in the high-risk ACS population. Another possible limitation could be that some initial bleeding events were not included as patients who died within 48 hours of hospitalization were excluded from the analysis. However, a validation analysis including early deaths (n=1311) did not alter the c statistic of the model (c statistic=0.71). The rate of major bleeding is higher in CRUSADE than in other studies, due to the complex patient population or the major bleeding definition. The definition of in-hospital major bleeding used in the present study has been published previously31 and is an adaptation of existing major bleeding definitions as applicable to the CRUSADE data collection methods.5,32,33 CRUSADE collected only hematocrit levels (not hemoglobin). History of prior bleeding or bleeding diathesis, which are recognized predictors of in-hospital bleeding,13 were also not collected in CRUSADE. Patients taking warfarin at admission were excluded, so additive risk was not considered. Finally, the c statistic of the CRUSADE in-hospital major bleeding model at 0.72 in the derivation and 0.71 in the validation cohort is modest but nevertheless better than other bleeding models.10,19
Conclusion
The CRUSADE bleeding score combines 8 baseline factors that predict the propensity for major bleeding into a simple validated tool to assist with risk assessment and optimize care of patients with NSTEMI.
Acknowledgments
The authors would like to thank the doctors, nurses, and patients participating in the CRUSADE Quality Improvement Initiative.
CRUSADE is funded by the Schering-Plough Corporation. Bristol-Myers Squibb/Sanofi Pharmaceuticals Partnership provides additional funding support. Millennium Pharmaceuticals, Inc., also funded this work. This work was also supported in part by a grant from the National Institute on Aging (R01 AG025312-01A1, PI Peterson).
Funding Sources
The CRUSADE Registry is funded by Millennium Pharmaceuticals and Schering-Plough Corporation. Bristol-Myers Squibb/Sanofi Pharmaceuticals Partnership provides an unrestricted grant in support of the program. This work was also supported in part by a grant from the National Institute on Aging (R01 AG025312-01A1, PI Peterson) and the American Heart Association.
Investigators had full, direct access to the data and performed the actual analysis. None of the sponsors played a role in conducting the analysis or in interpreting or reporting the results.
Footnotes
Conflict of Interest Disclosures
The authors have the following conflicts of interest to disclose.
Sumeet Subherwal: None.
Richard G. Bach: Research support from AstraZeneca, Schering-Plough Research Institute, Eli Lilly/Daiichi-Sankyo and The Medicines Company; speaker for Bristol-Myers Squibb/Sanofi Pharmaceuticals Partnership.
Anita Y. Chen: None.
Brian F. Gage: None.
Sunil V. Rao: Speaker and clinical investigator for Johnson & Johnson; consultant for sanofi-aventis, The Medicines Company; principal investigator for Momenta Pharmaceuticals.
L. Kristin Newby: Consultant for Astra Zeneca, Atherogenics, Biosite, CV Therapeutics, Novartis, Proctor & Gamble, Roche Diagnostics; principal or co-investigator for BG Medicine, Medicure, Schering-Plough; expert reviewer for Adolor; CEC reviewer for Inverness Medical; board member for Society of Chest Pain Centers.
Tracy Y. Wang: Principal investigator for Bristol-Myers Squibb, sanofi-aventis, Schering-Plough.
W. Brian Gibler: Grants from EMCREG-International, Millennium, Schering-Plough, sanofi- aventis, and Bristol-Myers Squibb.
E. Magnus Ohman: Consultant for Abiomed, Datascope, Inovise, Liposcience, Response Biomedical, The Medicines Company; principal investigator for Bristol-Myers Squibb, Eli Lilly, Millennium, sanofi-aventis, Schering-Plough, Daiichi Sankyo, The Medicines Company; speakers bureau for CV Therapeutics, The Medicines Company; stockholder in Inovise.
Matthew T. Roe: Investigator for Daiichi-Sankyo, Eli Lilly, Portola Pharmaceuticals, KAI Pharmaceuticals, Schering-Plough, sanofi-aventis; consultant for KAI Pharmaceuticals, Schering-Plough; CEC for Genentech, Novartis; speakers bureau for Schering-Plough.
Charles V. Pollack, Jr.: Speaker for Schering-Plough, sanofi-aventis; research support from sanofi-aventis, GSK; consultant to Schering-Plough, BMS, sanofi-aventis, The Medicines Company.
Eric D. Peterson: Research grants from Bristol-Myers Squibb/Sanofi Pharmaceuticals Partnership, Merck/Schering, and Schering Plough Corp.
Karen P. Alexander: None.
References
- 1.Task Force for Diagnosis and Treatment of Non-ST-Segment Elevation Acute Coronary Syndromes of European Society of Cardiology. Bassand JP, Hamm CW, Ardissino D, Boersma E, Budaj A, Fernández-Avilés F, Fox KA, Hasdai D, Ohman EM, Wallentin L, Wijns W. Guidelines for the diagnosis and treatment of non–ST-segment elevation acute coronary syndromes: The Task Force for the Diagnosis and Treatment of Non–ST-Segment Elevation Acute Coronary Syndromes of the European Society of Cardiology. Eur Heart J. 2007;28:1598–1660. doi: 10.1093/eurheartj/ehm161. [DOI] [PubMed] [Google Scholar]
- 2.Cannon CP, Weintraub WS, Demopoulos LA, Vicari R, Frey MJ, Lakkis N, Neumann FJ, Robertson DH, DeLucca PT, DiBattiste PM, Gibson CM, Braunwald E. TACTICS (Treat Angina with Aggrastat and Determine Cost of Therapy with an Invasive or Conservative Strategy)--Thrombolysis in Myocardial Infarction 18 Investigators. Comparison of early invasive and conservative strategies in patients with unstable coronary syndromes treated with the glycoprotein IIb/IIIa inhibitor tirofiban. N Engl J Med. 2001;344:1879–1887. doi: 10.1056/NEJM200106213442501. [DOI] [PubMed] [Google Scholar]
- 3.Anderson JL, Adams CD, Antman EM, Bridges CR, Califf RM, Casey DE, Jr, Chavey WE, 2nd, Fesmire FM, Hochman JS, Levin TN, Lincoff AM, Peterson ED, Theroux P, Wenger NK, Wright RS, Smith SC, Jr, Jacobs AK, Adams CD, Anderson JL, Antman EM, Halperin JL, Hunt SA, Krumholz HM, Kushner FG, Lytle BW, Nishimura R, Ornato JP, Page RL, Riegel B, American College of Cardiology. American Heart Association Task Force on Practice Guidelines (Writing Committee to Revise the 2002 Guidelines for the Management of Patients With Unstable Angina/Non-ST-Elevation Myocardial Infarction) American College of Emergency Physicians. Society for Cardiovascular Angiography and Interventions. Society of Thoracic Surgeons. American Association of Cardiovascular and Pulmonary Rehabilitation. Society for Academic Emergency Medicine ACC/AHA 2007 guidelines for the management of patients with unstable angina/non–ST-elevation myocardial infarction: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Writing Committee to Revise the 2002 Guidelines for the Management of Patients With Unstable Angina/Non–ST-Elevation Myocardial Infarction) developed in collaboration with the American College of Emergency Physicians, the Society for Cardiovascular Angiography and Interventions, and the Society of Thoracic Surgeons Endorsed by the American Association of Cardiovascular and Pulmonary Rehabilitation and the Society for Academic Emergency Medicine. J Am Coll Cardiol. 2007;50:e1–157. doi: 10.1016/j.jacc.2007.02.013. [DOI] [PubMed] [Google Scholar]
- 4.Ferguson JJ, Califf RM, Antman EM, Cohen M, Grines CL, Goodman S, Kereiakes DJ, Langer A, Mahaffey KW, Nessel CC, Armstrong PW, Avezum A, Aylward P, Becker RC, Biasucci L, Borzak S, Col J, Frey MJ, Fry E, Gulba DC, Guneri S, Gurfinkel E, Harrington R, Hochman JS, Kleiman NS, Leon MB, Lopez-Sendon JL, Pepine CJ, Ruzyllo W, Steinhubl SR, Teirstein PS, Toro-Figueroa L, White H, SYNERGY Trial Investigators Enoxaparin vs unfractionated heparin in high-risk patients with non–ST-segment elevation acute coronary syndromes managed with an intended early invasive strategy: primary results of the SYNERGY randomized trial. JAMA. 2004;292:45–54. doi: 10.1001/jama.292.1.45. [DOI] [PubMed] [Google Scholar]
- 5.Stone GW, McLaurin BT, Cox DA, Bertrand ME, Lincoff AM, Moses JW, White HD, Pocock SJ, Ware JH, Feit F, Colombo A, Aylward PE, Cequier AR, Darius H, Desmet W, Ebrahimi R, Hamon M, Rasmussen LH, Rupprecht HJ, Hoekstra J, Mehran R, Ohman EM, ACUITY Investigators Bivalirudin for patients with acute coronary syndromes. N Engl J Med. 2006;355:2203–2216. doi: 10.1056/NEJMoa062437. [DOI] [PubMed] [Google Scholar]
- 6.Yusuf S, Zhao F, Mehta SR, Chrolavicius S, Tognoni G, Fox KK, Clopidogrel in Unstable Angina to Prevent Recurrent Events Trial Investigators Effects of clopidogrel in addition to aspirin in patients with acute coronary syndromes without ST-segment elevation. N Engl J Med. 2001;345:494–502. doi: 10.1056/NEJMoa010746. [DOI] [PubMed] [Google Scholar]
- 7.Lincoff AM, Bittl JA, Harrington RA, Feit F, Kleiman NS, Jackman JD, Sarembock IJ, Cohen DJ, Spriggs D, Ebrahimi R, Keren G, Carr J, Cohen EA, Betriu A, Desmet W, Kereiakes DJ, Rutsch W, Wilcox RG, de Feyter PJ, Vahanian A, Topol EJ, REPLACE-2 Investigators Bivalirudin and provisional glycoprotein IIb/IIIa blockade compared with heparin and planned glycoprotein IIb/IIIa blockade during percutaneous coronary intervention: REPLACE-2 randomized trial. JAMA. 2003;289:853–863. doi: 10.1001/jama.289.7.853. [DOI] [PubMed] [Google Scholar]
- 8.Eikelboom JW, Mehta SR, Anand SS, Xie C, Fox KA, Yusuf S. Adverse impact of bleeding on prognosis in patients with acute coronary syndromes. Circulation. 2006;114:774–782. doi: 10.1161/CIRCULATIONAHA.106.612812. [DOI] [PubMed] [Google Scholar]
- 9.Manoukian SV, Feit F, Mehran R, Voeltz MD, Ebrahimi R, Hamon M, Dangas GD, Lincoff AM, White HD, Moses JW, King SB, 3rd, Ohman EM, Stone GW. Impact of major bleeding on 30-day mortality and clinical outcomes in patients with acute coronary syndromes: an analysis from the ACUITY trial. J Am Coll Cardiol. 2007;49:1362–1368. doi: 10.1016/j.jacc.2007.02.027. [DOI] [PubMed] [Google Scholar]
- 10.Moscucci M, Fox KA, Cannon CP, Klein W, López-Sendón J, Montalescot G, White K, Goldberg RJ. Predictors of major bleeding in acute coronary syndromes: the Global Registry of Acute Coronary Events (GRACE) Eur Heart J. 2003;24:1815–1823. doi: 10.1016/s0195-668x(03)00485-8. [DOI] [PubMed] [Google Scholar]
- 11.Rao SV, O’Grady K, Pieper KS, Granger CB, Newby LK, Van de Werf F, Mahaffey KW, Califf RM, Harrington RA. Impact of bleeding severity on clinical outcomes among patients with acute coronary syndromes. Am J Cardiol. 2005;96:1200–1206. doi: 10.1016/j.amjcard.2005.06.056. [DOI] [PubMed] [Google Scholar]
- 12.Rao SV, Eikelboom JA, Granger CB, Harrington RA, Califf RM, Bassand J-P. Bleeding and blood transfusion issues in patients with non–ST-segment elevation acute coronary syndromes. Eur Heart J. 2007;28:1193–1204. doi: 10.1093/eurheartj/ehm019. [DOI] [PubMed] [Google Scholar]
- 13.Spencer FA, Moscucci M, Granger CB, Gore JM, Goldberg RJ, Steg PG, Goodman SG, Budaj A, FitzGerald G, Fox KA, GRACE Investigators Does comorbidity account for the excess mortality in patients with major bleeding in acute myocardial infarction? Circulation. 2007;116:2793–2801. doi: 10.1161/CIRCULATIONAHA.107.694273. [DOI] [PubMed] [Google Scholar]
- 14.Fifth Organization to Assess Strategies in Acute Ischemic Syndromes Investigators. Yusuf S, Mehta SR, Chrolavicius S, Afzal R, Pogue J, Granger CB, Budaj A, Peters RJ, Bassand JP, Wallentin L, Joyner C, Fox KA. Comparison of fondaparinux and enoxaparin in acute coronary syndromes. N Engl J Med. 2006;354:1464–1476. doi: 10.1056/NEJMoa055443. [DOI] [PubMed] [Google Scholar]
- 15.Wiviott SD, Braunwald E, McCabe CH, Montalescot G, Ruzyllo W, Gottlieb S, Neumann FJ, Ardissino D, De Servi S, Murphy SA, Riesmeyer J, Weerakkody G, Gibson CM, Antman EM, TRITON-TIMI 38 Investigators Prasugrel versus clopidogrel in patients with acute coronary syndromes. N Engl J Med. 2007;357:2001–2015. doi: 10.1056/NEJMoa0706482. [DOI] [PubMed] [Google Scholar]
- 16.Antman EM, Cohen M, Bernink PJ, McCabe CH, Horacek T, Papuchis G, Mautner B, Corbalan R, Radley D, Braunwald E. The TIMI risk score for unstable angina/non–ST-elevation MI: a method for prognostication and therapeutic decision making. JAMA. 2000;284:835–842. doi: 10.1001/jama.284.7.835. [DOI] [PubMed] [Google Scholar]
- 17.Eagle KA, Lim MJ, Dabbous OH, Pieper KS, Goldberg RJ, Van de Werf F, Goodman SG, Granger CB, Steg PG, Gore JM, Budaj A, Avezum A, Flather MD, Fox KA, GRACE Investigators A validated prediction model for all forms of acute coronary syndrome: estimating the risk of 6-month postdischarge death in an international registry. JAMA. 2004;291:2727–2733. doi: 10.1001/jama.291.22.2727. [DOI] [PubMed] [Google Scholar]
- 18.Boersma E, Pieper KS, Steyerberg EW, Wilcox RG, Chang WC, Lee KL, Akkerhuis KM, Harrington RA, Deckers JW, Armstrong PW, Lincoff AM, Califf RM, Topol EJ, Simoons ML. Predictors of outcome in patients with acute coronary syndromes without persistent ST-segment elevation results from an international trial of 9461 patients. Circulation. 2000;101:2557–2567. doi: 10.1161/01.cir.101.22.2557. [DOI] [PubMed] [Google Scholar]
- 19.Nikolsky E, Mehran R, Dangas G, Fahy M, Na Y, Pocock SJ, Lincoff AM, Stone GW. Development and validation of a prognostic risk score for major bleeding in patients undergoing percutaneous coronary intervention via the femoral approach. Eur Heart J. 2007;28:1936–1945. doi: 10.1093/eurheartj/ehm194. [DOI] [PubMed] [Google Scholar]
- 20.Hoekstra JW, Pollack CV, Jr, Roe MT, Peterson ED, Brindis R, Harrington RA, Christenson RH, Smith SC, Ohman EM, Gibler WB. Improving the care of patients with non–ST-elevation acute coronary syndromes in the emergency department: the CRUSADE initiative. Acad Emerg Med. 2002;9:1146–1155. doi: 10.1111/j.1553-2712.2002.tb01569.x. [DOI] [PubMed] [Google Scholar]
- 21.Bhatt DL, Roe MT, Peterson ED, Li Y, Chen AY, Harrington RA, Greenbaum AB, Berger PB, Cannon CP, Cohen DJ, Gibson CM, Saucedo JF, Kleiman NS, Hochman JS, Boden WE, Brindis RG, Peacock WF, Smith SC, Jr, Pollack CV, Jr, Gibler WB, Ohman EM, CRUSADE Investigators Utilization of early invasive management strategies for high-risk patients with non–ST-segment elevation acute coronary syndromes: results from the CRUSADE Quality Improvement Initiative. JAMA. 2004;292:2096–2104. doi: 10.1001/jama.292.17.2096. [DOI] [PubMed] [Google Scholar]
- 22.Cockcroft DW, Gault MH. Prediction of creatinine clearance from serum creatinine. Nephron. 1976;16:31–41. doi: 10.1159/000180580. [DOI] [PubMed] [Google Scholar]
- 23.Liang K-Y, Zeger SL. Longitudinal data analysis using generalized linear models. Biometrika. 1986;73:13–22. [Google Scholar]
- 24.Rowe JW, Andres R, Tobin JD, Norris AH, Shock NW. The effect of age on creatinine clearance in men: a cross-sectional and longitudinal study. J Gerontol. 1976;31:155–163. doi: 10.1093/geronj/31.2.155. [DOI] [PubMed] [Google Scholar]
- 25.Spivak JL. Anemia in the elderly: time for new blood in old vessels? Arch Intern Med. 2005;165:2187–2189. doi: 10.1001/archinte.165.19.2187. [DOI] [PubMed] [Google Scholar]
- 26.Guralnik JM, Eisenstaedt RS, Ferrucci L, Klein HG, Woodman RC. Prevalence of anemia in persons 65 years and older in the United States: evidence for a high rate of unexplained anemia. Blood. 2004;104:2263–2268. doi: 10.1182/blood-2004-05-1812. [DOI] [PubMed] [Google Scholar]
- 27.Choussat R, Black A, Bossi I, Fajadet J, Marco J. Vascular complications and clinical outcome after coronary angioplasty with platelet IIb/IIIa receptor blockade. Comparison of transradial vs transfemoral arterial access. Eur Heart J. 2000;21:662–667. doi: 10.1053/euhj.1999.1945. [DOI] [PubMed] [Google Scholar]
- 28.Kiemeneij F, Laarman GJ, Odekerken D, Slagboom T, van der Wieken R. A randomized comparison of percutaneous transluminal coronary angioplasty by the radial, brachial and femoral approaches: the access study. J Am Coll Cardiol. 1997;29:1269–1275. doi: 10.1016/s0735-1097(97)00064-8. [DOI] [PubMed] [Google Scholar]
- 29.Cannon CP, Husted S, Harrington RA, Scirica BM, Emanuelsson H, Peters G, Storey RF, DISPERSE-2 Investigators Safety, tolerability, and initial efficacy of AZD6140, the first reversible oral adenosine diphosphate receptor antagonist, compared with clopidogrel, in patients with non–ST-segment elevation acute coronary syndrome: primary results of the DISPERSE-2 trial. J Am Coll Cardiol. 2007;50:1844–1851. doi: 10.1016/j.jacc.2007.07.053. [DOI] [PubMed] [Google Scholar]
- 30.Greenbaum AB, Ohman EM, Gibson CM, Borzak S, Stebbins AL, Lu M, Le May MR, Stankowski JE, Emanuelsson H, Weaver WD. Preliminary experience with intravenous P2Y12 platelet receptor inhibition as an adjunct to reduced-dose alteplase during acute myocardial infarction: results of the Safety, Tolerability and Effect on Patency in Acute Myocardial Infarction (STEP-AMI) angiographic trial. Am Heart J. 2007;154:702–709. doi: 10.1016/j.ahj.2007.06.001. [DOI] [PubMed] [Google Scholar]
- 31.LaPointe NM, Chen AY, Alexander KP, Roe MT, Pollack CV, Jr, Lytle BL, Ohman ME, Gibler BW, Peterson ED. Enoxaparin dosing and associated risk of in-hospital bleeding and death in patients with non–ST-segment elevation acute coronary syndromes. Arch Intern Med. 2007;167:1539–1544. doi: 10.1001/archinte.167.14.1539. [DOI] [PubMed] [Google Scholar]
- 32.Chesebro JH, Knatterud G, Roberts R, Borer J, Cohen LS, Dalen J, Dodge HT, Francis CK, Hillis D, Ludbrook P, Markis JE, Mueller H, Passamani ER, Powers ER, Rao AK, Robertson T, Ross A, Ryan TJ, Sobel BE, Willerson J, Williams DO, Zaret BL, Braunwald E. Thrombolysis in Myocardial Infarction (TIMI) trial, phase I: a comparison between intravenous tissue plasminogen activator and intravenous streptokinase. Clinical findings through hospital discharge. Circulation. 1987;76:142–154. doi: 10.1161/01.cir.76.1.142. [DOI] [PubMed] [Google Scholar]
- 33.An international randomized trial comparing four thrombolytic strategies for acute myocardial infarction. The GUSTO investigators. N Engl J Med. 1993;329:673–682. doi: 10.1056/NEJM199309023291001. [DOI] [PubMed] [Google Scholar]