Abstract
Background
Bleeding risk stratification in acute coronary syndrome is of highest clinical interest but current risk scores have limitations. We sought to develop and validate a new in‐hospital bleeding risk score for patients with acute myocardial infarction.
Methods and Results
From the nationwide SWEDEHEART (Swedish Web‐System for Enhancement and Development of Evidence‐Based Care in Heart Disease Evaluated According to Recommended Therapies) register, 97,597 patients with acute myocardial infarction enrolled from 2009 until 2014 were selected. A full model with 23 predictor variables and 8 interaction terms was fitted using logistic regression. The full model was approximated by a model with 5 predictors and 1 interaction term. Calibration, discrimination, and clinical utility was evaluated and compared with the ACTION (Acute Coronary Treatment and Intervention Outcomes Network) and CRUSADE (Can Rapid Risk Stratification of Unstable Angina Patients Suppress Adverse Outcomes With Early Implementation of the ACC/AHA Guidelines) scores. Internal and temporal validity was assessed. In‐hospital major bleeding, defined as fatal, intracranial, or requiring surgery or blood transfusion, occurred in 1356 patients (1.4%). The 5 predictors in the approximate model that constituted the SWEDEHEART score were hemoglobin, age, sex, creatinine, and C‐reactive protein. The ACTION and CRUSADE scores were poorly calibrated in the derivation cohort and therefore were recalibrated. The SWEDEHEART score showed higher discriminative ability than both recalibrated scores, overall (C‐index 0.80 versus 0.73/0.72) and in all predefined subgroups. Decision curve analysis demonstrated consistently positive and higher net benefit for the SWEDEHEART score compared with both recalibrated scores across all clinically relevant decision thresholds. The original ACTION and CRUSADE scores showed negative net benefit.
Conclusions
The 5‐item SWEDEHEART score discriminates in‐hospital major bleeding in patients with acute myocardial infarction and has superior model performance compared with the recalibrated ACTION and CRUSADE scores.
Keywords: acute myocardial infarction, bleeding, registry, risk score
Subject Categories: Myocardial Infarction, Complications, Acute Coronary Syndromes
Clinical Perspective
What Is New?
From a population of patients with acute myocardial infarction representative of a whole nation, we developed a simple 5‐item (age, sex, hemoglobin, creatinine, and C‐reactive protein) in‐hospital bleeding risk score that showed better model performance overall and in all important subgroups where the current guideline‐recommended CRUSADE (Can Rapid Risk Stratification of Unstable Angina Patients Suppress Adverse Outcomes With Early Implementation of the ACC/AHA Guidelines) score and the ACTION (Acute Coronary Treatment and Intervention Outcomes Network) score underachieved.
Our results also demonstrate that calibration as well as discrimination is of upmost importance when evaluating model performance.
What Are the Clinical Implications?
Bleeding risk assessment by the CRUSADE score is listed as one of the main quality indicators for acute myocardial infarction care by the European Society of Cardiology (ESC). However, the ESC guidelines give use of the CRUSADE score a cautious class IIb recommendation restricted to patients undergoing coronary angiography and the American guidelines do not recommend the use of a specific risk score for bleeding.
Our score seems valuable in the early assessment of patients and furthermore our findings stress the need for validation and, if necessary, recalibration before applying a risk score to a new population.
Advances in the management of acute coronary syndrome (ACS), including the use of invasive strategies, the implementation of dual antiplatelet therapy and more efficient antithrombotic treatment, has been associated with a decline in mortality across a number of countries.1, 2, 3 Yet, such improvements come at the price of an increased risk of bleeding and associated mortality and morbidity.4, 5, 6, 7, 8 Given that harm associated with bleeding may attenuate the net benefit of antithrombotic therapies,9, 10 there is international interest in initiatives to reduce bleeding complications in the context of ACS.
Guidelines from North America and Europe emphasize that bleeding risk as well as ischemic risk should be assessed.11, 12 While the GRACE (Global Registry of Acute Coronary Events) risk score13 is advocated for estimating future ischemic risk11, 12 and has been validated in numerous studies to show good or excellent model performance14 there is a paucity of in‐hospital bleeding risk models that demonstrate good model performance. Of the ACTION (Acute Coronary Treatment and Intervention Outcomes Network),15 ACUITY (Acute Catheterization and Urgent Intervention Triage Strategy),16 and CRUSADE (Can Rapid Risk Stratification of Unstable Angina Patients Suppress Adverse Outcomes With Early Implementation of the ACC/AHA Guidelines)17 bleeding risk scores, the latter score is the most established but is limited by weak discriminative ability especially among patients with ACS managed noninvasively, the elderly, patients prescribed oral anticoagulants (OACs), and patients with renal failure (glomerular filtration rate <60 mL/min).18, 19, 20, 21 In the European Society of Cardiology (ESC) guidelines,12 the CRUSADE score has a class IIb recommendation restricted to patients receiving coronary angiography.
The aim of the present study was to develop and validate a new risk score for in‐hospital major bleeding in a population including the aforementioned vulnerable subgroups. The SWEDEHEART (Swedish Web‐System for Enhancement and Development of Evidence‐Based Care in Heart Disease Evaluated According to Recommended Therapies) registry is an established national register highly representative of the Swedish ACS population, thereby enabling generalizable patient‐specific research into the care and outcomes of patients hospitalized with ACS.
Methods
The data and study materials will not be made available to other researchers for purposes of reproducing the results or replicating the procedure. However, a detailed description of the analytical method and the full equation is provided in Data S1.
The SWEDEHEART registry has been previously described.22 Briefly, it is a national registry of patients hospitalized for suspected ACS or undergoing coronary or valve intervention. All Swedish hospitals (n=72) participate, covering ≈90% of patients with acute myocardial infarction (MI) treated in hospitals in Sweden. The registry is regularly monitored with 95% to 96% agreement with regard to key variables between the registry and electronic health records.
In order to use a contemporary and reasonably large population, we selected all consecutive patients with acute MI, defined as discharge International Classification of Diseases, 10th Revision (ICD‐10) code I21 enrolled in the SWEDEHEART registry from January 1, 2009, until October 10, 2014 (n=109 714). After excluding 11 937 readmissions, multiple imputation was performed on the remaining 97 777 index hospitalizations. Subsequently, 180 (0.2%) patients with missing outcome data were excluded, resulting in a derivation cohort of 97 597 patients.
All Swedish residents are given a unique and permanent personal identification number that allows cross‐matching between several national registries. This made it possible to enrich baseline data by merging SWEDEHEART with the National Patient Registry, which contains information on discharge ICD codes from all admissions to Swedish hospitals since 1987.
All patients are informed about their participation in the registry and that they are allowed to opt out. Written consent is not required according to Swedish law. Patients were anonymized by a unique study identification number before the data analysis. The study protocol was approved by the regional ethics committee in Stockholm. The National Board of Health and Welfare approved the merging of registries.
The outcome was in‐hospital non–coronary artery bypass graft major bleeding defined as fatal, intracranial, or bleeding requiring blood transfusion or surgery (including endoscopic and vascular intervention) according to the SWEDEHEART registry. Other bleeding events were not recorded in the registry. For patients who underwent coronary artery bypass graft surgery during the same hospital stay, bleeding was only registered if it occurred before surgery.
All laboratory data (including hemoglobin, C‐reactive protein [CRP], and creatinine) were recorded on admission to the hospital. Renal function was measured as creatinine and the glomerular filtration rate was estimated using the Chronic Kidney Disease Epidemiology Collaboration equation.23 Heart failure at presentation was defined as any pulmonary rales or pulmonary edema. Previous bleeding was defined as any hospitalization with an ICD‐10 bleeding diagnosis (listed in Table S1) before the date of index acute MI.
Statistical Analysis
A detailed description of the statistical methods including an expanded list of references is presented in Data S1. Before any model fitting, 23 predictor variables (age, sex, body weight, hypertension, diabetes mellitus, previous MI, previous percutaneous coronary artery intervention, previous coronary artery bypass graft surgery, previous stroke, peripheral artery disease, chronic heart failure, previous bleeding, single aspirin therapy, dual antiplatelet therapy, OAC therapy, cardiopulmonary resuscitation before hospital, atrial fibrillation at hospitalization, cardiogenic shock, heart failure at hospitalization, ST‐segment–elevation MI, serum creatinine, CRP, and hemoglobin) were selected based on clinical relevance. In the derivation cohort, the percentage of missing values across the predictors and the bleeding outcome varied between 0 and 12% (Table S2). In total, 25% of the records were incomplete in the derivation cohort. Assuming a missing‐at‐random mechanism, the incomplete variables across the index population were imputed under fully conditional specification using a random forest‐based algorithm,24, 25 resulting in 25 imputed sets. Convergence was reached after 30 iterations. Before any model fitting, all patients with missing outcome data were removed as recommended by Von Hippel.26 Models were then fitted to the imputed data sets and the corresponding estimates were combined into one overall estimate.27
The full model including all predictors and 8 prespecified interactions (age and sex, age and diabetes mellitus, age and serum creatinine, age and hemoglobin, sex and body weight, sex and diabetes mellitus, sex and serum creatinine, and sex and hemoglobin) was fitted to the outcome variable using logistic regression. The continuous predictors (age, body weight, serum creatinine, hemoglobin, and CRP) were truncated at the first and 99th percentiles to limit the influence of extreme values. In addition, skewed predictors (body weight and serum creatinine) were log‐transformed. All continuous predictors were modeled using restricted cubic splines to allow for nonlinear relationships. Since the full model was too complex for routine use, a simplified approximate model was developed using the stepdown method.28 The approximate model was termed the SWEDEHEART score and presented in the form of a nomogram and an equation. Risk classes were created according to <0.5%, 0.5% to 1%, 1% to 2%, 2% to 4%, and >4% predicted risk of in‐hospital major bleeding.
Given that the ACTION and CRUSADE scores were poorly calibrated in the derivation cohort (Figure 1), recalibrated scores were obtained by fitting a logistic regression model using the ACTION and CRUSADE score, respectively, as the only predictor and the binary bleeding indicator as the outcome.29
Figure 1.
Calibration plots for the ACTION (Acute Coronary Treatment and Intervention Outcomes Network), recalibrated ACTION, CRUSADE (Can Rapid Risk Stratification of Unstable Angina Patients Suppress Adverse Outcomes With Early Implementation of the ACC/AHA Guidelines), recalibrated CRUSADE, and SWEDEHEART (Swedish Web‐System for Enhancement and Development of Evidence‐Based Care in Heart Disease Evaluated According to Recommended Therapies) scores. Deviations from perfect calibration (Ideal) represent bias in the predicted probabilities. For the ACTION and CRUSADE scores, the figure illustrates the observed proportion of bleeding in the derivation cohort (with 95% CIs) vs the proportion of bleeding in the previously published original derivation cohorts (Apparent). For the recalibrated ACTION and CRUSADE scores and the SWEDEHEART score, the figure illustrates the observed proportion of bleeding in the derivation cohort (with 95% CIs) vs the predicted probability of bleeding (Apparent) and its optimism‐corrected estimate (Optimism‐corrected). The apparent and optimism‐corrected calibration curves are almost indistinguishable, indicating no overfitting.
The C‐index was used to assess discrimination. Calibration was assessed graphically using smooth nonparametric regression and the corresponding CIs were obtained using 2000 bootstrap samples.30, 31 Discrimination and calibration were evaluated in the full derivation cohort and in predefined subgroups. The recalibrated ACTION and CRUSADE scores and the SWEDEHEART score were internally validated using 200 bootstrap samples. Temporal validity of the SWEDEHEART score was assessed using internal–external cross‐validation32 where each admission year was omitted in turn. The optimism (bias caused by overfitting) was estimated and optimism‐corrected performance estimates were obtained.
The clinical utility of the ACTION, CRUSADE, and SWEDEHEART scores was evaluated using decision curve analysis.33 Using a score to classify patients into low‐ and high‐risk bleeders requires a decision threshold, classifying all patients with a predicted bleeding probability above the threshold as high‐risk bleeders. Since the score is not perfect, the resulting decision will either be incorrect (a false‐positive bleed) or correct (a true‐positive bleed). The net benefit of using a score at a given threshold is defined as the difference between the proportion of true positives and the proportion of false positives weighted by the odds of the threshold, where the weight corresponds to the cost‐benefit ratio associated with incorrect and correct decisions. For a given threshold, a score is considered clinically useful if it has a positive net benefit and harmful if the net benefit is negative. For each score, the decision curve illustrates the net benefit corresponding to the decision thresholds. The score with the higher net benefit across the range of clinically relevant decision thresholds is preferred.
Methods and results are presented in line with the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) statement.34 All statistical analyses were performed with R (version 3.4.3, The R Foundation for statistical computing) using the add‐on packages mice (version 2.44) and rms (version 5.1–1).
Results
Baseline Characteristics and in‐Hospital Course
Baseline characteristics on admission of patients with and without major in‐hospital bleeding are listed in Table 1. The median time to discharge was 4 days (interquartile range, 3–6 days). Major bleeding occurred in 1356 patients (1.4%), and, of these, 50 (0.1%) were fatal, 114 (0.1%) were cerebral, and 1192 (1.2%) required surgery or red blood cell transfusion. Patients with a bleeding event were older (median 77 years versus 72 years), more often female (46% versus 35%) and had an overall higher preexisting burden of comorbidity and more extensive medication, including single aspirin therapy (36.8% versus 44.4%), dual antiplatelet therapy (4.1% versus 6.3%), or OAC therapy (5.6% versus 8.0%).
Table 1.
Baseline Characteristics
| No Major Bleed | Major Bleed | |
|---|---|---|
| (n=96 241) | (n=1356) | |
| Demography | ||
| Age, y | 72 (63–81) | 77 (69–83) |
| Weight, kg | 78 (69–90) | 74 (64–85) |
| Women | 33 779 (35.1) | 621 (45.8) |
| Medical history | ||
| Hypertension | 54 923 (57.1) | 916 (67.6) |
| Diabetes mellitus | 23 172 (24.1) | 411 (30.3) |
| Previous MI | 23 268 (24.2) | 398 (29.4) |
| Previous PCI | 13 295 (13.8) | 200 (14.7) |
| Previous CABG | 8286 (8.6) | 145 (10.7) |
| Previous PAD | 5641 (5.9) | 157 (11.6) |
| Previous stroke | 11 498 (11.9) | 259 (19.1) |
| Chronic heart failure | 13 647 (14.2) | 337 (24.9) |
| Previous bleeding | 5918 (6.1) | 170 (12.5) |
| COPD | 6986 (7.3) | 142 (10.5) |
| Previous cancer | 3176 (3.3) | 91 (6.7) |
| Medication on admission | ||
| β‐Blocker | 35 771 (37.2) | 633 (46.7) |
| RAS blockade | 59 934 (62.3) | 739 (54.5) |
| Calcium antagonist | 18 368 (19.1) | 324 (23.9) |
| Digoxin | 2052 (2.1) | 48 (3.5) |
| Statins | 28 056 (29.2) | 481 (35.5) |
| Diuretics | 23 301 (24.2) | 500 (36.9) |
| Aspirin | 35 389 (36.8) | 602 (44.4) |
| DAPT | 3933 (4.1) | 86 (6.3) |
| OAC | 5417 (5.6) | 108 (8.0) |
| Presentation | ||
| CPR before hospital | 1708 (1.8) | 37 (2.7) |
| Atrial fibrillation | 10 262 (10.7) | 213 (15.7) |
| Heart rate, bpm | 79 (66–93) | 86 (70–100) |
| Systolic blood pressure, mm Hg | 147 (130–167) | 140 (120–160) |
| Symptoms or signs of HF | 11 312 (11.8) | 288 (21.2) |
| Shock | 1214 (1.3) | 36 (2.7) |
| ST‐elevation | 31 991 (33.2) | 457 (33.7) |
| Laboratory data on admission | ||
| Hemoglobin, g/L | 139 (127–149) | 116 (101–131) |
| Anemia WHO definition | 19 729 (20.5) | 818 (60.3) |
| Creatinine, mmol/L | 84 (70–103) | 99 (77–137) |
| GFR by CKD‐EPI, mL/min | 74 (55–89) | 56 (37–76) |
| CRP | 5 (3–15) | 17 (5–66) |
Values are expressed as median (interquartile range) or number (percentage). bpm indicates beats per minute; CABG, coronary artery bypass graft; CKD‐EPI, Chronic Kidney Disease Epidemiology Collaboration; COPD, chronic obstructive pulmonary disease; CPR, cardiopulmonary resuscitation; CRP, C‐reactive protein; DAPT, dual antiplatelet therapy; GFR, glomerular filtration rate; HF, heart failure; MI, myocardial infarction; OAC, oral anticoagulant; PAD, peripheral artery disease; PCI, percutaneous coronary intervention; RAS, renin‐angiotensin system; WHO, World Health Organization.
Patients with a major bleeding event less frequently received an invasive coronary strategy (angiography 56.9% versus 77.2% and percutaneous coronary intervention 43.9% versus 60.3%). In addition, having a major bleeding event was associated with a 3‐fold incidence of new‐onset atrial fibrillation, cardiac arrest, and death, and a 4‐ and 5‐fold incidence of reinfarction and in‐hospital shock, respectively (Table 2).
Table 2.
In‐Hospital Course
| Variable | Major Bleeding | Total (N=97 597) | |
|---|---|---|---|
| No (n=96 241) | Yes (n=1356) | ||
| Treatment | |||
| Angiography | 74 284 (77.2) [0] | 772 (56.9) [0] | 75 056 (76.9) [0] |
| PCI | 58 002 (60.3) [0] | 607 (44.8) [0] | 58 609 (60.1) [0] |
| Fondaparinux | 48 749 (50.7) [159] | 458 (33.8) [4] | 49 207 (50.4) [163] |
| Heparin | 5495 (5.7) [159] | 77 (5.7) [4] | 5572 (5.7) [163] |
| LMWH | 10 057 (10.4) [159] | 307 (22.6) [4] | 10 364 (10.6) [163] |
| Diuretics | 18 444 (19.2) [92] | 646 (47.6) [2] | 19 090 (19.6) [94] |
| Inotropic drugs | 2934 (3.0) [88] | 193 (14.2) [4] | 3127 (3.2) [92] |
| Thrombolysis | 1599 (1.7) [0] | 35 (2.6) [0] | 1634 (1.7) [0] |
| Evaluation (echocardiography) | |||
| LVEF <50% | 30 220 (31.4) [23 891] | 595 (43.9) [351] | 30 815 (31.6) [24 332] |
| Complications | |||
| New‐onset atrial fibrillation | 3936 (4.1) [1603] | 158 (11.7) [6] | 4094 (4.2) [1609] |
| Reinfarction | 662 (0.7) [131] | 37 (2.7) [1] | 699 (0.7) [132] |
| Shock in hospital | 1960 (2.0) [1502] | 137 (10.1) [2] | 2097 (2.1) [1504] |
| CPR | 3038 (3.2) [15] | 138 (10.2) [0] | 3176 (3.3) [15] |
| Death | 4595 (4.8) [12] | 198 (14.6) [0] | 4793 (4.9) [12] |
Values are expressed as number (percentage) [missing]. CPR indicates cardiopulmonary resuscitation; LMWH, low‐molecular‐weight heparin; LVEF, left ventricular ejection fraction; PCI, percutaneous coronary intervention.
Model Performance and Validation
The full model including all predictors and interactions (Figure S1) had a C‐index of 0.81 (95% CI, 0.80–0.82). The internal validation indicated only modest overfitting with an optimism‐corrected C‐index and a calibration slope of 0.80 and 1.04, respectively. The approximate model including only the 5 most important predictors (hemoglobin, age, sex, creatinine, and CRP) and 1 interaction term (hemoglobin and sex) represented 92.7% of the full model and was termed the SWEDEHEART score, presented as a nomogram (Figure 2) and equation (Figure S2). The SWEDEHEART score had a C‐index of 0.80 (95% CI, 0.79–0.81) with an optimism‐corrected C‐index and a calibration slope of 0.80 and 0.99 respectively, indicating negligible overfitting. The internal‐external temporal validation indicated temporal stability with an optimism corrected C‐index of 0,80. (C in C‐index in cursive). The ACTION and CRUSADE scores had a C‐index of 0.73 (95% CI, 0.72–0.74) and 0.72 (95% CI, 0.71–0.74), respectively (Table 3), which is in agreement with the results presented in the original publications.15, 17
Figure 2.
Nomogram for the SWEDEHEART (Swedish Web‐System for Enhancement and Development of Evidence‐Based Care in Heart Disease Evaluated According to Recommended Therapies) score. To use the nomogram for an individual patient, the points for each predictor are read at the vertical line from the predictor value to the Points line (top). The total points are calculated and a vertical line from this value on the Total Points line to the Bleeding Risk line (bottom) provides the estimated probability of in‐hospital major bleeding risk. Total points <6 or >14 corresponds to a bleeding risk <0.5% or >10%, respectively. Example 1: men with hemoglobin=135, age=85 years, C‐reactive protein (CRP)=10, and creatinine=130. The points corresponding to each predictor are roughly 1.8, 2.2, 0.7, and 2.3. The total points are 7, corresponding to an in‐hospital major bleeding risk of ≈0.7%. Example 2: women with hemoglobin=135, age=85 years, CRP=10, and creatinine=130. The points corresponding to each predictor are roughly 3.3, 2.2, 0.7, and 2.3. The total points are 8.5, corresponding to an in‐hospital major bleeding risk of ≈1.2%.
Table 3.
C‐index for the ACTION, CRUSADE, and SWEDEHEART Scores
| No. | Bleeds | C‐Index | |||
|---|---|---|---|---|---|
| ACTION | CRUSADE | SWEDEHEART | |||
| Overall | 97 597 | 1356 | 0.73 (0.72–0.74) | 0.72 (0.71–0.74) | 0.80 (0.79–0.81) [0.80] |
| Sex | |||||
| Women | 34 400 | 621 | 0.68 (0.66–0.70) | 0.66 (0.64–0.68) | 0.74 (0.72–0.76) [0.74] |
| Men | 63 197 | 735 | 0.76 (0.74–0.78) | 0.76 (0.74–0.77) | 0.83 (0.81–0.84) [0.82] |
| Missing | 0 | ||||
| Age, y | |||||
| <75 | 56 002 | 572 | 0.77 (0.75–0.79) | 0.76 (0.74–0.78) | 0.81 (0.80–0.83) [0.81] |
| ≥75 | 41 594 | 784 | 0.67 (0.65–0.69) | 0.65 (0.64–0.67) | 0.76 (0.75–0.78) [0.76] |
| Missing | 1 | ||||
| Anticoagulation | |||||
| No | 91 204 | 1228 | 0.73 (0.72–0.75) | 0.72 (0.71–0.74) | 0.80 (0.79–0.81) [0.80] |
| Yes | 5525 | 108 | 0.71 (0.67–0.76) | 0.70 (0.65–0.74) | 0.77 (0.73–0.82) [0.77] |
| Missing | 868 | ||||
| Angiography | |||||
| No | 22 541 | 584 | 0.63 (0.61–0.65) | 0.59 (0.57–0.62) | 0.76 (0.74–0.79) [0.76] |
| Yes | 75 056 | 772 | 0.75 (0.73–0.77) | 0.74 (0.73–0.76) | 0.79 (0.78–0.81) [0.79] |
| Missing | 0 | ||||
| STEMI | |||||
| No | 64 730 | 896 | 0.74 (0.73–0.76) | 0.72 (0.71–0.74) | 0.82 (0.80–0.83) [0.81] |
| Yes | 31 304 | 447 | 0.74 (0.72–0.76) | 0.73 (0.71–0.75) | 0.77 (0.75–0.79) [0.77] |
| Missing | 1563 | ||||
| GFR | |||||
| <60 | 28 236 | 715 | 0.66 (0.65–0.68) | 0.65 (0.63–0.67) | 0.75 (0.73–0.77) [0.75] |
| ≥60 | 63 999 | 578 | 0.73 (0.71–0.75) | 0.71 (0.69–0.73) | 0.79 (0.77–0.81) [0.78] |
| Missing | 5362 | ||||
C‐index (95% CI) (if possible C in C‐index in cursive). ACTION indicates Acute Coronary Treatment and Intervention Outcomes Network; CRUSADE, Can Rapid Risk Stratification of Unstable Angina Patients Suppress Adverse Outcomes With Early Implementation of the ACC/AHA Guidelines; GFR, glomerular filtration rate; STEMI, ST‐segment–elevation myocardial infarction; SWEDEHEART, Swedish Web‐System for Enhancement and Development of Evidence‐Based Care in Heart Disease Evaluated According to Recommended Therapies.
The SWEDEHEART score achieved a consistently higher C‐index than the ACTION and CRUSADE scores in the subgroups of women, elderly (>75 years), patients managed noninvasively, patients taking OACs, those with ST‐segment–elevation MI/non–ST‐segment–elevation MI, and those with chronic kidney disease <60 mL/min (Table 3). In the subgroup of patients managed noninvasively, which constituted 25% of the study population, the CRUSADE score was only weakly discriminative, with a C‐index of 0.59 (95% CI, 0.57–0.62).
The ACTION and CRUSADE scores were poorly calibrated in the derivation cohort (Figure 1), and when recalibrated had an optimism‐corrected C‐index of 0.73 and 0.72, respectively, and a calibration slope of 1.0 in both cases, indicating no overfitting. The recalibrated ACTION and CRUSADE scores and the SWEDEHEART score showed comparable calibration (Figure 1) for bleeding probabilities between 1% and 4%, whereas the SWEDEHEART score demonstrated superior calibration for bleeding probabilities <1% and >4%. Calibration plots comparing the recalibrated ACTION and CRUSADE scores and the SWEDEHEART score in all subgroups are shown in Figure S3. The observed bleeding risks within the ACTION, CRUSADE, and SWEDEHEART risk classes are illustrated in Figures 3 and 4. The SWEDEHEART score demonstrated improved risk stratification within the ACTION and CRUSADE risk classes (Figures 3A and 4A), whereas the ACTION and CRUSADE scores did not improve the risk stratification within the SWEDEHEART risk classes (Figures 3B and 4B).
Figure 3.
Observed bleeding risk within the ACTION (Acute Coronary Treatment and Intervention Outcomes Network) and SWEDEHEART (Swedish Web‐System for Enhancement and Development of Evidence‐Based Care in Heart Disease Evaluated According to Recommended Therapies) risk classes. Observed bleeding risk and corresponding 95% CIs are illustrated within the ACTION and SWEDEHEART risk classes. In (A) the observed bleeding risk within the ACTION risk classes is shown for each SWEDEHEART risk class, demonstrating high variability of the observed bleeding risk and low agreement between the observed bleeding risk and the ACTION risk classes. In (B) the observed bleeding risk within the SWEDEHEART risk classes is shown for each ACTION risk class, showing low variability of the observed bleeding risk and high agreement between observed bleeding risk and the SWEDEHEART risk classes.
Figure 4.
Observed bleeding risk within the CRUSADE (Can Rapid Risk Stratification of Unstable Angina Patients Suppress Adverse Outcomes With Early Implementation of the ACC/AHA Guidelines) and SWEDEHEART (Swedish Web‐System for Enhancement and Development of Evidence‐Based Care in Heart Disease Evaluated According to Recommended Therapies) risk classes. Observed bleeding risk and corresponding 95% CIs are illustrated within the CRUSADE and SWEDEHEART risk classes. In (A) the observed bleeding risk within the CRUSADE risk classes is shown for each SWEDEHEART risk class, demonstrating high variability of the observed bleeding risk and low agreement between the observed bleeding risk and the CRUSADE risk classes. In (B) the observed bleeding risk within the SWEDEHEART risk classes is shown for each CRUSADE risk class, showing low variability of the observed bleeding risk and high agreement between observed bleeding risk and the SWEDEHEART risk classes.
Clinical utility was evaluated using decision curve analysis.33 Across the range of clinically relevant decision thresholds, the SWEDEHEART score had consistent positive and larger net benefit in comparison with the recalibrated ACTION and CRUSADE scores (Figure 5). For example, at a threshold of 2% in‐hospital major bleeding risk, the SWEDEHEART score will identify 2.0 and 2.3 additional in‐hospital major bleeds compared with the recalibrated ACTION and CRUSADE scores, respectively, in a population with 13.9 in‐hospital major bleeds per 1000 individuals, without increasing the number of false‐positives. Because of their poor calibration, the original ACTION and CRUSADE scores showed negative net benefit (Tables S3 and S4). In the same setting as above, the SWEDEHEART score will identify 11.3 additional in‐hospital major bleeds compared with the original ACTION and CRUSADE scores.
Figure 5.
Decision curve analysis. For each decision threshold, the net benefit of the recalibrated ACTION (Acute Coronary Treatment and Intervention Outcomes Network) and CRUSADE (Can Rapid Risk Stratification of Unstable Angina Patients Suppress Adverse Outcomes With Early Implementation of the ACC/AHA Guidelines) scores and the SWEDEHEART (Swedish Web‐System for Enhancement and Development of Evidence‐Based Care in Heart Disease Evaluated According to Recommended Therapies) score is shown. In addition, the net benefit assuming that all patients have a bleeding risk higher than the threshold (all risk high) as well as the net benefit assuming that all patients have a bleeding risk lower than the threshold (all risk low) are shown. For a given threshold, the difference in net benefit between two scores is the additional number of bleeds identified (per 1000) without increasing the number of false positive classifications. Across the range of decision thresholds, the SWEDEHEART score was consistently positive and had larger net benefit than the recalibrated ACTION and CRUSADE scores.
If CRP was not available, we made an alternate SWEDEHEART score without CRP. This alternate score represented 88.2% of the full model and had a C‐index of 0.79 (95% CI, 0.77–0.80) and an optimism‐corrected C‐index and calibration slope of 0.78 and 1.0, respectively. A more detailed description including nomogram, calibration curve, and equation for the SWEDEHEART score without CRP is shown in Table S5 and Figures S4 through S6.
Discussion
In this study we developed and validated a new risk score for in‐hospital major bleeding in nearly 100 000 patients with acute MI. Our new score, based on the 5 baseline variables (hemoglobin, age, sex, creatinine, and CRP) showed good discrimination and calibration across different subgroups. The ACTION score and the currently recommended CRUSADE score was, even after recalibration, inferior to the SWEDEHEART score and, as previously described, especially unfavorable in patients not undergoing coronary angiography19
Need for Validation and Recalibration of Scores
The ACTION and CRUSADE scores have already been externally validated in several populations. However, both scores were associated with negative net benefit in the derivation cohort, indicating that their use was harmful. This finding is of additional clinical importance since it stresses the need for validation and, if needed, recalibration before applying a risk score to a new population.
Comparison With Other Scores
What makes our score unique is that it was derived from a cohort including almost all patients hospitalized with acute MI in Sweden over a study period of almost 6 years. Of the entire index population of 97 777 patients, only 180 (0.2%) patients with missing outcome data were excluded. An internal–external cross‐validation was used to assess the temporal validity of the score within the geographical area. The ACTION and CRUSADE registers comprised selected patients on a national level since they only covered a small proportion of US hospitals. There is also lack of data regarding the proportion of patients with ACS who were included in the registries at participating hospitals. CRUSADE only included non–ST‐segment–elevation MI and excluded patients receiving OACs. Both the ACTION and CRUSADE derivation cohorts excluded patients with missing data on certain baseline variables as well as patients who died within 24 and 48 hours, respectively.15, 17 The ACUITY score was derived from 2 randomized controlled trials and thus included a strongly selected population.16
Overall Low Incidence of Bleeding
The incidence of major bleeding in our derivation cohort was low (1.4%) compared with ACTION (10.8%), ACUITY (7.3%), and CRUSADE (9.4%). There are many plausible explanations for this finding. Comparison of bleeding incidence is dependent not only on the definition but also the characteristics and treatment patterns of the population. Our bleeding definition did only include non–coronary artery bypass graft–related bleeds that were fatal, were intracranial, or that led to blood transfusion or intervention during a short observation period. Bleedings with only a large drop in hemoglobin were not counted. In the CRUSADE population, femoral access was dominating and glycoprotein IIb/IIIa blockers were used more frequently. Since then, several bleeding avoidance strategies have been implemented and most of the patients in our derivation cohort were treated through radial access, few received glycoprotein IIb/IIIa blockers, and new drugs (fondaparinux and bivalirudin) with a lower risk of bleeding were used. Finally, but not least important, when using a bleeding definition including transfusion, the bleeding incidence may also reflect the transfusion strategy in the study population. Our bleeding definition differed from the CRUSADE definition, which included hemoglobin drops and retroperitoneal bleeds. The transfused bleeds constituted over 6% in the CRUSADE population compared with only 1.2% in the SWEDEHEART population, which may suggest either more severe bleeds that led to transfusion and/or a more liberal transfusion strategy in the CRUSADE population. Geographical variation in transfusion strategy with a significantly higher incidence in US compared with non‐US countries has been described by Rao et al35 even though the rate of transfusion in patients undergoing percutaneous coronary intervention in the United States has declined over the past years36
There may be underreporting of bleeding events but the register is regularly monitored with 95% to 96% variable agreement.
Anemia as a Marker of Increased Risk
The strong association between anemia and bleeding is to some extent a result of our bleeding definition as well as the fact that preexisting anemia by itself may exacerbate the consequences of bleeding. Bleeding in patients with normal or high hemoglobin levels at baseline will more seldom lead to red blood cell transfusion, whereas bleeding in patients with anemia at baseline will more often be followed by an intervention. Nonfatal bleeding with large drop in hemoglobin without red blood cell transfusion or surgery was not registered as a bleeding event in the registry. Finally, preexisting anemia may reflect prior or ongoing occult bleeding, which will become manifest when treated with antithrombotic treatment in the course of an ACS.
Inflammation as a Marker of Increased Risk
The variable CRP may be questioned because it is nonspecific, but markers of inflammation are certainly of interest for prediction of bleeding events. A small study in the elderly has shown that the addition of CRP and previous bleeding improved the discriminative ability of the CRUSADE score37 In the ACUITY score and the PRECISE‐DAPT38 inflammatory status is reflected by white blood cell count. Recent work from the PLATO (Platelet Inhibition and Patient Outcome) and ARISTOTLE (Apixaban for the Prevention of Stroke in Subjects With Atrial Fibrillation) studies showed that growth differentiation factor 15, a marker of inflammation and cell ageing, was strongly associated with bleeding39, 40, 41 Elevated CRP has also been shown to be associated with both upper and lower GI bleeding42, 43 as well as malignancy.44 As expected, the alternate SWEDEHEART score without CRP explained less (88.2%) of the full model compared with the score with CRP (92.6%), but the alternate score without CRP was still useful, with only slightly lower discriminative ability.
Need for Risk Scores
Assessment of patients with ACS and an indication for treatment in clinical practice are often not as clear‐cut as in clinical trials and clinical guidelines. When deciding on intensity of antithrombotic treatment, physicians often have to weigh the ischemic risk against the bleeding risk. Objective assessment of ischemic risk by the GRACE score has been shown to be superior to physician‐estimated ischemic risk.45 With an accutare bleeding risk score the same should probabably hold true for assessment of bleeding risk. Bleeding risk assessment by the CRUSADE score is listed as one of the main quality indicators for in‐hospital risk assessment in acute MI by the ESC Acute Cardiovascular Care Association Quality of Care Working Group.46 Our new risk model is complex but still easy to use provided a calculator on a computer or smartphone or delivered by decision support. A simple head‐calculation score with dichotomized variables may be even more easy to use, but may result in less precise risk estimates and decisions about treatment. Still, we will need future trials that can more exactly define how our score should influence clinical decision making.
Strengths and Limitations
While there are strengths to our methodological approach including size, completeness, and the state‐of‐art derivation and validation, we acknowledge some study limitations.
Although our results should be considered as highly valid in Sweden (with similar patients with ACS and coronary care as the rest of the Western world), lack of external validation in other geographically defined populations in which the score will be used is still a limitation of the study. But this holds true for all scores. Our bleeding definition was limited by the variables available in our register and therefore we could not use the CRUSADE or any other standardized bleeding definition. Nor did we have information on type of bleeding and, thus, outcome associated with different subtypes of bleeding could not be evaluated.
Using a different bleeding definition and also the temporal aspect with implementation of bleeding avoidance strategies since the CRUSADE era may have limited the performance of the CRUSADE score in our study population. Furthermore, hematocrit level was unavailable in the derivation cohort and replaced by hemoglobin converted to hematocrit for calculation of the CRUSADE score, which may be less exact. We also lacked data regarding dosing of antithrombotic treatment, which may have influenced the explanatory value of the full model.
Conclusions
The SWEDEHEART score, based on the variables hemoglobin, creatinine, age, sex, and CRP, offers better prediction of major in‐hospital bleeding in acute MI than the ACTION score and the currently recommended CRUSADE score.
Sources of Funding
This study was funded by the Swedish Foundation for Strategic Research and the Swedish Heart‐Lung Foundation.
Disclosures
None.
Supporting information
Data S1. Supplemental Statistical Methods (detailed).
Table S1. Previous Bleeding ICD‐10 Diagnoses
Table S2. Missing Observations
Table S3. Net Benefit for ACTION and SWEDEHEART
Table S4. Net Benefit for CRUSADE and SWEDEHEART
Table S5. C‐Index for SWEDEHEART Score without CRP
Figure S1. The full model.
Figure S2. SWEDEHEART (Swedish Web‐System for Enhancement and Development of Evidence‐Based Care in Heart Disease Evaluated According to Recommended Therapies) score equation.
Figure S3. Calibration in subgroups.
Figure S4. Score equation for SWEDEHEART (Swedish Web‐System for Enhancement and Development of Evidence‐Based Care in Heart Disease Evaluated According to Recommended Therapies) without C‐reactive protein (CRP).
Figure S5. Nomogram for SWEDEHEART (Swedish Web‐System for Enhancement and Development of Evidence‐Based Care in Heart Disease Evaluated According to Recommended Therapies) score without C‐reactive protein.
Figure S6. Calibration for SWEDEHEART (Swedish Web‐System for Enhancement and Development of Evidence‐Based Care in Heart Disease Evaluated According to Recommended Therapies) score without C‐reactive protein.
(J Am Heart Assoc. 2019;8:e012157 DOI: 10.1161/JAHA.119.012157.)
References
- 1. Hall M, Dondo TB, Yan AT, Goodman SG, Bueno H, Chew DP, Brieger D, Timmis A, Batin PD, Deanfield JE, Hemingway H, Fox KA, Gale CP. Association of clinical factors and therapeutic strategies with improvements in survival following non‐ST‐elevation myocardial infarction, 2003–2013. JAMA. 2016;316:1073–1082. [DOI] [PubMed] [Google Scholar]
- 2. Szummer K, Wallentin L, Lindhagen L, Alfredsson J, Erlinge D, Held C, James S, Kellerth T, Lindahl B, Ravn‐Fischer A, Rydberg E, Yndigegn T, Jernberg T. Improved outcomes in patients with ST‐elevation myocardial infarction during the last 20 years are related to implementation of evidence‐based treatments: experiences from the SWEDEHEART registry 1995–2014. Eur Heart J. 2017;38:3056–3065. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Fox KA, Steg PG, Eagle KA, Goodman SG, Anderson FA Jr, Granger CB, Flather MD, Budaj A, Quill A, Gore JM; Investigators G . Decline in rates of death and heart failure in acute coronary syndromes, 1999–2006. JAMA. 2007;297:1892–1900. [DOI] [PubMed] [Google Scholar]
- 4. Eikelboom JW, Mehta SR, Anand SS, Xie C, Fox KA, Yusuf S. Adverse impact of bleeding on prognosis in patients with acute coronary syndromes. Circulation. 2006;114:774–782. [DOI] [PubMed] [Google Scholar]
- 5. Rao SV, O'Grady K, Pieper KS, Granger CB, Newby LK, Van de Werf F, Mahaffey KW, Califf RM, Harrington RA. Impact of bleeding severity on clinical outcomes among patients with acute coronary syndromes. Am J Cardiol. 2005;96:1200–1206. [DOI] [PubMed] [Google Scholar]
- 6. Mehran R, Pocock SJ, Stone GW, Clayton TC, Dangas GD, Feit F, Manoukian SV, Nikolsky E, Lansky AJ, Kirtane A, White HD, Colombo A, Ware JH, Moses JW, Ohman EM. Associations of major bleeding and myocardial infarction with the incidence and timing of mortality in patients presenting with non‐ST‐elevation acute coronary syndromes: a risk model from the ACUITY trial. Eur Heart J. 2009;30:1457–1466. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Manoukian SV, Feit F, Mehran R, Voeltz MD, Ebrahimi R, Hamon M, Dangas GD, Lincoff AM, White HD, Moses JW, King SB III, Ohman EM, Stone GW. Impact of major bleeding on 30‐day mortality and clinical outcomes in patients with acute coronary syndromes: an analysis from the ACUITY Trial. J Am Coll Cardiol. 2007;49:1362–1368. [DOI] [PubMed] [Google Scholar]
- 8. Ducrocq G, Schulte PJ, Becker RC, Cannon CP, Harrington RA, Held C, Himmelmann A, Lassila R, Storey RF, Sorbets E, Wallentin L, Steg PG. Association of spontaneous and procedure‐related bleeds with short‐ and long‐term mortality after acute coronary syndromes: an analysis from the PLATO trial. EuroIntervention. 2015;11:737–745. [DOI] [PubMed] [Google Scholar]
- 9. Ducrocq G, Schulte PJ, Budaj A, Cornel JH, Held C, Himmelmann A, Husted S, Storey RF, Cannon CP, Becker RC, James SK, Katus HA, Lopes RD, Sorbets E, Wallentin L, Steg PG. Balancing the risk of spontaneous ischemic and major bleeding events in acute coronary syndromes. Am Heart J. 2017;186:91–99. [DOI] [PubMed] [Google Scholar]
- 10. Valgimigli M, Costa F, Lokhnygina Y, Clare RM, Wallentin L, Moliterno DJ, Armstrong PW, White HD, Held C, Aylward PE, Van de Werf F, Harrington RA, Mahaffey KW, Tricoci P. Trade‐off of myocardial infarction vs. bleeding types on mortality after acute coronary syndrome: lessons from the Thrombin Receptor Antagonist for Clinical Event Reduction in Acute Coronary Syndrome (TRACER) randomized trial. Eur Heart J. 2017;38:804–810. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Amsterdam EA, Wenger NK, Brindis RG, Casey DE Jr, Ganiats TG, Holmes DR Jr, Jaffe AS, Jneid H, Kelly RF, Kontos MC, Levine GN, Liebson PR, Mukherjee D, Peterson ED, Sabatine MS, Smalling RW, Zieman SJ. 2014 AHA/ACC guideline for the management of patients with non‐ST‐elevation acute coronary syndromes: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. J Am Coll Cardiol. 2014;64:e139–e228. [DOI] [PubMed] [Google Scholar]
- 12. Roffi M, Patrono C, Collet JP, Mueller C, Valgimigli M, Andreotti F, Bax JJ, Borger MA, Brotons C, Chew DP, Gencer B, Hasenfuss G, Kjeldsen K, Lancellotti P, Landmesser U, Mehilli J, Mukherjee D, Storey RF, Windecker S, Baumgartner H, Gaemperli O, Achenbach S, Agewall S, Badimon L, Baigent C, Bueno H, Bugiardini R, Carerj S, Casselman F, Cuisset T, Erol C, Fitzsimons D, Halle M, Hamm C, Hildick‐Smith D, Huber K, Iliodromitis E, James S, Lewis BS, Lip GY, Piepoli MF, Richter D, Rosemann T, Sechtem U, Steg PG, Vrints C, Luis Zamorano J. 2015 ESC Guidelines for the management of acute coronary syndromes in patients presenting without persistent ST‐segment elevation: task force for the management of acute coronary syndromes in patients presenting without persistent ST‐segment Elevation of the European Society of Cardiology (ESC). Eur Heart J. 2016;37:267–315. [DOI] [PubMed] [Google Scholar]
- 13. Granger CB, Goldberg RJ, Dabbous O, Pieper KS, Eagle KA, Cannon CP, Van De Werf F, Avezum A, Goodman SG, Flather MD, Fox KA; Global Registry of Acute Coronary Events I . Predictors of hospital mortality in the global registry of acute coronary events. Arch Intern Med. 2003;163:2345–2353. [DOI] [PubMed] [Google Scholar]
- 14. Gale CP, Manda SO, Weston CF, Birkhead JS, Batin PD, Hall AS. Evaluation of risk scores for risk stratification of acute coronary syndromes in the Myocardial Infarction National Audit Project (MINAP) database. Heart. 2009;95:221–227. [DOI] [PubMed] [Google Scholar]
- 15. Mathews R, Peterson ED, Chen AY, Wang TY, Chin CT, Fonarow GC, Cannon CP, Rumsfeld JS, Roe MT, Alexander KP. In‐hospital major bleeding during ST‐elevation and non‐ST‐elevation myocardial infarction care: derivation and validation of a model from the ACTION Registry(R)‐GWTG. Am J Cardiol. 2011;107:1136–1143. [DOI] [PubMed] [Google Scholar]
- 16. Mehran R, Pocock SJ, Nikolsky E, Clayton T, Dangas GD, Kirtane AJ, Parise H, Fahy M, Manoukian SV, Feit F, Ohman ME, Witzenbichler B, Guagliumi G, Lansky AJ, Stone GW. A risk score to predict bleeding in patients with acute coronary syndromes. J Am Coll Cardiol. 2010;55:2556–2566. [DOI] [PubMed] [Google Scholar]
- 17. Subherwal S, Bach RG, Chen AY, Gage BF, Rao SV, Newby LK, Wang TY, Gibler WB, Ohman EM, Roe MT, Pollack CV Jr, Peterson ED, Alexander KP. Baseline risk of major bleeding in non‐ST‐segment‐elevation myocardial infarction: the CRUSADE (Can Rapid risk stratification of Unstable angina patients Suppress ADverse outcomes with Early implementation of the ACC/AHA Guidelines) Bleeding Score. Circulation. 2009;119:1873–1882. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Flores‐Rios X, Couto‐Mallon D, Rodriguez‐Garrido J, Garcia‐Guimaraes M, Gargallo‐Fernandez P, Pinon‐Esteban P, Aldama‐Lopez G, Salgado‐Fernandez J, Calvino‐Santos R, Vazquez‐Gonzalez N, Castro‐Beiras A. Comparison of the performance of the CRUSADE, ACUITY‐HORIZONS, and ACTION bleeding risk scores in STEMI undergoing primary PCI: insights from a cohort of 1391 patients. Eur Heart J Acute Cardiovasc Care. 2013;2:19–26. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Abu‐Assi E, Raposeiras‐Roubin S, Lear P, Cabanas‐Grandio P, Girondo M, Rodriguez‐Cordero M, Pereira‐Lopez E, Romani SG, Gonzalez‐Cambeiro C, Alvarez‐Alvarez B, Garcia‐Acuna JM, Gonzalez‐Juanatey JR. Comparing the predictive validity of three contemporary bleeding risk scores in acute coronary syndrome. Eur Heart J Acute Cardiovasc Care. 2012;1:222–231. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Manzano‐Fernandez S, Sanchez‐Martinez M, Flores‐Blanco PJ, Lopez‐Cuenca A, Gomez‐Molina M, Pastor‐Perez FJ, Sanchez‐Galian MJ, Cambronero‐Sanchez F, Guerrero Perez E, Garcia‐Narbon A, de Las Heras‐Gomez I, Valdes M. Comparison of the global registry of acute coronary events risk score versus the can rapid risk stratification of unstable angina patients suppress adverse outcomes with early implementation of the ACC/AHA guidelines risk score to predict in‐hospital mortality and major bleeding in acute coronary syndromes. Am J Cardiol. 2016;117:1047–1054. [DOI] [PubMed] [Google Scholar]
- 21. Faustino A, Mota P, Silva J; Researchers from the National Registry of Acute Coronary Syndromes PCS . Non‐ST‐elevation acute coronary syndromes in octogenarians: applicability of the GRACE and CRUSADE scores. Rev Port Cardiol. 2014;33:617–627. [DOI] [PubMed] [Google Scholar]
- 22. Jernberg T, Attebring MF, Hambraeus K, Ivert T, James S, Jeppsson A, Lagerqvist B, Lindahl B, Stenestrand U, Wallentin L. The Swedish Web‐system for enhancement and development of evidence‐based care in heart disease evaluated according to recommended therapies (SWEDEHEART). Heart. 2010;96:1617–1621. [DOI] [PubMed] [Google Scholar]
- 23. Levey AS, Stevens LA, Schmid CH, Zhang YL, Castro AF III, Feldman HI, Kusek JW, Eggers P, Van Lente F, Greene T, Coresh J; CKD‐EPI (Chronic Kidney Disease Epidemiology Collaboration) . A new equation to estimate glomerular filtration rate. Ann Intern Med. 2009;150:604–612. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Van Buuren S, Groothius‐Oudshoorn K. Multivariate imputation by chained equations in R. J Stat Softw. 2011;45:3. [Google Scholar]
- 25. Shah AD, Bartlett JW, Carpenter J, Nicholas O, Hemingway H. Comparison of random forest and parametric imputation models for imputing missing data using MICE: a CALIBER study. Am J Epidemiol. 2014;179:764–774. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Von Hippel P. Regression with missing Ys: an improved strategy for analyzing multiply imputed data. Sociol Methodol. 2007;37:83–117. [Google Scholar]
- 27. Marshall A, Altman DG, Holder RL, Royston P. Combining estimates of interest in prognostic modelling studies after multiple imputation: current practice and guidelines. BMC Med Res Methodol. 2009;9:57. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. Harrell FE Jr. Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis, 2nd ed Cham, Switzerland: Springer International Publishing; 2015:119. [Google Scholar]
- 29. Cox DR. Two further applications of a model for binary regression. Biometrika. 1958;45:562–565. [Google Scholar]
- 30. Austin PC, Steyerberg EW. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Stat Med. 2014;33:517–535. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Austin PC, Steyerberg EW. Bootstrap confidence intervals for loess‐based calibration curves. Stat Med. 2014;33:2699–2700. [DOI] [PubMed] [Google Scholar]
- 32. Steyerberg EW, Harrell FE Jr. Prediction models need appropriate internal, internal‐external, and external validation. J Clin Epidemiol. 2016;69:245–247. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006;26:565–574. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34. Collins GS, Reitsma JB, Altman DG, Moons KG; Group T . Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. The TRIPOD Group. Circulation. 2015;131:211–219. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35. Rao SV, Chiswell K, Sun JL, Granger CB, Newby LK, Van de Werf F, White HD, Armstong PW, Califf RM, Harrington RA. International variation in the use of blood transfusion in patients with non‐ST‐segment elevation acute coronary syndromes. Am J Cardiol. 2008;101:25–29. [DOI] [PubMed] [Google Scholar]
- 36. Sherwood MW, Wang Y, Curtis JP, Peterson ED, Rao SV. Patterns and outcomes of red blood cell transfusion in patients undergoing percutaneous coronary intervention. JAMA. 2014;311:836–843. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37. Koller L, Rothgerber DJ, Sulzgruber P, El‐Hamid F, Forster S, Wojta J, Goliasch G, Maurer G, Niessner A. History of previous bleeding and C‐reactive protein improve assessment of bleeding risk in elderly patients (>/= 80 years) with myocardial infarction. Thromb Haemost. 2015;114:1085–1091. [DOI] [PubMed] [Google Scholar]
- 38. Costa F, van Klaveren D, James S, Heg D, Raber L, Feres F, Pilgrim T, Hong MK, Kim HS, Colombo A, Steg PG, Zanchin T, Palmerini T, Wallentin L, Bhatt DL, Stone GW, Windecker S, Steyerberg EW, Valgimigli M; Investigators P‐DS . Derivation and validation of the predicting bleeding complications in patients undergoing stent implantation and subsequent dual antiplatelet therapy (PRECISE‐DAPT) score: a pooled analysis of individual‐patient datasets from clinical trials. Lancet. 2017;389:1025–1034. [DOI] [PubMed] [Google Scholar]
- 39. Hagstrom E, James SK, Bertilsson M, Becker RC, Himmelmann A, Husted S, Katus HA, Steg PG, Storey RF, Siegbahn A, Wallentin L, Investigators P . Growth differentiation factor‐15 level predicts major bleeding and cardiovascular events in patients with acute coronary syndromes: results from the PLATO study. Eur Heart J. 2016;37:1325–1333. [DOI] [PubMed] [Google Scholar]
- 40. Hijazi Z, Oldgren J, Lindback J, Alexander JH, Connolly SJ, Eikelboom JW, Ezekowitz MD, Held C, Hylek EM, Lopes RD, Siegbahn A, Yusuf S, Granger CB, Wallentin L; Aristotle and Investigators R‐L . The novel biomarker‐based ABC (age, biomarkers, clinical history)‐bleeding risk score for patients with atrial fibrillation: a derivation and validation study. Lancet. 2016;387:2302–2311. [DOI] [PubMed] [Google Scholar]
- 41. Wallentin L, Hijazi Z, Andersson U, Alexander JH, De Caterina R, Hanna M, Horowitz JD, Hylek EM, Lopes RD, Asberg S, Granger CB, Siegbahn A; Investigators A . Growth differentiation factor 15, a marker of oxidative stress and inflammation, for risk assessment in patients with atrial fibrillation: insights from the Apixaban for Reduction in Stroke and Other Thromboembolic Events in Atrial Fibrillation (ARISTOTLE) trial. Circulation. 2014;130:1847–1858. [DOI] [PubMed] [Google Scholar]
- 42. Tomizawa M, Shinozaki F, Hasegawa R, Shirai Y, Motoyoshi Y, Sugiyama T, Yamamoto S, Ishige N. Elevated C‐reactive protein level predicts lower gastrointestinal tract bleeding. Biomed Rep. 2016;4:711–714. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43. Tomizawa M, Shinozaki F, Hasegawa R, Togawa A, Shirai Y, Ichiki N, Motoyoshi Y, Sugiyama T, Yamamoto S, Sueishi M. Reduced hemoglobin and increased C‐reactive protein are associated with upper gastrointestinal bleeding. World J Gastroenterol. 2014;20:1311–1317. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44. Wang CS, Sun CF. C‐reactive protein and malignancy: clinico‐pathological association and therapeutic implication. Chang Gung Med J. 2009;32:471–482. [PubMed] [Google Scholar]
- 45. Chew DP, Junbo G, Parsonage W, Kerkar P, Sulimov VA, Horsfall M, Mattchoss S; Perceived Risk of I and Bleeding Events in Acute Coronary Syndrome Patients Study I . Perceived risk of ischemic and bleeding events in acute coronary syndromes. Circ Cardiovasc Qual Outcomes. 2013;6:299–308. [DOI] [PubMed] [Google Scholar]
- 46. Schiele F, Gale CP, Bonnefoy E, Capuano F, Claeys MJ, Danchin N, Fox KA, Huber K, Iakobishvili Z, Lettino M, Quinn T, Rubini Gimenez M, Botker HE, Swahn E, Timmis A, Tubaro M, Vrints C, Walker D, Zahger D, Zeymer U, Bueno H. Quality indicators for acute myocardial infarction: a position paper of the Acute Cardiovascular Care Association. Eur Heart J Acute Cardiovasc Care. 2017;6:34–59. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data S1. Supplemental Statistical Methods (detailed).
Table S1. Previous Bleeding ICD‐10 Diagnoses
Table S2. Missing Observations
Table S3. Net Benefit for ACTION and SWEDEHEART
Table S4. Net Benefit for CRUSADE and SWEDEHEART
Table S5. C‐Index for SWEDEHEART Score without CRP
Figure S1. The full model.
Figure S2. SWEDEHEART (Swedish Web‐System for Enhancement and Development of Evidence‐Based Care in Heart Disease Evaluated According to Recommended Therapies) score equation.
Figure S3. Calibration in subgroups.
Figure S4. Score equation for SWEDEHEART (Swedish Web‐System for Enhancement and Development of Evidence‐Based Care in Heart Disease Evaluated According to Recommended Therapies) without C‐reactive protein (CRP).
Figure S5. Nomogram for SWEDEHEART (Swedish Web‐System for Enhancement and Development of Evidence‐Based Care in Heart Disease Evaluated According to Recommended Therapies) score without C‐reactive protein.
Figure S6. Calibration for SWEDEHEART (Swedish Web‐System for Enhancement and Development of Evidence‐Based Care in Heart Disease Evaluated According to Recommended Therapies) score without C‐reactive protein.