External validation of the GRACE risk score and the risk–treatment paradox in patients with acute coronary syndrome

Abstract

Objectives

To validate the Global Registry of Acute Coronary Events (GRACE) risk score and examine the extent and impact of the risk–treatment paradox in contemporary patients with acute coronary syndrome (ACS).

Methods

Data from 5015 patients with ACS enrolled in the FORCE-ACS registry between January 2015 and December 2019 were used for model validation. The performance of the GRACE risk score for predicting in-hospital and 1-year mortality was evaluated based on indices of model discrimination and calibration. Differences in the delivery of guideline-recommended care among patients who survived hospitalisation (n=4911) per GRACE risk stratum were assessed and the association with postdischarge mortality was examined.

Results

Discriminative power of the GRACE risk score was good for predicting in-hospital (c-statistic: 0.86; 95% CI: 0.83 to 0.90) and 1-year mortality (c-statistic: 0.82; 95% CI: 0.79 to 0.84). However, the GRACE risk score overestimated the absolute in-hospital and 1-year mortality risk (Hosmer-Lemeshow goodness-of-fit test p<0.01). Intermediate-risk and high-risk patients were 12% and 29% less likely to receive optimal guideline-recommended care compared with low-risk patients, respectively. Optimal guideline-recommended care was associated with lower mortality in intermediate- and high-risk patients.

Conclusions

The GRACE risk score identified patients at higher risk for in-hospital and 1-year mortality, but overestimated absolute risk levels in contemporary patients. Optimal guideline-recommended care was associated with lower mortality in intermediate-risk and high-risk patients, but was less likely to be delivered with increasing mortality risk.

Key questions.

What is already known about this subject?

• Patients with acute coronary syndrome are a heterogeneous group with a varying risk of in-hospital and long-term mortality. Clinical risk scores, especially the Global Registry of Acute Coronary Events (GRACE) risk score, play an important role in estimating these risks.

What does this study add?

• The GRACE risk score remains a useful tool to identify patients at higher risk for in-hospital and 1-year mortality, although the absolute risk in contemporary patients is overestimated.

• Patients at high risk are less likely to receive guideline-recommended care, although high-risk patients potentially benefit most from these therapies in terms of absolute risk reduction.

How might this impact on clinical practice?

• In combination with more awareness of the risk–treatment paradox, risk stratification using risk scores could lead to increased implementation of evidence-based therapies, especially in high-risk patients.

Introduction

Patients with acute coronary syndrome (ACS) are a heterogeneous group with a varying risk of in-hospital and long-term mortality.^{1 2} Clinical risk scores play an important role in estimating these risks.^{1 2} Among the available risk scores, the Global Registry of Acute Coronary Events (GRACE) risk score is most used in clinical practice. The most recent version of the GRACE risk score (V.2.0) was originally derived in 32 037 patients enrolled in the GRACE registry (14 countries, 94 hospitals) between January 2002 and December 2007 and externally validated in the FAST-MI registry.³ Current guidelines recommend the GRACE risk score for estimating prognosis and make specific recommendations based on the GRACE risk score, for example an early invasive strategy is recommended within 24 hours in patients with non-ST-segment elevation acute coronary syndrome (NSTE-ACS) with a GRACE risk score >140.¹ Given the temporal improvements in both in-hospital and long-term survival of patients with ACS, continuous validation of the GRACE risk score in contemporary patient cohorts and subgroups with specific risk profiles, for example, women or the elderly, is warranted.^{4 5} Furthermore, it is well recognised that the delivery of guideline-recommended care is inversely related to the estimated mortality risk, the so called risk–treatment paradox.^{6 7} Even though there have been encouraging signs that this treatment gap between the different risk groups is narrowing, the reasons for and consequences of the risk–treatment paradox remain poorly understood.⁸

Thus, using data from the FORCE-ACS registry, our study aimed to (1) validate the GRACE risk score by assessing its performance for predicting in-hospital and 1-year mortality, (2) assess the performance of the GRACE risk score among subgroups based on sex, age, type of ACS and bleeding risk and (3) assess the delivery and impact of guideline-recommended care among the different GRACE risk score strata.

Methods

Study design and patient population

The rationale and design of the FORCE-ACS registry have been described previously.⁹ In brief, the FORCE-ACS registry is an ongoing prospective registry of nine Dutch hospitals. The primary aim of the registry is to provide insight into different aspects of the diagnosis, management and follow-up of patients with ACS. All participating hospitals are capable of performing coronary angiography and six hospitals have on-site percutaneous coronary intervention facilities. Patient management, including the use of invasive and pharmacological therapies, in all participating hospitals is performed according to relevant guidelines.^{1 2} From 2015 onwards, all consecutive adult patients admitted for (suspected) ACS were eligible for participation. Patients were followed up by questionnaires at 1, 12, 24 and 36 month(s) after initial admission. Written informed consent was obtained from each patient. The institutional review boards of all participating centres approved the protocol of the FORCE-ACS registry. The current study complies with the principles of the Declaration of Helsinki and reports according to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis statement.¹⁰

GRACE risk score

The GRACE risk score including the predicted in-hospital and 1-year mortality risk was calculated using the web calculator available at www.outcomes-umassmed.org/grace/acs_risk2/index.html. Patients were categorised as low-risk (<109), intermediate-risk (109–140) or high-risk (>140).

Clinical endpoints

The primary endpoints were in-hospital and 1-year mortality. All deaths were reviewed by two authors (NMRvdS and JA) who had full access to the patient’s (electronic) health record. Cause of death was classified as cardiovascular or non-cardiovascular. Cause of death was assumed to be cardiovascular unless a non-cardiovascular cause could be identified. The primary endpoints were evaluated in several subgroups: (1) men versus women, (2) <75 years versus ≥75 years, (3) NSTE-ACS versus ST-segment elevation acute coronary syndrome (STE-ACS) and (4) low bleeding risk (PRECISE-DAPT score <25) versus high bleeding risk (PRECISE-DAPT score ≥25).

To assess the association between the delivery of guideline-recommended care and 1-year mortality within each GRACE risk score stratum, each patient was classified as receiving optimal or suboptimal guideline-recommended care. Optimal guideline-recommended care was defined as undergoing coronary angiography during initial hospital admission and receiving all outpatient medications with a class I guideline recommendation (ie, aspirin, a P2Y₁₂-inhibitor, a beta-blocker, an angiotensin-converting enzyme (ACE) inhibitor or angiotensin receptor blocker (ARB) and a cholesterol-lowering drug (eg, statins, ezetimibe and/or a PCSK9-inhibitor)).^{1 2} In case of concurrent oral anticoagulation use (eg, because of atrial fibrillation) aspirin or a P2Y₁₂-inhibitor on top of an oral anticoagulant was also considered as optimal guideline-recommended care in line with the most recent guidelines.^{1 2} Only patients who survived their initial admission were included in this analysis as the critical condition of non-survivors might have prevented them from receiving guideline-recommended care. A complete case sensitivity analysis was performed to assess the effect of excluding patients who died during their initial hospital admission.

Statistical analysis methods

Continuous variables are presented as median and 25th–75th IQR. Discrete variables are presented as frequencies and percentages. The Mann-Whitney U and χ² test or Fisher’s exact test were used to compare quantitative and discrete variables, respectively.

Indices of discrimination and calibration were used to assess the performance of the GRACE risk score. To assess model discrimination (ie, the ability to distinguish (future) cases from non-cases), c-statistics with corresponding 95% CI were computed from the area under the receiver operator characteristic (ROC) curve. Based on current convention, discriminative ability was categorised as poor (c-statistic <0.70), moderate (c-statistic between 0.70 and 0.80) or good (c-statistic ≥0.80).¹¹ Subgroup differences were assessed by comparing c-statistics using the DeLong method. Calibration (ie, the ability to predict the absolute level of risk) was visually assessed by plotting observed versus predicted outcomes and formally tested using the Hosmer-Lemeshow goodness-of-fit test. A p value ≥0.05 for such a test would indicate adequate calibration.

Differences in the delivery of guideline-recommended care among the different GRACE risk categories were assessed using the χ² test. Subsequently, the cumulative incidence of postdischarge 1-year mortality among patients receiving optimal and suboptimal guideline-recommended care per GRACE risk stratum was estimated using the Kaplan-Meier method and differences between groups were assessed using the log-rank test. Additionally, the characteristics of patients receiving either optimal or suboptimal guideline-recommended care in the different risk categories were compared using previously described methods. Potential predictors for receiving optimal guideline-recommended care were investigated using multivariable forward stepwise logistic regression. Potential predictors considered for inclusion in the forward stepwise models were selected based on literature, clinical judgement and availability during hospital admission. Significance was set at a p value of <0.05. Statistical analyses were performed using SPSS V.26 (SPSS, Chicago, Illinois, USA) and R studio V.3.6.1 (Vienna, Austria).

Patient and public involvement

Patients and the wider public were not involved in the design or conduct of this study.

Results

Patient characteristics

From January 2015 until December 2019, 5817 patients were included in the FORCE-ACS registry. In total, 5181 patients were diagnosed with ACS at discharge. Vital status at 1-year follow-up was not available for 39 patients and the GRACE risk score could not be calculated in 127 patients. Hence, 5015 patients (28.7% women) were included for the model validation. A detailed flowchart is provided in online supplemental figure 1.

Compared with the derivation cohort of the GRACE risk score, patients in the validation cohort were more likely to be male and to have hyperlipidaemia (table 1). A smaller proportion of the patients in the validation cohort had hypertension, diabetes mellitus, chronic kidney disease and prior myocardial infarction compared with the derivation cohort. Patients in the validation cohort more often presented with abnormal cardiac enzymes, but had markedly lower serum creatinine levels. The rate of cardiac arrest at admission was almost twofold in the validation cohort as compared with the derivation cohort. Most of the patients in the validation cohort were classified as low risk (46.6%). The rest of the patients were divided among the intermediate-risk (36.5%) and high-risk (16.9%) categories.

Table 1.

Baseline characteristics of the FORCE-ACS validation cohort and the GRACE risk score derivation cohort

	FORCE-ACS validation cohort (n=5015)	GRACE derivation cohort (n=32 037)
Demographic characteristics
Age (years)	67 (58–76)	67 (56–76)
Body mass index (kg/m2)*	26.8 (24.4–29.7)	27 (24–30)
Female sex (%)	1441 (28.7%)	33%
Current or former smokers (%)	2931 (58.4%)	57%
Hypertension (%)	2808 (56.0%)	64%
Hyperlipidaemia (%)	2851 (56.8%)	51%
Diabetes mellitus (%)	1049 (20.9%)	26%
Chronic kidney disease (%)†	155 (3.1%)	7.6%
Atrial fibrillation (%)	439 (8.8%)	7.7%
Prior stroke or TIA (%)	459 (9.2%)	8.5%
Prior MI (%)	1100 (21.9%)	30%
Prior PCI (%)	1097 (21.9%)	19%
Prior CABG (%)	453 (9.0%)	13%
Clinical presentation
Heart rate (bpm)	73 (62–87)	76 (65–90)
SBP (mm Hg)	140 (124–157)	140 (120–160)
Serum creatinine (µmol/L)	81 (69–97)	90 (80–111)
ST-segment deviation (%)	2756 (55.0%)	53%
ST-elevation	2176 (43.4%)	36%
ST-depression	2068 (41.2%)	32%
Abnormal cardiac enzymes (%)	4718 (94.1%)	52%
Cardiac arrest at admission (%)	185 (3.7%)	1.9%
Killip class (%)
I	4416 (88.1%)	85%
II	514 (10.2%)	11%
III	33 (0.7%)	3.6%
IV	52 (1.0%)	0.8%
Clinical diagnosis (%)		N/A
NSTE-ACS	2965 (59.1%)
STE-ACS	2050 (40.9%)
GRACE risk score
GRACE risk score	111 (91–131)	N/A
GRACE risk category (%)		N/A
Low-risk	2335 (46.6%)
Intermediate-risk	1830 (36.5%)
High-risk	850 (16.9%)

Values are presented as median (25th–75th IQR) or number of patients (percentage).

*Body mass index was missing in 178 cases (3.5%).

†Chronic kidney disease was defined as a glomerular filtration rate <60 mL/min/1.73 m².

CABG, coronary artery bypass grafting; GRACE, Global Registry of Acute Coronary Events; MI, myocardial infarction; NSTE-ACS, non-ST-segment elevation acute coronary syndrome; PCI, percutaneous coronary intervention; SBP, systolic blood pressure; STE-ACS, ST-segment elevation acute coronary syndrome; TIA, transient ischaemic attack.

Performance of the GRACE risk score

The observed and predicted in-hospital and 1-year mortality rates in the overall cohort and the subgroups are shown in table 2. Causes of death are specified in online supplemental table 1. A total of 104 (2.1%) patients died in-hospital, 93 (1.9%) patients due to cardiovascular causes and 11 (0.2%) due to non-cardiovascular causes. At 1 year, 271 (5.4%) patients died, 204 (4.1%) patients due to cardiovascular causes and 67 (1.3%) due to non-cardiovascular causes. The distribution of mortality among the different risk groups demonstrated a clear gradient of risk (the higher the risk at baseline, the higher the observed mortality rate).

Table 2.

Observed and predicated rates of in-hospital and 1-year mortality in the FORCE-ACS validation cohort

	n	Event rates		Calibration*		Discrimination
		Observed event rate (%)	Predicted event rate (%)	χ2	P value	C-statistic (95% CI)
In-hospital mortality	5015	2.1	3.6	41.9	<0.01	0.86 (0.83 to 0.90)
Men	3574	1.8	3.5	37.0	<0.01	0.87 (0.82 to 0.92)
Women	1441	2.8	3.7	9.8	0.28	0.85 (0.79 to 0.91)
<75 years	3586	1.1	2.6	39.5	<0.01	0.85 (0.78 to 0.91)
≥75 years	1429	4.6	5.9	8.3	0.40	0.80 (0.75 to 0.86)
NSTE-ACS	2965	1.3	2.8	26.3	<0.01	0.85 (0.80 to 0.91)
STE-ACS	2050	3.2	4.7	18.6	0.02	0.86 (0.81 to 0.91)
PRECISE-DAPT score <25	3408†	0.8	2.3	41.6	<0.01	0.79 (0.69 to 0.88)
PRECISE-DAPT score ≥25	1510†	5.0	6.4	10.6	0.23	0.80 (0.75 to 0.85)
One-year mortality	5015	5.4	9.0	95.6	<0.01	0.82 (0.79 to 0.84)
Men	3574	4.6	8.7	87.3	<0.01	0.83 (0.79 to 0.86)
Women	1441	7.4	9.8	19.1	0.01	0.80 (0.76 to 0.84)
<75 years	3586	2.5	6.1	89.4	<0.01	0.79 (0.74 to 0.84)‡
≥75 years	1429	12.7	16.3	33.5	<0.01	0.70 (0.66 to 0.74)‡
NSTE-ACS	2965	5.3	8.6	48.7	<0.01	0.82 (0.79 to 0.85)
STE-ACS	2050	5.6	9.6	50.0	<0.01	0.82 (0.78 to 0.86)
PRECISE-DAPT score <25	3408†	1.8	5.7	103.0	<0.01	0.75 (0.69 to 0.82)
PRECISE-DAPT score ≥25	1510†	13.2	16.4	24.0	<0.01	0.69 (0.65 to 0.73)

*Hosmer-Lemeshow goodness-of-fit test.

†PRECISE-DAPT score could not be calculated in 97 cases (1.9%).

‡P value <0.05, differences between the c-statistics.

NSTE-ACS, non-ST-segment elevation acute coronary syndrome; STE-ACS, ST-segment elevation acute coronary syndrome.

The discriminative ability of the GRACE risk score for predicting in-hospital and 1-year mortality was good in the validation cohort with a c-statistic of 0.86 (95% CI: 0.83 to 0.90) and 0.82 (95% CI: 0.79 to 0.84), respectively (figure 1). Discrimination for predicting 1-year mortality in hospital survivors alone was similar (c-statistic: 0.79; 95% CI: 0.76 to 0.82). There were no differences in discriminative ability between subgroups for predicting in-hospital mortality. At 1 year, several subgroups demonstrated only moderate discrimination. The GRACE risk score predicted 1-year mortality less well in patients ≥75 years compared with patients <75 years (c-statistic: 0.70 (95% CI: 0.66 to 0.74) vs 0.79 (95% CI: 0.74 to 0.84), p=0.01). Discrimination was poor in patients with high bleeding risk (c-statistic: 0.69; 95% CI: 0.65 to 0.73). ROC curves for the various subgroups are provided in online supplemental figures 2–5.

Figure 1.

Receiver operating characteristic curve of the Global Registry of Acute Coronary Events risk score for in-hospital mortality (blue) and 1-year mortality (green) in the validation cohort. AUC, area under the curve.

Calibration of the GRACE risk score to predict in-hospital and 1-year mortality was inadequate, based on visual inspection of the plotted observed and predicted event rates and according to formal statistical testing (Hosmer-Lemeshow goodness-of-fit test p<0.01) (figure 2). The GRACE risk score overestimated in-hospital mortality risk in all subgroups, except in female patients, patients ≥75 years and patients at high risk of bleeding. At 1 year, the GRACE risk score overestimated the mortality risk in all subgroups. Calibration curves for the various subgroups are provided in online supplemental figures 6–9.

Figure 2.

Calibration plot of the Global Registry of Acute Coronary Events risk score for in-hospital mortality (blue) and 1-year mortality (green) in the validation cohort. Patients were divided into deciles based on the predicted risk of mortality, each data point represents one decile. The dashed line shows absolute agreement between the observed and predicted rates.

Delivery of guideline-recommended care

A total of 104 (2.1%) patients died during their initial hospitalisation. Hence, 4911 patients were included for the evaluation of delivery of guideline-recommended care (online supplemental figure 1). Based on the GRACE risk score, 2328 (47.4%) patients were classified into the low-risk, 1811 (36.9%) intermediate-risk and 770 (15.7%) high-risk category (table 3). Patients in the intermediate-risk group were 12% less likely to receive optimal guideline-recommended care as compared with patients in the low-risk group. Even more evident, patients in the high-risk group were 29% less likely to receive optimal guideline-recommended care as compared with patients in the low-risk group. Lack of beta-blocker (29.6%) and ACE-inhibitor or ARB (26.3%) use was the most common reason for being classified into the suboptimal guideline-recommended care group, a finding present in all three risk categories. Additionally, patients in the high-risk group were frequently classified into the suboptimal guideline-recommended care group because coronary angiography was not performed (14.6%).

Table 3.

Patient management for the whole cohort and separately in the low-risk, intermediate-risk and high-risk categories

	Entire cohort (n=4911)	Low-risk category (n=2328)	Intermediate-risk category (n=1811)	High-risk category (n=772)	P value
Invasive strategies
Coronary angiography (%)	4653 (94.7%)	2290 (98.4%)	1704 (94.1%)	659 (85.4%)	<0.01
PCI	3387 (69.0%)	1670 (71.7%)	1227 (67.8%)	490 (63.5%)
CABG	596 (12.1%)	268 (11.5%)	237 (13.1%)	91 (11.8%)
Medical therapy
Dual antiplatelet therapy (%)	4062 (82.7%)	2053 (88.2%)	1467 (81.0%)	542 (70.2%)	<0.01
Aspirin	4249 (86.5%)	2151 (92.4%)	1531 (84.5%)	567 (73.4%)
P2Y12-inhibitor	4615 (94.0%)	2188 (94.0%)	1704 (94.1%)	723 (93.7%)
Oral anticoagulants (%)	820 (16.7%)	205 (8.8%)	358 (19.8%)	257 (33.3%)	<0.01
Direct oral anticoagulant	378 (7.7%)	81 (3.5%)	165 (9.1%)	132 (17.1%)
Vitamin k antagonist	414 (8.4%)	110 (4.7%)	180 (9.9%)	124 (16.1%)
Other	28 (0.6%)	14 (0.6%)	13 (0.7%)	1 (0.1%)
Beta-blockers (%)	3456 (70.4%)	1655 (71.1%)	1251 (69.1%)	550 (71.2%)	0.32
ACE-inhibitor or ARB (%)	3621 (73.7%)	1770 (76.0%)	1345 (74.2%)	506 (65.5%)	<0.01
ACE-inhibitor	2999 (61.1%)	1529 (65.7%)	1076 (59.4%)	394 (51.0%)
ARB	651 (13.3%)	254 (10.9%)	282 (15.6%)	115 (14.9%)
Cholesterol lowering-drugs (%)	4546 (92.6%)	2200 (94.5%)	1659 (91.6%)	687 (89.0%)	<0.01
Statin	4408 (89.8%)	2128 (91.4%)	1608 (88.8%)	672 (87.0%)
Ezetimibe	357 (7.3%)	189 (8.1%)	125 (6.9%)	43 (5.6%)
Other	20 (0.4%)	13 (0.6%)	6 (0.3%)	1 (0.1%)
Guideline-recommended care
Optimal guideline-recommended care (%)	2465 (50.2%)	1282 (55.1%)	882 (48.7%)	301 (39.0%)	<0.01

ACE, angiotensin-converting enzyme; ARB, angiotensin receptor blocker; CABG, coronary artery bypass grafting; PCI, percutaneous coronary intervention.

In the intermediate-risk and high-risk category, optimal guideline-recommended care was associated with a significantly lower postdischarge mortality rate as compared with suboptimal guideline-recommended care (figure 3). The postdischarge mortality rate in patients receiving optimal guideline-recommended care was 0.5%, 2.7% and 5.6% in the low-risk, intermediate-risk and high-risk category, respectively. This was higher with 1.0%, 5.0% and 13.4% in the same categories in patients receiving suboptimal guideline-recommended care. A sensitivity analysis assessing the effect of excluding patients who died in hospital revealed that including these patients would strengthen the association between optimal guideline-recommended care and reduced 1-year mortality (online supplemental figure 10).

Figure 3.

All-cause mortality according to treatment status in the low-risk, intermediate- and high-risk categories.

Importantly, patients receiving optimal or suboptimal guideline-recommended care within the same risk strata had different characteristics (table 4) and several independent predictors for receiving optimal guideline-recommended care were identified (online supplemental table 2). Optimal-treated patients were younger and presented with STE-ACS more often compared with suboptimal-treated patients in all risk categories. STE-ACS diagnosis remained an independent predictor for receiving optimal guideline-recommended care in all risk categories, while age was an independent predictor in low-risk and high-risk patients. Optimal-treated patients at low and intermediate risk had a higher heart rate compared with patients receiving suboptimal guideline-recommended care, although heart rate was only an independent predictor in low-risk patients. Optimal-treated patients at low and high risk were less often female and presented with cardiac arrest more frequent. Female sex was an independent predictor for receiving suboptimal guideline-recommended care in these risk groups. Optimal treated patients at intermediate and high risk had lower creatinine levels compared with their counterparts receiving suboptimal guideline-recommended care, but this was not an independent predictor. Conversely, high-risk optimal-treated patients were more likely to present with cardiogenic shock than their optimally treated counterparts, although this was again not an independent predictor.

Table 4.

Characteristics of patients receiving suboptimal and optimal guideline-recommended care in the low-risk, intermediate-risk and high-risk categories

	Low-risk category			Intermediate-risk category			High-risk category
	Suboptimal (n=1046)	Optimal (n=1282)	P value	Suboptimal (n=929)	Optimal (n=882)	P value	Suboptimal (n=471)	Optimal (n=301)	P value
Age (years)	59 (52-66)	57 (52-64)	<0.01	73 (68-78)	71 (66-77)	<0.01	82 (76-86)	79 (73-84)	<0.01
≥75 years (%)	46 (4.4%)	36 (2.8%)	0.04	393 (42.3%)	302 (34.2%)	<0.01	373 (79.2%)	213 (70.1%)	<0.01
Female sex (%)	278 (26.6%)	286 (22.3%)	0.02	290 (31.2%)	289 (32.8%)	0.48	174 (36.9%)	84 (27.9%)	<0.01
Current smoker (%)	394 (37.7%)	507 (39.5%)	0.35	196 (21.1%)	195 (22.1%)	0.63	53 (11.3%)	45 (15.0%)	0.14
Hypertension (%)	491 (46.9%)	652 (50.9%)	0.06	542 (58.3%)	556 (63.0%)	0.05	303 (64.3%)	203 (67.4%)	0.38
Hyperlipidaemia (%)	560 (53.5%)	700 (54.6%)	0.61	558 (60.1%)	554 (62.8%)	0.28	276 (58.6%)	160 (53.2%)	0.13
Diabetes mellitus (%)	204 (19.5%)	225 (17.6%)	0.23	215 (23.1%)	188 (21.3%)	0.33	113 (24.0%)	69 (22.9%)	0.73
Prior stroke or TIA (%)	63 (6.0%)	62 (4.8%)	0.21	114 (12.3%)	88 (10.0%)	0.12	77 (16.3%)	34 (11.3%)	0.05
Prior MI (%)	225 (21.5%)	242 (18.9%)	0.11	212 (22.8%)	178 (20.2%)	0.15	139 (29.5%)	79 (26.2%)	0.32
Prior PCI (%)	239 (22.8%)	236 (18.4%)	<0.01	231 (24.9%)	200 (22.7%)	0.25	122 (25.9%)	57 (18.9%)	0.03
Prior CABG (%)	63 (6.0%)	61 (4.8%)	0.18	123 (13.2%)	79 (9.0%)	<0.01	76 (16.1%)	37 (12.3%)	0.14
Heart rate (bpm)	68 (59–78)	71 (62–82)	<0.01	73 (60–86)	75 (63–88)	<0.01	84 (68–103)	87 (70–106)	0.25
Heart rate <60 bpm (%)	263 (25.1%)	230 (17.9%)	<0.01	197 (21.2%)	141 (16.0%)	<0.01	64 (13.6%)	32 (10.6%)	0.23
SBP (mm Hg)	143 (130–159)	146 (130–161)	0.03	140 (122-157)	140 (123-156)	0.43	129 (113-14)	130 (118-147)	0.34
SBP <110 mm Hg (%)	41 (3.9%)	50 (3.9%)	0.97	101 (10.9%)	78 (8.8%)	0.15	87 (18.5%)	41 (13.6%)	0.08
Serum creatinine (µmol/L)	77 (67–90)	77 (67–88)	0.46	86 (72–101)	83 (70–99)	0.02	97 (79–124)	91 (75–110)	<0.01
eGFR <60 mL/min (%)*	69 (6.6%)	65 (5.1%)	0.11	199 (21.4%)	176 (20.0%)	0.44	213 (45.2%)	87 (28.9%)	<0.01
eGFR <30 mL/min (%)*	6 (0.6%)	5 (0.4%)	0.52	21 (2.3%)	6 (0.7%)	<0.01	40 (8.5%)	11 (3.7%)	<0.01
STE-ACS diagnosis (%)	305 (29.2%)	541 (42.2%)	<0.01	326 (35.1%)	475 (53.9%)	<0.01	169 (35.9%)	169 (56.1%)	<0.01
Cardiac arrest at admission (%)	0 (0.0%)	8 (0.6%)	0.01	19 (2.0%)	29 (3.3%)	0.1	55 (11.7%)	58 (19.3%)	<0.01
Cardiogenic shock (%)	0 (0.0%)	0 (0.0%)	N/A	2 (0.2%)	0 (0.0%)	0.5	17 (3.6%)	22 (7.3%)	0.02
PRECISE-DAPT score ≥25 (%)†	84 (0.8%)	87 (0.7%)	0.24	376 (40.5%)	311 (35.3%)	0.02	366 (77.7%)	211 (70.1%)	0.01

Values are presented as median (25th–75th IQR) or number of patients (percentage).

*Glomerular filtration rate was calculated using the modification of diet in renal disease equation.

†PRECISE-DAPT score could not be calculated in 46, 29 and 19 cases in the low-risk, intermediate-risk and high-risk category, respectively.

CABG, coronary artery bypass grafting; eGFR, estimated glomerular filtration rate; MI, myocardial infarction; PCI, percutaneous coronary intervention; SBP, systolic blood pressure; STE-ACS, ST-segment elevation acute coronary syndrome; TIA, transient ischaemic attack.

Discussion

The most important findings of the study are as follows: (1) the GRACE risk score retained good discrimination for prediction in-hospital and 1-year mortality, (2) the GRACE risk score overestimated the absolute in-hospital and 1-year mortality risk, (3) the delivery of optimal guideline-recommended care was inversely related to the estimated mortality risk and (4) optimal guideline-recommended care was associated with lower mortality in intermediate- and high-risk patients.

Performance of the GRACE risk score

The GRACE risk score showed good discriminative power for predicting in-hospital and 1-year mortality. These results align with previous external validation studies showing moderate-to-good discrimination.^{3 12–16} However, the GRACE risk score overestimated the absolute in-hospital and 1-year mortality risk, especially among low-risk patients. There are several plausible explanations for the overestimation in our cohort.

First, over the past decades there has been a substantial decrease in mortality, most likely due to (more complete) implementation of evidence-based therapies.^{4 5 17} Previous studies have confirmed that the GRACE risk score retains its discriminative accuracy and has not been influenced by better management or clinical outcomes.¹⁶ However, the GRACE risk score fails to account for the impact of these improvements on absolute risk prediction.¹⁶

Second, elevated cardiac biomarkers serve as an important predictor in the GRACE risk score. Almost all patients in our validation cohort had abnormal cardiac enzymes in contrast to only half of all patients in the GRACE derivation cohort. Discrepancies might be explained by the utilisation of high-sensitive cardiac troponin assays compared with traditional troponin assays at the time of model development. Introduction of these high-sensitive cardiac troponin has increased the detection of patients with only minor cardiac troponin release.¹⁸ The GRACE risk score does not differentiate between different troponin concentrations. Incorporating troponin concentration as a continuous or categorical instead of dichotomous variable might improve the performance of the model, but also provides practical concerns (eg, inter-hospital variability).

Third, geographical differences play a role in absolute risk prediction. The GRACE risk score was derived from a multinational cohort (14 countries) and geographic location has previously been identified as a predictor of mortality in patients with ACS.^{19 20} Overestimation of the mortality risk in this national (ie, Dutch) cohort might be a reflection of a geographical advantage.

Performance of the GRACE risk score in subgroups

Our study assessed the performance of the GRACE risk score for predicting in-hospital and 1-year mortality in several subgroups. Although discrimination remained good in most subgroups, we observed two trends: (1) the GRACE risk score lost discriminative power in subgroups at higher risk, and (2) the GRACE risk score overestimated absolute mortality risk especially in subgroups at lower risk. There are some general and subgroup-specific explanations for these trends.

First, discrimination in higher risk patients, such as elderly or patients at high bleeding risk, might be lower due to competing risks (eg, comorbidities such as cancer, neurocognitive disorders or overall frailty) for which the GRACE risk score predicts less accurately or not at all. The absolute rate of non-cardiovascular mortality was higher in patients ≥75 years and patients at high bleeding risk, although the proportion of non-cardiovascular mortality relative to overall mortality was fairly consistent across subgroups. Of note, discrimination remained at least moderate (c-statistic ≥0.70) in all subgroups, except for predicting 1-year mortality in patients at high bleeding risk. Moreover, we used the Hosmer-Lemeshow goodness-of-fit test to assess calibration, which has a tendency towards significance for larger sample sizes. In the validation cohort, the sample size of the subgroups at lower risk (ie, men, patients <75 years and patients at low bleeding risk) was more than twofold compared with their counterparts at higher risk, resulting in greater power to detect differences in predicted versus observed mortality rates in low-risk subgroups.

Our observation that absolute risk was overestimated especially in men was consistent with the findings of Gong et al in the CANRACE registry (n=14 422).²¹ It is well recognised that there are sex-related disparities in the management and outcome of patients with ACS.²² Treatment disparities might explain part of the inadequate calibration in men, who are more likely to receive guideline-recommended interventions, including undergoing coronary angiography and revascularisation more often compared with women.²³ Gong et al argued that because treatment is not a component of the GRACE risk score, it might affect its accuracy.²¹ Interestingly, Gong et al did not find a significant interaction between sex and the individual GRACE risk score predictors.²¹

Contrary to our findings, previous studies have shown that the GRACE risk score retains good discriminative power in older patients, even for 1-year mortality.²⁴ In our study, the GRACE risk score had moderate discriminative power for predicting 1-year mortality in older patients. Previous work has suggested that adding frailty to the GRACE risk score enhances discrimination for predicting 1-year mortality in patients ≥65 years, but the practicalities of incorporating a frailty assessment should be considered.²⁵ Explanations for poor discrimination for predicting 1-year mortality in patients at high bleeding risk, might coincide with the abovementioned. Furthermore, patients at high bleeding risk are more likely to prematurely discontinue guideline-recommended therapies, such as dual antiplatelet therapy, which is not considered by the GRACE risk score and might hamper its accuracy.

Risk–treatment paradox

In patients who survived their initial hospital admission, predicted mortality risk was inversely associated with the delivery of optimal guideline-recommended care. There have been encouraging signs that the treatment gap between the different risk groups is narrowing, but our study reinforces the notion that continued attention is warranted.^{4 5 8} Importantly, current guidelines emphasise the importance of (formal) risk stratification, especially since clinicians tend to overestimate procedure-related or therapy-related risks and underestimate disease-related risks based on clinical judgement alone.1 2 26 27 In combination with more awareness of the risk–treatment paradox, risk stratification based on risk scores could lead to increased implementation of evidence-based therapies, especially in high-risk patients. High-risk patients are often older and have more comorbidities (eg, chronic kidney disease) or competing risks (eg, high bleeding risk). Therefore, clinicians might be reluctant to subject patients to invasive procedures due to the risk of procedure-related complications, such as contrast-induced kidney injury or access site bleeding. Furthermore, clinicians might be disinclined to prescribe certain recommended medications in order to prevent polypharmacy, which is linked to non-adherence, especially in older and more frail patients.²⁸

Our study demonstrated that optimal guideline-recommended care was associated with lower 1-year mortality in intermediate-risk and high-risk patients. In the British MINAP and French FAST-MI registries guideline adherence was associated with decreased 30 days and 3 years mortality in a dose–response manner.^{29 30} Of note, these studies were mostly restricted to individual quality indicators or interventions rather than a cumulative measure of guideline-recommended care. In contrast, Saar et al used a cumulative measure of guideline-recommended care in the description of the in-hospital care for 3803 Estonian patients with non-ST-elevation myocardial infarction (NSTEMI). Patients with a low estimated 1-year mortality risk according to the GRACE risk score were more likely to receive guideline-recommended in-hospital medication (defined as aspirin, P2Y₁₂-inhibitors, ACE-inhibitors or ARBs, beta-blockers, statins and parenteral anticoagulants) and underwent coronary angiography more often than patients at intermediate or high risk. Importantly, Estonia has a relatively high cardiovascular mortality and almost half of patients were classified as high risk. Of note, Saar et al reported that only 70.2% of all patients underwent coronary angiography as compared with 94.7% in our cohort. This discrepancy might be explained in part by the inclusion of patients with STE-ACS in the FORCE-ACS registry, who practically always undergo coronary angiography, and the widespread coronary angiography capabilities in Dutch hospitals. There are also indications that the delivery of guideline-recommended care is associated with positive outcomes beyond 1 year. Hall et al showed that the differences in mortality between optimal and suboptimal-treated patients with NSTEMI increases over time and lasted up to 8 years, irrespective of risk category.^{7 31} Notably, Hall et al focused on patients with NSTEMI. The present study suggests that this effect was present across the ACS spectrum at least in the first year. Follow-up beyond 1 year is necessary to study the long-term effect of optimal guideline-recommended care.

It should be emphasised that there are important differences between patients receiving optimal and suboptimal guideline-recommended care even within risk strata. Differences in factors such as age and clinical presentation can significantly confound the association between guideline-recommended care and mortality. Furthermore, our results suggest that there might be sex disparities at play. Women were roughly 25% more likely to be classified as high risk, while women at low and high risk were less likely to receive optimal guideline-recommended care compared with their male counterparts even after considering relevant confounders. Moreover, there may be other unmeasured confounders (eg, frailty or medication compliance). Most studies do not report or correct for these differences and might therefore overestimate the potential of optimal guideline-recommended care.

Limitations

This study has several important limitations. First, 0.8% of patients were lost to follow-up before 1 year and the GRACE risk score could not be calculated in 2.5% of patients, because of missing values. In theory, this could have resulted in selection bias. However, this group only forms a small proportion of the total study population. Second, the association between guideline-recommended care and mortality as seen in this cohort study does not imply causality, especially since there were important differences between optimal-treated and suboptimal-treated patients within risk strata. Furthermore, drug-specific contraindications, patient refusals or allergies were not taken into account. Lastly, our study was conducted in the Netherlands and, as mentioned before, geographical differences in treatment and outcomes might have influenced the study results.

Conclusions

The GRACE risk score identified patients at higher risk for in-hospital and 1-year mortality, but overestimated absolute risk levels in contemporary patients. Optimal guideline-recommended care was associated with lower mortality in intermediate- and high-risk patients, but was less likely to be delivered with increasing predicted mortality risk.

Data availability statement

Data are available upon reasonable request.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

This study involves human participants and was approved by MEC-U (ref: ClinicalTrials.gov Identifier: NCT03823547) Participants gave informed consent to participate in the study before taking part.