External Validation of Eight Ruptured Abdominal Aortic Aneurysm Mortality Prediction Models Demonstrates Limited Predictive Accuracy

Abstract

Objective:

Over a dozen ruptured abdominal aortic aneurysm (rAAA) mortality risk prediction models currently exist; however, lack of external validation limits their applicability. This study aimed to evaluate the accuracy of eight common rAAA mortality risk prediction models in a large, contemporary, external validation cohort.

Methods:

A retrospective review of rAAA repairs at a multicentre integrated regional healthcare system with large central quaternary referral facility (2010 – 2020) was performed. Eight models were used to predict 30 day post-operative death, including the Updated Glasgow Aneurysm Score (GAS), Vascular Study Group of New England rAAA Risk Score, Harborview Pre-operative rAAA Risk Score, Modified Harborview Risk Score, Vancouver Scoring System (VSS), Artificial Neural Network Score, Dutch Aneurysm Score, and Edinburgh Ruptured Aneurysm Score. The models were assessed for discrimination, calibration, and clinical utility using receiver operating characteristic curves (area under the curve [AUC]), Hosmer–Lemeshow χ² test, Brier scores, and decision curve analysis. The proportion of unexpected survivors (survival despite > 80% predicted 30 day death) to expected deaths was compared across calculators, and both groups were compared using the model demonstrating the highest unexpected survival frequency.

Results:

Three hundred and fifteen rAAA repairs were included (mean age 73.6 ± 10.0 years; 72.1% male; 49.8% open repair) with a 30 day mortality rate of 32.1%. Three models had fair discrimination (AUC ≥ 0.70), with GAS having the highest AUC (0.74, 95% confidence interval 0.68 – 0.79). All models demonstrated poor to adequate calibration. Using VSS, unexpected survivors (n = 25) had less pre-operative shock (72% vs. 96%; p = .050) and statistically significantly less coagulopathy (median international normalised ratio 1.2 [interquartile range 1.1, 1.5] vs. 1.8 [1.3, 2.2]; p = .015) compared with expected deaths (n = 23).

Conclusion:

Current rAAA risk prediction models demonstrated only fair discrimination and poor to adequate calibration. These findings suggest that existing risk prediction models have not sufficiently captured important physiological characteristics associated with rAAA death and should be applied cautiously to clinical practice.

INTRODUCTION

Ruptured abdominal aortic aneurysm (rAAA) is a highly fatal surgical emergency, with an estimated post-operative mortality rate of 20 – 40%.^1,2 Only half of patients survive initial rupture to undergo surgical repair.^3,4 Regionalisation of rAAA to high volume centres and adoption of minimally invasive endovascular aortic aneurysm repair (EVAR) have reduced morbidity and mortality rates; however, early post-operative mortality > 30% and high rates of major post-operative complications make appropriate patient selection for repair challenging.^5,6 Several post-operative rAAA repair mortality risk prediction models have been developed to guide clinical decisions.^7–15 Few models have undergone external validation, and practical applicability for pre-operative risk prediction has been impaired by limitations, including development prior to widespread EVAR adoption,^7,8,15 the inclusion of intra-operative data,⁹ or inconsistent accuracy for the most critically ill patients.^7,8,16–18

This study aimed to comprehensively evaluate the accuracy and clinical utility of eight rAAA mortality risk prediction models in a large, contemporary patient cohort representative of a major North American healthcare system.

MATERIALS AND METHODS

Study design

This retrospective cohort study was exempt from informed consent by the University of Pittsburgh Human Research Protection Office (STUDY#19070316). Reporting followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines.¹⁹ Patients were identified using both a retrospective review of electronic health records and a prospectively maintained database. All consecutive adults (age > 18 years) undergoing rAAA repair at an integrated regional healthcare system with a large central quaternary referral facility staffed by a single hospital employed vascular group (2010 – 2020) were included. Patients either presented directly to the hospital of definitive repair or were transferred from an outside hospital (OSH) within or outside the abovementioned healthcare system. Patients having had a previous aortic repair, mycotic aneurysm, or trauma associated rAAA were excluded due to the increased complexity of the repair and decision making process or because they represented a separate disease process. Patients without complete data for calculating any risk model were excluded (Supplementary Table S1). To address any potential selection bias, a sensitivity analysis was performed to evaluate the models, separately excluding cases with missing variables required for each model.

Patient demographics, comorbid conditions, vital signs, laboratory values, repair type, and post-operative outcomes were collected. Pre-operative shock was defined as systolic blood pressure (SBP) < 90 mmHg, heart rate ≥ 120 beats/minute, or cardiopulmonary arrest pre-operatively. While the primary outcome of the investigated score models was either in hospital or 30 day death (Table 1), 30 day death was selected as the primary outcome because it is more commonly reported in contemporary literature and less affected by variability in hospital length of stay. Mortality data were determined via the Social Security Death Index and electronic health record death documentation.^20,21

Table 1.

Ruptured abdominal aortic aneurysm (rAAA) scoring system models and mortality prediction calculations.

Scoring system	Primary outcome	Mortality prediction model	Predicted mortality calculation/interpretation
Vascular Study Group of New England rAAA Risk Score (VSGNE)9	In hospital death	Score from 1–6 based on the presence of the following preoperative risk factors: age >76 years (2 points), cardiac arrest (2 points), loss of consciousness (1 point), suprarenal clamp* (1 point)	Scores of 0, 1, 2, 3, 4, and ≥5 correspond to mortality risk of 8%, 25%, 37%, 60%, 80%, and 87%, respectively
Updated Glasgow Aneurysm Score (GAS)8	30 day death	Updated GAS = age + 7 for open repair + 17 for shock† + 7 for myocardial disease‡ + 10 for cerebrovascular disease§ +14 for renal insufficiency‖	Calculate updated GAS score Calculate linear predictor = −5.3 + 0.052 × updated GAS Calculate predicted 30 day mortality = 1–(1/[1+exp{linear predictor}])
Vancouver Scoring System (VSS)12	In hospital 30 day death	VSS = −3.44 + 0.062 × age in years + 1.14 × loss of consciousness (yes = 1; no = −1) + 0.6 × cardiac arrest (yes = 1; no = −1)	Calculate VSS Calculate mortality risk = eVSS/(1+ eVSS)
Artificial Neural Network Score (ANN)13	In hospital death	Score from 0–4 based on the presence of the following preoperative risk factors (1 point per risk factor): age ≥70 years, CPR, loss of consciousness, shock†	Scores of 0, 1, 2, 3, and 4 correspond to mortality risk of 11%, 16%, 44%, 76%, and 89%, respectively
Dutch Aneurysm Score (DAS)14	30 day death	DAS = age in years × 0.074 + SBP/10 × −0.12 + 1 for CPR + (haemoglobin [mmol/L]/10)3¶ × −1.27	Calculate DAS Calculate ln(odds) with intercept = −4.73 + DAS Calculate predicted 30 day mortality = exp(ln[odds])/(1+exp[ln{odds}])
Edinburgh Ruptured Aneurysm Score (ERAS)15	30 day death	Score from 0–3 based on the presence of the following pre-operative risk factors (1 point per risk factor): GCS <15, SBP <90 mmHg, haemoglobin level <9 g/dL	Scores of ≥1, 2, and 3 correspond to mortality risk of 29%, 50%, and 80%, respectively
Harborview Pre-operative rAAA Risk Score (HRS)10	30 day death	Score from 0–4 based on the presence of the following pre-operative risk factors (1 point per risk factor): age >76 years, SBP ever <70 mmHg, creatinine >2.0 mg/dL, pH <7.2	Scores of 1, 2, 3, and 4 correspond to mortality risk of 22%, 70%, 80%, and 100%, respectively
Modified Harborview Risk Score (mHRS)11	30 day death	Score from 0–4 based on the presence of the following pre-operative risk factors (1 point per risk factor): age >76 years, SBP ever <70 mmHg, creatinine >2.0 mg/dL, INR >1.8	Scores of 0, 1, 2, 3, and 4 correspond to mortality risk of 17%, 43%, 54%, 84%, and 100%, respectively

rAAA = ruptured abdominal aortic aneurysm; CPR = cardiopulmonary resuscitation; SBP = systolic blood pressure; GCS = Glasgow Coma Scale; INR = international normalised ratio.

^*Patients undergoing endovascular repair were assigned a score of zero.

^†Systolic blood pressure <90 mmHg, heart rate ≥120 beats/minute, or cardiopulmonary arrest.

^‡Previous myocardial infarction and or ongoing angina.

^§Prior stroke or transient ischaemic attack.

^‖Serum creatinine >1.8 mg/dL, creatinine clearance <50 mL/min, or history of acute or chronic renal failure.

^¶To calculate haemoglobin in g/dL to mmol/L, multiply by 0.1551.

Ruptured abdominal aortic aneurysm risk prediction scoring systems

The estimated 30 day mortality rate following repair was calculated employing the following eight scoring systems: Updated Glasgow Aneurysm Score (GAS);⁸ Vascular Study Group of New England rAAA Risk Score (VSGNE);⁹ Harborview Pre-operative rAAA Score (HRS);¹⁰ Modified Harborview Risk Score (mHRS);¹¹ Vancouver Scoring System (VSS);¹² Artificial Neural Network Score (ANN);¹³ Dutch Aneurysm Score (DAS);¹⁴ and Edinburgh Ruptured Aneurysm Score (ERAS)¹⁵ (Table 1). Models were selected based on their frequent citation in the literature and alignment with the available data, minimising exclusions due to missing variables. Several other models were reviewed but excluded due to specific limitations (Supplementary Table S2).

Each prediction calculator generates the mortality risk either by applying multivariable logistic regression coefficients (GAS, VSS, DAS)^8,12,14 or through summation of unweighted binary variables and assigning the point scores to corresponding mortality risk (VSGNE, HRS, mHRS, ANN, ERAS).^9–11,13,15 In the published HRS model, patients scoring zero were not assigned a specific mortality rate, whereas those scoring one corresponded to a mortality rate of 22%. Therefore, in the current analysis, patients scoring zero were assigned to a mortality rate of < 22%.¹⁰

For each scoring system, the number of unexpected survivors (survival despite > 80% predicted mortality rate) and expected deaths (death within 30 days with > 80% predicted mortality rate) was quantified and both groups were compared.

Statistical analysis

Categorical variables were presented as frequencies with percentages and were compared with Fisher’s exact test. Continuous variables were presented as mean ± standard deviation or median with interquartile range and were compared with t test or Wilcoxon rank sum test, depending on their distribution. Statistical significance was determined by p < .050.

Each risk calculator was evaluated for discrimination, calibration, predicted accuracy, and net clinical benefit. Discrimination evaluates the model’s ability to distinguish between outcomes, while calibration measures how closely predicted probabilities align with observed outcomes. Well established and widely applied tests were used to evaluate these metrics. Discrimination was quantified with receiver operating characteristics curves (fair discrimination: area under the curve [AUC] ≥ 0.70).²² Pairwise tests of AUC equality were evaluated using DeLong’s test.²³ The Hosmer–Lemeshow χ² test determined the scoring systems’ calibration (adequate calibration: p > .050). The Brier score, a measure of both calibration and discrimination,²⁴ was also calculated as an adjunct to evaluate predictive accuracy. Additionally, decision curve analysis was performed to evaluate the net clinical benefit of the scoring models with fair discrimination.^25–27 Stata SE 18.0 (StataCorp, College Station, TX, USA) was used for all statistical analyses.

Subgroup analysis

To examine the robustness of scoring systems across repair type and transfer status, the scoring systems’ performance was explored in four subgroups: (i) patients transferred from OSH; (ii) patients presenting to the hospital where repair was performed; (iii) patients undergoing open surgical repair (OSR); and (iv) patients undergoing EVAR.

RESULTS

Patient characteristics

Of 427 consecutive adults undergoing rAAA repair (2010 – 2020), 67 were excluded due to prior aortic repair, mycotic aneurysm, or trauma associated rAAA, and 45 due to missing data. The final cohort included 315 patients (mean age 73.6 ± 10.0 years), with the majority being male (72.1%; n = 227) and identified as White race (79.4%; n = 250). A significant proportion of patients had evidence of pre-operative shock (39.4%; n = 124), and the majority (66.0%; n = 208) were transferred from OSH. The rAAA repair type was equally distributed (OSR 49.8%; n = 157), with 28.9% (n = 91) in hospital and 32.1% (n = 101) 30 day mortality rates. Patient demographics, medical comorbidities, and pre-operative characteristics are summarised in Table 2.

Table 2.

Demographics, medical comorbidities, pre-operative characteristics, and outcomes for patients (n = 315) included in the analysis of ruptured abdominal aortic aneurysm (rAAA) mortality predictive models.

Characteristic	rAAA repairs (n = 315)	Death within 30 days PO (n = 101)	Survival beyond 30 days PO (n = 214)	p value*
Patient demographics
Age – y	73.6 ± 10.0	77 ± 9.0	71.8 ± 9.9	<.001
BMI – kg/m2	26.8 (23.4, 30.9)	25.9 (23.4, 30.9)	27.2 (23.4, 30.9)	.44
Sex				.080
Female	88 (27.9)	35 (34.7)	53 (24.8)
Male	227 (72.1)	66 (65.3)	161 (75.2)
Ethnicity				.38
African American	8 (2.5)	2 (2.0)	6 (2.8)
American Indian	1 (0.3)	1 (1.0)	0 (0.0)
Asian	3 (1.0)	1 (1.0)	2 (0.9)
White	250 (79.4)	76 (75.2)	174 (81.3)
Declined	18 (5.7)	8 (7.9)	10 (4.7)
Not specified	35 (11.1)	13 (12.9)	22 (10.3)
Hispanic/Latino or not				.36
Hispanic or Latino	0 (0.0)	0 (0.0)	0 (0.0)
Non-Hispanic or Latino	256 (81.3)	79 (78.2)	177 (82.7)
Unknown	59 (18.7)	22 (21.8)	37 (17.3)
Medical comorbidities
Coronary artery disease	116 (36.8)	38 (37.6)	78 (36.4)	.90
Cerebrovascular disease	32 (10.2)	14 (13.9)	18 (8.4)	.16
Renal insufficiency†	34 (10.8)	14 (13.9)	20 (9.3)	.25
Pre-operative characteristics
Loss of consciousness	63 (20.0)	28 (27.7)	35 (16.4)	.023
GCS <15	79 (25.1)	40 (39.6)	39 (18.2)	<.001
Cardiac arrest	16 (5.1)	10 (9.9)	6 (2.8)	.012
Shock‡	124 (39.4)	55 (54.5)	69 (32.2)	<.001
SBP <90 mmHg	76 (24.1)	29 (28.7)	47 (22.0)	.21
SBP <70 mmHg	33 (10.5)	8 (7.9)	25 (11.7)	.43
Pre-operative laboratory values
Haemoglobin – g/dL	10.9 (9.2, 12.5)	10.3 (8.8, 11.6)	11.1 (9.4, 12.5)	.009
Creatinine – mg/dL	1.3 (1.0, 1.7)	1.4 (1.1, 1.9)	1.2 (1.0, 1.6)	.007
Blood pH <7.2	66 (21.0)	36 (35.6)	30 (14.0)	<.001
INR	1.3 (1.1, 1.6)	1.5 (1.2, 1.9)	1.3 (1.1, 1.5)	<.001
Transfer status				.90
Transferred from OSH	208 (66.0)	66 (65.3)	142 (66.4)
Presented directly to HDR	107 (34.0)	35 (34.7)	72 (33.6)
Repair type				<.001
Open repair	157 (49.8)	65 (64.4)	92 (43.0)
Endovascular repair	158 (50.2)	36 (35.6)	122 (57.0)
Death in hospital	91 (28.9)	90 (89.1)	1 (0.5)	<.001

Data are presented as mean ± standard deviation, median (interquartile range), or n (%). rAAA = ruptured abdominal aortic aneurysm; PO = post-operative; BMI = body mass index; GCS = Glasgow Coma Scale; SBP = systolic blood pressure; INR = international normalised ratio; OSH = outside hospital; HDR = hospital of definitive repair.

^*p<.050 represents a statistically significant difference between patients who died within 30 days and patients who survived beyond 30 days post-operatively.

^†Serum creatinine >1.8 mg/dL, creatinine clearance <50 mL/min, or history of acute or chronic renal failure.

^‡Systolic blood pressure <90 mmHg, heart rate ≥120 beats/minute, or cardiopulmonary arrest.

Scoring system predictive accuracy

All scoring systems demonstrated poor to fair discrimination²² for 30 day death (Fig. 1). Only GAS (AUC 0.74, 95% confidence interval [CI] 0.68 – 0.79), VSGNE (AUC 0.73, 95% CI 0.68 – 0.79), and HRS (AUC 0.70, 95% CI 0.65 – 0.75) demonstrated fair discrimination (Table 3), with no statistically significant differences on pairwise comparison. Notably, ERAS demonstrated a statistically significant difference from all other models except for DAS (Fig. 2). The mHRS, VSS, ANN, DAS, and ERAS models had poor discrimination. All scoring systems had adequate calibration except for HRS (p < .001) and mHRS (p = .007) (Table 3). In the sensitivity analysis separately excluding cases with missing variables required for each model (Supplementary Table S3), the models exhibited comparable performance compared with the primary analysis (Table 3).

Figure 1.

Receiver operating characteristic (ROC) curves for all analysed ruptured abdominal aortic aneurysm (rAAA) mortality prediction scoring systems. Performance comparison with a reference ROC of 0.5 is depicted. GAS, VSGNE, and HRS demonstrated fair discrimination (area under the curve ≥ 0.70). Full data are available in Table 3. GAS = Updated Glasgow Aneurysm Score; VSGNE = Vascular Study Group of New England rAAA Risk Score; HRS = Harborview Pre-operative rAAA Risk Score; mHRS = Modified Harborview Risk Score; VSS = Vancouver Scoring System; ANN = Artificial Neural Network Score; DAS = Dutch Aneurysm Score; ERAS = Edinburgh Ruptured Aneurysm Score.

Table 3.

Discrimination, calibration, and Brier score of the analysed ruptured abdominal aortic aneurysm (rAAA) risk prediction models.

Predictive model	Discrimination: AUC (95% CI)*	Calibration: Hosmere–Lemeshow test†	Discrimination and calibration: Brier score‡
Updated Glasgow Aneurysm Score (GAS)	0.74 (0.68–0.79)	.30	0.19
Vascular Study Group of New England rAAA Risk Score (VSGNE)	0.73 (0.68–0.79)	.70	0.19
Harborview Pre-operative rAAA Risk Score (HRS)	0.70 (0.65–0.75)	<.001	0.22
Modified Harborview Risk Score (mHRS)	0.69 (0.63–0.75)	.007	0.20
Vancouver Scoring System (VSS)	0.69 (0.63–0.75)	.55	0.23
Artificial Neural Network Score (ANN)	0.68 (0.62–0.74)	.17	0.21
Dutch Aneurysm Score (DAS)	0.63 (0.57–0.70)	.42	0.24
Edinburgh Ruptured Aneurysm Score (ERAS)	0.58 (0.53–0.63)	.32	0.21

AUC = area under the curve; CI = confidence interval; rAAA = ruptured abdominal aortic aneurysm.

^*AUC ≥0.7 represents fair discrimination.

^†High p value (>.050) suggests that there is no statistically significant difference between observed and predicted probabilities, indicating good calibration of the model.

^‡The Brier score ranges from 0 – 1, with a lower score indicating better performance.

Figure 2.

Pairwise comparisons of area under the curve (AUC) of the ruptured abdominal aortic aneurysm (rAAA) mortality risk prediction models. The three models demonstrating fair discrimination with an AUC ≥ 0.70 (GAS, VSGNE, and HRS) showed no statistically significant difference on pairwise comparison. The ERAS model, which had the lowest AUC in the analysis, demonstrated a statistically significant difference from all other models except for DAS. GAS = Updated Glasgow Aneurysm Score; VSGNE = Vascular Study Group of New England rAAA Risk Score; HRS = Harborview Pre-operative rAAA Risk Score; mHRS = Modified Harborview Risk Score; VSS = Vancouver Scoring System; ANN = Artificial Neural Network Score; DAS = Dutch Aneurysm Score; ERAS = Edinburgh Ruptured Aneurysm Score.

Using decision curve analysis, both GAS and VSGNE demonstrated superior net clinical benefit across a broader spectrum of threshold probabilities compared with HRS, a result consistent with their AUCs and Brier scores (Fig. 3).

Figure 3.

Decision curve analysis of the ruptured abdominal aortic aneurysm (rAAA) mortality prediction scoring systems demonstrating fair discrimination. Decision curve analysis is a statistical method used to evaluate models or tests by considering the clinical consequences of false positive or false negative results. The y axis represents the net benefit, while the blue and red lines represent the clinical strategies of intervention for all and intervention for none, respectively. The threshold probability reflects the point at which a patient (or decision maker) would opt for treatment, and this can significantly differ based on their goals of care.²⁶ However, it is assumed that it would be clinically reasonable for most patients to seek intervention when the mortality risk is < 20% and to opt against any intervention when the mortality risk is > 80%. Therefore, the net clinical benefit of the three models with fair discrimination (GAS, VSGNE, and HRS) was compared. Both GAS and VSGNE showed a positive net benefit over intervention for all and intervention for none clinical strategies across a broader range of threshold probabilities in comparison with HRS, a result consistent with their receiver operating characteristic curve analyses and Brier scores. GAS = Updated Glasgow Aneurysm Score; VSGNE = Vascular Study Group of New England rAAA Risk Score; HRS = Harborview Pre-operative rAAA Risk Score.

Unexpected survival analysis

The unexpected survival analysis was conducted across all models; however, only VSS had a sufficiently large proportion of unexpected survivors for adequate analysis (Fig. 4). In the VSS cohort, unexpected survivors (n = 25; 7.9%) had higher median body mass index (28.4 kg/m² vs. 24.7 kg/m²; p = .051), a lower rate of pre-operative shock (72% vs. 96%; p = .050), and statistically significantly less coagulopathy (median international normalised ratio [INR] 1.2 vs. 1.8; p = .015) compared with expected deaths (n = 23; 7.3%) (Table 4).

Figure 4.

Distribution of unexpected survival and expected death among all the ruptured abdominal aortic aneurysm (rAAA) mortality risk prediction models. Unexpected survival was defined as survival despite > 80% predicted death, while expected death was defined as death within 30 days with > 80% predicted death. The VSS demonstrated the highest frequency of unexpected survival in comparison with all other models, which showed seven cases or fewer. GAS = Updated Glasgow Aneurysm Score; VSGNE = Vascular Study Group of New England rAAA Risk Score; HRS = Harborview Pre-operative rAAA Risk Score; mHRS = Modified Harborview Risk Score; VSS = Vancouver Scoring System; ANN = Artificial Neural Network Score; DAS = Dutch Aneurysm Score; ERAS = Edinburgh Ruptured Aneurysm Score.

Table 4.

Characteristics of unexpected survivors (n = 25) compared with expected deaths (n = 23) within the Vancouver Scoring System (VSS) cohort.

Characteristic	Expected deceased (n = 23)	Unexpected survivors (n = 25)	p value*
Patients demographics
Age – y	76.0 ± 6.8	77.4 ± 6.7	.47
BMI – kg/m2	24.7 (20.3, 29.7)	28.4 (24.2, 32.7)	.051
Sex			1.0
Female	6 (26)	7 (28)
Male	17 (74)	18 (72)
Ethnicity			.91
African American	1 (4)	0 (0)
White	19 (83)	21 (84)
Declined	1 (4)	1 (4)
Not specified	2 (9)	3 (12)
Hispanic/Latino or not			.61
Hispanic or Latino	0 (0)	0 (0)
Non-Hispanic or Latino	22 (96)	22 (88)
Unknown	1 (4)	3 (12)
Medical comorbidities
Coronary artery disease	9 (39)	10 (40)	1.0
Cerebrovascular disease	4 (17)	4 (16)	1.0
Renal insufficiency†	2 (9)	3 (12)	1.0
Pre-operative characteristics
Loss of consciousness	23 (100)	25 (100)	e
GCS <15	21 (91)	20 (80)	.42
Cardiac arrest	10 (43)	6 (24)	.22
Shock‡	22 (96)	18 (72)	.050
SBP <90 mmHg	8 (35)	9 (36)	1.0
SBP <70 mmHg	2 (9)	6 (24)	.25
Pre-operative laboratory values
Haemoglobin – g/dL	9.3 (8.1, 11.1)	10.7 (9.3, 12.4)	.059
Creatinine – mg/dL	1.3 (1.1, 1.6)	1.4 (1.2, 1.7)	.59
Blood pH <7.2	11 (48)	10 (40)	.77
INR	1.8 (1.3, 2.2)	1.2 (1.1, 1.5)	.015
Transfer status			.78
Transferred from OSH	13 (57)	13 (52)
Presented directly to HDR	10 (43)	12 (48)
Repair type			.15
Open repair	14 (61)	9 (36)
Endovascular repair	9 (39)	16 (64)
Death in hospital	20 (87)	0 (0)	<.001

Data are presented as mean ± standard deviation, median (interquartile range), or n (%). BMI = body mass index; GCS = Glasgow Coma Scale; SBP = systolic blood pressure; INR = international normalised ratio; OSH = outside hospital; HDR = hospital of definitive repair.

^*p<.050 represents a statistically significant difference between both groups.

^†Serum creatinine >1.8 mg/dL, creatinine clearance <50 mL/min, or history of acute or chronic renal failure.

^‡Systolic blood pressure <90 mmHg, heart rate ≥120 beats/minute, or cardiopulmonary arrest.

Subgroup analysis

For patients transferred from OSH, GAS (AUC 0.74, 95% CI 0.67 – 0.81), VSGNE (AUC 0.73, 95% CI 0.66 – 0.80), and HRS (AUC 0.71, 95% CI 0.64 – 0.77) had fair discrimination with no statistically significant differences in their AUCs on pairwise comparisons. For patients presenting to the hospital of definitive repair, VSGNE (AUC 0.74, 95% CI 0.64 – 0.84), GAS (AUC 0.71, 95% CI 0.61 – 0.82), and mHRS (AUC 0.70, 95% CI 0.60 – 0.80) demonstrated fair discrimination with no statistically significant difference in their AUCs (Table 5).

Table 5.

Subgroup analysis evaluating calibration and discrimination of ruptured abdominal aortic aneurysm (rAAA) scoring systems across patient transfer status.

Predictive model	Discrimination: AUC (95% CI)*		Calibration: Hosmere–Lemeshow test†		Brier score‡
	Transferred from OSH	Presenting to HDR	Transferred from OSH	Presenting to HDR	Transferred from OSH	Presenting to HDR
Updated Glasgow Aneurysm Score (GAS)	0.74 (0.67–0.81)	0.71 (0.61–0.82)	.25	.63	0.19	0.19
Vascular Study Group of New England rAAA Risk Score (VSGNE)	0.73 (0.66–0.80)	0.74 (0.64–0.84)	.034	.75	0.19	0.19
Harborview Pre-operative rAAA Risk Score (HRS)	0.71 (0.64–0.77)	0.69 (0.59–0.78)	<.001	.017	0.22	0.22
Modified Harborview Risk Score (mHRS)	0.68 (0.62–0.75)	0.70 (0.60–0.80)	.001	.70	0.20	0.19
Vancouver Scoring System (VSS)	0.69 (0.61–0.76)	0.70 (0.59–0.80)	.68	.53	0.23	0.25
Artificial Neural Network Score (ANN)	0.67 (0.60–0.75)	0.69 (0.60–0.79)	.36	.21	0.21	0.21
Dutch Aneurysm Score (DAS)	0.63 (0.55–0.71)	0.64 (0.52–0.76)	.44	.38	0.24	0.24
Edinburgh Ruptured Aneurysm Score (ERAS)	0.58 (0.52–0.64)	0.60 (0.51–0.69)	.25	.87	0.21	0.21

AUC = area under the curve; CI = confidence interval; OSH = outside hospital; HDR = hospital of definitive repair; rAAA = ruptured abdominal aortic aneurysm.

^*AUC ≥0.70 represents fair discrimination.

^†p>.050 suggests that there is no statistically significant difference between observed and predicted probabilities, indicating good model calibration.

^‡The Brier score ranges from 0 – 1, with a lower score indicating better performance.

All scoring systems demonstrated adequate calibration for transferred patients, except for VSGNE (p = .034), mHRS (p = .001), and HRS (p < .001), while for patients presenting to the hospital of definitive repair, all models had adequate calibration except for HRS (p = .017). Brier scores for all models, except for mHRS and VSS, were equal in both groups, indicating comparable predictive accuracy (Table 5).

The subgroup analysis for repair type showed that AUCs for OSR were higher compared with EVAR in all models except for ANN (0.66 vs. 0.70; p = .56) and ERAS (0.58 vs. 0.60; p = .78). The mHRS had the highest AUC (0.76, 95% CI 0.68 – 0.83) for OSR, whereas GAS demonstrated the highest AUC (0.71, 95% CI 0.61 – 0.81) for EVAR. The AUCs among models with fair discrimination for OSR (mHRS, HRS, GAS, VSGNE, and VSS) ranged from 0.72 – 0.76, with no statistically significant differences on pairwise comparisons. The HRS and mHRS were the only models with inadequate calibration for both OSR (HRS, p < .001; mHRS, p = .001) and EVAR (HRS, p = .007; mHRS, p = .032) (Table 6).

Table 6.

Subgroup analysis evaluating calibration and discrimination of ruptured abdominal aortic aneurysm (rAAA) scoring systems across patient repair type.

Predictive model	Discrimination: AUC (95% CI)*		Calibration: Hosmere–Lemeshow test†		Brier score‡
	OSR	EVAR	OSR	EVAR	OSR	EVAR
Updated Glasgow Aneurysm Score (GAS)	0.73 (0.65–0.81)	0.71 (0.61–0.81)	.31	.38	0.20	0.17
Vascular Study Group of New England rAAA Risk Score (VSGNE)	0.73 (0.66–0.81)	0.67 (0.59–0.76)	.96	.10	0.20	0.17
Harborview Pre-operative rAAA Risk Score (HRS)	0.73 (0.66–0.81)	0.67 (0.59–0.76)	<.001	.007	0.24	0.19
Modified Harborview Risk Score (mHRS)	0.76 (0.68–0.83)	0.63 (0.54–0.71)	.001	.032	0.20	0.19
Vancouver Scoring System (VSS)	0.72 (0.64–0.80)	0.68 (0.58–0.77)	.12	.52	0.22	0.24
Artificial Neural Network Score (ANN)	0.66 (0.58–0.74)	0.70 (0.61–0.79)	.55	.34	0.24	0.18
Dutch Aneurysm Score (DAS)	0.68 (0.59–0.76)	0.61 (0.50–0.72)	.41	.41	0.23	0.26
Edinburgh Ruptured Aneurysm Score (ERAS)	0.58 (0.52–0.65)	0.60 (0.51–0.68)	.052	.73	0.24	0.18

AUC = area under the curve; CI = confidence interval; OSR = open surgical repair; EVAR = endovascular aortic aneurysm repair; rAAA = ruptured abdominal aortic aneurysm.

^*AUC ≥0.70 represents fair discrimination.

^†p>.050 suggests that there isno statistically significant difference between observed and predicted probabilities, indicating good model calibration.

^‡The Brier score ranges from 0 – 1, with a lower score indicating better performance.

DISCUSSION

This study evaluated the predictive accuracy of eight rAAA mortality scoring systems in a large contemporary cohort of patients. Overall, the examined systems demonstrated poor to fair discrimination and poor to adequate calibration. Furthermore, there were no statistically significant AUC differences in subgroup analysis, undermining their accuracy in the external validation cohort.

These results align with the 2018 Society for Vascular Surgery and 2024 European Society for Vascular Surgery guidelines, suggesting that these scoring models may support discussions with physicians, patients, and families regarding transfer and treatment decisions. However, critical decisions, such as withholding treatment or opting for palliation, should not rely solely on these scoring systems.^28,29

Discussion of specific models

All of the existing prediction models were developed using retrospective data and used a limited number of predictor variables in each model. These differed between models, resulting in a wide variety of predictors (Table 1). Each model design is likely to be limited by missing data or selection.

The GAS was the first predictive model developed in 1994 on OSR patients but was later updated to include a variable for OSR or EVAR, one shock variable, and three variables for pre-existing comorbid conditions.^7,8 The VSGNE was developed using 242 patients undergoing OSR. The six point score was derived from a logistic regression model using stepwise elimination and was internally validated by calculating optimism by bootstrapping 1 000 replications.⁹ Further attempts were made to incorporate a more modern experience with EVAR using more recent VSGNE and Vascular Quality Initiative (VQI) data.³⁰ The VSS was developed on 157 rAAA OSRs utilising three variables: age, unconsciousness, and cardiac arrest.¹² Similarly, the ERAS model used 105 OSRs and uses only three variables (low haemoglobin, hypotension, and Glasgow Coma Scale < 15).¹⁵ Model development was limited by difficulty with attaining statistical significance in regression modelling.

The DAS was developed using the largest cohort of 508 prospectively collected patients and used four variables: age and three pre-operative physiological indicator variables (SBP, cardiopulmonary resuscitation [CPR], and haemoglobin).¹⁴ The model used a higher EVAR percentage, although remaining relatively low (24%) compared with current real world EVAR use and the current cohort. The ANN score was developed on 107 patients predominantly undergoing OSR using an artificial neural network.¹³ Despite use of deep learning, the model used only four variables (age, loss of consciousness, shock, and CPR), incorporated 14% EVAR patients, and performed marginally better than the equivalent model trained using traditional regression.

The HRS used 303 patients, implementing an EVAR first approach to rAAA repair halfway through the study cohort enrolment, hence the higher EVAR proportion (24%). The model uses two variables encompassing haemorrhagic shock elements (pH and hypotension) and two variables potentially affecting mortality risk from multiple organ failure (advanced age and creatinine). The AUC for predicted 30 day death of the HRS model was 0.81, but unlike the other validated scoring systems, internal validation or optimism calculations using bootstrapping or other methods were not performed.¹⁰ Finally, the mHRS, developed using 360 patients, is the most contemporary predictive model, similar to HRS but replacing pH < 7.2 with INR > 1.8.

While GAS, DAS, and VSS used regression models with coefficients to calculate the individual mortality risk prediction, the remaining models adopted point scores corresponding to mortality rates. Among these, only VSGNE scores correlated with predicted mortality rates, whereas ANN, ERAS, HRS, and mHRS scores corresponded to observed mortality rates. Failure to assess and report calibration is a common issue with potential ramifications since poor calibration, defined as a disagreement between the estimated and observed number of events, can result in misleading predictions.^31,32 This may explain the poor calibration observed in these models.

Previous validation attempts

Several publications have attempted external validation of existing rAAA prediction models, all with significant limitations. Half of these were published before 2006 and primarily evaluated the GAS.³³ Six recent studies have performed external validation of more modern scores, given the increased number of these models within the past decade.^{17,18,33–36}

van Beek et al. studied 449 patients in a retrospective multicentre cohort in Amsterdam using GAS, VSS, and ERAS. The GAS and VSS performed best with fair discrimination but consistently overestimated the mortality rate, while ERAS performed poorly.¹⁷ Vos et al. studied 347 patients from a retrospective multicentre study, also from the Netherlands, with the addition of DAS. All models showed fair discrimination at best but adequate calibration.³³ Reite et al. studied 177 patients using the Hardman index, VSS, GAS, and ERAS in a multicentre retrospective Norwegian cohort and found generally poor discrimination with all AUCs < 0.7.³⁵ The primary drawback for all these studies is the inclusion of older data in the cohort resulting in almost exclusive use of OSR (85% in van Beek et al.,¹⁷ 75% in Vos et al.,³³ and 100% in Reite et al.³⁵), limiting their generalisability to contemporary cohorts where patients treated with OSR are more likely to have complex anatomy and not reflecting the current trend of EVAR first practice for rAAA.

Three recent studies have attempted to validate multiple scoring systems using smaller but more contemporary cohorts with higher proportions of EVAR. Hansen et al. evaluated DAS, HRS, and VSGNE in 38 patients (EVAR 88%).³⁴ Discrimination was high, with VSGNE having the highest AUC (0.86). Thompson et al. likewise applied the GAS, Hardman index, VSS, ERAS, HRS, VSGNE, and ANN scores to a retrospective 60 patient cohort (EVAR 56%).¹⁸ Discrimination and calibration were not reported in this study, but categorisation of high mortality risk ranged widely and a comparison of actual mortality to predicted mortality showed underestimation by all systems. Ciamarella et al. compared the HRS, DAS, and VSGNE scores in 49 patients (EVAR 67%) and demonstrated adequate discrimination, but the small cohort resulted in wide confidence intervals.³⁶ All studies demonstrated a wide range of variability in results, probably due to the small sample sizes compared with the developmental cohorts as well as the high variability in presentation and physiological condition of patients with rAAA.

Model validation and outcomes

To address as many of these drawbacks as possible, this study was designed around a retrospective cohort of a similar size or larger than the cohorts used to develop the models and large enough to account for the variable presentation of patients with rAAA. The beginning of cohort enrolment also coincided with the shift to EVAR first rAAA repair at the study institution, resulting in a cohort more reflective of contemporary rAAA management.

Overall, although three models showed fair discrimination in mortality prediction, none had an AUC > 0.74. The findings from the analysis of unexpected survivors and the sensitivity analysis showed significantly different measures of coagulopathy, shock, and OSR rates, as well as differing associations with transfer status. This suggests that critical physiological and systematic information affecting the prognosis of any specific patient may not be adequately captured in these models. This probably reflects a multifactorial cause in the development of each model given the heterogeneity of locales, transfer patterns, treatment paradigms at each institution, available data for training a model, selection bias, and, critically, the evolution of rAAA management over time. The focus on 30 day death may also impact the accuracy of any prediction models to capture physiological nuances associated with differing modes of death in the early and late periods. Although conducted on a less accurate model, the unexpected survival analysis allowed for data exploration at a granular level and offered insights into the overall poor model performance.

The low incidence and high clinical acuity of rAAA overall presents a challenge to truly understand the effects of pre-operative physiology on death and post-operative outcomes. Many recent studies on rAAA have prioritised sample size over accuracy and granularity of data. All but one of the prediction models had low dimensionality with four or fewer variables in the final model, limiting the ability to apply important physiological information. In addition, model development typically uses measurements at a single point in time and has not accounted for changes in patient condition over time or for laboratory and other clinical values affected by resuscitation.

Notably, the VSGNE model includes suprarenal clamping, which can only be definitively assessed intra-operatively. While the authors suggest using pre-operative computed tomography as a proxy for the need for suprarenal clamping, the model was built on this intra-operative variable, which may limit its validity as a pre-operative tool.

Limitations

This study has certain limitations common to validation studies based on retrospective data. Not all models provided regression coefficients and instead used integer point scoring, potentially diminishing the predictive accuracy. However, the models were used in this study as intended in practice. Notably, 45 (10.5%) of 427 patients were excluded due to missing data. Cohort specific biases should be considered. While the cohort reflects a large North American central referral system for rAAA treatment, it may have limited generalisability to settings with different patient transfer systems or healthcare infrastructure. This risk was mitigated by the cohort’s relatively large size, the use of a contemporary approach, and consistency of the results with prior studies.^{17,18,33–36}

Conclusion

None of the rAAA prediction models evaluated in this study reached a level of good discrimination in a large retrospective cohort that reflects contemporary rAAA treatment paradigms. Such prediction models may be used to estimate a patient’s risk, which may be useful during a goals of care discussion at the bedside when time is critical. However, recognising their many limitations should limit their utility in definitive decision making based solely on the predicted risk. Future models should be derived from contemporary, preferably multicentre, cohorts with current rAAA treatment paradigms and should assess factors that are as representative as possible of a complete physiological picture including multiple organ dysfunction, coagulopathy, and surgical stress in order to mitigate these limitations.

WHAT THIS PAPER ADDS.

External validation of ruptured abdominal aortic aneurysm (rAAA) mortality scoring systems is limited and has significant drawbacks. This study addresses these issues using a large, contemporary, retrospective cohort from a large multicentre health system. The evaluated models demonstrated poor to fair discrimination and calibration, with no statistically significant differences between models in subgroup analysis, reflecting their limited utility in clinical decision making. This study’s cohort reflects modern rAAA management, highlighting the need for future models to incorporate complete physiological profiles, including multiple organ dysfunction, coagulopathy, and surgical stress, in order to enhance accuracy and clinical relevance.

APPENDIX A. SUPPLEMENTARY DATA

Supplementary data to this article can be found online at https://doi.org/10.1016/j.ejvs.2025.02.017.