External Validation of the CRAX2MACE model

Abstract

Background:

Single photon emission computed tomography (SPECT) myocardial perfusion is frequently used to predict risk of major adverse cardiovascular events (MACE). We performed an external validation of the CRAX2MACE score, developed to estimate 2-year risk of MACE in patients with suspected coronary artery disease (CAD).

Methods:

Patients who underwent clinically indicated SPECT with available follow-up for MACE were included (N=2,985). The prediction performance for MACE (revascularization, myocardial infarction, or death) within 2-years for CRAX2MACE was compared with stress and ischemic total perfusion deficit (TPD) using area under the receiver operating characteristic curve (AUC). Calibration was assessed with calibration plots, Brier score and the Hosmer-Lemeshow test.

Results:

MACE occurred within 2 years in 243 (8.1%) patients. The AUC for CRAX2MACE (0.710, 95% CI 0.677 – 0.743) was significantly higher compared to stress TPD (AUC 0.669, 95% CI 0.632 – 0.706, p=0.010) and ischemic TPD (AUC 0.664, 95% CI 0.627 – 0.700, p<0.001). The model had acceptable goodness-of-fit (p=0.103) and was well-calibrated with Brier score of 0.071.

Conclusions:

CRAX2MACE had higher predictive performance for 2-year MACE than quantitative perfusion in an external population. The current model is simple to use and could be implemented to assist physicians when estimating patient risk.

Introduction:

Single-photon emission computed tomography myocardial perfusion imaging (SPECT MPI) is commonly used for risk stratification in patients with suspected CAD. Regional myocardial ischemia has been established as an important predictor of cardiovascular events¹. However, the prevalence of myocardial ischemia in patients undergoing SPECT imaging has diminished over time ². As a corollary, consideration of other clinical and imaging information is increasingly important. However, to date holistic risk-prediction models, integrating ischemia with other imaging and clinical variables, have been underutilized³.

The cardiovascular risk assessment for major adverse cardiac events (MACE) at 2-year (CRAX2MACE) model incorporates several clinical and imaging variables to predict 2-year risk of MACE⁴. The model can be calculated quickly with routinely available information and demonstrated excellent discrimination for MACE with area under the receiver operating characteristic curve (AUC) of 0.79 during internal validation, significantly outperforming quantitative perfusion alone ⁴. However, a recent study questioned the external validity of the CRAX2MACE score, demonstrating AUC of 0.612 for a composite outcome of all-cause mortality, non-fatal myocardial infarction (MI) or late coronary revascularization⁵.

While external validation frequently demonstrates reduced predictive performance due to differences in patient populations, the reported accuracy was significantly lower than would be expected by using quantitative perfusion assessment alone. Accordingly, we performed an external validation study of the CRAX2MACE model using data from our institution.

METHODS

Study Population:

This was a retrospective study of 2,985 patients who underwent SPECT-MPI as part of clinical care between September 1, 2014 and December 31, 2018 at the University of Calgary with available follow-up for MACE. CRAX2MACE was derived to predict outcomes in patients with suspected CAD as described in detail previously⁴. The score was derived using a logistic regression model in 3,896 patients and validated using 1,946 patients from a single center, imaged between 2001 and 2008. Details of the CRAX2MACE model components and calculation are shown in Supplemental Table 1. In our primary analysis we included all patients, but we performed a supplemental analysis after excluding patients with known CAD. Known CAD was defined as either a history of previous myocardial infarction (MI) (n=354) or revascularization (n=320)⁶. The study was approved by the University of Calgary Conjoint Health Research Ethics Board.

Clinical Information

Details regarding past medical history were prospectively collected in the Alberta Provincial Project for Outcome Assessment in Coronary Heart Disease (APPROACH) database. History of cardiac hospitalization in the preceding 3 years was obtained using the Discharge Abstract Database using previously validated ICD-10 codes⁷.

Image Acquisition

Patients underwent a ^99mTc-Sestamibi rest-stress SPECT MPI with a GE Discovery 570 CZT scanner (GE, Boston, USA). Image acquisition protocols and radiotracer doses have been described previously⁸. Stress testing was conducted with symptom-limited exercise stress (n=1,683) or pharmacological stress (n=1,302).

Image Analysis

All image datasets were de-identified and transferred to Cedars-Sinai Medical Center, where quality control was performed by experienced core laboratory technologists without knowledge of the clinical data. Stress and rest images were analyzed by Quantitative Perfusion SPECT (QPS) software (Cedars-Sinai Medical Center, Los Angeles, CA) as previously described to quantify total perfusion deficit (TPD)⁹. with ischemic TPD calculated as the difference between stress and rest values. Left ventricular ejection fraction was calculated from post-stress gated images. The presence of transient ischemic dilation (TID) was established using previously derived thresholds for abnormal TID ratio¹⁰. The CRAX2MACE score was calculated using age, sex, diabetes, recent cardiac hospitalization, stress TPD, ischemic TPD, post-stress LVEF, and TID as previously described⁴.

Clinical Outcomes:

Our primary outcome was MACE, defined as late revascularization (coronary artery bypass grafting or percutaneous coronary intervention >90 days post MPI), admission for non-fatal MI, or all-cause mortality within the first 2-years of follow-up, similar to the original derivation study⁴. Follow up for MACE was obtained through the Discharge Abstracts Database, National Ambulatory Care Reporting system and Alberta Vital Statistics. We also assessed a secondary outcome of death or MI.

Statistical analysis:

Continuous variables were summarized as mean (standard deviation [SD]) if normally distributed and as median (interquartile range [IQR]) otherwise. The prediction performance of the CRAX2MACE score was assessed with AUC and compared with stress TPD and ischemic TPD using DeLong’s method¹¹. AUC was also assessed with a time-dependent ROC analysis (timeROC package in R version 4.1.2). Lastly, we assessed model calibration with Brier scores, calibration plots, and the Hosmer-Lemeshow goodness-of-fit test using deciles of risk.

RESULTS

Patient population

In total, 2,985 patients were included with median age 67.4 (IQR 59.2 to 75.00) and 1,625 (54.4 %) male patients. During the first 2-years of follow-up, MACE occurred in 243 (8.1%) patients (first event 49 late revascularization, 46 myocardial infarct, 148 deaths). Table 1 compares patient characteristics of patients who experienced MACE compared to those who did not.

Table 1: Population characteristics.

Population characteristics stratified by occurrence of major adverse cardiovascular events (MACE) within the first 2 years of follow-up. CAD – coronary artery disease, IQR – interquartile range, LVEF – left ventricular ejection fraction, TPD – total perfusion deficit.

	No MACE within 2 years  n = 2742	MACE within 2 years  n = 243	P-value
Age, median (IQR)	67.2(58.9, 74.8)	71.2(63.2, 77.4)	<0.001
Male, n(%)	1477(53.9)	148(60.9)	0.037
Hypertension, n(%)	1619(59.0)	160(65.8)	0.041
Diabetes, n(%)	670(24.4)	106(43.6)	<0.001
Dyslipidemia, n(%)	1380(50.3)	137(56.4)	<0.001
Recent cardiovascular hospitalization, n(%)	198(7.2)	38(15.6)	<0.001
Previous myocardial infarction, n(%)	305(11.1)	49(20.2)	<0.001
Previous revascularization, n(%)	471(17.2)	64(26.3)	<0.001
Pharmacologic Stress, n(%)	1140(41.6)	162(66.7)	<0.001
Stress TPD (%), median(IQR)	2.1(0.5–4.9)	4.7(1.9–12.0)	<0.001
Stress TPD < 5%	2083(76.0)	124(51.0)	<0.001
Stress TPD 5 – 9.9%	397(14.5)	52(21.4)
Stress TPD 10 – 19.9%	160(6.0)	34(14.0)
Stress TPD 20 – 29.9%	65(2.4)	23(9.5)
Stress TPD ≥ 30%	32(1.2)	10(4.1)
Ischemic TPD(%), median(IQR)	1.6(0.4–3.4)	3.0(1.2–5.5)	<0.001
Ischemic TPD < 5%	2375(86.6)	173(71.2)	<0.001
Ischemic TPD 5 – 9.9%	295(10.8)	49(20.2)
Ischemic TPD 10 – 19.9%	66(2.4)	19(7.8)
Ischemic TPD ≥20%	6(0.2)	2(0.8)
LVEF(%), median(IQR)	66(57–73)	60(45–69)	<0.001
LVEF > 44%	2525(92.1)	192(79.0)	<0.001
LVEF 40 – 44%	66(2.4)	13(5.4)
LVEF 30 – 39.9%	90(3.3)	21(8.6)
LVEF <30%	61(2.2)	17(7.0)
Transient ischemic dilation, n(%)	134(4.9)	12(4.9)	1.000

Prediction Performance

Figure 1 shows AUC for 2-year MACE. The AUC for CRAX2MACE (0.710, 95% CI 0.677 – 0.743) was significantly higher compared to stress TPD (AUC 0.669, 95% CI 0.632 – 0.706, p=0.010) and ischemic TPD (AUC 0.664, 95% CI 0.627 – 0.700, p<0.001). AUC was also higher for CRAX2MACE when assessed with time-dependent analysis (AUC 0.718) compared to stress TPD (AUC 0.695) or ischemic TPD (AUC 0.672, p<0.01 for both). Prediction performance for death or MI within 2 years was also superior for CRAX2MACE (AUC 0.706, 95% CI 0.669–0.742) compared to stress TPD (AUC 0.656, 95% CI 0.615 – 0.696, p=0.004) or ischemic TPD (AUC 0.648, 95% CI 0.607 – 0.689, p<0.001), Figure 2. Supplemental Figures 1 and 2 shows prediction performance in patients without known CAD.

Figure 1: Prediction Performance for 2-year MACE.

Prediction performance for major adverse cardiovascular events (MACE) during 2-years of follow-up. MACE included revascularization, non-fatal myocardial infarction, and all-cause mortality. AUC – area under the receiver operating characteristic curve, CI – confidence interval, TPD – total perfusion deficit.

Figure 2: Prediction Performance for 2-year Death or MI.

Prediction performance for death or non-fatal myocardial infarction (MI) during 2-years of follow-up. AUC – area under the receiver operating characteristic curve, CI – confidence interval, TPD – total perfusion deficit.

Model Calibration

Figure 3 shows predicted 2-year risk of MACE and observed 2-year rate of MACE for deciles of CRAX2MACE score. The Brier score was 0.071, suggesting good calibration. The Hosmer-Lemeshow test demonstrated acceptable goodness-of-fit (chi-square 13.4, p=0.099). Incidence of MACE at 2-years in patients at low risk (estimated risk <5%; n=1,328, 44.5%) was 3.2%, at moderate risk (estimated risk 5–9.9%; n=1,185, 39.7%) was 9.4%, at high-risk (estimated risk 10–19.9%, n=326, 10.9%) was 14.1% and at very high risk (estimated risk ≥20%; n=146, 4.9%) was 29.5%. Supplemental Figure 3 outlines calibration in patients without known CAD.

Figure 3: Calibration plot for CRAX2MACE model.

Calibration plot outlining predicted risk and observed rate of major adverse cardiovascular events (MACE) during 2-years of follow-up.

Discussion:

We performed an external validation study of the CRAX2MACE model using a large retrospective cohort. We identified better discrimination for 2-year MACE with the CRAX2MACE model compared to quantitative perfusion alone. Additionally, the model demonstrated good predictive performance for hard cardiac events and good calibration between predicted and actual risk. As expected, our results demonstrate reduced accuracy compared to the original derivation study. However, the overall accuracy remained better compared to current quantitative methods, suggesting that the model maintains reasonable predictive performance in an external population.

The CRAX2MACE score demonstrated good predictive performance for 2-year MACE in this external population, with an AUC of 0.71. The model demonstrated good calibration and maintained high predictive performance for hard events. One major advantage of the score is that it can be readily calculated with routinely acquired SPECT MPI parameters. However, a larger number of potentially important clinical, stress, and imaging features can be efficiently integrated using machine learning. This approach has been used to identify patients with a low risk of obstructive CAD and low risk of MACE for stress only imaging^12–13. However, in its current form CRAX2MACE may offer a simple and readily implemented, yet still accurate, method to help physicians predict MACE risk following SPECT MPI.

Our results may differ from the derivation study and previous external validation for several reasons. Our patient population was more contemporary (2014 to 2018) compared to Leslie et al. (2001 to 2008). Consistent with improvements in medical therapies and decreasing rates of cardiovascular events over time ^2,14, in our study the prevalence of MACE at 2 years was significantly lower (8.1%) compared to the derivation study (14.0%). In spite of this difference, calibration was good in both our study (Brier score 0.071) and the study by Megna et al (Brier score 0.061)⁵. As expected for an external validation population, there are significant differences in many patient characteristics for our cohort compared to the other two. For example, use of pharmacologic stress was lower in the cohort of Leslie et al. (34.8%), and comparable between our cohort (43.6%) and the cohort of Megna et al (44.1%)^4,5. Our cohort was older (median age 67.4) compared to either the cohort of Megna et al (mean age 62) or Leslie et al. (mean age 65.1) ^4,5. Stress TPD also varied between cohorts with mean stress TPD of ~4% in Megna et al., ~10% in Leslie et al., and 4.4% in our cohort ^4,5. This at least partially explains the decreased predictive performance for CRAX2MACE in our study compared to the validation of Leslie et al⁴. These differences likely also explain some of the difference in results compared to the previous external validation study by Megna et al⁵. Importantly, in our study higher stress and ischemic TPD values, which make up a significant component of the CRAX2MACE score, occurred in patients who experienced MACE. This was not the case in the study by Megna et al.⁵, which would be expected to decrease model discrimination. However, these markers have been previously reported as the most robust markers of risk in other populations^1,15, and therefore should not prohibit generalizability of the CRAX2MACE model. Importantly, prediction performance for hard cardiac events (non-fatal MI and all-cause mortality) was also significantly higher compared to quantitative perfusion alone in our study. Regardless, given the discrepancy in results, additional validation in sites outside North America could be considered.

New Knowledge Gained

The CRAX2MACE model demonstrates good discrimination of 2-year MACE and acceptable goodness-of-fit and calibration in an external patient population. The CRAX2MACE model is simple to apply in clinical practice and outperforms quantitative analysis alone

Conclusions

CRAX2MACE had high predictive performance for 2-year MACE in an external population. To improve the prognostic accuracy and relevance of CRAX2MACE in an era of decreasing ischemia, the model may benefit from further adaptation.

ABBREVIATIONS

APPROACH

Alberta Provincial Project for Outcome Assessment in Coronary Heart Disease

AUC

area under the receiver operating characteristic curve

CAD

coronary artery disease

CRAX2MACE

cardiovascular risk assessment for major adverse cardiac events

MACE

major adverse cardiovascular events

MI

myocardial infarction

MPI

myocardial perfusion imaging

SPECT

single photon emission computed tomography

TPD

total perfusion deficit