Prognostic Value of Machine Learning–based Time-to-Event Analysis Using Coronary CT Angiography in Patients with Suspected Coronary Artery Disease

Abstract

Purpose

To assess the long-term prognostic value of a machine learning (ML) approach in time-to-event analyses incorporating coronary CT angiography (CCTA)–derived and clinical parameters in patients with suspected coronary artery disease.

Materials and Methods

The retrospective analysis included patients with suspected coronary artery disease who underwent CCTA between October 2004 and December 2017. Major adverse cardiovascular events were defined as the composite of all-cause death, myocardial infarction, unstable angina, or late revascularization (>90 days after index scan). Clinical and CCTA-derived parameters were assessed as predictors of major adverse cardiovascular events and incorporated into two models: a Cox proportional hazards model with recursive feature elimination and an ML model based on random survival forests. Both models were trained and validated by employing repeated nested cross-validation. Harrell concordance index (C-index) was used to assess the predictive power.

Results

A total of 5457 patients (mean age, 61 years ± 11 [SD]; 3648 male patients) were evaluated. The predictive power of the ML model (C-index, 0.74; 95% CI: 0.71, 0.76) was significantly higher than the Cox model (C-index, 0.71; 95% CI: 0.68, 0.74; P = .02). The ML model also outperformed the segment stenosis score (C-index, 0.69; 95% CI: 0.66, 0.72; P < .001), which was the best performing CCTA-derived parameter, and patient age (C-index, 0.66; 95% CI: 0.63, 0.69; P < .001), the best performing clinical parameter.

Conclusion

An ML model for time-to-event analysis based on random survival forests had higher performance in predicting major adverse cardiovascular events compared with established clinical or CCTA-derived metrics and a conventional Cox model.

Keywords: Machine Learning, CT Angiography, Cardiac, Arteries, Heart, Arteriosclerosis, Coronary Artery Disease

Supplemental material is available for this article.

© RSNA, 2023

Summary

Employing machine learning in time-to-event models that integrate clinical and coronary CT angiography–derived predictors may improve precision of personalized risk predictions in patients with suspected coronary artery disease.

Key Points

• ■ In this retrospective analysis of 5457 patients with suspected coronary artery disease, a machine learning–based time-to-event model slightly but significantly outperformed a Cox model (C-index, 0.74 vs 0.71, respectively; P = .02) in predicting major adverse cardiovascular events based on coronary CT angiography–derived and clinical predictors.

• ■ The assessed machine learning model also demonstrated higher prognostic value compared with established coronary CT angiography–derived or clinical predictors (best performance by segment stenosis score: C-index, 0.69; P < .001).

Introduction

In the last decade, coronary CT angiography (CCTA) has been one of the most transformative imaging modalities in the evaluation of obstructive coronary artery disease (CAD) and assessment of atherosclerotic plaque (1). Due to its high negative predictive value, CCTA has been recently recommended by several practice guidelines as a first-line strategy in ruling out obstructive CAD, particularly among symptomatic individuals with low to intermediate risk (2–4). Besides exclusion of obstructive CAD and thus ischemia, CCTA can assess the extent of nonobstructive CAD and thereby may guide further medical therapy. Numerous CCTA-derived risk scores and parameters, such as the segment involvement score (SIS), segment stenosis score (SSS), and grading of tissue features of coronary atherosclerosis, have demonstrated incremental predictive power for future cardiovascular events compared with clinical variables alone (5–7). However, existing scores that integrate CCTA-derived and clinical parameters (5,8) are currently not widely used.

Recently, analysis of CCTA data using machine learning (ML) algorithms has opened new avenues in the diagnosis and prognostication of cardiovascular disease and the personalized risk assessment of primary and secondary cardiovascular prevention. ML employs tools from computer science and statistics to identify generalizable predictive patterns and latent relations within multivariable data sets. Particularly in a scenario where both clinical and imaging information is readily available at the workplace and can be integrated in an automatic or semiautomatic manner, ML applications have the potential to maximize the predictive value that can be extracted from various examinations. Indeed, growing evidence demonstrates the superiority of ML-based risk modeling when compared with CCTA-derived risk scores or clinical parameters (9–11). Variability in the progression of CAD, however, requires predictive models to take into account the time to an event of interest. Accordingly, survival analysis (12), or more generally, time-to-event analysis, is an established method to predict major adverse cardiovascular events (MACE) in non-ML studies and is also used in clinical studies across medical disciplines. This approach goes beyond classifying whether an event occurs or not, a strategy that is typically used in the data science field. However, to our knowledge, ML-based time-to-event analysis has not yet been performed for assessing the prognostic value of CCTA-derived information.

We aimed to investigate the long-term prognostic value of an ML-based time-to-event modeling approach using CCTA-derived measures and clinical variables for MACE occurrence in patients with suspected CAD.

Materials and Methods

Study Sample

This retrospective study, which was approved by the local ethics committee, included patients with suspected but not proven CAD who underwent CCTA between October 2004 and December 2017. Exclusion criteria included a patient age younger than 18 years, a diagnosis of congenital heart disease, the occurrence of an acute life-threatening situation, or lack of a stable sinus rhythm during examination. Among those who met the inclusion criteria, 2011 patients undergoing CCTA before November 2008 were included in a study from 2019 that assessed the long-term prognostic value of CCTA by means of conventional statistics (13).

Before the examination, all patients provided written informed consent and completed a questionnaire about personal information such as age, weight and height, symptoms, and current medication. Additionally, cardiac risk factors were assessed in the questionnaire. Hypertension was defined as a systolic blood pressure greater than 140 mm Hg or administration of antihypertensive treatment. Diabetes mellitus was defined as fasting blood glucose level greater than 7 mmol/L or abnormal oral glucose tolerance test, as defined by the World Health Organization, and use of insulin or oral antidiabetic therapy. Smoking status was categorized as never, previously, and currently smoking. Family history of CAD was defined as the presence of CAD in first-degree relatives younger than 65 years for women or younger than 55 years for men. Additionally, laboratory results for total cholesterol, low-density lipoprotein, high-density lipoprotein, and triglycerides were collected. Based on the cardiovascular risk factors, two risk scores were calculated: the Morise risk score (14) and the Framingham risk score (15).

Follow-up information was recorded during clinical visits, through questionnaires sent by mail, or by phone contact. The reported events were verified through the electronic medical records or direct contact with an attending physician. A national registry to follow up with patients was not available. The primary combined end point of the study was MACE, defined as the composite of all-cause death, myocardial infarction, unstable angina, or late revascularization (>90 days after CCTA).

CT Protocol

Image acquisition was performed using four different CT scanners (Siemens Healthineers) during the study period (details are provided in Table S1).

Patient heart rate and blood pressure were monitored before the examination. In the absence of contraindications, patients were administered intravenous β-blocker medication, targeting a heart rate of less than 60 beats per minute, as well as sublingual nitrates to pursue vasodilation of coronary arteries in case of a systolic blood pressure higher than 100 mm Hg. The CT acquisition protocol has been previously described in detail (6).

Plaque Assessment from CCTA

Coronary arteries were segmented using the first 15 of the original 18 segments according to the simplified American Heart Association classification (16). Vessel segments greater than 1.5 mm in diameter were interpreted by two experienced radiologists, of whom at least one had read a minimum of 400 cardiac CT scans (R.A., A.W., E.H., S.A.M., M.H.), and any disagreement was solved by consensus. Stenosis severity caused by coronary artery plaques was evaluated and categorized as none (0%), minimal (1%–24%), mild (25%–49%), moderate (50%–69%), and severe (≥70%). The minimum vessel diameter size required for evaluation was 1.5 mm. Plaque composition was determined for each diseased segment and graded as noncalcified, calcified, or mixed plaques. Noncalcified plaques were defined as a tissue structure of at least 1 mm² with lower attenuation compared with the contrast-enhanced lumen. Calcified plaques were defined as a visible calcification of more than 1 mm² with a signal intensity greater than that of the contrasted vessel lumen. These plaques were further characterized as spotty if the diameter was less than 3 mm in any direction or gross if the extent of the calcification was greater than or equal to 3 mm in any direction. Plaques meeting both the criteria for noncalcified and calcified plaques were classified as mixed plaques.

CCTA-derived Characteristics

Several conventional CCTA-derived parameters were calculated based on plaque assessment. CAD severity was categorized as normal, nonobstructive in the case of less than 50% stenosis, and obstructive in the case of greater than 50% stenosis, with obstructive CAD further categorized as one-vessel obstructive, two-vessel obstructive, and three-vessel obstructive (17,18). The SSS and SIS were determined as described by Min et al (19). Last, the CAD Reporting and Data System (CAD-RADS) categories were calculated as proposed by Cury et al (20).

Time-to-Event Models Integrating Clinical and CCTA-derived Data

A total of 18 clinical and CCTA-derived variables were included, comprising demographic parameters, clinical cardiovascular risk factors and risk scores, CCTA-derived characteristics, CCTA-derived risk scores, and the CT scanner model generation. A full list of predictors is provided in Table S2.

A Cox proportional hazards model (Cox model) was built as the reference. Here, the proportional hazards assumption was validated by testing for independence between scaled Schoenfeld residuals and time according to the method proposed by Grambsch and Therneau (21) (see Table S3 and Fig S1). To minimize overfitting, a feature selection was implemented by recursive feature elimination with cross-validation based on squared standardized regression coefficients (22). For the ML model, random survival forests as an ensemble of survival trees with log-rank test statistics as a splitting criterion were employed (23,24). By fitting decision trees on various bootstrap samples of the data set with random subsets of variables, a feature selection is performed during model training (25). Both models were trained, validated, and tested using repeated nested cross-validation (26). Details on the cross-validation framework are provided in Appendix S1. Both models were implemented in Python (version 3.7.3; Python Software Foundation) using scikit-survival (27) and sckit-learn (28). The code used to model and analyze the data has been made publicly accessible on GitHub (https://github.com/DHM-CCTA-ML/CCTA_ML_TimeToEvent).

Statistical Analysis

Categorical variables are presented as frequencies and percentages. Continuous variables are presented as means ± SDs or medians (IQRs) when variables are not normally distributed. Spearman rank correlation coefficient was calculated for all pairs of predictors including separate variables representing the four different CT scanners employed in our study. From the trained time-to-event models, the relevance of features was assessed by permuting individual variables in the test data within the cross-validation framework and measuring the corresponding decrease in performance.

The predictive power of variables and integrative models was assessed by Harrell concordance index (C-index), which was calculated together with 95% CIs (29). C-indexes were compared according to the method described by Kang et al (30), and statistical significance was considered for P < .05. In addition, Kaplan-Meier analyses were conducted for both integrative models with patients stratified by deciles of the estimated risk. Statistical analyses were performed in Python (version 3.7.3), including the library ELI5 (31) to perform the permutation analysis, MATLAB R2019a (MathWorks), and R software (version 3.6.1; R Foundation) (32), including the compareC package (30).

Results

Patient Clinical and CT Characteristics

During the study period, 7893 patients with suspected CAD underwent CCTA. A total of 141 patients were excluded; 50 who were younger than 18 years, four with congenital heart disease, 73 without a stable sinus rhythm, and 14 due to a life-threatening situation. A total of 2295 patients were lost to follow-up (Fig 1). Thus, 5457 patients were included (61 years ± 11, 3648 [66.8%] male patients) in the final analysis. Table 1 shows all characteristics of the study patients. The median CAD 10-year risk calculated using the Framingham risk score was 8.6 (IQR, 5.1–14.4); the pretest risk assessed by the Morise score was low in 930 patients (17.0%), intermediate in 4257 patients (78.0%), and high in 270 patients (4.95%). A comparison of the characteristics between study patients and patients lost to follow-up is provided in Table S4.

Figure 1:

Study flowchart with inclusion and exclusion criteria. CAD = coronary artery disease, CCTA = coronary CT angiography, MACE = major adverse cardiovascular events.

Table 1:

Patient Characteristics

During the median follow-up of 7.3 years (IQR, 4.5–9.8 years; full range, 1–10 years) among all included patients, MACE was observed in 304 patients (5.57%). All-cause mortality was observed in 136 patients (2.49%), myocardial infarction in 26 patients (0.48%), unstable angina in two patients (0.004%), and late revascularization in 156 patients (2.86%). Among these patients, cardiac death occurred in 57 patients (1.04% of 5457 patients) and noncardiac death occurred in 79 patients (1.45%).

The distribution of patients with different scanner generations included 1144 patients (21.0% of 5457 patients) examined with a 64-slice single-source CT scanner, 970 (17.8%) with a 64-slice dual-source scanner, 1828 (33.5%) with a 128-slice dual-source CT scanner, and 1515 (27.8%) with a 192-slice dual-source CT scanner.

Patient CCTA-derived Characteristics

Table 2 shows the CCTA-related findings among the study patients. No CAD was observed in 1076 patients (19.7% of 5457 patients), whereas 3013 patients (55.2%) were diagnosed with nonobstructive CAD and 1368 patients (25.1%) showed obstructive CAD. The median scores for SSS and SIS were 4 and 3, respectively. Most of the patients were diagnosed with CAD-RADS 2 (35.0%).

Table 2:

Coronary CT Angiography Findings in Study Patients

Variables representing the different CT scanner models showed a more pronounced correlation with CCTA-derived data (-0.18 < ρ < 0.24) than with clinical parameters (-0.12 < ρ < 0.10) (Fig 2).

Figure 2:

Matrix depicts correlation between predictors. Labels in black correspond to the CT scanner generation, in dark blue to clinical features, and in light blue to coronary CT angiography–derived parameters. CAD = coronary artery disease, CAD-RADS = Coronary Artery Disease Reporting and Data System, DSCT = dual-source CT, SSCT = single-source CT.

Prediction of MACE by the Integrated Time-to-Event Models

A median of 15 (IQR, 14–16) features was selected for the Cox model within the inner cross-validation folds. Figure 3 shows feature importance of the Cox and ML models, respectively. The highest permutation importance (calculated by the mean decrease of the C-index) for the Cox model was observed for obstructive CAD at 0.129, followed by age at 0.055, the CT scanner generation at 0.039, and the SSS at 0.038. All other clinical and CCTA-derived parameters showed a permutation importance of less than 0.02. For the ML model, the highest permutation importance was demonstrated for age at 0.029, followed by CT scanner generation at 0.02, SSS at 0.015, and the Framingham risk score at 0.013. All other variables showed a permutation importance of less than 0.005.

Figure 3:

Graph depicts the importance of features in integrated time-to-event models. The bars represent the permutation importance of the predictors included in (A) the Cox model and (B) the machine learning model for time-to-event prediction based on random survival forests. Clinical features are shown in dark blue and coronary CT angiography–derived parameters are in light blue. CAD = coronary artery disease, CAD-RADS = Coronary Artery Disease Reporting and Data System.

The predictive power of the ML model (C-index, 0.74; 95% CI: 0.71, 0.76) was significantly higher compared with the Cox model (C-index, 0.71; 95% CI: 0.68, 0.74; P = .02). The ML model also outperformed all clinical (C-indexes, 0.48–0.66; all P < .001; best performance by patient age: C-index, 0.66; 95% CI: 0.63, 0.69) and CCTA-derived (C-indexes, 0.44–0.69; all P < .001; best performance by SSS: C-index, 0.69; 95% CI: 0.66, 0.72) predictors (Table 3, Fig 4). Kaplan-Meier curves stratified by the predicted risk represent this finding and highlight the potential for risk stratification (Fig 5).

Table 3:

Predictive Power of Coronary CT Angiography–derived Measures, Clinical Parameters, and Integrated Models for the Prediction of Major Adverse Cardiovascular Events

Figure 4:

Receiver operating characteristic curves for the prediction of major adverse cardiovascular events. The assessed machine learning (ML) model showed a statistically significantly higher concordance index (C-index) compared with the Cox-based regression model (P = .02). ML model performance was also significantly better compared with all clinical and coronary CT angiography–derived parameters or established risk scores integrating respective features (all P < .001). FRS = Framingham risk score, Spotty calc. = spotty calcification, SSS = segment stenosis score.

Figure 5:

Kaplan-Meier analysis stratified by the predicted risk of major adverse cardiovascular events. The Kaplan-Meier curves per decile of risk predicted by (A) the Cox model and (B) the evaluated machine learning model represent the potential of both models for risk stratification. The higher concordance index (C-index) for the machine learning model compared with the Cox model, shown in Figure 4, suggests an improved risk stratification (0.74 vs 0.71, P = .02).

Discussion

In this study, an ML model for time-to-event analysis based on random survival forests demonstrated a higher accuracy (C-index, 0.74; 95% CI: 0.71, 0.76) in the prediction of MACE than a conventional Cox model that integrates clinical and CCTA-derived features (C-index, 0.71; 95% CI: 0.68, 0.74; P = .02). Moreover, we found that the assessed ML model also outperformed established clinical and CCTA-derived risk predictors, including patient age (C-index, 0.66; 95% CI: 0.63, 0.69; P < .001) and the SSS (C-index, 0.69; 95% CI: 0.66.0.72; P < .001).

Since we undertook similar efforts to optimize the performance of both models in an algorithmic modeling approach, these results indicate that an ML model, specifically a random survival forest, can benefit from characteristic ML capabilities such as modeling nonlinear data, advanced feature selection, and ensemble learning (25,33). Therefore, ML may improve the risk stratification in patients with suspected CAD by an enhanced integration of clinical and CCTA-derived risk predictors.

Today, risk stratification for future cardiovascular events is largely performed using risk scores based on clinical data and, more recently, on the extent of CAD, degree of stenosis, or plaque composition measured through CCTA imaging parameters (34). The recent development of CCTA-derived scores has substantially improved risk stratification beyond clinical cardiovascular risk factors of individuals with suspected CAD (5–7). With the increasing complexity of available data, ML has been applied across the cardiovascular domain and has provided substantial improvements in prognostic performance, with the highest accuracy shown from the combination of imaging and clinical modalities rather than scores alone (1).

Corroborating our observations, previous studies on CCTA found that comprehensive ML models outperform established clinical or CCTA-derived risk predictors. Using data from a multicenter study, Motwani et al (11) evaluated a large panel of parameters and observed that their ML model exhibited a higher area under the receiver operating characteristic curve (AUC, 0.79) compared with the Framingham risk score of 0.61 or individual CCTA severity scores (SSS, 0.64; SIS, 0.64; Duke index, 0.62) for predicting all-cause mortality. In a large study from Nakanishi et al (35), an ML model that included clinical and CT-derived parameters achieved an AUC of 0.85 for predicting cardiovascular disease deaths, demonstrating higher performance than ML models that integrate clinical (AUC, 0.82) or CT data (AUC, 0.80) exclusively.

Beyond that, only a few studies in this field directly compared ML with more traditional statistical approaches based on the same set of predictors. An ML model from Tesche et al (36) made to predict MACE in individuals with suspected CAD showed higher discriminatory power (AUC, 0.96) than a conventional logistic regression analysis (AUC, 0.92) that combines CT risk scores with adverse plaque features and clinical parameters. In an algorithmic modeling approach employing nested cross-validation, Johnson et al (37) assessed different classifiers to predict all-cause mortality and found that the performance of a conventional logistic regression classifier (AUC, 0.74) was inferior to that of nonlinear models, including k-nearest neighbors (AUC, 0.77) and bagged trees (AUC, 0.77).

The findings of these studies are consistent with our results. However, while we assessed different time-to-event models (which represent the established statistical approach for these data), to our knowledge, our study was the first in this field to compare an ML-based survival model with Cox regression. Studies from other fields, however, corroborate the notion that ML approaches have the potential to outperform traditional Cox modeling. Spooner et al (38) demonstrated that both random forest–based models and various boosted models showed higher performances than Cox regression in predicting dementia when applied to two separate high-dimensional data sets. Jung et al (39) found that a model based on random survival forests showed the highest C-index, at 0.74, in predicting survival in patients with resectable upper gastrointestinal cancer when compared with a classic Cox proportional hazards model (C-index, 0.64) and some other ML approaches.

The results of our feature importance analysis showed relative concordance among the top three parameters for the Cox model (obstructive CAD, age, CT scanner generation) and the evaluated ML model (age, CT scanner generation, SSS). However, variables that typically show a nonlinear relation with the occurrence of cardiovascular events, such as age (15), demonstrated a higher relative importance in the random survival forest–based ML model than in the Cox model. This observation suggests that ML algorithms are better equipped to model nonlinear relations than more traditional statistical approaches. In addition, we observed overall lower permutation importance for the assessed ML model than for the Cox model. This finding indicates that our ML model is more robust to variances in a single predictor, presumably by a higher intrinsic complexity that allows for compensation in the absence of meaningful information provided by a specific variable. Interestingly, CT scanner generation was among the features with highest importance in both models, even though this variable alone represents a poor predictor of MACE in our evaluation. We hypothesize that the relevance of CT scanner generation as a predictor for MACE lies in its correlations with CCTA-derived parameters, as observed by the correlation analysis in our study. Therefore, these correlations likely reflect improvements in imaging technology that may have influenced the presentation and rating of coronary lesions. It should be noted in this context, though, that information provided by multiple predictors in a similar fashion may result in diminished feature importance compared with variables presenting unique informative content, such as CT scanner generation, due to the nature of this analysis. Considering another hypothesis, we cannot rule out that CT scanner generation might represent different risk profiles of patients undergoing CCTA over time (even as we do not observe a substantial correlation between CT scanner generation and clinical risk factors). However, as both the Cox model and ML model were trained with the same data, this aspect does not interfere with our conclusions.

In the fast-moving era of personalized medicine, with “big data” simultaneously becoming more available, ML represents a powerful platform for the use of complex data in predictive modeling (1). Particularly with the improved integration of different sources of health data, the high performance of ML applications may pave the way for more data-driven models in everyday clinical practice. In this context, introducing time-to-event analysis to ML models that integrate clinical and CCTA-derived predictors has the potential to further increase the precision of personalized risk predictions in patients with suspected CAD compared with not only traditional statistical models, but also previous ML approaches by harnessing the temporal information in follow-up data. It should be highlighted that the observed or predicted time to an event can convey valuable clinical information beyond the occurrence of an adverse event per se, which has also accordingly been considered in more traditional methods for time-to-event analysis.

Our study had some limitations. First, there may have been selection bias given the retrospective nature of the study among patients from an urban area and due to differences in the demographic or clinical characteristics between study patients and patients lost to follow-up. Second, this was a single-center study without external validation. However, we performed repeated nested cross-validation that ensures that the models are always trained, optimized, and tested with previously unseen data. Third, the predictors were not transformed as part of preprocessing to facilitate the handling of nonlinear relations by linear regression models. Doing so might have improved performance of the Cox model but is not expected to positively impact ML model performance. Finally, due to the relatively low pretest risk of the study sample, the fraction of patients with MACE was quite low. This may have influenced precision of the time-to-event models. However, this potential limitation applies to both assessed approaches, and we found no further reasons to suspect that censoring occurred in relation to the risk for MACE.

In conclusion, an ML model for time-to-event analysis employing random survival forests showed higher performance for the long-term prediction of MACE compared with a Cox model based on clinical and CCTA-derived variables in our large study of patients with suspected CAD. By employing ML in time-to-event models that integrate clinical and imaging data, our approach promises new avenues to improve both the risk stratification in patients with CAD and the precision of personalized cardiovascular medicine.