Machine Learning to Predict Abnormal Myocardial Perfusion from Pre-test Features

Abstract

Background:

Accurately predicting which patients will have abnormal perfusion on MPI based on pre-test clinical information may help physicians make test selection decisions. We developed and validated a machine learning (ML) model for predicting abnormal perfusion using pre-test features.

Methods:

We included consecutive patients who underwent SPECT MPI, with 20,418 patients from a multi-center (5 sites) international registry in the training population and 9,019 patients (from 2 separate sites) in the external testing population. The ML (extreme gradient boosting) model utilized 30 pre-test features to predict the presence of abnormal myocardial perfusion by expert visual interpretation.

Results:

In external testing, the ML model had higher prediction performance for abnormal perfusion (area under receiver-operating characteristic curve [AUC] 0.762, 95% CI 0.750 – 0.774) compared to the clinical CAD consortium (AUC 0.689) basic CAD consortium (AUC 0.657), and updated Diamond-Forrester models (AUC 0.658, p<0.001 for all). Calibration (validation of the continuous risk prediction) was superior for the ML model (Brier score 0.149) compared to the other models (Brier score 0.165 to 0.198, all p<0.001).

Conclusions:

ML can predict abnormal myocardial perfusion using readily available pre-test information. This model could be used to help guide physician decisions regarding non-invasive test selection.

INTRODUCTION

Myocardial perfusion imaging (MPI) is frequently used to diagnose or risk stratify patients with known or suspected coronary artery disease (CAD) (1–3). Abnormal regional myocardial perfusion can be used to detect obstructive CAD with high diagnostic accuracy (4). Additionally, the presence of abnormal regional myocardial perfusion can be used to identify a group of patients with a higher risk of major adverse cardiovascular events(5). As a result of these findings, the volume of SPECT MPI has grown to 15–20 million scans performed annually worldwide(6). However, the prevalence of abnormal perfusion has been decreasing since SPECT MPI was first implemented clinically, dropping from 40.9% in 1991 to 8.7% in 2009 (7). As a result, it is becoming increasingly important to ensure appropriate patient selection. This is particularly relevant since patients with normal regional myocardial perfusion on SPECT MPI may be more effectively risk stratified by using measures such as absolute myocardial blood flow (8, 9) or coronary artery calcification (10, 11). Additionally, patients with a low likelihood of abnormal perfusion may be candidates for stress-first imaging (and evaluated for stress only imaging) to reduce radiation exposure(12). However, SPECT (or PET) MPI could be particularly useful in patients with a high risk of ischemia since this may help target more aggressive therapies. For example, ischemia can be used to predict symptom benefit from revascularization (13)(14). However, to date, most pre-test risk models have focused on a patient’s likelihood of having anatomically defined obstructive CAD rather than abnormal regional myocardial perfusion.

Most pre-test risk prediction models have also been developed using statistical regression methods (15). Machine learning (ML) has the potential to surpass these traditional methods by efficiently integrating a large number of variables. ML may also identify non-linear relationships and higher-order interactions between variables (16), which is not possible with traditional statistical methods. This approach has been used to predict risk of major adverse cardiovascular events and to identify patients for stress-only imaging who have a low probability of obstructive CAD (17) or adverse events (18). In this work, we propose a ML score to predict likelihood of abnormal perfusion solely from pre-test patient factors (without imaging information). The model was trained using patients from the multi-center, international REgistry of Fast Myocardial Perfusion Imaging with NExt generation SPECT (REFINE SPECT)(19), then tested using data from two external sites.

MATERIALS AND METHODS

Study Populations

The internal testing population from included consecutive patients from 5 sites undergoing SPECT MPI between 2009 and 2014 (n=20,418) as previously described (19). The external testing population included 9,019 consecutive patients from the University of Calgary (n=2,985) and Oklahoma Heart Hospital (n=6,034). The study protocol complied with the Declaration of Helsinki and was approved by the institutional review boards at each participating institution. The overall study was approved by the institutional review board at Cedars-Sinai Medical Center.

Clinical Data

Clinical data was collected using site-specific mechanisms as previously described. Clinical indications for testing were recorded at the time of imaging. Clinical data included: age, sex, body mass index, past medical history, symptoms, and family history of CAD. Chest pain symptoms were classified as typical chest pain or atypical chest pain using standard definitions. Past medical history included: diabetes mellitus, hypertension, dyslipidemia, peripheral vascular disease, myocardial infarction (MI), percutaneous coronary intervention (PCI), coronary artery bypass grafting, and history of smoking. History of CAD was defined as previous myocardial infarction or previous revascularization(20).

Imaging protocols and interpretation

Imaging protocols for the REFINE SPECT population have been described previously (19). Experienced cardiologists interpreted perfusion with access to clinical history, rest and stress hemodynamic and ECG findings, and all available imaging data at the time of clinical imaging. Visual subjective interpretation of stress perfusion was assessed with summed stress scores (SSS) using the 17-segment American Heart Association model(21), or overall reader interpretation.

De-identified image datasets were transferred to the core laboratory (Cedars-Sinai Medical Center) where automated quantitation of SPECT MPI studies was performed by experienced technologists (19). Myocardial contours were generated automatically with Quantitative Perfusion SPECT (QPS) /Quantitative Gated SPECT (QGS) software (Cedars-Sinai Medical Center, Los Angeles, CA). Myocardial perfusion was quantified by total perfusion deficit (TPD) which incorporates severity and extent of perfusion abnormalities and is more reproducible compared to visual ischemia scoring (22, 23).

Outcomes

Abnormal perfusion was defined as SSS>3 if segmental scoring was available and as reader interpretation of abnormal or definitely abnormal if not (24). All studies in the external population were assessed using SSS, with SSS>3 defined as abnormal (24). We also performed a sensitivity analysis, where abnormal perfusion was defined as stress TPD > 5%.

Machine Learning

ML considered 30 pre-test variables as outlined in Supplemental Table 1. The variables were categorized as demographic, past medical history, indication, ECG, and vital signs. Resting ECG findings were included, since those could be determined prior to testing, but stress ECG findings were not considered. The overall proportion of missing values within the dataset was 0.35%. Missing values were imputed with the population’s median value for numerical variables, or with a distinct ‘missing’ label for categorical variables. For the model development, we utilized the currently leading ML method, extreme gradient boosting approach–XGBoost (25) implemented as xgboost package (version 0.82.1). XGBoost is a leading ML method which combines multiple decision trees as weak predictors in an ensemble learning method and has been shown to provide improved prediction performance compared to alternative methods (26). Variable importance was assessed using the built-in XGBoost gain metric, which is a measure of how important a particular feature is for the overall prediction performance.

Internal Training and Testing

ML models were trained with stratified 10-fold cross-validation to separate data into training and testing samples (27, 28). The whole dataset was randomly split into 10 folds with a similar proportion of patients with abnormal perfusion in each fold. During internal testing, 8 folds (80% of the population) were used for testing, one-fold (10% of the population) for validation, and one-fold (10%) used for testing. The process was repeated 10 times, with a different fold used for testing and validation in each iteration. Thus, only unseen data was used for testing each model and a total of 10 models were assessed. The advantages of the 10-fold cross-validation over single split-sample approach are well documented in the ML literature and include reducing variance in prediction error, related to arbitrarily splitting data, leading to a more accurate estimate of model performance and maximizing the data for both training and validation, without overfitting or overlap between test and validation data (27).

Comparison Risk Models

We compared prediction performance for the ML model to the CAD consortium basic (basic CAD) and clinical (clinical CAD) models (15) as well as the updated Diamond-Forrester (DF) model (29). The basic CAD model considers age, sex, and presenting symptoms (15). The clinical CAD model considers age, sex, presenting symptoms, diabetes, hypertension, dyslipidemia, and smoking (15). Both of the CAD consortium models were developed in a population of 5,677 patients who underwent coronary computed tomography angiography or invasive angiography and were designed to predict obstructive CAD. The updated DF model, a contemporary modification of the original DF model(30), considers age, sex, and type of chest pain and was modified using a cohort of 2260 patients with chest pain referred for invasive coronary angiography to ensure better calibration in current populations (29). These models were all developed for use in patients with suspected CAD.

External testing:

During external testing, the ML model was trained using all patients from the internal population. The model was then tested using patients from the two external sites.

Statistical Analyses

Categorical variables were summarized as number (proportion) and compared with a Chi-square or Fisher Exact test as appropriate. Continuous variables were summarized as mean (standard deviation [SD]) if normally distributed and median (interquartile range [IQR]) otherwise. Receiver operating characteristic (ROC) curves for identifying patients with abnormal perfusion. Area under the ROC curve (AUC) was used to compare models using the method described by Delong et al. (31). Model calibration was assessed with calibration graphs and Brier scores (32, 33). Cost estimates for cost per patient identified with abnormal perfusion were based on CPT re-imbursement code 78451 (34). All statistical tests were two-sided with a p-value <0.05 considered significant. Analyses were performed with Stata version 14.2 and R 4.1.2.

RESULTS

Details of the internal and external populations are shown in Table 1. As expected, there were significant differences between the two populations. In particular, patients in the external population were older and less likely to have previous CAD. An outline of the internal and external testing procedures is shown in Figure 1.

Table 1:

Baseline population characteristics

	Internal Population (N=20418)	External Population (N=9019)	p-value
Age, median (IQR)	64 (56, 73)	68 (60, 75)	<0.001
Male, n (%)	11642 (57.0%)	4871 (54.0%)	<0.001
Typical chest pain, n (%)	1224 (6.0%)	133 (1.5%)	<0.001
Atypical chest pain, n (%)	4589 (22.5%)	3208 (35.6%)	<0.001
Outpatient, n (%)	18417 (90.2%)	8169 (90.6%)	0.326
Inpatient, n (%)	1535 (7.5%)	832 (9.2%)	<0.001
Emergency department, n (%)	466 (2.3%)	18 (0.2%)	<0.001
Body mass index, median (IQR)	27.3 (24.6, 30.9)	29.4 (25.8, 33.3)	<0.001
Past Medical History, n (%)
Hypertension	12920 (63.3%)	5736 (63.6%)	0.600
Dyslipidemia	12903 (63.2%)	1979 (21.9%)	<0.001
Diabetes mellitus	5212 (25.5%)	2726 (30.2%)	<0.001
Peripheral vascular disease	2420 (11.9%)	2834 (31.4%)	<0.001
Prior CAD	5796 (28.4%)	1606 (17.8%)	<0.001
Prior myocardial infarction	2768 (13.6%)	677 (7.5%)	<0.001
Prior PCI	3967 (19.4%)	839 (9.3%)	<0.001
Smoking, n (%)	3875 (19.0%)	2841 (31.5%)	<0.001
Family History, n (%)	5642 (27.6%)	4720 (52.3%)	<0.001
Exercise stress, n (%)	9457 (46.3%)	5318 (59.0%)	<0.001
Abnormal perfusion, n (%)	4106 (20.1%)	2003 (22.2%)	<0.001

Population characteristics for the internal and external populations. CAD – coronary artery disease, IQR – interquartile range, PCI – percutaneous coronary intervention.

Figure 1: Internal and External Testing Procedures.

Summary of the internal and external testing procedures. Internal testing (left) was performed using 10-fold cross-validation where the internal testing population was randomly split into ten folds. In each of 10 iterations, 8 folds are used for training, one-fold for validation and one-fold for testing, with a different fold used for testing in each of 10 iterations. The results from the ten different testing sets are combined to provide an estimate of model performance. In external testing (right), model training and validation is performed with patients from REFINE SPECT and tested in two external sites.

Internal Testing

Of the 20418 patients included in the internal population, 4106 (20.1%) patients were classified as having abnormal perfusion. Prediction performance for abnormal perfusion was higher for ML (AUC 0.829, 95% CI 0.822 – 0.836) compared to the CAD clinical (AUC 0.716, 95% CI 0.708 – 0.725), CAD basic (AUC 0.699, 95% CI 0.691 – 0.708) and updated DF model (AUC 0.700, 95% CI 0.691 – 0.708, all p<0.001) (results in Figure 2). We performed a separate analysis where existing clinical scores were included as ML features, with no significant difference in ML prediction performance (AUC 0.829, 95% CI 0.822 – 0.836).

Figure 2: Prediction performance.

Prediction performance for abnormal perfusion from pre-test information in internal testing (left) and external testing (right). The machine learning model (XGBoost) had higher prediction performance compared to the coronary artery disease (CAD) consortium basic or clinical models as well as the updated Diamond-Forrester (DF) model (all p<0.001). AUC – area under the receiver operating characteristic curve, CI – confidence interval.

The calibration graph for the ML model is shown in Figure 3. The Brier score for ML (0.119) was superior compared to the other models (0.149 to 0.184, all p<0.001). The optimal threshold, determined by Youden index, was 0.196 (74% sensitive, 77% specific). However, we also identified a highly sensitive (95% sensitive) threshold (>0.0731, n = 6221 [30.5%] below threshold, 37% specific; 97% negative predictive value [NPV]) and an ultra-sensitive (99% sensitive) threshold (>0.0452, n = 2643 [12.9%] below threshold, 16% specific; 98% NPV) for abnormal perfusion. The top 25 variables by feature importance (a measure of how important a variable is for the model predictions) from the internal cross-validation are shown in Figure 4.

Figure 3: Model Calibration.

Calibration graph showing the predicted proportion of patients with abnormal perfusion and actual proportion of patients with abnormal perfusion by decile of machine learning (ML) score. The model calibration for the internal population (top) was higher compared to the external population (bottom), where there was a tendency to underestimate prevalence of abnormal perfusion.

Figure 4: Feature Importance.

Feature importance ranked according to feature gain. Higher feature gain means that the variable was more useful for categorizing patients as having normal or abnormal perfusion. Prior coronary artery disease (CAD) included a history of previous myocardial infarction (MI), percutaneous coronary intervention (PCI), or coronary artery bypass grafting. BMI – body mass index, CP – chest pain, ECG – electrocardiogram, LBBB – left bundle branch block, PVD – peripheral vascular disease.

External Testing

Of the 9,019 patients included in the external testing population, 2,003 patients (22.2%) were classified as having abnormal perfusion. Prediction performance for abnormal perfusion was higher for ML (AUC 0.762, 95% CI 0.750 – 0.774) compared to the CAD clinical (AUC 0.689, 95% CI 0.677 – 0.702), CAD basic (AUC 0.657, 95% CI 0.643 – 0.670) and updated DF model (AUC 0.658, 95% CI 0.645 – 0.671, all p<0.001) (Figure 2).

Model calibration in the external testing population is shown in Figure 3. All models demonstrated modest calibration. The Brier score for ML (0.149) was superior compared to other models (Brier score 0.165 to 0.198, all p<0.001). The optimal threshold from the internal testing population (>0.196, n = 2,394) was 56% sensitive, 82% specific. The highly sensitive threshold (>0.0731, n = 3089 [34.2%] below threshold) had sensitivity 89% and specificity 41% (NPV 93%). The cost per patient with abnormal perfusion identified for patients below the highly sensitive threshold would be $16,300, compared to $4,007 above the cut-off. The ultra-sensitive threshold (>0.0452, n = 1399 [15.5%] below threshold) had sensitivity 96% and specificity 19% (NPV 95%). The cost per patient with abnormal perfusion identified for patients below the ultra-sensitive threshold would be $22,997, compared to $4,738 above the cut-off.

Sensitivity Analyses

The ML model had higher prediction performance compared to traditional models in patients with a history of CAD (Supplemental Figure 1) and in patients without a history of CAD (Supplemental Figure 2). Additionally, the ML model had higher performance for predicting abnormal quantitative perfusion (stress TPD > 5%, Supplemental Figure 3).

DISCUSSION

We developed a ML model to predict abnormal perfusion on MPI using traditional pre-test features and evaluated it with both internal cross-validation and external testing. We demonstrated that the ML model was able to incorporate pre-test clinical information to improve prediction performance for abnormal regional myocardial perfusion compared to existing risk models. All models demonstrated modest calibration, but this was superior for the ML method compared to traditional methods. These results suggest the ML could be used to predict likelihood of abnormal perfusion on MPI using pre-test features to may help guide physicians when choosing diagnostic tests. Physicians could potentially utilize these models to identify high-risk patients in whom extent of myocardial ischemia may help guide aggressive therapies(13, 14, 35, 36). Alternatively, patients with a low likelihood of abnormal MPI who have already been sent for testing could be scheduled for stress-first imaging.

We found that discrimination for abnormal perfusion was higher with ML compared to existing risk scores. This was an expected finding, since ML can objectively integrate a larger number of features to classify patients compared to existing prediction models. Additionally, existing risk models have primarily been developed to predict presence of obstructive CAD(15), which may not necessarily be predictive of perfusion abnormalities. In another study, Juarez-Orozco et al. developed a ML model to predict the presence of abnormal myocardial perfusion (37), but also considered functional imaging variables which were not utilized in the present model (since they would only be available after MPI was performed). During internal cross-validation, the model had an AUC of 0.72 (37). Rouhani et al. developed a conventional statistical model for identifying patients with normal MPI from pre-test features, demonstrating an AUC of 0.68 in the derivation cohort (38). External validation is a critical step for evaluating the generalizability of a model and typically demonstrates lower prediction performance (39) as was seen in our study. Assessing model calibration (40), a measure of how well predicted probability matches actual probability, is also important in external populations since this can significantly influence the clinical impact of a risk model. For example, a model could have excellent discrimination (when assessed as a continuous measure) but identify all of the patients as high-risk and therefore have low clinical utility. In our study, the ML model had better calibration compared to existing models in both internal and external testing.

The results of our study may also help physicians better understand which clinical predictors of abnormal perfusion are most important. Features such as prior CAD and exercise stress had high feature importance (a measure of how important a variable was for the model predictions) but are not considered in any of the comparison risk scores (which were developed for patients with suspected CAD) (15). We also confirmed the significance of other features, like typical chest pain (41), which are considered in existing models. While we included features with partially redundant information (for example, known CAD, previous MI, and previous revascularization) ML algorithms are inherently capable of handling interactions between these variables to maximize prediction performance (16). Lastly, out of a possible 23 test indications we identified that suspected angina, troponin elevation, and cardiomyopathy had the highest feature importance. The clinical information required for the ML model is routinely available from electronic medical records, which could be utilized to enable automatic risk predictions when ordering tests. This task could potentially be further simplified by considering ML models with reduced features (42), and integrating multiple imputation methods for handling missing values (43).

The proposed ML model could potentially be used by physicians to help guide test selection. Current guidelines suggest that physicians decide between testing options based on patient likelihood of obstructive CAD (1–3), but suggest that anatomic or functional imaging are appropriate in intermediate risk patients. Our model can help guide physicians to choosing SPECT MPI for patients where it is most likely to change management. In patients with a low probability of abnormal perfusion, subsequent management is more likely to change in response to information regarding absolute myocardial blood flow (8, 9) or anatomic testing (10, 11). From a health care system perspective, the cost required to identify a patient with abnormal perfusion is significantly higher in patients with a low probability of abnormal perfusion and some of those patients could potentially be managed without further imaging (1–3). Nuclear cardiology laboratories could potentially use this tool to aid in patient triaging or to select patients already referred for SPECT MPI for stress-first, and potentially stress-only, imaging to reduce health care costs and radiation exposure. Lastly, physicians could potentially use the ML predictions, which provides a robust estimate of pre-test probability, when adjudicating equivocal MPI findings.

Our study has a few important limitations. The score was derived and tested in patients undergoing SPECT MPI and separate studies would be required to determine if the scores can also predict probability of abnormal myocardial in patients undergoing PET MPI or stress echocardiography. Additionally, while ML demonstrated superior discrimination compared to other risk score the overall prediction performance was poor for all models in patients with known CAD. This suggests that alternative approaches to prediction may be helpful in this important sub-group. Lastly, future studies are needed to determine if using these risk scores to guide diagnostic test selection improved clinical outcomes or reduced costs.

CONCLUSIONS

ML demonstrated high prediction performance for abnormal myocardial perfusion using readily available pre-test information. This model could be used to help guide physician decisions regarding testing strategies.

NEW KNOWLEDGE GAINED.

Using external testing, machine learning identified patients who are more likely to have abnormal myocardial perfusion using readily available clinical information. The model demonstrated higher prediction performance and better calibration compared to traditional risk models.

ABBREVIATIONS

AUC

area under the receiver operating characteristic curve

CAC

coronary artery calcium

CAD

coronary artery disease

CI

confidence interval

ECG

electrocardiogram

MI

myocardial infarction

ML

machine learning

MPI

myocardial perfusion imaging

PCI

percutaneous coronary intervention

PET

positron emission tomography

REFINE SPECT

REgistry of Fast Myocardial Perfusion Imaging with NExt generation SPECT

SPECT

single photon emission computed tomography

SSS

summed stress score

TPD

total perfusion deficit