Validation of a fully automated deep learning-enabled solution for CCTA atherosclerotic plaque and stenosis quantification in a diverse real-world cohort

Coronary CT angiography (CCTA) has proven to be a reliable test for the evaluation of coronary artery stenosis severity and for quantification of the overall burden of coronary atherosclerosis providing incremental prognostic information. Recent advances in CT technology allow for semi-automated measurements of coronary atherosclerotic plaque characteristics with high accuracy as compared to intravascular imaging.¹ However, semi-automated plaque quantification is time-consuming and requires a high level of human expertise. Recently, a novel deep-learning system for CCTA-derived measurements of plaque volume and stenosis severity showed excellent agreement with expert reader measurements and intravascular ultrasound (IVUS), at a fraction of the analysis time taken by experts (3 min vs 25 min).² Moreover, a significant prognostic value was evidenced upon application of this novel system within an external cohort (SCOT-HEART trial) predominantly comprising individuals of White European ancestry. We aimed to validate this fully automated deep learning-enabled solution against expert readers’ semiautomatic measurements using a clinical practice-based US population from diverse race/ethnicity backgrounds.

The patient population consisted of 201 consecutive patients (781 plaque lesions) undergoing a clinically indicated CCTA for the evaluation of CAD from two sites: Oklahoma Heart Institute (n = 121) and Montefiore Health System (n = 80). Images were acquired with ≥64 detector row scanners from different vendors (Siemens, Philips, and GE). Two ground truther’s expert readers (L.S. and D.S.B.), one for each cohort, performed visual and semiautomated measurements with validated research software (Autoplaque [APQ] version 2.5; Cedars-Sinai Medical Center, Los Angeles, CA, USA). Vessel segmentation adjustments were done manually by the experts. Independently, a fully automated deep-learning analysis was performed using APQ v3.0 (Fig. 1, Panel 1), without the expert’s intervention, and the following measurements were compared in a case-by-case approach: total plaque volume (TPV, mm3), calcified plaque volume (CP, mm3), non-calcified plaque volume (NCP, mm3), diameter stenosis (DS, %). Agreement between deep-learning and expert readers was assessed using the intraclass correlation coefficient (ICC) and Bland-Altman analysis. Correlation was evaluated with Spearman’s rank correlation coefficient. The image segmentation performance was assessed by calculating Dice coefficients.

Fig. 1.

Panel 1) Case example using fully automated advanced plaque analysis. Panel 2) Correlation (A) and Bland-Altman analysis (B) for total plaque volume. RCA, right coronary artery.

Overall, there were 106 (53%) men, and 95 (47%) women, with a mean age of 60 12 years. The race/ethnicity background distribution was the following: Non-Hispanic White 43% (n = 86), Hispanic 33% (n = 66), Non-Hispanic Black 14% (n = 28), Asian 3% (n = 6), and Others 7% (n = 15). There was an excellent correlation between the fully automated solution and clinician expert reader semiautomatic measurements for TPV (r = 0.944), NCP (r = 0.939), CP (0.944), and DS (r = 0.895) with p < 0.0001 for all. These measurements also demonstrated very strong-excellent intraclass correlation agreement ranging from 0.717 to 0.941 (p < 0.0001 for all). The correlation and Bland-Altman analysis for TPV (mm3) are shown in Fig. 1, Panel 2. The image segmentation performance was excellent for all plaque characteristics as shown by the Dice coefficients ranging from 0.849 to 0.934.

This study represents the first US-based external validation of a novel, deep-learning enabled fully automated approach for evaluation of CCTA plaque burden and stenosis grading in comparison to the ground truth of Level III expert readers using semi-automated measurements.

Quantitative plaque analysis provides an improved assessment across the entire coronary tree compared to visual assessment and semi-quantitative scores.¹ However, barriers to the implementation of CCTA quantitative plaque analysis in clinical practice include the time to perform this assessment, the level of expertise required, and the lack of standardization of software analysis tools. Recently, Choi et al.³ used a series of FDA-cleared convolutional neural networks to perform CAD-RADS stenosis categorization in 232 patients, with close agreement seen with the consensus of expert readers (κ coefficients of 0.72 at the vessel level and 0.81 at the patient level). Nevertheless, when compared to our cohort, their study had less representation of the female sex (37% vs 47% in our cohort), race/ethnicity background was not reported, and the exams were performed using only 2 CT scanners.

This study is not without limitations. First, although we showed a robust performance of deep learning across several different CT vendors and scan parameters, we excluded CCTA studies of poor image quality that were deemed uninterpretable by expert readers. Second, we did not perform validation with intracoronary imaging in this study but both software versions had been validated already against those modalities. Despite these limitations, our study is one of the first to demonstrate strong agreement between a fully automated deep-learning and expert human readers in quantifying atherosclerotic plaque and stenosis across all-comers clinical practice patients. Our findings show potential for future clinical application.

Data sharing statement

Data can be shared upon reasonable request pending ethics and institutional approvals.