Deep Learning-Based Lumen and Vessel Segmentation of Intravascular Ultrasound Images in Coronary Artery Disease

Author's summary

Deep learning algorithms allow automatic learning without explicit programming, potentially improving diagnostic accuracy. The Efficient-UNet showed good performance in delineating vascular geometry on the grayscale intravascular ultrasound image. This data-driven approach may support clinicians in evaluating coronary artery morphology and making clinical decisions during percutaneous coronary intervention.

Abstract

Background and Objectives

Intravascular ultrasound (IVUS) evaluation of coronary artery morphology is based on the lumen and vessel segmentation. This study aimed to develop an automatic segmentation algorithm and validate the performances for measuring quantitative IVUS parameters.

Methods

A total of 1,063 patients were randomly assigned, with a ratio of 4:1 to the training and test sets. The independent data set of 111 IVUS pullbacks was obtained to assess the vessel-level performance. The lumen and external elastic membrane (EEM) boundaries were labeled manually in every IVUS frame with a 0.2-mm interval. The Efficient-UNet was utilized for the automatic segmentation of IVUS images.

Results

At the frame-level, Efficient-UNet showed a high dice similarity coefficient (DSC, 0.93±0.05) and Jaccard index (JI, 0.87±0.08) for lumen segmentation, and demonstrated a high DSC (0.97±0.03) and JI (0.94±0.04) for EEM segmentation. At the vessel-level, there were close correlations between model-derived vs. experts-measured IVUS parameters; minimal lumen image area (r=0.92), EEM area (r=0.88), lumen volume (r=0.99) and plaque volume (r=0.95). The agreement between model-derived vs. expert-measured minimal lumen area was similarly excellent compared to the experts' agreement. The model-based lumen and EEM segmentation for a 20-mm lesion segment required 13.2 seconds, whereas manual segmentation with a 0.2-mm interval by an expert took 187.5 minutes on average.

Conclusions

The deep learning models can accurately and quickly delineate vascular geometry. The artificial intelligence-based methodology may support clinicians’ decision-making by real-time application in the catheterization laboratory.

Graphical Abstract

INTRODUCTION

Intravascular ultrasound (IVUS) is a useful tool for planning percutaneous coronary intervention (PCI) as it provides information to the operators on lesion severity, plaque characteristics, reference vessel size, and lesion length. By using validated imaging criteria, IVUS has been used for clinical decision-making and PCI optimization through the correction of stent under-expansion and edge problems and prevention of procedural complications to eventually prevent stent failure, such as stent thrombosis and restenosis.¹⁾,²⁾,³⁾ Cross-sectional and volumetric measurements of the lumen and external elastic membrane (EEM) as well as the plaque plus media thickness between those two structures are used to evaluate vessel geometry. Device sizing or stent optimization depends on the geometrical assessment of the stenotic segment. About the natural history of coronary atherosclerosis, the plaque burden, tissue composition, and vascular remodeling constitute essential predictors of the risk of adverse cardiac events.⁴⁾ Meticulous segmentation of IVUS images is crucial in assessing vessel geometry, including the area and diameter, in quantifying atheroma and defining the region of interest (ROI) for plaque characterization. A single IVUS pullback contains thousands of images; therefore, the manual segmentation of the lumen and frame-by-frame measurement of the EEM constitutes a time-consuming and error-prone task. Moreover, the traditional IVUS criteria usually measured within one selected frame may be insufficient to reflect the status of an entire vascular segment. Therefore, a rapid, accurate algorithm for automatic segmentation of the whole sequence of pullback images is needed to facilitate the on-site utilization of IVUS and real-time decision-making in the catheterization laboratory.

Recently, convolutional neural networks (CNNs) have been adopted in many computer vision applications for lesion detection and semantic segmentation in various domains. A few studies have demonstrated the good performance of deep learning algorithms to delineate the lumen and vessel of coronary arteries; however, those studies comprised models that were developed using images from a small number of cases and cannot be applied to the general population with diverse lesion characteristics.⁵⁾,⁶⁾,⁷⁾ Furthermore, the agreement of the model-derived cross-sectional and volumetric measurements with expert analysis has not been validated adequately. Thus, this study was conducted to develop a CNN-based automatic segmentation algorithm and to evaluate the performance of the algorithm for the calculation of quantitative IVUS parameters in patients with coronary artery disease.

METHODS

Ethical statement

The protocol for this retrospective data analysis was approved by the Institutional Review Board of Asan Medical Center (2016-1281), and the requirement of written informed consent from the participants was waived.

Study population

Between November 2012 and July 2015, 1,357 patients underwent pre-procedural IVUS to assess for coronary stenosis with angiographic diameter stenosis >40% on visual estimation at Asan Medical Center, Seoul, Republic of Korea. In patients with IVUS pullbacks of 2 or more vessels, the vessel with the most severe stenosis was selected. After excluding 228 cases with stented lesions, 47 cases with chronic total occlusion, 14 cases with poor imaging quality due to severe non-uniform deformity or air bubble artifact, and 5 cases with technical errors in the imaging files, a final cohort of 1,063 coronary arteries was included in this retrospective analysis. The patients were randomly assigned (4:1) to the training and test sets. The independent data set of 111 IVUS pullbacks obtained between January and March 2016 was also used to assess the vessel-level performance.

Acquisition of intravascular ultrasound

After intracoronary administration of 0.2 mg nitroglycerin, grayscale IVUS imaging was performed using a motorized transducer pullback (0.5 mm/s) and a commercial scanner (Boston Scientific/SCIMED, Minneapolis, MN, USA) consisting of a rotating 40-MHz transducer within a 3.2-F imaging sheath. An ROI was defined as the segment from the ostium to a point located 10 mm distal to the lesion (maximal plaque thickness >0.5 mm).

Model development

The experienced user manually labeled the lumen and vessel boundaries in every IVUS frame with a 0.2 mm interval (approximately every 12th frame). Lumen segmentation was undertaken based on the interface between the lumen and the leading edge of the intima. A discrete interface at the border between the media and the adventitia corresponded approximately to the location of the EEM.

The overall workflow of model development is shown in Figure 1. The proposed method comprised three steps: First, we decomposed an IVUS image into multiple 2-dimensional (2D) segmentations of frames. For 2D segmentation, the fully convolutional network (FCN) with pre-trained weights from EfficientNet trained on an ImageNet database was used (Supplementary Appendix). The skip connections to the model that combined the hierarchical features from the convolutional layers with the different scales were applied. The neural network model, which we named Efficient-UNet, was utilized for developing the proposed segmentation method (Supplementary Figure 1). Among the various subtypes demarcated based on the complexity of EfficientNet (B0–B7), the EfficientNet-B2 layer was applied because of the inference time and performance. Supplementary Table 1 shows the encoding layer performance of the proposed network. After leveraging the data generated in consecutive frames, the five displacement values of 0, 1, 2, 3, and 4 frames were applied. Five extracted masks were fused to generate one result mask using the Ensemble.

Figure 1. Workflow for developing the Efficient-UNet for IVUS segmentation.

IVUS = intravascular ultrasound.

Using a mini-batch size of 8 images, the Adam optimizer for 100 epochs was applied with an initial learning rate of 0.0001. Due to multiple classes’ nature, cross-entropy was used as a loss function. The detailed methods of implementation details, data augmentation, and pre-and post-processing are described in the Supplementary Appendix.

Cross-sectional images were segmented into three compartments: 1) the adventitia, including the pixels outside the EEM (coded as “0”); 2) the lumen, including the pixels within the lumen border (coded as “1”); and 3) the plaque, including the pixels between the lumen border and EEM (coded as “2”). To calibrate the pixel dimensions, grid lines were automatically applied in the IVUS images, and the pixel spacing was calculated for extracting the IVUS parameters.

To assess the extent of overlap and distance between the model-predicted vs. human-measured lumen and the frame-level EEM areas, five metrics—the dice similarity coefficient (DSC), Jaccard index (JI), surface DSC, mean surface distance, and Hausdorff Distance—were derived.

To compare the model’s (vs. experts) performance in the independent dataset that included the 111 IVUS pullback, the manual segmentation and IVUS measurement were undertaken with the offline software (Echoplaque 3.0; Indec Systems, Mountain View, CA, USA).

Statistical analysis

All values are expressed as mean±1 standard deviation (continuous variables) or count and percentage (categorical variables). Continuous variables are compared using unpaired t-tests, and categorical variables are compared using χ² statistics. A p value <0.05 was considered statistically significant. Between the ground truth and predicted values, Bland-Altman was used to evaluate the agreement, and Pearson product-moment correlation was used to assess the linear correlation. Statistical analyses of the patient and lesion characteristics at baseline were performed using SPSS (version 10.0; SPSS Inc., Chicago, IL, USA).

RESULTS

Clinical and lesion characteristics

In the study cohort, the mean age was 64.0±9.6 years, and 76% were men. The target vessels were the left anterior descending artery in 75%, the left circumflex artery in 3%, the right coronary artery in 19%, the ramus intermedius in 2%, and the left main coronary artery in 1% of the cohort. The incidences of diabetes mellitus, hypertension, and hyperlipidemia were 33%, 54%, and 31%, respectively.

Frame-level performance of the model

Based on the evaluation metrics, the baseline model performance for lumen and EEM segmentation has been summarized in Table 1. For lumen segmentation, Efficient-UNet (vs. DeepLabv3Plus vs. RefineNet vs. UNet) showed a higher DSC (0.93±0.05 vs. 0.91±0.06 vs. 0.89±0.11 vs. 0.92±0.05) and JI (0.87±0.08 vs. 0.85±0.09 vs. 0.82±0.14 vs. 0.86±0.09) and demonstrated better DSC (0.97±0.03 vs. 0.95±0.05 vs. 0.95±0.06 vs. 0.96±0.04) and JI (0.94±0.04 vs. 0.91±0.08 vs. 0.91±0.09 vs. 0.92±0.07) for EEM segmentation. Table 2 summarizes the evaluation metrics of EfficientNet-B2 with different input channels. The ensemble of the five models for displacement in consecutive frames shows the best performance for lumen (DSC, 0.94±0.04) and EEM (DSC, 0.97±0.02) segmentation.

Table 1. Performance of IVUS segmentation models in the test set.

Model	Lumen					EEM
	DSC	JI	SDSC	MSD (mm)	HD (mm)	DSC	JI	SDSC	MSD (mm)	HD (mm)
Efficient-UNet (proposed model)	0.93±0.05	0.87±0.08	0.65±0.19	0.04±0.03	0.16±0.15	0.97±0.03	0.94±0.04	0.78±0.18	0.03±0.03	0.13±0.14
DeepLabv3Plus (xception)	0.91±0.06	0.85±0.09	0.59±0.20	0.08±0.12	0.40±0.67	0.95±0.05	0.91±0.08	0.73±0.19	0.08±0.12	0.42±0.66
4-cascaded RefineNet (resnet101)	0.89±0.11	0.82±0.14	0.56±0.24	0.06±0.05	0.20±0.16	0.95±0.06	0.91±0.09	0.74±0.21	0.05±0.08	0.24±0.31
UNet	0.92±0.05	0.86±0.09	0.63±0.19	0.07±0.08	0.39±0.62	0.96±0.04	0.92±0.07	0.74±0.19	0.08±0.11	0.43±0.61

Values are shown as mean±standard deviations.

DSC = dice similarity coefficient; EEM = external elastic membrane; HD = Hausdorff distance; IVUS = intravascular ultrasound; JI = Jaccard index; MSD = mean surface distance; SDSC = surface dice similarity coefficient.

Table 2. Performance of EfficientNet-B2 with different input channels in the test set.

No of displacement consecutive frames	Lumen					EEM
	DSC	JI	SDSC	MSD (mm)	HD (mm)	DSC	JI	SDSC	MSD (mm)	HD (mm)
0	0.93±0.05	0.86±0.08	0.62±0.19	0.08±0.10	0.50±0.79	0.97±0.03	0.93±0.05	0.76±0.18	0.06±0.08	0.42±0.69
1	0.93±0.05	0.87±0.08	0.65±0.19	0.04±0.03	0.16±0.15	0.97±0.03	0.94±0.04	0.78±0.18	0.03±0.03	0.13±0.14
2	0.93±0.05	0.87±0.08	0.64±0.20	0.06±0.06	0.48±0.77	0.96±0.04	0.93±0.07	0.75±0.20	0.06±0.11	0.42±0.68
3	0.93±0.05	0.87±0.08	0.64±0.21	0.05±0.04	0.45±0.73	0.97±0.03	0.94±0.05	0.77±0.18	0.04±0.04	0.37±0.62
4	0.93±0.05	0.87±0.08	0.65±0.19	0.04±0.04	0.17±0.24	0.96±0.03	0.93±0.05	0.76±0.19	0.04±0.07	0.19±0.31
Ensemble*	0.94±0.04	0.89±0.06	0.76±0.19	0.04±0.06	0.18±0.27	0.97±0.02	0.94±0.04	0.82±0.16	0.03±0.03	0.13±0.17

Values are shown as mean±standard deviations.

DSC = dice similarity coefficient; EEM = external elastic membrane; HD = Hausdorff distance; JI = Jaccard index; MSD = mean surface distance; SDSC = surface dice similarity coefficient.

^*Ensemble of the five models for displacement consecutive frames.

Vessel-level performance of the model

The vessel-level performances were tested in the independent test set that included 111 IVUS pullbacks with lesion lengths of 32.0±14.1 mm (including 64.0±28.2 frames). Table 3 presents a comparison of the Efficient-UNet- vs. expert-measured vessel-level IVUS parameters. Table 4 shows the correlations between the proposed model-derived and expert-measured IVUS parameters per vessel. Figure 2 shows the Bland-Altman plots between the proposed model-derived and expert-measured minimal lumen area. The correlation between the proposed model-derived and expert-measured minimal lumen area is shown in Figure 3. The agreement between model-derived vs. expert-measured minimal lumen area within the lesion was similarly excellent compared to the experts' agreement. The cross-sectional and volumetric IVUS measurements derived from the model vs. experts were compared in Supplementary Figures 2 and 3. The intra- and interobserver variations in the core laboratory analysis are summarized in Supplementary Table 2. Their limits of agreement (LOA) are shown in Supplementary Table 3. The interobserver variations are demonstrated in Supplementary Figures 4 and 5, and their LOA were included in Supplementary Table 4.

Table 3. Vessel-level comparison of the proposed model-derived vs. expert-measured IVUS parameters.

IVUS parameters		Efficient-UNet	Expert 1	Expert 2
Within lesion segment
	Minimal lumen area (mm2)	2.7±1.0	2.5±1.1	2.6±1.0
	EEM at the MLA site (mm2)	13.1±4.5	13.1±4.9	12.6±4.8*
	Plaque burden at the MLA site (%)	78.4±7.1	78.8±9.1	77.2±9.6
	Mean lumen diameter (mm)	1.8±0.3	1.8±0.3	1.8±0.3
	Lumen volume (mm3)	189.6±91.1	179.4±85.1	174.9±83.8
	EEM volume (mm3)	488.2±241.0	464.1±224.8	443.3±214.3
	Plaque volume (mm3)	298.6±163.6	285.7±150.2	268.3±140.6
Proximal reference 5-mm segment
	Minimal lumen area (mm2)	8.9±2.5	9.7±2.8	9.3±2.7
	EEM at the MLA site (mm2)	17.9±4.9	18.2±5.1	17.5±4.8
	Plaque burden at the MLA site (%)	50.0±8.6	46.3±9.1	46.3±8.5
	Mean lumen diameter (mm)	3.3±0.5	3.5±0.5	3.4±0.5
Distal reference 5-mm segment
	Minimal lumen area (mm2)	5.5±2.1	5.7±2.3	5.5±2.2
	EEM at the MLA site (mm2)	11.4±4.4	10.3±4.1	9.9±3.8
	Plaque burden at the MLA site (%)	49.9±12.6	43.6±11.1*	43.5±10.5*
	Mean lumen diameter (mm)	2.6±0.5	2.6±0.5	2.6±0.5

EEM = external elastic membrane; IVUS = intravascular ultrasound; MLA = minimum luminal area.

^*p values <0.05 (vs. model).

Table 4. Correlations between Efficient-UNet-derived and expert-measured IVUS parameters per vessel.

IVUS parameters		Correlation coefficient†
		vs. Expert 1*	vs. Expert 2*
Within lesion segment
	Minimal lumen area (mm2)	0.92	0.93
	EEM at the MLA site (mm2)	0.88	0.87
	Plaque burden at the MLA site (%)	0.86	0.85
	Mean lumen diameter (mm)	0.92	0.93
	Lumen volume (mm3)	0.99	0.99
	EEM volume (mm3)	0.99	0.99
	Plaque volume (mm3)	0.95	0.96

EEM = external elastic membrane; IVUS = intravascular ultrasound; MLA = minimum luminal area.

^*All p values <0.001; ^†Pearson product-moment correlation coefficient for assessing the linear correlation.

Figure 2. Bland-Altman between the proposed model-derived and expert-measured minimal lumen area.

Figure 3. Correlation between the proposed model-derived and expert-measured minimal lumen area.

The inference time of the lumen and EEM segmentation by Efficient-UNet was 0.132 seconds per section. The model-based lumen and EEM segmentation for a 20-mm lesion segment required 13.2 seconds, whereas manual segmentation with a 0.2-mm interval by an expert took 187.5 minutes on average.

DISCUSSION

Quantitative IVUS measurements are based on the lumen and vessel segmentation and provide useful information on the coronary artery morphology to support interventionists in clinical decision-making. Given the lack of digital solutions for accurately identifying the vascular geometry, IVUS analysis has been conducted by experts who manually annotate the lumen and EEM borders. However, this approach has many pitfalls, including high intra- and interobserver variabilities. Although every 60th (1-mm interval) or 30th (0.5-mm interval) image is selected and segmented manually, volumetric analysis in a whole pullback usually takes several hours. As frame-by-frame manual annotations of the entire IVUS pullback are laborious and time-consuming, the on-site intraprocedural IVUS assessment procedure depends on a quick analysis that involves only a few sections.

In interventional cardiology, there is an unmet need for automated IVUS analysis. However, IVUS segmentation programs have not been employed widely in real-world clinical practice, possibly due to the poor performance and limited availability of such programs. There are many challenges of IVUS images, including common artifacts (e.g., guide-wire artifacts, non-uniform rotational distortion, motion artifacts, reverberations, side lobe artifacts, and blood-speckle artifacts). Especially in attenuated or calcified plaques with acoustic shadowing, stented segments, and bifurcation sites, delineating the EEM border with a consistent criterion may prove challenging.

A large number of IVUS segmentation methodologies that incorporate edge-tracking and gradient-based techniques,⁸⁾ snake models,⁹⁾ or probabilistic-based approaches¹⁰⁾ have been studied; however, the performance of the abovementioned techniques for detecting EEM and lumen borders was suboptimal. CNNs have been designed to automatically and adaptively ascertain the spatial hierarchies of features through backpropagation. This data-driven approach can be utilized in medical imaging to develop various predictive models. Recently, CNNs have been adopted in many computer vision applications, including lesion detection and semantic segmentation. The performance of deep learning models for automatic IVUS segmentation has since increased. The previous approaches were based on a U-Net architecture that comprised encoder and decoder schemes. Yang et al.¹¹⁾ demonstrated that an FCN called Dual Path U-Net outperformed conventional computer vision-based approaches for segmenting the lumen and EEM.

Based on the rapid evolution of the algorithm, the encoder component of U-Net has been changed. Nishi et al.¹²⁾ developed the IVUS segmentation model by applying DeepLabv3. The DSCs of the model were 0.83 in overall and 0.92 in native coronary arteries. From an engineering point of view, our current model using both U-Net and Efficient-Net showed better DSCs (0.93±0.05 and 0.97±0.03 for lumen and EEM segmentation, respectively). Moreover, we compared the diagnostic accuracies among Efficient-UNet vs. various algorithms (Table 1). In the clinical aspect, the previous study was focused on the matrices-based evaluation only at the frame level, while our current study validated both frame- and vessel-level performance. We tested the cross-sectional and volumetric parameters generally used in actual practice for evaluating lesion severity and prognostic implication and determining treatment strategy. Although there is a concern about the poor reproducibility of IVUS measurement potentially affecting the quality of data labeling, the degree of interobserver variance was not considerable in this study. We also validated the excellent agreement between the model-derived vs. the expert-measured IVUS parameters by meticulous core laboratory analysis. The quick and accurate segmentation of IVUS images can save time and expenses for manual contouring and reduce the clinicians’ workload. By eliminating the effect of human subjectivity or uncertainty of interpretation, this data-driven approach will be helpful for the effective diagnosis of coronary atherosclerosis and clinical decision-making.

Nonetheless, there are some limitations of this study. First, the cases with poor image quality from severe non-uniform rotational deformity or air bubble artifacts were excluded from this analysis. Model performance might decrease in non-selected cases in real-world practice without an expert preview. Second, our models developed using 40-Mhz IVUS images cannot be extrapolated to those obtained by 20- or 60-MHz IVUS. As another pitfall, this current study does not include external validation. The performance of the models needs to be validated in a multicenter cohort. As this analysis included only native coronary arteries, the models should be tuned for segmenting stented lesions. The models’ ability to delineate the neointimal border should be tested to evaluate the lesions with in-stent restenosis. More cases with intimal dissection, intraluminal thrombus, and nodular calcification should be included for model training to improve the segmentation performance. To assess the complex bifurcation lesions, the establishment of expert consensus for the lumen and EEM segmentation of side branches is a prerequisite for labeling the training data.

In conclusion, the deep learning models performed well in the lumen and EEM segmentation at both frame and vessel levels. This artificial intelligence-based methodology can accurately and quickly delineate vascular geometry and has considerable potential for real-time application in the catheterization laboratory.

SUPPLEMENTARY MATERIALS

Supplementary Appendix

Algorithm for IVUS segmentation

Supplementary Table 1

The encoding layer performance of the proposed network

Supplementary Table 2

Intra- and inter-observer variabilities of IVUS measurements

Supplementary Table 3

LOA of Bland-Altman between model-derived and expert-measured IVUS measurements

Supplementary Table 4

LOA of Bland-Altman between inter-observer of IVUS measurements

Supplementary Figure 1

The architecture of the proposed Efficient-UNet framework for semantic segmentation.

Supplementary Figure 2

Bland-Altman between model-derived and expert-measured IVUS measurements.

Supplementary Figure 3

Correlation between the proposed model-derived and expert-measured IVUS measurements.

Supplementary Figure 4

Bland-Altman between inter-observer of IVUS measurements.

Supplementary Figure 5

Correlation between inter-observer of IVUS measurements.