Machine deep learning accurately detects endoleak after endovascular abdominal aortic aneurysm repair

Abstract

Objective

The objective of this study was to develop a machine deep learning algorithm for endoleak detection and measurement of aneurysm diameter, area, and volume from computed tomography angiography (CTA).

Methods

Digital Imaging and Communications in Medicine files representing three-phase postoperative CTA images (N = 334) of 191 unique patients undergoing endovascular aneurysm repair for infrarenal abdominal aortic aneurysm (AAA) with a variety of commercial devices were used to train a deep learning pipeline across four tasks. The RetinaNet object-detection convolutional neural network (CNN) architecture was trained to predict bounding boxes around the axial CTA slices that were then stitched together in two dimensions into a smaller region containing the aneurysm. Multiclass endoleak detection and segmentation of the AAA, endograft, and endoleak were performed on this smaller region. Segmentations on a single randomly selected contrast from each scan included 33 full and 68 partial segmentations for endograft and AAA and 99 full segmentations for endoleak. A modified version of ResNet-50 CNN was used to detect endoleak on individual axial slices. A three-dimensional U-Net CNN model was trained on the task of dense three-dimensional segmentation and used to measure diameter and volume with a specially designed loss function. We made use of fivefold cross-validation to evaluate model performance for each step, splitting training and testing data at each fold, such that multiple scans from the same patient were preserved with the same fold. Algorithm predictions for endoleak were compared with the radiology report and with a subset of CTA images independently read by two vascular specialists.

Results

The localization portion of the network accurately predicted a region of interest containing the AAA in 99% of cases. The best model of binary endoleak detection obtained an area under the receiver operating characteristic curve of 0.94 ± 0.03 with an optimized accuracy of 0.89 ± 0.03 on a balanced data set. An introduced postprocessing algorithm for determining maximum diameter was used on both the predicted AAA segmentation and ground truth segmentation, predicting on average an absolute diameter error of 2.3 ± 2.0 mm by 1.4 ± 1.7 mm for each measurement, respectively. The algorithm measured AAA and endograft volume accurately (Dice coefficient, 0.95 ± 0.2) with an absolute volume error of 10.1 ± 9.1 mL. The algorithm measured endoleak volume less accurately, with a Dice score of 0.53 ± 0.21 and an average absolute volume error of 1.2 ± 1.9 mL.

Conclusions

This machine learning algorithm shows promise in augmenting a human's ability to interpret postoperative CTA images and may help improve surveillance after endovascular aneurysm repair. External validation on larger data sets and prospective study are required before the algorithm can be clinically applicable.

Clinical Relevance

This manuscript describing the application of machine learning for endoleak has clinical relevance to endovascular abdominal aortic aneurysm follow-up. The techniques described herein may be more broadly applied to the diagnosis of aortic disease. In the future, vascular surgeons will benefit from integrating such artificial intelligence algorithms into their practice.

Article Highlights.

• •Type of Research: Basic science
• •Key Findings: A machine deep learning algorithm demonstrated acceptable accuracy in the detection of endoleak on computed tomography scans.
• •Take Home Message: This machine learning algorithm shows promise in augmenting a human's ability to interpret postoperative computed tomography angiography images and may help improve surveillance after endovascular aneurysm repair.

Endoleaks, a well-recognized consequence of endovascular aneurysm repair (EVAR), occur in up to 20% of patients.¹ Whereas the clinical significance of various endoleak types has been debated, the presence of an endoleak is a prognostic marker for poor late outcomes, including abdominal aortic aneurysm (AAA) sac expansion, secondary interventions, and late rupture.2, 3, 4, 5 Surveillance imaging is required after EVAR, and although duplex ultrasound has surfaced as an alternative noninvasive imaging modality, three-phase computed tomography angiography (CTA) remains the “gold standard,” particularly for early imaging and surrounding reinterventions.^1,6 Whereas CTA has good intraobserver and interobserver variability for aneurysm diameter, other parameters, such as diagnosis of endoleak and measurement of AAA volume, have variable accuracy and have received less attention in the literature.^7,8 The diagnostic accuracy of diameter changes in the identification of AAA volume increase has been challenged with a report of good specificity but poor sensitivity.⁹

In recent years, machine deep learning techniques have been used in a variety of health care applications, particularly within the medical imaging field.¹⁰ Whereas success has been reported using segmentation techniques to measure AAA diameter and volume, none have applied sophisticated deep learning techniques to the automated diagnosis of endoleak coupled with these measures.11, 12, 13, 14, 15, 16 Our objective was to develop a deep learning algorithm for endoleak detection and AAA diameter and volume that can augment the human's ability to read CTA images after EVAR to reduce subjectivity and variability while increasing the accuracy of the diagnosis.

Methods

Digital Imaging and Communications in Medicine files representing CTA images (N = 334) of 191 unique, consecutive patients undergoing EVAR at the University of Vermont Medical Center from 2006 to March 2018 with commercially available devices were used to train a deep learning-based pipeline across varied tasks. Three-phase CTA with non-contrast-enhanced arterial and venous phases with 1.5-mm slice thickness were obtained as part of the established standard of care at periodic intervals after EVAR and deidentified by converting them into Neuroimaging Informatics Technology Initiative (NIfTI) file format. Throughout the study period, CTA scans were routinely obtained at 1 month and 1 year after EVAR. In the past 5 years, select patients were followed up by duplex ultrasound after the 1- or 2-year scan at the discretion of the attending surgeon. The mean time between EVAR and each CTA scan was 629 days. Although some patients' CTA data were used more than once, in such cases, the patient's imaging data were never split in a case in which the algorithm would be able to learn explicitly from the patient's earlier scan.

The algorithm (Endo-Detecto) was developed in stages to carry out four tasks: endoleak detection, maximum AAA diameter, AAA volume, and endoleak volume. The pipeline began with a localization network designed to predict bounding boxes around the AAA on all axial slices of the CTA scans using the RetinaNet, one-stage detector styled, convolutional neural network (CNN) architecture (Fig 1).¹⁷ Predicted two-dimensional bounding boxes were then stitched together to represent a smaller 128 × 128-voxel region of interest containing primarily the AAA. This smaller region was used as input for detection of endoleak and multiclass AAA segmentation. A ResNet-50 style CNN was trained at this stage to output axial slice by slice predictions for the presence of endoleak.¹⁸ Prediction generalization from slice by slice was conducted by assigning the maximum individual slice prediction made across both venous and arterial contrast phases to that set of scans. For the localization and binary endoleak detection tasks, 760 contrast phase scans were drawn from the available scans, which constituted all arterial and venous phase CTA scans with an endoleak as well as a random mix of arterial and venous phase scans from cases without endoleak with the pre-contrast scan used for every case. The 334 scans were evenly split by case and control and labeled as endoleak yes or no. Next, the classification threshold (ie, that a given prediction needs to be greater than in order to be labeled yes for endoleak) was optimized as the value that maximizes area under the curve receiver operating characteristics on the training set. During training, various data augmentation strategies, including random rotations and flips, were applied to reduce overfitting. In addition, various hyperparameters were investigated, making use ultimately of the following training techniques; snapshot ensembles >100 epochs with a starting cyclic learning rate of 0.01 along with the built-in Keras stochastic gradient descent optimizer were employed to improve predictive performance. We made use of fivefold cross-validation as a method of evaluating endoleak detection performance, splitting training and testing data at each fold by patient.

Fig 1.

Overview of the pipeline illustrating use of three modified convolutional neural networks (CNNs). AAA, Abdominal aortic aneurysm; CTA, computed tomography angiography; 3D, three-dimensional.

AAA-predicted bounding boxes were stitched together into a three-dimensional (3-D) region of interest. A 3D U-Net CNN, with residual connections, was trained on the task of segmenting AAA and endograft within the cropped 3D AAA volumes. As generating full dense segmentations is time intensive (∼1 hour per scan), the 3D U-Net architecture was modified with a specially designed multiclass Dice loss capable of accepting partial 3D segmentations according to the following formula:

$$Loss=\frac{\forall i\in |classes|\frac{2\ast \sum Tru{e}_i\cap \mathrm{Pr}e{d}_i}{\sum Tru{e}_i+\sum \mathrm{Pr}e{d}_i\cap Seen}}{|classes|}$$

where classes refer to the different input classes (AAA and endograft), seen refers to a binary mask indicating whether that voxel was seen while performing partial segmentation, and True_i and Pred_i refer to the ground truth label and predicted label for class i.

We performed manual aortic aneurysm dense segmentation on 101 scans (33 complete, 68 partial segmentation) to train the segmentation task where scans are defined as one contrast series. Partial segmentations are composed of three to five manually labeled slices within each view (axial, sagittal, coronal), requiring on average 10 minutes. Complete segmentations included labeling of every slice. For the diameter task, 168 contrast scans (both arterial and venous per CTA scan) were used because the non-contrast-enhanced scans were lower resolution. We further made use of the 33 fully segmented scans in reporting cross-validated network performance on the prediction of AAA and endograft volume. Minor postprocessing was applied on the outputted predictions from the 3D U-Net, automatically fixing outliers in predicted class on the basis of surrounding voxel classes. Another algorithm was then applied to extract maximal anterior-posterior diameters from outputted network predictions.

The gold standard for endoleak detection and maximum aneurysm diameter in the axial plane was the official radiology report, obtained from electronic medical records of each patient. In addition, a subset of 100 CTA images were independently read for endoleak and diameter by an interventional radiologist and a vascular surgeon (C.M., D.B.). Interobserver and human to machine AAA diameter measurement variability was evaluated by Bland-Altman analysis. Precision was tested by calculating the coefficient of repeatability, which represents the value below which the absolute difference between two repeated test results may be expected to lie with a probability of 95%.

The study followed the principles outlined in the Declaration of Helsinki. The University of Vermont Institutional Review Board has approved the research protocol with a waiver of informed consent.

Results

AAA localization

The number of scans varied for each patient, ranging from 1 to 12 over time, and is shown in Fig 2. The RetinaNet-based localization network had an accuracy of 98.7% in capturing the AAA within the region of interest on the CTA image. We base this estimation of accuracy on manual review and correction of 314 previously unseen scans that were supplied to the localization network. These scans were composed of 91,575 axial slices, of which 412 (0.4%) required minor manual operator revision, defined as small changes made to a bounding box or to a slice missed before or after the predicted aneurysm region. These revisions may be caught by automated padding. We observed only two predicted slices with a more significant bounding box error. A significant number of AAA predictions were missed altogether on only four full scans.

Fig 2.

Distribution of computed tomography (CT) scans across patients.

Binary endoleak classifier

As computed across fivefold cross-validation, the binary classification portion of the pipeline correctly identified the presence of an endoleak in 147 of 167 scans. Absence of endoleak was correctly reported in 151 of 167 scans. The resulting sensitivity and specificity for endoleak detection were 90% and 88%, respectively, with an accuracy of 89% (Fig 3). The positive and negative predictive values were 89% and 91%, respectively. The area under the receiver operating characteristic curves per validation fold were between 0.89 and 0.97, with an average of 0.94 ± 0.03 (Fig 4). The distribution of endoleaks was as follows: type I, n = 15; types I and II, n = 1; type II, n = 141; types II and III, n = 2 and 3; type III, n = 1; and indeterminate.

Fig 3.

Endoleak confusion matrix depicting true and false positives and negatives.

Fig 4.

Receiver operating characteristic curve for binary classification of endoleak.

Segmentation

As determined by cross-validated performance on 33 fully segmented scans, the algorithm accurately segmented the AAA with a Dice coefficient of 91% ± 5% and intersection over union of 84% ± 9% (Table). The accuracy of endograft segmentation was higher with a Dice coefficient of 95% ± 3% and intersection over union of 90% ± 5%.

Table.

Results of segmentation of aneurysm and endograft

Category	Dice coefficient, %	Intersection over union, %	Absolute volume, mL	Volume, % change
AAA	91 ± 5	84 ± 9	4.5 ± 3.4	5.6 ± 3
Endograft	95 ± 3	90 ± 5	5.2 ± 6.7	4.7 ± 6
AAA and endograft	95 ± 2	91 ± 4	6.3 ± 5.6	3.3 ± 3

AAA, Abdominal aortic aneurysm.

AAA diameter

We compared the deep learning algorithm prediction of maximum diameter in two directions with 84 official radiology reports that listed both maximal AAA diameters and with the average of two expert readers. In comparing the average of two readers with the deep learning algorithm, the coefficient of repeatability was 13.72 (Fig 5, A). In comparing the radiology report with the deep learning algorithm, the coefficient of repeatability was 15.06 (Fig 5, B). Last, we compared the 33 fully segmented scans between the algorithm for extracting maximal diameter on human ground truth and the same algorithm on predicted segmentation, with a coefficient of repeatability of 5.42 (Fig 5, C).

Fig 5.

Bland-Altman plots of abdominal aortic aneurysm (AAA) diameter measurements (sum of both maximal measurements M1 + M2) comparing deep learning (DL) algorithm with (A) average (Avg) of two expert human readers and (B) the radiology report (84 cases) and (C) comparing algorithm on ground truth vs predicted segmentation (22 cases). CR, Coefficient of repeatability; SD, standard deviation.

AAA volume

We computed 3D segmentation network performance on the 33 fully segmented scans with fivefold cross-validation, in which all partial segmentations were included within the training set within all folds. The network obtained a Dice coefficient of 91% ± 5% for AAA volume with a 4.5 ± 3.4 mm³ absolute volume error. The network obtained a Dice coefficient of 91% ± 5% for predicting endograft volume with a 5.2 ± 6.7 mm³ absolute volume error.

Endoleak volume

The algorithm's ability to measure endoleak volume was less accurate, with Dice coefficient of 0.53 ± 0.2 and an average error of 1.2 ± 1.9 mL.

Discussion

We report the novel application of three CNNs for the detection of endoleak on CTA images after EVAR. CNNs are a subclass of deep neural networks most commonly applied to the analysis of images. CNNs were designed to resemble the connectivity of neurons as found in the animal visual cortex, in which individual neurons respond to stimuli within one region of the visual field and different neurons overlap to cover the entire area. Deep learning approaches have overwhelmingly outperformed state of the art in many traditional computer vision tasks, such as image classification and object detection.¹⁹ Specific deep network architectures, like the CNN, have been used for classification and segmentation with great success, obtaining state-of-the-art results in classic vision problems. These deep neural network configurations have been shown to learn useful distributed feature representations for many complex, high-dimensional data sets and to construct sophisticated domain-specific features without relying on precise human-crafted input representations.^20,21 The Endo-Detecto algorithm accurately diagnosed the presence or absence of an endoleak on post-EVAR CTA images, reaching the level of a human reader. Whereas CTA has high accuracy for the diagnosis of endoleak with a reported sensitivity of 83% and specificity approaching 100%, relatively little attention has been devoted to the physician variability in endoleak diagnosis.^7,22 One small study reported sensitivities of 81% to 85% in the detection of type II endoleak using two untrained observers.⁸ The algorithm readily identified the region of the AAA and segmented the aneurysm from the device reliably with minimal human optimization. The algorithm measured AAA diameter comparably to both the radiology report and two independent readers. Unlike radiology reports, which sometimes omitted diameter values, the algorithm automatically reports two diameter measurements and can be set to report as many as needed. The variability in AAA volume was within the 5% range generally acceptable in clinical practice. The ability to measure endoleak volume was not sufficiently accurate and will require further training and optimization.

A machine learning algorithmic approach to the interpretation of computed tomography (CT) scans after EVAR has several possible clinical applications. First, several studies have advocated for the addition of AAA volume to the clinical follow-up and decision-making for endoleak treatment.²³ Relatively small changes in diameter can be associated with substantial changes in volume.²⁴ Second, patients are subjected to many CT scans, both for lifelong postoperative surveillance and for other indications. Review of prior imaging for comparison purposes can be expedited using a machine learning algorithm, thus improving the speed, ease, and rigor of follow-up. Comparisons of endoleak, AAA diameter, and AAA volume can be quickly tabulated for reference. Third, the volume of type II endoleak and source and diameter of the feeding artery have been reported to have prognostic value.25, 26, 27, 28 However, parameters such as the volume of endoleak are currently not readily available from CT scans. In an effort to reduce radiation exposure and intravenous administration of iodinated contrast media, some radiologists have suggested a staged approach to imaging with a review of the non-contrast-enhanced scan before conducting an arterial and venous phase. For patients with a decrease in diameter and volume, the contrast phase of the scan could be avoided in appropriately selected EVAR follow-up cases. However, at this time, this protocol is not practical because of the need for real-time reading by the radiologist and communication with the physician ordering the image. An automated program could solve this problem by enabling radiologists to focus their time and workflow. Finally, automation of endoleak volume would also be useful in clinical research to further refine the natural history of this common problem. Expanding the algorithm's capability to other anatomic parameters, such as the detection of thrombus position and coupling it with patient data, may allow endoleak prediction in the future.

Limitations to this work include that it was a single-center study without validation across a larger series of CTA studies. Further work is needed to train and to test the algorithm on edge cases. For example, we excluded CT images with prior embolization in this analysis. The algorithm must also be tested across a greater variety of devices as some have specific characteristics that may affect the interpretation.²⁹ We did not design the algorithm to determine the type of endoleak or to distinguish thrombus as others have reported.¹⁵ Further work is needed to identify characteristics that predict the source and origin of the endoleak. Finally, institutional imaging protocols, imaging parameters, and CT scanner manufacturers vary with respect to routine three-phase imaging factors, slice thickness, intravascular iodinated contrast media volume, and other technical specifications that must be considered in validating the algorithm.

Conclusions

This machine learning algorithm shows promise in augmenting a human's ability to interpret postoperative CTA images and may help improve post-EVAR surveillance.

Author contributions

Conception and design: SH, SW, DB

Analysis and interpretation: SH, CM, SW, DB

Data collection: SH, MP, CM, DB

Writing the article: SH, DB

Critical revision of the article: SH, MP, CM, SW, DB

Final approval of the article: SH, MP, CM, SW, DB

Statistical analysis: SH

Obtained funding: DB

Overall responsibility: DB

From the Society for Vascular Surgery