Aortic Annulus Detection Based on Deep Learning for Transcatheter Aortic Valve Replacement Using Cardiac Computed Tomography

Abstract

Background

To propose a deep learning architecture for automatically detecting the complex structure of the aortic annulus plane using cardiac computed tomography (CT) for transcatheter aortic valve replacement (TAVR).

Methods

This study retrospectively reviewed consecutive patients who underwent TAVR between January 2017 and July 2020 at a tertiary medical center. Annulus Detection Permuted AdaIN network (ADPANet) based on a three-dimensional (3D) U-net architecture was developed to detect and localize the aortic annulus plane using cardiac CT. Patients (N = 72) who underwent TAVR between January 2017 and July 2020 at a tertiary medical center were enrolled. Ground truth using a limited dataset was delineated manually by three cardiac radiologists. Training, tuning, and testing sets (70:10:20) were used to build the deep learning model. The performance of ADPANet for detecting the aortic annulus plane was analyzed using the root mean square error (RMSE) and dice similarity coefficient (DSC).

Results

In this study, the total dataset consisted of 72 selected scans from patients who underwent TAVR. The RMSE and DSC values for the aortic annulus plane using ADPANet were 55.078 ± 35.794 and 0.496 ± 0.217, respectively.

Conclusion

Our deep learning framework was feasible to detect the 3D complex structure of the aortic annulus plane using cardiac CT for TAVR. The performance of our algorithms was higher than other convolutional neural networks.

Graphical Abstract

INTRODUCTION

Aortic valve stenosis is a famous valvular heart disease in the world.1 Transcatheter aortic valve replacement (TAVR) should include analyzing a number of significant valve parameters before the surgical treatment. Palmer et al.2 developed algorithms to segment the aortic root. The main issues in reconstruction of aortic valve shape are visualization of valve function in real time and measurement of aortic root diameter in clinical settings. Another study aimed to reconstruct an aortic valve for patient-specific modeling using cardiac computed tomography (CT) images.3 Previously, methods for measuring valve annuli motion using cardiac magnetic resonance (MR) have utilized semi-automated feature tracking methods.4 Most cardiac MR tracking tools monitor the blood-myocardium boundary using maximum likelihood estimation.5

Deep learning6 is a subfield of machine learning inspired by artificial neural networks.7 Recently, an explosion of deep learning methods, such as convolutional neural networks (CNNs), multilayer perceptron, and recurrent neural network, has occurred with promise for predicting disease risks from medical datasets. Another study using deep learning8 robustly identified ten valve landmarks from orientations of three different long-axis MR images acquired in clinical setting. Recently, semantic segmentation using deep learning is important to infer meaningful parts with predefined annotations for each pixel in the medical datasets.9,10,11 Several studies have introduced segmentation algorithms based on CNNs combined with conditional random fields.12 Further, a fully convolutional network (FCN)13 and DeepLab14 have been introduced to improve the outputs of segmentation. Segmentation of three-dimensional (3D) volumetric medical images using 3D U-net has been successful.15,16 Cascade architectures which integrate segmentation and detection architectures and cascaded 3D FCN can perform more accurately using segmentation network combined with a region proposal network.18,19,20,21,22,23 Some researchers have developed methods based on CNNs to segment the atrium and ventricle in cardiac imaging.24,25 Astudillo et al.26 proposed CNNs architecture to automatically extract landmarks for preoperative TAVR.

In this study, we present our deep learning architecture to find 3D complex structures of the aortic annulus plane without finding specific landmarks for TAVR using cardiac CT and evaluate the performance of our algorithms (Annulus Detection Permuted AdaIN network [ADPANet]) by comparing with other CNNs (transformer-based 3D U-net, and original 3D U-net).

METHODS

This study retrospectively reviewed consecutive patients treated for aortic stenosis with TAVR using hospital information system records. The dataset consisted of selected scans from patients who underwent TAVR between January 2017 and July 2020. Total datasets were collected and preprocessed using cardiac CT. The model was trained using nnU-net,16 transformer-based U-net,27 and our model, including the permuted adaIN28 layer, root mean square error (RMSE) loss function, and a 3D U-net architecture, using a limited training dataset and ground truth were delineated manually by three radiologists. Our network is shown in Fig. 1. First, a radiologist found the main points noncoronary cusp (NCC), left coronary cusp (LCC), and right coronary cusp (RCC) to draw annuls plane about moving the CT scan view in AVIEW in Figs. 1 and 2.

Fig. 1. Overall, the deep learning architecture to automatically detect the aortic annulus.

ADPANet = Annulus Detection Permuted AdaIN network, NCC = noncoronary cusp, LCC = left coronary cusp, RCC = right coronary cusp, RCO = right cardiac output, LCO = left cardiac output.

Fig. 2. Examples of the gold standard for the annulus plane using cardiac computed tomography with AVIEW.

Datasets and pre-processing

The dataset consisted of 72 scans from patients who underwent TAVR between January 2017 and July 2020 (Table 1).

Table 1. Schematic representation of the study population.

Characteristics		Training (n = 51)	Tuning (n = 10)	Test (n = 11)
Age, per patient		46 ± 14	38 ± 17.1	26 ± 6.3
Gender, per patient
	Male	32	4	3
	Female	19	6	7

Cardiac CTs were scanned using dual-layer spectral CT (IQon; Philips Healthcare, Best, The Netherlands). Intravenous contrast material (370 mg/mL iodine; Iopamiro; Bracco, Milan, Italy) was injected via an 18-gauge catheter placed in the antecubital vein. The protocol involved an initial injection (60–80 mL) of contrast medium at a rate of 5 mL/s, followed by 30 mL of saline at the same rate. CT was scanned in the cranio-caudal direction during an inspiratory breath-hold with retrospective electrocardiogram gating through tube current modulation. The bolus-tracking technique was triggered with a threshold of 100 Hounsfield units in the ascending aorta. The parameters for scanning were as follows: voltage, 120 kVp; current, 158–300 mAs; collimation, 64 × 0.625 mm; pitch, 0.16; gantry rotation time, 0.27 s; field of view, 257 mm; image matrix, 512 × 512; slice thickness, 0.9 mm; and slice increment, 0.45 mm. The CT dose-length product (in mGy · cm) was recorded for patients exposed to radiation during a CT scan. Effective dose (in mSV) was estimated by multiplying the dose-length product by the conversion factor (k = 0.017 mSv/mGy · cm) for the chest examination.

For the training, tuning, and test sets (about 80%:10%:10%) using a limited dataset, ground truth was delineated manually by three cardiac radiologists (S.H.H., and S.P. with 12, and 5 years of experience in cardiac imaging, respectively) with AVIEW platforms, version 1.1.40 (Coreline Software©, Seoul, Korea; https://www.corelinesoft.com) after manually finding NCC, LCC, and RCC using AVIEW.

The entailed labeling voxels on every cardiac CT slice included non-background (i.e., annulus plane) and background using the AVIEW software in Fig. 2.

All input datasets were resampled to 256 × 256 pixels (XY spatial size), including 120 slices along the Z direction, with the median voxel spacing of their corresponding dataset at the batch time. In total, 250 batches were used per epoch. We used third-order spline interpolation for cardiac CT and nearest-neighbor interpolation for the respective segmentation masks. Each cardiac CT image was normalized by the z-score method for intensity normalization. For efficient training, we performed aggressive data augmentation.

Architecture for training

We customized nnU-net based on 3D U-net16 to detect and localize the aortic annulus plane using cardiac CT in Fig. 3. The network includes an encoder and decoder parts. The left side diminished the dimensionality, whereas the right side expanded the original dimensionality. The encoder is similar to CNNs aggregating semantic features as spatial information is reduced. This network was customized in the encoder and decoder for more accurate semantic and spatial information to determine the aortic annulus plane. We added permuted AdaIN17 to the activation layers of the 3D convolution layers before batch normalization (BN).

Fig. 3. The architecture of ADPANet.

ADPANet = Annulus Detection Permuted AdaIN network.

$$\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{m}\mathrm{u}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{ }\mathrm{A}\mathrm{d}\mathrm{a}\mathrm{I}\mathrm{N}\mathrm{ }\left(p\mathrm{A}\mathrm{d}\mathrm{a}\mathrm{I}\mathrm{N}\right)\mathrm{ }=\mathrm{ }\left\{\begin{array}{c}x, probability P\\ (p{IN}^{\pi }\left(x_1\right),⃛,p{IN}^{\pi }\left(x_N\right) otherwise\end{array}\right.$$ (1)

where π is a uniformly selected permutation and p is a hyperparameter fixed ahead of training.

In addition, we first defined the BN operation:

$${\mu }_c\left(x\right)\mathrm{ }=\mathrm{ }\frac{1}{N}\mathrm{ }\sum _{n=1}^N{\mathrm{\mu }}_{nc}\left(x\right)$$ (2)

$${\sigma }_c = \sqrt{\frac{1}{NHW}\sum _{n=1}^N\sum _{h=1}^H\sum _{w=1}^W{(x_{nchw}-{\mu }_c\left(x\right))}^2 + \epsilon }$$ (3)

$$BN\left(x\right) = \gamma \left(\frac{x - {\mu }_c\left(x\right)}{{\sigma }_c \left(x\right)}\right) + \beta$$ (4)

for some parameters γ and β. After applying BN, we have:

$${\mu }_{nc}\left(BN(x\right)) = \frac{\gamma }{{\sigma }_c \left(x\right)}·\left({\mu }_{nc}\left(x\right) - {\mu }_c\left(x\right)\right) + \beta$$ (5)

$${\sigma }_{nc}\left(BN(x\right)) = \frac{\gamma ·{\sigma }_{nc} \left(x\right)}{{\sigma }_c \left(x\right)}$$ (6)

With pAdaIN, when statistics are swapped (i.e., not identity):

$${\mu }_{nc}\left(BN\left(p\mathrm{A}\mathrm{d}\mathrm{a}\mathrm{I}\mathrm{N}\right(x\right)\left)\right) = \frac{\gamma }{{\sigma }_c \left(x\right)} · \left({\mu }_{\pi \left(n\right)c}\left(x\right) - {\mu }_c\left(x\right)\right) + \beta$$ (7)

$${\sigma }_{nc}\left(BN\left(p\mathrm{A}\mathrm{d}\mathrm{a}\mathrm{I}\mathrm{N}\right(x\right)\left)\right) = \frac{\gamma · {\sigma }_{\pi \left(n\right)c} \left(x\right)}{{\sigma }_c \left(x\right)}$$ (8)

Eq. 7 and Eq. 8 follow from pAdaIN shifting channel-wise statistics. After applying pAdaIN, the statistics of channel c for sample n are the same as those of the previous sample π(n). Hence, BN maintains to swap statistics and scales them using batch-wise statistics.

To expand the missing spatial information, U-net was used to perform this step as the decoder, which received semantic features from the low vertex of “U” and recombined all layers of the left part with those of the right part for skip connection. The two networks (encoder and decoder) consisted of convolution layers (or transposed) and upsampling or downsampling layers. A distinctive feature of 3D U-net was the concatenated architecture of encoder and decoder parts to improve segmentation. These features were achieved by integrating detailed anatomy from the down-sampling including the broader contextual features to the up-sampling path. In this study, each epoch consisted of 250 batches. In this network, leaky ReLU activation functions were replaced with random ReLUs, and loss functions were used, including dice coefficients, boundary loss functions, and binary cross-entropy (BCE). Adam optimization approach for stochastic gradient descent was used with an initial learning rate of 3 × 10⁻⁴ and a l2 weight decay of 3 × 10⁻⁵. The calculation of losses was conducted using the dice similarity coefficient (DSC), as shown in Eq. (9). The loss functions, dice loss (DL), and BCE are defined in Eq. (10) and Eq. (11), respectively. The volumes of the gold standard and inference for deep learning are defined by V_gs and V_seg, respectively.

$$\mathrm{D}\mathrm{S}\mathrm{C}\mathrm{ }\left(V_{seg},\mathrm{ }{V}_{gs},\right)\mathrm{ }=\mathrm{ }\frac{2|V_{seg} \cap { V}_{gs}|}{\left|V_{seg}\right| + \left|V_{gs}\right|},$$ (9)

$$\mathrm{D}\mathrm{L}\mathrm{ }=\mathrm{ }1\mathrm{ }-\mathrm{ }\frac{2\left|V_{seg}\cap { V}_{gs}\right|}{\left|V_{seg}\right| + \left|V_{gs}\right|},$$ (10)

L (y, s) = −y log s − (1 − y) log (1 −s), (11)

$$RMSE\mathrm{ }\left(P_g,P_i\right) = mean\left(\sqrt{\frac{j={\sum }_{j=1}^n{\left(P_{g,j}- P_{i,j}\right)}^2}{n}}\right)$$ (12)

where s and y denote the output probability and the corresponding gold standard, respectively.

In addition, the RMSE loss function (12) was used to measure the accuracy of the aortic annulus plane using cardiac CT.

where P_g is the pixel coordinate of the gold standard and P_i is the pixel coordinate of the inference.

Although we manually find NCC, LCC, and RCC using AVIEW, our network infers the presence and location of the aortic annulus plane in the multi-planar reconstruction view in Fig. 4 without any temporal consistency including three points. Therefore, images for time-series were not required.

Fig. 4. Aortic annulus plane in the multi-planar reconstruction view.

LCC = left coronary cusp, NCC = noncoronary cusp, RCC = right coronary cusp.

Experimental setups

To detect the aortic annulus of patients who underwent TAVR in CT, each axial section of volume was presented sequentially to the 3D CNN. The model was built in Python 3.8, using PyTorch. Training and testing were performed using PyTorch 3.8 and NVIDIA Titan RTX GPU (24 GB). For maximization of speed for training and optimization of the memory in GPU, larger input tiles and the batch size of four were applied. Adam optimizer was set with an initial learning rate of 3 × 10⁻⁴ and a l2-weight decay value of 3 × 10⁻⁵. The learning rate was decreased by a factor of 0.2 if the losses did not perform after 30 epochs. The optimal model was selected during training per epoch with the tuning datasets among the training datasets. We compared our algorithms (ADPANet including a Gaussian noise layer and pAdaIN) 3D U-net with others, including origin 3D U-net16 and transformer-based 3D U-net.26

Ethics statement

The Institutional Review Board for human investigations at the Korea University Anam Hospital (2020AN0355) approved the study protocol and the informed consent was waived due to the retrospective characteristic of this study. All datasets were de-identified to preserve privacy of patients. All processes were conducted in accordance with the relevant regulations and guidelines.

RESULTS

Performance evaluation for detection of aortic annulus plane

The total dataset consisted of 72 scans (Table 1). The proposed ADPANet and other CNNs were evaluated using clinical 3D cardiac CT. The accuracy of the aortic annulus plane was determined by obtaining DSC and RMSE, which are calculated by the distance between the gold standard labels and the inferences of the deep learning in Table 1. The DSC and RMSE achieved between the results of inference and gold standard were compared for 11 cases in Table 1. These were used in our and other algorithms (Table 2).

Table 2. Dice similarity coefficient and the root mean square error for detection of the aortic annulus plane using ADPANet, original 3D U-net, and transformer-based 3D U-net for test dataset (11 cases).

Measurements	ADPANet	Transformer-based 3D U-net	Original 3D U-net
RMSE	55.078 ± 35.794	58.862 ± 39.538	69.656 ± 65.230
DSC	0.496 ± 0.217	0.417 ± 0.261	0.495 ± 0.259

Data are shown as below: First (RMSE), p = 0.316 for ADPANet and transformer-based 3D U-net and p = 0.167 for ADPANet and original 3D U-net; Second (DSC), p = 0.711 for ADPANet and transformer-based 3D U-net and p = 0.172 for ADPANet and original 3D U-net.

ADPANet = Annulus Detection Permuted AdaIN network, 3D = three-dimensional, RMSE = root mean square error, DSC = dice similarity coefficient.

The outputs are presented in Figs. 5 and 6. In test dataset after training and tuning, (a), (b), and (c) show the areas of the aortic annulus plane in the axial, sagittal, and coronal view, and 3D volume rendering in cardiac CT, and (d) shows the aortic annulus plane. The green area is the gold standard by experts, and the red area is output of deep learning, including ADPANet, original 3D U-net, and transformer-based 3D U-net. The best cases among deep-learning algorithms were shown in Fig. 5. The worst cases among deep-learning algorithms were shown in Fig. 6.

Fig. 5. The best case among the detection of the aortic annulus plane. The top row is the output from ADPANet, the middle row is the output from transformer-based 3D U-net, and the bottom row is the output from original 3D U-net (green: gold standard, red: inference).

ADPANet = Annulus Detection Permuted AdaIN network, 3D = three-dimensional.

Fig. 6. The worst case among the detection of the aortic annulus plane. The top row is the output from ADPANet, the middle row is the output from transformer-based 3D U-net, and the bottom row is the output from original 3D Unet (green: gold standard, red: inference).

ADPANet = Annulus Detection Permuted AdaIN network, 3D = three-dimensional.

We also compared our algorithm with other algorithms in the literature. Table 2 lists the DSCs and RMSEs for detection of the aortic annulus plane using cardiac CT. The average RMSE values for the aortic annulus plane using cardiac CT with ADPANet, transformer-based 3D U-net, and original 3D U-net were 55.078 ± 35.794, 58.862 ± 39.538, and 69.656 ± 65.230, respectively. The average DSC values for the aortic annulus plane using cardiac CT with ADPANet, original 3D U-net, and transformer-based 3D U-net were 0.496 ± 0.217, 0.417 ± 0.261, and 0.495 ± 0.259, respectively.

DISCUSSION

Localization of the aortic annulus plane is important because of the variation of the aortic valve shape in the patients and the leaflet thickness, which is much smaller than two pixels in 3D cardiac CT with a resolution of 0.49 × 0.49 × 1.25 mm. Our study presents deep learning-based aortic annulus detection using 3D cardiac CT for automatically localizing and segmenting the aortic annulus plane. ADPANet is a customized nnU-net based on 3D U-net including permuted AdaIN. This is important to detect and localize the aortic annulus plane using cardiac CT. This layer helps to detect the aortic annulus plane around the critical areas, including shape and texture information from specific areas on CT.

Deep learning has been proposed for inferring complex structures and specific areas using a large volume of datasets.24,25,27 Meaningful segmentation outputs for the medical imaging have been obtained using 3D U-net.20 Other researchers have developed cascaded deep learning networks involving detection algorithms using multipath networks.18,19 Tang et al.22 proposed a cascaded architecture integrating segmentation and detection using the backbone of the VGG-16 network.20 Roth et al.21 also presented a cascading method with a second-stage FCN to detect and segment objects in target boundary regions. We customized 3D U-net to obtain robust and accurate results, including the RMSE loss and pAdaIN.

To evaluate the detection of the aortic annulus plane using ADPANET, the outputs of our algorithm were compared with those of others. The average RMSE value (56.078 ± 35.794) for detecting the aortic annulus plane using ADPANet outperformed that of transformer-based 3D U-net (58.862 ± 39.538) and original 3D U-net (69.656 ± 65.230). The average DSC value for ADPANet (0.496 ± 0.217) was higher than those reported by other CNNs (0.417 ± 0.261 and 0.495 ± 0.259). Using our algorithm, we localized the 3D aortic annulus plane, which is too complex to be detected using cardiac CT without any temporal consistency.

Previous studies28,29,30,31,32,33,34,35 have investigated structural and biomechanical interactions between the origin aortic root and a deployed transcatheter valve during TAVR. One limitation of these studies is that a 3D aortic valve model developed using 3D volume datasets often needs to be manually annotated by humans for experts. These methods are time-consuming for annotation, and prevent prompt collaboration with clinicians, to provide accurate model predictions.

Currently, various studies36,37,38,39,40,41,42 have demonstrated automatic segmentation and geometric reconstruction of specific areas for the shape of the four cardiac chambers, ventricular outflow tract, and aorta in cardiac medical images. For estimating valve shape, Zheng et al.43 have visualized 3D reconstructions of specific areas of the aorta using 3D C-arm CT. Pouch et al.44 have presented registration techniques to reconstruct the shape of aortic leaflet in 3D echocardiographic images with only three landmarks delineated by the clinician. Ionasec et al.45have developed methods for reconstructing mitral-aortic complex annotated (over 600 cases) from 3D volume for learning. These techniques require real-time reconstruction and visualization in the clinical field. In particular, analysis for TAVR requires a reconstructed aortic annulus plane or landmark point detection using cardiac CT. Otherwise, we didn’t directly compare our algorithms with previous studies36,37,38,39,40,41,42 because previous algorithms weren’t different from our methodology. However, there are several limitations in this study. We need to increase the number of datasets for training and tuning to improve accuracy.9,10 Next, our study excluded patients with aortic valve calcifications, which are prevalent in 18–53% of the general population aged > 50 years,46,47,48,49 accounting for approximately 82% of aortic stenosis.50 We aimed to decrease parameters such as calcium information to improve the performance of deep learning model in detection of the aortic annulus plane. Since we have verified the feasibility of the deep learning model in patients without aortic valve calcifications, we need to check whether the deep learning model can be trained well even when including patients with aortic valve calcifications. Finally, further validation using more datasets or multi-center datasets and comparison of performances with state-of-the-art network, is important to verify our algorithm detecting aortic annulus plane.

In conclusion, our deep learning architecture is feasible to detect the 3D complex structure of the aortic annulus using cardiac CT for TAVR comparing with other algorithms. It is expected that the time devoted to reconstructing the aortic annulus will be significantly reduced by applying the deep learning model to the clinical setting. More research is needed to verify the feasibility of deep learning in patients with aortic valve calcification and validate models in other medical centers.