CardioVision: A fully automated deep learning package for medical image segmentation and reconstruction generating digital twins for patients with aortic stenosis

Abstract

Aortic stenosis (AS) is the most prevalent heart valve disease in western countries that poses a significant public health challenge due to the lack of a medical treatment to prevent valve calcification. Given the aging population demographic, the prevalence of AS is projected to rise, resulting in a progressively significant healthcare and economic burden. While surgical aortic valve replacement (SAVR) has been the gold standard approach, the less invasive transcatheter aortic valve replacement (TAVR) is poised to become the dominant method for high- and medium-risk interventions. Computational simulations using patient-specific models, have opened new research avenues for optimizing emerging devices and predicting clinical outcomes. The traditional techniques of generating digital replicas of patients’ aortic root, native valve, and calcification are time-consuming and labor-intensive processes requiring specialized tools and expertise in anatomy. Alternatively, deep learning models, such as the U-Net architecture, have emerged as reliable and fully automated methods for medical image segmentation. Two-dimensional U-Nets have been shown to produce comparable or more accurate results than trained clinicians’ manual segmentation while significantly reducing computational costs. In this study, we have developed a fully automatic AI tool capable of reconstructing the digital twin geometry and analyzing the calcification distribution on the aortic valve. The developed automatic segmentation package enables the modeling of patient-specific anatomies, which can then be used to simulate virtual interventional procedures, optimize emerging prosthetic devices, and predict clinical outcomes.

2. Introduction

Aortic stenosis (AS), the most prevalent valvular disease, is increasingly being reported in elderly individuals, regardless of gender [1]–[3], and will soon become a universal health burden for developed countries [4]. AS is characterized by the narrowing of the aortic valve, resulting from limited leaflet dynamics caused by calcification. Therefore, a more precise characterization of the condition could be achieved by using quantitative indices to assess the extent of calcium build-up [5]. Despite the prevalence and potential public health threat of AS [6], there is no medicinal treatment to prevent valve calcification as the major factor contributing to the pathogenesis [7]. The disease will continue to be a significant healthcare and economic problem [8] due to the ever-increasing aging of the global population [9] and its high prevalence in the elderly, affecting nearly 10% of patients aged 80–89 years old [2]. In North America alone, the number of individuals with symptomatic severe AS is expected to grow from 0.8 to 1.4 million between 2025 and 2050. According to literature predictions, there will also be five million cases of AS by 2030 in the United States [10]; the prognosis for Europe is even more bleak, with a projected rise from 1.3 to 2.1 million patients [6]. Although AS numbers are increasing, the relationship between calcium phenotype and disease severity is poorly understood. Moreover, there is still limited understanding of the temporal and spatial course of calcification and uncertainty in clinical decision-making regarding which individuals should undergo correction and what type of surgery would be most appropriate.

As our understanding of AS has grown more profound, so has the complexity surrounding its pathogenesis and diagnosis. More evidence suggests that mechanical forces, genetics, inflammation, and calcification contribute to AS development [8], [11]. Initially, oscillatory shear stresses experienced by valvular endothelial cells damage the endothelial layer and consequently alter local cellular gene expression and increase accumulation of inflammatory cells [12]. A detrimental feedback loop is initiated once these cellular changes alter the macrostructure of the leaflets instigating calcification [11]. Hypertrophy and potential heart failure are results of increased valve leaflet stiffness and reduced orifice area that, in turn, increase resistance to flow and impose a greater workload on the left ventricle [7],[13]. Without proper treatment for the initial symptoms of AS, the chance of survival can be as low as 11.6% [14].

Fortunately, timely intervention can mitigate many of the consequences of AS. Deemed the gold standard approach [15], surgical aortic valve replacement (SAVR) is now increasingly being replaced by the less invasive transcatheter aortic valve replacement (TAVR) for high-risk and, more recently, medium-risk interventions [16], [17]. In fact, since 2018, there have been more reported cases of TAVR than SAVR, reflecting its growing popularity and effectiveness [18]. TAVR entails deploying a balloon- or self-expandable bioprosthetic valve within the diseased native leaflets [19]. Due to its significantly less invasive nature, TAVR reduces patient recovery time, making it a more cost-effective alternative to SAVR [20], [21]. Additionally, TAVR is also offered as a viable option for certain patients who are considered inoperable for SAVR due to their age and various comorbidities [20], [21]. Nevertheless, there are reported drawbacks to the TAVR procedure, which emphasize the need for improvements in device design and therapy practices. Permanent pacemaker implantation (PPI) and heart conductance disturbances are the most frequent complications reported for TAVR [22]. Other serious complications include the occurrence of strokes, paravalvular leaks, reduced quality of life after the procedure, and coronary obstructions [23], [24]. While patient-specific aortic valve measurements, such as calcification score or general valve morphology, are used for TAVR instrumentation selection [25], their predictive power concerning the occurrence of TAVR complications is limited [26]. A deeper understanding of these aspects would be crucial for improving the long-term success and safety of TAVR interventions.

Computational simulations leveraging three-dimensional, patient-specific models have presented new opportunities for investigating and gaining mechanistic insight into the challenges associated with treating AS and optimizing emerging devices [27]. Furthermore, creating accurate digital twin models of patients enables the efficient and precise extraction of relevant measurements for TAVR planning and serves as the foundation for virtual intervention or pre-surgery training in challenging cases [28], [29]. In addition, it allows for the prediction of clinical outcomes, including the occurrence and extent of possible complications [30], and offers valuable morphology information for the development of potential personalized TAVR supporting tools and valve implants [31]. Besides, patient-specific aortic root shape analysis and elaborate valve calcification metrics are presumed to be powerful predictors for TAVR mortality rates and the probabilities of complication [32], [33]. However, the application of computational modeling is challenged by the tedious and time-consuming process of generating digital replicates of patient anatomy, which significantly hinders the development of large-scale studies. To build the three-dimensional mechanical environment of diseased aortic valves for studying mechanics or performing virtual intervention, it is necessary to accurately segment the geometry of the ascending aorta as well as the pathologic valve, including any calcification. Generating digital replicas of the structure thus requires distinctive approaches for the aortic root, native valve, and the calcification thereof.

Patient-specific segmentation of aorta is traditionally performed semi-automatically using the thresholding approach— a laborious process that requires anatomical expertise and dedicated tools. Using this labor-intensive method, a trained individual may spend over 20 minutes to precisely mark the aortic root and calcification alone; this is not including the time needed to generate the patient-specific aortic valve, which is vital for comprehensive calcification analysis in AS. While annotation by a trained physician is still considered the gold standard for image segmentation in terms of accuracy, alternative automated methods are increasingly matching this level of accuracy and reliability. The advent of deep learning models, which require smaller training datasets, has led to a promising upsurge in reliable methods capable of producing fully automated and affordable medical segmentation. Since its inception, the U-Net architecture has gained popularity as a convolutional neural network (CNN) architecture that exhibits high precision even with limited training data, making it the most widely used architecture for medical segmentation tasks [34], [35]. Segmentation of volumetric data can be done with both 2D and 3D U-Nets, with the former method processing each slice individually rather than in its entirety. Comparing 2D and 3D U-Nets to segment thoracic aorta imaged via CT, the Dice score was reported to be slightly improved for the 2D scenario (0.978 vs 0.972); however, it was not a true reflection of the precision in the segmentation of the anatomy and indeed was due to errors in segmenting distal portions of the thoracic aorta. Such a limitation is less concerning for AS studies as they generally involve segmentation of the ascending aorta [36]. It can thus be concluded that 2D U-Nets can yield similar, or even more accurate, outcomes than the manual aorta segmentations done by trained clinicians compared with the 3D U-Nets, all while significantly reducing the computational costs associated with the latter.

Although there are deep-learning-based segmentation approaches available for echocardiography images [37], creating three-dimensional models of native patient-specific aortic valves based on CT images requires an alternative approach to segmentation, primarily due to the poor visibility of the valve leaflet tissue in the CT scans. The leaflets are thus often reconstructed through a parametric approach, where anatomical landmarks are extracted either directly from the CT images or indirectly from geometrical models of the aortic root. The outline of the commissures and basal attachment points serves as a foundation for creating the shell elements of the geometrical model, enabling the accurate reconstruction of the leaflets in three-dimensional models of native patient-specific aortic valves [38]–[41]. Aortic valve models can be described mathematically as a combination of parametric curves defining the commissures and axial convexity of the cusp, which compared against 3D-transesophageal echocardiogram measurements with the average error equaling only 0.78mm, yields 3% of error for subjects with aortic valve diameter of 27mm [42]. The lack of an automated package capable of reconstructing the digital twin, encompassing patient-specific aortic valve and calcification, alongside automatic analysis of the calcium score, served as the main motivation for this study to develop deep learning and image processing techniques as a valuable support for clinical decision-making. Our developed pipeline provides a fundamental calcium distribution analysis, revealing the calcium burden on the left and right coronary and noncoronary leaflets.

Our objective is to harness the power of the individual methods mentioned earlier to create patient-specific models by developing a novel method for automatic landmark detection. We have developed CardioVision (CV) as a fully automatic, AI-driven pipeline for medical image segmentation and geometry reconstruction. CV streamlines, standardizes, and automates the generation of AS digital twins, making them suitable for large-scale computational studies of pathogenesis and intervention. This paper examines the theoretical foundation for CV and discusses its utility. In addition to providing medical image segmentation (tested for left ventricles [43]) and digital twin reconstruction, CV performs calcification analysis by measuring and visualizing calcification on individual aortic valve leaflets. Python programming language, in conjunction with Docker virtualization technology [44] that bestows cross-platform capability (tested on Windows, Linux, and MacOS), was employed for developing the pipeline. CV is hosted on GitHub (https://github.com/amirrouh/cardiovision), wherein the supplementary documentation allows training the AI model on custom images.

3. Materials and Methods

Our automated digital replica generator for AS patients comprises multiple distinct modules (Fig. 1). The deep learning CNN suite (DCNN) reconstructs the aorta geometry using raw patient CT imaging as input. The reconstructed geometry is then further processed to detect the valve landmarks and generate the aortic valve geometry using a parametric model described in Section 3.3. Subsequently, the automatically reconstructed geometries are imported into the calcium detector module, which identifies calcifications present on the aorta, aortic valve, and left ventricular outflow tract (LVOT). The assembly module processes this information and ultimately generates the final representation of the patient-specific model in stereolithography (STL) format.

Fig. 1:

Deep learning powered and fully automatic pipeline to generate patient-specific models of aortic stenosis cases. The blue and red arrows show the training and prediction processes, respectively.

3.1. Data Usage and Preprocessing

We have secured a unique repository of CT scans for AS patients, approved by our institutional review board (IRB), which comprised 35 cases for training and testing the DCNN for aorta segmentation. Double oblique multiplanar reconstruction (MPR) was directly performed on the raw CT images using the DICOM Software viewer (Visage Imaging Inc.) to rotate the axial, sagittal, and coronal planes and unveil a clear axial view on the aortic valve. This process ensured that the commissures, as well as the basal attachments points of the valve, were approximately located on the same or nearby slices, facilitating the landmark detection. Additionally, the image size was reduced to confine training duration and minimize computational cost. The raw CT images were cropped to only include the aortic root and LVOT, while the slice thickness was set to the lowest possible value of 0.6mm (Fig. 2).

Fig. 2:

Cropping procedure used to export the patient CT images, focusing on the aortic root and LVOT regions.

All CT scans used in this study were deidentified and converted to Nearly Raw Raster Data (NRRD) format in adherence to ethical guidelines established by our IRB. Two independent trained experts created ground truth labels through manual segmentation of the aorta for randomly assigned patients using an open-source image processing software (3D-Slicer). These labels were subsequently stored as binary label NRRD files, serving as the basis for training and validation of the deep learning model.

3.2. Aortic Root Generator

Aortic root geometry was reconstructed using an in-house DCNN module, which effectively converted the patient-specific CT images into a 3D representation of the aortic root, parts of the ascending aorta, and LVOT.

3.2.1. Data Preprocessing

The employed CT scans and their corresponding labels were normalized to ensure voxel intensity ranged between −1 and 1, and then cropped/padded to 512×512 pixels. Given the limited number of patients available for training (35), augmentation techniques such as scaling (from 0.9 to 1.1) and rotating (from −10 to 10 degrees) were employed to effectively expand the training dataset.

3.2.2. Model Structure

The Keras Python library, widely popular among the computer vision community due to its performance and pythonic structure, was used for the deep learning-powered automatic segmentation. The CT scans and corresponding labels were used to train the DCNN model, which incorporated the U-Net architecture (Fig. 3) comprising two major components: (i) an encoder using convolutional and pooling layers to extract critical features from the input images and (ii) a decoder including convolutional layers and up-sampling to predict the output segmentation [45]. Alongside the network structure, multiple network hyperparameters were tuned to obtain high accuracy while optimizing the training time (Table 1). The DCNN model was trained on a Quadro RTX 6000 GPU, equipped with 4,608 Cuda cores and 24 GB of GPU memory, allowing for efficient and effective model training.

Fig. 3:

Deep convolutional neural network (DCNN) architecture used for semantic image segmentation.

Table 1:

DCNN configurations and parameters

Loss function	Convolutional Layers Activation functions	Optimizer	Number of Parameters
Dice	Rectified Linear Units (ReLU)	ADAM	492474

The Adam optimization algorithm represents a stochastic gradient descent approach that dynamically adjusts the learning rates of model parameters based on historical gradient information. By incorporating exponential moving averages to estimate both mean and uncentered variance of gradients, it accelerates convergence and adeptly handles sparse gradients. These attributes collectively position Adam as a widely embraced and essential algorithm within the realm of deep learning [46]. To ascertain the quantity of learnable parameters within the neural network, we computed this by summing up the learnable parameters associated with each individual layer. This value is determined by multiplying the kernel size by the number of kernels present in that particular layer.

3.2.3. Model training

Patients were randomly shuffled and divided into two groups: 70% for training and 30% for testing. Using 4-fold cross-validation, we achieved reliable performance from the network and conducted a comprehensive analysis across all patients. In more detail, the patients used for testing were divided into four groups, and ensemble training was performed four times on the remaining patients. The classifier was trained using the Dice similarity coefficient measure [47] with a learning rate set at 10⁻⁴, and $L2$ regularization was implemented. The batch size was set to 2. To ensure the network’s robust performance, five separate subsets of the training-validation dataset were used to train the segmentation network and create five different models [45]. Employing majority voting, the candidate labels produced by each of the five models were combined. This approach is referred to as ensemble training within this work [45].

3.2.4. Model finalization

After the aortic geometry was generated by the DCNN suite, several image processing steps were applied. First, dilations of the labels were eroded using a cubic structuring element to remove small holes within the volume. Next, to smoothen the generated label, a median filter was applied utilizing a ball-shaped footprint of 5 voxels. Following smoothening and artifact removal, the largest connected component was isolated, effectively removing unwanted small particles.

3.3. Aortic Valve Generator

The underlying aortic valve design was developed to accurately represent the patient-specific morphological features while establishing a robust foundation for future automatic solid body generation, essential for various computational simulations. While the initial design drew inspiration from previous literature [38], [39], [42], it was subsequently updated and efficiently redeveloped to fulfill the additional requirements of the automated workflow for solid body model generation [39], [42]. Consequently, the aortic valve geometry was parametrically reconstructed based on seven anatomical landmarks, ensuring an accurate and customizable representation tailored to each patient’s unique anatomy.

3.3.1. Automatic Landmark Detection

To accurately reconstruct the aortic valve, several landmarks are required to outline the commissural and basal shape of the leaflets. The valve is axially confined by commissural and basal planes at the level of the three commissural (P4 – P6) and basal attachment points (P1 – P3). Specifically, P1 represents the basal attachment point of the left cusp, P2 the right cusp, and P3 the non-coronary cusp. P4 corresponds to the commissural attachment point between the left and right cusps, P5 between the right and non-coronary cusps, and P6 between the left and non-coronary cusps. Ultimately, P7 represents the point of coaptation of the valve (Fig. 4).

Fig. 4:

Enumeration of the aortic valve landmarks, illustrating the basal (left) and the commissural attachment points (middle), along with the point of coaptation on the average plane (right).

An automatic geometry-based method was developed to detect the landmarks for patient-specific valve reconstruction using the 3D geometry of the aorta obtained from the DCNN module. We assumed that the commissural and basal attachment points, respectively, resided on the same vertical CT slice. The method utilized the first derivation of the aortic cross-section area with respect to the z-axis (normal to the CT images) to detect the top (commissural plane) and bottom (basal plane) CT slices, where 6 out of 7 landmarks reside. The vertical coordinates of the top and bottom planes were determined by counting the desired slice number for the top slice (the first slice in the CT scan) multiplied by the slice distance. The following equations were used to calculate the slice number:

$$A_z=\partial A/\partial z$$ (1)

$$id{x}_a=argmax\left(A_z\right)$$ (2)

$$id{x}_b=argmin\left(A_z\right)$$ (3)

where $\mathrm{A}$ is the aortic cross-section area, and $id{x}_a$ and $id{x}_b$ represent slice numbers for commissural and basal planes, respectively.

To detect the basal attachment landmarks (P1 – P3) on the bottom slice, we located the intersections between the minimum enclosed triangle of the aorta and the guidelines connecting the triangle vertices to the center of the triangle (Fig 5a). The three identified points on the isolated slice were considered the basal attachment points. The same enclosing triangle was then translated, without rotation, to the detected top CT slice so that the triangle centroid overlayed the center of the aortic cross-section area. The midpoints of the triangle were connected to the centroid via a new set of guidelines, and their first intersections with the aortic cross-section were considered the top three landmarks, P4 – P6 (Fig. 5b). Given the locations of basal and commissural landmarks, the last landmark, P7, was located by taking the average coordinates of all identified landmarks.

Fig. 5:

Landmark Detection. (a) Bottom landmark guidelines connecting bottom enclose triangle vertices to the triangle centroid; (b) Top landmark guidelines connecting centroid of the bottom enclosed triangle transferred to the commissural plane to the mid points of the opposite edges; (c) Fitting ellipse calculated from the aortic cross-section on coronary plane; (d) Fitted plane transferred to the bottom landmark for landmark assignment; (e) Calculating the landmark distance to the major axis of fitted ellipse to assign the landmarks.

Once the basal and commissural landmarks were located, it became necessary to identify the individual landmarks and assign them to their corresponding number, P1 – P6. A critical anatomical feature, the location of coronary arteries, was thus extracted and leveraged to enumerate the landmarks, linking their relative positions to coronary arteries. The location of the coronary arteries was calculated as follows:

$$ecc=\sqrt{1-{\left(\frac{L_2}{L_1}\right)}^2}$$ (4)

$$id{x}_c=argmax\left(ecc\right)$$ (5)

Where $L_1,L_2$, and $ecc$ represent the minor axis, major axis, and the eccentricity of the enclosed ellipses, respectively. $id{x}_c$ corresponds to the slice number at which the eccentricity is maximum due to the appearance of the coronaries. The isolated slice was designated as the coronary plane (Fig. 5c), and the normal horizontal (on x-y plane) distances between the landmarks on the basal plane and the projection of the major axis of the ellipse on the coronary plane were calculated. The landmark with the greatest distance was identified as the non-coronary landmark, followed by the right coronary, and lastly, the left coronary landmark (Fig. 3d). With the non-coronary, left-coronary, and right-coronary landmarks identified, the commissural landmarks were detected accordingly (Fig. 5e).

3.3.2. Aortic Valve Construction

The aortic valve was constructed with two primary objectives in mind, both of which were essential to ensure the viability of the models for computational solid body simulation: achieving a watertight seal between adjacent cusps without intersections and maintaining a continuous curvature without any sharp edges or points. The leaflet shape is formed by two curves, the leaflet free edge curve, which defines the in-plane curvature on the commissural plane, and the guide curve, which governs the axial convexity of the cusp. The curves were generated by interpolating through the anatomical landmarks, using the two adjustable parameters per cusp $\left(C_{l,r,nc}{M}_{l,r,nc}\right)$ and derived construction points (marked with an apostrophe) resulting from the projection of the basal landmarks onto the commissural plane and vice versa (Fig. 6 a). The construction of the aortic valve is explained for the left cusp, as it is analogous for the non-coronary and right cusps.

Fig. 6:

(a) Side view of the design with the anatomical landmarks and derived construction points marked in blue, construction points in red, and parameters in green (b) Free edge curves and guide curves; (c) The spline guiding points on the cross splines resulting in a homogenous point cloud; (d) Offsetting the leaflet point cloud along the normals, resulting in a solid body point cloud.

Each cusp has two adjustable parameters, $C_l$ and $M_l$, which control the orifice opening and axial curvature, respectively. The landmarks and construction points were transformed into a local coordinate system $\left(x^{’},y^{’}\right)$ with the origin set to P7′, which corresponds to the projection of landmark P7 on the commissural plane, and the $y^{’}$ axis passing through P1′, representing a similar projection for landmark P1. To confine the individual cusps and avoid any intersection with the neighboring leaflets, straight boundary lines were inserted from the commissural landmarks P4 and P6 to the projected point of coaptation P7′. The leaflet free edge curve is composed of three sub-curves. The apex sections of the curve were modelled using a parabolic spline, whereas the outer sections were modelled as straight lines to ensure congruency with the constraining boundary lines. The points of transition between the spline and straight lines were defined as $B_4$ and $B_6$. The leaflet free edge curve is hence comprised of a straight line from P4 to transition point $B_4$, a parabolic spline interpolated through points $B_4,C_l$, and $B_6$, followed by another spline from $B_6$ to P6. To calculate the relative position of the transition points $B_4$ and $B_6$ on the boundary lines, the following model was employed. This explanation focuses on one half of the free edge curve, as the process is analogous for the other half.

The parabolic spline was approximated as a 2^nd-degree polynomial degree with the following form:

$$f_1\left(x\text{'}\right)=ax{\text{'}}^2+bx\text{'}+c$$ (6)

with $b=0$, and $c=C_{ly\mathrm{’}}$. The straight line from $P4$ to $B_4$ follows the form:

$$f_2\left(x\text{'}\right)=mx\text{'}$$ (7)

where $m=\frac{P{4}_{y\text{'}}-P{7}^{\prime }{}_{y\text{'}}}{P{4}^{\prime }{}_{x\text{'}}-P^{\prime }{}_{x\text{'}}}$

The entire shape of the leaflet free edge curve should only depend on one parameter $\left(C_l\right)$, which corresponds to the chosen orifice opening size. To determine parameter $a$, as well as the location of $B_4$, with respect to $C_l$, continuity equations with respect to the 1^st and 2^nd derivatives were applied resulting in:

$$a=\frac{m^2}{4c}$$ (8)

The location of the transition point is thus:

$$B_{4x\text{'}}=\frac{2c}{m}$$ (9)

$$B_{4y\text{'}}=2c$$ (10)

The guide curve was similarly modelled as a parabolic spline interpolated through points $C_l$ via $M_l$ to the basal attachment landmark P1. $M_l$ represents a point on a line connecting P7″ and P1′ with the convexity of the cusp being altered by shifting parameter $M_l$ along the line.

The design was implemented using the open-source Python libraries: VTK, Open3D, and PyMeshLab. The points located on the leaflet free edge curve, the guide curve, and the cross splines resulted in a homogenous point cloud, which provided a foundation for surface meshing and solid body generation (Fig. 6 b and c). A uniform valve thickness was enforced by estimating and aligning the normal of each point in the point cloud before offsetting the individual points along their normal by a defined distance (Fig. 6 d).

To create the closed surface shell, the four individual enclosing point clouds, namely the upper and lower leaflet as well as the two edge point clouds, were extracted and subsequently meshed using the ball pivoting algorithm and then fused, resulting in the generation of a closed surface representation [48].

3.4. Calcification segmentation and Final Geometry

The calcium burden of the aortic valve can be quantified using various semi-automated methods. The Agatston score [49], for instance, defines calcium regions based on the Hounsfield unit (HU) using thresholding techniques, which has been extensively used in the literature and proven to be reliable in accurately detecting calcifications in CT images [50]. Fig. 7 shows how thresholding voxel intensities led to the detection of calcium in a sample CT slice. Toward the end of this section, we will demonstrate how these identified calcifications, along with the aortic root and valve, were used to create the final geometry of the digital twin.

Fig. 7:

Calcium detection using thresholding techniques

As calcification is known to impair the physiological function of the aortic valve leaflets [13], [51], it becomes clinically insightful to accurately quantify and visualize the spatial position of calcifications on the leaflet. To segment the calcification, the raw CT scan was first masked with the volume of the generated aorta. Subsequently, a threshold was applied, whereby any voxel value above 900 HU was labelled as calcified tissue with individual calcification labels prescribed for each detected calcification. To discern whether a calcification cluster resides within the aortic valve, a new mask was generated to depict the cross-sectional area of the leaflet, extending downward beyond the sinotubular junction. The reconstructed leaflets, initially generated as PLY files, were converted into isotropic arrays by voxelizing the point cloud using the Python library Open3D. By converting the leaflets to an array with the same size and resolution of the aorta and calcifications, we ensured that the calcifications were appropriately positioned with respect to the leaflets. While anatomical features were employed to guide leaflet construction, the resultant leaflets remain simplified renditions of their real counterparts. Consequently, segmented calcifications might not adhere accurately to these approximated leaflets, leading to their placement as independent entities within the aortic root—misaligned with anatomical observations. To address this disparity, a calcium repositioning procedure was subsequently undertaken. Should the center of mass of a calcification cluster lie within the confines of the generated leaflet mask, the entire cluster is then adjusted vertically, either upwards or downwards, to align its center of mass with the generated leaflets. This repositioning produces a more anatomically accurate representation of calcifications occupying the volume inside the aortic root.

After the successful completion of the segmentations, a Python script was used to generate 3D reconstructions of the aorta, aortic valve, and LVOT (Fig. 8).

Fig. 8:

Steps taken to produce the STL geometry for the aorta (orange), calcifications (blue), and leaflets (green).

The generated geometries of both the aorta and the calcifications were converted into VTK objects using marching cubes applied to the 3D label map [52]. A Laplacian smoothing operation was then executed on the generated mesh to achieve more evenly distributed vertices. During this process, every vertex was modified in accordance with the average position of its connected vertices, with the extent of displacement controlled by a relaxation factor. This algorithm was executed for 300 iterations with a relaxation factor of 0.1. The VTK file was then converted to the widely used STL file format, suitable for visualization and engineering studies. The leaflets were converted from PLY to STL utilizing a surface reconstruction algorithm that determined the alpha shape of the cloud. This process involved utilizing Open3D, a Python package endowed with the capability to create a generalized convex hull [53].

4. Results and Discussions

In this section, the practical applicability of the theoretical foundations underpinning the CV package was examined by generating a digital model for a representative case entirely devoid of any user interaction. In conjunction with the generation of geometries, the performance of various modules was evaluated employing quantified metrics in the case of deep learning and resorting to visualizations for other image processing approaches.

4.1. Geometry Reconstruction

To showcase the fully automatic process of geometry reconstruction, a sample set of CT images was randomly selected. The outcome of the process is comprehensively explained as the raw data progresses through the pipeline. As the aortic root serves as the reference component for the assembly, the automated reconstruction of geometry begins with the aortic root and LVOT geometries.

4.1.1. Aorta and LVOT

The CT image of the randomly selected patient was passed to the DCNN suite to generate the aorta geometry, which was then compared with the ground truth geometries manually segmented by a trained clinician (Fig. 9). The obtained geometry, including the left and right coronary arteries, underwent a series of postprocessing steps, including median smoothing. The coronary arteries were utilized to detect the landmarks associated with the left, right, and non-coronary cusps. The geometry of the aortic root was also used to identify the calcifications associated with the aortic valve and within the aortic root geometry. As a result, the aortic root and LVOT were considered as the reference components of the total assembly.

Fig. 9:

Automatic generation of the patient-specific aortic root and LVOT (left) compared to the ground truth obtained from manual segmentation conducted by experienced clinicians (right)

4.1.1.1. Deep Learning Performance

Evaluating the performance of our semantic model to segment aorta requires metrics capable of gauging the extent of overlap and accuracy between the predicted segmentations and the ground truth. We adopted the following definitions as standard evaluation measures, particularly pertinent to semantic segmentation within the realm of medical imaging [54]. In the subsequent equations, TP (true positive) and TN (true negative) indicate properly categorized pixels, while FP (false positive) indicate pixels incorrectly labeled as aortic root and FN (false negative) indicate pixels mistakenly labeled as background.

Intersection over Union (IoU): Also known as the Jaccard index, this metric quantifies the overlap between the predicted and the actual aorta region. A higher $\mathrm{I}\mathrm{o}\mathrm{U}$ signifies a more accurate prediction. Notably, it remains resilient against imbalanced datasets, as it does not consider true negatives.

$$IoU=\frac{TP}{TP+FP+FN}$$ (11)

Dice Score (or Dice coefficient): It shares similarities with $\mathrm{I}\mathrm{o}\mathrm{U}$ but places greater emphasis on the true positives. This metric is particularly pertinent for imbalanced datasets, a common occurrence in medical imaging, where the area of interest is often overshadowed by the background.

$$DiceScore=\frac{2TP}{2TP+FP+FN}$$ (12)

Precision:

It represents the proportion of correctly identified positive pixels for aorta segmentation that are indeed true positives. Improved precision results in a reduction of false positive identifications.

$$Precision=\frac{TP}{TP+FP}$$ (13)

Recall (or Sensitivity):

It indicates the proportion of genuine positive aorta pixels that were correctly detected. Enhanced recall corresponds to a decrease in false negatives.

$$Recall=\frac{TP}{TP+FN}$$ (14)

Pixel-wise Accuracy:

This metric determines the proportion of pixels, both positive and negative, that are correctly classified throughout the image. While useful for a general understanding of accuracy, it might be suboptimal for imbalanced datasets and may yield a deceptively high score if the majority class is identified correctly.

$$PixelAcc.=\frac{TP+TN}{TP+FP+TN+FN}$$ (15)

In the specific context of aorta segmentation, the emphasis should be placed on regions of interest, particularly highlighting the positive class. Due to the potential imbalance in such datasets, where the aorta region might be scarce or relatively small, relying solely on pixel-wise accuracy could yield a skewed perspective. In these scenarios, metrics such as $\mathrm{I}\mathrm{o}\mathrm{U}$ and Dice Score offer more insightful evaluations.

The performance metrics for the aortic root generator module reveal satisfactory outcomes for the deep learning suite (Table 2). Furthermore, visual validation was conducted by juxtaposing the predicted models with the geometries meticulously reconstructed by experienced clinicians.

Table 2:

Performance metric values for the aorta segmentation module

IoU	Dice	Precision	Recall	Pixel-wise Accuracy
95.6%	97.8%	97.8%	97.8%	99.9%

4.1.2. Aortic Valve Reconstruction

Based on the obtained aortic geometry, the corresponding valve landmarks were detected, from which the patient-specific geometry of the valve was generated (Fig. 10c). The design of the aortic valve geometry was strategically focused on capturing the principal morphological attributes, all while preserving simplicity to facilitate seamless future solid body generation. The approximate location and orientation of the valve with respect to the aortic root and coronaries were calculated to represent the calcifications relative to the three cusps and two coronaries. These valuable representations can provide surgeons with greater insight into the accumulation of calcium at different positions on the valve, which in turn unveils the extent of calcification severity and offers potential indications for treatment strategies.

Fig. 10:

(a) Detected commissural landmarks identified on the top plane; (b) Detected basal landmarks identified on the bottom plane; (c) 3D representation of all the landmarks; (d) Final reconstructed model of the aortic valve.

4.2. STL Generator

Thereafter, the normals of the triangular meshes are determined and the file is exported as an STL (Fig.11). The generated valve is then positioned according to the detected landmarks on the aorta geometry within the final assembly model (Fig. 11a). To distinguish and assign the calcification region exclusive to the aortic valve, the normal projection of calcification should be on the leaflet, and finally a vertical translation is required to better visualize the calcifications on the leaflets.

Fig. 11:

The process of calcification adjustment and segregation; (a) All the detected calcium islands within the defined mask; (b) Isolation and adjustment of the detected calcifications on the aortic valve leaflets (dark green). A mask, representing the cross-sectional area of the generated leaflets, is created below the sinotubular junction (yellow). Calcifications (orange in a) whose center of mass falls within the mask are translated onto the leaflet, resulting in improved visualization (purple); (c) Patient-specific calcium distribution on individual leaflets extracted automatically. Left, right, and non-coronary calcifications are shown in yellow, blue, and red.

4.3. Calcium burden Analysis

While the Agatston calcium burden index is widely used in real-life practice [55], it has limitations in providing detailed information about the spatial distribution and functional implications of calcifications. Yet, the distribution of calcification within the aortic valve has been reported to considerably affect the hemodynamics of the valve [56]. As a result, the calcification pattern of the aortic valve can offer crucial insights into the severity of valve disease and the potential complications associated with interventions [57]. Numerical simulation results of a recent study highlighted the presence of complicated flow patterns on the aortic side of the valve, particularly in the region where focalized distribution of valve calcification is commonly observed. These flow patterns have the potential to significantly modulate the blood flow dynamics and shear stress [58], which may ultimately derive aortic stenosis and even left ventricular remodeling [59]. Our developed pipeline marks a significant milestone enabling the precise quantification of calcification distribution and spatial positioning within the aortic valve for the first time. This breakthrough provides invaluable insights, essential for accurate diagnosis and effective therapy planning.

Our pipeline goes even beyond the detection of aortic valve calcification and automatically detects LVOT geometry and calcification, presenting diverse clinical applications. For instance, in cases of TAVR, where the calcification pattern of the device landing zone is known to be linked to residual aortic valve regurgitation [30], CV allows for the evaluation of the calcification pattern in conjunction with the valve anatomy. This comprehensive assessment aids in making well-informed treatment decisions and guiding interventions for optimal patient outcomes. To showcase the utility of CV in analyzing calcium burden, we have implemented an automated calculation of calcification burden on individual leaflets (Fig. 11).

After reconstructing the patient-specific leaflets and calcifications, CV categorizes calcifications based on the specific leaflet they are present on and identifies any continuous calcification areas, known as calcification islands, sorting them by size. We compare the outcome of our segregated analysis with the traditional calcium burden metric measured by our collaborating clinicians (Table 3).

Table 3:

Calcium burden (volume) calculated using the proposed platform

	Agatston calcium burden	Right coronary leaflet	Left coronary leaflet	Non-coronary leaflet
Volume (mm3)	2087	470.1	885.2	731.7

Our integrated pipeline enables precise identification of calcium islands and analysis of their distribution on individual leaflets, facilitating accurate calculation of calcium per leaflet. By analyzing the radial (R) and angular (α) position of the center of mass, as well as the arc angle (β) of an individual calcium island, we gain valuable insights into patient-specific calcium distribution characterization and its potential utility in determining AS severity (Fig. 12 and Table 4).

Fig. 12:

Representative calcium distribution analysis using the automatically reconstructed digital twin. The blue plane includes the top landmarks of the aortic valve. The distribution of the selected, isolated calcium island (red) is characterized where R represents the radial position of its center of mass (dark sphere) in relation to the aortic orifice center (white sphere). The projection of the R vector on the landmark plane is measured as r. The location of each calcium island is determined using two parameters (r and α) where α is the minimum angular distance between the r and any of the given boundaries (dashed lines) or an individual leaflet. The arc angle (β) is defined to quantify the angular stretch of the isolated calcium island.

Table 4:

Representative quantification of calcium morphology and distribution

Surface area (mm2)	Volume (mm3)	R (mm)	r (mm)	α (rad)	β (rad)
27.7	10.1	10.26	2.2	0.16	0.48

4.4. A Concrete Example of Clinical Translation

Demonstrating the practical application of our platform, we utilized the toolbox to reconstruct digital models of the aortic root and calcified valve for 45 randomly selected severe AS patients. This allowed us to analyze their calcium distribution characteristics (in axial, radial, and angular directions) along with leaflet-specific metrics (Fig. 13). Furthermore, we leveraged this information to predict clinical events, showcasing the potential clinical relevance of CV. The metrics used to explore the radial and angular calcification distribution in 2D and 3D included mean and variance volumes, mean and variance distances, arc angles, and angular positions of calcifications. These metrics were calculated for different sub-volumes based on the three leaflets and the axial root direction, extending the scope of our analysis. Additionally, we conducted an independent analysis of the calcification close to the two coronary ostia. In total, we extracted 57 metrics, providing a comprehensive assessment of the spatial distribution and characteristics of calcification within the aortic valve and root.

Fig. 13:

a) Automated extraction of the aortic root and coronaries; b) Detection of coronary artery ostia, individual valve leaflets, and calcifications; c) Pearson correlation factors for calcification metrics and the outcome of permanent pacemaker placement installation (PPM); d) Principal component analysis with support vector machine decision boundary for PPM and no PPM classes

Using principal component analysis (PCA) [60] and a support vector machine (SVM) model [61], we assessed the predictive potential of extracted patient-specific parameters for post-TAVR permanent pacemaker installation (PPM). The PCA analysis was conducted with the scikit-learn implementation, which projected the extensive calcification metrics vector into two dimensions, represented by the two principal components, using singular value decomposition. To improve the results, we scaled the input vectors from 0 to 1 for each dimension [62]. This approach allowed us to effectively reduce the dimensionality of the data and identify the most significant patterns and variations in the patient-specific parameters related to post-TAVR PPM. To address the issue of missing values in the extracted calcification metrics for individual patients resulting from zero-value divisions, we replaced these missing values with the overall mean of the dataset for the specific metric.

A SVM model was then trained on the 2D PCA vector using the scikit-learn SVM support vector classification Python module. The SVM model was used to predict the non-probabilistic binary non-linear boundary between patients who received pacemakers and those who did not undergo PPM. SVM aims to find a hyperplane, or a set of hyperplanes, that can effectively distinguish between two groups of data points in a binary classification problem by maximizing the distance between the decision boundary and the closest training data point of each class. In our SVM model, the shape of the decision boundary is determined by the kernel function, specifically in this case, a radial basis function (RBF). The kernel function allows us to map the data into a higher-dimensional space, making it easier to find a separating hyperplane. Additionally, the complexity of the model is controlled by the regularization parameter C (Table 5). The parameter γ in the RBF kernel controls how far the influence of a single training sample extends and is therefore essential to prevent overfitting by regulating the smoothness of the decision boundary. To correct for the imbalance in the number of PPM versus no PPM patients, the SVM class weight parameters were set to a 1:3 ratio. This means that the SVM gave more importance to correctly classifying the minority class (PPM patients) by penalizing misclassifications in that class more heavily, while still aiming to correctly classify the majority class (no PPM patients).

Table 5:

Support Vector Machine Parameters

Kernel	Regularization (C)	Kernel coeff. (γ)	Tolerance	Class weight (0)	Class weight (1)
rbf	1.0	0.5	0.001	1	3

The study revealed significant associations (p < 0.05) between several spatial calcification parameters and the outcome of PPM, including calcium load in the lower root, mean calcium size across leaflets, calcium volume within ±10° of the right coronary ostium, and calcium volume in the right leaflet. Despite the limited number of patients, the SVM model demonstrated promising accuracy, with accuracies above 70%. A larger dataset and optimized model parameters are expected to further enhance the predictive power.

4.5. Limitations

Although our developed pipeline proves to be valuable for various applications, it is essential to acknowledge some limitations. One of the primary limitations is the relatively small size of our training dataset, which includes only 35 sets of CT images. This might restrict the generalization ability of our deep learning models to a broader population. However, despite this limitation, the accuracy of our developed segmentation tool was found to be competitive with reported semi-automatic and automatic methods in the existing literature. To improve the performance and robustness of the models, a larger and more diverse training dataset could be beneficial. The limited resolution of imaging modalities presented challenges in accurately locating aortic valve landmarks directly from the CT images. To address this limitation, we opted for an alternative approach by approximating the aortic valve landmark locations based on the aortic geometry. This enabled us to derive a parametric valve model design that could be used to generate solid body models of the valves more efficiently. Continuous updates and enhancements are planned to address these limitations and ensure the platform’s versatility and reliability for clinical translation.

5. Conclusion

We developed a fully automatic and highly integrated pipeline capable of generating a complete digital replica for AS cases, including the aorta, aortic valve, and calcification from clinical CT images. The integration of multiple components within our pipeline ensures seamless data flow and efficient processing, enabling accurate and innovative characterization of the calcification spatial distribution, extraction of several quantifiable metrics of calcium morphology and distribution, and leveraging them to predict adverse clinical outcomes. CV showcased commendable performance through quantified metrics and visual comparisons against manually segmented models undertaken by trained clinicians. Indeed, this pipeline has a wide range of applications, including but not limited to visualizing diseased valve and the aorta to assist with diagnosis and clinical decision-making, generating digital substrates for computational modeling of patient-specific hemodynamics and virtual intervention, and evaluating the performance of emerging prosthetic devices. Its versatility makes it a valuable asset for advancing research and clinical practice in the field of aortic valve disease management.

Our ultimate goal is to provide clinicians and researchers with a powerful tool that not only improves the understanding of aortic stenosis and valve-related conditions but also assists in making informed and timely decisions in clinical practice. With the ability to analyze a comprehensive set of metrics for the calcium phenotype, we gain valuable insights into the disease and its impact on valve function. Leveraging this information, we can identify meaningful patterns and associations between the calcification metrics, clinical events, and post-op outcomes. This predictive capability holds the potential to revolutionize treatment planning using digital medicine and virtual surgery or intervention, ultimately improving patient outcomes.

Our platform’s capability to efficiently and rapidly generate patient-specific models for large cohorts is streamlining large-scale computational studies and in silico clinical trials for aortic stenosis. This advancement brings us closer to the long-term goal of conducting personalized medicine and treatments, where tailored approaches can be applied to individual patients based on their unique characteristics and needs.

Highlights.

• Aortic stenosis, the most common valve disease, lacks a preventative treatment for valve calcification,

• Computer models for TAVR enhancement hindered by digital twin generation burden Our AI tool automatically creates digital replicas of case-specific anatomy from CT

• Digital twin allows calcification distribution mapping, informing diagnosis/therapy Our tool feeds virtual surgery and device optimization to improve patient outcome