Augmented reality visualization of biomechanical wall stresses on abdominal aortic aneurysms using artificial intelligence

Abstract

The number of medical images taken has continued to increase year over year for an aging population in the United States. It has been shown that patients understand their diagnoses better when shown a 2D or 3D image of their respective diseases. However, clinicians do not regularly show patients their images as it requires additional time and processing. In this experiment, we demonstrate the use of augmented reality to visualize abdominal aortic aneurysms using a previously developed artificial intelligence engine. Our group further expanded the number of cases used for training the stress prediction model to a total of 274 patients (206 used for training or ~ 5.4 million nodes, and 68 for testing or ~1.8 million nodes). Medical images undergo automated segmentation, and wall stresses are predicted on the 3D surface of aneurysms to view a heat map. The pipeline includes introducing elements into the Microsoft HoloLens 2 ecosystem to view models and additional analytics, enabling clinicians and patients to view the biomechanical status without the need for a computational or imaging expert. The proposed clinical workflow would allow a local server to process medical imaging data, generate point clouds, predict wall stresses on individual points, and create a 3D model with a colormap to view in augmented reality. The study revealed that neural networks and ensemble boosted tress models predicted the wall stresses more accurately (when compared to ground truth finite element analysis results). The approach is not limited to the HoloLens 2 ecosystem but can be used with other emerging augmented or virtual reality hardware systems.

Summary:

Patient understanding of their diagnosis improves when they are shown medical images by clinicians. An artificial intelligence-based clinical workflow has been developed to visualize the biomechanical status using augmented reality, providing additional information to clinicians and patients. Our research improves the tools that are available to clinicians and patients to help provide a better understanding of diagnosis and potentially prognosis.

Fig. 1.

A study presented by Phelps et al. 2017 surveyed patients while interaction with clinicians. The patients were surveyed an gave a score based on their understanding, accuracy, trust, anxiety, and overall satisfaction of their hospital visit when an image was not shown (verbal diagnosis given), 2D, and 3D medical images shown. It was found that patient understanding, accuracy, trust, anxiety, and satisfaction was statistically significant when comparing 2D image presented vs. no image, 3D image presented vs. no image (readapted from Phelps et al. 2017). Anxiety was not statistically significant when patients were shown the images of their diagnosis.

Fig. 2.

The aorta is a major conduit for oxygenated blood flow from the heart to the rest of the body. The abdominal aorta leads to the bifurcation to the lower limbs. When the terminal aorta begins to dilate 50 % or greater, it becomes defined as an abdominal aortic aneurysm (AAA). This condition is highly prevalent in about 2.5 million US citizens over the age of 65 with 200,000 new diagnoses each year. Clinicians will track the diameter through medical imaging surveillance before a maximum diameter threshold is met (5.0 cm for women, and 5.5 cm for men). After the threshold is breached, clinicians will provide clinical intervention through surgery. However, there are still a relatively large number of patients rupturing before this threshold, a critical event that leads to a death (75–90 % mortality). Biomechanics researchers have investigated the relationship between wall stress and wall strength to understand which patients may be at risk of rupture, a potentially better tool than simply measuring diameter.

Fig. 3.

A) Initial medical image for an AAA that can be images using computed tomography, magnetic resonance imaging, and ultrasound. The medical image data uses the DICOM header file embedded inside each individual 2D image file (that makes an entire 3D volume or stack in the axial direction). Image segmentation occurs here after the medical images are acquired. Automated segmentation (using a trained 2D U-NET) or manual/semi-automated image segmentation is used for this process before B) A 3D point cloud is generated from the manual/semi-automated or automated segmentation. The 3D point cloud uses pixel dimensions for X,Y, and Z directions. A scaling factor is applied using information from the DICOM header file (X and Y pixel spacing) and relative position (image patient position) for the distance between 2D axial slices. The scaling factors (pixels/mm) are applied to the 3D point cloud to convert into ‘real-world’ units. C) The 3D point cloud is converted into a 3D surface. This is achieved my computing the normal of each vertex or node and finding a neighborhood of points. A Poisson reconstruction is applied to create a 3D surface of the aneurysm wall. D) In a previous study, several machine learning models were trained to predict the wall stresses using a pre-processing script. The pre-processing script calculated the relative position of the vertices to the centroid of the AAA, intraluminal thrombus thickness, and principal curvatures. The processed data is input into the ML models to predict wall stresses for each vertex or node. E) The heatmap of stresses needs to be applied to the 3D surface. This process is performed by taking the predicted wall stresses and applying a color map to each vertex defined as a red, green, and blue color (normalized between 0 and 1). The colors based on the predicted wall stresses are then baked onto the surface using Blender, and exported as an ‘.obj’,’.vrml’, or ‘.glb’ file for visualization. F) The 3D model file is then uploaded to the augmented or virtual reality environment for visualization. The file can be accessed via cloud or local data storage and visualized in a number of 3D viewers that both the clinician and patient can view.

Fig. 4.

Stress prediction results using linear regression (traditional statistical method) and different machine learning models (ensemble boosted trees, decision trees, and neural networks. The figure shows the ground truth (from finite element analysis) on the first column, and the results of the various methods showing the anterior and posterior views. The wide neural network performed the best among the trained models. The wall stress map shows elevated stress banding on the posterior section of the aneurysm.

Fig. 5.

A) The rendered 3D models with the heat map baked onto the surface are exported and uploaded into the augmented reality system. There are three models loaded using the file system and operating system of the hardware (Microsoft HoloLens). The three models represent the stress predictions from a fine tree model, ground truth (stress analysis), and a trilinear neural network. B) A 2D image of the analytics used for the example 3D model with stress predictions (91st percentile for PWS, and 72nd percentile for MWS) can be visualized in the HoloLens environment while simultaneously viewing the 3D models. The data processed is from a database that is growing and can be used to compare a patient aneurysms to a growing population.

Biographies

Timothy K. Chung is research faculty in the Department of Bioengineering at the University of Pittsburgh’s Swanson School of Engineering. He earned his PhD in Biomedical Engineering from the University of Iowa, focusing on cardiovascular biomechanics. His research interests include computational modeling of cardiovascular diseases, experimental mechanical testing of biological soft tissues, artificial intelligence, and augmented reality. Dr. Chung has been involved in interdisciplinary projects, such as developing AI-based tools to predict patient outcomes for conditions like abdominal aortic aneurysms and endometriosis.

Pete H. Gueldner is a Ph.D. candidate in Bioengineering at the University of Pittsburgh, specializing in vascular biomechanics and aneurysm research. Originally from Texas, he earned his bachelor’s degree in Biomedical Engineering from the University of Texas at San Antonio. As a National Science Foundation Graduate Research Fellow and American Heart Association Pre-doctoral Fellow Pete focuses on studying abdominal aortic aneurysms, using innovative testing to better understand diseased tissue mechanics.

Nathan L. Liang is a board-certified vascular surgeon and Associate Professor of Surgery and Bioengineering at the University of Pittsburgh. He completed his medical degree at Texas A&M University College of Medicine, followed by a vascular surgery residency at UPMC, where he also earned a master’s in clinical research. Dr. Liang specializes in complex aortic surgeries, including both traditional and minimally invasive endovascular techniques, and he leads the Complex Endovascular Aortic Interventions program at UPMC. His research focuses on applying advanced statistical and machine learning methods to improve surgical and endovascular treatments for aortic conditions and acute pulmonary embolism. An active member of multiple professional societies.

David A. Vorp is the John A. Swanson Professor of Bioengineering and Senior Associate Dean for Research and Facilities at the University of Pittsburgh’s Swanson School of Engineering. Dr. Vorp’s research specializes in vascular biomechanics, mechanopathobiology, and tissue engineering, with a focus on aortic aneurysms and tissue-engineered vascular grafts. He has authored over 145 peer-reviewed papers and has received numerous honors, including the Van C. Mow Medal and the Carnegie Science Award. He is also a Fellow of the American Heart Association, the American Institute for Medical and Biological Engineering, the American Society of Mechanical Engineers, and the Biomedical Engineering Society.

Data availability

The authors do not have permission to share data.