Automatic Detection of Vegetations With Transesophageal Echocardiography in Infective Endocarditis Using Artificial Intelligence

Abstract

Infective endocarditis (IE) is a life-threatening condition frequently associated with endocardial lesions known as vegetations. Detection and characterization of these lesions are critical for a proper diagnosis and management of the disease according to the current standard practice, but current human analysis techniques present severe limitations such as very basic set of measurements and high inter-operator variability. This is a retrospective observational study across 7 hospitals with 329 IE patients. An AI-based model was trained to detect vegetations in transesophageal echocardiographic (TEE) images. We measured the accuracy of the system both in terms of vegetation detection at the frame level (i.e., answering the question “is there any vegetation in this image and, if so, where is it?”) and vegetation diagnosis at the patient level (i.e., “does this patient have a vegetation?”). Two different architectures, YOLO and DETR, were evaluated within the AI-based model framework, and a comparative analysis of their performance was performed. The model exhibited strong diagnostic capability, achieving an area under the receiver-operating characteristic curve (AUROC) of 0.91 (average positive predictive value = 0.81, true positive rate = 0.83). Vegetation detection at the frame level also achieved promising performance metrics (positive predictive value = 0.83, true positive rate = 0.75). The algorithm achieved high-performance metrics detecting vegetations and identifying patients with vegetations, which can facilitate and accelerate IE diagnosis by non-expert cardiologists.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10278-025-01722-0.

Introduction

Left-sided infective endocarditis (IE) is a serious, life-threatening condition that involves infection of the endocardial surface, potentially leading to the formation of heart valve vegetations, which are mobile structures composed of platelets, fibrin, microbial microcolonies, and inflammatory cells. Robust evidence supports a strong correlation between the presence and characteristics of vegetations and clinical outcomes, particularly due to their close association with serious complications such as systemic or pulmonary embolism.[1–6]

Therefore, diagnosis and characterization of vegetations are essential components of disease management. The presence of vegetations is not only a major criterion for the diagnosis of infective endocarditis,[7] but also a predictor of poor prognosis [8]: vegetations are more frequent in patients with definitive endocarditis compared with those with possible endocarditis, and the former have higher in-hospital mortality.[9] Current standard practice relies on transesophageal echocardiography (TEE) as the primary imaging modality for their detection, and parameters such as size, mobility, and location are considered for risk evaluation [1]. Searching for vegetations is mandatory in bloodstream infections of Enterococcus faecalis, Staphylococcus aureus, and possibly Streptococci, given the high probability of IE.[10] Most importantly, vegetation size, defined as the maximal length of the vegetation in a 2D cine loop, is a particularly relevant metric used to guide surgical indication.[1, 2, 11]

However, current vegetation assessment techniques have been shown to have limited reliability: surgery does not appear to improve prognosis in patients with large vegetations as the sole risk factor,[12] and significant inter-observer variability exists in measuring vegetation size (surgical indication may change in more than 40% of patients depending on the operator performing the assessment). [13].

In this scenario, there is a need for more advanced and reproducible vegetation evaluation methods to improve risk prediction, disease management, and surgery indication. Several recent works apply artificial intelligence (AI) techniques to this end.[14–19] However, most of them are models aimed at estimating mortality risk under specific conditions (SABIER score for patients with Staphylococcus aureus bacteremia [16], SYSUPMIE for early mortality after surgery [17], with a remarkable accuracy of AUC = 0.81) and only employ structured clinical features, without direct analysis of the image itself. Other works aim at studying new molecular biomarkers or proteomics, yielding very good diagnostic results but limited prognostic power (sensitivity = 0.78 for predicting mortality[18] and AUC = 0.83 for identifying patients in NYHA IV).[19] However, none of these approaches considers advanced features of vegetations, while it is already known that they have a significant impact on embolic risk (for instance, systemic embolisms were more frequent in filiform and raceme-shaped vegetation than in sessile vegetation).[20].

In this field, some early works have begun exploring the application of AI to process raw echocardiographic imaging of vegetations. In medical imaging, neural networks have been successfully employed over cardiac imaging data [21–24], and particularly echocardiography [25], for autonomous segmentation, measurement, diagnosis, optimizing imaging protocols, mortality/event prediction, and clinical risk stratification. In the field of TEE, AI has been employed to detect and track meaningful structures (most notably the mitral annulus).[26–29] And in endocarditis, in particular, Esmaely et al.[30] have applied simple machine learning models to manually segmented valve contours, achieving an AUC of 0.88 for identifying valves with vegetations. However, in their work, an expert physician is still required to manually analyze, locate, and segment the contour of the vegetation.

The purpose of the present study is to implement an AI model to automatically detect vegetations in TEE loops of patients with left-sided IE. This system could be directly employed to assist in the assessment of vegetation presence, particularly in low-volume centers where physicians may be less familiar with the diagnostic work-up when endocarditis is suspected.[31] But, more importantly, it may serve as the foundation for future advanced image-based characterization of vegetations, computed from programmatically derived (i.e., operator-independent) parameters such as size, velocity, trajectory, and morphological features. These quantifiable metrics could ultimately enable highly accurate risk stratification and support personalized clinical decision-making.

Material and Methods

A. Data Collection and Curation

A database was developed compiling data from 329 retrospective patients diagnosed with left-sided IE between 2014 and 2025 in the seven participating centers applying the following inclusion criteria: (1) definitive diagnosis of IE based on the modified Duke criteria; (2) left-sided IE; (3) availability of TEE images in standard DICOM format with visible vegetations. Collected data included standard DICOM TEE images and clinical variables from the endocarditis episode from admission to discharge. All clinical data and imaging were acquired according to standard-of-care procedures. Table 1 summarizes the baseline clinical and demographic characteristics of the study population.

Table 1.

Baseline characteristics of patients with IE. *Seventeen percent of patients had surgical indication but did not receive the intervention

Age	67.6 ± 12.9
Male	239 (73%)
Diabetes mellitus	106 (32.3%)
Previous renal failure	64 (19.6%)
Prior vascular disease	71 (25%)
Immunosuppression	29 (8.9%)
Location Mitral Aortic Both	134 (40.7%) 143 (43.5%) 52 (15.8%)
Prosthetic Mechanical Biological	18 (5.5%) 61 (18.5%)
Microorganism Staph. aureus Staph. coagulase negative Strep. Viridans	53 (18.1%) 23 (7.9%) 29 (9.9%)
Outcome Surgery* In-hospital mortality	167 (52%) 81 (24.8%)

Between 4 and 12 cine loops were obtained from each patient, each one comprising between 15 and 203 frames on standard 2D echocardiographic longitudinal or three-chamber views (excluding x-plane and color Doppler) captured in standard-of-care exams (no specific acquisition protocol). We used all available loops from each patient to avoid bias from arbitrary selection. Poor image quality on the TEE was not an exclusion criterion. The selected cine loops were cleaned with basic pre-processing algorithms to eliminate information outside the ultrasound cone, erase labels and annotations, normalize intensity, and reduce noise.

In addition, for validation purposes, a control dataset was compiled with TEE images of patients without vegetations. This dataset includes 70 randomly selected individuals who underwent a TEE within the last 5 years in the coordinating center and neither had IE nor were suspected of having it.

B. Data Labeling

Once the dataset is curated, the next step involves the precise labeling of vegetations within each echocardiographic video. This labeling was performed using the Visual Object Tagging Tool (VoTT), an open-source annotation tool that facilitates frame-by-frame labeling of objects within videos.

Each frame of the echocardiographic cine loops was examined, and the vegetation was marked with bounding boxes as shown in Fig. 1. Two cardiac imaging experts carried out the annotation process to ensure accuracy and consistency. This comprehensive annotation strategy was selected to allow the system to learn relevant imaging patterns that replicate the typical behavior of imaging experts. A total of 50,172 vegetations were manually marked in 167,369 frames extracted from 2517 cine loops belonging to 329 patients. It is important to note that the manual labels used for model training and validation were indeed performed by experts utilizing all available information (clinical data and multiple TEE views per patient).

Fig. 1.

Annotated vegetation in a preprocessed frame of an echocardiogram from a patient diagnosed with endocarditis

C. Model Description

Two non-pretrained, state-of-the-art architectures were evaluated for the model implementation: a YOLOv12[32] neural network and a RT-DERT[33] neural network. Both architectures were configured to take a still echocardiographic image (640 × 640 pixels) as input and to output the coordinates of bounding boxes of detected vegetations, if any. For each detected vegetation, the model estimates a confidence value between 0 and 1: A higher confidence value indicates a greater likelihood that the marked structure is a true vegetation.

D. Model Training

To implement and validate the model, the sample must be divided into three separate datasets: training (for model implementation), model optimization, and test (for performance measurements over a completely independent sample). The dataset was partitioned on a per-patient basis, ensuring that all images from a given patient were confined to a single subset. Therefore, all videos from a single patient are assigned to the same set. This approach was adopted instead of splitting by loop or frame in order to prevent data leakage. Including images from the same patient in both sets could allow the model to memorize patient-specific features, thereby compromising its ability to generalize. Such leakage would likely lead to a significant overestimation of model performance during validation.

In order to mitigate patient selection bias due to dataset splitting, ten different random splits of the dataset into training, model optimization and test were performed, and results averaged across the splits.

The training set was enhanced using standard data augmentation techniques, such as rotation, scaling, inversion, and mosaic data augmentation[34] to increase spatial variability in the dataset and improve the robustness of the trained model. Figure 2 shows an example of the different versions of an input image generated during this data augmentation process.

Fig. 2.

Example of synthetic training samples produced by data augmentation from a single image

E. Validation

We measured the accuracy of the system both in terms of vegetation detection at the frame level (i.e., answering the question “is there any vegetation in this image and, if so, where is it?”) and vegetation diagnosis at patient level (i.e., “does this patient have a vegetation?”).

At the frame level, standard performance metrics for automated object detection were computed. A detection is considered a true positive if its bounding box is at least 50% overlapped with the manually labeled vegetation bounding box, or a false positive otherwise. Overlapping is measured in terms of Intersection-over-Union (IoU), explained in supplementary data section d.

To measure the diagnostic accuracy at the patient level, we implemented a simple vegetation diagnosing system to integrate vegetation detection results along all frames of each cine loop. For each loop, the ratio of frames with a positive vegetation detection vs. total number of frames in the sequence is computed. Low ratios normally mean spurious wrong detections in isolated frames, while high ratios are consistent detections across the entire video sequence. The computed ratio is the output of the diagnostic system and could be loosely understood as the probability of the patient having a vegetation: if the ratio is over a configurable threshold, the patient is considered positive. While simple, this approach builds on the good accuracy of the frame-level detection and provides robust generalizability across heterogeneous scenarios, considering the variability in both vegetation attachment sites and mobility patterns. Other specific approaches such as restricting the analysis to the systolic phase (closed valve) may seem advantageous, but could lose generalizability. This closed valve strategy for instance would be less effective in prosthetic valves, where vegetations are frequently anchored to the annulus and thus exhibit reduced displacement, making their detection more challenging.

The diagnostic accuracy of this system is validated over a sample population of 130 patients, including the test dataset of each split (60 positive cases) and the control dataset (70 negative cases) (Fig. 3). Receiver-operating characteristic (ROC) curves for the diagnostic system have been computed for each split.

Fig. 3.

Dataset split and population size through training and validation

Results

Our model demonstrated strong diagnostic performance with both architectures in discriminating between patients with and without vegetations when applied to cine loops. The YOLOv12-based model achieved an average area under the ROC curve (AUROC) of 0.91 (positive predictive value = 0.81, true positive rate = 0.83), while the RT-DETR-based model achieved an average AUROC of 0.9 (positive predictive value = 0.81, true positive rate = 0.79). The YOLOv12 architecture contained 272 layers and 2,568,243 parameters, while the RT-DETR model was ten times more complex with 32,808,131 parameters across 457 layers. Some specific examples of operation are illustrated in Fig. 4.

Fig. 4.

Three examples (A, B, C) of the vegetation diagnosis test over cine loops. Patient A is an endocarditis with a vegetation on the mitral valve; the detector is correct in all five frames presented, properly locating the vegetation in the four frames where it is visible and not detecting anything in the frame where it is not present. Patient B is a case without infective endocarditis, correctly identified in four frames, with only one frame in the loop mistakenly interpreted as showing a vegetation. Patient C is a prolapse in P1 with a ruptured secondary chord misclassified as a vegetation in the last three frames of the loop. Setting up the vegetation diagnosis system with a threshold of 80% (at least four out of five frames to consider a positive) would result in Patient A being a true positive, patient B a true negative, and patient C a true negative

For vegetation detection on a per-frame basis, the YOLOv12 architecture achieved a mean positive predictive value of 0.83 (indicating that 83% of detected vegetations were correct) and a mean true positive rate of 0.75 (indicating that 75% of all vegetations were successfully detected and localized), both with a standard deviation below 0.04, reflecting strong convergence of the model during training. The RT-DETR architecture achieved a mean positive predictive value of 0.84 and a mean true positive rate of 0.76, both with a standard deviation below 0.04, likewise indicating robust convergence.

Detailed comparative results for each split are provided in supplementary data: Tables 3–6. Specifically, Table 3 shows standard metrics to characterize how the detections overlap with the manual label gold standard, in order to evaluate how well the model not only finds vegetation but also precisely determines its size and position. Figure 5 shows some examples of specific cases.

Fig. 5.

Three (A, B, C) specific examples of vegetation detection over the test group. Left images are manually labeled ground truth boxes, and right images are detections identified by the AI model, including its associated confidence estimation. Example A shows a detection with > 95% overlap. Example B shows two detections in a frame where there is only one vegetation: one of them is a false positive if the confidence threshold is under 0.4. Example C shows another potential false positive: a wrong detection where no vegetation is present

Discussion

This study introduces an AI model developed to analyze TEE image sequences for the detection of vegetations associated with left-sided infective endocarditis. Two state-of-the-art architectures, YOLOv12 and RT-DETR, were compared. The models’ performance was validated at the frame level using standard object detection metrics, as well as its utility as a diagnostic tool for discriminating between patients with and without vegetations. Although they are conceptually and architecturally different, their results in this scenario have been similar. The RT-DETR model is however more complex, with about ten times more parameters than YOLOv12, which results in longer training and operation times.

While scores might not immediately suggest human-expert-level performance, it is crucial to consider the significant difference in available information between an expert human annotator and our proposed system. Expert clinicians typically leverage a wealth of additional data for identifying vegetations, including multiple echocardiographic views, real-time dynamic assessment across cine loops, and comprehensive patient clinical information. In contrast, our system operates solely on the information contained within each individual image frame.

To the best of our knowledge, there is no similar model published in the literature, so it is not possible to directly compare our results to previous state of the art. The most similar work is the vegetation contour discriminator of Esmaely et al.[30], with an AUC of 0.88 for identification of vegetations in manually extracted contours of heart valves. Our system shows a slightly superior AUC of 0.91, and identifies and locates vegetations over raw echocardiographic images without human intervention.

The automated vegetation detection system proposed in this work addresses the critical need for consistent and reliable vegetation characterization in the diagnosis of IE. Given the high mortality rate and severe complications associated with IE, such as embolic events, the ability to accurately and consistently detect vegetations is crucial. Furthermore, the automated detection of vegetations is a step towards automatically computing vegetation measurements, currently essential for risk stratification and therapeutic planning. Accurate measurement of vegetation size can help predict embolic events, as larger vegetations are associated with a higher risk.[5, 6] In fact, some of the indications to perform surgery by the international guidelines[1, 2] rely on the size of the vegetation and provide a very specific cut-off above which a patient must be sent to surgery.

Our proposed system presents several advantages: first, it has the potential to eliminate both intra- and inter-operator variability in vegetation measurement, thereby enabling a standardized assessment and supporting more consistent clinical decision-making. Second, by providing objective and reproducible measurements, the algorithm has the potential to become the basis for future updates to clinical guidelines, allowing cut-off values for management decisions to be based on standardized and quantifiable parameters. Third, this system may establish the foundation for a novel advanced vegetation characterization, incorporating features such as motion dynamics (e.g., acceleration, velocity and trajectory) and morphological features (shape, density and others) that may offer new insights into disease behavior and prognosis. And fourth, given its diagnostic performance, the tool is suitable for integration into clinical workflows as a decision support system, particularly in smaller healthcare centers and other settings with limited access to expert imaging interpretation.

The proposed system is designed to detect vegetations at the frame level; however, to determine whether a suspicious mass is a vegetation, clinicians usually review the entire video to assess its motion, which makes it reasonable to expect that an AI model incorporating a temporal component could perform better. This approach would face several challenges. Labeling becomes extremely demanding in models that require all frames containing the object to be annotated, as unannotated frames may be interpreted as negative cases. This is not only an exponential increase in the labeling workload, but often infeasible, since during parts of the cardiac cycle when vegetations are partially obscured, it is difficult to distinguish them from cardiac valves or abscesses. In addition, it would also imply a risk of underfitting, since the complexity of the input would increase (video sequence vs. frame) while the number of available samples would decrease exponentially (while we could extract 50,172 vegetation frames from our 329 patients, only 2517 cine loops would be available for video training).

Therefore, our model was designed to detect vegetations on a frame-by-frame basis, followed by a temporal evaluation of these detections, providing a more efficient and practical solution.

The number of frames varies substantially across studies, such that patients with fewer frames are more susceptible to false detections, as each misclassification exerts a disproportionately greater influence on the overall results. This limitation is particularly evident in older ultrasound examinations acquired with earlier-generation systems, which are typically characterized by lower temporal resolution and reduced image quality. Despite these considerations, no exclusion criteria based on frame count or number of loops were imposed, given that short loops are frequently observed in routine clinical practice.

A key limitation of the current system is that it is designed to operate within a clinical context of suspected endocarditis. Consequently, the model was not specifically trained to differentiate vegetations from other intracardiac masses. This means anomalies such as Lambl’s excrescences, thrombus, or ruptured chordae tendineae may currently be misidentified as vegetations. Future developments should address this by extending the training dataset to include a broader range of intracardiac formations, enabling the system to accurately distinguish these structures and expand its clinical applicability. Other limitations include the exclusion of right-sided IE from the population and the relatively small number of patients with prosthetic valves, particularly mechanical. The system would need to be trained with additional datasets focused on these populations before being fully applicable in these contexts, particularly in the latter case since artifacts caused by the prosthesis may confound the algorithm.

Conclusions

The results of this study demonstrate the effectiveness of a new automated vegetation detection algorithm in TEE images. The algorithm achieved high-performance metrics in detecting and locating vegetations down to the frame level and diagnosing the presence of vegetations in TEE video sequences at the patient level. This system could be directly employed to assist in the assessment of vegetation presence, but, more importantly, it may serve as the foundation for future advanced image-based characterization of vegetations, computed from programmatically derived (i.e., operator-independent) parameters such as motion velocity, trajectory, and morphological features. This can be extremely useful to reduce inter-operator variability in the vegetation measurement task, often subject to significant variability among operators, and to identify new advanced imaging biomarkers for personalized risk characterization and disease management.

Supplementary Information

Below is the link to the electronic supplementary material.

Abbreviations

IE

Infective endocarditis

TEE

Transesophageal echocardiographic

CNN

Convolutional neural networks

AI

Artificial intelligence

IOU

Intersection-over-Union

AUC

Area under the receiver-operating characteristic curve

ROC

Receiver-operating characteristic

Author Contribution

Study conception and design was carried out by Alberto San Román, Daniel Pinilla-García, and Carlos Baladrón. Patient recruitment and data collection were performed by all authors. Data and image processing, analysis, and labeling were carried out by Daniel Pinilla-García, Luis Llamas-Fernández, Itziar Gómez-Salvador, Manuel Carrasco-Moraleja, Teresa Sevilla, Javier López, J. Alberto San Román, and Carlos Baladrón. Model design, development, and training were performed by Daniel Pinilla-García, Luis Llamas-Fernández, and Carlos Baladrón. The manuscript was written by Daniel Pinilla-García, Luis Llamas-Fernández, Itziar Gómez-Salvador, Manuel Carrasco-Moraleja, Teresa Sevilla, Javier López, J. Alberto San Román, and Carlos Baladrón. All authors reviewed, read, and approved the final manuscript.

Funding

This work was supported by Instituto de Salud Carlos III (Grant number PI23/01072), Gerencia Regional de Salud de Castilla y León (Grant number GRS2929/A1/2023), and “La Caixa” Foundation (Grant number CI24-10743).

Instituto de Salud Carlos III,PI23/01072,Fundación La Caixa,CI24-10743,Gerencia Regional de Salud de Castilla y León,GRS2929/A1/2023

Declarations

Ethics Approval

Approval was granted by the Ethics Committee “CEIm de las Áreas de Salud de Valladolid” (Date 13MAR2024/No PI-24–200-C). The ethics committee approved a waiver of informed consent.

Consent to Participate

The ethics committee approved a waiver of informed consent.

Consent for Publication

No personal data of the patients has been published.

Competing Interests

The authors declare no competing interests.