Automated kidney stone segmentation from flexible ureteroscopy videos using a U-net model: A preliminary feasibility study

Abstract

Manual identification of kidney stones during flexible ureteroscopy can be challenging due to motion, variable illumination, bleeding, dust from fragmentation, and limited visual contrast. While machine learning has been applied to nephrolithiasis using radiological imaging, relatively few studies have explored direct segmentation of kidney stones from ureteroscopic video, which is most relevant for intraoperative guidance. We conducted a preliminary feasibility study using 12 recorded flexible ureteroscopy procedure videos from 12 distinct patients (approximately 11 h of footage). Frames were extracted and annotated using a semi-automated pipeline combining manual bounding boxes with automated mask generation. A 2D U-Net model with a DenseNet-121 encoder was trained to segment kidney stones from video frames. Data were split at the video and patient level into training (8 videos), validation (2 videos), and test (2 videos) sets. Model performance was evaluated using pixel accuracy and Dice coefficient, and inference speed was measured on an NVIDIA T4 GPU. On the validation set, the model achieved a pixel accuracy of 94% and a Dice coefficient of 73%. On the held-out test set, it achieved a pixel accuracy of 92% with a Dice coefficient of 76%. Segmentation performance was highest when stones were clearly visible and degraded under challenging visual conditions such as bleeding, dense dust, and severe motion blur. The average inference time was 32.85 ms per frame, corresponding to approximately 30 frames per second. This study demonstrates the feasibility of deep learning–based kidney stone segmentation from real-world flexible ureteroscopy video frames. While results are preliminary and limited by dataset size, single-center data, and lack of clinical integration, the findings highlight both the potential and the challenges of intraoperative video-based stone segmentation. Future work will focus on dataset expansion, multi-center validation, and integration into real-time surgical workflows for controlled evaluation.

Introduction

Nephrolithiasis is a common health problem worldwide affecting around 12% of men and 7% of women and is known to induce a range of symptoms such as discomfort and pain as well as potential ramifications like ureteral obstruction, infections and kidney failure¹. The prevalence of kidney stone disease is on the rise globally with the occurrence of nephrolithiasis in the United States increasing from 3.2% in 1976 to 1980, to 5.2% in the years 1988–1994, and further to 8.8% during 2007–2010². Conventional surgical methods used nowadays encompass the endoscopic route through flexible ureteroscopy, however, current methods of manually detecting and treating kidney stones during endoscopic surgeries could be subject to potential human subjectivity, limited sensitivity due to blurry vision, and variable surgeon skill levels and proficiency, all of which can lead to incomplete stone removal, prolonged surgery duration, and increased risk of complications.

Although machine learning has been increasingly applied in nephrolithiasis research, most existing studies focus on radiology-based data such as CT or MRI for stone detection and segmentation, or on classifying stone types from static endoscopic images. In contrast, relatively fewer studies have addressed what is most relevant during surgery: direct segmentation of kidney stones from ureteroscopic video streams, where automated visual guidance could support the surgeon in real time.

Several notable efforts have explored deep learning–based segmentation in ureteroscopic settings. Lazo et al. demonstrated the feasibility of U-Net–based lumen segmentation in ureteroscopy, highlighting the potential of deep learning despite challenging endoscopic visuals³. Setia et al. investigated computer vision–enabled detection of kidney stone fragments during ureteroscopy and laser lithotripsy, including early intraoperative integration, using data from both digital and fiberoptic ureteroscopes and achieving detection of fragments smaller than 270 μm⁴. Stoebner et al. analyzed kidney stone segmentation in endoscopic video feeds by comparing multiple deep learning architectures, reporting U-Net + + as the best-performing model, closely followed by U-Net⁵. In addition, Gupta et al. proposed a motion-based semantic segmentation approach that leverages temporal information across consecutive frames for multi-class segmentation during ureteroscopy and lithotripsy⁶.

Despite these promising studies, existing work remains constrained by small or limited datasets and by the challenging visual conditions inherent to endoscopic surgery, including bleeding, dust from fragmentation, motion blur, illumination changes, and variation in scope types. These factors can substantially affect robustness and generalizability. To explore segmentation performance under such real-world conditions, the present study is trained and evaluated on a flexible ureteroscopy dataset consisting of 12 complete procedure videos (approximately 11 h of footage). Importantly, performance is evaluated on previously unseen procedures using a strict video-level split, providing a more realistic assessment of generalization across intraoperative variability.

The primary objective of this study is to evaluate the feasibility of intraoperative kidney stone segmentation using deep learning models trained on real-world flexible ureteroscopy video frames. Rather than aiming to establish clinical readiness or definitive robustness, this work investigates whether meaningful segmentation can be achieved under realistic surgical conditions. The dataset includes variability in lighting, motion, scope types, and visual artifacts, allowing qualitative assessment of performance and failure modes on previously unseen procedures, which can inform future model development and validation.

Materials and methods

We collected 12 videos of recorded flexible ureteroscopy procedures that were conducted on 12 distinct patients at the American University of Beirut Medical Center, culminating in around 11 h of footage. All data was deidentified and Institutional Review Board (IRB) approval was secured through the Human Research Protection Program (HRPP) at the American University of Beirut (SUR.AE.03) following their guidelines and protocols. Patients provided written informed consent for video recordings of their surgeries for research purposes in the preadmission unit before surgery and all their questions were answered. All methods were performed in accordance with the World Medical Association Declaration of Helsinki ethical principles and guidelines.

Three different types of digital single-use ureteroscopes were used in these procedures LithoVueTM (Boston Scientific, Marlborough, MA), UscopeUE3022 (Zhuhai Pusen Medical Technology Co, Ltd, Zhuhai, China) and EU-scope™ (Innovex Medical Devices Co. Shanghai, China). An 80 W holmium laser with 272micron laser fibers was utilized for all cases. The Proximie⁷ platform was used to record the cases.

Out of the 12 videos, 8 were used for training, 2 for validation and 2 for testing. Video frames were extracted using OpenCV. Frames were first manually filtered to remove non-informative content (e.g., frames outside the surgical field or with complete occlusion). The remaining frames were annotated through a semi-automated pipeline. Computer science students, supervised by clinical experts, manually drew bounding boxes around visible stones and clinically relevant fragments using LabelMe. Stones were distinguished from surrounding tissue, bleeding, and debris based on visual characteristics such as irregular shape and characteristic white to yellow–dark coloration. Pixels corresponding to the blue laser fiber were excluded using color-based filtering.

To assist mask generation, the Segment Anything Model (SAM), developed by Meta, was applied to each frame to produce object-level masks. The resulting masks were filtered using the Supervision Python library to retain only those masks whose coordinates overlapped with the manually annotated bounding boxes. This process produced grayscale segmentation masks for frames containing stones and empty masks for frames without stones. All generated masks were qualitatively reviewed by clinical experts to verify correspondence with stone regions. This semi-automated approach was used to improve annotation efficiency; formal inter- and intra-rater reliability analysis was not performed and is acknowledged as a limitation of this preliminary study.

The final dataset consisted of 1,141 training frames (684 stone-positive and 457 stone-negative), 688 validation frames, and 421 test frames.

All images and corresponding masks were used to train a 2D U-Net segmentation model. The model was implemented using the keras_unet_collection library and consisted of five encoder–decoder levels with filter sizes of 64, 128, 256, 512, and 1024, with two convolutional blocks per level. A DenseNet-121 encoder pre-trained on ImageNet was used, with batch normalization enabled and all layers trainable. Input images were resized to 256 × 256 × 3, and the network produced a single binary segmentation mask using a Sigmoid activation function.

Training was performed using the Adam optimizer with default TensorFlow/Keras learning rate settings, a batch size of 8, and up to 100 epochs with shuffled batches. Binary cross-entropy loss was used, and performance was monitored using both pixel accuracy and the Dice coefficient to account for class imbalance. L2 regularization with a weight decay of 0.01 was applied to reduce overfitting. The trained model is computationally compatible with frame-wise inference suitable for real-time processing, pending integration into a complete intraoperative system. Source code for model training and evaluation is provided at the end of the manuscript.

Model performance was evaluated using two metrics. Pixel accuracy was computed as the ratio of correctly classified pixels to the total number of pixels. The Dice coefficient was used as the primary segmentation metric, as it better reflects overlap quality between predicted and ground truth masks in the presence of class imbalance.

Results

Twelve flexible ureteroscopy procedure videos were used for model training, validation, and testing. The procedures addressed the four most common kidney stone types: weddellite, whewellite, uric acid, and struvite stones. Using the described training and evaluation protocol, the model achieved a pixel accuracy of 94% and a Dice coefficient of 73% on the validation set, and a pixel accuracy of 92% with a Dice coefficient of 76% on the held-out test set.

Model performance varied across visual conditions present in the endoscopic frames. In particular, reduced segmentation quality was observed in frames affected by bleeding and dense dust generated during stone fragmentation. In such cases, the predicted masks sometimes partially overlapped with visually similar artifacts or failed to fully delineate the stone region (Fig. 1). The model was unable to detect stones in frames that were completely blurred; however, mild blur did not consistently prevent stone segmentation. In contrast, when stones were clearly visible and frames were well focused, the model was able to accurately delineate stone boundaries and generate masks that closely matched the underlying stone geometry (Fig. 2).

Fig. 1.

A: Kidney stone visualized, blood and debris surrounding it. Laser pointer in blue. B: Mask superimposed on small section of the stone.

Fig. 2.

A: Kidney stone visualized. Laser pointer in blue. B: Mask superimposed fully and correctly over kidney stone.

The dataset included variability in lighting conditions, scope types, and stone motion during laser lithotripsy across training, validation, and test sets. Within this setting, the model demonstrated the ability to segment stones and larger fragments in previously unseen procedure videos, while exhibiting reduced performance under visually challenging conditions as described above. To quantitatively assess the impact of intraoperative visual artifacts, a subgroup analysis was performed on the test set frames. Frames were categorized into two groups: “Clear” (well-focused with minimal artifacts) consisting of 333 frames and “Blurry” (presence of bleeding, dense dust or significant motion blur) representing the remaining 88 frames. The model achieved a Dice coefficient of 0.78 and an accuracy of 93% in the “Clear” subgroup, compared to 0.73 Dice coefficient and an accuracy of 98% in the “Blurry” subgroup. This 5% drop in Dice score quantitatively supports the qualitative observation that intraoperative debris and poor visibility negatively impact segmentation precision.

Inference speed was evaluated on the test set using an NVIDIA T4 GPU. The model achieved an average inference time of 32.85 ms per frame, corresponding to approximately 30 frames per second.

Discussion

Utilizing AI for endoscopic image analysis is a topic being explored in different specialties and procedures ranging from gastrointestinal endoscopy, laparoscopy, otolaryngology scopes, to cystoscopy and ureteroscopy in urology⁸. Machine learning models are being incorporated into detecting kidney stones; however, this is still minimal and has been mainly limited to using CT images as input for the predicting model to produce segmentation of kidney stones⁹ or using MR images as input for the detection and classification models¹⁰. These developed models aim at aiding in the diagnosis of nephrolithiasis in the early stages more than their actual segmentation. Other work used endoscopic images of kidney stones to classify them according to their type and composition¹¹, with successfully classifying the four most abundant types of kidney stones but failing at identifying less frequent stone types or stones with mixed compositions. These endoscopic imaging techniques are not being used to detect and segment the kidney stones. While there are available models that can segment kidney stones, using CT and MR images, only preliminary work has been published on the endoscopic segmentation of kidney stones.

In this study, we evaluated the feasibility of segmenting kidney stones from real-world flexible ureteroscopy video frames using a deep learning–based approach. Performance was assessed on previously unseen procedure videos using a strict video- and patient-level split, with the goal of characterizing segmentation behavior under realistic intraoperative conditions rather than establishing clinical readiness. This work should therefore be interpreted as a preliminary feasibility investigation.

Several important limitations must be considered. First, despite using 12 complete procedure videos, the number of annotated frames remains relatively small and may not capture the full spectrum of intraoperative variability. In addition, manual frame filtering prior to annotation may introduce selection bias by underrepresenting the most challenging visual conditions. Second, all data were collected from a single institution using a limited set of scope models, and no external validation was performed. Third, annotations were generated through a semi-automated pipeline combining manual bounding boxes with SAM-based mask extraction, which may introduce systematic labeling errors, particularly in frames affected by bleeding, dust, or unclear stone boundaries. Finally, although inference speed was measured on a T4 GPU, the system has not been validated as a fully integrated real-time tool in the operating room, where latency, hardware constraints, and safety requirements are more stringent. The gap between training and held-out test performance also suggests that some degree of overfitting cannot be ruled out, particularly given the limited dataset size and model complexity, and further underscores the need for larger datasets and external validation.

Analysis of failure cases provides insight into the challenges of endoscopic stone segmentation. The model performed best when stones were clearly visible and frames were well focused, but segmentation quality degraded in the presence of bleeding, dense dust from fragmentation, and severe motion blur. Bleeding can reduce contrast and introduce textures and colors that resemble stone surfaces, leading to false positives or incomplete segmentation. Dust and debris generate small, bright, irregular patterns that blur the boundary between stone, fragments, and background. Motion blur and defocus remove fine-grained edge information that is critical for pixel-level segmentation. The quantitative subgroup analysis confirms that while the model is resilient to mild artifacts, dense dust and motion blur lead to a measurable performance decline (Dice 0.73 vs. 0.78). These results show that the model has trouble distinguishing stones when laser dust makes the image look blurry, and because the software only analyzes one image at a time, it cannot easily separate the stone from these visual distractions.

While this study focuses on segmentation feasibility, automated stone localization could support future applications such as continuous stone tracking, intraoperative decision support during laser lithotripsy, or integration with robotic platforms. However, these applications are not demonstrated by the present experiments and would require substantial engineering effort, integration into the surgical workflow, and rigorous safety validation. In particular, any coupling of segmentation output to laser control mechanisms would demand extremely high reliability, low latency, and robust fail-safe design.

Future work should prioritize expanding the annotated dataset, incorporating a broader range of intraoperative conditions—especially bleeding, dusting, and blur—and performing external validation using multi-center data to assess generalizability. On the methodological side, robustness may be improved through temporal modeling approaches that leverage information across consecutive frames and through targeted augmentation strategies designed to address common visual artifacts. Before clinical deployment, the system should be evaluated end-to-end in a real-time surgical workflow using recorded live streams or controlled pilot studies.

Conclusion

Artificial intelligence and machine learning are increasingly being explored to support intraoperative decision-making across surgical disciplines. In the context of nephrolithiasis, automated segmentation of kidney stones from ureteroscopic video has the potential to assist surgeons by improving visualization during stone treatment.

In this study, we evaluated the feasibility of segmenting kidney stones from real-world flexible ureteroscopy video frames using a deep learning model trained on intraoperative data. On held-out procedure videos, the model achieved a validation accuracy of 94% with a Dice coefficient of 73%, and a test accuracy of 92% with a Dice coefficient of 76%, while exhibiting clear performance degradation under challenging visual conditions such as bleeding, dust, and severe blur.

These findings demonstrate the potential of video-based stone segmentation while highlighting important limitations related to dataset size, visual variability, and lack of clinical integration. Future work will focus on expanding the dataset, performing multi-center validation, and integrating the model into a real-time surgical workflow for controlled evaluation prior to any clinical deployment.

Author contributions

**Dr. Albert El Hajj** Conceptualization, Supervision, Writing-Review and Editing; **Dr. Rida Assaf** Methodology, Software, Validation, Formal Analysis, Investigation, Writing and Editing; **Anthony Bou Mrad** Formal Analysis, Writing- Original draft, Writing- Review and Editing; **Eva Malik** Writing- Original draft, Writing- Review and Editing; **Khaled Gheith** Methodology, Software, Validation, Formal Analysis, Investigation; **Joudy Al Ashkar** Methodology, Software, Validation, Formal Analysis, Investigation; **Nour Obeid** Methodology, Software, Validation, Formal Analysis, Investigation.

Funding

No funding was used for this project.

Data availability

The datasets generated and analyzed during this study are not publicly available due to patient privacy and institutional restrictions. De-identified data may be made available from the corresponding authors upon reasonable request and subject to appropriate ethical and data-use approvals.

Declarations

Competing interests

The authors declare no competing interests.

Ethics approval

Our methods and procedures were performed in accordance with the World Medical Association Declaration of Helsinki ethical principles and guidelines.

Patient consent statement

Patients were consented in writing and verbally for approval of recording their surgeries intraoperatively using a form filled out preoperatively.