Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2026 Oct 15.
Published in final edited form as: IEEE Trans Med Robot Bionics. 2024 Oct 14;6(4):1405–1408. doi: 10.1109/tmrb.2024.3475827

ENTRI: Enhanced Navigational Toolkit for Robotic Interventions

Manish Sahu 1, Hisashi Ishida 2, Laura Connolly 3, Hongyi Fan 4, Anton Deguet 5, Peter Kazanzides 6, Francis X Creighton 7, Russell H Taylor 8, Adnan Munawar 9
PMCID: PMC13664639  NIHMSID: NIHMS2212678  PMID: 42877242

Abstract

Image-guided robotic interventions represent a transformative frontier in surgery, blending advanced imaging and robotics for improved precision and outcomes. This paper addresses the critical need for integrating open-source platforms to enhance situational awareness in image-guided robotic research. We present an open-source toolkit, named ENTRI, that seamlessly combines a physics-based constraint formulation framework, AMBF, with a state-of-the-art imaging platform application, 3D Slicer. ENTRI facilitates the creation of highly customizable interactive digital twins, that incorporate processing and visualization of medical imaging, robot kinematics, and scene dynamics for real-time robot control. Through a feasibility study, we showcase real-time synchronization of a physical robotic interventional environment in both 3D Slicer and AMBF, highlighting low-latency updates and improved visualization. The source code and supplementary materials for this study are available at https://github.com/LCSR-CIIS/ENTRI.

Index Terms—: Robot-assisted surgery, surgical navigation, image-guided surgery, robot operating system, digital twin

I. Introduction

Image-guided robotic interventions represent an evolving frontier in surgical procedures, seamlessly blending robotic and imaging technologies to equip surgeons with enhanced information, refine precision, and improve outcomes [1]. Active imaging serves as a navigation tool for guiding surgical interventions [2], while robotic assistance enhances proficiency by refining tool tip precision and mitigating the impact of hand tremors [3].

The development of intelligent image-guided robotic systems necessitates the integration of complementary situational awareness [4], incorporating contextual models of patients and intra-operative devices. These models interpret surgical situations, delivering real-time feedback tailored to the ongoing procedure. Physics-based robotics simulations facilitate dynamic interaction with robotic models in a surgical virtual environment [5]. Presently, 3D Slicer [6], an open-source medical imaging platform, stands as a widely adopted tool for research and prototyping in image guidance [7]. Simultaneously, Robot Operating System (ROS) [8] is a widely popular open-source communication middleware for robotics research. However, 3D Slicer lacks a straightforward interface for surgical robotics applications [9]. To fully leverage the potential of image-guided robotic surgery, there is a crucial need to integrate open-source navigation and robotics platforms.

Developing complementary situational awareness for image-guided robot-assisted research encounters a significant challenge: the absence of flexible open-source frameworks that empower researchers to swiftly develop and test systems tailored to specific surgical contexts. The simulation requirements for a situation-aware system are multi-faceted, encompassing the need for a real-time, physics-based dynamic virtual environment. This environment should seamlessly integrate various components, including surgical robots and patient anatomy, and extend its capabilities to provide real-time feedback aligned with safety constraints and the surgical context. Moreover, to ensure wide robotics compatibility, the platform must accommodate any robot supported in ROS.

This paper outlines the system development work focused on integrating 3D Slicer and a physics-based simulation environment, Asynchronous Multibody Framework (AMBF) [10], using ROS as the underlying communication middleware. The objective is to empower researchers by merging the visualization and registration capabilities of 3D Slicer with the dynamic and constraint-based simulation capabilities of AMBF. Through this integration, researchers gain a comprehensive toolset to advance projects in medical robotics, contributing to the overall progression of the field.

To facilitate seamless integration, ROS modules were developed for both AMBF and 3D Slicer, functioning as ROS nodes. These modules act as a bridge, conveying contextual information between 3D Slicer and AMBF. This collaborative approach leverages the strengths of both tools, with 3D Slicer providing calibration, registration, and navigation visualization, and AMBF delivering real-time physics-based constraints. The incorporation of ROS not only facilitates communication but also opens the door to potentially integrate other open-source projects. To showcase the capabilities of our integrated system, we conducted a feasibility study, demonstrating low-latency, real-time visualization of robots within the 3D Slicer environment.

II. Related Work

Image-guidance in medical robotics:

3D Slicer [6] is the most widely used medical imaging platform for research and prototyping, offering robust functionalities such as segmentation, registration, and three-dimensional visualization of medical image data. Its dedicated image-guided therapy extension, SlicerIGT [7], further enriches its capabilities by providing image registration and interfacing with external devices to support image guidance. An open-source communication protocol, OpenIGTLink [11], further extends its utility for IGT research by providing a direct interface with commercial tracking systems [12]. To foster image-guided robotics research, there have been attempts to integrate 3D Slicer and ROS by using a ROS interface with OpenIGTLink [13] or a ROS-IGTL bridge [14] implemented for the KUKA lightweight robot (LWR). However, these interfaces merely facilitate low-level data exchange between the two environments [15]. To enable more synchronous and seamless integration between the two platforms, Connolly et al. [15] developed a SlicerROS2 module to facilitate bi-directional data transfer between the Slicer Medical Reality Modelling Language (MRML [6]) scene to the ROS scene graph (Tf [16]).

Dynamic Simulation:

Within the ROS community, RViz is the default visualization tool, however, it does not simulate robot kinematics, dynamics or interaction. While several open-source simulators such as Gazebo [17] and Virtual Robot Experimentation Platform (V-REP) [18] support features for robot dynamics, these are not suitable for real-time surgical applications [10]. Asynchronous Multi-Body Framework (AMBF) [10], which enables the simulation of complex surgical robots and environments, is modular, real-time, and asynchronous, offering direct access to software (e.g., system libraries or drivers) and hardware components (input and haptic devices, optical tracking, video input, etc.). Additionally, AMBF provides a hierarchical software plugin interface for the development of customized applications. Leveraging these design choices and capabilities, AMBF has been successfully expanded into an immersive simulation system for surgical purposes [19], [20].

In this work, we aim to integrate 3D Slicer and AMBF using Robot Operating System (ROS) to utilize the extensive capabilities of 3D Slicer for visualization, processing, and registration of medical imaging data, and the physics-based constraints of AMBF for simulating the interaction of a robot with the anatomy. These constraints can be used, for example, to enforce virtual fixtures, where the real-time simulation being aware of the robot, anatomy, and the environment, feeds back the force or velocity commands to the robot to avoid critical regions. The physical realism of the environment, anatomy, and robot can be fine-tuned in AMBF to balance the computational cost. In our prior work, we implemented the computation of Signed Distance Fields (SDF) of patient anatomies in AMBF to improve task performance [21], [22]. This functionality in AMBF, thanks to its modularity, can enable navigational cues such as graphical overlays and heatmaps in addition to virtual fixtures.

III. Methods

We provide the system overview and functionality of our toolset in this section.

A. System Design

The functional requirements for prototyping of image-guided robotic systems necessitate real-time visualization of the surgical scene in 3D Slicer, coupled with bidirectional data transfer between the simulated scene and the Slicer MRML scene. The system design addresses these requirements by representing the scene in both AMBF and 3D Slicer. A crucial component of the design is providing communication between the MRML scene in 3D Slicer and the scene in AMBF represented by AMBF Description Format (ADF) [19]). To this end, we developed two modules - AMBF-ROS plugin and Slicer-ROS module. These modules contribute to the system by providing a high-level real-time description of the environment.

B. System Components and Setup

Figure 2 illustrates the workflow for digitally representing the core elements of our system, which include the patient, the robot, and the surgical tools. The models for the robot and surgical tools are derived from 3D CAD designs obtained from their respective manufacturers. These CAD files can be converted into URDF/ADF formats compatible with our system using existing conversion tools like the SolidWorks to URDF converter (sw_urdf_exporter) or the Fusion360 to URDF converter (fusion2urdf). Patient models are created by segmenting volumetric images—such as MRI or CT scans—in 3D Slicer, yielding a 3D surface model. This segmentation model can be converted into an ADF file using the Slicer ADF Plugin, resulting in the digital representation of the surgical scene in both 3D Slicer and AMBF. To ensure accurate spatial alignment of all components, we rely on optical tracking systems to track surgical tools and patient anatomy. Calibration and registration routines are executed using the SlicerIGT modules [7] to achieve precise alignment. Real-time updates of the 3D Slicer scene are facilitated through AMBF, which communicates via ROS to synchronize both environments.

Fig. 2.

Fig. 2.

Open in a new tab

The workflow of replicating a surgical scene in 3D Slicer and AMBF.

C. Scene Representation and Updates

1). AMBF:

Our system provides support for both URDF and a customized message structure representing scene data, ADF, encompassing body names (rigid bodies) and system path for meshes. This customization is essential for the AMBF simulator, which handles relative poses of rigid bodies at a lower level, allowing direct queries of rigid body poses through the ADF file.

2). 3D Slicer:

The 3D Slicer scene is organized according to MRML nodes that correspond to different data types. For this work, the links of the robot are represented as vtkMRMLModelNodes and their respective positions are stored in vtkMRMLLinearTransformationNodes. This follows the same conventions as the SlicerROS2 module in [15] where the robot description file and stereolithography (STL) or Object (OBJ) files for each link are used to generate these representations in Slicer.

Updates:

To synchronize the scene in 3D Slicer with the simulation environment, we use the ROS Transformation package (tf). During the initialization phase, the function “registerRobotMeshPaths()” in AMBF sends the filepaths of the robot’s meshes to the ROS parameter server. Subsequently, 3D Slicer reads this information to populate the scene using the function “LoadAMBFModelToScene”. During runtime, tracking data from the surgical scene is transmitted to AMBF, which processes the scene and broadcasts the tf for each rigid body. The ROS node associated with 3D Slicer retrieves this information at a frequency of 200 Hz to update the visualization in the 3D viewer, ensuring that the virtual representation accurately reflects the real-time state of the simulation.

IV. Experiments

The primary focus of our work is to evaluate the real-time behavior of the system in a surgical robotics use case, specifically assessing communication performance for surgical scene updates.

A. Surgical Robotics Use Case

To demonstrate the applicability of our system and assess its efficacy, we performed a feasibility experiment using a Robotic ENT (Ear, Nose, and Throat) Microsurgery System (REMS) which is a pre-clinical version developed by Galen Robotics (Baltimore, MD). The virtual models of REMS are loaded into AMBF and with their movements synchronized by referencing the actual state of the real REMS. The phantom was registered using optical tracking and the 3D Slicer registration toolkit. Our system demonstrated efficient real-time tracking of the robotic system and anatomy, updating the virtual representations seamlessly in both AMBF and 3D Slicer.

B. Performance Evaluation

For evaluating the performance of the system, REMS model comprising 25 rigid bodies and a skull model was loaded into AMBF. The 3D Slicer ROS module was designed to update at a maximum frequency of 200Hz. The evaluation consisted of actuating the robot within AMBF, and for each update, we measured the round-trip delay (RTD). The results are presented in Fig. 5, indicating Mean:19.98 ms, Median:18.99 ms, std:4.40 ms. Given that our system focuses on the unilateral communication from AMBF to 3D Slicer, we can reasonably assume that there is approximately 10 ms delay between the components. These statistics show that the RTD for the ROS module is stable at around 19 ms (9.5 ms for one-way delay). Considering the lowest reported latency of 16 ms that impacts humancomputer interaction (HCI) tasks [23], we believe the latency of our system will not significantly affect the operator’s tasks in real-world applications.

Fig. 5.

Fig. 5.

Open in a new tab

Latency analysis: Round Time Delay (RTD) over 1000 frames.

V. Discussion and Conclusion

In this work, we addressed an essential need for an open-source navigation and robotics platform integration by seamlessly integrating 3D Slicer, a widely adopted medical imaging platform, with the physics-based simulation environment AMBF, using ROS as the underlying communication middleware. The developed ROS modules, acting as nodes for both AMBF and 3D Slicer, serve as a bridge, facilitating real-time synchronization of scene states between the simulation and the 3D Slicer. Our implementation demonstrates the feasibility of achieving low-latency, real-time visualization of robotic interventions within the 3D Slicer environment. This integration represents a substantial contribution to the field of robotic intervention research and development, harmonizing the strengths of 3D Slicer in visualization and registration with AMBF’s dynamic and constraint-based simulation capabilities. The collaborative approach leverages ROS to enhance communication and holds the potential for future integration with additional open-source projects, fostering a more comprehensive toolset for researchers in medical robotics.

For future work, we aim to enhance the system by developing assistive shared control functions tailored for various robot-assisted ENT procedures. Recognizing surgeons’ familiarity with the intuitive 3D Slicer interface, we plan to empower them with interactive planning within 3D Slicer, while incorporating assistive functions based on the physics interactions between the anatomy and surgical tools into AMBF. Computational feedback generated within AMBF will be dynamically reflected back to the 3D Slicer interface, offering surgeons a comprehensive and intuitive platform for planning intricate surgical procedures. This integration of computational feedback with the 3D Slicer interface aims to provide surgeons with a cohesive and user-friendly environment that aligns with their preferences and enhances their decision-making processes.

Fig. 1.

Fig. 1.

Open in a new tab

Overview of integration of 3D Slicer and AMBF. Different features for each software tool are highlighted around their respective logos.

Fig. 3.

Fig. 3.

Open in a new tab

Higher level structure of Slicer-ROS Module connected with AMBF Simulator. AMBF simulator and 3D slicer communicate Scene data and kinematics using ROS as an intermediary middleware.

Fig. 4.

Fig. 4.

Open in a new tab

Overview of the experimental setup for the simulated robotic sinus procedure. Galen robotics system holds an endoscope aimed at the sinus phantom (Screen A). AMBF simulator acquires motion data from the Galen robot, achieving real-time synchronization with the 3D Slicer application through the integration of a novel pipeline (Screen B).

Acknowledgment and Disclosures

Russell Taylor and Johns Hopkins University (JHU) may be entitled to royalty payments related to the technology discussed in this paper, and Dr. Taylor has received or may receive some portion of these royalties. Also, Dr. Taylor is a paid consultant to and owns equity in Galen Robotics, Inc. These arrangements have been reviewed and approved by JHU in accordance with its conflict of interest policy. This work was presented as a podium presentation at the HamlynSymposium on Medical Robotics 2023.

This work was supported in part by a research contract from Galen Robotics; in part by NIDCD K08 under Grant DC019708; in part by a research agreement with the Hong Kong Multi-Scale Medical Robotics Centre; and in part by Johns Hopkins University internal funds.

Contributor Information

Manish Sahu, Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD 21218 USA.

Hisashi Ishida, Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD 21218 USA.

Laura Connolly, Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD 21218 USA, and also with the School of Computing, Queen’s University, Kingston, ON K7L 3N6, Canada.

Hongyi Fan, Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD 21218 USA.

Anton Deguet, Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD 21218 USA.

Peter Kazanzides, Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD 21218 USA.

Francis X. Creighton, Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD 21218 USA, and also with the Department of Otolaryngology-Head and Neck Surgery, Johns Hopkins University School of Medicine, Baltimore, MD 21205 USA

Russell H. Taylor, Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD 21218 USA, and also with the Department of Otolaryngology-Head and Neck Surgery, Johns Hopkins University School of Medicine, Baltimore, MD 21205 USA.

Adnan Munawar, Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD 21218 USA.

References

  • [1].Fichtinger G, Troccaz J, and Haidegger T, “Image-guided interventional robotics: Lost in translation?” Proc. IEEE, vol. 110, no. 7, pp. 932–950, Jul. 2022. [Google Scholar]
  • [2].Cleary K and Peters TM, “Image-guided interventions: Technology review and clinical applications,” Annu. Rev. Biomed. Eng, vol. 12, pp. 119–142, 2010. [DOI] [PubMed] [Google Scholar]
  • [3].Taylor R et al. , “A steady-hand robotic system for microsurgical augmentation,” Int. J. Robot. Res, vol. 18, no. 12, pp. 1201–1210, 1999. [Google Scholar]
  • [4].Simaan N, Taylor RH, and Choset H, “Intelligent surgical robots with situational awareness,” Mech. Eng, vol. 137, no. 9, pp. S3–S6, 2015. [Google Scholar]
  • [5].Choi H et al. , “On the use of simulation in robotics: Opportunities, challenges, and suggestions for moving forward,” Proc. Nat. Acad. Sci, vol. 118, no. 1, 2021, Art. no. e1907856118. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [6].Fedorov A et al. , “3D slicer as an image computing platform for the quantitative imaging network,” Magn. Reson. Imag, vol. 30, no. 9, pp. 1323–1341, 2012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [7].Ungi T, Lasso A, and Fichtinger G, “Open-source platforms for navigated image-guided interventions,” Med. Image Anal, vol. 33, pp. 181–186, Oct. 2016. [DOI] [PubMed] [Google Scholar]
  • [8].Quigley M et al. , “ROS: An open-source robot operating system,” in Proc. ICRA Workshop, vol. 3, Kobe, Japan, 2009, p. 5. [Google Scholar]
  • [9].Connolly L et al. , “An open-source platform for cooperative, semiautonomous robotic surgery,” in Proc. ICAS, 2021, pp. 1–5. [Google Scholar]
  • [10].Munawar A and Fischer GS, “An asynchronous multi-body simulation framework for real-time dynamics, haptics and learning with application to surgical robots,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), Nov. 2019, pp. 6268–6275. [Google Scholar]
  • [11].Tokuda J et al. , “OpenIGTLink: An open network protocol for image-guided therapy environment,” Int. J. Med. Robot, vol. 5, no. 4, pp. 423–434, 2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [12].Lasso A, Heffter T, Rankin A, Pinter C, Ungi T, and Fichtinger G, “PLUS: Open-source toolkit for ultrasound-guided intervention systems,” IEEE Trans. Biomed. Eng, vol. 61, no. 10, pp. 2527–2537, Oct. 2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [13].Tauscher S, Tokuda J, Schreiber G, Neff T, Hata N, and Ortmaier T, “OpenIGTLink interface for state control and visualization of a robot for image-guided therapy systems,” Int. J. Comput. Assist. Radiol. Surg, vol. 10, pp. 285–292, Mar. 2015. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [14].Frank T, Krieger A, Leonard S, Patel NA, and Tokuda J, “ROS-IGTL-bridge: An open network interface for image-guided therapy using the ROS environment,” Int. J. Comput. Assist. Radiol. Surg, vol. 12, pp. 1451–1460, Aug. 2017. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [15].Connolly L et al. , “Bridging 3D slicer and ROS2 for image-guided robotic interventions,” Sensors, vol. 22, no. 14, p. 5336, Jul. 2022. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [16].Foote T, “tf: The transform library,” in Proc. TePRA, 2013, pp. 1–6. [Google Scholar]
  • [17].Koenig N and Howard A, “Design and use paradigms for gazebo, an open-source multi-robot simulator,” in Proc. IROS, vol. 3, 2004, pp. 2149–2154. [Google Scholar]
  • [18].Rohmer E, Singh SP, and Freese M, “V-REP: A versatile and scalable robot simulation framework,” in Proc. IROS, 2013, pp. 1321–1326. [Google Scholar]
  • [19].Munawar A et al. , “Fully immersive virtual reality for skull-base surgery: Surgical training and beyond,” Int. J. Comput. Assist. Radiol. Surg, vol. 19, pp. 51–59, Jan. 2024. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [20].Shu H et al. , “Twin-S: A digital twin for skull base surgery,” Int. J. Comput. Assist. Radiol. Surg, vol. 18, pp. 1077–1084, May 2023. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [21].Ishida H et al. , “Haptic-assisted collaborative robot framework for improved situational awareness in skull base surgery,” in Proc. ICRA, 2024, pp. 3588–3594, doi: 10.1109/ICRA57147.2024.10611187. [DOI] [Google Scholar]
  • [22].Ishida H et al. , “Beyond the manual touch: Situational-aware force control for increased safety in robot-assisted skullbase surgery,” Int. J. Comput. Assist. Radiol. Surg, vol. 19, pp. 1273–1280, May 2024, doi: 10.1007/s11548-024-03168-1. [DOI] [PubMed] [Google Scholar]
  • [23].Attig C, Rauh N, Franke T, and Krems JF, “System latency guidelines then and now–is zero latency really considered necessary?” in Proc. Int. Conf. Eng. Psychol. Cogn. Ergon., 2017, pp. 3–14. [Google Scholar]

RESOURCES