Intelligent magnetic manipulation for gastrointestinal ultrasound

Abstract

Diagnostic endoscopy in the gastrointestinal tract has remained largely unchanged for decades and is limited to the visualization of the tissue surface, the collection of biopsy samples for diagnoses, and minor interventions such as clipping or tissue removal. In this work, we present the autonomous servoing of a magnetic capsule robot for in-situ, subsurface diagnostics of microanatomy. We investigated and showed the feasibility of closed-loop magnetic control using digitized microultrasound (μUS) feedback; this is crucial for obtaining robust imaging in an unknown and unconstrained environment. We demonstrated the functionality of an autonomous servoing algorithm that uses μUS feedback, both on benchtop trials as well as in-vivo in a porcine model. We have validated this magnetic-μUS servoing in instances of autonomous linear probe motion and were able to locate markers in an agar phantom with 1.0 ± 0.9 mm position accuracy using a fusion of robot localization and μUS image information. This work demonstrates the feasibility of closed-loop robotic μUS imaging in the bowel without the need for either a rigid physical link between the transducer and extracorporeal tools or complex manual manipulation.

Introduction

Diseases of the gastrointestinal (GI) tract are numerous and severe, accounting for approximately 8 million deaths annually worldwide (1). They include cancers, such as colorectal, gastric, and esophageal, and inflammatory bowel disease (IBD), e.g. ulcerative colitis and Crohn’s disease (1, 2). High grade bowel cancers are associated with high mortality (3), while chronic IBD conditions are associated with poor quality of life (4) and increased risk of further complications that include bowel cancer (5). Diagnosis and management of GI diseases are typically performed using optical flexible endoscopy (FE). This is the current gold standard despite numerous drawbacks and very few design improvements since its introduction (6). FE relies on a gastroenterologist, or operator, guiding the semi-rigid insertion tube of the endoscope through the mouth or anus while steering only its tip. This distal end contains a camera for inspecting the mucosa (surface anatomy) for areas of interest and an instrument channel for performing interventions (7). Advancing (pushing, pulling, and twisting) the endoscope by manipulating the proximal end results in tissue stretching that causes patient discomfort, and in extreme cases, can perforate the bowel (8). A further limitation of FE is that inspection of the small bowel is challenging and is not routinely conducted; it can be costly, long in duration, difficult to perform and painful for the patient (9). This has motivated the development and use of capsule endoscopy (CE) – a disruptive technology in which a patient swallows a small diagnostic capsule (10). CE has the advantages of ease of administration, minimal discomfort, and the ability to transit the entire GI tract; however, capsules are limited by passive locomotion (via peristalsis) and cannot carry out important interventions (11, 12).

GI disease diagnosis under current practice can be onerous and time consuming. Biopsies are invasive (introducing the risk of adverse events), incur additional costs such as increased workload (13–16) and cannot be performed using CE. Many mucosal defects are readily identifiable while others can be more subtle and obscure in appearance. Therefore, guidelines recommend multiple biopsies of tissue with abnormal appearance and in some cases, normal appearance; the route taken is dependent on the disease under investigation (14). After collection, the tissue specimens are studied in a pathology laboratory and a patient may be requested to return for a follow-up FE, adding further cost and prolonging disease treatment. In-situ histology that is both non-invasive and facilitates rapid diagnosis would therefore be extremely beneficial (17).

Several imaging modalities have been investigated to enhance in-situ capabilities of diagnostic devices that may expedite, or eliminate, post-procedural pathology. Examples include: narrow-band imaging (18), auto fluorescence imaging (19), Raman spectroscopy (20), x-ray imaging (21), computed tomography (CT) (22), positron emission tomography (PET) (23), magnetic resonance imaging (MRI), optical coherence tomography (24) and endoscopic ultrasound (EUS) (25). Of these modalities, EUS is safe, low-cost, and has the ability to generate higher resolution cross-sectional images than CT, PET, or MRI; furthermore, it has already been demonstrated to provide accurate staging of GI malignancies, including rectal cancer (26, 27). Recent advancements in EUS technology have enabled the study of thin (<3 mm) bowel layers by using high-frequency (> 20 MHz), and therefore high resolution, ultrasound (US) - called microultrasound (μUS) (27–29). This ability to accurately measure the dimensions of subsurface tissue anatomy is known to be beneficial, as an increase in tissue thickness can be indicative of inflammation and disease (27). Only recently, μUS technology has been integrated into ingestible capsules (28, 30) that have been shown to generate US images that correlate closely with GI tissue histology (31). For the first time, this offers the potential to perform EUS, and specifically μUS, in all regions of the GI tract.

Although these capsules have acquired μUS images in the small bowel, they are limited in their capabilities as they are passive and so subject to the environment for both acoustic coupling, via tissue conforming to the capsule, and locomotion, via peristalsis. This uncontrolled approach makes operator-led targeted diagnosis and intervention impossible. It also fundamentally limits US image acquisition in other areas of the GI tract, such as the stomach and colon, where the cavity is significantly larger than the capsule. In these situations, the capsule is less constrained by the environment and acoustic coupling with the tissue cannot be guaranteed. The problem is compounded by the presence of air, which is an effective barrier to acoustic coupling. A robotic system that enables robust, targeted imaging in all locations in the GI tract by actively engaging with the environment could solve this. Furthermore, knowledge of the pose of the transducers could allow the sizes of lesions and their relative location in the GI tract to be identified. If implemented clinically, such a system, offering targeted in-situ diagnostics could reduce procedure time, improve pathology turn-around time, and enhance image-guided diagnostic yield, while also reducing physical and cognitive burden on the user.

The majority of previous work on robotic US can be grouped into three categories: (Cl) robot-assisted probe manipulation, where robot motion is open-loop with respect to US imaging (32–36) and robotics is used to aid in teleoperation; (C2) visual servoing using US image feedback for the steering of a tool, where US is used as a method of tracking (visualizing) a robotic tool and aiding its navigation towards a target (37–44); and (C3) visual servoing using US image feedback for the steering of the US transducer, where the US field of view can be actively, and even autonomously, adjusted (45–49). The latter two categories involve the processing of US images and a robotic control system that operates in closed-loop with respect to the US information. To the best of our knowledge, all previous work in this field has concerned the actuation of extracorporeal robots or robots that have been partially inserted in a human via rigid links, e.g. transrectal US (TRUS) robots (50). These robotic probes must maintain contact with the tissue to facilitate the transfer of energy in the form of US (51) and are limited to a working range that is proximal to the anus and thus are not feasible for navigating a fully mobile in-vivo μUS transducer.

In this work, we investigate the feasibility of magnetically actuating a μUS probe in the form of a tethered robotic capsule endoscope (RCE). Magnetic actuation allows for the application of force and torque on the intracorporeal device and does not require a rigid-link connection between an extracorporeal actuation unit (i.e. permanent- or electro-magnet) and an intracorporeal robot. Magnetic manipulation and μUS complement each other, with magnetic attraction inherently providing intimate contact with the tissue, which is crucial for reliable acoustic coupling. The μUS transducer signals can also be used in the robotic control loop to provide real-time feedback on the interaction between the RCE and its environment, supplementing the existing feedback modalities – vision, pose and force - available for intelligent magnetic manipulation (52–54). In this approach, we continuously acquire ID signals from a μUS transducer in the RCE, process these signals, and use the feedback in a magnetic actuation control loop. We have developed an autonomous control routine that facilitates the searching of a transducer configuration (i.e. RCE pose) that results in μUS image acquisition, without the user needing to learn to manually navigate the robot.

Our work contributes to the aforementioned categories C2 and C3 as we use US feedback to inform robot motion and use robot motion to reposition the μUS transducer, i.e. the RCE. This is the first application of: US-based servoing of an in-vivo device with embedded transducers, μUS-based servoing using ID US signals, and the use of intelligent magnetic manipulation (i.e. probes actuated with a non-rigid link) to facilitate probe access to remote regions of the GI tract. This technology advances the fields of medical robotics, EUS, and GI endoscopy, with the following key benefits: (i) the addition of a level of environmental perception that can facilitate autonomous control (55), improving overall performance and usability; (ii) the ability to implement precise RCE, and thus μUS transducer, motions with the use of magnetic manipulation; (iii) a method of analyzing μUS image quality during operation and adjusting robotic control accordingly to ensure effective μUS image acquisition; (iv) the potential to use this technology in the entire GI tract, given minor adjustments, or removal, of the tether.

The RCE system:

To show feasibility of the proposed approach, we leverage our previous work on intelligent magnetic actuation for painless colonoscopy (53) to develop both a novel RCE and autonomous μUS-robot control strategy. Our system (Fig. 1) consists of an extracorporeal permanent magnet (EPM) and an intracorporeal permanent magnet (IPM), the latter being embedded in the RCE. The RCE has magnetic field sensing capability that facilitates real-time localization of the device; the accuracy of which is approximately 2 mm and 3° (53). Throughout this manuscript, we assume this low-pass filtered localization feedback to be a ground truth. We use pose feedback for closed-loop magnetic control where positions, orientations, and velocities of the RCE can be commanded. We note that the pose repeatability of the serial arm, and thus EPM, is ±0.15 mm (LBR Med 14 R820, KUKA). This control resolution is thus the minimum bound on the control repeatability of the RCE. Environmental factors and field modelling have further effects on RCE repeatability. The forces and torques applied on the RCE are estimated using the point-dipole magnetic model. Using an on-board transducer and an oscilloscope, our system receives signal feedback in the form of waveforms. The logarithmic representation of these waveforms results in an A-scan. The waveforms are processed to develop an echo-signal-rating, or ESR. The ESR is an amplitude-based metric that determines the maximum magnitude of the received acoustic signal. It is computed by applying digital filters to a raw voltage output in digital form from μUS transducers. A high ESR rating for an acoustic waveform implies that high amplitude echoes are present in the signal, and thus acoustic coupling and adequate alignment with the environment exists. The diagnostic task of determining the content of echoes is left to the operator. Our system can capture diagnostic images while the RCE moves and can use information from the A-scans to influence magnetic control. We use this feedback to autonomously capture μUS signals in an uncertain and unconstrained environment where acoustic coupling cannot easily be confirmed. The acquisition of sequential A-scans facilitates the generation of B-scans, i.e. two-dimensional US images with a logarithmic representation of echo amplitude. A graphical representation of our RCE moving through the colon lumen while building a B-scan is shown in Fig. 1(A) and, an overview of our system is shown in Fig. 1(B). A video describing the overall concept can be found in the Supplementary Materials (Movie S1).

Fig. 1. System Description.

(A) A conceptual image of the in-vivo RCE showing real-time acquisition of localized μUS images where ultrasonic features consistent with the histological layers of the bowel wall have been annotated. The RCE contains (a) μUS transducers, (b) an LED light source, (c) an irrigation channel, (d) a camera, (e) an I PM with circuitry that facilitates real-time pose estimation of the device, and (f) a soft, flexible tether. (B) The benchtop setup, showing key components, as well as a close-up view of the RCE prototype.

Evaluation:

We conducted a series of benchtop experiments to investigate the feasibility of this concept. It included: (1) RCE Control Development – where tests were performed in a silicone phantom to better understand how to manipulate the RCE to acquire μUS images and characterize the relationship between control parameters and image quality; (2) Servoing Using Ultrasound Feedback – where the feasibility of closing the robotic control loop using US echoes was explored; (3) Benchtop System Validation – where the ESR method was validated on an acoustically realistic agar phantom. The demonstration of the system’s ability to autonomously acquire spatially relevant US images was also demonstrated during these tests; (4) In-vivo Validation – the first demonstration of a RCE acquiring μUS waveforms in a living porcine model.

Results

RCE Control Development:

We first characterized the robotic system by studying the relationship between RCE control parameters—i.e. tilt, roll, and RCE-substrate magnetic contact force—and μUS images. We found that the successful acquisition of μUS signals of sufficient amplitude by our RCE is sensitive to four main criteria: (i) the presence of an US coupling medium, (ii) transducer-tissue contact force, (iii) transducer tilt and (iv) transducer roll - all with respect to the substrate. The conventions used for the tilt and roll of the RCE can be seen in Fig. S1 (A) in the supplementary material. We conducted benchtop experiments using a silicone phantom (Fig. S1(B)) to characterize this sensitivity and validate the system’s ability to effectively manipulate the transducers (i.e. RCE).

We conducted ten instances of each test, with an operator teleoperating the EPM (i.e. serial manipulator): increasing coupling force from 0.25 N to 2.5 N (RCE tilt = 0°), adjusting RCE tilt from −10° to 10° (coupling force = 0.6 N), and adjusting RCE roll in a span of 50° (coupling force = 0.6 N). We found that a higher force can strengthen US signals; however, the dependence of signals on coupling force is not linear or repeatable (Fig. 2(A)). We found that strong acoustic coupling exists in a ± 3° tilt range of the RCE. In roll-characterization trials, we found that acoustic coupling was strong in a roll range of approximately ± 10.5°. These results show that contact force facilitates acoustic coupling (tight contact), and tilt and roll determine the transducer alignment, which is necessary to ensure that reflected echoes are successfully received.

Fig. 2. B-Scans generated during RCE benchtop trials.

(A) B-scan images with the corresponding tilt and tissue-coupling force, collected during a force sensitivity test. The black arrow identifies a wall-echo from our phantom. (B) Autonomous echo detection “Echo maintenance” routine with three disturbances. The wall-echo from the phantom is indicated by black arrows. The change in the observed depth of the back wall is attributed to variability in the phantom thickness, RCE contact force, RCE tilt, and thickness of coupling medium.

Sample filtered time-series of waveforms used for characterizing the force, tilt, and roll ranges are shown in Fig. 2(A), Fig. S2(A), Fig. S2(B), and Fig. S3(A). The signal strength of μUS waveforms is subject to significant attenuation and scattering without a coupling medium (in our case we used water and US gel). For this reason, the RCE was designed with an embedded water channel for on-demand, in-situ irrigation. When considering just the substrate, acoustic coupling is likely to be better in an in-vivo setting when compared with our silicone phantom, owing to the continuous secretion of mucus (56). A minimum contact force should be maintained but the force characterization results suggest that attempting to characterize a force-to-echo-strength relationship is likely not practical owing to a lack of robust repeatability i.e. a known force does not necessarily induce a standard echo quality. Tissue contact force should also be kept to a minimum to avoid mucosal damage. A previous study has shown that no damage was inflicted on porcine mucosa when irrigation pressures of up to 3 bar were applied (57). The maximum (worst-case) pressure expected from our RCE is 0.75 bar. This was calculated by assuming a RCE tilt angle of 45°, tissue indentation of 1 mm and a maximum contact force achievable by our current system of 2.5 N. During the tilt and roll characterization tests, and all other experimental work, the contact force did not exceed 1.5 N.

Experiments were subsequently performed to determine whether giving calibration-based pose commands, and autonomously approaching them, is sufficient for robust GI μUS imaging. We used the characterization results to determine the parameters that should be controlled and appropriate ranges for them. Tests were also performed to assess whether μUS signals could be successfully acquired while the RCE moves along a linear trajectory; these were conducted using both teleoperation and autonomous navigation. Table 1 summarizes this work. While the RCE was able to reach desired configurations and traverse paths, reliable imaging was not always achieved. This was attributed primarily to the lack of μUS feedback and the necessity for both acoustic coupling and transducer alignment.

Table 1. Autonomous RCE motion (open-loop in μUS).

A summary of the tests carried out to assess the RCE’s ability to acquire μUS signals while in motion. Two configurations (A and B) were chosen to show the efficacy and repeatability of autonomous positioning for μUS acquisition in a defined environment. All the repetitions were performed on a horizontally orientated silicone phantom (Fig. S1(B)).

Test	Description	Control strategy	Results
RCE approach configuration (A)	From arbitrary start conditions, approach −2° RCE tilt and 1 N coupling force.	Autonomous	Configuration reached. 40% of repetitions acquired μUS images (n=5)
RCE approach configuration (B)	From arbitrary start conditions, approach 3° tilt and 0.9 N coupling force.	Autonomous	Configuration reached. 100% of repetitions acquired μUS images (n=5)
Linear trajectory	Move RCE forwards in linear trajectory (8 cm) while maintaining 0–1° tilt and 0.6 N coupling force.	Teleoperated	Trajectory complete in 100% of repetitions. Weak acoustic coupling achieved in all trials. Intermittent regions of strong echoes observed (n=5).
		Autonomous	Trajectory complete in 100% of repetitions. Weak acoustic coupling achieved in all trials. Intermittent regions of strong echoes observed (n=5).

Servoing Using Ultrasound Feedback:

The acquisition of μUS echoes is dependent on successful acoustic coupling and alignment of the transducer with a substrate. Simply manipulating the RCE to a configuration or through a trajectory is insufficient for reliable μUS imaging (Table 1). Therefore, the ESR was used to provide a quantitative real-time measurement of the acoustic coupling; without this metric, a control loop could not be closed with μUS.

Our approach focuses on the amplitude of the received signal and compares it to an empirically set threshold (ESR_thresh) at and above which echoes can be clearly seen. Although an ESR below ESR_thresh may show potentially useful, but less clear, echoes, the threshold serves as a measure of the signal strength that the system strives to attain. We developed an autonomous echo-finding algorithm that uses this ESR for attaining clear acoustic images. The rating method and algorithm were evaluated during benchtop experiments. The user and autonomous system were first tasked with using the ESR to manipulate the RCE, adjusting tilt and force, to acquire echoes, i.e. Echo detection, before the system’s ability to autonomously react to external disturbances, i.e. Echo maintenance, was evaluated. Table 2 summarizes these tests and results. As seen in Table 2, the results suggest feasibility of our ESR method and its effectiveness as feedback for robust diagnostic imaging.

Table 2. Echo detection (closed loop in (μUS).

A summary of the tests carried out to assess the efficacy of using μUS feedback. During Echo detection, two substrate orientations were chosen to show repeatability and environmental adaptation. During Echo Maintenance, the external disturbances were in the form of pulling the tether and manually disturbing the pose of the EPM; these were included to simulate the presence of uncertainties that may be encountered in clinical practice. All repetitions were performed on a silicone phantom (Fig. S1(B)).

Test	Description	Substrate orientation	Control strategy	Results
Echo detection	From arbitrary start pose, adjust force and tilt until an echo is detected (i.e. ESRthresh exceeded)	Horizontal (tilt = 0°)	Teleoperated	Echoes found in 100% of repetitions (n=5)
			Autonomous	Echoes found in 100% of repetitions (n=5)
		Tilted*	Teleoperated	Echoes found in 100% of repetitions (n=5)
			Autonomous	Echoes found in 100% of repetitions (n=5)
Echo maintenance†	Adjust force and tilt to find echo, then react to external disturbances and attempt to maintain it.	Horizontal (tilt = 0°)	Autonomous	Echoes found and reestablished after disturbances in 80% of repetitions (n=5)‡

^*A fixed, but unknown (to the system) arbitrary value

^†An example B-scan from these tests is shown in Fig. 2(B)

^‡The unsuccessful repetition was attributed to a lack of US gel

Benchtop System Validation:

The previous experimental trials focused on the development of robotic control and the method of closing the loop with μUS feedback. Further tests were performed to assess the performance of the system on a more acoustically realistic phantom (Fig. S1(C)) – an agar-based material with embedded features. Additional information on this phantom can be found in the Supplementary Materials. These tests demonstrate the capability to find echoes autonomously using the ESR method and to combine RCE localization and manipulation to complete an autonomous linear trajectory while gathering images with clear features. The latter demonstrates the acquisition of spatially relevant US images, i.e. B-scans with a spatially-accurate horizontal axis. The results from these tests are summarized in Table 3. The echo detection results provide further support for the ESR method used. The teleoperated results were weaker than those seen on the silicone phantom, with the operator repeatedly failing to achieve reliable acoustic coupling. This was attributed to (i) the increased challenge of manipulating the capsule on a flatter, lower friction substrate, where the curvature of the environment does not aid alignment and (ii) the time limit set on the experiments, showing the increased cognitive burden of teleoperating the RCE (This requires the simultaneous monitoring and control of capsule pose, contact force and ESR output with sufficient reaction speed). The spatial scanning results (with an example B-scan shown in Fig. S4) suggest that it is possible to autonomously acquire spatially relevant US images by combining locomotion with autonomous echo detection – a capability that is necessary for future diagnostic tasks.

Table 3. Benchtop validation tests.

A summary of the tests carried out to validate the ESR control algorithm and demonstrate the capability of using the system to create spatially relevant ultrasound images – performed on an agar-based phantom. To better mimic the uncertainty in in-vivo conditions, the phantom was tilted by approximately 2°, unknown to the system.

Test	Description	Control strategy	Results
ESR validation	From arbitrary start pose, adjust tilt until acoustic coupling is achieved (i.e. ESRthresh exceeded). Experiment duration: 90 s.	Teleoperated (n=10)	Robust acoustic coupling achieved in 70% of repetitions, small amplitude echoes observable in other 30% of repetitions (n=10)
		Autonomous (n=10)	Acoustic coupling achieved in 100% of repetitions (n=10)
Spatial scanning (Combined locomotion and echo detection)*	Autonomous linear motion while maintaining acoustic coupling. When coupling is lost, RCE enters echo-search routine by adjusting tilt.	Autonomous (n=10, 3 features in each trajectory)	100% of large (10 mm wide) features located. 66.7% of small (3 mm wide) features located. Error in measuring length of gaps between features: 0.98 ± 0.91 mm (n=28 of 40 possible). Error in measuring all feature widths: 1.08 ± 0.89 mm (n=50 of 60 possible).

^*A representative image taken during one of these repetitions is shown in Fig. S4 in the supplementary material

In-vivo Validation:

The robotic system was evaluated in an in-vivo porcine model (Fig. 3(A)), with the primary objectives of (i) successfully merging μUS, robotic and magnetic technologies in-vivo, (ii) comparing teleoperated and autonomous probe manipulations in terms of μUS image acquisition performance and (iii) assessing the feasibility of performing targeted μUS for GI screening in a living organism. The experiments, summarized in Table 4, included both teleoperated and autonomous echo detection experiments, where the goal was to manipulate the RCE to acquire distinct in-vivo μUS images of the bowel wall.

Fig. 3. Experimental setup and results for in-vivo evaluation of our RCE.

(A) The in-vivo experimental setup in the endoscopy suite. (B) A B-Scan acquired during an in-vivo trial and the corresponding ESR with the set threshold (red dashed line). Features, characteristic of bowel wall layers, are annotated with black arrows. A deeper, tissue feature is shown with a white arrow.

Table 4. Concept validation tests (in-vivo).

A summary of the tests carried out to validate the concept – performed in a living porcine model. The Autonomous echo detection tests were monitored by the clinician for safety reasons, but no assistance was given to the system.

Test	Description	Results
Teleoperated echo detection	Adjust force and tilt manually to find and maintain US coupling (i.e. ESRthresh exceeded)	High cognitive burden. Only low amplitude echoes seen and only in 40% of repetitions (n=5)
Autonomous echo detection*	Adjust force and tilt autonomously to find and maintain US coupling (i.e. ESRthresh exceeded)	Low cognitive burden. Echoes seen in 100% of repetitions, with 60% being very distinct (n=5)

^*An in-vivo B-scan taken during these tests is shown in Fig. 3(B).

The level of electrical interference between the subsystems was acceptable and the RCE integrity was maintained throughout the tests. Tissue from the animals was inspected post mortem and no visual signs of trauma were seen. These results show that teleoperation was inadequate in this context; it was associated with high cognitive burden due to the required controlling of multiple variables simultaneously and in a complex, dynamic environment. The poor teleoperated results (Table 4) also show the importance of proper transducer alignment as inadequate alignment resulted in poor acoustic coupling and fewer received echoes, despite there being continuous magnetic attraction (RCE-tissue contact). During autonomous operation, the robot was able to find an echo and react to a change in the ESR to briefly maintain the configuration at which the echo was acquired (Fig. 3(B)). A video summarizing the in-vivo work can be found in the Supplementary Materials (Movie S3).

Discussion

With this work, we set out to answer the following fundamental questions: “Can magnetic actuation be used to facilitate targeted μUS imaging of the bowel wall?” and “What level of robotic assistance is necessary for robust μUS imaging?” In doing so, our main contributions have been: (i) the development and in-vivo evaluation of the first RCE capable of targeted μUS imaging of the GI tract, (ii) the implementation of real-time μUS echo processing to enhance the robot’s environmental awareness and (iii) the development of an autonomous control strategy that uses both magnetic field and μUS feedback for more robust μUS image acquisition. We have demonstrated that closed-loop robotic control using intelligent magnetic manipulation and μUS feedback is both feasible and advantageous as it enables our RCE to acquire submillimeter-scale, localized images of the bowel wall with minimal input from the clinician. The successful bench-top and in-vivo validations of autonomous μUS acquisition has been crucial in reaching the important milestone of determining the pre-clinical feasibility of the concept. The methods we applied in this work rely on μUS waveform feedback owing to its high resolution; however, the methods are relevant for other frequencies of US feedback. The core concepts are the use of an acoustic signal-rating and magnetic servoing to achieve a clear acoustic image.

Magnetic fields not only allow safe and efficient transmission of force and torque through living tissue without the need for a rigid connection with the intracorporeal probe, but also inherently lend themselves to the continuous transducer-tissue coupling that is crucial for μUS imaging. With the integration of magnetic control, our system is capable of precise, repeatable, and autonomous positioning of a RCE to visualize regions of interest. Precise manipulation is necessary given the minimum contact force threshold and the small range of tilt (+/− 3°) and roll (+/− 10.5°) required for successful μUS signal acquisition. We demonstrated successful manipulation in both benchtop and in-vivo environments where the RCE contact force and orientation were adjusted to achieve clear μUS signals. This is the first demonstration of robotic μUS imaging in the bowel and an important milestone to support the need for further investigation toward enabling early in-situ diagnosis anywhere along the GI tract.

Our method of echo detection was highly effective at quantifying the presence and strength of μUS echoes and robotic assistance (a degree of autonomy as defined in (55)) was crucial for the robust acquisition of μUS images. The RCE, in an unknown in-vivo environment, used μUS feedback as a sensing modality to both manipulate the RCE and “feel” the surrounding tissue, and to gain clinically-relevant histological information from the tissue in the form of μUS images of the bowel wall layers. Without real-time feedback, the user would not have known when coupling was achieved; autonomous echo detection would also have been technically impossible to implement. A significant benefit of using closed-loop control in this context is robustness and repeatability when imaging the bowel wall in an unstructured environment. Teleoperation of our RCE proved to be challenging as no intuitive mapping between magnetic field input and US image quality exists. This resulted in a significant cognitive burden as the operator was able to observe the ESR while attempting to maintain a contact force threshold and a narrow tilt range. Advancements in US array technology may reduce this sensitivity; however, the inherent limitation of manual effort to control many variables will still exist.

We found that approaching a location (using the on-board camera, for example) then activating the autonomous echo-finding routine using the ESR was a robust and intuitive imaging method. On benchtop, we demonstrated that the RCE is effective at traversing linear trajectories while maintaining acoustic coupling. This enabled us to develop spatially-relevant B-scans of an agar phantom with lateral spatial accuracy of approximately 1 mm. Our results suggest the potential of using our proposed methods to scan large areas and produce images that facilitate the observation and measurement of features. Closed-loop control was found to be essential, particularly in-vivo, as it greatly increased usability and improved overall performance, with the results showing a clear advantage compared to teleoperation. The robotic system can perform the complex task of simultaneous magnetic and US control more effectively by handling the numerous control variables quickly and accurately.

Whereas the ESR index-based control worked successfully, we believe the efficacy of our methods can be improved with a higher rate of μUS data acquisition. Our system was designed for pre-clinical feasibility and captured A-scans at less than 3 Hz. This resulted in cases where the RCE could not react fast enough to sudden interactions with obstacles in the environment. The data acquisition rate could be increased with a high performance analogue-to-digital data acquisition system, similar to that already used in the US literature (58). With a higher acquisition rate, the gradient of the ESR could be monitored and the system could detect the continuous change in signal strength and allow for appropriate motion compensation. This would also allow the system to acquire data for reconstructed 3D images of the bowel wall by continuously collecting and merging localized μUS data as the RCE moves.

The current method of magnetic actuation does not enable RCE roll-control about its central axis. It has been shown that magnetic control can be extended to all axes when manipulating the distribution of dipole elements in an actuated device (59); however, this comes with an increase in system complexity and a potential cost of implementation. As a simple solution, we chose to offset the internal magnet from the geometric center of the RCE, thus introducing a corrective torque; we also flattened the top of the RCE to increase roll resistance. These features result in a unique minimum energy configuration of the RCE in which the transducers are in contact with tissue. The most viable long-term and complete solution to this is to replace the single element μUS transducers with curved μUS arrays and completely avoid the need for precise roll control of the RCE body; an approach that has been previously explored and shown to be feasible (60).

With achievable design modifications, the RCE could be suitable for use in any region in the GI tract. Two key areas for modification are the tether and the overall size of the RCE. A tether was included to simplify the on-board electronics and provide a failsafe method to recover the RCE during in-vivo tests. If the tether is made significantly thinner and more flexible— or if it is removed completely and replaced with wireless technology— the RCE could extend its reach to the entire GI tract. Wireless technology has already been implemented in both μUS (28) and magnetic actuation (52) where human studies have shown that a wireless magnetic capsule for gastroscopy achieves similar procedural accuracy to standard gastroscopy (61). To access narrower lumens, such as those in the small bowel, the RCE must be miniaturized, with the two most significant size reductions being made by replacing the camera with a smaller alternative (e.g. 160K CMOS camera, 1.6 mm diameter, FUJIKURA) and by reducing the diameter of the IPM. While a reduction in IPM volume would reduce the applied magnetic wrench, this can be compensated with an increase in the volume of the EPM.

Upon collection of more US data and classification by experts, additional information will be gained about the environment and the clinical efficacy of using μUS for in-situ diagnosis. Using methods such as deep neural networks (62), real-time generated A- and B-Scans may potentially be processed on pre-trained networks to quickly determine if the transducer is in contact with tissue or debris or if an echo pattern corresponds to healthy or diseased tissue (63). Our algorithm picks out maxima of acoustic waveforms, but additional information is also available which artificial intelligence can more readily process. We hypothesize that the relationships between physical landmarks, i.e. the histological layers of the bowel wall, and echo signals could be learned by methods such as convolution. This could result in an algorithm that characterizes the physical (i.e. acoustic) properties of tissue, rather than the properties of the waveform as is done in the ESR. Such an AI driven approach may lend itself to autonomous diagnoses in the future. Additional processing of the B-scan images could also include feature tracking. Combined with higher data acquisition rates, this may allow micro-scale motion encoding and precise RCE pose control, e.g. for image stabilization.

MATERIALS AND METHODS

The aim of the experimental portion of this work was to evaluate the new RCE technology and determine the clinical feasibility of conducting robot-assisted μUS imaging of the GI tract wall. The objectives of this study included the evaluations of (i) the interplay between our US system and the magnetic actuation system; (ii) the ability to magnetically manipulate the RCE while acquiring μUS images; (iii) the feasibility of using real-time processing of μUS images in feedback control; (iv) the feasibility of using this feedback to find echoes autonomously; (v) the ability to acquire spatially relevant images via simultaneous locomotion and echo detection and; (vi) in-vivo performance to show pre-clinical feasibility. We begin this section with an overview of our hardware systems and provide a description of the data-flows. We then elaborate our experimental methods.

System overview

The system, Fig. 4, consists of: a custom-built RCE with embedded μUS transducers, a serial manipulator with a permanent magnet at its end effector for applying fields and gradients to the RCE, custom circuitry for magnetic localization and force and torque estimation, a μUS data acquisition system, custom software system on a desktop PC. Movie S1 provides an overview of the RCE concept.

Fig. 4. System schematic.

A diagram that summarizes the major components in the system and the data flow between them.

The RCE system

The RCE was developed and fabricated for in-vivo use. It comprises a 3D printed shell (Standard Clear resin, Form 2 3D printer, Form Labs), a camera (CMOS, MO-T1003–65-N01, Misumi), an LED light-source, an irrigation channel, a permanent magnet (cylindrical, axially magnetized, 11.11 mm diameter and length, NdFeB, N52 grade, D77-N52, K&J Magnetics, USA) encased with localization circuitry (custom flex circuit (53)) and two miniature μUS transducers. The RCE was assembled by hand, sealed with epoxy (Permabond 4UV80HV) and coated with Parylene C for biocompatibility and lubricity (64). The current prototype, measuring 21 mm in diameter and 39 mm in length, includes a tether (6 mm diameter, PEBAX material) for power and data transmission, and as a failsafe method to retrieve the device. The RCE is manipulated via the magnetic wrench between the small IPM and the larger EPM (cylindrical, axially magnetized, 101.6 mm diameter and length, NdFeB, N52 grade, ND_N-10195, Magnetworld AG, Germany) positioned at the end effector of a medical grade, 7 degree-of-freedom serial robotic manipulator (14 kg payload, LBR Med, KUKA AG).

The force estimation strategy relies on the recovery of RCE pose information. This is done using a localization strategy that generates 6 DoF, low-pass filtered, pose information at 100 Hz with approximately 2 mm and 3° accuracy (53). Throughout this manuscript, we assume that our localization reading is the ground-truth. This assumption is justified by the validation of the method described elsewhere (53). The magnetic control method used in this work relied on linearizing magnetic wrenches and using them to compute necessary joint values of a serial manipulator; it is based on previous control work (53, 65, 66). Autonomous control routines in this study utilized a nonholonomic velocity controller for translation, where velocity was commanded along a trajectory and motion orthogonal to the path was treated as a position error. Orientation control was position-level, i.e. specific angle tilts and pans were commanded as opposed to commanding angular velocities.

The control system was implemented using the robotic operating system (ROS) (67). An overview of magnetic actuation is included in the Supplementary Materials. Vectors are described using bold lettering, v, and unit vectors are described using $\widehat{v}$. All points and vectors are represented in the global frame. The dipole model is used for magnetic control and the underlying equations for the magnetic field, force (f_m), and torque (τ_m), are described in the Supplementary Materials Magnetic actuation of devices is achieved by modifying the direction and gradient of an external field; these modifications result in changes in applied torque and force, respectively. This applied torque and force, or wrench, results in motion of the actuated device. The governing control relation for robot actuation is shown in Eq. (1); this expression is similar to prior works in the field (53, 68). This relationship is a linearization of the nonlinear relationship between magnetic wrench and EPM twist. The position and heading of the EPM are indicated using p_e and ${\widehat{m}}_e$ respectively, where m_e is the magnetic moment of the magnet. The implementation of proportional-integral-derivative control is denoted by pid(). The heading of the RCE is denoted by ${\widehat{m}}_i$ and the use of the subscript “d” indicates a desired value that is commanded by a higher-level control. The heading error e_h is computed using ${\widehat{m}}_i\times {\widehat{m}}_{i_d}$ and the error magnitude is the angle between these vectors. The magnetic actuation Jacobian is $J_e\in {ℝ}^{6\times 6}$. The pseudo-inverse of J_e is given by $J_e^{+}$.

$$\left[\begin{array}{c}\delta p_e\\ \delta {\widehat{m}}_e\end{array}\right]={\left[\begin{array}{cc}\frac{\partial f_m}{\partial {\stackrel{\rightharpoonup }{p}}_e}& \frac{\partial f_m}{\partial {\widehat{m}}_e}\\ \frac{\partial {\tau }_m}{\partial {\stackrel{\rightharpoonup }{p}}_e}& \frac{\partial {\tau }_m}{\partial {\widehat{m}}_e}\end{array}\right]}^{+}\left[\begin{array}{c}\delta f_{m_d}\\ \delta {\tau }_{m_d}\end{array}\right]=J_e^{+}{\left[pid\left(e_{p_{xy}}\right),pid\left(f_z\right),pid\left(e_h\right)\right]}^T$$ (1)

The desired EPM motion is then converted to joint-level commands for the serial robot using the relationship between EPM motion and joint commands, Eq. (2) (68), where $\mathbb{S}\left(\right)$ indicates the skew-symmetric form of the cross-product operation, the purpose of which is to compensate for the symmetry of the EPM’s field about its magnetization axis. The identity matrix is represented as ${\mathbb{I}}_3$ and a matrix of zeroes is represented by ${\mathbb{O}}_3$. The geometric Jacobian J_r is that of the serial manipulator.

$$\left[\begin{array}{c}\delta p_e\\ \delta {\widehat{m}}_e\end{array}\right]=\left[\begin{array}{cc}{\mathbb{I}}_3& {\mathbb{O}}_3\\ {\mathbb{O}}_3& \mathbb{S}{\left({\widehat{m}}_e\right)}^T\end{array}\right]J_r(q)\delta q=J_A(q)\delta q$$ (2)

The changes in joint values are computed via Eq. (3), where ξ is a small damping scalar, integrated, and sent to the serial manipulator’s internal controller.

$$\delta q=J_A{(q)}^T{\left(J_A(q)J_A{(q)}^T+\xi {\mathbb{I}}_3\right)}^{-1}\left[\begin{array}{c}\delta p_e\\ \delta {\widehat{m}}_e\end{array}\right]$$ (3)

The tilt (α) and roll (γ) of the RCE are visualized in Fig. S1(A). The tilt is computed from Eq. (4). The roll is defined as rotation of the RCE about its major axis from a nominal orientation.

$$\alpha ={\text{cos}}^{-1}\left(\frac{{\widehat{m}}_i^T\left({\mathbb{I}}_3-\widehat{z}{\widehat{z}}^T\right){\widehat{m}}_i}{\Vert \left({\mathbb{I}}_3-\widehat{z}{\widehat{z}}^T\right){\widehat{m}}_i\Vert }\right)$$ (4)

Ultrasound transducers and system

We used two, 5 mm diameter PVDF (polyvinylidene fluoride) μUS transducers with a center frequency of 30 MHz and a physical focus at 6 mm (f# =1.2) mounted on the top surface of the RCE with the faces embedded approximately 1.5 mm into the RCE to bring the focal point closer to the luminal surface of the bowel wall. PVDF was used instead of more common piezoceramic because of its flexibility and transducer manufacturability; a 30 MHz transducer required a conductive film of 9 μm in thickness, making it easy to achieve a spherical focus in a small package. Data was acquired from one transducer, while a second one was installed for redundancy. The transducers were connected to a custom printed circuit board (PCB) inside the RCE. This PCB was connected to external hardware via individually shielded micro-coaxial cables (42 AWG core, 9442 WH033, Alpha Wire, USA) that provided some electrical shielding from the other RCE circuitry. The US transducers were pulsed during transmission, and the received echoes were amplified by a commercial pulser-receiver (DPR300, JSR Ultrasonics, Imaginant Inc, USA) with 70 dB gain prior to digitization with a 2-channel, 70 MHz, oscilloscope (DSOX2002A, Keysight Technologies). After digitization, signals were envelope detected using a Hilbert transform and logarithmically compressed.

Autonomous echo-finding algorithm

We implemented a waveform processing algorithm in Python to compute the ESR. This ROS node interfaced with an oscilloscope using the Python Universal Serial Bus Test and Measurement Class library and a Keysight Technologies API to acquire waveforms from the oscilloscope at a maximum rate of 3 Hz. Waveform processing consisted of: (i) trimming the raw waveform – an array of voltages – to a desired starting processing depth; (ii) applying a forward-backward linear filter using SciPy; (iii) flattening the signal using a low pass filter to make relative echo magnitudes appropriate; (iv) and computing the maximum peak-to-peak waveform amplitude range in the waveform, which is the ESR. This ESR is then filtered over time. The ESR is an amplitude-based metric that informs an operator, or computing system, of the presence of acoustic information in a signal. We note that the clinical interpretation of this signal is left to the operator. As the metric is amplitude-based, the ESR is not highly sensitive to filter parameters. To demonstrate what the μUS images that are used to compute the ESR look like, we refer the reader to Fig. S3 and Fig. S5(B) which are time-series images of the filtered waveforms that correspond to the figures shown in Fig. 2 and Fig. 3(B). Here, we emphasize the relative clarity with which an echo can be identified (when comparing Fig. 2 with Fig. S3). As the μUS transducer is used as both transmitter and receiver in the system, residual energy from the transmit pulse is seen in the form of ‘ringdown’. We therefore implemented an additional ringdown-compensation filter by subtracting a moving average of the ringdown portion of the signal at each time step as follows:

$${\stackrel{\rightharpoonup }{p}}_{Br}={\stackrel{\rightharpoonup }{p}}_B-\frac{\sum _{i=1}^n{\stackrel{\rightharpoonup }{p}}_B}{n}$$ (5)

Here, ${\stackrel{\rightharpoonup }{p}}_B\in {ℝ}^{\text{m}}$, is a vector that represents the waveform values (voltages) of length m. The number of waveforms that are used in the filter is indicated by n. The length of this vector corresponds to the depth of an A-scan. The ringdown-filtered waveform segment is referred to by ${\stackrel{\rightharpoonup }{p}}_{Br}\in {ℝ}^{\text{m}}$. This filter assists with visualizing areas of interest at smaller tissue depths which may be otherwise obscured by the ringdown feature. It is demonstrated in Fig. S5, where Fig. S5(A) shows the B-scan without the ringdown-filter, and Fig. S5(B) shows a time-series of filtered waveforms combined with ringdown compensation. The waveforms in Fig. S5(B) are used by our real-time algorithm to compute the ESR.

The echo-finding algorithm is implemented using a low US acquisition rate and aimed to show the feasibility of autonomous echo acquisition and optimization. Our autonomous method, illustrated in Fig. 5, continuously increments the desired tilt of the RCE while maintaining a constant, predefined, tissue contact force. The algorithm is activated while the RCE is in an arbitrary pose – with or without transducer-tissue coupling. The algorithm then goes to and maintains a force while tilting the RCE between hard-coded tilt limits. The control process obtains ESR updates from a separate echo-processing script and continuously checks whether the echo is above a desired threshold (ESR_thresh). The ESR_thresh is determined prior to algorithm usage via a calibration procedure where an operator teleoperates the RCE until acoustic coupling is obtained. The value of the ESR that results in an adequate acoustic signal is then used as ESR_thresh This approach was used because of the high variability between acoustic characteristics of different substrates. Once the control process observes an ESR value that is above ESR_thresh, the controller switches states from “searching” to “maintaining RCE configuration”. Here, the EPM makes motions that maintain the RCE in the pose in which distinct echoes were observed. In the case that the control process observes a loss of echo, it switches back to the “search” state (tilting routine).

Fig. 5. Block diagram control schematic of our RCE system.

The US system components are shown in yellow, the magnetic control system is shown in blue, and the auxiliary inputs and outputs are shown in green.

Benchtop Testing

Benchtop testing was performed with the primary goals of characterizing and refining the system in a known environment. It can be summarized in these categories: (i) RCE control development, where we characterized the interplay between RCE motion and μUS signal quality; (ii) servoing using US feedback, where we conducted preliminary validation of our ESR metric; and (iii) system validation, where comprehensive experiments were conducted to validate US servoing on a more acoustically relevant phantom. For characterization trials, a phantom tissue substrate, 200 mm in length and ~7 mm in thickness, was made from cast silicone (Ecoflex 00–30 silicone and 10% Slacker by mass). The phantom was placed in a semi-cylindrical acrylic tube (77 mm internal diameter) and attached to an adjustable frame. The choice of silicone and its thickness were made to allow the RCE to deform into the substrate while also allowing repeatable/comparable measurements. For validation tests, an agar-based formulation was used to develop a phantom that was acoustically closer to tissue (69–71). It included fabric mesh to simulate features of convoluted shape, micro-scale particles to provide attenuation and speckle, and several regularly sized and spaced features along its length. A full description of this phantom can be found in the Supplementary Material. Acoustic coupling gel was used as both a coupling medium and lubricant. The individual performing the experiments had experience with operating the robotic arm and the construction of the RCE. The benchtop platform and a summary of the testing can be seen in Movie S2.

Characterizing the effect RCE motion and force had on echo quality was done using the semi-cylindrical silicone substrate placed in a horizontal configuration. Given a known substrate orientation, RCE pose feedback provided the relative tilt between the transducers and the phantom. To ensure that echoes are most effectively reflected to the transducer, the transducer’s focus should be as close to normal to the substrate. Knowledge of transducer-tissue contact force, transducer tilt and transducer roll sensitivity give perspective as to which parameters of the RCE are important in acquiring and improving localized μUS images. Given that the magnetization axis of the magnet in the RCE is along its body, the RCE’s roll cannot be controlled. However, we placed the magnet off the center body axis of the RCE to induce a restorative torque to keep the transducers biased towards a vertical orientation (72).

The effect that force (RCE-substrate contact force) had on the μUS signal was characterized by conducting ten force-varying motions while data was acquired from a μUS transducer; the force was increased from 0.25 N to 2.5 N under teleoperation while tilt was kept as horizontal (i.e. 0°) as possible. To characterize the effect that RCE tilt had on the μUS signal, a similar protocol was used, but this time the operator adjusted the RCE tilt between ±10° via teleoperation, while force was kept constant at 0.6 N. The roll characterization varied in protocol in that the RCE was rolled by manually applying a torque to the device. The roll angle was varied in a 50° range while force was kept nearly constant at 0.6 N.

Appropriate control strategies and parameter (force and tilt) values for acquiring μUS signals during RCE motion were chosen based on the results from the characterization work. In the first set of experiments, the RCE autonomously approached two configurations of force and tilt. The two configurations were chosen to show control repeatability and were (A) −2° tilt and 1 N coupling force and (B) 3° tilt and 0.9 N coupling force. The phantom substrate was orientated horizontally. Before each repetition, the user placed the RCE into an arbitrary decoupled pose and force. The test was then initiated, and the system autonomously approached the desired configuration while μUS signals were continuously acquired. This autonomous maneuver can be seen in Movie S2. Tests were done to assess the feasibility of moving the RCE through a linear trajectory while continuously acquiring localized μUS signals. The RCE was placed into a desired configuration (0–1° tilt and 0.6 N force) before being moved along an 8 cm linear trajectory under both teleoperated and autonomous control regimes. Both the system and user were tasked with maintaining the desired configuration during locomotion in an attempt to maintain μUS coupling without ESR feedback. Localization data could be combined with continuously-acquired μUS signals to generate spatially-relevant μUS images, i.e. 2D B-scans of depth and width/distance. Although our recorded position is 3D, the scans are plotted in 2D with respect to the distance traveled by the RCE.

An objective assessment of ESR was not conducted as a universal standard indicator for US image quality does not exist. The ESR was first assessed subjectively by visually confirming that increases in ESR correspond to improvements in echo amplitude seen in processed B-scans. Two test protocols were then used to assess whether the ESR could be effectively used as a form of feedback in the robotic control loop (these preliminary tests were done using the silicone phantom). In the first, we conducted echo detection trials in both a horizontal phantom and a tilted phantom to simulate the unknown environment configuration that would be present during use in-vivo. Both teleoperation and autonomous control were used. The RCE was first placed in an arbitrary start pose (transducers decoupled from substrate) before the tilt was adjusted within hard limits until ESR_thresh was exceeded (i.e. μUS coupling achieved). A trial was considered successful if multiple distinct μUS echoes were captured. During the autonomous tests, once the RCE found acoustic coupling, it used magnetic closed-loop control to maintain its configuration; simultaneously, the system continued to check for a loss of acoustic coupling. The second protocol was conducted to observe whether the system could autonomously react to external disturbances and attempt to maintain μUS coupling. From an arbitrary start pose, the RCE was tasked with autonomously detecting an echo and then maintaining that configuration in the presence of disturbances. Disturbances were induced either by teleoperating the EPM to override the autonomous routine and cause unwanted motion in the RCE or by pulling on the tether of the RCE to decouple it acoustically from the substrate. Movie S2 shows this use of the ESR and an example of the disturbances used.

The acoustically-relevant agar phantom was used firstly, to validate autonomous echo detection (i.e. the ESR methodology) and secondly, for additional μUS servoing experiments that demonstrate combined echo detection and autonomous locomotion. In the former we repeated the Echo detection tests performed on the silicone phantom under both teleoperated and autonomous control regimes (n=10). In the latter, we conducted 20 closed-loop teleoperation trials where autonomous servoing was used. In 10 trials, the RCE traversed an area where large acoustic features were present. In the other 10 trials, smaller acoustic features were present. The travel distance (trajectory length) was 15 cm in all trials. We developed and validated a linear traversal strategy for the RCE where the system entered a μUS servoing mode when acoustic coupling was lost. Once coupling was recovered, the RCE attempted to maintain the orientation in which the high amplitude echo, i.e. high ESR, was achieved. This methodology proved to be robust and the majority of markers were identified. We were able to measure widths of markers and spacing between them with approximately 1 mm accuracy.

In-vivo Testing

The intended outcome of these tests was to show the core functionality (i.e. acquiring μUS echoes using a magnetically actuated RCE) and was an important pre-clinical milestone that was necessary to determine whether this technology should be explored further. Prior to the in-vivo trial, the capsule was coated with a conformal film of Parylene C using a vacuum deposition tool (SCS PDS 2010, Specialty Coating Systems, IN, USA). The surface was primed with A174 silane adhesion promoter before deposition. Parylene C is a USP Class VI polymer that is commonly used for its biocompatibility, lubricity and moisture barrier properties (64). This coating provided an additional layer of protection between the capsule and the surrounding tissue. Adhesion testing of Parylene C to the resin and leak testing of the capsule were also conducted prior to the in-vivo trial, as detailed in the Supplementary Material.

The trial was performed at a dedicated large animal research facility in the Roslin Institute, University of Edinburgh. The study was conducted under Home Office (UK) License (Procedure Project License (PPL): PF5151DAF) in accordance with the Animal (Scientific Procedures) Act 1986. The choice of a porcine as a model is based on GI anatomy comparable with human (73, 74). This includes bowel dimensions, particularly lumen caliber and bowel wall structure. The trial was conducted with the porcine (British Landrace, Female, 37 kg) placed under general, terminal anesthesia. The system was used in an operating theatre that had access to a mobile fluoroscopy machine (“C-arm” - Ziehm Vision FD). The C-arm was only used for visualizing the RCE in-situ and was not necessary for its operation. Prior to trial initiation, the bowel was cleaned gently with water to remove any residual fecal material and the ESR_thresh was calibrated in-situ. This was the main source of feedback during tests and provided a quantitative measure of transducer-tissue contact to inform RCE motion.

The RCE was inserted per rectum and was advanced approximately 20 cm into the colon by a combination of pushing the tether and magnetic actuation. Air and water were administered through the irrigation channel when deemed necessary to either distend the bowel to advance the RCE, or for improving transducer coupling. During teleoperated echo-detection experiments, the RCE was first placed in a decoupled configuration. The test was then started, and the user increased the contact force and adjusted tilt within a predefined range until multiple distinct echoes were found or until no echo was found after more than one minute of tilting motions. The RCE was then decoupled before completing the next repetition. During autonomous echo-detection tests, the user placed the RCE in the colon and decoupled the transducer in the same way as the teleoperated experiments; however, in this case, the autonomous routine was started by pressing a button. The system then performed a search routine in which the RCE was tilted within a 10° range (−15° to −25°) and the force was increased. When an echo was detected, the system autonomously attempted to maintain the current configuration (tilting back and forth to compensate for environmental disturbances). If lost again, the search routine resumed. Movie S3 summarizes the in-vivo work. Once the experiments were completed, the animal was euthanized, and necropsy was performed. There was no evidence of colonic perforation or gross trauma.