Enabling the future of colonoscopy with intelligent and autonomous magnetic manipulation

Abstract

Early diagnosis of colorectal cancer significantly improves survival. However, over half of cases are diagnosed late due to demand exceeding the capacity for colonoscopy - the “gold standard” for screening. Colonoscopy is limited by the outdated design of conventional endoscopes, associated with high complexity of use, cost and pain. Magnetic endoscopes represent a promising alternative, overcoming drawbacks of pain and cost, but struggle to reach the translational stage as magnetic manipulation is complex and unintuitive. In this work, we use machine vision to develop intelligent and autonomous control of a magnetic endoscope, for the first time enabling non-expert users to effectively perform magnetic colonoscopy in-vivo. We combine the use of robotics, computer vision and advanced control to offer an intuitive and effective endoscopic system. Moreover, we define the characteristics required to achieve autonomy in robotic endoscopy. The paradigm described here can be adopted in a variety of applications where navigation in unstructured environments is required, such as catheters, pancreatic endoscopy, bronchoscopy, and gastroscopy. This work brings alternative endoscopic technologies closer to the translational stage, increasing availability of early-stage cancer treatments.

Introduction

With over 19 million procedures performed every year in the EU and US, colonoscopy is the “gold standard” for managing colorectal diseases ¹. The need for colonoscopy is expected to rise by 16% in the next decade ², with the primary benefit being early detection and prevention of colorectal cancer (CRC) - the third most common malignancy worldwide ³. Preventive colon screening can increase early-stage detection rates for CRC where a patient’s 5-year survival rate is over 90%; survivability drops drastically to less than 10% when diagnosed at a late stage ⁴. Availability of colonoscopy is largely hindered by the aging design of the flexible endoscope (FE) used for this procedure ⁵. Originally introduced in the 1960’s ⁶, FEs have several drawbacks and have seen very few improvements. Specific design limitations of the FE ⁷ include: (i) Inherent complexity of the device preventing a single use approach, thus requiring cleaning and sterilisation ⁸. (ii) Patient pain due to tissue stretching as the endoscope is pushed through the colon, limiting social acceptance, and introducing risks such as tissue perforation and anaesthesia related adverse events. (iii) Lack of intuitiveness requiring highly trained personnel, resulting in a long and expensive training process ⁹ and shortage of endoscopists with respect to demand ¹⁰. This reduces the availability of early screening and increases the potential loss of human lives. Overcoming the limitations of FEs would allow colonoscopies to become ubiquitous and have a significant impact in early detection of malignant diseases.

The limitations of FEs have motivated the development of alternative approaches. Lack of intuitiveness and ease-of-use has been addressed by robotic actuation of conventional endoscopes ^11,12 with shorter procedure duration, but has not addressed pain and reprocessing issues. Wireless capsule endoscopes (WCE) ¹³, and internally actuated robotic devices ^14,15, address issues of pain and discomfort; however, wireless devices fail to provide therapeutic functionalities such as biopsy and removal of polyps. Moreover, the complexity of internally actuated mechanisms results in cumbersome design and prevents a significant cost reduction. Magnetically actuated endoscopes ^16–20 have demonstrated potential to reduce pain, reduce cost, enhance diagnostic capabilities ²¹, and improve therapeutic interventions. Although promising, translating magnetically actuated endoscopes for clinical use has failed due to challenges in control. External actuating magnetic fields, generated by varying electromagnetic coils ²² or by moving permanent magnets ²³, commonly mounted on robotic manipulators (Fig. 1), are nonlinearly related to the motion of the magnetic endoscope. Giving the user the complex and unintuitive task of guiding the endoscope by controlling the field requires experience and results in unsatisfactory procedure times ²⁴. Developing advanced control strategies capable of assisting and offering an intuitive user experience with reduced procedure times would serve to act to enable the clinical translation of magnetic colonoscopy, with the overarching goal of widening and improving patient care.

Fig. 1-.

Overview of the robotic MFE system. The magnetic endoscope (bottom right) is equipped with an endoscopic camera, an insufflation channel, and a working channel. Illumination is provided by an LED. A KUKA LBR Med robotic arm is used to manipulate an external permanent magnet. The endoscopic video feed is projected on a monitor with a graphical interface showing parameters such as relative robot speed and inter-magnetic distance.

Work thus far on improving navigation in magnetic endoscopy has been shown in magnetic endoscopes for gastric screening ²⁵, catheter steering ²⁶ and bronchoscopy ²⁷. In the context of mobile and complex environments such as the colon, navigation has only been shown for following simple, pre-defined trajectories ²⁸- failing to provide a substantial proof of clinical feasibility. The colon is an unstructured and dynamic environment, consisting of convoluted soft tissue, which is subject to significant variabilities due to gravity, varying patient position, peristalsis, and insufflation. Furthermore, the colon contains obstacles such as tissue folds, water, and debris. Pre-defined trajectories would soon become inaccurate in this ever-changing environment. To represent a practical and clinically viable alternative, intelligent control of magnetic endoscopes must be advanced significantly.

We hypothesise that controlling magnetically manipulated endoscopes with the introduction of with superior levels of intelligence and autonomy could increase their navigational performance. This would ultimately reduce procedure times and the mental and physical burden placed on the operator, allowing more focus on the clinical aspects of the procedure as reduced training is needed for the manual manipulation of the endoscope. This would have a positive effect on the availability of the procedure. Autonomy for magnetic endoscopes can be contextualised in the general trend towards enhanced autonomy that is gaining momentum in the field of medical robotics. Inspired by the standardisation of autonomy levels in self-driving cars ²⁹, the medical robotics community is converging towards the definition of six levels of autonomy ^30–32 characterised by increased intelligence. In this work, the discussion on our magnetic endoscope and development of autonomous control will support an analysis of how general definitions can be specified for robotic endoscopy and the features required to reach each autonomy level. Our novel contribution to the field of machine intelligence is the ability to explore, for the first time in robotic colonoscopy, how different levels of computer assistance may improve the procedure and reduce user workload.

The main scientific questions we investigate in this work are: (i) How can intelligent control strategies overcome the inherent complexities of controlling magnetic intra-corporeal endoscopes? (ii) What level of autonomy is required to enable a non-expert operator to navigate a magnetic endoscope in an unstructured environment such as the colon, while maintaining procedure duration comparable to a FE? (iii) Can effective, intelligent control strategies reduce the physical and mental burden of the operator?

A successful outcome to these questions, combined with a technology such as the magnetic flexible endoscope (MFE) (Fig. 1), designed for painless colonoscopy, could provide a major improvement, and welcomed disruption in early detection and treatment of colorectal diseases. The MFE has been developed by our group over the last 12 years ^17,24,33, in this paper we present for the first time a comprehensive approach to autonomous navigation of the endoscope.

Besides being crucial for colonoscopy, this work is applicable to several other endoscopic applications where the environment is unstructured and poses significant challenges for effective navigation. This would also reduce pure dependency on manual expertise. With robotic assistance in navigation, training resources can be directed towards the cognitive aspects of endoscopy such as recognition of pathology, differential diagnosis, and creation of treatment plans.

To investigate these scientific questions, we have developed a control methodology that allows simplified user inputs and image-based, autonomous navigation, capable of computing motion based on a real-time visual analysis of the environment. This methodology was comparatively tested in benchtop and in-vivo (porcine model) settings with non-expert users. In doing so, we provide the following contributions:

1. The first demonstration of intelligent and autonomous control enabling non-expert users to successfully perform magnetic colonoscopy by travelling a significant distance in-vivo, and with a duration comparable to standard FE.

2. A framework to define the increasing levels of autonomy in medical robotics applied to robotic flexible endoscopy.

3. An analysis into the autonomous features required to overcome the complexities of magnetic manipulation in unstructured tubular cavities.

4. The development of intelligent and autonomous control strategies for magnetic endoscopy which have enabled a reduction in exertion for the user.

A schematic overview of our approach to control is provided in Fig. 2 and described in Supplementary Video 1. The navigation system is composed of several elementary blocks, organised in three main layers. Each layer provides a set of features characterised by increasing autonomy, relying on functionalities offered by the underlying layers.

Fig. 2 -.

Schematic overview of the control layers associated to autonomy levels. In the first layer, where no autonomy is available, the user manually controls the robot end effector in 5 degrees of freedom (DOF) in an attempt to manipulate the MFE. In the second layer, the user controls the endoscope and the system carries out suitable motions of the robot by taking into consideration localisation information and magnetic field interaction. In the third layer the user has discrete control over the endoscope, the lumen is detected and followed autonomously. In the standard definition of the autonomy levels, these correspond to Level-0, 1 and 3, respectively. Level-2 is defined as “task autonomy” and, in robotic endoscopy, describes autonomous execution of tasks such as retroflexion or guided biopsy.

The first and most simple layer is defined as “direct robot operation”. In this layer, the user is manipulating the robot (that holds an external permanent magnet (EPM)) to influence MFE motion. This layer exhibits functionality offered by the mechanical platform and an elementary level of manual control whereby the user must themselves control variations in the interacting magnetic fields. The functionality offered can be associated to Level-0, as the manipulator is a mere executor of the movements imparted by the human operator (with the addition of some safety constraints). This layer serves to act as a comparative baseline for the subsequent developments in control and autonomy defined in this work.

In the second layer, user inputs are directly focused on navigating the endoscope through the colon, while the system carries the burden of generating a suitable magnetic control action to accomplish the desired endoscope motion. In this layer, the presence of the robot is inconsequential to the user whose inputs directly control the endoscopes tip, via the video feed. With this, the user intuitively instructs how they wish the endoscope camera to move inside the colon. Using real-time positional information of the MFE (accuracy 5mm (±1mm) and 6° (±0.8°), 100Hz, Supplementary Fig. 1), provided by previous work ³⁴ on a magnetic localisation algorithm and a Hall effect/ IMU sensor circuit in the tip of the MFE, this level of control computes the best motion strategy to perform the required action and subsequently operates the robot to adapt the magnetic field accordingly. We define this layer as “intelligent endoscope teleoperation,” which can be associated to Level-1 or robotic assistance, following the classification provided by ^30,31. The human operator maintains continuous control over navigation, while the robot assists with magnetic manipulation.

In the third layer, the system governs motion of the MFE based on a real-time analysis of the endoscopic video feed, combined with the knowledge of the endoscopes pose from the localisation system. The direction of motion is computed by an image analysis algorithm that detects the centre of the lumen. The endoscope is then autonomously steered and advanced through the colon using the navigation control developed in the underlying layer. The local real-time knowledge of the anatomy acquired through the image-analysis is crucial for enabling this level of autonomy. If desired, the user can override the system’s choice by clicking on the desired location in the image. To highlight the autonomous features, we define this layer as “semi-autonomous navigation”. This layer can be associated to Level-3 or conditional autonomy, where the system generates task strategies and relies on the operator to approve or override the choice. In our system, the navigation task is performed autonomously but under supervision of the operator who can perform discrete control actions and override the autonomous control to select a different output of orientation.

In the discussion of levels of autonomy, Level-2 has been omitted. This level, defined as “Task autonomy”, describes a system that carries out semi-autonomous motion but is dependent on a human-in-the-loop to indicate the end target and waypoints of that motion. Examples of Level-2 in the context of endoscopy are motion along pre-defined trajectories ²⁸, autonomous retroflexion ³³ and stabilisation of the endoscope’s tip during interventional tasks (e.g. biopsy). Although task autonomy is promising and could contribute to the goal of simplifying the overall procedure, this work is focused on navigation inside the colon. As the shape of the colon is not fixed and changes frequently, waypoints and end targets of predefined-trajectories under Level-2 control would need to be constantly updated by the user. From the technical viewpoint, the features required to perform this task are the same as Level-1 as the user remains in continuous control, hence the discussion of level-2 has been omitted in this work.

Benchtop and in-vivo results

Experimental validation

We conducted a set of experiments to evaluate the developed control strategies, with their respective performances being scored in terms of navigation and user workload. We first designed an experiment to assess the effectiveness of the endoscope orientation controller, shown in Extended data Fig. 1 and Supplementary Video 2, and then conducted a benchtop study where users untrained in colonoscopy were asked to make multiple attempts at navigating the MFE in a latex phantom, using the various intelligent control strategies. Finally, we conducted an in-vivo study on two porcine models, with the goal of further comparing the performance and ease-of-use of different control methods in a living being.

Benchtop experimental results

To compare the different strategies, we performed a comparative trial on a benchtop platform (Supplementary Video 3). A latex simulator was configured into a standard colon shape used by gastrointestinal practitioners during training (Fig3–c) and then covered from view (Fig3–a). 10 novice participants (no endoscopy experience) were instructed to navigate the MFE from the rectum to the cecum as fast as possible, 5 times for each control strategy (15 total per user). Each task was repeated five times before proceeding to the following task, all the participants performed the tasks in the same order. Each participant completed all the tasks on the same day, but different participants were admitted to the lab on different days. The end of the navigation task (the cecum) was placed and clamped at 9 haustral folds from the end of the colon as per manufacturer instruction. This resulted in a rectum-to-cecum distance of 100 cm. A test was labelled as complete upon navigating from the rectum to the cecum in 20 minutes or less. Users were given a lead-in time of 20-minutes for each of the 3 control strategies to become familiar with the controls before initiation of the trial. The choice of a 20-minutes time limit is based on ³⁵, which reports that the average cecal intubation time for a trainee in a standard colonoscopy is 14.1 minutes, as well as the time limit chosen in a colonoscopy simulator study ³⁶. Detailed data on each task are available in Supplementary Dataset 1.

Fig. 3 -.

Benchtop experimental setup and results. (a) The experimental setup. The user is manipulating the joystick with the right hand and feeding the tether with the left hand. The phantom is covered, and the endoscopic video feed is visible in the user interface. (b) Successful completion times for each control strategy: direct robot operation: n=29, intelligent endoscope teleoperation.: n=48, semi-autonomous navigation: n=50. Red bars indicate median, edges are 25th and 75th percentiles, whiskers indicate range, and red crosses denote outliers. P-values computed using the Kruskal Wallis test. (c) Detail of the latex phantom representing a human colon (M40, Kyoto Kagaku Co., Ltd). Anatomical features are reproduced by template fixations provided by the manufacturer, the standard configuration has been chosen. (d) Example completed trajectory of the MFE using direct robot operation. (e) Example completed trajectory of the MFE using intelligent endoscope teleoperation. (f) Example of endoscope path during a semi-autonomous execution. The user override is represented in yellow and the autonomous motion in blue.

After every attempt, users were asked to complete a NASA task load index (TLX) questionnaire ³⁷. The NASA TLX is a widely used workload assessment instrument, aimed at scoring human perceived workload on six subjective subscales: mental demand (how mentally demanding was the task?), physical demand (how physically demanding was the task?), temporal demand (how hurried or rushed was the pace of the task?), performance (how successful were you at completing the task?), effort (how hard did you have to work to accomplish your level of performance?), and frustration (how frustrated, insecure, discouraged, irritated, stressed or annoyed were you?). All subscales range from 0 (very low) to 100 (very high) with an exception for performance, which ranges from 0 (perfect) to 100 (failure).

Overall completion rates (percentage successfully navigated from the rectum to the cecum in 20 minutes or less) for direct robot operation, intelligent teleoperation and semi-autonomous navigation were 58% (29/50), 96% (48/50) and 100% (50/50), respectively. As shown by Fig. 3–b, out of all successful attempts, direct robot operation presented the slowest average completion time of 11 min. 8 s. ± 3 min. 59 s and had the MFE commonly produce convoluted trajectories (example shown in Fig. 3–d). This was often because the user would position the MFE in an undesired manner, get stuck, then would have to pull back the tip via the tether, readjust the position of the MFE and try again. Intelligent teleoperation and semi-autonomous navigation were significantly faster and comparable to each other with average completion times of 4 min. 6 s. ± 2 min. 8 s, and 4 min. 14 s. ± 1 min. 31 s, respectively. These results outperform colonoscopies carried out on the same phantom by novice users, which, in another study ³⁶, lasted an average of 17min. ± 8 min. More details are provided in the discussion. The completed trajectories of the MFE using endoscope teleoperation (example shown in Fig. 3–e) and autonomous navigation (Fig. 3–f) were much more direct and smoother compared to robot operation, as the MFE was able to be more easily positioned, and reaching the cecum did not require the user to inefficiently withdraw and retry difficult sections. P-values (Fig. 3–b) indicate statistical significance when comparing completion times.

Regarding ease-of-use (Table 1), users found direct robot operation to be significantly more demanding in all NASA task load categories. High levels of effort and frustration resulted from the endoscope losing magnetic coupling with the EPM. In different relative poses of the two magnets, user commands produced different changes in magnetic forces and torques, appearing to the users as a random effect on the movement of the MFE. Main points of failure using direct robot operation were the hepatic and splenic flexures, with the lack of an intuitive connection between command and motion making these tight turns particularly difficult to navigate. Out of the three control strategies, semi-autonomous navigation presented the lowest user workload scores in all categories. The performance of the autonomous system let the users take on more of a monitoring role which in turn, made the task much less demanding.

Table 1-.

NASA Task Load index, mean user workload ratings from benchtop trial results. High, orange shaded values indicate poor user experience and low, green shaded values indicate good user experience.

	Benchtop unweighted mean workload ratings (lower score better)
Subscale	Direct robot operation	Intelligent endoscope teleoperation	Semi-autonomous navigation
Mental Demand	79	29	18
Physical Demand	57	23	15
Temporal Demand	68	34	22
Performance	54	18	15
Effort	81	27	18
Frustration	74	24	17
Mean workload	71	25	17

In the 50 successful semi-autonomous repetitions, the MFE was autonomously operated for on average 91% of the total time required to navigate from the rectum to the cecum, with 12 completed procedures being performed fully autonomously without any manual override necessary. Out of the procedures requiring manual intervention, users most commonly needed to give an input via the joystick in the rectum due to the multiple sharp turns found in quick succession that placed the lumen behind and out of view of the camera. An example of semi-autonomous execution is shown in Fig3–f.

In-vivo experimental results

After the benchtop study highlighted the improved ease-of-use and performance associated with increased MFE autonomy, we performed an in-vivo study on a porcine model (two female Yorkshire-Landrace porcine, 33kg and 35kg), (Supplementary Video 3). The primary objectives of the experiments were: (i) highlighting shortcomings of simple robot teleoperation in magnetic manipulation; (ii) comparing the benefits provided to non-trained users by the increasing intelligent control strategies in a variable and tortuous environment such as the porcine colon; (iii) and, given that a porcine colon is comparatively more difficult to navigate than a human colon, provide a strong indicator for the potential of the system in the less demanding human anatomy. The increased tortuosity of the porcine colon results from its highly spiralled structure (Supplementary Fig. 3). This continuously spiralling trajectory arguably creates more points of tissue-MFE contact and increased friction that requires a higher magnetic force to overcome. Furthermore, the colon loops present a navigation challenge that requires continuous rotation of the IPM and EPM, often reaching the limits of magnetic actuation or the robotic manipulators joints. As a result, repositioning of the animal (e.g. rotation) or reconfiguring of the robotic manipulator joints may be necessary and thereby extend the overall procedure time.

The experimental scheme was completed by two operators with no prior endoscopic experience. The experiments were designed to compare the use of a conventional flexible endoscope (FE) (Olympus PCF-160AL) and the various levels of control strategies developed for the MFE. At the beginning of experiment, each user was given a 10-minute lead-in period and instructed to use a standard FE to travel as far possible inside the porcine colon. After 10 minutes, the end point - the furthest distance reached in the colon - was tattooed to serve as a comparable distance marker for subsequent attempts (Fig. 4–b). Travelled distance was measured using the incremental markings on the endoscope insertion tube. At every iteration, if the end-point distance reached surpassed the marker, the new furthest point reached was measured, tattooed, and updated to be the new target.

Fig. 4 –

In-vivo results. (a) Experimental setup of the in-vivo trial. The robotic arm is operating the MFE, the endoscopic video feed is displayed on the user interface. (b) Detail of tattoo marker used to identify the maximum distance reached with a conventional endoscope. (c) is the path travelled by the MFE using autonomously assisted control (Level-3) reaching 85cm. Two anatomical loops can be observed, with the MFE being able to successfully overcome the difficult turns. (d) Is an example video frame of the system autonomously detecting and steering the MFE towards the porcine lumen (green dot).

Subsequently, each user attempted to navigate the porcine colon with the MFE using the different control strategies (Fig. 4–a). Trials were divided into sets, in the order of one direct robot operation, one intelligent teleoperation, and one semi-autonomous navigation. The number of sets performed was four on the first animal and three on the second animal, as available time was affected by limiting factors such as the risk of prolonged anaesthesia. During every repetition, the time required to reach the tattooed marker and the position of the EPM and MFE were recorded. The user completed a NASA task load index after each attempt to compare ease-of-use between the different approaches. Detailed data are available in Supplementary Dataset 2.

Completion times and completion rates for the two users are reported in Supplementary Table-1. On the first animal, the tattooed distance reached using the standard FE was 45cm. Significant tortuosity in the colon prevented any further distance to be achieved. The user was then able to perform 4 attempts using each MFE control strategy. As time allowed 4 sets to be completed using the MFE, the fastest 4 attempts using the FE were used in this comparison. Average completion times were 1 min 39 s for the standard FE, 9 min 4 s for direct robot operation, 2 min 20 s for intelligent endoscope teleoperation, and 3 min 9 s for semi-autonomous navigation. The same approach was followed on the second animal. During the initial phase with the conventional FE, the user reached a notable distance of 85 cm which became the tattooed-distance-target for following attempts. A faecal blockage prevented any further distance to be achieved. Although the difference between distances travelled in the first and the second animal is significant, this is quite common in experiments involving animals, where the colon is tortuous, prone to gas retention (that can cause the bowel to press into and collapse neighbouring lumens) and difficult to clean before the procedure (e.g. humans undergo a rigorous bowel preparation that requires ingestion of fluids in a closely followed protocol – this cannot be performed on animals).

Time allowed 3 sets of attempts to be completed using the MFE, with the fastest 3 standard FE attempts being used in comparison. Average completion times for the second user were 3 min 29 s for the standard FE, 8 min 36 s for intelligent endoscope teleoperation, and 9 min 39 s for semi-autonomous navigation. The lowest level, direct robot operation, was unable to reach the marker.

The trajectory of the MFE during one of the autonomously assisted control trials (Level-3, User 2, 85cm target) is shown in Fig. 4–c, with the on-board camera image detecting the lumen shown in Fig. 4–d. The trajectory shows the MFE being able to overcome two loops and several tortuous bends. Regarding user workload, shown in Table 2, both users found that using the standard FE and direct robot operation were the most demanding in all NASA workload categories. Direct robot operation was more demanding than a standard FE for most mental workload categories. A FE, while difficult to master, has a physical-cable-link between the control interface and the tip, resulting in a direct and predictable response in tip movement. The absence of intelligent control for direct robot operation and physical link between the interacting magnetic fields meant that the user would have to mentally predict the result of their next input, given the current state of the magnetic system, often resulting in frustration when motions of the MFE did not move as predicted. Intelligent teleoperation and semi-autonomous navigation were significantly less demanding for the novice user. Similarly, to the benchtop experiments, in autonomous mode the user had the ability to override the motion with manual control. During the semi-autonomous repetitions, the MFE was navigated in autonomous mode for on average 87% of the time required to reach the marker for User-1 (45cm total distance), and 78% for User-2 (85cm total distance). This remarkable result has been obtained under the supervision of a veterinary surgeon, who continuously verified the safety of the procedure. Such high rate of autonomy indicates that the semi-autonomous mode, in conjunction with the use of safety measures such as limited minimum inter-magnetic distance, provides satisfactory safety levels.

Table 2 -.

NASA Task Load index, mean workload ratings on porcine models. High, orange shaded values indicate poor user experience and low, green shaded values indicate good user experience.

	In-vivo unweighted mean workload ratings (lower score better)
Subscale	Standard FE	Direct robot operation	Intelligent endoscope teleoperation	Semi-autonomous navigation
Mental Demand	66	72	14	11
Physical Demand	60	40	14	9
Temporal Demand	44	59	17	28
Performance	46	59	20	32
Effort	61	61	19	10
Frustration	63	81	18	28
Mean workload	60	60	18	19

Discussion

In this work, we enable intelligent and autonomous navigation of magnetic endoscopes in complex environments such as the colon and define how increasing levels of autonomy can be applied to robotic endoscopy. We discuss the features required to enable each autonomy level and synthesise an integrated control scheme in which software layers with higher intelligence capitalise on features offered by the underlying layers. The effectiveness of our techniques was tested on benchtop and in-vivo, in a porcine model. With respect to the scientific questions outlined in the introduction:

1. We show that the inherent complexity of navigating magnetic endoscopes with a single external permanent magnet can be overcome by the developed intelligent control strategies. These were able to mask the unintuitive nature of interacting magnetic fields and field gradients. In particular, the simultaneous use of localisation and an advanced closed-loop control strategy is crucial to achieve satisfactory procedure times. The availability of a reliable localisation mechanism can be substituted by estimation-based techniques ²⁷, or visual feedback ³⁸ in applications where the environment is more structured and constrained, such as navigation in lungs or cardiovascular apparatus; however, the inherent complexities of colonoscopy require internal (such as in this work), or external localisation ³⁹. Moreover, we show how an effective control strategy can overcome the limitations imposed by the actuation of a single permanent magnet. Systems based on different magnetic field sources such as coils ⁴⁰ or rotating permanent magnets ⁴¹, may provide similar capabilities with comparable results, although continuous rotation of the endoscope (and consequently, of the camera) may hinder the integration of vision, localisation and control.

2. The minimum level of autonomy required for a non-expert to effectively navigate a complex environment such as the colon is Level-1. In ³⁶, the time required to reach the cecum with a conventional endoscope, on the same phantom used for this study, has been evaluated on 32 novice users and 21 experienced endoscopists. An average of 17min. ± 8 min. is required to completely untrained operators, decreasing to 11min. ± 7min. after 11 hours of training. Additionally, the experienced colonoscopists performed the same test, resulting in an average procedure duration of 7min ± 5 min. The results of this study show that endoscope teleoperation and semi-autonomous navigation outperform conventional colonoscope for novice and newly trained operators, reducing the time to reach the cecum to a value comparable to that of experienced clinicians.

3. Autonomous navigation introduces a significant step towards the autonomous execution of colonoscopy, thus providing substantial benefits in terms of reducing mental and physical workload. Moreover, the degree of autonomy enabled by this feature, similar to other tasks like surgical suturing, has the potential to revolutionise clinical workflow, requiring minimal and discontinuous intervention from the operator. In the future, the robot speed will be increased to achieve faster motion and further reduce procedure duration.

Other examples of the manipulation of magnetic endoscopes are available in literature ⁴²; however, they either lack localisation ²⁷, an endoscope tether (required for interventional capabilities) ⁴³, or an intelligent control system ⁴⁴, limiting the translation to clinical use. This paper shows the first example of a tethered magnetic endoscope successfully navigating the colon of a porcine model by means of a blended use of magnetic localisation, closed-loop robotic control, and elaboration of the endoscope camera image.

The results shown in this paper build upon the work carried out in 12 years of development, in which the foundations of the MFE platform have been developed. In previous works, our group evaluated different control strategies, aimed at tackling particular aspects of navigation such as pre-defined trajectories on benchtop ³⁴, levitation ⁴⁵, or overcoming obstacles ⁴⁶, but this is the first example of full control of the navigation process, successfully piloting the MFE in a porcine model. Additionally, we demonstrate autonomous navigation of magnetically manipulated endoscopes, in-vivo, for the first time. By adopting a fusion of magnetic and visual feedback, we have developed a system that can make endoscopic inspection of the bowel autonomous and more user friendly when compared to using conventional endoscopes. Our aim is to reduce the complexity of endoscopic procedures by automating the manual aspects of endoscope manipulation, thus reducing the burden on the operator, and enabling more focus for the clinical aspects of the procedure. This work may facilitate the adoption of colonoscopy by requiring a reduced skillset for the navigation of magnetic endoscope devices, thus allowing previously required training resources to be better utilised on the diagnosis and treatment of patients. Considering the unmet demand for colonoscopy and the expected raise in preventive screening campaigns in the next decade, the results of this work may substantially contribute to saving human lives. This work is also crucial as a scientific foundation for transitioning to clinical trials, where other crucial hypotheses, such as acceptability and level of pain associated to the procedure, can be tested.

With 19-million colonoscopies performed every year in the US and EU, and a constantly increasing demand, this technology has a disruptive potential to revolutionise the current practice. Potential impacts of this work also concern the control of magnetic endoscopes for other applications such as gastroscopy ⁴⁷ and bronchoscopy ⁴⁸. The framework defined in this work could be adopted with different endoscope designs and lays the groundwork for the development of additional levels of autonomy.

The methods applied in this work strongly rely on endoscope localisation, which operates under the assumption of no distortion in the magnetic field produced by the EPM. This can be a challenging restriction in clinical environments, although the sensitive workspace is limited to the patient’s abdominal area and the availability of MRI-compatible devices may significantly mitigate this hurdle. Moreover, the effective workspace is limited by the strength of the magnets. This may have a negative effect on patients with high body mass index, for which the minimum safe EPM-to-endoscope distance is higher. The magnets strength also affects the manoeuvrability of the endoscope; stronger magnets would require smaller EPM motion, thus improving the endoscopes reactivity and disturbance rejection. The most viable solution would be to revise the design of the endoscope, currently limited using standalone cameras that require dedicated cabling and reduce the space available for the magnet. To further validate the methods described in this paper, in the future we will consider an extensive trial on benchtop phantoms with complex configurations such as alpha loops, that may be significantly more complex to navigate.

For the benchtop study, the purpose was to validate our control methods and test the hypothesis that a non-expert can navigate a colon with low mental and physical exertion by using the MFE and increasing robotic assistance. Therefore, a cohort of complete novices was deemed most appropriate. For future work, a study involving users of various skill levels would be interesting and would indicate the learning curve of this technology - something we have explored previously ⁴⁹ and intend to explore in future works with the next generation of the hardware platform.

The findings of this work also open the way toward the development of other autonomous tasks in endoscopy. Further benefits could be found through the development of autonomous control strategies to aid in therapeutic tasks such as biopsy and polypectomy. The current diagnostic practice relies on the operators experience and training in analysing the endoscopic image. We hypothesise that in the future, artificial intelligence and autonomous navigation may be coupled to ultimately improve patient care (diagnosis and therapy) and that, with the development of dedicated control strategies (that completely integrates a vision module with artificial intelligence) higher levels of autonomy (i.e. Level-4) will be possible. Furthermore, this work is particularly timely, with the global pandemic COVID-19 severely restricting endoscopy practice under instruction by governing bodies ⁵⁰. Standard endoscopy requires multiple staff and proximity of staff and patient. This is problematic since FEs generate significant aerosols that can readily spread infection between the multiple, grouped personnel. The MFE and the control developed here demonstrate the potential for robotic endoscopy procedures to be performed with fewer staff and, with minor adjustments to the MFE system (such as a simple tether feeder), the option to reduce contact between staff and patient. This may facilitate procedures with considerably lower risk of viral infection or cross-contamination and endoscopy practise to be unhindered by any future pandemics.

Intelligent Control and Autonomous Navigation

The experiments described in this work are aimed at evaluating the performance of different levels of autonomy required to successfully navigate the colon with the MFE platform. In this section, we initially describe the system and the features provided by each layer. We then discuss the validation process of the autonomous lumen detection algorithm.

System overview

The main components of the system are a robotic arm with an EPM mounted on the tip and the magnetic endoscope. The endoscope, shown in Fig. 1, is composed of a 3D printed shell, a localisation circuit, an endoscopic camera and an intracorporeal permanent magnet (IPM) that is immersed in the field produced by the EPM.

The interaction between EPM and IPM, shown in Supplementary Fig. 4, is provided by magnetic coupling. Forces (f_l) and torques (τ_l) exerted on the IPM, computed with respect to the world reference frame O_w are described by the magnetic dipole model:

$${\mathit{f}}_{\mathit{l}}=\mathit{\nabla }({\mathit{m}}_{\mathit{I}}\cdot {\mathit{B}}_{\mathit{E}})\hspace{1em}\hspace{0.17em}\hspace{0.17em}{\mathit{\tau }}_{\mathit{l}}={\mathit{m}}_{\mathit{I}}\times {\mathit{B}}_{\mathit{E}}$$ (1)

where m_E and m_I are the magnetic moments of the EPM and IPM expressed with respect to the global coordinate frame O_w, B_E is the vector representing the magnetic field generated by the EPM in the IPM position and m_E, m_I are the vectors describing the cartesian positions of EPM and IPM. A 5 degrees of freedom (DOF) manipulator could theoretically move the EPM in any pose, generating all feasible combinations of forces and torques accordingly. In practice, a 7-DOF manipulator, such as the KUKA LBR Med R820 used in this work, provides enhanced dexterity and minimises the risk of reaching joint limits. The system was implemented via ROS (Robot Operating System), with the robotic manipulator controlled in joint space. For all tests, user inputs were given via a 6DOF joystick (3D Space Pilot, 3D connexion Inc. USA). This high DOF joystick was necessary as control Level-0 required the user to control motions of the EPM in multiple DOF (pitch, yaw, and up, down, forwards and backwards translation). This joystick was then also used for endoscope teleoperation and autonomous navigation to remain ergonomically consistent across all tests. After completing this study it was noted that the more successful intelligent control levels required simplified user inputs with reduced DOF. As such, a more appropriate joystick can be used for future development, e.g. a PlayStation 3 navigation controller (Fig. 2) (Sony Corporation, Japan).

Direct robot operation

This layer provides the core functionality of teleoperating the EPM to indirectly produce effects on the MFE. The end effector of the robotic arm is operated via a joystick interface connected to a low-level controller. Operator commands provided through the joystick interface constitute the users request to move the robot end effector. Inputs, that include angular displacements $\delta {\widehat{\mathit{m}}}_{\mathit{E}}\in {ℝ}^3$ (where ${\widehat{\mathit{m}}}_{\mathit{E}}$ is the unit vector associated to m_E) and linear displacements $\delta {\mathit{p}}_{\mathit{E}}\in {ℝ}^3$, are gathered and transformed in joint angles variations $\delta \mathit{q}\in {ℝ}^7$ by means of the differential relation:

$$\delta \mathit{q}={\mathit{J}}^{†}{\mathit{W}}_{\mathit{a}}\mathit{s}\left[\begin{array}{c}\delta {\mathit{p}}_{\mathit{E}}\\ \delta {\widehat{\mathit{m}}}_{\mathit{E}}\end{array}\right]+\mathit{\gamma }(\mathit{I}-{\mathit{J}}^{†}\mathit{J}\text{)}\delta {\mathit{q}}_0$$ (2)

Where ${\mathit{J}}^{†}\in {ℝ}^{7\times 6}$ is the pseudoinverse of the robot’s Jacobian and ${\mathit{W}}_{\mathit{a}}\in {ℝ}^{6\times 6}$ is a suitable weighting matrix. In order to avoid joint limits, a modified version of the saturation in the null space algorithm ⁵¹ has been adopted. This involves scaling the task by means of the scaling factor $\mathit{s}\in {ℝ}^6$ (Supplementary Alg. 2) and injecting a suitable action $\delta {\mathit{q}}_{\text{0}}\in {ℝ}^7$ in the null space of the robot’s Jacobian, multiplied by a scaling factor γ, a function of robot velocity gain, manually tuned. In this work, 𝛿q₀ was chosen in such a way to minimise the distance of the robot’s joint angles from the central position (Supplementary Alg. 3). For safety reasons, the absolute rotation and vertical height of the end effector has been limited and are adjustable through the user interface. The robot joint angles q(t) are computed at every time step of the control algorithm by time integration (forward Euler method).

The position and orientation of the endoscope tip are shown to the user in conjunction with the robot pose, by means of a virtual 3D environment. Simultaneously, the video feed of the camera embedded in the capsule is presented to the user. The robot’s end effector is teleoperated under the assumption that the endoscope would follow the motion of the EPM, due to magnetic coupling. This approach, although very simple from a computational and architectural viewpoint presents several drawbacks: (i) The magnetic coupling is nonlinear, hence similar variations of the EPM pose might not result in the same effect on the endoscope (ii) the orientation of the camera is not aligned with the perspective of the user (gravity could be in any direction in the camera frame), increasing the mental effort required to the operator. (iii) the control action applied to the endoscope is suboptimal, as the human presence in the loop can significantly reduce performance.

Owing to the axial symmetry of the permanent magnets, rotations about their longitudinal axis have no effect on the magnetic field. For this reason, the orientation in the roll axis of the endoscope is not controllable and therefore the camera alignment on that axis is ungovernable. The operator is required to mentally compute the rotation of the image with respect to the horizon and consider it when operating the robot. Endoscopists usually tackle this complexity by a trial-and-error process, but experience is required, and the operator can undergo considerable stress. Moreover, the effect generated on the endoscope by the EPM motion can significantly change in different relative poses due to the nonlinearity of the magnetic coupling, thus adding complexity to the navigation task.

During preliminary tests, several users reported severe difficulties in separately requesting end-effector rotation and translation. This might be due to the complexity of pushing the controller joystick without inducing any rotation and vice-versa. To ease the teleoperation, EPM motion has been restricted by preventing movements along the Y axis and rotations around the roll axis, as the first corresponds to lateral motion of the endoscope, while the second is a rotation around the magnetisation axis. Moreover, two operating modalities have been defined with the aim of separating motion and orientation control of the endoscope. In the first, pitch rotation of the EPM is prevented. In the second, linear motions are nullified in favour of rotation control. This feature is obtained by assigning suitable weights to the matrix W_a, shown in (Supplementary Eq. 1).

Intelligent endoscope teleoperation

The main feature of this layer is to mask the complexity of teleoperating the robot for a desired motion of the endoscope, as inducing an effect on the IPM by commanding motions of the EPM is unintuitive. This subsystem provides direct control of the endoscope’s tip to the user, thus overcoming the limitations of the lower layer in terms of ease of use.

Intelligent teleoperation of the MFE is enabled by a real-time localisation system thoroughly described in ³⁴. This is based on a particle filter estimation of the tip pose with respect to a pre-computed map of the magnetic field, generated by the EPM ⁵². The system is capable of estimating orientation and position of the MFE tip with an accuracy of 5mm (±1mm) and 6° (±0.8°) in static and dynamic conditions. Taking advantage of the sensing provided by the localisation, a closed-loop control scheme aimed at navigating the endoscope was explored.

To develop a control system based on a linear model, the magnetic dipole model for forces and torques described in equation (1) can be expressed with respect to the position and orientation of the magnets and locally linearized, resulting in the following differential relation:

$$\left[\begin{array}{c}\delta {\mathit{f}}_{\mathit{l}}\\ \delta {\mathit{\tau }}_{\mathit{l}}\end{array}\right]=\left[\begin{array}{l}\frac{\partial {\mathit{F}}_{\mathit{m}}}{\partial {\mathit{p}}_{\mathit{E}}}\hspace{1em}\frac{\partial {\mathit{F}}_{\mathit{m}}}{\partial {\mathit{p}}_{\mathit{I}}}\hspace{1em}\frac{\partial {\mathit{F}}_{\mathit{m}}}{\partial {\widehat{\mathit{m}}}_{\mathit{E}}}\hspace{1em}\frac{\partial {\mathit{F}}_{\mathit{m}}}{\partial {\widehat{\mathit{m}}}_{\mathit{I}}}\\ \frac{\partial {\mathit{\tau }}_{\mathit{m}}}{\partial {\mathit{p}}_{\mathit{E}}}\hspace{1em}\frac{\partial {\mathit{\tau }}_{\mathit{m}}}{\partial {\mathit{p}}_{\mathit{I}}}\hspace{1em}\frac{\partial {\mathit{\tau }}_{\mathit{m}}}{\partial {\widehat{\mathit{m}}}_{\mathit{E}}}\hspace{1em}\frac{\partial {\mathit{\tau }}_{\mathit{m}}}{\partial {\widehat{\mathit{m}}}_{\mathit{I}}}\end{array}\right]\hspace{0.17em}\left[\begin{array}{c}\delta {\mathit{p}}_{\mathit{E}}\\ \delta {\mathit{p}}_{\mathit{I}}\\ \delta {\widehat{\mathit{m}}}_{\mathit{E}}\\ \delta {\widehat{\mathit{m}}}_{\mathit{I}}\end{array}\right]={\mathit{J}}_{\mathit{F}\mathit{A}}({\mathit{p}}_{\mathit{E}},{\mathit{p}}_{\mathit{I}},{\widehat{\mathit{m}}}_{\mathit{E}},{\widehat{\mathit{m}}}_{\mathit{I}})\left[\begin{array}{c}\delta {\mathit{p}}_{\mathit{E}}\\ \delta {\mathit{p}}_{\mathit{I}}\\ \delta {\widehat{\mathit{m}}}_{\mathit{E}}\\ \delta {\widehat{\mathit{m}}}_{\mathit{I}}\end{array}\right]$$ (3)

where ${\mathit{F}}_{\mathit{m}}\in {ℝ}^3$ and ${\mathit{\tau }}_{\mathit{m}}\in {ℝ}^3$ are the nonlinear expressions of magnetic forces and torques (the complete expression can be found in Supplementary Eq. 2) ⁵³, p_E, ${\mathit{p}}_{\mathit{I}}\in {ℝ}^3$ are the positions of the EPM and IPM, ${\widehat{\mathit{m}}}_{\mathit{E}}$ and ${\widehat{\mathit{m}}}_{\mathit{I}}\in {ℝ}^3$ are the unit vectors representing the orientation of EPM and IPM in the world reference frame, and 𝛿f_lin, $\delta {\mathit{\tau }}_{\mathit{l}\mathit{i}\mathit{m}}\in {ℝ}^3$ represent the variation of F_m and ${\mathit{\tau }}_{\mathit{m}}\in {ℝ}^3$ with respect to a local configuration change. Assuming a constant pose of the endoscope, equation (3) can be simplified to:

$$\left[\begin{array}{c}\delta {\mathit{f}}_{\mathit{l}}\\ \delta {\mathit{\tau }}_{\mathit{l}}\end{array}\right]={\mathit{J}}_{\mathit{F}}({\mathit{p}}_{\mathit{E}},{\mathit{p}}_{\mathit{I}},{\widehat{\mathit{m}}}_{\mathit{E}},{\widehat{\mathit{m}}}_{\mathit{I}})\left[\begin{array}{l}\delta {\mathit{p}}_{\mathit{E}}\\ \delta {\widehat{\mathit{m}}}_{\mathit{E}}\end{array}\right]$$ (4)

Although the magnetic dipole model is globally nonlinear, the local linearization and constant endoscope pose are reasonable assumptions as the motion of the endoscope is slow (~0.01m/s) with respect to the frequency of the control loop (100Hz). The Jacobian J_F is computed at every time step; thus, the simplified linear model is locally valid and provides satisfactory performances. The orientation control is carried out by a closed loop system, described by the following expression:

$$\mathit{\delta }{\mathit{\tau }}_{\mathit{l}}={\mathit{R}}_{\mathit{G}}^{\mathit{I}}\mathit{p}\mathit{d}\hspace{0.17em}\text{(}\overline{\delta {\mathit{\vartheta }}_{\mathit{a},\mathit{I}}},\overline{{\mathit{\omega }}_{\mathit{a},\mathit{I}}})$$ (5)

The pd() function computes a proportional-derivative control action with respect to the user input $\overline{\delta {\mathit{\vartheta }}_{\mathit{a},\mathit{I}}}\in {ℝ}^3$ and the current endoscope angular velocities $\overline{{\mathit{\omega }}_{\mathit{a},\mathit{I}}}\in {ℝ}^3$ expressed in local coordinates (the overbar indicates local reference frame). ${\mathit{R}}_{\mathit{G}}^{\mathit{I}}\in {ℝ}^{3\text{x3}}$ is the rotation matrix describing the endoscope orientation w.r.t. the global reference frame, shown in Supplementary Fig. 5.

The control of the linear motion of the endoscope is not based on the linearized magnetic model of equation (5). When a linear motion is required, the orientation of the endoscope ${\widehat{\mathit{m}}}_{\mathit{I}}$ is projected on the horizontal plane by the proj_xy() operator and multiplied by the motion command $\overline{\delta {\mathit{X}}_{\mathit{a},\mathit{I}}}\in {ℝ}^3$. A damping term ($\alpha \left[1-\frac{F_{m_Z}}{F_{m_Z,max}}\right]$) is introduced to maintain the EPM in the proximity of the endoscope: F_mz is the force exerted by the EPM along the z (vertical) direction, F_mZ,max is the maximum value of the same force and 𝛼 is a weighting constant. Finally, W_FF ∈ {0,1} is an activation term, thus enabling the feedforward term when the motion is commanded. The overall control function is shown in equation (6). The computation of the pseudoinverse of the Jacobian ${\mathbf{J}}_{\mathbf{\text{F}}}^{†}$ is carried out by means of weighted/damped least squares the algorithm is shown in Supplementary Alg. 4.

$$\left[\begin{array}{l}\delta {\mathit{p}}_{\mathit{E}}\\ \delta {\widehat{\mathit{m}}}_{\mathit{E}}\end{array}\right]={\mathit{J}}_{\mathit{F}}^{†}\left[\begin{array}{c}\mathbf{0}\\ \mathit{\delta }{\mathit{\tau }}_{\mathit{l}}\end{array}\right]+W_{FF}(\overline{\delta X_{a,I}})\mathit{p}\mathit{r}\mathit{o}{\mathit{j}}_{\mathit{x}\mathit{y}}({\widehat{\mathit{m}}}_{\mathit{I}})\hspace{0.17em}(\overline{\delta X_{a,I}}-\alpha \left[1-\frac{F_{m_Z}}{F_{m_Z,max}}\right])$$ (6)

Preliminary trials have shown that a linearized approach to teleoperation of the endoscope might induce drift in the EPM with respect to the optimal pose (i.e. exactly above the endoscope), thus resulting in a reduced controllability of the MFE. In the normal motion state, the robot is controlled by equation (6). If the magnetic coupling is not optimal (condition number of ${\mathit{J}}_{\mathit{F}}\gg 1$), the system enters a “recoupling” state and the magnet is brought back to the optimal condition, equation (7) controls the robot motion. The p_i() functions computes a proportional integral control action to move the EPM directly above the endoscope, maintaining the orientation in the X-Y plane.

$$\mathit{\delta }{\mathit{p}}_{\mathit{E}}=\mathit{p}\mathit{i}\left[\begin{array}{l}{\mathit{p}}_{\mathit{I},\mathit{x}}-{\mathit{p}}_{\mathit{E},\mathit{x}}\\ {\mathit{p}}_{\mathit{I},\mathit{y}}-{\mathit{p}}_{\mathit{E},\mathit{y}}\\ {\mathit{h}}_{\mathit{r}\mathit{e}\mathit{s}\mathit{e}\mathit{t}}-{\mathit{p}}_{\mathit{E},\mathit{x}}\end{array}\right]\hspace{1em}\mathit{\delta }{\widehat{\mathit{m}}}_{\mathit{E}}=\mathit{p}\mathit{i}(\mathit{p}\mathit{r}\mathit{o}{\mathit{j}}_{\mathit{x}\mathit{y}}({\widehat{\mathit{m}}}_{\mathit{I}}-{\widehat{\mathit{m}}}_{\mathit{E}}))$$ (7)

Similarly, it is possible to take advantage of the instants when no motion is commanded on the endoscope to maximise the magnetic manipulability. When the joystick is not generating input to the control system, the control action is switched to a different state, where the translational dynamics is controlled by equation (7), while the rotational dynamics is described by equation (8). The rotational displacements of the EPM are computed as an optimisation problem where the wrenches applied to the endoscope are minimised to prevent any undesired motion of the endoscope.

$$\delta {\widehat{\mathit{m}}}_{\mathit{E}}=min\hspace{1em}{\mathit{J}}_{\mathit{F}\mathit{A}}\left[\begin{array}{c}\delta {\mathit{p}}_{\mathit{E}}\\ 0\\ \delta {\widehat{\mathit{m}}}_{\mathit{E}}\\ 0\end{array}\right]$$ (8)

Autonomous navigation

This layer is aimed at further enhancing autonomy by offering autonomous navigation capabilities for the MFE application. To autonomously navigate through the colon, we leverage a combination of the magnetic manipulation algorithms defined in the previous sections, and image processing to autonomously detect the direction of the colon. With this directional information we:

1. Autonomously steer the MFE camera frame towards the centre of the colon lumen.

2. Autonomously advance the MFE forwards through the colon, once aligned to the lumen.

Owing to the inherent, highly variable mobility of the colon (introduced by patient body movement, breathing, peristalsis and low modulus of tissue), we sought to adopt an approach to autonomous navigation that is devoid of pre-defined trajectories. Our approach to leverages a real-time understanding of the colon’s pathway using image processing. Multiple groups have developed image-processing techniques to infer motion direction in endoscopic images, remarking that future benefits would be found in the application of these techniques to the active control of endoscopes ⁵⁴, such as the work we present here. Our chosen approach to inferring direction in the colon is irrespective of features specific to the colon, such as alternative methods using haustral folds ⁵⁵. The absence of feature-specific function enables the autonomous navigation work we present here to be transferable to other magnetic endoscope devices, designed for navigation in other tubular cavities.

To detect the colon lumen in the endoscope image we build upon the adaptive threshold segmentation algorithm presented by ⁵⁴, with the pseudocode of this algorithm shown in Supplementary Alg. 1. The image is first segmented (Extended data Fig. 2–a/b) to remove all but the darkest and most distinct region with the assumption that this area most likely contains the distal lumen. This segmentation is performed using the red channel of the RGB image, as this channel amplifies the distinction between bright and dark regions in the predominantly red-shaded colon.

The image is then down sampled by 50% to reduce computational complexity and converted to grayscale. The corresponding grey-level histogram of the image contains distinct valley points that can be used to separate pixels in two classes: a non-lumen region class and a lumen-region class. To define an optimal threshold for separating pixels into these two classes, each possible threshold value is measured for its class separability using a discriminant criterion measure. The threshold that returns the maximum value for this measure gives the threshold that most effectively segments the image. However, multiple regions can remain in the image after this segmentation. For this each region is scored on its likelihood to contain the lumen, with the highest scoring region being a function of the largest area and the darkest average pixel intensity ⁵⁴. With this final region, all but the darkest pixels are removed, with the centre mass point of these remaining pixels being the final centre-of-lumen estimate.

To advance the endoscope in the colon, we assume that the camera should be directed towards the lumen before any forward motion is requested. The orientation control builds upon what described in the previous section, the input to the endoscope orientation controller $\overline{\delta {\vartheta }_{\mathit{a},\mathit{I}}}$ is generated as described by equation (9). A proportional controller aligns the centre of the image (𝑥_C, 𝑦_C) with the detected lumen (𝑥_i, 𝑦_i), as shown in Extended data Fig. 2–c.

$$\overline{\delta {\mathit{\vartheta }}_{\mathit{a},\mathit{I}}}=\mathit{\beta }\left[\begin{array}{l}{x}_C-x_l\\ y_C-y_l\end{array}\right]$$ (9)

The velocity input imparting translational motion to the endoscope $\overline{\delta X_{a,I}}$ is directly proportional to the alignment between the endoscopic image and the centre of the lumen, as described by equation (10) and shown in Extended data Fig. 2–d.

$$\overline{\delta X_{a,I}}=1-e^{\kappa \sqrt{{(x_C-x_l)}^2+{(y_C-y_l)}^2}}$$ (10)

With the linear velocity being throttled by the positioning of the lumen, priority is given to steering the endoscope. The endoscope is advanced through the colon only when the lumen is towards the centre of the image, thus preventing the endoscope from being driven into a tissue fold or against bends.

However, if the endoscope is directed against the colon wall and the lumen is not visible, the autonomous system needs to respond and avoid advancing the endoscope towards an incorrectly identified lumen. For this reason, the FAST (Features from Accelerated Segment Test) detection algorithm is used to identify discernible edges within the image. In no-lumen scenarios (Extended data Fig. 3–a), there is a distinct reduction in the number of features in the image when compared to images containing a significant portion of colon lumen (Extended data Fig. 3–b). In particular, the presence of haustral folds and overlapping tissue flaps constitutes a satisfactory set of features.

With a threshold set for the number of features based off experience with the system, the controller will detect when no-lumen can be found and initiate a mitigation routine. This routine involves the system moving the EPM away from the MFE so that, being free from counteracting magnetic torque, the endoscope can naturally align to the lumen of the colon as the user pulls back slightly on the tether. Once the lumen has been relocated, autonomous navigation resumes.

Additional info on in-vivo experiments

The in-vivo trials were performed in the Large Animal Experimental Facility at the University of Leeds under Home Office (UK) License (Procedure Project License: PC71ADE55) in accordance with the Animal (Scientific Procedures) Act 1986 and the NC3Rs guidelines. The reporting has been carried out in accordance with the ARRIVE guidelines.

The female Yorkshire-Landrace pigs were 33kg and 35kg and were placed the in supine position under general, terminal anaesthesia, with a water enema being administered after general sedation to clean the bowel prior to the start of the study. Users performing the trial were trained in animal welfare and husbandry and were also supervised by a trained gastroenterologist and named veterinary surgeon. When navigating the colon during in-vivo experiments, users had access to air insufflation to distend the colon, suction to remove stool and debris, and water irrigation to clean the endoscopes camera. Necropsy was performed and did not reveal any gross trauma or perforation of the colon.

Extended Data

Extended Data Fig. 1 – Validation of the magnetic manipulation algorithm.

The first experiment sort to verify that the closed-loop controller could manipulate the EPM in such a way that the magnetic torque imparted on the MFE would accurately and precisely control the direction of the MFE camera frame. The experiment was carried out on a testing rig consisting of a straight tract of latex colon model (M40, Kyoto Kagaku Co., Ltd) with a LED reference point mounted at one end of the tract. The MFE was then positioned so that its camera could observe the LED (10cm separation distance). A simple image-thresholding algorithm was then used to detect the LED in the MFE image. The robot closed-loop controller, based on a proportional-derivative control approach, thoroughly described in Materials & Methods, autonomously steered the MFE to trace two predefined motions in the image plane, arranged in either a sinusoidal or circular trajectory with the tracked LED point used as a positional reference. Upon the LED aligning to the first pixel point of the trajectory, the target was updated to the next point along the trajectory and repeated until complete (Supplementary Video 2). Each trajectory was repeated 5 times with the circular path having an average pixel position error of 6.54 ± 0.94, and the sinusoidal path having an average pixel position error of 7.73 ± 1.45. Given a pixel-to-millimetre-conversion described in Supplementary Fig. 2, this experiment shows that the orientation controller can steer the MFE image plane towards a target, with a positional accuracy of about 5mm.

Extended Data Fig. 2 – Example of lumen detection algorithm.

where (a) is the original MFE image and (b) is the segmented centre of the colon and (c) is the centre mass point of the lumen (x_l, y_l) which will be steered to the centre of image (x_C, y_C) and (d) represents the change in linear velocity of the MFE, given the distance between the estimated lumen (x_l, y_l) and centre of image (x_C, y_C).

Extended Data Fig. 3 – Feature detector for autonomous navigation.

The action of the autonomous controller is dependent on the absence, or presence of a lumen. When no clear lumen is present in the MFE image (Extended data Fig. 3–a), the FAST feature detector will return a low number of features, with a feature defined as a discernible edge in the image. Features are shown here as green circles. When a clear lumen is present in the MFE image (Extended data Fig. 3–b), the FAST feature detector will instead return a high number of features.

Data Availability

The authors declare that all the data supporting the findings of this study are available within the paper and its supplementary information files