Navigation benchmarking for autonomous mobile robots in hospital environment

Abstract

The widespread adoption of robotic technologies in healthcare has opened up new perspectives for enhancing accuracy, effectiveness and quality of medical procedures and patients’ care. Special attention has been given to the reliability of robots when operating in environments shared with humans and to the users’ safety, especially in case of mobile platforms able to navigate autonomously. From the analysis of the literature, it emerges that navigation tests carried out in a hospital environment are preliminary and not standardized. This paper aims to overcome the limitations in the assessment of autonomous mobile robots navigating in hospital environments by proposing: (i) a structured benchmarking protocol composed of a set of standardized tests, taking into account conditions with increasing complexity, (ii) a set of quantitative performance metrics. The proposed approach has been used in a realistic setting to assess the performance of two robotic platforms, namely HOSBOT and TIAGo, with different technical features and developed for different applications in a clinical scenario.

Introduction

Over the past years, the adoption of robotics in healthcare environments has pushed forward significantly, providing a wide range of solutions to improve the accuracy, effectiveness and quality of medical procedures and patient care^1,2. Such robotic devices are designed to automate repetitive tasks, improve operational efficiency, reduce costs, and provide support to the clinical staff, as well as to enhance patients’ care experience. They are able to carry out a wide variety of tasks ranging from the delivery of drugs and supplies to the cleaning and disinfection of the environments, up to the assistance in rehabilitation treatment³. Recently, the new version of the standard ISO 8373:2012 has defined medical robots as those “intended for use as medical electrical equipment or medical electrical systems”, also specifying that a “medical robot is not regarded as an industrial robot or a service robot”⁴. Evidently, the widespread adoption of robotic devices in healthcare draws attention to their reliability and to the safety of the users during interaction with them, especially for platforms operating in unstructured and sensitive contexts^5,6. In a clinical environment, the user’s interaction with a mobile robot may be unexpected. This is particularly common in tasks that involve the robot autonomous navigation, which may come across physical obstacles, patients, or other clinical operators while moving around. Although the adoption of robotic devices can guarantee numerous advantages for improving the effectiveness of patients’ treatment, also relieving clinical staff from repetitive and burdensome procedures, the analysis of the state of the art has shown that there are no clear and well-established performance, safety and environmental requirements for their adoption in a clinical scenario. These requirements would be paramount to ensure that a system is safe and effective in the spaces for which it has been conceived from the early stages of its design.

Hence, the aim of this paper is to propose a quantitative method, grounded on the use of a motion capture system, to assess the autonomous navigation capability of mobile robots designed for hospital applications, taking into account the physical constraints that may arise from the environment in which they are used. To achieve these objectives, a set of standardized batches of tests has been devised, to be carried out in a controlled but realistic hospital scenario involving both static and dynamic obstacles, as well as quantitative performance metrics. These tests were conducted on two different Autonomous Mobile Robots (AMRs), i.e. HOSBOT and TIAGo, as they were designed for two types of clinical operations in which robotics is widely adopted, namely logistics for the transport of hospital materials/equipment and assistance to healthcare personnel through mobile manipulators with reach-and-grasp capabilities. The proposed approach derives from the analysis of the characteristics of the hospital context and aims to be not dependent on the specific mobile platform tested. The method has been tested on two different AMRs, in order to highlight the robustness and flexibility of the approach for the validation of mobile robots in hospital contexts. Such an approach is also expected to be used for benchmarking purposes and for collecting useful data to prospectively identify guidelines for the adoption and proper functioning of medical robots with autonomous navigation capabilities.

The paper is organized as follows: Sect. Robot navigation in hospitals: performance requirements and assessment presents the current state of the art regarding mobile robots and the current challenges for their adoption in medical settings, Sect. Benchmarking method for robot navigation in hospital environment presents the proposed benchmarking approach and Sect. Application of the benchmarking method shows an application of the proposed approach. The obtained results are highlighted and discussed in Sect. Results and discussion; lastly, Sect. Conclusions presents the conclusions and possible further developments.

Robot navigation in hospitals: performance requirements and assessment

In the scientific literature related to robots for hospital environments there are many examples of systems conceived to guarantee a high quality of patients’ care, able to autonomously navigate and support patients with motor and cognitive disabilities in performing their rehabilitation tasks^3,7, adopted to assist with surgical procedures or to transfer items among different areas⁶. Additional applications also include preventing the spread of infections, reducing human error, supporting clinical operators by relieving them from repetitive or not urgent tasks and allowing them to focus attention on more high-priority activities. They are also adopted for many clinical tasks such as sterilization, surface and wall disinfection, remote monitoring, logistics, medical testing, social care and interaction⁸. In this context, the assessment of robots performance is necessary to ensure safe working conditions for the machine itself and for the humans^9,10, also in accordance with the ISO 13482:2014 standard¹¹, that provides guidelines on the safety of medical robots and sets out requirements to ensure that associated risks are ideally eliminated or reduced to an acceptable level.

A hospital ward represents a very complex and unstructured scenario. Developing robotic devices capable of autonomous navigation in such environments can be extremely challenging. Indeed, hallways, waiting areas, and patient rooms can be crowded with medical staff, patients, visitors, and various medical equipment. Along the hospital ward, there are single or multiple rooms for bedridden patients with at least one bed, a table, a chair for visitors, and usually a private bathroom. Examination rooms can be equipped with medical instruments and diagnostic equipment necessary to evaluate the patient’s condition, such as medical supplies, drug administration equipment, mobile diagnostics, and other devices needed for patient’s care^12,13. The European Commission has recently provided a framework for the design and construction of hospitals, which includes the distribution of the space for each room, the different departments and services , taking into account the accessibility, the separation among different functions, and the management of patients and staff flow. More in general, there are no overarching regulations and the scenario can vary meaningfully from one country to another¹⁴. For example, the dimension of assisted living facilities in the USA should be at least 11 ${\text{m}}^2$ per resident for their living and dining areas, whereas in Utah this value is about 30 ${\text{m}}^2$¹⁵. About intensive care units, the recommended surface for patient rooms is about 19 ${\text{m}}^2$ per bed in the USA¹⁶, and 25 ${\text{m}}^2$ for single-patient rooms in EU. In Italy the floor area in hospital rooms must not be less than 7 ${\text{m}}^2$ per bed in multi-bed rooms and 9 ${\text{m}}^2$ per bed in single-bed rooms¹⁷. It is hence clear that autonomously navigating robots should adapt to very heterogeneous and unstructured conditions and comply with the presence of humans¹⁸. They should be able to detect and avoid potential collisions, predict human movements, and plan paths that minimize risks to both robot and humans, also in response to different walking speeds, changes in direction, and social norms in different environments^18,19.

Evidently, the implementation of robots able to autonomously navigate in the hospital can be very demanding. Algorithms for autonomous navigation take care of managing localization, mapping and path planning, typically by using data from different types of sensors²⁰. Indeed, most of the navigation algorithms proposed in the scientific literature take advantage from the map of the environment in which the robot has to operate. Moreover, these approaches use data from sensors, such as LIDARS, cameras, or GPS, to estimate the position and orientation of the robot in the environment, through the Extended Kalman Filter (EKF) or particle filter, to manage uncertainty and improve the accuracy of the position estimation²¹. A navigation module is also in charge of computing the optimal route to a desired destination within the map, by using planning algorithms such as Rapidly Exploring Random Trees, $\text{A}_{}^{\ast }$, and Dynamic $\text{A}_{}^{\ast }$, and identifies the sequence of actions required to reach the target destination, taking into account physical constraints such as obstacles, speed limits and navigation preferences^22,23. In parallel, the environment map is updated with sensor data; this map has information about the location of obstacles, landmarks, or other features relevant to navigation^24,25. However, it is crucial to conduct extensive testing to assess the robot ability to avoid collisions with humans and negotiate any other static or dynamic obstacles during navigation. Each evaluation method can be used with different quantitative metrics. In the literature, some navigation performance metrics are proposed. Security metrics, dimensional metrics, and smoothness metrics were introduced to classify different key performance indicators²⁶. Security metrics are concerned with measuring distances between the vehicle and obstacles. Dimensional metrics, on the other hand, focus on optimal trajectories. Smoothness metrics evaluate the energy and time spent on decision-making, such as the bending energy and smoothness of curvature. However, in hospital environments, safety and task success are of greater importance than smoothness metrics.

These metrics are typically employed in tests carried out in structured environments with at most static obstacles. Indeed, available studies adopting these metrics propose generalised assessment methods and do not focus on their application in real settings, which are subject to changes and dynamic behavior.

Metrics to evaluate navigation performance in the presence of humans include success rate in reaching the target, path regularity, speed, and acceleration profiles. Studies adopting these metrics have mainly been carried out in pedestrian environments, addressing the problem of crowded places and not those of clinical settings²⁷. In particular, they analyse the effect of robots presence on humans in terms of possible alterations to free gait. Other types of studies, on the other hand, aim to evaluate socially aware robot navigation²⁸, by proposing evaluation metrics which apply to a different type of scenario but also in this case they may not be suitable for a clinically relevant context.

Few studies quantified the navigation performance of robotic devices in a hospital environment, e.g. for logistics and disinfection purposes²⁹. During the Covid-19 emergency, a robotic assistant for used in a real Covid treatment center was presented³⁰. Preliminary evaluations of the robot navigation capabilities were carried out in a laboratory scenario, using performance indicators. Nonetheless, in this case, none of the adopted indicators took into account the presence of obstacles, i.e. the evaluation was limited to free navigation. Furthermore, the study by Fang et al.³¹ proposes a novel visual SLAM algorithm, specifically designed to outperform classical methods in hospital environments. However, the performance indicators used to quantify the effectiveness of the proposed approach only focus on the trajectory error, computed with respect to a gold standard trajectory.

In summary, from the analysis of the literature, it emerges that few results were obtained in real clinical settings, no standard approaches have been proposed for quantitative assessment, and no minimum level of performance of the AMRs (at least in terms of speed, path to follow, and achievement of the desired target) have been identified for their adoption in such complex and sensitive scenarios. The aim of this paper is to fill the gap highlighted in the literature in terms of quantitative evaluation of the use of autonomous navigation robots in the specific case of a real hospital setting. To this purpose, the paper proposed a structured benchmarking protocol composed of standardised tests and devised a broad set of metrics, taking into account the different application scenarios in which a robot may operate, considering the presence of medical equipment, furniture, patients and healthcare workers.

Benchmarking method for robot navigation in hospital environment

This section describes a novel benchmarking method that includes protocol and metrics to provide a quantitative assessment of the navigation capability of a medical mobile robot acting in clinical environments. The proposed method is shown in Fig. 1.

Figure 1.

Overview of the proposed benchmarking method.

Benchmarking protocol

The benchmarking method is based on a protocol structured in four different batches of tests with progressive increase of path complexity, from free linear motion to complex scenarios with various combinations of static and dynamic obstacles.

More in detail, the proposed benchmarking protocol, as shown in Fig. 2, is composed of the following batches of tests:

• No Obstacle (NO): it consists of covering the distance $d_{\mathit{AB}}$, forth and back, in the absence of obstacles. Tests are carried out at set speeds indicated as $v_0$, $v_1$, and $v_2$.

• Static Obstacle (SO): it consists of covering the distance $d_{\mathit{AB}}$, forth and back, with one or two static obstacles in different configurations. In the case of double obstacles, these are placed at 3 different distances, as a function of the robot footprint $d_{\text{base}}$: $d_1=d_{\text{base}}$, $d_2=1.5d_{\text{base}}$, $d_3=2d_{\text{base}}$. Tests are carried out at the set speeds $v_0$, $v_1$, and $v_2$. For each combination of number/size/distance of obstacles and robot speeds, 3 repetitions are recorded. The SO tests includes the following configurations:

  • Single: one static obstacle is positioned in the middle of the linear path $d_{\mathit{AB}}$ (Fig. 2a).
  • Double Parallel (${\text{SO}}_{//}$): two obstacles of the same size are placed in the middle of the linear path $d_{\mathit{AB}}$ (Fig. 2b in grey).
  • Double Perpendicular (${\text{SO}}_{\perp }$): two obstacles of the same size are placed on a line perpendicular to $d_{\mathit{AB}}$ (Fig. 2b in yellow).
• Mobile Obstacle (MO, Fig. 2c): it consists of covering the distance $d_{\mathit{AB}}$, forth and back, in the presence of a moving obstacle, that is represented by a (second) testing mobile base teleoperated along the set path at fixed linear and angular speeds. The tested robot covers $d_{\mathit{AB}}$ at two different speeds $v_0$ and $v_1$, with the moving obstacle traveling at the same speed values. Hence, three different conditions are tested:

  • Passing (MO_P): the robot and the moving obstacle are facing along the path (Fig. 2c, in purple the dynamic obstacle);
  • Crossing (MO_C): the moving obstacle moves with a trajectory perpendicular to the line $d_{\mathit{AB}}$ (Fig.  2c, in orange the dynamic obstacle);
  • Overtaking (MO_O): the moving obstacle surpasses the robot at the maximum allowed speed $v_2$. In this case the tested robot moves only at $v_0$ (Fig. 2c, in green the dynamic obstacle).

• Complex Environment (CE): it consists of replicating the Crossing Test condition, at the speeds $v_0$ and $v_1$, by adding a static obstacle in the centre of the path, as shown in Fig. 2d.

The tests are also conveniently summarised in Table 1.

Figure 2.

Proposed batches of tests: (a) single static obstacle, (b) double static obstacle (parallel and perpendicular), (c) mobile obstacle (passing in purple, crossing in orange, overtaking in green, (d) complex environment. $S_1$, $S_2$, $S_3$, $S_4$ are the different sizes of the obstacles used, $d_1$, $d_2$, $d_3$ the distances at which they are placed in the case of tests with double obstacles.

Table 1.

Overview of the different batches of tests and configurations.

Batch of test
Name	Acronym	Configuration	Description
No obstacle	NO	–	Cover the distance $d_{\mathit{AB}}$ forth and back
Static obstacle	SO	Single double parallel double perpendicular	Cover the distance $d_{\mathit{AB}}$ forth and back with one or two static obstacle
Mobile obstacle	MO	Passing crossing overtaking	Cover the distance $d_{\mathit{AB}}$ forth and back with a mobile obstacle in different configurations
Complex environment	CE	Crossing with static obstacle	Cover the distance $d_{\mathit{AB}}$ forth and back with a mobile obstacle crossing and a single static obstacle

For each tested configuration, the robot travels a linear path from point A to point B and vice versa. Although the travelled distance may depend on numerous factors, including the dimensions of the rooms and corridors, safety standards, the size of the hospital and the specific use of each area, $d_{\mathit{AB}}$ is conventionally set at 5 m, taking into account relevant room characteristic sizes described in Sect. Robot navigation in hospitals: performance requirements and assessment. The values of robot speeds, $v_0=0.2$ m/s, $v_1=0.6$ m/s, and $v_2=1$ m/s, have been selected to be comparable to the speed of human walking indoors, representative for both visitors and clinical staff under normal and emergency conditions. In the CE condition, the robot is tested at speed $v_0$ and $v_1$. Indeed, such values have been selected to evaluate the performance of robotic systems in everyday clinical contexts, in which the robot should be able to navigate with static obstacles and slowly moving people. Consequently, the typical lower limit of human walking, ranging from 0.6 to 1.3 m/s³², was considered as the maximum speed. Moreover, four square static obstacles $S_1,S_2,S_3,S_4$ with an edge of [0.03, 0.15, 0.30, 0.60] m are used, which fall within the dimensions normally adopted for small hospital furnishings³³ and general obstacles that a robot may encounter along its path, such as medical equipment of different types and sizes. Three trials are performed for each configuration.

Benchmarking metrics

The proposed method includes a performance analysis to quantitatively assess the navigation algorithm. This analysis is based on data acquired during the testing phases in the real scenario to evaluate how the robot moves and interacts with its surroundings. More in detail, the benchmarking method includes the recording of the robot trajectories, (e.g. by using a motion capture system with reflective markers positioned on the robot, as described in more detail in Subsect. Experimental testing). From the obtained trajectories, the following performance indicators are extracted:

• Completion Time (CT): time required for the robot to move from point A to point B and vice versa. It is computed as:

  $$\begin{array}{c}\hfill CT=t_f-t_0\end{array}$$ 1

  where $t_0$ and $t_f$ are the starting and ending time instants. This indicator is important for assessing the robot capability in achieving the assigned goals.
• Path Length (PL): total distance to move from point A to point B and vice versa. It is computed as:

  $$\begin{array}{c}\hfill PL=\frac{{\int }_{t_0}^{\text{f}}\dot{p}(t)dt}{d_{\text{AB}}}\end{array}$$ 2

  where $\dot{p}(t)$ is the linear velocity of the robot and $d_{\text{AB}}$ is the total distance traveled by the robot. PL can be influenced by various factors, such as the complexity of the environment, the robot planning algorithms, and its ability to avoid unnecessary detours or backtracking. A shorter path length indicates a more efficient navigation strategy, as the robot can reach its goal with minimal movement, conserving energy and time.
• Distance Error (DE): error made in reaching the desired position. It is computed as:

  $$\begin{array}{c}\hfill DE=\frac{\Vert x_{\mathit{goal}}-x(t_f)-T_p\Vert }{T_p}\end{array}$$ 3

  where $x_{\mathit{goal}}$, $x(t_f)$ and $T_p$ are the desired and reached positions in the reference frame and the tolerance in reaching goal position, respectively, assuming the same tolerance on both navigation axes. It is a measure of the accuracy of the robot in reaching its destination. High DE may be related to various reasons, including, inaccuracies in its sensors that can lead to errors in the robot localization (i.e., erroneous determinations of its own position in the environment), limitations in the accuracy of its movement control, and environmental factors such as uneven terrain or dynamic obstacles. Low DE indicates that the robot navigation system is accurate and reliable in reaching its targets, whereas a high DE suggests that the robot movements may be less accurate or subject to errors.
• Orientation Error (OE): error made in reaching the desired orientation. The computation of this parameter depends on the representation adopted for the orientation. In its general form it can be expressed as:

  $$\begin{array}{c}\hfill OE=\frac{\Vert o_{\mathit{goal}}-o\left(t_f\right)-T_o\Vert }{T_o}\end{array}$$ 4

  where $o_{\text{goal}}$, $o\left(t_{\text{f}}\right)$ and $T_o$ are the desired and reached orientation in the reference frame and the tolerance in reaching goal orientation, respectively. The sources are similar to DE.
• Success Rate (SR): percentage of successfully completed navigation tasks or missions out of the total number of attempted ones. It is computed as:

  $$\begin{array}{c}\hfill SR=\frac{N_{\mathit{succ}}}{N_{\mathit{tot}}}100\end{array}$$ 5

  where $N_{\mathit{succ}}$ and $N_{\mathit{tot}}$ are the number of completed and attempted assigned tasks, respectively. A test is defined as failed if DE or OE is greater than their tolerances, or if a collision occurs while performing the task. SR is a key performance metric that indicates how effectively the robot can reach its destinations. A high success rate indicates that the robot navigation system is reliable and effective in accomplishing its tasks, while a low success rate suggests that the robot may have encountered challenges or difficulties in completing its navigation objectives. The success rate is influenced by various factors, including the accuracy of the location of the robot, the efficiency of its path planning algorithms, the quality of its sensors, the adaptability of its control algorithms, and the complexity of the environment in which it operates. Moreover, the success rate is essential for evaluating the reliability and safety of autonomous robots in real-world applications. For safety-critical tasks, achieving a high success rate is of utmost importance to ensure that the robot can accomplish its mission with minimal errors or failures.
• Minimum Distance from Obstacle (MDO): minimum distance between the robot and any obstacles in its surroundings during its navigation. It is computed as:

  $$\begin{array}{c}\hfill MDO=\text{min}\{D(t)\}\end{array}$$ 6

  where D(t) is the function distance between robot and obstacle over time. MDO is a metric used to assess how effectively a robot can navigate through an environment while maintaining a safe distance from obstacles. This metric is analyzed to ensure that the robot maintains a safe margin or clearance from objects, walls, or other obstacles to avoid collisions and guarantee safe navigation. The metric is important in environments with dynamic obstacles, crowded spaces, or tight passages, where the risk of collision is higher.
• Time Cruise Speed (TCS): percentage of time the robot moves at its cruise speed with respect to the total time to complete the task. It is computed as:

  $$\begin{array}{c}\hfill TCS=\frac{T_{v_{\mathit{set}}}}{\mathit{CT}}100\end{array}$$ 7

  where $T_{v_{\mathit{set}}}$ is the time intervals at which the robot moves at its cruise speed and the CT is previously defined. This metric can be used to evaluate the efficiency and energy optimization of robot navigation during navigation.

Table 2 summarises all the performance indicators described above.

Table 2.

Proposed performance indicators.

Benchmarking metrics
Name	Acronym	Definition
Completion time	CT	Time required to travel the desired length by the robot back and forth
Path length	PL	Desired length to be travelled by the robot back and forth
Distance error	DE	Error made in reaching the desired position
Orientation error	OE	Error made in reaching the desired orientation
Success rate	SR	Percentage of successfully completed navigation tasks or missions out of the total number of attempted ones
Minimum distance from obstacle	MDO	Minimum distance between the robot and any obstacles during its navigation
Time cruise speed	TCS	Percentage of time the robot moves at its cruise speed with respect to the total time to complete the task

Application of the benchmarking method

This section describes the application of the method presented in Sect.  Benchmarking method for robot navigation in hospital environment to two MARs used to perform different tasks in hospitals.

Experimental testing

The experimental testing has been performed in a simulated hospital environment, shown in Fig. 3, that replicates the physical layout and characteristics of a real hospital, including patient’s beds, corridors, waiting areas, and furnishings. The simulated hospital environment offers a safe and controlled space to test and validate robots without jeopardizing patients’ safety. Moreover, the autonomous navigation performance of the robotic platforms has been evaluated by using the Vicon Vero optoelectronic system version 2.2 (Vicon Motion Systems Ltd., UK), composed of 8 cameras with a resolution of 2048$\times$1088 MP. Reflective spherical markers were placed on the robots and obstacles so that cameras could record the trajectories covered by the robotic platforms during the trials. Specifically, 8 markers (diameter: 14 mm) were placed on the robots (4 on the bottom base and 4 on the top) and 4 markers (diameter: 14 mm) were placed on the obstacles.

Figure 3.

The simulated hospital environment adopted for experimental testing.

To demonstrate the efficacy of the navigation benchmarking approach, it has been tested on two AMRs: HOSBOT (HOSpital roBOT, designed and developed by Scuola Superiore Sant’Anna, Pisa, Italy) and TIAGo (PAL Robotics S.L., Barcelona, Spain) shown in Fig.  4a and b, respectively.

Figure 4.

The tested robotic platforms: HOSBOT (a) and TIAGo (b).

These robots were chosen to cover the maximum set of tasks that mobile robots can perform in an unstructured clinical scenario, namely logistics and assistance to healthcare personnel and patients. As described in more detail below, the two AMRs have different physical characteristics, and this choice was made to make the proposed algorithm generalisable, regardless of the intrinsic characteristics of the robotic platforms to be used.

HOSBOT is a mobile robotic platform for hospital logistics comprised of an independent commercial AMR (in this version a RB-1, Robotnik Automation S.L., Valencia, Spain) and a rack containing smart containers, customized and optimized to be transported by the selected robot. It is equipped with a customized rack (named SmartRack) that lifts up to 35 mm to allow connection and transportation. The rack has a wheeled structure that carries RFID-equipped containers, called SmartBoxes, as shown in Fig. 4b. The separation between the carrier and the carried structures (SmartRack and SmartBoxes) provides a high level of flexibility, enabling the simultaneous management of multiple SmartRacks by using just a single mobile robotic platform. The mobile robot features a circular geometry with an outer diameter of 0.50 m, whereas the smart rack has a square base structure with 0.60 m side. Once again, this parameter can be adjusted by leveraging the modularity of the system. The weight of the HOSBOT system is 55 kg, consisting of 20 kg for the mobile base RB-1 and 35 kg for the SmartRack with full load (profile structure plus smartboxes). The height of the system (in its current configuration) is 1.50 m, chosen to house several smartboxes for the transport of consumables while also not completely obstructing the view of people crossing its path. However, this parameter can be potentially customized according to the design need. Safe autonomous navigation is guaranteed by a laser scanner and RGB cameras mounted on the frontal part of the AMR.

On the other hand, TIAGo integrates an anthropomorphic arm with 7 Degrees of Freedom (DoFs), a gripper to enable manipulation, a RGB-D camera and a vocal synthesizer. The base footprint size of the TIAGo ($d_{\text{base}}$) is 0.54 m, the weight is 72 kg, and the height varies from 1.10 to 1.45 m, depending on the elongation of the torso. All its components have already been integrated into the ROS-based robotic architecture. The integration of all these components allows the robot to be used flexibly in different applications, including tasks involving human–robot interaction. Thanks to its modularity it is suitable for monitoring and supporting the rehabilitation task of the upper limb, replicating the therapist-patient binomial of conventional treatment⁷. TIAGo has also been adopted in hospital scenarios during the recent pandemic for logistics and environmental disinfection tasks in a Covid Center³⁰.

HOSBOT and TIAGo adopt both Simultaneous Localization and Mapping (SLAM) algorithms³⁴ for autonomous navigation. This allows them to create a map of the surrounding environment and to establish their position within that map, simultaneously. The SLAM navigation algorithm works in two main phases: mapping and localization. During the mapping phase, the robot explores the environment and collects sensory data to create a static map of the features of the environment, such as walls, objects, and obstacles. The robot uses its sensors, such as cameras or laser sensors (LIDAR), to detect obstacles and gather information about their distance and location. The robot compares the current sensor data with the existing map to calculate its real-time location. This can be done by using point matching or information filtering algorithms, which compare the detected environment features with those present in the map. Moreover, the tested robotic platforms adopt GMapping³⁵. It estimates the robot current position within the map. The particle filter represents different assumptions about the position of the robot, maintaining a distribution of particles representing the different possible positions. The adopted parameters for global and local planners and costmaps are reported in Table 3 for both HOSBOT and TIAGo.

Table 3.

Adopted navigation parameters for HOSBOT and TIAGo.

Global and local planners	HOSBOT	TIAGo
Update frequency of the global planned path	5 Hz	1 Hz
Tolerance at the goal point	0.3 m
Update frequency of the local planned path	5 Hz	1 Hz
Controller frequency	10 Hz
Explored samples in the x velocity space	3	10
Explored samples in the y velocity space	3	0
Explored samples in the theta velocity space	10	20
Tolerance in reaching goal position	0.1 m	0.2 m
Tolerance in reaching goal orientation	0.01 rad	0.2 rad
Maximum translational velocity	1.5 m/s
Maximum rotational velocity	2.0 rad/s

Costmap parameters	HOSBOT	TIAGo
Global and local costmap update frequency	5 Hz	10 Hz
Local obstacle range	0.3 m	3.5 m
Local raytrace range	5.5 m	4.0 m
Local width and height	10 m	5.0 m
Local resolution	0.050 m	0.025 m

Statistical analysis

The Wilcoxon paired-sample test has been applied for a comparative analysis of each performance metric at different velocities for both HOSBOT and TIAGo, after verifying that the data do not belong to a Gaussian distribution. The significance level was set at $p=0.05$.

Moreover, a correlation analysis has been performed to evaluate the strength and direction of the linear relationship between the performance indicators. Such a statistical approach, based on the Pearson index, has been adopted to calculate the correlations between the selected metrics and evaluate the correlation among the proposed metrics, and monitor any redundancies. The derived correlations are considered very low if $\rho \le 0.19$, low if $0.20\le \rho \le 0.39$, moderate if $0.40\le \rho \le 0.59$, strong if $0.60\le \rho \le 0.79$ and very strong if $0.80\le \rho \le 1.0$.

Results and discussion

This section presents an analysis of the results obtained with the proposed method on HOSBOT and TIAGo in a simulated hospital environment. More in detail, Fig. 5 representatively shows HOSBOT trajectories during the execution of the proposed batches of tests at speed $v_0$. From left to right, examples of the trajectories are shown for NO, for SO with a static obstacle of size ${\text{S}}_2$, for ${\text{SO}}_{//}$ with obstacles of size ${\text{S}}_1$, for ${\text{SO}}_{\perp }$ with obstacles of size ${\text{S}}_1$. For MO and CE, the figure shows ${\text{MO}}_P$, ${\text{MO}}_C$, ${\text{MO}}_O$ with moving obstacle at $v_1$, CE with moving obstacle at $v_1$ and static obstacle of size ${\text{S}}_2$. In this case TIAGo is used as moving obstacle. As evident, the robot tries to plan and execute a path even in the presence of obstacles along its route. This ensures that the robot reaches the target pose, within its tolerance, thanks to the navigation algorithm. However, this may not be enough for an hospital applications. In fact, it is necessary to ensure the total absence of collisions and an execution time adequate for the distance to cover. Similar recordings were also obtained for TIAGo under the same conditions, with HOSBOT used as moving obstacle.

Figure 5.

Representative trajectories recorded for HOSBOT during the execution of the proposed batches of tests at speed $v_0$. The trajectories are shown in blue and those of the TIAGo, used as a moving obstacle, are in orange. The same experiments were performed with the TIAGo under testing and by using the HOSBOT as a moving obstacle (not shown for the sake of brevity).

Figure 6 shows the mean values and their confidence intervals of the proposed metrics at $v_0$, $v_1$, $v_2$ for both AMRs. The results shown for each performance indicator have been normalized with respect to the full range of values obtained for each type of test of the batches, among all the tested speeds. Bar plots report also the significant statistical differences. The specific values of p are shown in Table 4.

Figure 6.

Results obtained for the performance indicators at different speed values ($v_0$, $v_1$, $v_2$) for each banch of tests. The results are normalized (HOSBOT on the left, TIAGo on the right). Asterisks denote statistically significant differences (*: $0.01<p\le 0.05$; **: $0.001<p\le 0.01$; ***: $0.0001<p\le 0.001$; ****: $p\le 0.0001$).

Table 4.

Statistically significant differences and values of p for HOSBOT and TIAGo.

Benchmarking metric	Batch of test	Velocity values	$\mathit{p}$
			HOSBOT	TIAGo
		$v_0-v_1$	$1.34{10}^{-6}$	$4.72{10}^{-15}$
		$v_0-v_2$	$3.59{10}^{-8}$	$3.36{10}^{-15}$
	SO	$v_1-v_2$	$5.19{10}^{-3}$	-
	MO	$v_0-v_1$	–	$1.45{10}^{-3}$
CT	CE	$v_0-v_1$	–	$4.06{10}^{-2}$
		$v_0-v_1$	–	$8.12{10}^{-4}$
		$v_0-v_2$	–	$1.26{10}^{-7}$
	SO	$v_1-v_2$	–	$1.17{10}^{-2}$
PL	CE	$v_0-v_1$	–	$4.06{10}^{-2}$
DE	SO	$v_0-v_2$	–	$1.02{10}^{-2}$
OE	SO	$v_0-v_2$	–	$2.06{10}^{-3}$
		$v_0-v_1$	–	$2.58{10}^{-4}$
MDO	SO	$v_0-v_2$	–	$1.73{10}^{-5}$
		$v_0-v_1$	$2.82{10}^{-5}$	$5.20{10}^{-11}$
	SO	$v_0-v_2$	$4.68{10}^{-7}$	$3.36{10}^{-13}$
TCS	MO	$v_0-v_2$	–	$3.93{10}^{-3}$

As expected, the CT in NO decreases as speed increases for both robots. In particular, it decreases more than half in the transition from speed $v_0$ to $v_1$ (for HOSBOT, CT decrease from $51.73\pm 0.81$ s to $20.19\pm 0.16$ s for $v_0$ and $v_1$, whereas for TIAGo CT reduces from $48.10\pm 0.83$ s to $22.50\pm 1.02$ s). The CT also decreases in the transition from speed $v_1$ to $v_2$ with a smaller reduction than in the previous case. The PL for HOSBOT is greater than the one for TIAGo in the three tested velocities in the NO condition. HOSBOT performs a 10% longer path in the case of $v_1$ and $v_2$. The higher values of errors in reaching the target in terms of orientation were obtained in the case of higher cruising speeds for both HOSBOT and TIAGo. Reducing the speed corresponds to an increase in the accuracy of reaching the assigned target. The highest accuracy (i.e. the minimum values of position and orientation error) were obtained from HOSBOT at speed $v_1$ ($0.05\pm 0.02$ m and $0.02\pm 0.01$ rad). The obtained results in NO are due to the different structures and related weight, although the two robots have similar path planning parameters, as reported in Table 3. In SO condition, CT decreases with the same trend shown in the previous case. TIAGo is able to cover a shorter distance ($8.31\pm 1.80$ m) in case of $v_0$. The highest OE is recorded for TIAGo at $v_2$, the lowest is for HOSBOT at the same speed value. The obtained results show that both robots were able to calculate an alternative route to reach the target without colliding with the obstacles. The robots have adapted their path, even if the obstacles are placed in a different position, as often happens in an extremely crowded clinical scenario. In addition, the high success rates and low number of collisions, recorded even on repeated tests, demonstrate good reliability of both platforms.

As described in Sect.  Benchmarking method for robot navigation in hospital environment and shown in Fig. 6, the autonomous platforms were tested in MO scenario only at two different speeds ($v_0$ and $v_1$). In the case of MO, navigation becomes even more complex and requires greater adaptability and responsiveness. The navigation algorithm has shown a good ability to detect and follow moving obstacles, especially in the case of crossing, thanks to the detection of the obstacle which occurs earlier than in other cases. In this case, it is not only necessary to detect obstacles but also to constantly monitor them to predict their future states. Furthermore, as the speed increases, the results of the indicators tend to get worse, so the robot takes more time to reach the target with a low level of accuracy in position and orientation.

In a complex environment, the planned path may become irregular or even impossible to follow as complexity increases. The mobile base should be able to adapt its path in real-time to avoid moving obstacles. This obviously generates an increase in the CT, SR, and PL. Navigating in the presence of moving obstacles requires greater attention to safety. The mobile base should slow down or stop quickly if obstacles get too close or move unexpectedly and the obtained results depend on the navigation algorithm. In general, the error in reaching the target in terms of position and orientation worsens as the complexity of the path and the speed increase. This analysis shows that, even in complex scenarios, the robots have managed to avoid collisions and guarantee a very high success rate. This is a fundamental requirement to ensure the safety of people and things in any conditions. Furthermore, a good trade-off between the size of the robot and the dimension of the environment should be at least greater than 10% of the characteristic size of the robot to guarantee the minimum level of maneuvering space, by assuming HOSBOT as the worst case in terms of dimension. Indeed, HOSBOT has a 10% higher PL in the case of $v_1$ and $v_2$. The PL represents the length of the path planned by the robot and reflects the amount of space required for navigation. The PL represents the length of the path planned by the robot and reflects the amount of space required for navigation and for ensuring adequate maneuvering space in a hospital environment for both HOSBOT and TIAGo. Therefore, determining the 10% PL value is based on the trade-off between the size of the robot and the need to have sufficient space to navigate effectively and safely. The CT increases with the complexity of the environment, although for many clinical tasks, it is not an extremely compelling requirement. The error of reaching the target in position or orientation can be a limiting factor in some clinical applications, such as in the case of interacting with patients and clinical staff to perform an assigned task. In terms of statistical significance, it can be seen that the results for CT and TCS have similar trends for both robots, showing a reduction in the time required to complete the task as well as in the time the robot holds a cruise speed, particularly for SO condition, influenced also by the selected speed. Significant differences are observed in DE (Deviation Error) and OE (Orientation Error) for the TIAGo robot. These differences may arise from several factors. Firstly, the inherent physical attributes of the TIAGo, such as its heavier weight compared to the HOSBOT, result in distinct inertial properties. Secondly, the TIAGo encounters obstacles, potentially of similar sizes, further contributing to the observed differences. The robot may therefore find it more difficult to re-optimize its trajectory, especially in a small space, once the obstacles have been overcome. With TIAGo, a statistically significant difference was also obtained for PL in the case of SO, among all speed value combinations. This also suggests that at high speeds the system makes more errors in maintaining the optimal path, although it manages to reach the target. Inertia could also affect this situation, for example in cases when the robot has to turn or change direction to avoid an obstacle. It is important to understand from these findings that speed, environmental complexity, safety considerations and robot manoeuvrability are linked each other, even if described by different benchmarking metrics.

The purpose of this paper and the discussion of the results is not to specifically compare the two robots under consideration. Rather, the aim is to provide a general evaluation method that can be applied to any mobile robot in a hospital environment, by taking into account the requirements that depend on the context of use (e.g. logistics, disinfection, rehabilitation). The testing of two robots with different characteristics allows for further generalization of the proposed method , which does not depend on the physical characteristics of the robot under analysis (in terms of base footprint, weight, and height). By contextualising these results within the broader framework of robot evaluation, the study aims to provide insights for optimising navigation algorithms in hospital scenarios. Furthermore, the analysis shows how different factors influence navigation performance, with a tendency for performance indicators to deteriorate as speed increases. This highlights the importance of considering speed as a critical variable when optimising navigation algorithms, especially in environments where accuracy is paramount. Therefore, the adaptability of navigation and the crucial role of safety become extremely important, especially in sensitive contexts such as hospitals, always considering the need to adapt the robot performance to the specific requirements. In these circumstances, the planned path may deviate or become impractical due to obstacles or spatial constraints; consequently, the mobile base must dynamically adapt its trajectory in real time to avoid obstacles, resulting in increased Completion Time (CT), Success Rate (SR), and Path Length (PL). The need for such adaptability requires robust navigation algorithms capable of effectively handling complex scenarios. Navigating in environments with moving obstacles requires increased attention to safety considerations. The ability of the robot to quickly slow down or stop in response to unexpected movements or close encounters with obstacles is critical to ensuring the safety of both people and assets in the environment. This emphasis on safety is in line with the overall goal of deploying autonomous robots in sensitive environments such as hospitals, where the well-being of patients is paramount. The obtained results suggest the importance of the size of the environment relative to the dimensions of the robot in facilitating manoeuvrability. Ensuring sufficient manoeuvring space is essential for effective navigation, particularly in environments with limited spatial constraints. This consideration is specially relevant in hospital environments where the presence of medical equipment, personnel and patients requires agile and unobtrusive navigation.

The overall results for both robots combined from the pairwise correlation analysis among the indicators are shown in Fig. 7. As evident, the highest values for correlation coefficients are obtained for the $DE-OE$ and $CT-TCS$ pairs, with correlation coefficients of 0.48 and 0.47, respectively. The moderate positive correlation between CT and TCS suggests that the robot does not immediately reach cruising speed, so when the task duration is longer, it keeps the set speed for a longer time. The indicators concerning errors exhibit similar behavior when compared to CT and PL. Indeed, a negative correlation is obtained for both DE and OE compared with the PL and CT indicators, so the error in terms of position and orientation is greater when the task is traveled in less time (which corresponds to less distance). A moderate correlation is found between PL and DE ($\rho =-0.4$), while it is low between PL and OE ($\rho =-0.37$), between CT and DE ($\rho =-0.23$), and between CT and OE ($\rho =-0.2$). Regarding the minimum distance recorded during the tasks between the robots and obstacles (static or dynamic), there is a low correlation between MDO and TCS ($\rho =0.33$), and a very low correlation between MDO and DE ($\rho =0.2$) and between MDO and CT ($\rho =0.18$). As the time needed to perform the task increases, the accuracy in reaching the target position is higher. The results of the statistical analysis for the pairs of indicators suggest that they have a relationship of their information content and the proposed protocol shows a good level of redundancy. On the other hand, considering the results obtained from the pairwise correlation, it is still possible to adopt a minimum set of indicators. The pairs CT and PL, DE and OE, TCS and MDO, had a positive correlation from the pairwise comparison analysis. By examining how the performance indicators correlate with each other, it is possible to gain insights into how different aspects of robot performance are related. Correlation analysis helps to assess potential trade-offs between performance metrics. For instance, a strong negative correlation between task duration and position error suggests that completing tasks more quickly may come at the expense of higher error rates in positioning. If certain indicators are correlated, it suggests that they may capture similar aspects of performance, so a set of them can be chosen to prioritize or refine measurements to capture different aspects of performance to provide a more comprehensive evaluation.

Figure 7.

Pairwise correlations among the proposed performance indicators for both robots.

Conclusions

In this work a structured method for benchmarking autonomous navigation of mobile medical robots in hospital environments has been presented. The paper provides a complete framework to be used for quantifying useful performance to prospectively ensure safety in presence of spatial constraints in the environment. The proposed benchmarking method is based on multiple tests with conditions of increasing complexity and on the assessment of the navigation path of the robotic devices by means of an optoelectronic system. In this paper two different robotic platforms underwent testing and a set of performance indicators was caluclated to have a synthetic measure of the features of the missions carried out.

The obtained results suggest the truthfulness of the selected parameters to define the minimum set of characteristics that a robotic system should have to be used in a hospital scenario. Besides, the two tested robotic devices (i.e. HOSBOT and TIAGo) showed a performance adequate for their use in a real scenario, also according to the non-standard criteria for sizing clinical environments reported in literature.

Future efforts will be devoted to collect data from autonomous navigation robots in hospital environments implementing different navigation algorithms to evaluate the impact of each algorithm on autonomous navigation performance and technical requirements of the environment.

Author contributions

C.R. and F.S. analyzed the literature, designed the paper, proposed the benchmarking method, tested the robotic platforms with the proposed approach, analyzed the experimental data and wrote the manuscript. C.T. organized the experimental sessions, acquired the data and contributed to write the manuscript. N.L.T. contributed to design the benchmarking method, to prepare the setup and collect data, to write the manuscript. M.C. and G.C. contributed to the design of the experiments, discussed the results and contributed to the manuscript writing. L.Z. contributed to the design of proposed platform, discussed the results, wrote the paper and supervised the study. All the authors read and approved the final version of the manuscript.

Data availibility

All data generated during and/or analysed in the current study are not openly accessible but are available from the corresponding author upon reasonable request.

Competing interests

The authors declare no competing interests.