Area-Coverage Planning for Spray-based Surface Disinfection with a Mobile Manipulator

Abstract

The use of robots has significantly increased to fight highly contagious diseases like SARS-COV-2, Ebola, MERS, and others. One of the important applications of robots to fight such infectious diseases is disinfection. Manual disinfection can be a time-consuming, risky, labor-intensive, and mundane, and humans may fail to disinfect critical areas due to the resulting fatigue. Autonomous or semi-autonomous mobile manipulators mounted with a spray nozzle at the end-effector can be very effective in spraying disinfectant liquid for deep disinfection of objects and surfaces. In this paper, we present an area-coverage planning algorithm to compute a path that the nozzle follows to disinfect surfaces represented by their point clouds. We project the point cloud on a plane and produce a polygon on which we generate multiple spray paths using our branch and bound-based tree search area-coverage algorithm such that the spray paths cover the entire area of the polygon. An appropriate spray path is chosen using a robot capability map-based selection criterion. We generate mobile manipulator trajectories using successive refinement-based parametric optimization so that the paths for the nozzle are followed accurately. Thereafter, we need to make sure that the joint velocities of the mobile manipulator are regulated appropriately such that each point on the surface receives enough disinfectant spray. To this end, we compute the time intervals between the robot path waypoints such that enough disinfectant liquid is sprayed on all points of the point cloud that results in thorough disinfection of the surface, and the particular robot path is executed in the minimum possible time. We have implemented the area-coverage planning and mobile manipulator motion planning on five test scenarios in simulation using our ADAMMS-SD (Agile Dexterous Autonomous Mobile Manipulation System for Surface Disinfection) robot. We benchmark our spray path generation algorithm with three competing methods by showing that the generated paths are significantly more efficient in terms of area coverage and reducing disinfectant wastage. We also show the time interval computation between successive waypoints results in thorough disinfection of surfaces.

1. Introduction

A once-in-a-century pandemic [1] has resulted from the spread of the novel coronavirus disease 2019 (COVID-19), caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). This has resulted in several million people losing their lives and many more having serious health consequences. There have been other dangerous viruses like Ebola, MERS, etc., in recent years that have resulted in severe health hazards. In the future, pandemics from influenza-type viruses like SARS-CoV-2 may become more frequent due to the increased intimate contact with certain wild life [2]. One of the biggest challenges with several viruses is that they are highly contagious. Studies have shown that they can remain infectious on metal, glass, wood, fabrics, and plastic surfaces for several hours to several days [3]. Although not the prime mode of spread, they can spread via surfaces. When a coronavirus-infected person coughs or sneezes, the infected droplets land on the surrounding surfaces. When these surfaces are touched by another person (even after several hours) who then touches their mouth, eyes, or nose, they can contract the virus [4]. Locations where there are a constant influx and out-flux of people like washrooms, train stations, public transport vehicles, offices, restaurants, grocery stores, gyms, etc., are places where the virus can easily spread through surfaces. Hence, it is important to disinfect regularly touched surfaces in such environments to reduce the spread of viruses.

Surface disinfection can play an important role in preventing the spread of viruses. Small surfaces like doorknobs, switches, taps, touch screens, etc., can be disinfected manually. However, manually disinfecting large areas regularly can be extremely risky, time-consuming, labor intensive, and mundane. There is potential for a person disinfecting to miss or inadequately disinfect surfaces due to fatigue. Moreover, it can be expensive due to the disposable protective gear required to execute the job. Alternatively, autonomous robots can be used for performing this task in an efficient and effective manner. The main advantage of autonomous robots for disinfection is that they can operate 24/7 without putting humans at risk. They can work at all times by performing disinfection and auto-recharging. This liberates time for healthcare workers to focus on other critical activities, such as interacting with patients. They can even self disinfect. Hence, it is beneficial to automate this repetitive task by using disinfection robots [5].

Robots for disinfection can be divided into two distinctive branches, UV light-based, and spray-based disinfection robots. These robots can be mobile robots, manipulators, mobile manipulators, or even drones, each operating autonomously or semi-autonomously with human-in-loop [6]. One of the ways to disinfect spaces with UV light [7] is with mobile robotic systems that mount a large UV source. Such a robot can be used to disinfect large rooms by moving around to different locations and switching on the UV light source. These robots can be driven and operated manually or autonomously and have been used commercially in hospitals, food banks, airplanes etc [8], [9]. However, these robots are used for general disinfection and not deep disinfection of specific objects or surfaces like handles, mugs, medical equipment, etc., that may be exposed to the virus. For disinfection of such objects, mobile manipulators [10] like ADAMMS-UV [6] that manipulate and disinfect occluded objects using a UV light wand can be used. Although UV light provides an effective solution for disinfection, especially of electronics and sensitive equipment, it may not be useful for disinfection of reflecting surfaces, fabrics, and rough surfaces as UV light is not highly penetrating. Even slight occlusion can result in a shadow being cast on the infected surface rendering the UV light ineffective. Moreover, for effective disinfection, the intensity of the UV light needs to be high, which may not always be possible. Furthermore, it is challenging to recognize if a surface has indeed been disinfected with UV light as there is no residue left to detect. And finally, disinfecting in the presence of humans may be challenging due to exposure risks associated with UV light.

Liquid spray-based disinfection can be divided into two categories: spraying disinfectant with and without wiping and wiping with moist rag. NASA-JSC [11] have introduced protocols for wiping to protect the health of the residing astronauts and the safety of the internal environment of the International Space Station (ISS). Wiping-based disinfection can be very effective for thorough disinfection, however it can be challenging for robots to automate the wiping motion. Moreover, sprays used for disinfection typically evaporate after being sprayed. Furthermore, wiping can be challenging for fabrics, or flexible objects. Liquid spray-based disinfection robots spray disinfectant liquid on surfaces and can be used on fabrics, rough surfaces, and places where UV light may not reach. With enough dosage, surfaces can be very effectively disinfected. The main advantage of liquid spray-based disinfection is that the disinfectant spray droplets can be detected with appropriate vision-based technologies, and we can measure how effectively surfaces are sterilized. Although UV disinfection may be effective in disinfection of large areas at a time [9], they may not be very effective at places where deep disinfection is required. In such cases, spray-based disinfection may be useful. On the other hand, spray-based disinfection may not be effective where critical instruments need disinfection and may get damaged due to the liquid. However, for most objects, disinfection using a spray that evaporates can be very effective. Several mobile robots have been developed recently that spray liquid disinfectants as they move around. They are effective in disinfection of corridors by performing uniform spraying and do not focus on any particular equipment or surface. To increase the effectiveness of spray disinfection, mobile manipulators XDBOT [12] (eXtreme Disinfection roBOT, developed at NTU) and ADAMMS-SD (Agile Dexterous Autonomous Mobile Manipulation System for Surface Disinfection, developed at our lab [13]) are used to spray the liquid in particular directions and disinfect desired surfaces thoroughly. The nozzle for the spray is attached to the end-effector of the UR5 manipulator of ADAMMS-SD is shown in Fig. 1. The manipulator gives the robot the ability to focus the spray of the liquid on desired surfaces due to its wide reachability in distance and orientation.

Fig. 1.

The spray nozzle attached to the end-effector of the UR5 robot of ADAMMS-SD.

The focus of this paper is to develop a novel algorithmic framework for effective and efficient liquid spray-based disinfection of surfaces and large objects, like shown in Fig. 2 with a spray nozzle attached to the end-effector of a mobile manipulator. In this work, a human operator decides the area to be sprayed on. We assume that the operator extracts an area and the corresponding point cloud that needs to be sprayed on. Hence, given the point cloud, the goal for the robot is to compute configuration space (or joint) trajectories of the mobile manipulator such that the entire surface of the object is sprayed with enough disinfectant to guarantee thorough disinfection. The robot trajectory should be such that the nozzle at the end-effector follows a path that covers the entire area with the spray from the nozzle. We can potentially spray in a zig-zag fashion on the surface to cover the area; however, the resulting path may not cover the entire area, and if it does, it may end up wasting disinfectant liquid by going on areas of the surface multiple times. Hence, to make the spraying process efficient and effective, we have developed a spray path generation algorithm. For this, we record a point cloud of the surface to be sprayed and project it onto a plane. The 2D polygon generated by the edge of the projected point cloud is used as the area to be covered. We have developed a branch and bound-based approach for generating a spray path that covers the entire area of the polygon using the sides of the polygon. This gives us spray paths that, if followed, will result in covering the entire area of the polygon. Moreover, the spray remains inside the polygon, resulting in reduced wastage of the disinfectant fluid, making the process more efficient in terms of reducing spray fluid wastage. We choose a spray path using a capability map-based selection criterion for the mobile manipulator. We then map the chosen spray path on the original point cloud and determine the nozzle tip workspace path based on the spray model that we have used. In our disinfection framework, once the spray path is available, one can use any constrained trajectory generation algorithm that can incorporate the constraints on the rate of spray motion and frustum of cone orientation. In this work, we use a sequential constraint refinement-based non-linear parameter optimization-based approach for trajectory generation of the robot. The advantages of using an optimization-based approach in such situations are explained in Section 2. The trajectory that this method or any other method develops is then refined to a time-optimal trajectory for the specific configurations that result in complete disinfection of the surface being sprayed. For this, we present a linear programming-based method that determines the amount of time the robot should take between every two consecutive waypoints such that the robot sprays a quantity of disinfectant that is greater than a threshold for each point on the surface. This, in turn, gives us the rates for each degree of freedom such that the appropriate time intervals between successive waypoints are executed. Hence, the main contributions of this study are:

Fig. 2.

A simulation scene in CoppeliaSim [14] of ADAMMS-SD spraying liquid disinfectant in a washroom.

• 1.an algorithm for generating polygon area covering spray paths
• 2.a selection criterion for selection of appropriate spray path for the mobile manipulator
• 3.spline-based problem formulation for constrained optimization and a sequential successive refinement based method for solving the highly non-linear optimization problem for robot motion planning
• 4.a linear programming based methodology for determining the time intervals between consecutive waypoints that ensures enough disinfectant dosage for each point on the surface

We have tested our disinfection framework on five distinct test cases in simulation and one experimental demonstration.

2. Related work

Manipulators and mobile manipulators have been used for a variety of purposes in literature [15], [16], [17], [18], [19], [20], [21], [22], [23], [24]. In this paper, we have studied several methodologies for developing the spray disinfection system. We study the literature for the areas of spray modeling for robotic liquid spraying, area coverage with robotic manipulator and constrained trajectory generation for mobile manipulators.

2.1. Spray and spray gun trajectory modeling

Spray modeling has been studied in the literature in regards to robotic spray painting. The common theme in literature is to use a specific spray deposition model on the specific surface and plan paths and velocities such that there is uniform deposition along the path that covers the surface. Bo et al. [25] presented a fast planning framework for robotic spray painting on planar, cylindrical, conical, and spherical surfaces. They considered an elliptical double-beta static coating growth rate model as a reference model for the spray trajectory generation. Based on the thickness requirements, they determined the spray gun velocity and stroke distance. Arikan and Balkan [26] modeled the single painting stroke paint thickness distribution as a beta distribution that varies as the paint gun moves along a line. A trajectory planning tool for automotive painting was developed by Conner et al. [27]. They present an analytic deposition model for electrostatic rotating bell atomizers where the deposition thickness profile is taken to be a 1D Gaussian revolved about the $z$-axis to make it into an asymmetric volcano model. This deposition model is used to determine the velocities of the spray gun as it travels over objects that have changing surface curvatures so as to have uniform spray coating. A similar deposition model is presented by Chen et al. [28], where they consider a rectangular model for spatial paths for spray painting. Chen and Zhao [29] developed a deposition rate function on a plane according to experimental data, which they then used for multi-objective constraint optimization to determine the tool trajectory based on the requirement that the deposition is uniform. In the deposition model by Duncan et al. [30] that is used by Kout and Muller [31], the particles fly on straight rays sharing a starting point inside the gun, and the deposited amount is taken to be the fraction absorbed by the surface like the refraction of light. They use this model to find a finite set of spray gun configurations that cover the surface and minimize the error between the target coating and the coating achieved by those configurations. Gleeson et al. [32] presented an approach for spray paint optimization that can minimize paint thickness deviations from a target thickness by fitting a spline function to the experimental data to get the applicator footprint profile. This profile is then projected onto the geometry, and the deposition model on curved surfaces is generated. This is used in an optimization routine to get the desired paint thickness. There are several such methods in literature well documented in a survey paper by Chen et al. [33]. The paper discusses deposition and spray models as well as CFD-based methods to generate deposition layers. A Gaussian model of the deposition rate of the spray during the cold spray process was discussed by Zhang et al. [34]. We have incorporated this model for our disinfectant spray since the spray can nozzle we used for our method showed Gaussian-like distribution and determining the parameters of the distribution was relatively straightforward as compared to other spray deposition mode. We have explained the determination of the parameters in Section 9.

As seen in literature, there are several methods for modeling spray painting. However, the problem of spray disinfection is inherently different from the problem of robotic spray painting. For spray painting, the requirement is that throughout the surface, the paint thickness must be uniform. Any deflection from this requirement can be easily visually spotted. Hence, for any commercial application for spray painting, there is a need for rigorous modeling of the spray deposition. It terms of the actual nozzle motion, the nozzle tip does not have much flexibility on the orientation as compared to the normal to the surface due to the uniformity requirements. On the other hand, spray disinfectant deposition requires that the entire surface be covered in at least a threshold amount of disinfectant. This may result in certain areas of the surface receiving just enough dosage, whereas other areas may receive a significantly large dosage. Even though this may not be desired, such a situation is acceptable as the disinfectant in most cases evaporates. This relaxes the constraints on the nozzle orientation with respect to the normal to the surface. Moreover, since there is no aesthetics when spraying disinfectant, uncertainties in the surface can be tolerated as compared to spray painting.

2.2. Area coverage path planning

The problem we are trying to solve involves spraying the disinfection on the entire surface of the point cloud, which is a coverage path planning problem for the nozzle attached to the end-effector of the robot. Such problems have been studied extensively in literature in the survey paper by Galceran and Carreras [35]. Area coverage is required for several robotic applications like lawn mowing [36], agricultural field plowing [37], bush trimming [38], spray painting [39], CNC machining [40], cleaning [41], finishing [42], inspection [43] etc. These problems involve coverage planning for mobile robots however, we are interested in solving the coverage path planning for the end-effector of a manipulator. One of the effective methods for solving the coverage path planning problem is to formulate it as a traveling salesman problem (TSP) [44]. Kaljaca et al. [38] formulated the bush trimming problem with a manipulator mounted with a trimming tool at the end-effector as a TSP by using the mesh of the desired shape of the bush. Each triangle on the mesh was considered as a vertex for the TSP. Such an approach is effective only when there are not a large number of vertices. A framework for 3D surface coverage by redundant manipulator was implemented by Hess et al. [45]. Here different inverse kinematic solutions for the robot are treated as individual nodes in a graph which is modeled as a generalized traveling salesman problem (GTSP). GTSP is where the nodes of a graph are subdivided into clusters, and at least one node in each cluster needs to be visited. This method can work for mobile manipulators however, the nonholonomic nature of the mobile base can make it challenging to find feasible edges for the graph. When a point cloud with thousands of points is available, like in our case, solving TSP directly on it is infeasible. Hence, it is necessary to translate the point cloud to more tractable geometry. One of the ways of doing this is to project the point cloud to a plane and extract a polygon to spray on. Leidner et al. [46] presented a method to perform cleaning and wiping chores with a redundant robot Rollin’ Justin. The medium of the wiping task is described by a particle distribution, whereas the motion plan is generated, splitting the task into a high-level planning module and a specific control for the required cleaning action. The objective was to optimize the Cartesian path length. Paus et al. [47] present a combined approach for robot placement and coverage planning for mobile manipulators. The approach takes constraints like collision and stability into account to determine appropriate base placements and solve the TSP for the end-effector. Yang et al. [48] solve the problem of non-revisiting coverage task with minimal discontinuities for non-redundant manipulators. In our method, the mobile base and the manipulator move in a coordinated fashion to cover the entire area. For industrial manipulator, coverage path planning for determining eddy currents in aeronautical parts is discussed by Olivieri et al. [49] using a zig-zag pattern. Glorieux et al. [50] implemented a non-random targeted viewpoint sampling strategy for coverage planning to cover the entire area for camera-based inspection of large parts.

Coverage planning is one of the important aspects in CNC machining that has been optimized by Yao and Gupta [40]. In CNC, the most common cutting paths are either zig-zag or inward spiral. These paths provide advantages since they are easy to generate however, they may not be the shortest paths. The shortest path in such cases can be a combination of zig-zag and spiral paths. They present an approach for generating such combinations using a search-based method that chooses different combinations of the edges of the polygon and eliminating such combinations based on certain pruning heuristics. For generating paths for spraying on the surface, similar approaches can be used. The difference is that the cutting tool must remain perpendicular to the surface, whereas the spray can be at an angle to the normal to the surface. Kalburgi et al. [51] use a spanning tree-based coverage path planning algorithm based on approximate cellular decomposition for CNC machine. The cutter tool needs to pass through all the region that is required to be removed without any gaps.

Typically, humans either spray on a surface in a zig-zag pattern or in a spiral pattern to cover the entire surface (more on this has been discussed in Section 9). Inspired by this, we want to find paths for the mobile manipulator that is a combination of zig-zag and spiral paths in order to generate the optimal path for the particular approach. In our work, we implement an approach based on a branch and bound depth-first tree search-based approach for area coverage of a 2D polygon generated by projection of the given point cloud of the surface. This method gives us the flexibility to choose one of the several paths it generates based on the ease of the robot the follow the path when projected on the 3D point cloud. This method is fundamentally different from the TSP-based approaches presented in literature as it is more systematic and produces paths based on desired heuristics.

2.3. Constrained trajectory generation

The area coverage paths generated need to be followed by the robot, which requires the use of constraint motion planning algorithms. Researchers have explored sampling-based approaches to generate constrained point-to-point trajectories for high-DOF robotic systems like manipulators and mobile manipulators [52], [53], [54], [55], [56], [57], [58] as well as mobile robots [59], [60]. Dalibard et al. [61] proposed an approach where task constrained Jacobian-based local motions were generated to connect nodes in the bi-directional RRTs for generating whole-body motions in humanoids. Berenson et al. [62], [63], [64] presented sampling-based point-to-point planners to generate robot trajectories that satisfy certain kind of constraints. The sampling for new nodes and possible targets was done by projecting random samples on constraint manifolds in configuration space. Hence, all the possible nodes of the trees satisfy the constraints of the task. Task-constrained mobile manipulation planning has been done by Burget et al. [65], which is an extension of Informed-RRT* [66]. Tangent Space-RRT [67] and Tangent Bundle-RRT [68] are another class of methods for constrained sampling-based motion planning. Here lazy projection is implemented where samples are generated in the tangent space of the constrain manifold. They are projected on the manifold only when the tangent space approximation of the manifold is worse than a certain threshold to save computation time. In Atlas-RRT [69], [70], this is further extended for precomputing the tangent space approximations of the entire manifold at several points. Since there is no further need for projection on the manifold during the entire time of planning, this further saves computation time. Oriolo et al. [71] developed a motion planner for mobile manipulators to follow given end-effector paths. Padri et al. [72] presented a sampling-based approach for task constraint motion planning for mobile manipulators with nonholonomic constraints. They sample points on the point cloud, which is the constraint manifold in the workspace, and use the null-space of the constrained Jacobian to integrate the nonholonomic constraints, similar to the work done by Thakar et al. [73]. One of the disadvantages of using sampling-based methods for the constraints like the ones presented in this work is that projecting on the constraint manifold can be challenging since it can be narrow owing to the strict constraints on the pose of the end-effector. Also, in several cases, the length of the path can be high, which can result in the singular configurations that may not be projected on the manifold.

Huang et al. [74] used B-Spline representation of the joints to achieve various optimization objectives. Shi et al. [75] used a NURBS (Non-uniform rational B-spline) representation of the joints for similar objectives. A genetic algorithm is employed to minimize cost functions with joint limits in both methods. Menasri et al. [76], discretized the trajectory and, at each step, performed a search for a new position of the end effector in the workspace to reach the final position. Because of the redundancy of the robot, any end-effector position can be achieved by infinitely many robot configurations. This enables them to find the best configuration that allows for avoiding obstacles and the singularities of the robot. The genetic algorithm-based approaches can be computationally expensive and may fail to find a feasible trajectory for complex motion generation problems like for mobile manipulators system since many of the constraints are conflicting in nature.

Constrained Quadratic Programming (CQP) [77], Quadratic Programming (QP) [78] and non-linear programming [79], [80], have been used to generate a time-optimal trajectory for robots like mobile manipulators by minimizing pose error which is approximated using the Jacobian. Stilman et al. [81], [82] proposed an approach for high degree of freedom robot motion generation by combining sampling-based and Jacobian-based methods. Shankar et al. [83] presented a QP-based approach to solving local motion planning for whole-body manipulation for mobile manipulators. Their formulation can take collision avoidance as constraints of QP or as part of the objective function. Escande et al. [84] presented an approach for trajectory planning of humanoids using hierarchical quadratic programming. The main idea of their approach is to solve the QP for one constraint at a time. The solutions for the latter constraints are found in the null space of the prior constraints. Giftthaler et al. [85] approached the trajectory planning for mobile manipulators using sequential linear–quadratic optimal control. It is often challenging to represent many constraints (e.g., mesh-to-mesh collision) in a quadratic format. Moreover, Jacobin-control-based approaches may not converge without proper initialization [86]. Also, Jacobian approximations may be accurate enough in certain areas of the configuration space and may result in trajectories that do not satisfy the constraints adequately.

Toussaint et al. [87] discussed the use of optimization for robotic applications. Kieffer et al. [88] addressed the path and trajectory generation in the robot configuration space from a given end-effector path in the robot workspace. They solved the path-finding problem in configuration space by representing each joint motion as a parametric curve and then converted it into the trajectory. Martin et al. [89] considered a parametric representation of each joint motion using B-Splines and formulated the minimum-effort trajectory generation problem as an SQP. Their objective was to minimize torque over the course of the trajectory. The work presented a recursive algorithm for computing the objective function and the gradients of objective and constraint functions. Halladay et al. [90], [91] presented an approach for generating configuration space trajectories for a given workspace path in the presence of obstacles. They use different types of distance metrics to measure the difference between the desired path and the generated path after path constrained optimization. The performance of optimization-based approaches with parametric representation for trajectory generation is highly dependent on the initial seed. Generating the seed trajectories or seed parameters for a given problem can be challenging. Finding an appropriate parametric representation and a good optimization seed can significantly accelerate the optimization process as demonstrated by Kabir et al. [92], [93], [94]. Here an initial seed is determined by computing inverse kinematic (IK) solutions at a high resolution. Multiple B-splines are used to represent different areas of the workspace paths using the path curvatures. The control points of the B-splines are used as the optimization variables, with the objective function being the trajectory execution time. The collision and several other constraints are applied. A successive refinement-based method is used to solve the highly non-linear and non-convex optimization problem. This ensures that multiple constraints do not contradict each other resulting in quick computation of the solution as compared to single-stage optimization where all constraints are applied together. This method works well when the workspace path is complex as it can break down the path into multiple simpler segments when the length of the spray path is high.

3. Problem description

3.1. Preliminaries

Let the workspace of the robot be denoted as $\mathcal{W}\subset {\mathbb{R}}^3$. Let $\mathbf{q}=\{x,y,\varphi ,{\theta }_1,{\theta }_2,\dots ,{\theta }_n\}$ be the configuration of mobile manipulator with a manipulator with $n$ joints. $(x,y,\varphi )$ is the pose of the mobile base, i.e the location and the orientation. The maximum forward velocity of the mobile base is $V_{max}$, and its maximum turning velocity is ${\omega }_{max}$. The maximum joint velocity of each joint is ${\dot{\theta }}_{max}$. Let the geometry of the mobile manipulator at a configuration $\mathbf{q}$ be represented as a set of rigid bodies $\mathcal{M}\left(\mathbf{q}\right)\subset \mathcal{W}$. Let $\mathcal{O}\subset \mathcal{W}$ be the set of workspace obstacles. Let $\mathcal{C}$ be the configuration space of the mobile manipulator containing all the valid configurations of the robot. The set of joint configurations that lead to collision is denoted by ${\mathcal{C}}_{obs}=\{\mathbf{q}\in \mathcal{C}:\mathcal{M}\left(\mathbf{q}\right)\cap \mathcal{O}\ne 0̸\}$. The set of joint configurations that are collision-free is ${\mathcal{C}}_{free}=\{\mathbf{q}\in \mathcal{C}\setminus {\mathcal{C}}_{obs}\}$. Let $\mathbf{T}$ be the homogeneous transformation matrix representing the pose of the end-effector (EE) in $\mathcal{W}$. $\mathbf{T}\subset SE\left(3\right)$ (Special Euclidean Group). For a given joint configuration $\mathbf{q}$, we can find $\mathbf{T}$ in the world frame of reference by applying Forward Kinematics (FK). We use the dot notation to extract quantities (e.g. $\mathbf{T}.p$ denotes the position component and $\mathbf{T}.R$ the rotation matrix). $\mathbf{q}.q_b$ denotes the pose of the mobile base and $\mathbf{q}.q_m$ denotes the $n$-D configuration of the manipulator.

The nozzle and a depth camera are attached to the end-effector of the robot, and the corresponding homogeneous transformation matrices are ${\mathbf{T}}_N$ and ${\mathbf{T}}_C$. ${\mathbf{T}}_N$ and ${\mathbf{T}}_C$ can be determined using FK and their transformation matrices from the end-effector. The robot captures the point cloud of the surface to disinfect with the depth camera in the camera frame of reference, which is then transformed to the world frame of reference using ${\mathbf{T}}_C$. We use the method of continuous point cloud capturing method described in [95] to capture a point cloud (details in Section 5), where the robot moves the end-effector pointing the depth camera at the surface to be disinfected to record the point cloud of the desired surface. This point cloud may contain areas that do not need disinfection, like the floor or any walls. Hence, a human operator then extracts the section of the point cloud for the surface that needs to be disinfected. We denote this extracted point cloud in the world frame of reference as ${\mathcal{P}}_W$. The point cloud ${\mathcal{P}}_W$ contains $N_p$ number of points.

3.2. Problem statement

A mobile manipulator is given with $3+n$ degrees of freedom and the corresponding maximum forward, turning, and joint velocities. A nozzle attached at the end-effector (Fig. 1) of the mobile manipulator sprays disinfectant liquid with a spray model $\mathcal{S}(\alpha ,D_n,D_f,d_o,\sigma )$ (Fig. 3, described in detail in Section 4). Also, given is a continuous point cloud ${\mathcal{P}}_W$ with $N_p$ number of points of the surface to be disinfected. The objective is to determine a collision-free, time-optimal configuration space trajectory of the mobile manipulator such that each of the $N_p$ points of the point cloud receives at least a minimum threshold dosage ($H_o$) of disinfectant as the mobile manipulator moves.

Fig. 3.

The spray cone showing the spray angle $\alpha$ and the distances $D_f$ and $D_n$ that form a frustum of the cone. The rate of deposition is modeled as a Gaussian distribution for the circular cross-section at a distance $D_n$.

4. The disinfectant spray model

The spray from the nozzle is assumed to be conical with a cone angle that we call the spray angle ($\alpha$). The effectiveness of the spray decreases with the distance from the nozzle tip. We define a distance $D_f$ to be the cut-off distance from the nozzle tip, after which the effectiveness of the spray is minimal. We also define a distance $D_n$ to be a cut-off distance from the nozzle tip, such that if the distance of a surface from the nozzle tip is less than $D_n$, the spray liquid starts flowing due to the high concentration. This may not be desirable when spraying sensitive equipment or when there is no option to wipe it with a cloth. Hence, the spray from the nozzle is considered effective if the surface being sprayed is within the frustum of the cone formed by the cone angle $\alpha$, and the distances $D_f$ and $D_n$ from the nozzle tip. This frustum of the cone is shown in Fig. 3. The rate of deposition of the spray liquid on a surface perpendicular to the direction of the spray at a distance $D_n$ can be modeled as a Gaussian distribution with a standard deviation $\sigma$ and zero mean. [27], [28], [34], [96] The deposition rate at the center of the circular cross-section is $d_o$. The parameters $\sigma$, $d_o$, $D_n$ and $D_f$ can be determined experimentally. This spray model is denoted by $\mathcal{S}(\alpha ,D_n,D_f,d_o,\sigma )$.

The spray model briefly described Fig. 3 gives the liquid disinfectant deposition rate (or the disinfectant height build-up rate) if a surface was to intersect the spray cone at a distance $D_n$ perpendicular to the nozzle axis. As mentioned, the frustum of the cone formed between $D_n$ and $D_f$ is where spray disinfection is actually effective. Hence, it is necessary to determine the rates of liquid disinfectant deposition at any point within the volume of frustum of the cone between $D_n$ and $D_f$.

The liquid disinfectant deposition rate at a distance $D_n$ as shown in Fig. 3 is given as follows [34]:

$$d_R(D_n,r)=d_o{e}^{-\frac{1}{2}\frac{r^2}{{\sigma }^2}}$$ (1)

where, $r$ is the radial distance from the spray cone axis. Based on the continuity equation, we can derive the deposition rate $d_R(z,r)$ where $z\in [D_n,D_f]$ and $r\in [D_{n}tan\alpha ,D_{f}tan\alpha ]$. The deposition rate at a distance $z$ from the nozzle tip is given as follows (derivation in Appendix):

$$d_R(z,r)=d_o\frac{D_n^2}{z^2}{e}^{-\frac{1}{2}\frac{r^2}{{\sigma }^2}\frac{D_n^2}{z^2}}$$ (2)

Using the Eq. (2), we can determine the deposition rate at any point within the frustum of the spray cone. Hence, for any point belonging to the point cloud ${\mathcal{P}}_W$, we can determine the deposition rate normal to the local plane at that point. As shown in Fig. 4, the distance of point $P$ on the point cloud ${\mathcal{P}}_W$ from the nozzle tip is $D_P$, and the angle that the vector from the nozzle tip to the point $P$ makes with the spray axis is $\beta$. This gives $z=D_{P}cos\beta$ and $r=D_{P}sin\beta$. The normal to the point cloud at point $P$ is ${\hat{n}}_P$. The angle that ${\hat{n}}_p$ makes with the vector from point $P$ to the nozzle tip is $\gamma$. Hence, the rate of deposition at point $P$ normal to the surface of the point cloud can be approximated by the following equation, assuming that the disinfection fluid deposited normal to the surface does not flow after being sprayed.

$$P.d_R=d_o\frac{D_n^2}{D_P^2}{sec}^2\beta e^{-\frac{1}{2}\frac{D_n^2{tan}^2\beta }{{\sigma }^2}}cos\gamma$$ (3)

Fig. 4.

A point ${\mathcal{P}}_W$ is sprayed on, and a point $P$ with its coordinates in the world coordinates are $(x_w,y_w,z_w)$.

We represent the deposition rate for point $P$ using dot notation as $P.d_R$. The direction of the normal at point $P$, ${\hat{n}}_p$ can be determined by planar segmentation of the point cloud $P_W$ at point $P$. Hence, using Eq. (3), we can evaluate the deposition at any point of the point cloud as long as it lies inside the frustum of the spray cone. It should be noted that the Eq. (3) is an approximation and can change for different type of disinfection fluid. Our method is agnostic to the spray model chosen, however, we show the results using the deposition rate modeled as Eq. (3). Also, we have neglected the impact of gravity in the derivation of Eq. (2).

5. Overview of approach

First the robot captures a point cloud and ensure that the uncertainties in the point cloud are minimized. To this end, we use the method described in [95] to capture continuous point clouds. The method generates a set of camera locations around the part that allow coverage of desired surfaces. The locations are constrained to keep the camera frame within a distance from the surface. The distance constraint minimizes the uncertainty in the captured pointcloud. A robot trajectory is generated in the joint space satisfying the acceptable distance constraint. Pointclouds are continuously captured as the robot executes the trajectory. A pointcloud merging algorithm then produces the output constructed pointcloud representing the part within a sub-millimeter average error (0.6–0.9 mm). Thereafter, a human operator precisely extracts the area of the point cloud that needs to be disinfected. This extracted point cloud is denoted by ${\mathcal{P}}_W$.

In order to thoroughly disinfect a surface given by the point cloud ${\mathcal{P}}_W$, the mobile manipulator has to move the nozzle attached to its end-effector such that each point of the continuous point cloud passes within the frustum of the spray cone at some point during the robot motion. Hence, it is necessary to determine a path consisting of waypoints with position and orientation for the nozzle tip such that as this path is followed in sequence, points of the point cloud pass through and out of the frustum of the spray cone. By the end of the path, all points should have passed through the frustum i.e. the frustum should have passed through all the points. This path is such that each waypoint of this path $\in SE\left(3\right)$. And to determine the workspace path of the nozzle tip, a sequence of points of the point cloud must be determined, which will be called the spray path henceforth. The framework and an overview of the approach for determining the mobile manipulator trajectory for spray disinfection is shown in Fig. 5.

Fig. 5.

The framework for robotic disinfection describing the sequence of operations to determine the final trajectory of the mobile manipulator for disinfection of the surface represented by the point cloud.

The first step is to determine the sequence of points on the point cloud, which we call the spray path. A spray path waypoint is a point where the spray cone axis intersects the point cloud. This point should be on or inside the effective frustum of the spray cone. There can be several spray paths that can be effective in disinfecting the entire point cloud; we select one path based on a selection criterion. This selection criterion is based on the reachability of the mobile manipulator and is explained in detail in Section 6.6. Once we have chosen a spray path, based on the constraints of the spray cone that each point of the point cloud must lie inside the frustum, and the robot motion constraints, we determine the configurations of the mobile manipulator for each waypoint on the spray path using sequential constraint-based optimization. This is explained in details in Section 7. Once we have the mobile manipulator and hence the nozzle path, we determine the velocities of each of the mobile manipulator configurations so that each point of the point cloud receives enough dosage of the disinfectant to sterilize it. This is done by determining how much time the robot should take between every two consecutive waypoints. Details of this are explained in Section 8.

6. Spray path generation

As mentioned earlier, a spray path is a sequence of points of the point cloud, where each point is the intersection of the axis of the spray cone and the point cloud. This sequence should be such that each point of the point cloud should at least once be inside the effective frustum of the spray cone. This can be formulated as an area coverage problem, where the spray should cover the entire area of the surface.

6.1. Planar polygon generation

To solve this problem, the first step is to get a planar version of the point cloud. We can get the planar version by either projecting the point cloud on a plane or unwrapping the point cloud into a plane [97]. In this paper, we project the point cloud onto a plane such that we get its area projection which may or may not be its maximum area projection. The projection operation is shown in Fig. 6. It starts with uniformly sampling a large number of points on the point cloud, and at each of those points, we determine the normal direction to the surface locally using planar segmentation. We then align all the unit normals to one of the sides of the point cloud, i.e., above or below, or upwards or downwards. Then we take a vector sum of the normals to determine the dominant normal direction as shown in Fig. 6(b). We then project each point of the point cloud to the plane perpendicular to the dominant normal, as shown. We keep a record for each 3D point and the corresponding 2D point on the plane which we use to reproject the generated path on the 3D point cloud in Section 6.5 We then determine the boundary points of the planar projected set of points.

Fig. 6.

(a) The point cloud to be disinfected is shown (b) The projection of the point cloud on a plane perpendicular to the dominant normal of the point cloud is shown along with the polygon containing all the projected points.

The goal is to form a bounding polygon with these boundary points. This polygon cannot be a convex hull of all the points due to the concavity that may result from the projection operation. Moreover, we cannot form a polygon with all the boundary points as vertices since that will result in a large number of very small polygon edges, which is not desirable. Hence we must choose a subset of those boundary points as the vertices of a polygon. Here, $(P_1,P_2,\dots ,P_m)$ represent the $m$ boundary points in a sequence. The point $P_1$ is such that the angle formed by the two segments joining $P_1$ to its two adjacent points (for $P_1$ these points are $P_2$ and $P_m$) is the smallest amongst all $m$ points. We choose $P_1$ to be one of the vertices of the desired polygon. From $P_1$ we go to $P_2$ and sequentially to $P_m$, and find the first point $P_i$ such that the distance between $P_1$ and $P_i$ is greater than a length $l_o$. If all points between $P_1$ and $P_i$ lie within a lateral offset distance of ${\delta }_o$ from the line joining $P_1$ and $P_i$, then the next vertex of the polygon adjacent to $P_1$ is $P_i$. If such a $P_i$ does not exist, we choose the furthest point from $P_1$ such that all points between that point and $P_1$ are within ${\delta }_o$ distance laterally from the line joining that point and $P_1$, as the next vertex of the polygon. This point will be at a distance $<l_o$ from $P_1$. Once the adjacent vertex to $P_1$ is decided (call that $P_b$), we repeat this process to determine the adjacent vertex to $P_b$. This process gives a polygon, but the maximum length of an edge would be about $l_o$. Hence, even a possible straight edge much longer than $l_o$ will be divided into multiple segments. We remove the intermediary points of such a long straight edge by checking for the angle of the polygon at each point. If the angle is close to $180°-{\alpha }_o$, we remove the polygon vertex. ${\alpha }_o$ is a small angle that can be tuned. Similarly, the parameters $l_o$, $l_o^{\prime }$ and ${\delta }_o$ can be chosen appropriately. This methodology gives us a polygon, an example of which is shown in Fig. 6(b). This polygon is denoted as ${\mathcal{G}}_o$.

6.2. The spray radius

The algorithm for computation of the spray paths on the polygon ${\mathcal{G}}_o$ generated by the methodology described in Section 6.1 is explained in Algo. 1. Before going into the detail of the spray path generation algorithm, it is important to discuss the spray radius or radii that will be used for the algorithm. As can be seen in Fig. 4, when the point cloud is sprayed, the spray radius is at least $R_n$. If the spray cone axis is not orthogonal to the point cloud, the projection will be an ellipse. The minimum length of the ellipse axes would be $R_n$. Hence, we assume that we spray with a radius of $R_n$, which is the minimum radius or axis length of the ellipse. As can be seen in Fig. 7, when the point cloud is projected on the plane, the radius of the spray circle also decreases on the projection plane. We call the radius of spray on the plane to be the spray radius $R_s$. The spray radii vary for different regions of the point cloud, as can be seen in Fig. 7. We sample several points on the point cloud and compute the spray radius on the plane on those points and take the minimum of those radii to be the spray radius $R_s$. However, if the minimum of those radii is very small as compared to $R_n$, it will result in the spray segments very close to each other at a place where the projected radius is not as small. Hence in such a case, we have different $R_s$ for different regions of the projected point cloud or polygon. For the purpose of explaining the algorithm, we assume there is a single $R_s$ for the entire projected polygon.

Fig. 7.

The minimum radius of the spray circle on the point cloud is $R_n$. However, when the point cloud is projected on a planar surface, the radius of the spray circle also needs to be projected. There are several projected radii, which are called ${R_s}_i$ for different angles of projection of the plane.

6.3. The spray path generation algorithm

The spray path algorithm is shown in detail in Algo. 1 and illustrated in Fig. 9. We are given the projected polygon ${\mathcal{G}}_o$ of the point cloud ${\mathcal{P}}_W$, and the spray radius $R_s$ and the goal is to determine a sequence of segments on the polygon ${\mathcal{G}}_o$ such that the entire polygon and the point cloud are covered with disinfectant. We call this sequence of segments the spray path. Once we have the spray path on the polygon, we will project those on the point cloud and generate mobile manipulator end-effector waypoints for its motion planning. The first step for the generation of the spray paths is to generate a polygon ${\mathcal{G}}_{\text{off}}$ that is offset inside the given polygon ${\mathcal{G}}_o$ by a distance equal to the spray radius $R_s$. If we spray along the segments of the polygon ${\mathcal{G}}_{\text{off}}$, the boundary of the polygon ${\mathcal{G}}_o$ will be sprayed. A simple spray segment is shown in Fig. 8.

Fig. 9.

The spray algorithm. The red area has not yet been disinfected; the blue area is the disinfected area. The polygon with a blue boundary is the offset polygon of the polygon to be sprayed. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 8.

A single spray segment. We assume that when the spray starts, the spray starts from the boundary of the polygon to avoid a corner to be left unsprayed. We will discuss in Section 9 how we modify the spray paths to spray the unsprayed corners.

We get the offset polygon and add all the individual segments of the offset polygon to a stack (lines 6–7 of Algo. 1, column 1. of Fig. 9). We then choose the latest segment in the stack (line 10) and expand it (line 11). Expanding a segment is basically spraying the segment and getting the polygon remaining after spraying that line segment. This is shown in Fig. 9, column 2. The polygon sprayed is in blue, which is essentially a trapezium with a height equal to twice the spray radius $R_s$. It can be pointed out that due to the circle, the corner of the polygon (the bottom right-hand corner in black in Fig. 8) will not be sprayed. However, since the spray radius is the minimum possible radius, it is expected that the corner will be sprayed when the actual larger circle (or ellipse) of the frustum of the spray cone is used. In the case that the corner angle is very small, a large area of the corner would remain unsprayed. In such a case, spraying this segment will automatically be eliminated, as will be explained in the next steps of the algorithm. Also, when the segment is sprayed, the checked area in Fig. 8 will remain unsprayed. However, as we move to the next segment, this part will be sprayed. Hence we assume that the checked portion is also sprayed, and hence the sprayed polygon in blue is a trapezium. Also, it must be noted that for the very first segment being expanded, we start the spray from the edge of the polygon as shown by the red dotted line in Fig. 8. As can be seen in Fig. 9, this trapezium is shown in column 2. for $s_1$ and $s_7$ since those segments are convex segments of the polygon. A convex segment is where all the vertices of the polygon lie on the same side of the polygon. On the other hand, segment $s_2$ is a non-convex segment where at least one vertex lies on the opposite side of the segment as compared to others. For a non-convex segment, the sprayed polygon is terminated at the end of the segment, as can be seen in the second polygon for column 2. of Fig. 9.

The remaining polygon after spraying a segment is the next polygon ${\mathcal{G}}_{\text{next}}$ (line 11). We then find the offset polygon of ${\mathcal{G}}_{\text{next}}$ as shown in column 3 (line 15). Then we go on to determine the next segments, which would be the children of the current segment. For a convex segment being expanded, there are three children, the two adjacent segments (denoted by $s_{1b}\&s_{1c}$ and $s_{7b}\&s_{7c}$) and one parallel segment ($s_{1a}$ and $s_{7a}$). For a non-convex segment, there are only the two adjacent segments as shown for $s_2$ as $s_{2a}$ and $s_{2b}$. All the children segments are derived from the offset polygon of the next polygon ${\mathcal{G}}_{\text{next}}$. These segments are generated and pushed onto the stack (lines 46–50). The next polygon ${\mathcal{G}}_{\text{next}}$ then becomes the current polygon (line 51), and we continue the process by popping the most recent addition to the stack till all the segments have been expanded. Hence, this is a depth-first search with a branching factor of 3. Here the root node is a virtual node with the number of segments of the first offset polygon as the number of children.

The condition that the leaf node has reached for a branch is that the offset polygon of the current polygon does not exist. This is possible only when the current polygon is small, as shown in Fig. 10. Once the leaf node is generated, the path for that branch of the tree from the root to the leaf is generated by connecting the spray segments sequentially with connecting segments as shown in Fig. 9 column 3. The final spray segment is generated for the polygon with no offset polygon by drawing a line through the end of the previous segment in the direction of the principal axis of the polygon. The dashed line in the polygon with no offset polygon shown in Fig. 10 is such a final segment. This terminates a single spray path for that particular branch of the search tree. Next, we go into the details of the termination conditions for generating the spray path and the branch pruning heuristics used to accelerate the optimal path generation.

Fig. 10.

The remaining polygon which has no offset polygon.

The first branch pruning condition is that if the next polygon ${\mathcal{G}}_{\text{next}}$ (line 11) and its offset polygon ${\mathcal{G}}_{\text{off}}$ (line 15) are not continuous, then we do not expand the current segment since. An example of discontinuous offset polygon is shown in Fig. 11. The discontinuity will require the spray path to join the two discontinuous regions, making the spray path longer. We prune the branches of the search tree in order to avoid such a situation. The pruning of such polygons happens in lines 12–14 and 17–19 of Algo. 1.

Fig. 11.

Due to a narrow section of the polygon, the offset polygon is discontinuous.

If the offset polygon of the next polygon does exist and is continuous, then we have the next pruning heuristic. For this pruning heuristic, we measure the area sprayed by the current incomplete path as compared to the polygon the spray is supposed to spray on. For the purpose of the algorithm, for example, in Fig. 9 column 2, after we spray on the red segment, we consider the blue trapezium as the sprayed polygon. However, as can be seen in Fig. 8, the actual sprayed area will be the blue plus the checked area, and the bottom right corner (black area) will remain unsprayed. This unsprayed area can be significant when the angles of the polygon are small. Hence, we need to determine the difference between the polygon we expect to spray (blue) and the area it actually sprays. Although we mentioned earlier that the larger spray radius would result in the corner getting sprayed on, that is not always true. Hence, we prune branches of the search tree that result in paths that have large unsprayed regions. We measure this unsprayed area by having a dense grid of points in the polygon and count the points inside the sprayed polygon vs. the actual sprayed area. The difference between the two numbers is the number of grid points not sprayed and can be used as a measure of the area not sprayed. In Algo. 1, we generate the incomplete spray path in line 20 and get the number of grid points this incomplete path actually sprayed on in line 21. We determine the number of grid points that are inside the trapeziums of every path segment and take their difference to get the number of grid points not sprayed on by the incomplete path. This number of grid points not sprayed resembles the measure of the area that the incomplete path should have sprayed but did not. If this number of grid points not sprayed is greater than the least number of points not sprayed documented till now, we prune this branch (line 24–26). It must be noted that the measure of the best path is not the length of the path but the smallest number of grid points it ends up not spraying on.

If the offset polygon of the next polygon does not exist (like in Fig. 10), this means that we have reached the leaf node. We compute the final path as described earlier. This is shown in line 28 of Algo. 1. We similarly get the grid points the final path actually sprays on and the grid points it should spray on in lines 29 and 30, respectively. If the number of grid points the final spray path does not spray on is less than the current least number of grid points not sprayed on, we find the length of this final spray path and make it the best path length till now and record the least number of grid points not sprayed on, in lines (34 and 33 respectively). We store this is as one of the best area covering path. If the number of grid points the final spray path does not spray on is equal to the current least number of grid points not sprayed on, we check if the length of the current path is less than the best path length recorded till now, we make this the best path length and store this path as one of the best area covering path till now (lines 41 and 42). We continue to the next iteration after that in line 45.

The algorithm presented is an any-time algorithm, where it keeps on finding shorted paths that cover more and more of the area. We can terminate the algorithm once it has found a path. If the number of grid points not sprayed is zero and the lengths of the progressive paths computed do not change by more than a certain percentage, we terminate the algorithm. The resulting paths can be zig-zag, spiral, or a combination of the two, as shown in column 5 of Fig. 9. We record all the paths that have the least amount of grid points not sprayed on since a longer path may be better than a shorter one to spray on with a mobile manipulator.

6.4. Two zig-zag paths for same spray segments

As shown in Fig. 9 column 4 first polygon, there are two parallel spray segments. There are two ways to connect the two segments, one from either end of the segment. Hence, there can be two ways a number of parallel segments can be connected. The two ways are shown in Fig. 12. When a path is to be determined in lines 20 or 28, in such a scenario, we determine both parallel paths and keep the one or both if they satisfy the conditions of being spraying on equal to or more of the grid points as compared to the previous best spray paths.

Fig. 12.

The two different parallel paths from the same spray segments.

6.5. Re-projection of the spray path

Once the spray path is computed on the 2D polygon, we need to project it back to the 3D point cloud. As mentioned in Section 6.1, we project the points of the 3D point cloud on the 2D surface. As we do so, we keep a record of the projected 2D point corresponding to each 3D point of the point cloud. Once the 2D spray path is generated we determine the points from the 2D projection of the point cloud that are on closest to the line segments of the spray path. Since we have a correspondence between the 3D point cloud points and the 2D projected points, we map those 2D points closest to the spray path segments to the 3D points of the point cloud. This way we project the spray path generated for polygon coverage back to the original point cloud ${\mathcal{P}}_W$.

6.6. Spray path selection criterion

As the spray path generation algorithm terminates, it returns several planar spray paths, all of which cover the area of the polygon. In this section, we explain the selection criterion for the spray path from the multiple paths generated. The mobile manipulator has a nonholonomic mobile base and a robotic arm mounted on it. For every pose, i.e., the position and orientation of the mobile base, the manipulator can reach different locations in the workspace. The capability map [98], [99] of the manipulator shown in Fig. 13. It shows scaled reachability, which is the ease with which each point in the workspace is reachable, i.e., the reachability of the robot for each point in the workspace. The capability map of a manipulator is computed by sampling a large number of its joint configurations and measuring the number of times each 3D voxel in the workspace is reached by the end-effector.

Fig. 13.

The mobile manipulator and the capability map of the manipulator at the corresponding to the pose of the mobile base.

For the endpoints of each segment of the spray path as shown in Fig. 14(b), we generate a pose for the nozzle tip of the robot by moving by a distance $D_n$ from Fig. 3, in the direction normal to the point cloud at that point. We do this for all the segment endpoints (blue points in Fig. 14(b)). The spray path segments are thus transformed into nozzle tip segments. For each spray path, we check if every nozzle tip segment lies inside the capability map, i.e., if any point on the segment has the scaled reachability close to zero. This implies that in order to spray on that segment, the mobile base will have to move. We want to make sure that the mobile base moves as little as possible when the robot is performing disinfection. Hence, we choose whichever spray path has the least number of segments going outside the capability map at the initial configuration of the mobile manipulator as the spray path to execute.

Fig. 14.

(a) The planar path generated by the spray path generation algorithm and (b) corresponding path on the point cloud.

7. Motion planning for mobile manipulator

Once the spray paths on the projected polygon are generated, we move on to the motion planning of the mobile manipulator such that the nozzle at the end-effector of the robot sprays along the spray paths projected back on to the point cloud as shown in Fig. 14. Before we plan the motions, we need to select an appropriate spray path from the multiple paths generated by the algorithm.

7.1. Nozzle pose constraints

For a spray path on the point cloud, like in Fig. 14(b), we create nozzle tip waypoints by choosing points at regular intervals along the path. We make sure that the corner points for the spray path are included in the nozzle tip waypoints. We add a frame at each point with the $z$-axis along the normal direction to the plane at that point. Each waypoint is a 4 × 4 transformation matrix $\in SE\left(3\right)$. We denote this sequence of 4 × 4 transformation matrices along the spray path as nozzle tip waypoints. There are $N_W$ spray path waypoints. The sequence of nozzle tip waypoints represents a workspace curve ${\mathcal{W}}_p\left(s\right)=\{{\mathbf{W}}_p\left(i\right):i=1,2,\dots ,N_W\}s\in [0,1]$. Where $s$ is a spray path length parameter; $s=0$ implies $i=1$ and $s=1$ implies $i=N_W$. One of the nozzle tip waypoints ($\mathbf{T}={\mathbf{W}}_p\left(5\right)$) is shown in Fig. 15. For each of these nozzle tip waypoints, there are constraints on the feasible nozzle configurations. The spray cone axis must intersect the 3D position of the spray path waypoint, $\mathbf{T}.p$. Moreover, $\mathbf{T}.p$ must lie within $D_n$ and $D_f$ from the spray path position, i.e., within the frustum of the spray cone.

Fig. 15.

The spray cone axis should intersect the spray path waypoint inside the frustum of the cone.

When a plane intersects the frustum of the spray cone, the intersection is a circle or an ellipse, as can be seen in Fig. 15. However, if the angle of intersection is skewed, the intersection can be a truncated ellipse or a parabola. This is not desirable since we want to make sure that the entire spray is effective. Hence, it is necessary to make sure that the nozzle tip is at a relative orientation from the spray path waypoint such that the intersection of the frustum of the spray cone and the plane to the point cloud at the waypoint is an ellipse or a circle and not a truncated one. In order to make sure of that, there are limits to the angle between the vector from the waypoint to the nozzle tip and the waypoint $Z$-axis (i.e., the normal to the point cloud at the waypoint).

As mentioned earlier in the section, the spray cone axis intersects the nozzle tip waypoint position. In Fig. 16 the red dotted line is the plane to the point cloud at the nozzle tip waypoint positions and the dotted arrow is the normal to the plane at the point. $D_{z1}$ and $D_{z2}$ are the distances of the waypoint to the nozzle tip. In Fig. 16(a), it can be seen that $D_{z1}$ is such that, $D_n<D_{z1}<D_f$ and angle ${\alpha }_1<{\alpha }_2$. In order to make sure that the intersection ellipse are not truncated, the angle $\beta$ between the normal to the plane and the spray cone axis should be such that $-{\alpha }_1<\beta <{\alpha }_1$. In Fig. 16(b) $D_{z1}$ is such that, $D_n<D_{z1}<D_f$ and angle ${\alpha }_2<{\alpha }_1$, hence here the angle $\beta$ between the normal to the plane and the spray cone axis should be such that $-{\alpha }_2<\beta <{\alpha }_2$. Hence, the constraint on the angle $\beta$ is $-{\alpha }_{\text{lim}}<\beta <{\alpha }_{\text{lim}}$ where ${\alpha }_{\text{lim}}=min({\alpha }_1,{\alpha }_2)$ with ${\alpha }_1=arctan\left(\frac{D_f-D_z}{R_f}\right)$ and ${\alpha }_2=arctan\left(\frac{D_z-D_n}{R_n}\right)$.

Fig. 16.

The limits on the angle the spray waypoint transformation matrix $z$-axis makes with the line joining the waypoint position and the nozzle position. The dotted red line represent the instantaneous plane at the spray point position.

7.2. Robot motion constraints

For each nozzle tip waypoint, the robot configuration $\mathbf{q}$ should be such that the pose of the nozzle ${\mathbf{T}}_N=\text{FK}\left(\mathbf{q}\right)$ satisfy the constraints described till now. Due to the nonholonomic nature of the mobile base motion, there are constraints on the mobile base configurations. This nonholonomic constraint must be satisfied between every two consecutive mobile base configurations $\mathbf{q}.q_b$. Moreover, the constraints for the joint limits, joint velocities, the mobile base forward and turning velocities should be satisfied. And finally, the robot’s motion should be free of self-collision and external collisions. There can be three kinds of collision- robot–robot collision, robot self-collision, and robot–object collision. In this work, we approximated each rigid body as a collection of spheres [100] and considered the signed distance among sphere pairs for scoring collisions. If the distance between two spheres is less than the sum of the radii of the two spheres, those two spheres are in collision. We can use this sphere representation for collision between the manipulator and the mobile base and for external collision with the point cloud.

7.3. Motion planning problem formulation

The goal is to find a sequence $\mathcal{Q}$ of configurations $\mathbf{q}$ such that the nozzle sprays along the spray path and the mobile manipulator moves from the starting nozzle tip waypoint to the final nozzle tip waypoint in the least amount of time. This ensures that the disinfection of the surface represented by the point cloud happens in a time-optimal manner. Moreover, we also want to make sure that the mobile base moves as little as possible so as to mitigate the uncertainties associated with the mobile base localization. With this objective, the motion planning problem is formulated as a non-linear optimization problem. Given the $N_W$ nozzle tip waypoints ${\mathbf{W}}_p$, the robot model and its sphere representation and the constraints due to the spray model $\mathcal{S}$, the objective is to find the configuration space trajectory for the mobile manipulator $\mathcal{Q}$. The problem statement is as follows:

$$\underset{\mathcal{Q}}{\text{minimize}}{w}_1\mathbf{T}+w_2\sum _{i=1}^{N_W-1}\mathit{dist}({\mathbf{q}}_i.q_b,{\mathbf{q}}_{i+1}.q_b)s.t.$$ (4)

$${\text{C}}_{\text{nozzle pose}}\left(\mathbf{q}\right)\le 0$$ (5)

$${\text{C}}_{\text{nonholonomic}}(\mathbf{q},\dot{\mathbf{q}})=0$$ (6)

$${\text{C}}_{\text{collision}}\left(\mathbf{q}\right)\le 0$$ (7)

$${\text{C}}_{\text{joint limits}}\left(\mathbf{q}\right)\le 0,{\text{C}}_{\text{velocity limits}}\left(\dot{\mathbf{q}}\right)\le 0$$ (8)

where, $\mathbf{T}$ is the motion execution time for the robot. We have considered a surrogate function to estimate the execution time. The idea is that we will be moving each joint with maximum displacement between two waypoints at its maximum velocity and regulate the velocities of other joints such that they all complete motion at the same time. Hence, $△t$ is the time the slowest configuration variable (i.e., degree of freedom) takes as the robot goes from the $i$th to the $i+1$st waypoint.

$$\mathbf{T}=\sum _{i=1}^{i=N_W-1}△t_i$$ (9)

$$△t_i=\text{max}(({\mathbf{q}}_{i+1}^{\mathbf{j}}-{\mathbf{q}}_i^{\mathbf{j}})/{\dot{\mathbf{q}}}_{\text{max}})$$ (10)

Here $\mathit{dist}({\mathbf{q}}_i.q_b,{\mathbf{q}}_{i+1}.q_b)$ is the distance covered by the mobile base between two consecutive waypoints. $w_1$ and $w_2$ are the weights corresponding to each objective. The nozzle pose constraints ${\text{C}}_{\text{nozzle pose}}\mathbf{q}$ have been described in Section 7.1. We can represent the nozzle pose constraints as nozzle pose error by converting it into an exponential error function [92].

$$\text{Nozzle Pose Error}=exp\left(w_a(D_n-z_N)\right)+exp\left(w_a(z_N-D_f)\right)+exp\left(w_b\sqrt{x_N^2+y_N^2}\right)+exp\left(w_c(-{\alpha }_{\text{lim}}-\beta )\right)+exp\left(w_c(\beta -{\alpha }_{\text{lim}})\right)$$ (11)

where, $(x_N,y_N,z_N)$ is the position of the nozzle with respect to a spray path waypoint computed using the robot FK and the homogeneous matrix associated with the spray path waypoint. Computation of $\beta$ is as described in Section 7.1. $w_a,w_b,\&w_c$ are the weights associated with each error. Similarly, the constraints due to the nonholonomic constraints of the mobile base can be converted into the nonholonomy error.

$$\text{Nonholonomy Error}=exp(w_h\text{abs}((x_{i+1}-x_i)sin{\varphi }_i-(y_{i+1}-y_i)cos{\varphi }_i))$$ (12)

where $(x_i,y_i,{\varphi }_i)$ and $(x_{i+1},y_{i+1},{\varphi }_{i+1})$ are the mobile base poses ${\mathbf{q}}_{\mathbf{i}}.q_b$ and ${\mathbf{q}}_{\mathbf{i}}+\mathbf{1}.q_b$ for two consecutive robot configurations. Hence we can change the objective function to include the nozzle pose and the nonholonomy errors. The objective function is as follows:

$$w_1\text{Total Time}+w_2\text{Total MB Distance}+\text{Nozzle Pose Error}+\text{Nonholonomy Error}$$ (13)

with the collision, joint limits, and joint velocity limit constraints as is. Here Total Time is $\mathbf{T}$ and Total MB Distance is the distance traveled by the mobile base, $\mathit{dist}({\mathbf{q}}_i.q_b,{\mathbf{q}}_{i+1}.q_b)$ from Eq. (4). We solve this motion planning problem with non-linear optimization with spline representation for the joint configurations.

7.4. Spline representation and seed generation

The trajectory $\mathcal{Q}$ of the robot that needs to be determined is a sequence of robot configurations $\mathbf{q}$ for each of the $N_W$ nozzle tip waypoints. For a mobile manipulator, the number of degrees of freedom is $n+3$, where $n$ is the degrees of freedom of the manipulator. Generating path-constrained trajectories is computationally challenging for such high-DOF robotic systems. This is due to a large number of highly non-linear constraints and a large number of configurations $(N_W\times (n+3))$ that are to be solved for. Hence, we convert the trajectory generation problem into a discrete parameter optimization problem. In this paper, we approximate each of the configuration variable trajectories as a third-order B-Spline. Representation of the $j$th configuration variable, ${\theta }_j$, will be,

$${\theta }_j(s,x^j)=\sum _{i=1}^{N_{\text{cp}}}{R}_{i,k}\left(s\right)x_i^j,s\in [0,1]$$ (14)

$$R_{i,k+1}\left(s\right)=\frac{s-{\gamma }_i}{{\gamma }_{i+k}-{\gamma }_i}{R}_{i,k}\left(s\right)+\frac{{\gamma }_{i+k+1}-s}{{\gamma }_{i+1}}{R}_{i+1,k}\left(s\right)$$ (15)

$$R_{i,1}\left(s\right)=\left\{\begin{array}{cc}1,\hfill & \text{if}{\gamma }_i\le s<{\gamma }_{i+1}\hfill \\ 0,\hfill & \text{otherwise}\hfill \end{array}\right)$$ (16)

In Eqs. (15), (16), ${\gamma }_i$ are the knots. The knots of a B-spline are the points where the piece-wise polynomials of the B-Spline meet. The map between the nozzle tip path length parameter ($s$) and time ($t$) is the following, $s=0$ when $t=t_i$ and $s=1$ when $t=t_f$. $N_{\text{cp}}$ is the number of control points for the one dimensional spline curve representing ${\theta }_j$, $x^j$ is the vector of control points for ${\theta }_j$, and $R_{i,k}\left(s\right)$ are the basis functions parameterized with nozzle tip path-length parameter $s$ of the workspace curve ${\mathcal{W}}_p\left(s\right)$.

In this work, we will represent the trajectories as splines and the trajectory generation problem as a parameter optimization problem. Each spline will represent the motion of one degree of freedom. Given the spline representation, the optimization problem is to find the number of control points ($N_{\text{cp}}$) and the optimal set of control points ($X$) for the configuration variables. $X$ is a vector created by vertically stacking the vectors $x^j,\forall j$. For a mobile manipulator system with $n+3$ degrees of freedom, the length of the vector $X$ will be $(n+3)\times N_{\text{cp}}$. We need to determine the control points $X$ such that the robot configurations satisfy the constraints on all $N_W$ nozzle tip waypoints.

The optimization variable for the problem instance in Eq. (4) is now the control points $X$. For an $X$, we can determine the robot configurations for any $s\in [0,1]$ giving the trajectory of the entire mobile manipulator. If we initiate the optimization routine with a random seed (or initial solution), then we may not be able to find a feasible solution within a reasonable computation time-bound. We may land on an infeasible solution, as well as a highly sub-optimal solution, owing to the numerous local minima in this class of problems. We present a method to generate an approximate initial seed by finding an inverse kinematics (IK) solution for the mobile manipulator at a feasible nozzle pose for every nozzle tip waypoint.

We uniformly sample $N_s=N_W/10$ nozzle tip waypoints, making sure that endpoints of individual segments are included in $N_s$. With an initial mobile manipulator configuration (${\mathbf{q}}_{\text{seed}}$) as the seed, we compute a feasible IK solution for the $N_s$ nozzle tip waypoints by solving the following optimization problem.

$${\mathbf{q}}_i=\underset{\mathbf{q}}{\text{argmin}}{\left(\text{Nozzle Pose Error}\right)}_i:i=1,2,\dots ,N_s$$ (17)

Nozzle pose error computation for a configuration $\mathbf{q}$ is done according to Eq. (11). Once we have the configurations for all the $N_s$ nozzle tip waypoints, we need to assign a nozzle tip path-length parameter $s\in [0,1]$ to each configuration. We assign the parameter $s$ to a nozzle tip waypoint based on the length along the nozzle tip path from the first waypoint position to the particular waypoint position. If the length along the nozzle tip path to a waypoint position is $l$, then $s=l/L$, where $L$ is the nozzle tip path length. We assign the corresponding $s$ to the configuration $\mathbf{q}$ obtained from solving the Eq. (17). We then determine the control points and knot vectors for $(3+n)$ third-order B-splines, one for each configuration variable (i.e degree of freedom) ${\theta }_j$ like in Eq. (14). We stack the control points for each configuration variable to get the seed ($X_o$) of the optimization problem (4). $X_o$ is a $(n+3)\times N_{\text{cp}}$ vector.

7.5. Successive constraint refinement for solving multi-stage optimization

Solving the full-blown problem considering all the constraints together can be computationally slow. Each constraint has one or several attraction basins in the optimization parameter space. Having all constraints together may result in opposing attraction canceling out, causing the solution to get stuck in local minima or at an infeasible location [93]. Hence, it is necessary to solve a problem with a few constraints first and successively add constraints to the previous solutions. Moreover, selecting the right sequence for adding constraint components to the optimization procedure can help to converge quickly to the desired feasible region.

We have identified the appropriate sequence after extensive experimentation to obtain a feasible solution quickly for a complex optimization problem described in (4). In the following methodology, the $\text{solveNLOpt}$ function takes in the seed control points, the objective function and the constraints, and uses non-linear programming to determine a locally optimal solution. The following steps describe our approach.

• 1.$X_1\leftarrow \text{solveNLOpt}(X_o,\text{Objective},\text{Constraints})$$\text{Objective}:\text{Nozzle Pose Error}$, $\text{Constraints}:\Phi$$\text{Objective}:\text{Nozzle Pose Error}$, $\text{Constraints}:\Phi$
• 2.$X_3\leftarrow \text{solveNLOpt}(X_2,\text{Objective},\text{Constraints})$$\text{Objective}:\text{Nozzle Pose Error},\text{Nonholonomy Error}$, $\text{Constraints}:\Phi$
• 3. $X_4\leftarrow \text{solveNLOpt}(X_3,\text{Objective},\text{Constraints})$ $\text{Objective}:\text{Nozzle Pose Error},\text{Nonholonomy Error},\text{Total Time}$ $\text{Constraints}:\Phi$
• 4. $X_5\leftarrow \text{solveNLOpt}(X_4,\text{Objective},\text{Constraints})$ $\text{Objective}:\text{Nozzle Pose Error},\text{Nonholonomy Error},\text{Total Time},\text{Total MB Distance}$ $\text{Constraints}:\Phi$
• 5. $X_6\leftarrow \text{solveNLOpt}(X_5,\text{Objective},\text{Constraints})$ $\text{Objective}:\text{Nozzle Pose Error},\text{Nonholonomy Error},\text{Total Time},\text{Total MB Distance}$ $\text{Constraints}:\text{Joint Limits},\text{Velocity Limits}$
• 6. $X_7\leftarrow \text{solveNLOpt}(X_6,\text{Objective},\text{Constraints})$ $\text{Objective}:\text{Nozzle Pose Error},\text{Nonholonomy Error},\text{Total Time},\text{Total MB Distance}$ $\text{Constraints}:\text{Joint Limits},\text{Velocity Limits},\text{Collision}$

We do not have any constraints for steps 1–4 as the focus is only on reducing the objective function in each step. We add the joint and velocity limit constraints thereafter and then the collision avoidance constraint in the end. The sequence shown here has been obtained empirically after extensive experimentation.

8. Computation of time intervals between waypoints

In the previous section, the trajectory generated by the optimization-based motion planner satisfies the robot motion and nozzle tip path-related pose constraints. Hence, it results in the disinfectant sprayed along the spray path. The surrogate for time in the optimization described in Eq. (10), measures the total time required to execute the task. It measures the time between two consecutive configurations as the maximum of the times required by each degree of freedom to move between the configurations at the maximum possible speed. Hence, this $△t$ is the least amount of time feasible between two consecutive configurations. Even though this results in the shortest trajectory execution time, it may not be enough to thoroughly disinfect every point on the path. Each point on the path needs enough amount of disinfectant to be thoroughly sanitized. If the robot moves too quickly, it may not spray enough disinfectant on some points for them to be sanitized. Hence, it is necessary to regulate the velocities of the configuration variables or degrees of freedom so that the nozzle moves with a velocity that results in thorough disinfection of every point on the path. We call this regulation of velocities as determination of time intervals between successive spray path waypoints. One way to regulate the velocities is to determine the amount of time between every two consecutive configurations of the trajectory so that the points on the surface that are sprayed receive enough disinfectant. As mentioned in Section 7.4, the number of nozzle tip waypoints used for seed generation for spline parameters and optimization is 10% of the total number of nozzle tip waypoints. Using the splines generated by the optimization-based motion planner, we generate new dense nozzle tip waypoints by computing the pose of the nozzle using forward kinematics of the configurations. An example of the dense nozzle tip waypoints is shown in Fig. 17.

Fig. 17.

The spray deposition rate for point $P_k$ as the spray passes through it. $c_0$ to $c_8$ are the circles (or cross-sections of the spray cone frustum with the point cloud) as the nozzle moves from left to right.

Consider a point $P_k$ on the point cloud as shown in Fig. 17. Each circle in the figure is the intersection of the spray cone frustum with the point cloud. It can be a circle or an ellipse, but here we consider it to be a circle for explanation purposes. The spray moves from left to right, $P_k$ is initially just outside $c_0$. It is inside $c_1$ to $c_4$ where it moves towards the center of the spray, and hence the spray deposition rate increases according to Fig. 3 and Eq. (3). This spray deposition rate may or may not be Gaussian. In $c_4$, $P_k$ is the closest to the center as compared to any circle. After that, the spray deposition rate decreases, as shown. The area under the deposition rate curve is the total disinfectant deposited at point $P_k$. This total disinfectant deposited must be greater than or equal to $H_o$ as mentioned in Section 3.

Since the nozzle tip waypoints are discrete, we cannot compute the deposition rates of the point when the spray moves from $c_i$ to $c_{i+1}$. Hence, it is difficult to compute the accurate area under the deposition rate curve. Therefore, we compute a conservative area by measuring the deposition rates at the discrete instances. From $c_0$ to $c_1$, $P_k$ is outside and then inside the circle. Hence, the deposition rate is zero initially and increases as it goes inside the circle. Hence, for $c_0$ to $c_1$, we assume the spray deposition rate to be zero. Once $P_k$ is inside the circle, the deposition rates are greater than zero at every discrete waypoint. In this case, we take the conservative approximation of the area to be the minimum of the deposition rates at the two points consecutive waypoints multiplied by the time difference between the two waypoints (which we are yet to determine). Hence, we get the area under the curve approximated as the bars as shown in Fig. 17. This area for the point $P_k$ of the form $a_{k1}△t_1+a_{k2}△t_2+a_{k3}△t_3+a_{k4}△t_4+a_{k5}△t_5+a_{k6}△t_6$. Here, we know the coefficients $a_{ki}$. This area should be less than $H_o$. We sample $N_r$ points on the point cloud and get the equation of the area for each of those points. Hence, we have $N_w-1$ $△t$s to determine. Moreover, we know that each $△t$ should be greater than or equal to the $△t$ obtained from Eq. (10) from Section 7 since that is the least amount of time feasible between any two waypoints. Furthermore, we want to make sure that the total time taken to go from the first nozzle tip waypoint to the last is minimized. Hence, we have the following linear programming formulation to determine the unknowns $△t_1$ to $△t_{N_W-1}$.

$$\text{min}cx$$

$$\text{s.t.}Ax\le H_o^{\prime }$$

$$x\ge △t^{\prime }$$ (18)

where, $x$ is a column vector of size $N_W-1$ of $△t_i$s, $c$ is a row vector of ones of size $N_W-1$. Hence $cx$ is the total trajectory execution time. $Ax\le H_o^{\prime }$ is the equation that encodes all the linear equations for the computation of the area for all $N_r$ points. Hence, the size of matrix $A$ is $N_r\times N_W-1$. $H_o^{\prime }$ is a $N_r\times 1$ vector with all entries $H_o$. Moreover, as mentioned, each $△t_i$ should be less than the minimum possible time permitted by the robot between every two consecutive waypoints. This is encoded in the $x\ge △t^{\prime }$. $△t^{\prime }$ is a column vector with each entry for the minimum feasible time difference between two consecutive waypoints. This equation can be solved using any linear programming methodology. Since this is a set of linear equations, solving which is computationally very fast, a large number of samples from the point cloud ($N_r$) can be taken.

Spraying at a point constantly so that it receives $H_o$ disinfectant is important rather than spraying on it at discrete times. It is possible that one point may lie inside several sets of consecutive circles. We want to make sure that only one consecutive set is used in determining the total spray deposited. Hence, we choose the set where the average deposition rate is the highest. This may help in driving down the corresponding $△t$s. The $△t$s obtained from this completes the trajectory of the mobile manipulator that guarantees complete disinfection of the points on the surface represented by the point cloud by spraying enough disinfectant on them. We can now regulate the velocity and acceleration of each degree of freedom or configuration variable so as to reach the next waypoint in at least the corresponding $△t$.

8.1. Addressing uncertainties in point clouds

As mentioned before, we use the method described in [95] for capturing continuous point clouds as the robot arm moves from a start to an end configuration and merging them to get a more accurate point cloud. The uncertainties in the resulting point clouds for each point are about 0.6–0.9 mm in the normal direction to the plane of the point cloud after extensive testing of the method in several scenarios. Hence, for the method described in this section, we can compute each coefficient $a_{ki}$ i.e the disinfectant spray rate assuming the worst case by displacing the particular point on the point cloud by 1 mm so as to reduce the disinfectant rate. This will ensure that our method computes the time intervals between waypoints such that each point receives enough disinfectant to guarantee disinfection in the worst case scenario in terms of point cloud uncertainty.

9. Results and discussion

9.1. Test cases

We have tested our methodology on five diverse test situations where disinfection is more likely needed on a daily basis. Fig. 18 shows the 5 test cases and the corresponding point cloud of the area to be disinfected. The test case (a) has a washbasin sink and adjacent area that needs disinfection. The test case (b) has a work area with a computer. We assume that the spray disinfectant evaporates, and hence it is safe to use on electronics. A pair of chairs are shown in the test case (c), representing a waiting area or a movie theater. Test case (d) represents two doors of a busy hallway that needs disinfection due to constant usage. The final test case (e) shows a hospital bed that needs disinfection. The corresponding scenarios and their point clouds were recorded in the robot simulation software CoppeliaSim [14], and the algorithms in this paper have been implemented in MATLAB. The depth camera is mounted on the robot end-effector using which we capture several point clouds at different positions and orientations and stitch them together to generate the large point clouds shown.

Fig. 18.

The 5 test cases and the corresponding point clouds on which we tested our algorithm. (a) A wash basin sink (b) A workplace table and computer (c) Chairs representing a movie theater or waiting area (d) Passage doors (e) A hospital bed.

9.2. Discussion on generated spray paths

The projected polygon for the point clouds and the corresponding planar spray paths are shown in Fig. 19. The spray paths show the maximum polygon area covering path possible for the given polygon. The spray radius $R_s$ is taken to be 8 cm to compute the offset polygons, whereas the actual spray radius is 9 cm. This is done to increase the area covered by the spray path and not corners at acute angles. Of the maximum area covering paths, the ones shown are the ones that have been selected by the mobile manipulator capability map-based selection criterion from Section 6.6. The polygons and the corresponding spray paths are shown in Fig. 20. The solid lines in the path correspond to the segments generated by the spray path generation algorithm. The dashed lines correspond to the connecting segments between two consecutive segments generated by the algorithm. The starting dashed segment denoted by s is the starting spray path segment. This starting segment goes beyond the edge of the polygon so as to not leave an unsprayed region at the start. The terminal segment denoted by t is the final segment. The direction and the starting point of this segment are determined as mentioned in Section 6.3. Although these paths should cover the entire area of the polygon, in the case of polygons with sharp angles like in cases (a) & (e) in Fig. 20, the corners may remain unsprayed. The unsprayed areas of the polygon have also been highlighted therein. The percentage area not sprayed is observed to decrease or remain the same with each iteration of the spray path generation algorithm, as shown in Fig. 22.

Fig. 19.

The polygons for the 5 test cases and the corresponding spray paths. The solid parts represent segments from the Algo. 1, and the dashed parts represent the connecting segments. The areas of the polygon not sprayed are highlighted with circles for each test case. The “s” corresponds to the dashed starting segment of the spray path and “t” stands for the solid terminal segment of the spray path. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 20.

The spray path for the 5 test cases plotted on the corresponding point clouds.

Fig. 22.

Percentage area not sprayed vs. the number of spray path iterations for all the 5 test case scenes.

The performance of the Spray Path Generation algorithm for all the five test cases is shown in Table 1. For the test case (a), there are eight different spray paths generated, with the best path not spraying on about $0.6\text{\%}$ of the area. The area not sprayed on is typically the corner points when the angles are really small, as shown in Fig. 20(a). During the algorithm, the most expensive step is the expansion of a segment, i.e., computing of its children and the offset polygon. Hence, test cases that have a large number of segments expanded have a higher computation time. Moreover, we observed that when a polygon has concavities, it takes longer to compute its offset polygon. Hence, in the test case (c), where the polygon has concavities, it takes a similar time for spray path generation as for test case (b), which has almost double the number of expanded polygon segments. The case (d) has a convex spray polygon similar to a rectangle. The algorithm computes the multiple spray paths. We terminate the algorithm when the last 5 test paths have zero grid points not sprayed on, and the length of the path does not decrease more than a set factor. Hence, for case (d), no branches are pruned since all attempts to compute the path are successful. The polygon for test case (e) as shown in Fig. 20(e) has several narrow extensions. These are smaller in width as compared to the spray radius; hence the offset polygon does not show any of these narrow extensions. Hence, these areas of the polygon tend to be ignored during the computation of the spray path. However, some of the extensions can be covered due to the starting segment and the final segment, as can be seen in the figure whereas, other extensions as highlighted are not sprayed. Hence, we get only a single path that covers the maximum possible area, as shown in the figure. We observe a similar situation with respect to narrow extensions not being sprayed in the test case (a) and (c) as well, where there are narrow extensions represented by small acute angles at some of the polygon vertices. Hence, the percentage of the area not sprayed is higher for the test case (a) and (e).

Table 1.

The table describing the performance of the Spray Path Generation algorithm on the 5 test cases from Fig. 18.

To study the polygons that have narrow extensions and/or acute angles at any of the vertices, we define a term called the spray efficiency of any point inside the polygon. We uniformly sample a dense set of points inside the polygon. At each of the sampled points, we draw a circle with that point as the center and the spray radius as the radius. The ratio of the area of that circle that lies inside the polygon to the area of the circle is called the spray efficiency. We assign this spray efficiency to all the points inside the intersection of that circle and the polygon. We repeat this for every sampled point. Whenever the spray efficiency of any particular point is being assigned, we check if the previous spray efficiency was greater than, less than, or equal to the current spray efficiency. If it is less than the current spray efficiency, we make the spray efficiency of that point to be the current spray efficiency. This computation ensures that any point well inside the polygon has a spray efficiency of 1 and points near corners or in narrow extensions have an efficiency less than 1. The spray efficiency color map is shown in Fig. 21(a). It can be seen that inside the narrow extensions, the spray efficiency is significantly lower than at points well inside the polygon. The points that have a spray efficiency less than 1 imply that spraying on those points will waste the spray disinfectant liquid. It can be seen that there is a similarity in the areas of the polygon that have lower spray efficiency and the areas that the spray path does not spray on from Fig. 20(e). Since the spray path start and end segments are computed in a different way as compared to the rest of the segments, the spray path ends up spraying a larger percentage of the area as compared to the area that Fig. 21(a) would suggest. It should be noted that having a spray efficiency less than 1 at a point does not mean that the spray path will not be able to spray on it. The spray efficiency color map of a polygon helps us get an understanding of the areas of the polygon that are less likely to be sprayed on by the spray path.

Fig. 21.

(a) This shows the spray efficiency across the polygon for the test case (e). It can be seen that at the corners and at the ends of narrow extensions, the spray efficiency is low as compared to the other parts of the polygon. (b) Bar plot showing the percentage of area that does not have spray efficiency of 1 for all the 5 test cases. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

We need to make sure that the areas of the polygon that have a spray efficiency of less than one are being sprayed by the spray path. Spray efficiency not equal to 1 does not mean that the point will not be sprayed on. Hence, when comparing the percentage of the area not sprayed from Table 1 to the bar plot in Fig. 21, the actual percentage is much lower. It is still a finite number for test cases (a), (c), and (d). We need to make sure that the areas marked in red in Fig. 20 are sprayed on. We determine the areas in red from Fig. 20 by computing the sprayed area and subtracting it from the original polygon. Once we have the areas, we find the closest spray path waypoint to the centroid of the local unsprayed area (in red) and move towards the centroid from the waypoint till the entire unsprayed area is inside a spray circle. We travel this small segment from the nearest waypoint and back to the waypoint and continue to the next waypoint. The spray paths mapped onto the original point clouds for all the five test cases are shown in Fig. 20. The small extensions to the spray paths are highlighted. This ensures that the entire spray polygon is sprayed on.

Table 2.

The performance of the optimization-based motion planner for the mobile manipulator for the five test cases.

9.3. Bench-marking

The spray paths generated by the spray path generation algorithm and the additional segments to cover the narrow extensions cover the entire area of the polygon. The algorithm generates several spray paths, and we choose one from several paths that cover the maximum area and is the shortest. There can be a few simple approaches to generate spray paths without following our algorithm. We compare our method with three potentially competing methodologies for generating spray paths. Zig-zag and spiral paths are typically the most common ways humans intuitively spray on the polygon. These methods may not be optimal in terms of the area coverage and even length of the path. Paths generated by our algorithm are essentially a combination of zig-zag and spiral paths optimized to cover the entire area and not waste spray by staying inside the boundary of the polygon. We describe three competing methods for generating spray paths as follows:

• •Longest Convex Edge Zig-Zag Path: For any polygon, we can generate a zig-zag path like in Fig. 12 with the parallel lines oriented in any direction. In this method, we take the longest convex edge of the polygon and generate a zig-zag spray path with that edge as the starting segment. The offset for the zig-zag parallel lines is twice the spray radius. The idea behind using the longest convex edge for getting the direction of the parallel segments of the path is that it will result in a greater chance of the spray path remaining inside the polygon. We want to make sure that the spray liquid is not wasted when the path is outside the polygon, which can happen when a zig-zag path is being sprayed on. The assumption here is that there is at least one convex edge for the polygon.
• •Principle Component Zig-Zag Path: This method is similar to the previous one, with the difference being that the parallel segments of the zig-zag path aligned with the direction of the principle component of the polygon as opposed to the longest convex edge. The idea behind this method is that each spray segment sprays a large area along the segment. This reduces the number of spray segments required. In this and the longest convex edge zig-zag path, we start from a vertex of the polygon and move in a zig-zag fashion in the directions perpendicular to the principal component or the direction to the longest convex edge till there is no polygon left to disinfect.
• •Spiral Path: A spiral path sprays along the perimeter of the polygon and then goes inside layer by layer. This ensures coverage of the entire area.

The effectiveness of a spray path can be studied by measuring quantities like the length of the path, the percentage of the area it covers, the length of the path where the spray disinfectant is wasted, i.e., it is sprayed outside the polygon. We compare the paths generated by the above methodologies with the paths generated by our method with respect to these quantities in order to gauge the performance of our algorithm with human-intuition-based simple approaches. The paths generated for the test scene (a) are shown in Fig. 23. The comparison is shown in Table 3, Table 4. It can be seen that for most test cases, the length of the path generated by the spray path generation algorithm is higher as compared to the paths generated by the other three methods. However, the percentage of the area not sprayed is significantly higher in each case for the three competing methods. Moreover, the percentage of spray wasted, which is the ratio of the sprayed area by the path outside the spray polygon to the total area sprayed inside the polygon, is also significantly higher for the three methods. The wastage happens because the spray path no longer stays inside the polygon but can go outside to spray corners like Fig. 23(b). The spray path generation algorithm is targeted towards producing paths that cover the entire area and do not go outside the polygon. This is evident from the percentage of the area not sprayed and the percentage of the spray path wasted for all the 5 test cases as seen in Table 4. Hence, our algorithm ensures that there are no inefficiencies during the spraying operations with respect to spray wastage and ensuring the entire area is covered.

Fig. 23.

For the test case (a): (a) The longest convex edge zig-zag path (b) principle component zig-zag Path and (c) Spiral path.

Table 3.

Table comparing the longest convex edge zig-zag path to the principle component zig-zag path for all the 5 test cases.

Table 4.

Table comparing the Spiral path to the path generated by the spray path generation algorithm.

9.4. Robot motion generation

The successive constraint refinement approach is used for robot motion generation to follow the spray paths from Fig. 20. The performance of the successive refinement methodology is shown in Table 2. The number of control points for splines representing each degree of freedom depends on the length of the nozzle tip path and the resolution at which we consider points on the nozzle tip paths. For each point on the nozzle tip path at the considered resolution, we determine a feasible IK configuration of the robot using Eq. (17). The seed spline which fits all the IK solutions is then successively refined using successive refinement methodology. We compare the performance of the successive refinement methodology with a single-stage optimization with all the constraints being applied together. From Table 2, we observe that single-stage optimization generates a solution within a 2000 s time bound only for the test cases (a) and (b). For the next three test cases, single-stage optimization fails to find a solution within the given time-bound. The algorithm was implemented in MATLAB; hence the computation time was observed to be large. The collision count shows a clear picture in terms of the performance of the two methods. In the single-stage, the collision function is called several times in each iteration. However, for sequential optimization or successive refinement, only at the last stage do we include the collision constraints. Hence, the collision count and the computation time for successive refinement are significantly lower than single-stage optimization. The end-effector pose error is taken to be the magnitude of the third term in Eq. (11). The maximum values of this error and the non-holonomy error from Eq. (12) are lower than 10⁻⁶ is each test case for successive refinement based approach.

For determining time intervals between successive waypoints, we sample points on the point cloud with a resolution of 1 cm. For all the sampled points, we measure the amount of spray disinfectant deposition rate at each waypoint according to Eq. (2). From the robot motions, we have the time taken to move between every two consecutive waypoints based on the maximum joint velocities. This is the fastest the robot can physically move between the waypoints, and hence the resulting trajectory is the time-optimal trajectory. However, it may result in not enough disinfectant sprayed on all the sampled points on the point cloud. We solve the linear programming problem in Eq. (18) to get the time taken to move between successive waypoints so that each point gets enough disinfectant. This increases the trajectory execution time as compared to before appropriate time intervals are applied between waypoints as shown in Table 5. The trajectory execution times for all test cases before and after determining time intervals between successive waypoints are shown. We also show the percentage of the sampled points that do not receive enough disinfectant dosage before appropriate time intervals between waypoints. For test case (b), more than two-thirds of the points do not receive enough dosage, whereas for (d), only 13.3% of the points do not receive enough dosage. Based on the trajectory execution time before and after determining time intervals between successive waypoints, we can observe the benefits in terms of the extra time required. To implement this, we regulate the joint velocities appropriately between the waypoints so as to reach the next waypoint by taking at least the prescribed time difference from Eq. (18). After determining the appropriate time intervals between successive waypoints, each point receives enough dosage.

Table 5.

Comparison between the trajectory execution time before and after determining appropriate waypoint intervals and the corresponding percentage of point cloud points that did not receive enough disinfectant.

The trajectory generated has the mobile base moving as the robot sprays on the surface. The trajectory for the mobile base for test case (a) has been shown in Fig. 27. If we constrain the robot to remain stationary or add constraints such that it can remain at a few points close the surface, the mobile base path and behavior will change accordingly.

Fig. 27.

The mobile base trajectory for the test case (a).

9.5. Physical experiment

We have conducted physical experiments with ADAMMS-SD for spraying on a scene replicating the test case (b). The spray cone parameters for the physical experiments are shown in Fig. 24. We used a Lysol disinfectant readily in stores for disinfection. We conducted experiments to determine the distances $D_n$ and $D_f$. We take $D_n$ to be approximately the distance lower than which the spray liquid starts flowing on the surface, and $D_f$ as the distance where the spray starts behaving as a projectile when the nozzle is spraying in the horizontal direction. The spray angle was found to be $14°$. By placing a measuring tube at the center of the spray at a distance $D_n$ we determined $d_o$, the rate at the center of the spray to be approximately $1.1\text{mm}/\text{s}$. We use these same parameters for our simulation experiments as well. For effective disinfection, it is important that the treated surface be visibly wet for a certain length of time which is called the contact time. As per CDC guidelines, the contact time can be about 15 s for most objects in the home or office environments. We spray a surface at a distance $D_n$ from the nozzle tip for a certain amount of time and observe if it remains visibly wet for 15 s. We record the time and multiply it by $d_o$ to get the threshold disinfectant required any point to be $H_o$ as mentioned in Section 3. We obtain this threshold to be approximately $1.8\text{mm}$. We use this threshold value for determining time intervals between successive waypoints in Section 8 and for simulation experiments.

Fig. 24.

The spray model for the physical experiment.

For the experiment, we kept the mobile base stationary since the entire scene workspace was reachable for the manipulator. We have regulated the velocities of the end-effector between waypoints according to the data provided by the time interval between successive waypoint determination methodology. The entire task is completed in about 17 s. (Task and nozzle trajectories are shown in Fig. 25, Fig. 26)

Fig. 25.

The sequence of nozzle locations at 10 different locations of the nozzle tip path.

Fig. 26.

(a) The scene for physical experiment (b) The spray path on the polygon (c) The point cloud and the overlayed spray path.

10. Conclusions & future work

In this paper, we present a system and method for spray disinfection using mobile manipulators. The mobile manipulator has a spray nozzle and a depth camera attached to the end-effector. It captures a point cloud of the object or an area to be disinfected. The point cloud is projected and converted into a 2D polygon. We have presented an algorithm for determining a spray path that covers the entire area of the polygon. The algorithm produces non-intuitive paths that cover the entire area of the polygon. We use a branch and bound-based method with pruning heuristics to accelerate the computation of the spray path. The algorithm produces several spray paths, and we choose the one that enables the mobile manipulator to reach the maximum number of segments at the initial pose. We then project the spray path on the original point cloud to get the nozzle tip path. We move on to determining the mobile manipulator motions using a successive refinement approach for non-linear optimization to find the control points of the spline representation of the robot. This gives us the fastest possible path for the robot configurations. This path may not spray enough dosage to disinfect all the points of the area. We then determine time intervals between successive waypoints of the path by adding time delays between them. This results in sufficient spraying on every point of a coarsely sampled point cloud.

We have shown that the spray path algorithm generates a path that covers the majority of the area. We modify the path to make sure that narrow extensions and corners of the polygons are sprayed sufficiently. The first spray path is generated quickly, as shown in Table 1 and the final path is generated within a few seconds. The successive refinement or multi-stage optimization for motion generation results in significantly faster computation of the path as compared to single-stage optimization. For the test cases where there is no time out, the successive refinement approach computes a solution  30 times faster. Moreover, the errors for the pose of the end-effector and the non-holonomic constraints of the mobile base are of the order 10⁻⁶. The trajectory generated by the optimization algorithm results in a large percentage of points not receiving enough disinfectant. After determining time intervals between successive waypoints, it results in 100% of the coarsely sampled points on the point cloud receiving enough disinfectant. The trajectory execution time increases to between  2–5 times as a result.

Limitations and Failures: There are a few limitations on the situations where the spray path generation algorithm may not perform well. Cases where there is a large depth where the nozzle tip may not be able to reach, is one such situation. Moreover, the algorithm only sprays on the area projected as the polygon, and there may be a need to generate a different polygon from different locations if the object being disinfected is large. Moreover, in cases where the point cloud projection of a large area results in a small area in 2D, it may not be sprayed completely. In such a situation unwrapping the point cloud as mentioned in Section 6.1 may be a better solution. If the spray path is long, a single spline with a large number of control points may perform worse since the optimization variable size will quickly rise due to the high degrees of freedom of the robot. In such situations, it is necessary to break the splines into a number of splines and have spline connectivity constraints in the optimization. The final limitation is that when the point cloud to be sprayed is large, even a sparse sampling of points may result in a large number of points for the matrix $A$ of the linear programming problem formulation. It may take a significant amount of time to create the problem instance in such cases. Moreover, if we make the sampling even sparse to reduce this time, the number of points being disinfected may not be accurate, and there may be areas of the point cloud that do not receive enough disinfectant even after determining time intervals between successive spray path waypoints.

It should also be noted that we have used a simplistic model of the disinfectant spray from the nozzle. Since we have neglected gravity, to robot spraying in close to upward directions will result in different spray depositions at points than what our model will expect. Having a detailed spray model including gravity and better spray flow model on steep surfaces will make our interval generation method produce accurate results for thorough disinfection.

Future Work: We plan to create a robust physical robotic platform that is able to perform all the tasks quickly with optimized implementations of all algorithms. Currently, we have a human extract a point cloud of the desired surface from the recorded point cloud. We plan to have a method to filter out the undesired parts of the point cloud, like floor or walls behind the objects to be disinfected. Moreover, we also plan to have a human feedback on important areas of the point cloud, like a door knob or handles or any objects that need more thorough disinfection. We plan to modify our algorithm to spend more time on such objects or have a denser spray path to ensure their thorough disinfection. Furthermore, the system currently has no feedback mechanism on whether the physical robot has sprayed enough to actually disinfect the areas. We plan to have a deep learning-based approach to identify how well an area has been sprayed on and if there are any corners that are not ‘wet’ yet and need spraying. This feedback mechanism will significantly increase the effectiveness of the system. In the future, we also plan to have exhaustive experimentation of the robot following the sprayed paths and completely disinfecting the entire object from all sides instead of a single direction projection of it implemented in this work.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Biographies

Shantanu Thakar is currently pursuing a Ph.D in Robotics and AI with Prof. S K. Gupta at the Viterbi School of Engineering at the University of Southern California. His research is focused on Physics aware motion planning for high degree of freedom robotic systems and AI for robot task and motion planning. He completed his BTech and MTech in Mechanical Engineering from Indian Institute of Technology Bombay (IITB) in 2015.

Rishi Malhan is a third-year Ph.D student in Robotics and artificial intelligence (AI) under Prof. S K. Gupta at the University of Southern California. He is interested in realizing smart robotic assistants through advances in AI. His research interests are robot motion planning, numerical optimization, reinforcement learning, deep learning, etc.

Prahar M. Bhatt is a Ph.D. candidate at the Aerospace and Mechanical Engineering Department at the University of Southern California (USC). He is currently researching at the Center for Advanced Manufacturing, USC, with Dr. Satyandra K. Gupta. As a researcher, Prahar has a demonstrated history of working in the areas of artificial intelligence, computer vision, motion planning, machine learning, non-linear optimization, and robotics.

Satyandra K. Gupta is the Smith International Professor with the Aerospace and Mechanical Engineering Department and the Director of the Center for Advanced Manufacturing with the University of Southern California, Los Angeles, CA, USA. His current research interests include robotics, computer-aided design and manufacturing automation.

Appendix.

We derive the spray deposition rate mentioned in Eq. (2). Given that at a distance $D_n$ from the nozzle tip, the deposition rate is as defined from Eq. (1)(Eq. (A.2)), we want to find what is the deposition rate at a any distance $z$ from the nozzle. We assume that gravity does not impact the spray close to the nozzle.

$$d_R(D_n,r)=d_o{e}^{-\frac{1}{2}\frac{r^2}{{\sigma }^2}}$$ (A.1)

To derive this, essentially we need to determine the constants $d_q$ and ${\sigma }_q$ of the following equation (see Fig. A.28):

$$d_R(z,r)=d_q{e}^{-\frac{1}{2}\frac{r^2}{{\sigma }_q^2}}$$ (A.2)

Fig. A.28.

The spray deposition rate modeled as a Gaussian changes as the distance from the nozzle tip.

The volume of the spray per second remains constant at any cross-section perpendicular to the spray direction. Hence, the volume at $D_n$ per second is equal to the volume at a distance $z$.

$$dV=d_R(D_n,r)dA$$ (A.3)

$$dA=2\pi rdr$$ (A.4)

$$dV=d_{o}2\pi r{e}^{-\frac{1}{2}\frac{r^2}{{\sigma }^2}}dr$$ (A.5)

here, $dA$ is the area of a ring centered at the cone axis. Integrating, we get the volume at $D_n$ to be

$$V_{R_n}=2\pi d_o{\sigma }^2(1-e^{-\frac{R_n^2}{2{\sigma }^2}})$$ (A.6)

By applying conservation of volume, the volume rate at $D_n$ is equal to the volume rate at $z$,

$$2\pi d_o{\sigma }^2(1-e^{-\frac{R_n^2}{2{\sigma }^2}})=2\pi d_q{\sigma }_q^2(1-e^{-\frac{R_q^2}{2{\sigma }_q^2}})$$ (A.7)

$$R_q=R\frac{z}{D_n}$$ (A.8)

Now assuming that the spray rate distribution standard deviation $\sigma$ changes linearly with distance we get,

$${\sigma }_q=\sigma \frac{z}{D_n}$$ (A.9)

$$d_q=d_o\frac{D_n^2}{z^2}$$ (A.10)

hence, the deposition rate at any distance $z$ and a radial distance $r$ is given by:

$$d_R(z,r)=d_o\frac{D_n^2}{z^2}{e}^{-\frac{1}{2}\frac{r^2}{{\sigma }^2}\frac{D_n^2}{z^2}}$$ (A.11)

This the Eq. (2).