Design and Control of a Highly Redundant Rigid-flexible Coupling Robot to Assist the COVID-19 Oropharyngeal-Swab Sampling

Abstract

The outbreak of novel coronavirus pneumonia (COVID-19) has caused mortality and morbidity worldwide. Oropharyngeal-swab (OP-swab) sampling is widely used for the diagnosis of COVID-19 in the world. To avoid the clinical staff from being affected by the virus, we developed a 9-degree-of-freedom (DOF) rigid-flexible coupling (RFC) robot to assist the COVID-19 OP-swab sampling. This robot is composed of a visual system, UR5 robot arm, micro-pneumatic actuator and force-sensing system. The robot is expected to reduce risk and free up the clinical staff from the long-term repetitive sampling work. Compared with a rigid sampling robot, the developed force-sensing RFC robot can facilitate OP-swab sampling procedures in a safer and softer way. In addition, a varying-parameter zeroing neural network-based optimization method is also proposed for motion planning of the 9-DOF redundant manipulator. The developed robot system is validated by OP-swab sampling on both oral cavity phantoms and volunteers.

I. Introduction

The outbreak of novel coronavirus pneumonia (COVID-19) is affecting the entire world, which has caused a large number of deaths with an increase in the spread of COVID-19. To control the spread of COVID-19 at the early stage, oropharyngeal-swab (OP-swab) sampling is commonly adopted with respect to sample collection and specimen sources for diagnosis [1]. However, protecting the safety of medical staff during the sampling process raises a new challenge because of susceptibility to infection from person to person through respiratory droplets and contact transmission in an unprotected environment [2]. A variety of related studies have reported that respiratory droplets, feces and urine are the routes of transmission [3]. To address these issues in OP-swab sampling, robotics could play a key role in disease prevention.

Considering the high risk of infection of COVID-19 and negative tests of nasopharyngeal swabs (NP-swab) caused by irregular sampling, it is necessary to design an OP-swab sampling robot to assist healthcare staff through remote access. Robot-assisted OP-swab sampling is a promising technique because it relieves the burden from medical staff, is convenient for large-scale deployment, is cost-effective, and offers sampling standardization. As reported in [4], an OP-swab robot could speed up the sampling process especially because there is a lack of qualified healthcare workers. In [5] and [6], a semi-automatic OP-swab robot (compared to the traditional human method) was developed with a teleoperation scheme that achieved good performance in clinical practice with a 95% success rate. The OP-swab robot system mainly includes a snake-like manipulator, two haptic devices(Omega.3 and Omega.6) for teleoperation, an endoscope for visualization (assistance operating in the oral cavity) and a force-sensing system for safety protection. All OP-swab sampling processes were controlled by experts’ teleoperation, where one haptic device (Omega.3) control the tongue depressor and another haptic device (Omega.6) operate the manipulator. Wang et al. [7] have designed a low-cost robot for assistance in sampling of NP-swab; a swab gripper is attached to the rotation link with its extruded active 2-degree-of-freedom (DOF) end-effector for actuating the swab and a generic 6-DOF passive arm for global positioning. In [8], the team from the University of Southern Denmark and the Lifeline Robotics company developed the first automatic swab robot for COVID-19 sampling. In [8], the robot is integrated with a UR5 manipulator, a visual system and dexterous rotatable rigid connectors assembled with OP-swab which could perform the sampling task quickly using an automated scheme.

Of note, medical security remains vital to OP-swab sampling because human throat is fragile and easily injured. Nonetheless, although many desirable results were obtained, the abovementioned end-effectors were designed on the basis of a rigid body structure, which may cause physical injuries of the oral cavity during improper operation or other medical malpractices. On the basis of our previous work [9], a novel micro-pneumatic actuator (MPA) for throat swab sampling is developed to achieve flexible collection and integrate a force sensor, which provides safe, stable, and reliable sampling experience, as shown in Fig. 1. Furthermore, force-sensing actuation offers compliance, which is helpful against shocks, particularly during interaction with oral cavity. Compared with the rigid body-based sampling robots [5], [7] and [8], the developed MPA is made of soft material, which is safer and lighter. Moreover, MPA is smaller and has a 7.5 mm cross-sectional diameter, which is convenient for working in human oral cavity.

Fig. 1.

9-degree-of-freedom (DOF) redundant rigid-flexible coupling robot.

To avoid contact with the OP-swab specimen and collision with the manipulator, it is essential to consider the constraints of the oral cavity space during motion planning. Inspired by our previous work [10] on remote center of motion (RCM) techniques in minimally invasive surgical robots, the second to the last link of the OP-swab sampling robot is constrained with the oral cavity center (OCC) constraint, which guarantees the absence of collision between the inserted end-effector with the oral cavity. Although mechanical implementation is usually safer for an open minimally invasive surgery, programmable RCM is cost effective and convenient to be implemented [11]. The traditional Cartesian adaptive control in [11] was applied for surgical control with the RCM constraint but without considering the physical limits of the manipulator, such as the joint angle and velocity. For practical applications, we propose an optimization strategy for the multi-constraint problem, where the joint angle and velocity, and the OCC constraint-based kinematic model are derived as a quadratic programming problem. In this study, a comparison between the varying-parameter zeroing neural network (VP-ZNN) and the traditional gradient descent method [12] is proposed for the optimization problem that has achieved superior performance, which can converge at the finite time owing to the optimal solution changing over time.

To test the developed OP-swab sampling robot, which is shown in Fig. 1, the experiments are simultaneously conducted in both the human oral cavity phantom and volunteers, regarding the sampling time, operation complexity for medical staff, safety and effectiveness. The framework of COVID-19 OP-swab sampling using the RFC robot is shown in Fig. 2. The contributions of this study can be summarized as follows.

• 1)A rigid-flexible coupling robot is designed to assist COVID-19 OP-swab sampling, where the MPA for OP-swab sampling is developed to achieve flexible sampling and thereby provides compliant, safe, stable, and reliable sampling.
• 2)An improved motion planning method is proposed for the 9-DOF highly redundant robot-assisted OP-swab sampling system, which guarantees efficiency and accuracy, where the multi-constraint kinematics model is derived as an optimization problem; then, the VP-ZNN is applied to solve the optimization problem.
• 3)Substantive practical experiment testing of the developed OP-swab sampling robot system is demonstrated in the human oral cavity phantom and volunteers.

Fig. 2.

Control framework of COVID-19 OP-swab sampling with a redundant rigid-flexible coupling robot.

II. Design of the Rigid-Flexible Coupling Robot

A. Redundant Rigid-Flexible Coupling Robot

The rigid-flexible coupling robot for OP-swab sampling consists of a UR5 manipulator and a self-developed rigid-flexible coupling manipulator. The overall structure of the 9-DOF redundant rigid-flexible coupling robot is shown in Fig. 1. The self-developed manipulator has 3 DOFs, including a linear motor (prismatic joint), a servo- motor (revolute joint), and an MPA that can change the offset distance of throat swab from the center.

B. Design, Modeling, and Fabrication of MPA

The design of MPA refers to the wrinkle shape of the elastomer robot [13]. Fig. 3 shows that MPA consists of effective air chambers. The sectional view of MPA is shown in Fig. 3(a), which contains a chamber part and a cover part. These two parts are bonded together to form a single-DOF MPA. is the wall thickness of the air chamber; is the distance between the air chambers, and is the total length of MPA. When MPA is inflated, the air chambers will expand and repel each other, which causes MPA to bend. The bending process of MPA is shown in Fig. 3(b).

Fig. 3.

(a) Cross section of the micro-pneumatic actuator(MPA). (b) bending process of MPA. and (c) 2D modeling of MPA.

Assuming that all air chambers have the same bending under the same pressure, it is easy to obtain the geometric relation:

Owing to the small value of after bending, the chord length is approximately equal to the arc length at the non-tensile layer. The distance between the center line and the non-tensile layer is , and the arc length is also approximately equal to the chord length; thus, is

The overall motion model is shown in Fig. 3 (c), where the chord length is

The forward kinematics of MPA is modeled as

According to the abovementioned information, we design MPA with 12 air chambers. However, the number of effective air chambers, i.e., the number of effective joints is because only half of the head chamber and tail chamber will bend. The wall thickness of each chamber is 1.5 mm, the width of each air chamber is 4.5 mm and the total length is 80 mm.

The chamber part and the cover part of MPA are made by 3D printing with a 50-shore hardness. The air pressure is controlled by the proportional valve ITV1050-312 L. Then, a deformation experiment of MPA was conducted, and the experimental results are shown in Fig. 4(a), where the gravity is directly perpendicularly to the plane of the paper. The relationship between the bending angle and air pressure is expressed in Fig. 4(b). It is observed that the bending angle is approximately proportional to the air pressure. After data fitting, the linear equation for the angle () and pressure (kPa) is written as

The correlation coefficient is 0.99 990 and the standard deviation is 0.6464, which confirms that and are linearly dependent. By combining equations (1) and (5), we obtain

Finally, the end-point position of MPA can be calculated by combining equations (2), (3), (4), and (6).

Fig. 4.

Experimental results of MPA. (a) Deformation of MPA with a 50-shore hardness under different air pressures. (b) Relationship between bending angle and air pressure.

C. Force Sensing

To detect the contact force during OP-swab sampling, a strain gauge sensor (BF350-3AA23T0) attached to MPA is adopted. It is necessary to explore the mapping relationship between the force and voltage for force measurement calibration. Actually, the current output voltage is related to two factors, i.e., voltage generated by the force, and voltage caused by the air pressure without a load. Therefore, we have . After 20 group experiments, the relationship between no-load voltage and air pressure is obtained by linear fitting: (mV). Force calibration was performed using the electronic balance (CX-668). The relationship between and is approximately linear; thus, we know that

The details are shown in Fig. 5. The correlation coefficient is 0.99 461 and the standard deviation is 2.65 733.

Fig. 5.

Relationship between and .

III. Optimization Control for the 9-DOF Redundant Manipulator

A. Manipulator Kinematics Model With OCC Constraints

The coordinate system of the 9-DOF redundant RFC manipulator is shown in Fig. 6. The forward kinematics model [14], [15] of the 9-DOF RFC robot can be defined as follows:

where denotes the Jacobian matrix. The joint angle and velocity constraints are defined as follows:

where and present the lower and upper bounds of , respectively; and and denote the lower and upper bounds of the joint velocity , respectively.

Fig. 6.

Coordinate system of the 9-DOF redundant RFC manipulator.

Actually, the constraints in (9) can be converted as

where is the positive constant. Therefore, according to (10) and (11), the joint angle and velocity constraints can be rewritten as a new constraint in the velocity level:

Because the RFC robot is a 9-DOF highly redundant manipulator, there are infinite solutions of in (8) by the inverse kinematics. However, the convergence rate, accuracy, and computational complexity of the pseudoinverse-type solution in inverse kinematics cannot satisfy the requirements. Consequently, we need to identify an optimization solution under multiple constraints; thus, the inverse kinematics problem can be expressed as a new optimization problem. The first-priority optimization problem associated with OP-swab sampling is expressed as follows:

where represents the reference velocity associated with OP-swab sampling tasks. The weight matrix is set as the identity matrix.

Considering the OCC constraint, the link- of the RFC manipulator needs to pass through the OCC, and performs the sampling tasks, which is different from the traditional RCM constraint in the last link. The geometric relationship is shown in Fig. 7. For the -DOF OP-swab sampling robot, the forward kinematics mapping function of Cartesian position and can be defined as follows:

Fig. 7.

Constraint with the oral cavity center.

Unlike common RCM constraints, should be always on the straight line straight line between and (second to the last link), where is the end position of the link and is the end position of the link . During the actual OP-swab sampling, we want to keep the error of the OCC constraint as small as possible. Lines 1 and 2 are constructed as follows: , , respectively. Utilizing the relationship between and the vector projection, can be further written as follows:

where is the length of the second to the last link. The derivative of in (17) with respect to time is reformulated as follows:

where denotes the Jacobian matrix of the OCC constraint. Regarding the OCC constraint task, we want to keep the OCC error at the minimum value.

In this section, it is necessary to find a feasible solution for multiple constraints and guarantee that the tracking error and OCC error always remain in a small range. For COVID-19 sampling tasks, we aim to reformulate OCC (18) and joint physical limits (15) in an optimization scheme and design a method to solve the optimization problem. Consequently, by simultaneously taking OCC, end-effector task, and joint physical constraints into account, the new multi-task optimization problem can be formulated as

where and . Without the explicit expression of and , the actual trajectory will drift and the position error cannot converge to zero from a random initial position. To overcome this issue in (19), a feedback item associated with position signals is integrated into and :

Of note, we should manage the priority strategy for the multiple-task optimization problem by different weights. Therefore, the objective function is defined as

The optimization problem in (22) can be rewritten as follows:

where , and are the constants employed to prioritize different tasks.

B. Neural Network Design

In this section, we describe the design of the VP-ZNN, which is used to solve the optimization problem in (22). First, the multiple-task optimization problem in (22) is converted into an equivalent problem; thus, the VP-ZNN is employed to solve it.

The Lagrange function of (22) constraints is formulated as follows:

where and ; indicates the gradient of (24). As the KKT condition defined in [16], if is continuous, the optimization solution satisfies the following condition:

The state decision variable . The problem in (25) is equivalent to the following:

where and . The bounds of state variable are expressed as

Therefore, the bound of state variable are defined as

The error model of the novel VP-ZNN is expressed as

To ensure the model error convergence to zero, we define the following formulation,

where is the constant that can adjust the convergence rate; denotes the activation function, and , which makes 1. Clearly, the error in (28) converges to zero with exponential convergence.

The function in (28) is expanded as

We further modify the VP-ZNN in (29) as

For comparison, the traditional gradient descent-based recurrent neural network is denoted as

IV. Experiments

In this section, we present the tests with an oral cavity phantom and volunteers using the OP-swab robot system, approval from the Institutional Review Board of The Chinese University of Hong Kong, Shenzhen was obtained(IRB number CUHKSZ-D-20 210 002). Then, the advantages and disadvantages are summarized and discussed. The parameters of the VP-ZNN are set as: , , , , , and .

A. Demonstration With Phantom Experiments

For safety reasons, first, we conduct the experiments with an oral cavity phantom, which will help to confirm the safety of the OP-swab robot system. The oral cavity phantom has a 1:1 size of human oral cavity, which enables natural simulation of workflow across OP-swab sampling and allows us to work extremely close to conditions in practice. The procedures of experiments are as follows:

• •The phantom oral cavity is detected and segmented with the RealSense Camera, which is configured on the RFC manipulator. Because the sampling areas are the left and right tonsils and the palate area, we recognize and locate the target position by Mask R-CNN.
• •The desired sampling trajectories are obtained by online motion planning with an oral cavity constraint, in which we locate the target position in the first step and then generate the trajectories. The desired trajectories are a piecewise straight line from the palate area to the left tonsils to the right tonsils in the Cartesian space, where the relationship between each axis and time is the minimum jerk curve.
• •The robot is driven to perform the sampling tasks. In this phase, the desired joint trajectories are obtained from optimization control methods. Moreover, multiple protection mechanisms with force sensing are activated during robot operation.

The experimental scenarios of the oral cavity phantom are shown in Fig. 8. We collect the dataset for the oral cavity phantom which is trained by Mask R-CNN [17]. To obtain the category and location of the oral sampling areas, an oral visual detector is trained and obtained. First, an oral cavity image dataset is created by collecting from RealSense D435i. Second, the Mask R-CNN with a Inception v4 module as its backbone is trained on the oral cavity image dataset and an oral cavity detector is obtained. Finally, oral cavity coordinates are mapped to depth images to obtain the depth values of the oral cavity. We achieved a 95% recognition success rate on the model, and the sampling time is less than 20 s.

Fig. 8.

Demonstration with an oral cavity phantom. In our experiments, the 1:1 human oral cavity is tested.

B. Demonstration With Volunteers Experiments

On the basis of many trials of the OP-swab RFC robot system with the oral cavity phantom, the robustness and safety were significantly improved. Then, we conducted a number of volunteer experiments and achieved milestone significance for the future clinical application of OP-swab sampling. The procedures of experiments are the same as those described in Sec. IV-A. The dataset of the human oral cavity is collected by 29 volunteers and trained using Mask R-CNN.

The experimental scenarios of different subjects are shown in Fig. 9. The desired trajectories are the same as in Section IV-A. We design the piecewise straight-line trajectories from the palate to the right tonsils to the left tonsils in the Cartesian space, and the relationship between each axis and time is the minimum jerk curve. Fig. 10 shows the tracking results associated with one of the trials of planning trajectories. The motion trajectories in the Cartesian space are shown in Fig. 10(a). Figs. 10(b) and 10(c) show that the joint trajectories located in the physical limits and joint velocity are smooth within the velocity limits, respectively. Fig. 10(d) shows the tracking error in the Cartesian space, where all errors rapidly converge to a small value (0.02 mm). In addition, we added a comparison experiment shown in Fig. 10(d), where the proposed method has a faster convergence rate and smaller errors than [10]. The average recognition rate, control precision, sampling time, and sampling contact force are recorded and shown in Table. I. The comparative experiments were conducted to examine the effectiveness of robotic OP-swab sampling. The swab quality was verified according to the threshold cycle (Ct) value of the selected reference gene (RNase P) by the RT-PCR test [18]. Fig. 11 shows the RT-PCR test results compared with those for the manual scheme. OP-swab with Ct values of and are considered as qualified and unqualified samples, respectively, and the details are shown in Table II. The test results show that the samples are qualified.

Fig. 9.

Sampling process from vision detection to sampling tasks. In our experiments, many volunteers were tested, and the recognition rate, control precision, sampling time, and sampling contact force were recorded.

Fig. 10.

Tracking results: the manipulator track of the desired trajectories (sampling tasks) in the Cartesian space. (a) The 3D trajectories of the manipulator; (b) the joint angle trajectories; (c) the joint velocity trajectories (located in bounds); and (d) comparison experiments with [10].

TABLE I. Average Sampling Parameters.

Metric Types	Phantom	Volunteers
Recognition rate	95%	93%
Control precision	0.2 mm	0.2 mm
Sampling time	18 2 s	20 4 s
Sampling force	150 mN	150 mN

Fig. 11.

RT-PCR test results.

TABLE II. Ct Distribution.

Ct	(24-27)	(27-30)	(30-33)	(33-37)	>37	Qualified rate
Manual	11	16	3	0	0	100%
Robotic	5	21	3	0	1	96.67%

V. Conclusion and Future Work

In this study, we designed a 9-DOF redundant RFC robot to assist the COVID-19 OP-swab sampling. Moreover, we formulate the sampling tasks, physical limits, and OCC constraints as a novel optimization problem. Then, a VP-ZNN is proposed to solve the multi-constraint optimization problem online. The experimental results of the oral cavity phantom and volunteer experiments demonstrate the effectiveness of the designed robot system and proposed control methods. The average sampling time on phantoms and volunteers are 18 s and 20 s respectively. To maintain the sterility of the arm and effector between patients, the MPA is designed as a disposable device with a quick connector that is easily to be replaced, where the disposable protective film is attached to the MPA. In the future, we will focus on improving the robustness of the system and move forward to clinical testing.

Acknowledgment

The authors would like to thank the Prof. Xi Zhu, Mr. Yi Liang, Ms. Kerui Yi and Mr. Xiaochuan Lin for their suggestions and engineering contributions to this work.

Funding Statement

This work was supported in part by the Chinese University of Hong Kong, Shenzhen, in part by the Shenzhen Institute of Artificial Intelligence and Robotics for Society, in part by the Shenzhen Fundamental Research under Grants JCYJ20180508162406177, JCYJ20190806142818365 and JCYJ20210324115604012, in part by the National Natural Science Foundation of China under Grants U1613216, 62006204, and 61903100 from the Chinese University of Hong Kong, Shenzhen.