A collaborative robot for COVID-19 oropharyngeal swabbing

Abstract

The coronavirus disease 2019 (COVID-19) outbreak has increased mortality and morbidity world-wide. Oropharyngeal swabbing is a well-known and commonly used sampling technique for COVID-19 diagnose around the world. We developed a robot to assist with COVID-19 oropharyngeal swabbing to prevent frontline clinical staff from being infected. The robot integrates a UR5 manipulator, rigid–flexible coupling (RFC) manipulator, force-sensing and control subsystem, visual subsystem and haptic device. The robot has strength in intrinsically safe and high repeat positioning accuracy. In addition, we also achieve one-dimensional constant force control in the automatic scheme (AS). Compared with the rigid sampling robot, the developed robot can perform the oropharyngeal swabbing procedure more safely and gently, reducing risk. Alternatively, a novel robot control schemes called collaborative manipulation scheme (CMS) which combines a automatic phase and teleoperation phase is proposed. At last, comparative experiments of three schemes were conducted, including CMS, AS, and teleoperation scheme (TS). The experimental results shows that CMS obtained the highest score according to the evaluation equation. CMS has the excellent performance in quality, experience and adaption. Therefore, the proposal of CMS is meaningful which is more suitable for robot-sampling.

1. Introduction

The outbreak of coronavirus disease 2019 (COVID-19) has had a global impact, resulting in millions of deaths. To control the spread of COVID-19, oropharyngeal (OP) swabbing is commonly used to collect samples for diagnosis [1]. However, preventing the infection of medical staff during the sampling process due to interpersonal susceptibility through respiratory droplets and contact in an unprotected environment has become a new challenge [2]. Several studies have reported that respiratory droplets, feces, and urine are transmission routes [3], [4]. Robots can play a significant role in addressing these issues. Considering the high infection risk of COVID-19, an OP-swabbing robot can assist medical staff in collecting samples remotely. This would be especially convenient and cost-effective for large-scale deployment and standardized sampling. [5] reported that OP-swabbing robots can address the lack of qualified healthcare workers to speed up the sampling process. [6] developed a semiautomatic OP-swabbing robot for teleoperation that has a 97% success rate in clinical applications. The OP-swabbing robot mainly comprises a snake-like manipulator, two haptic devices for teleoperation, an endoscope for visualization of oral cavity, and a force-sensing system for safety. The entire OP-swabbing process is teleoperated by an expert, with one hand holding the Omega.3 haptic device to control the tongue depressor and the other hand holding the Omega.6 haptic device to operate the manipulator. Wang et al. [7] designed a low-cost robot to assist with OP-swabbing, where the swab gripper is attached to a rotation link with an extruded active two-degrees-of-freedom (DOF) end-effector for actuating the swab and a generic 6-DOF passive arm for global positioning. The swapping robot for COVID-19 diagnosis was developed by a team from the University of Southern Denmark and the Lifeline Robotics Company [8]. The robot integrating a UR5 manipulator, visual system, and dexterous and a rotatable rigid connector was demonstrated to perform the OP-swabbing task quickly and automatically.

Because the human throat is fragile and easily injure, medical safety remains a critical issue for OP-swabbing. Although the above end-effectors obtained desirable results, they were designed with rigid body structures. Thus, they may cause physical injuries to the oral cavity because of improper operation or other medical malpractices. Motivated by [9] work, a novel micro-pneumatic actuator (MPA) is developed for throat swabbing, as well as integrated force sensor for safe, stable, and reliable sampling. Force-sensing actuation offers compliance to mitigate shocks, which is mainly useful for interactions with the oral cavity.

In addition to safety, the sampling period and complexity are also significant challenges for developing a reasonable OP-swabbing schemes. The main drawback of [6] is the complex operation, which leads to an uncomfortable experience and inefficient sampling. One solution is to automate the OP swabbing process. However, automated scheme rely excessively on the visual system for positioning accuracy. It is difficult to achieve the required accuracy for detection, recognition, measurement, and control. Furthermore, robots cannot be adjusted online in some unexpected situations, such as when the positioning error is relatively large and the subject has a strong pharyngeal reflex.

In this study, we propose a rigid–flexible coupling (RFC) manipulator to assist with OP-swabbing. As shown in Fig. 1, the RFC robot comprises a UR5 manipulator, RFC manipulator, force-sensing and control subsystem, visual subsystem, and haptic device. The RFC manipulator consists of a flexible MPA and two rigid joints. At the same time, we conduct an extended study on the basis of the previous work [10], that is, we propose a collaborative manipulation scheme (CMS) to integrate the various advantages of the automatic scheme (AS) and teleoperation scheme (TS) for robot-assisted OP-swabbing. The contributions of this paper are as follows:

Fig. 1.

The detailed design and application scenario of proposed collaborative robot for OP-swab sampling. (a) Application scenario. (b) RFC manipulator. (c) Components of collaborative robot.

• 1.An intrinsically safe and accurate RFC manipulator for OP-swabbing was developed. A PI control system is proposed to improve the adaptability of AS in different oral spaces and under mild pharyngeal reflex conditions.
• 2.For accurate positioning, a visual detection and localization module is proposed that considers the spatial characteristics of the oral cavity.
• 3.An evaluation metrics for OP-swabbing robot is proposed, five typical factors are identified.
• 4.A novel control scheme named CMS is proposed. The CMS performs excellent according to the comparison experiments. The CMS has perfect adaptability, good acquisition quality, and high security, making it an efficient and feasible idea for robot-sampling control.

The remainder of this paper is organized as follows. Section 2 presents tasks analysis and evaluation metrics. In Section 3, the design, model, and control of the RFC manipulator is presented. The visual detection and localization of the oral cavity is described in Section 4. Section 5 presents the control schemes. The experiments are discussed in Section 6. Finally, Section 7 introduces the conclusions and future work.

2. Task analysis and evaluation system

2.1. Sampling area, force, and paths

According to China’s prevention and control plan for COVID-19 [11], a manual OP-swab sampling process entails wiping the pharyngeal tonsils on both sides, back and forth, at least three times, and then wiping the posterior pharyngeal wall up and down at least three times. Therefore, the sampling areas comprise the pharyngeal tonsils and posterior pharyngeal wall, as shown in Fig. 2(a). The yellow ellipses represent the left and right tonsils. The posterior pharyngeal wall, excluding the uvula is represent by blue ellipses.

Fig. 2.

Sampling areas and paths. (a) Sampling areas. (b) Sampling paths. (c) Sampling paths on 3D oral cavity phantom. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

The prevention and control plan does not offer a clear guideline for the force required for robot’s automated sampling. Recently, a team studied sampling force and found no significant difference in sample quality when the force was varied from 10–60 g, although the incidence rate of sore throats in patients was higher when the sampling force exceeded 40 g [6]. Therefore, a sampling force of $<$ 40 g was set for the robot prototype.

The preferred sampling path for OP-swabbing was also not specified in the prevention and control plan. According to the survey conducted, sampling paths differed among hospitals. Because a well-defined sampling path is necessary for robotic sampling, we sorted some sampling paths with the help of doctors, as shown in Fig. 2(b). The red and black curves represent the sampling paths for tonsils, and the blue curve represents the posterior pharyngeal wall. An oral cavity phantom with a 1:1 human scale was used to draw the space curves to obtain more information about the sampling paths in the oral cavity. The curves shown in Fig. 2(b) were drawn in a digital oral cavity phantom by three-dimensionally (3-D) scanning the oral cavity phantom, as shown in Fig. 2(c), where the range is defined by the same color areas near the curves. The spatial fluctuations of the three types of curves can be seen from Fig. 2(c), where the two types of curves of the tonsils are more complicated and changeable. Considering that the complicated and changeable sampling path can verify the performance of the robot system to a greater extent, and the posterior pharyngeal wall is more prone to occur pharyngeal reflexes than the tonsils, we chose the red curve of the tonsils as the sampling path of the experiments.

In reality, not only the spatial variation of the sampling path, but also the individual differences in oral structure and the appearance of pharyngeal reflexes must be considered. These factors have complicated the RFC robot’s sampling path. Therefore, in order to reduce the influence of these factors, the robot system should recognize and locate the target through a vision system, and realize the sampling path through the control schemes. In addition, the force constraints in robot manipulation should be guaranteed through a force sensing system.

2.2. Evaluation metrics for OP-swab Robot

To compare research findings, it is necessary to define common metrics for OP-swab robot performance evaluation. Five important factors are identified that should be taken into consideration. These common metrics are elaborated as below:

(1) Efficiency: The average sampling time is used to represent the efficiency.

(2) Quality: The threshold cycle (Ct) value is used to represent the sample quality.

(3) Experience: The test subjects are asked to evaluate the sampling experience on three scales: great, not bad, and terrible. This evaluation corresponded to scores of 10, 6, and 3, respectively. A great experience meant that the robot completed the OP-swabbing gently and efficiently. A not bad experience meant that the sampling task is accomplished without criticism and praise. A terrible experience meant that the subject felt strong discomfort or the sampling by the robot tooks a long time.

(4) Safety: The maximum sampling force is used to represented the safety.

(5) Adaption: This metric is evaluated by operator who response for clinical OP-swab sampling. This metric shows the capability of robot dealing with unexpected situation, such as strong pharyngeal reflex.

An equation for overall performance evaluation is also derived as follow,

$$Score=\sum _{i=1}^5{\lambda }_{i}Scor{e}_i$$ (1)

where $Scor{e}_{\mathrm{i}}$ and ${\lambda }_{\mathrm{i}}$ are the score and the weight of each factor, respectively.

3. Design and control of RFC manipulator

3.1. Design, fabrication, and analysis of the micro-pneumatic actuator

Bionic soft actuators have become a popular research topic recently [12], [13], [14]. A pneumatic finger, also known as a pneumatic manipulator, is a soft actuator made of a soft-shell material with air chambers inside. The air pressure in the air chambers controls the movement of the finger. Festo [13] developed a bionic pneumatic elephant trunk that showed a flexible, dexterous, and safe grasping performance. The soft and dexterous MPA proposed for use in OP-swabbing is shown in Fig. 1(c). A short swab is inserted at the end of the MPA, and a force sensor is attached to MPA. The MPA comprises two parts glued together: the cover and chamber structure. The cross-section is shown in Fig. 3(a). When the MPA is inflated, each air chamber expands like a balloon. The air chambers repel each other as they expand, causing the MPA to bend. The MPA has $n_t$ air chambers, but only half of the head and tail air chambers can bend, so the number of effective free joints is $n_f=n_t-1$. Figs. 3(b) and (c) show the MPA bending process. If all air chambers bend in the same direction under the same pressure, the geometric relation is easy to derive:

$$\theta =2\gamma /n_f,$$ (2)

where $\gamma$ is the bending angle of MPA as shown in Fig. 4(a) and $\theta$ is the bending angle of each chamber as shown in Figs. 3(c). Because of the small value of $\theta$ after bending, the arc length $c$ is approximately equal to chord length in the non-tensile layer. The MPA has a length of $l_f$ mm, and the air chambers have a thickness of $b$ mm. $m$ is the distance between the centerline and non-stretchable layer, and the arc length $c^{\prime }$ is approximately equal to the chord length in the centerline. Thus, $c^{\prime }$ is

$$c^{\prime }=c+2msin(\theta /2)$$ (3)

Fig. 3.

PneuNets structure of MPA. (a) Cross-sectional view of MPA. (b) and (c) MPA bending process.

Fig. 4.

Experimental results of MPA. (a) The deformation of MPA under different air pressures. (b) The relationship between joint angle and air pressure.

The MPA cavity structure is relatively complicated and difficult to fabricate using conventional methods. An MPA with a consistent and repeatable performance can be fabricated using a high-resolution 3-D printer. We used a 3-D light-curing printer (Lite600, Uniontech, China) that can print soft plastic (PU) materials with 0.1 mm accuracy and layer thickness of 0.05–0.25 mm. The chambers and cover of the MPA were printed and then glued together, which dimensions are as follows: $n_t=12$, $c=5.5$ mm, $l_f=90$ mm, and $b=1.5$ mm. A deformation experiment was conducted, where the gravity was perpendicular to the plane of the paper and the results are shown in Fig. 4. The air pressure is controlled by a high accuracy proportional valve ITV1050-312L with 0.1 kPa in the control framework as shown in Fig. 5 The bending angle and air pressure were found to have a positive correlation. The binomial fitting tool in origin was used to fit the relationship between the bending angle $\gamma$ [${}^{\circ }$] and air pressure $P$ [kPa] to the following binomial:

$$\begin{array}{c}\gamma =7.514265\times 10^{-5}\times P^2+0.141515P-0.10571\hfill \end{array}$$ (4)

Fig. 5.

Control scheme of MPA.

The binomial correlation coefficient $R$-Square(COD) of the experimental results was 0.99926, and the standard deviation $S$ was 0.06244. Thus, Eq. (4) can be used to predict the degree of bending of the MPA.

3.2. Modeling and workspace analysis for RFC manipulator

The RFC manipulator has three degrees of freedom (DOFs), which include linear motor (prismatic joint), servomotor (revolute joint), and MPA that can change the distance of the throat swab offset center. The overall structure is shown in Fig. 1(b). The relationship between the movement position in space and the control variables was analyzed to develop a kinematic model to control the RFC manipulator. The kinematic model comprises three mappings, as shown in Fig. 6. The device-actuator space mapping ($f_3$) is essential for robot control and can be derived from the actuator-task space mapping ($f_1$) and task-device space mapping ($f_2$). The coordinate frame setting and nomenclature are shown in Fig. 7(a). The actuator-task space mapping ($f_1$) is given by

$$\left\{\begin{array}{c}{x}^t=\left[dsin\left(n_f\frac{\theta }{2}\right)+hsin\left(\beta \right)\right]cos\left(\alpha \right)\hfill \\ y^t=\left[dsin\left(n_f\frac{\theta }{2}\right)+hsin\left(\beta \right)\right]sin\left(\alpha \right)\hfill \\ z^t=\left[dcos\left(n_f\frac{\theta }{2}\right)+hcos\left(\beta \right)\right]+z_p\hfill \end{array}\right)$$ (5)

where $\beta =2\gamma$, $\theta$ is derived from Eq. (2) (4), and $d$ is derived from the geometric relationship in Fig. 7 as

$$d=\frac{c^{\prime }sin(n_f\theta /2)}{sin(\theta /2)}$$ (6)

where $c^{\prime }$ is derived from Eq. (3).

Fig. 6.

Mapping relationships of RFC manipulator.

Fig. 7.

(a) Coordinate frame setting and nomenclature. (b) The reachable workspace of MPA with revolute joint and workspace of RFC manipulator.

The teleoperation scheme’s control mode for the RFC manipulator is incremental, where the end-effector is not controlled by the knob’s position but by the stroke in each movement cycle when the button is pressed. The inverse mapping ($f_2^{-1}$) is

$$\left\{\begin{array}{c}\hfill x_i^t=x_{i-1}^t+k[x_i^T-x_{i-1}^T]\hfill \\ \hfill y_i^t=y_{i-1}^t+k[y_i^T-y_{i-1}^T]\hfill \\ \hfill z_i^t=z_{i-1}^t+k[z_i^T-z_{i-1}^T]\hfill \end{array}\right)$$ (7)

where $x^T$ is the desired position and the vector $k$ includes the scaling factors. The RFC manipulator’s workspace was drawn using the established kinematic model (length h of 50 mm). The maximum radius of rotation was 87.8 mm, as shown in Fig. 7(b). This is sufficient to meet the spatial requirements for swabbing the human oral cavity.

3.3. Force sensing and calibration

To detect the radial force, a strain gauge (BF350-3AA23T0) was used and installed as shown in Fig. 1(c). We built a Wheatstone bridge and designed a set of operational amplifier circuits with AD623, which can convert resistance to the output voltage. The AD conversion module acquires the output voltage and sends it to the ROS system. The current output voltage $U_{now}$ is related to two factors: the voltage $\Delta U$ generated by the force and the voltage $U_{F=0}$ caused by the air pressure without a load. Therefore, we can obtain

$$\Delta U=U_{\mathrm{now}}-U_{F=0}$$ (8)

We drove the MPA from 0 to 150 kPa in intervals of 10 kPa and recorded the output voltage. After 20 groups of experiments, the relationship between the no-load voltage and air pressure was obtained by linear fitting:

$$U_{F=0}=-4.8809P+U_1\left(mV\right)$$ (9)

where $U_1$ is the initial voltage without air pressure. Force calibration was performed using the electronic balance (CX2000). The experimental setup is shown in Fig. 8(a). At different air pressures, the MPA bent and generated the target contact force, and we recorded the current force $F$ and voltage $U_{now}$. The voltage $\Delta U$ caused by the current force $F$ was then calculated using Eqs. (8), (9). The calibration results are shown in Fig. 8(b). The relationship between $F$ and $\Delta U$ is approximately linear, and we obtained

$$F=0.0937\Delta U+0.7507$$ (10)

Fig. 8.

Force calibration. (a) Setup. (b) Measure results.

The correlation coefficient $R$ was 0.9982, and the standard deviation $S$ was 0.6027.

3.4. PI control system

A PI control system with a real-time force feedback function is designed to allow the automatic scheme to adapt to different cavity space shapes. Fig. 9 presents the details. The control system contains the planner, controller, actuator, manipulator, and sensor. The planner consists of an inverse kinematics module, which is responsible for converting the coordinates in the Cartesian kinematic space into a control quantity for each joint. For the control of MPA, it is necessary to convert the bending angle $\theta$ into air pressure. The PI controller compares the deviation between the operating and measured values and adjusts it using $K_p$ and $K_i$ parameters to achieve stable and reliable closed-loop control. Actuators 1–3 are pneumatic drivers, rotary servo, and linear motors, which control joints 1–3. Thus, the RFC manipulator is controlled in a closed-loop.

Fig. 9.

Control framework of RFC manipulator.

4. Visual subsystem

In this section, for accurate positioning, a visual detection and localization module is proposed that considers the spatial characteristics of the oral cavity, which contains a visual network, as shown in Fig. 10.

Fig. 10.

The oral detection, segmentation, and localization network for visual subsystem.

Visual sensors are mainly used in the visual part of the system to detect and segment the sampling areas inside the oral cavity, and to locate the sampling points according to the oral structure. The centers of the oral cavity and oral sampling areas such as tonsils and the posterior pharyngeal wall need to be obtained for visual localization. Therefore, these areas need to be detected. However, direct detection and segmenting of the tonsils and posterior pharyngeal wall are extremely difficult. This is because the tonsils and posterior pharyngeal wall have fewer features and their edges are not clear enough to be detected and segmented accurately. Therefore, the detection and segmentation parts of the oral cavity are designed as shown in Fig. 11(a). There are three labels, namely mouth (green area), sampling area (red area), and uvula (yellow area). The mouth can determine the position of the oral cavity, the sampling area can determine the left and right tonsils, and the uvula can determine the position of the posterior pharynx. Furthermore, because these areas have many features, they can be detected and segmented more easily.

Fig. 11.

Schematic diagram of oral segmentation labels and Oral position relationship. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

The oral detection and segmentation network uses Mask R-CNN [15], which is a type of instance segmentation network, but the Mask R-CNN has a complex structure and it lacks real-time performance. In this study, real-time performance is critical for oral detection and localization. Therefore, we improved the Mask R-CNN and introduced an Inception v2 [16] module to its backbone. The Mask R-CNN Inception v2 network structure is composed of Inception v2, Region Proposal Network (RPN), Fully Convolutional Networks (FCN), and fully connected layers (FC), as shown in Fig. 10. Among them, Inception v2 is used as the network’s backbone for image feature extraction. The target regions are selected using RPN. The FC part is used for the detection and localization of oral areas, and the FCN part is used for the semantic segmentation of oral areas. Furthermore, Inception v2 is used to replace the ResNet/FPN part of the original Mask R-CNN. It is a deformed network of GoogLeNet [17]. To reduce calculation, its structure uses a separable convolution with a depth multiplier of eight. It can effectively increase the network’s processing speed. Additionally, the depth information output by the RealSense camera has also been refined using a filter to smooth the edges of the depth images and averaging the depth values of the same pixel in multiple frames.

After detecting and segmenting the sampling area of the oral cavity, the results can be used to locate the sampling points to provide position support for the robot arm motion-planning path. The position of the posterior pharyngeal wall is directly below the uvula (red dot), as shown in Fig. 11(b). The positions of the left and right tonsils are in the lower left and right corners of the ${\mathrm{S}}_{\mathrm{area}}$ (green dots). The position of the mouth is in the center of the mouth (black dot).

The above network was trained and verified. A total of 20 volunteers was collected to make an oral dataset. A total of 50,000 images containing the mouth, ${\mathrm{S}}_{\mathrm{area}}$, and uvula labels was produced, of which 40,000 were used as the training set and 10,000 were used as the test set. The results of verification on the test dataset are shown in Table 1 and Fig. 12.

Table 1.

Results of image segmentation and measurement algorithm.

Accuracy (%)	mouth	S_area	uvula	average
	99.27	98.84	92.88	97
Speed (ms)	120
Error (mm)	1.3

Fig. 12.

Example of oral sampling area detection and localization.

Table 1 shows that using the above-mentioned oral image algorithm, the average detection accuracy of oral parts can be obtained at 97.00%, which meets the system’s detection requirements. The algorithm processes each image (640 × 480 pixels) at a speed of 120 ms, which meets the systems’ real-time requirements. Additionally, the average measurement error of the sampling location is 1.3 mm, which also meets the system’s measurement accuracy. Therefore, the oral visual detection and localization method designed in this study can meet the requirements of an automated oral swab-sampling robot.

5. Control schemes

The proposed OP-swabbing system has two main control schemes: the TS [6] and AS [8]. We applied these two schemes as well as the proposed CMS to our robot.

5.1. TS and AS

For the TS, an expert controlled the haptic device Touch throughout the sampling process. The OP-swabbing with a 9-DOF robot was divided into two phases. First, touch mapping was used to control the 6-DOF UR5 robot’s arm from the initial position to near the tonsil. Then, the touch mapping was switched to the 3-DOF RFC manipulator. The expert controlled the manipulator to swab the tonsils.

For the AS, the entire motion planning is conducted in the order of numbering in the figure, with the light green, light blue, and black curves representing the first, second, and third trajectories, respectively, and the two red curves representing the sampling paths of the left and right tonsils, respectively, as shown in Fig. 13. The first trajectory is from the robot’s initial position to the set point near the left tonsil, the second trajectory is from the end point of the sampling path of the left tonsil to the set point near the right tonsil, and the third trajectory is from the end point of the sampling path of the right tonsil back to the robot’s initial position.

Fig. 13.

Motion trajectories and sampling paths.

After object detection in the oral cavity by the depth camera (Realsense D435i), the tonsil coordinates are used as the landmark of the sampling path. The procedures of the AS are as follows:

• •According to the obtained coordinates of the left tonsil, the set point, which is the end point of the first trajectory, are set near the left tonsils.
• •With the MPA’s bending direction is to the left and the air pressure is zero, the robot runs the first trajectory. After reaching the set point, the MPA is brought into contact with the surface of the tonsils by inputting air pressure, which makes the starting point of the sampling path obtained.
• •After setting a constant force value and displacement of the servo linear motor whose moving direction parallel to the $Z$-axis direction, the constant force control system is turned on, and the sampling path of the left tonsil is completed with the fixed angle of rotating motor which make the $y$ coordinate unchanged.
• •Repeat the preceding steps to make the robot run the second trajectory and the sampling path of the right tonsil.
• •Finally, run the third trajectory to end the sampling process.

The spatial constraints of the oral cavity space must be considered for motion planning, to avoid human injury during the sampling process. To constrain the end-effector of the OP-swabbing robot and prevent collision with the oral cavity, the remote center of motion (RCM) technique used for minimally invasive surgical robots [18], [19] was used. Although mechanical implementation is safer for complex structures and calibration procedures, a programmable RCM based on a control algorithm is more reasonable and flexible [20]. In our previous work [10], the second to the last link of the OP-swab sampling robot is constrained with the oral cavity center (OCC) constraint, which guarantees the absence of collision between the inserted end effector with the oral cavity. What is more, a varying-parameter zeroing neural network (VP-ZNN) is applied to solve the optimization problem.

5.2. CMS

As we know, TS and AS have their own advantages and disadvantages. TS has the advantages of high adaptability, especially for abnormal conditions such as the pharyngeal reflex. However, the main disadvantage of TS is the complicated operation, which leads to an uncomfortable experience and inefficient sampling. AS has the advantages of high efficiency and stable sampling intensity. However, AS still has shortcomings, mainly the high precision requirements for object detection and robot control, which leads to poor adaptability and fault tolerance of the robot system. Therefore, we propose a collaborative manipulation scheme (CMS) which combines the AS and TS. As shown in Fig. 14, the procedure of CMS comprises two phases: autonomous movement and teleoperation. In the autonomous phase, the robot is driven close to the uvula or tonsil based on visual information and OCC-based control, which reduces the operation time and complexity. In the teleoperation phase, the expert uses the haptic device to control only the RFC manipulator to sample the tonsils in the oral space, which make the operation simple. CMS combines the advantages of automation and teleoperation, which has perfect adaptability, good sampling experience and high efficiency.

Fig. 14.

The control workflow of RFC robot with collaborative manipulation scheme.

The similarities and differences of the three schemes are shown in Table 2.

Table 2.

Similarities and differences of the three schemes.

	Visual	Human operation	Phase1	Phase2
TS	✗	✓	Teleoperation	Teleoperation
AS	✓	✗	Automatic	Automatic
CMS	✓	✓	Automatic	Teleoperation

6. Experiments

6.1. RFC manipulator performance experiments

6.1.1. Intrinsic safety

The OP-swabbing working area is sensitive, and humans are prone to react during the sampling process. Therefore, the sampling tool must take into account the impact of human reactions and must not cause human injury. We defined intrinsic safety as a sampling tool’s characteristics that enable it to avoid human injury in the event of an impact. We tested the performances of different sampling tools, such as RFC manipulators and ordinary swabs under an impact that was simulated as a displacement.

The sampling tool was positioned vertically downward, as shown in Fig. 15(a). The force gauge was fixed to the horizontal table to determine the contact force of the endpoint, and the linear motor controlled the RFC manipulator or swab by gradually increasing the displacement. Each sampling tool is tested several times. The experimental results are shown in Fig. 15(b). The blue line represents the relationship between the force and displacement of the RFC manipulator. Even as the displacement increased, the force increase maintained a gentle slope until it reached the limit value. The purple line represents the swab results, which exceeded the measurement range of the force gauge (2000 mN) at the next point. The force changed from 43 to 1534 mN within a displacement of 1.92 mm (30.72–32.64 mm), whereas the blue line increased from 38 to 816 mN within a displacement of 18.4 mm (30.84–49.24 mm). Compared with the swab, the RFC manipulator’s contact force was limited to a safe range. Thus, the RFC manipulator can effectively resist impacts and avoid accidents.

Fig. 15.

Intrinsic safety experiment.

6.1.2. Repeatability positioning accuracy experiment

The repeatability of the positioning accuracy was tested to verify the reliability and accuracy of the MPA. A paper with 1 mm coordinates was stuck on a flat wooden board. A straight wire with a diameter of about 0.2 mm was fixed at the end of the MPA. The camera was fixed approximately 20 cm above the coordinate paper. To ensure no friction between the paper and MPA, the wire was kept approximately 2 mm away from the paper. A picture was taken with a fixed camera at a pressure of 250 kPa. To eliminate the influence of gravity on the experimental results, the direction of gravity was perpendicular to the paper surface, as shown in Fig. 16(a). The air pressure was increased from −50 kPa to 0, 50, 100, 150, 200, and 250 kPa in sequence. The pressure was maintained at each level for more than 10 s, and photos of the MPA in a curved state were taken. Each air pressure was tested 10 times, and the coordinates were recorded. A minimum circle enclosing method was used to enclose all the coordinates. The radius of the enclosing circle was defined as the repeatability error.

Fig. 16.

Repeatability experiment.

As shown in Fig. 16(b), the result shows that the maximum error of MPA is ±0.8mm at 250 kPa, which proves that it has high repeatability positioning accuracy and meets the robot sampling requirements.

6.2. Radial constant force control experiments

In this subsection, we introduce the constant force control experiments. The difference between the operating force $F_{set}$ and measured force $F_{now}$ is $e_1\left(t\right)$, and therefore

$$e_1\left(t\right)=F_{\mathrm{set}}-F_{\mathrm{now}}$$ (11)

The constant force control formula is as follows.

$$\Delta P_{out}=K_P{e}_1\left(t\right)+K_I{\int }_0^t{e}_1\left(t\right)dt$$ (12)

As shown in Fig. 17(a), we use the different size of 3D printed tonsil models represent the various oral geometry. We created four tonsil models that are 0.75, 1, 1.25, and 1.5 times that of humans. Which represent the differences in oral space between individuals, such as enlarged and smaller tonsils, and oral size of adults and adolescents. First, we sprayed ink on the throat swab and set the constant force to 2 g. The model is placed on an electronic balance, and a recording camera is fixed above it. Then the RFC manipulator sampling on the four models with red curves as we showed in Fig. 2(c). After the robot runs to the set point which near the tonsils, the air pressure provided the constant force control, the servo motor maintained a fixed angle, the required path can be completed only by retreating the linear motor for a distance. The sampling results is shown in Fig. 17(b), four paths on different tonsil models are clearly and continuous, which proves that with the help of constant force control, the autonomous scheme could adjusting the various oral geometry among different individuals. Furthermore, we record the sampling force provided by the electronic balance (a) and the force provided by the sensor (s), which represent the actual force and sensor force respectively. The constant force control performance of 0.75, 1, 1.25, and 1.5 models is shown in Fig. 17(c), (d), (e), and (f). The blue line represents the actual force, the orange line represents the sensor force, the gray line represents the desire force. The maximum error of force control is 0.41 g, and the error between the sensor’s measured value and the actual value is 0.25 g.

Fig. 17.

Constance force control on model 0.75,1,1.25,1.5.

6.3. Control schemes comparison experiments

We followed the evaluation metrics in Section 2.2 to compare three schemes. In this experiment, sixteen volunteers (nine males and seven females) were sampled in three schemes and manual. The group contains male and female of five ages ranging from 10–20, 20–30, 30–40, 40–50, and 50–60, respectively. After each sampling, we recorded the sampling time, maximum sampling force, and experience.

When the sampling was completed, 240 samples were applied to the COVID-19 Real-Time PCR test with QuantStudio™ 5 in the clinical laboratory at the People’s Hospital of Long-gang District, Shenzhen, China. The swab quality was determined based on the Ct value of the selected reference gene (RNase P) in the RT-PCR test [11], [21]. For quality, Ct values of less than and greater than 45 were considered to indicate qualified and unqualified samples, respectively, according to the specification from the Shanghai BioGerm Medical Technology Co., Ltd, which provided the Nucleic Acid Detection Kit. Besides, if the RT-PCR test result is undetermined, it means this sample is invalid and failed.

Fig. 18 shows the experimental results. The time distribution is shown in Fig. 18(a), the sampling time of the AS was mainly distributed in 40–60 s, and it was significantly shorter than that of the other two schemes. Fig. 18(b) and Fig. 18(c) is refer to distribution and box plot of force, respectively. These two figures show that the collection intensity of CMS and TS is relatively scattered, while the AS is stable and concentrated. In Fig. 18(d), the Ct value of three schemes and manual is similar, which means that in our 240 samples, the quality of the robot sampling was as good as that of the manual sampling, and the automatic sampling was as good as other two modes which operated by expert. Fig. 18(e) display the distribution of experience.

Fig. 18.

Comparison experiments results of three control schemes.

Overall, Table 3 presents the distribution for the factors. The efficiency score of TS is set as 60 point, which is considered as a pass line;CMS with a 73 s average sampling time achieved a score of 75 points; and AS achieved a score of 85 points. For quality, the score of quality is equal to the quaand AS achieved a score of 85 points. For the score of safety, The three schemes of AS, CMS, and TS achieved scores of 90, 80, and 70 points for safety, respectively, as the maximum average force increased. Additionally, the experience score is the ten-point system changed to a hundred-point system. Finally, for evaluation of adaptation metric, TS and CMS achieved a score of 100 point. TS and CMS not only have good adaptability for different sizes of oral space and people but also can deal with various emergencies. Although AS can handle slight pharyngeal reflex, it falls short in other unexpected situations, such as strong pharyngeal reflex. Therefore, AS achieved a score of 80 point in adaptation. Fig. 18(f) shows three schemes scores in each factor.

Table 3.

Experimental data distribution.

	Time					Ct value				Experiences			Maximum force
	$\le$40	(40-60]	(60-80]	(80-100]	(100-120]	$\le$27	(27-32]	(32-37]	Invalid	Terrible(3)	Not bad(6)	Great(10)	$\le$2	(2.01-4)	(4.01-6)	(6.01-8)	(8.01-10)
AS	8	52	0	0	0	8	40	12	0	8	27	25	0	60	0	0	0
TS	0	2	21	20	17	4	48	8	0	7	41	12	5	24	19	9	3
CMS	0	13	30	14	3	1	50	9	0	5	30	25	3	29	22	5	1
M	60	0	0	0	0	1	8	51	0	15	29	16	–	–	–	–	–

The average and score in each factors is shown in Table 4. The overall score with proposed control schemes were also calculated. The weight of these five factors ranges from ${\lambda }_1$-${\lambda }_5$. In this study, each of factor has the same weight because these five factors are equally important in our opinion, hence, we chose 0.2 for ${\lambda }_1$-${\lambda }_5$. Therefore, CMS received a score of 85.8, and AS received a score of 85.6, indicating that both of these schemes performed well when the five factors were considered. CMS has an advantage in terms of adaptation, but a disadvantage in terms of efficiency and safety. AS has an advantage in terms of efficiency and safety, but a disadvantage in terms of adaptability. For TS, the control should be optimized to reduce the time, which got a score of 79.

Table 4.

Experimental data average and score (${\lambda }_{\mathrm{1}}$-${\lambda }_{\mathrm{5}}$$=$ 0.2).

	Time		Quality		Experience		Force		Adaption	Score
	Average (s)	Score1	Qualified rate (%)	Score2	Average	Score3	Average (g)	Score4	Score5
AS	46	85	100	100	7.3	73	2.3	90	80	85.6
TS	91	60	100	100	6.5	65	4.5	70	100	79
CMS	73	75	100	100	7.4	74	3.9	80	100	85.8
M	6	–	100	–	6.3	–	–	–	–	–

Compared with the robot system (RS) proposed by [6], the CMS had a longer sampling time. The CMS had an average maximum sampling force of 3.9 g during the 60 sampling experiments, which is much less than that of the RS (10–60 g). The gentle sampling force of the RFC robot will not have any effect during the sampling of elastic surfaces in tonsils. The RS has an advantage in terms of efficiency, whereas the CMS has the advantage in terms of safety and user experience. The manual samples obtained a qualified rate of 100% which is considered as a reference. The CMS required a much longer sampling time than a manual collection but offered a significantly better experience because of the soft and light MPA. Finally, the OP-swabbing robot can separate the subject and operator to effectively prevent the spread of the virus and protect the medical staff.

7. Conclusions and future work

In this study, a novel and intrinsically safe RFC manipulator for OP-swabbing was developed. A visual detection and localization module was designed for the recognition and positioning task in the oral cavity and a CMS was developed for high quality, adaptive and accurate robot-assisted OP-swabbing. Based on the proposed equation for comparison on three schemes, CMS performed excellent when combining five factors. Furthermore, the MPA disinfection scheme is first disinfected with alcohol and then apply a disposable protective film. For future study, the precision of the force-sensing system will be improved, and a multi-dimensional force control system will be developed. Second, the MPA can be improved by developing a bidirectional bending actuator. Third, a visual servo module will be developed to realize real-time recognition and positioning. Finally, to maintain the sterility of the arm and effector between patients, the MPA will be designed as a disposable device with an easily replaceable quick connector.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Biographies

Yongquan Chen received his Ph.D. degree from the Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China in 2014. He is currently a Research scientist in Institute of Robotics and Intelligent Manufacturing, The Chinese University of Hong Kong, Shenzhen, and director of Research Center on Unmanned Systems(RCUS) of Shenzhen Institute of Artificial Intelligence and Robotics for Society, Shenzhen, China. His current research interests include design, sensing and control of robotics system and multi-agent systems.

Qiwen Wang received his B.S. degree in Measurement and Control Technology and Instruments, Hunan University of Science and Technology, China in 2015. He is currently a Research assistant in Shenzhen Institute of Artificial Intelligence and Robotics for Society (AIRS). His research interests include soft actuator, design and control of robotics system.

Chuliang Chi received his B.E. degree in Mechanical Design Manufacture and Automation from College of Mechanical Engineering, Dongguan University of Technology, China in 2019. He is currently a Research assistant in Shenzhen Institute of Artificial Intelligence and Robotics for Society (AIRS), Shenzhen, China. His current research interests include mechanical design, sensing and control of robotics system.

Chengjiang Wang received the B.E. degree in Mechatronics engineering, from Changchun University of Science and Technology, Changchun, Jilin, China in 2010, and the M.E. degree in Mechatronics engineering, from Harbin Institute of Technology, Shenzhen China in 2012. He is currently pursuing a Ph.D. degree in Computer and Information Engineering. His research interests include spherical robot, computer vision, power line inspection robot, and redundant manipulator.

Qing Gao was born in Tangshan, China. He received his Ph.D. degree in the State Key Laboratory of Robotics, Shenyang Institute of Automation (SIA), Chinese Academy of Sciences (CAS), Shenyang, China. Currently, he is working in Shenzhen Institute of Artificial Intelligence and Robotics for Society (AIRS), The Chinese University of Hong Kong, Shenzhen (CUHKSZ), Shenzhen, China. His research interests include robotics, artificial intelligence, machine vision and human–robot interaction.

Heng Zhang received the B.E. degree in 2017, and M.S. degree in 2020, both in Electrical Engineering and Automation, from Taiyuan University of Technology, Taiyuan, Shanxi, China. He is currently a visiting student in Shenzhen Institute of Artificial Intelligence and Robotics for Society, Guangdong, China. His research interests include design, sensing and control of logistics robotics system and medical robots.

Zheng Li is a senior member of IEEE, member of RAS and ASME. He received the Ph.D. degree in robotics from The Chinese University of Hong Kong, Hong Kong in 2013. After working as a research fellow in National University of Singapore, he returned to The Chinese University of Hong Kong in 2015. Currently, he is an Associate Professor with the Department of Surgery, Multiscale Medical Robotics Centre and Chow Yuk Ho Technology Centre for Innovative Medicine. His research interests include design, kinematics modeling, sensing and control of flexible and soft robots for medical applications.

Zonggao Mu received the B.E. degree in Mechatronics engineering, from Shandong University of Science and Technology, Qingdao, Shandong, China in 2011, and the M.E. degree in 2013 and the Ph.D. degrees in 2018, both in Mechatronics engineering, from Harbin Institute of Technology, Shenzhen China. He was a Postdoctoral in control engineering in Harbin Institute of Technology and a Visiting Scholar in The Chinese University of Hong Kong, Hong Kong, China. He is currently an Associate Professor with the School of Mechanical Engineering, Shandong University of Technology, Zibo, China. His current research interests include the planning and control of space robotics, medical robots and hyper-redundant manipulators.

Ruihuan Xu received his PhD degree in Clinical Test and Diagnosis from Peking Union Medical College in 2009. She worked in clinic lab as a directer of District People’s Hospital of Shenzhen City. She also is a doctoral supervisor in clinical laboratory diagnostics of Guangzhou University of traditional Chinese medicine. Her research interest is psychoneuroimmunology. In this work she is responsible for 2019-nCov RNA detection.

Zhenglong Sun received his Ph.D. degree from Nanyang Technological University (NTU), Singapore in 2013. He is currently an Assistant Professor in the School of Science and Engineering at the Chinese University of Hong Kong, Shenzhen, China. His research interests include medical robotics, human robot collaboration and multi-modal perception. He is a senior member of IEEE, and has published over 50 journal and conference publications in the fields of robotics and biomedical engineering.

Huihuan Qian received Ph.D. degree from the Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China in 2010. From 2010 to 2015, he was with The Chinese University of Hong Kong, Hong Kong SAR, China. He is currently an Assistant Professor in the School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, and Vice President in Shenzhen Institute of Artificial Intelligence and Robotics for Society, Shenzhen, Guangdong, China. His current research interests include robotics, human–machine interaction, and energy.

Appendix A. Supplementary data

The following is the Supplementary material related to this article.

Video S1

This video mainly introduces experimental demonstrations of a collaborative robot for COVID-19 oropharyngeal swabbing.