Research on acupuncture robots based on the OptiTrack motion capture system and a robotic arm

Abstract

OBJECTIVE:

To propose an automatic acupuncture robot system for performing acupuncture operations.

METHODS:

The acupuncture robot system consists of three components: automatic acupoint localization, acupuncture manipulations, and De Qi sensation detection. The OptiTrack motion capture system is used to locate acupoints, which are then translated into coordinates in the robot control system. A flexible collaborative robot with an intelligent gripper is then used to perform acupuncture manipulations with high precision. In addition, a De Qi sensation detection system is proposed to evaluate the effect of acupuncture. To verify the stability of the designed acupuncture robot, acupoints’ coordinates localized by the acupuncture robot are compared with the Gold Standard labeled by a professional acupuncturist using significant level tests.

RESULTS:

Through repeated experiments for eight acupoints, the acupuncture robot achieved a positioning error within 3.3 mm, which is within the allowable range of needle extraction and acupoint insertion. During needle insertion, the robot arm followed the prescribed trajectory with a mean deviation distance of 0.02 mm and a deviation angle of less than 0.15°. The results of the lifting thrusting operation in the Xingzhen process show that the mean acupuncture depth error of the designed acupuncture robot is approximately 2 mm, which is within the recommended depth range for the Xingzhen operation. In addition, the average detection accuracy of the De Qi keywords is 94.52%, which meets the requirements of acupuncture effect testing for different dialects.

CONCLUSION:

The proposed acupuncture robot system streamlines the acupuncture process, increases efficiency, and reduces practitioner fatigue, while also allowing for the quantification of acupuncture manipulations and evaluation of therapeutic effects. The development of an acupuncture robot system has the potential to revolutionize low back pain treatment and improve patient outcomes.

1. INTRODUCTION

In recent years, acupuncture has been shown to be effective and efficient at treating several diseases worldwide, especially pain-related diseases.^1,2 Originating from Yinyang and Wuxing, acupuncture treatment regulates and restores the Qi and Shen of the human body through stimulating several acupuncture points corresponding to the disease.² In this way, pain can be relieved, and the diseases can be cured.

Traditionally, acupuncturists use millimeter-thin needles to stimulate acupuncture points corresponding to diseases.³.They make them De Qi with manipulations such as lifting-thrusting and twirling-rotating, which can ensure the effectiveness of the acupuncture.⁴ Then the needles are maintained in the human body for some time, during which the acupuncturists manipulate the needles at intervals to stimulate the acupuncture points.⁵ All millimeter-thin needles should be removed when the acupuncture ends for patient safety. Successful acupuncture manipulation is based on the knowledge and experience of acupuncturists.⁶ However, due to the lack of experienced acupuncturists, electroacupuncture, rather than traditional acupuncture, is widely used to alleviate the imbalance between the number of patients and the number of acupuncturists.^7,8 Electroacupuncture can only implement up and down lifting motions, which may diminish the therapeutic effect of acupuncture.⁹ Furthermore, manual needle removal may result in some needles being missed and cause harm to a patient's body.

With the rapid advancement of modern technology, automated acupuncture has become possible. An acupuncture robot is currently being developed to perform acupuncture work. The robot can perform acupuncture manipulations precisely and objectively by quantifying factors such as the depth, speed, and insert angle. Additionally, the robot can perform needle retrieval operations, reducing the workload of acupuncturists. The acupuncture robot system comprises three parts:^10,11 (a) selection and automatic location of acupoints; (b) needle insertion and acupuncture point stimulation; (c) evaluation of the De Qi sense. The development of acupuncture robots has garnered increased amounts of attention since the turn of the century. Litscher et al ^12,13 were the first to quantify the measurable effects of acupuncture, sparking research on acupuncture robotics. Zhang et al ¹⁴ digitized human acupoints based on digital meridian theory and fractal geometry and established a small-world network for selecting effective acupoints. Liu et al ¹⁵ explored the topological rules of the acupoint-indication network based on graph theory and achieved intelligent acupoint selection. However, in these studies, the selection and automatic location of acupoints were focused on, and acupuncture manipulations and the detection of acupoint simulation effects were ignored. Lan et al ¹⁶ introduced an integrated approach to locate facial acupoints in 2D images using a 3D morphable model and landmark points. They realized the coordinate transformation between the camera and the robot arm while developing approaches for detecting De Qi events. Su et al¹⁷ presented a hand-eye-force coordination system that can control the acupuncture robot, locate acupoints, estimate the force of needle insertion, and control the force during acupuncture manipulation. The limitation of this system is that the acupoints should be manually labeled. The study of specific acupuncture manipulations has been largely neglected in previous robot-assisted acupuncture systems, with researchers mainly focusing on the selection and coordination of acupoints. However, the clustering analysis method and deep-learning-based computer vision algorithms were adopted to identify and classify acupuncture manipulations, which is crucial for the quantification, objectification, and standardization of acupuncture practices.^{18,⇓,⇓,⇓-22}

In this work, the authors propose a new approach to the design of an automated acupuncture system that utilizes a motion capture system for acupoint recognition and a robot to perform acupuncture manipulations. The physician marks the acupoints for treatment based on the patient's symptoms, and the OptiTrack motion capture system recognizes the marked acupoints. The robot operating system then receives the coordinates of the acupoints and plans an appropriate acupuncture path. The robot system performs acupuncture operations, including needle pickup, needle insertion at each acupoint, and lifting-thrusting and twirling-rotating manipulations at specific acupoints to stimulate acupoints. During the acupuncture operation, the patient's speech is recognized in real time to determine whether the patient has De Qi, and the acupuncture operation is adjusted accordingly. Finally, the acupuncture robot removes all needles after the acupuncture operation.

2. METHODS

In this work, the acupuncture robot system consists of three components: automatic acupoint localization, acupuncture manipulations, and De Qi sensation detection. First, the OptiTrack motion capture system captures the world coordinates of the acupoints based on the preplaced markers. The acupoint coordinates should be converted to those in the robot control system using coordinate transformation computation. Next, the robotic arm and an intelligent gripper are designed as the acupuncturist's hand to perform the acupuncture manipulations. After the acupuncture manipulations, De Qi sensation is detected by keyword detection of patients' speech feedback, to realize real-time evaluation and adjustment of the acupuncture effect. In the acupuncture robot system, the current position feedback and the visual feedback are proposed to update the motion direction of the robot in real time, which ensures the accurate location of the acupoint. The flowchart of the acupuncture robot system is as follows.

2.1. The automatic acupoint location system based on the OptiTrack motion capture system and coordinate transformation

In the process of acupuncture treatment of diseases, different patients have different pain sites and different pathological causes.²³ The acupuncturist must first determine the acupoints according to the patient's pain location and condition. Based on the main symptoms, the corresponding treatment guideline is “to relax the tendons and activate the channels, pass through the meridians and relieve pain”. The localized Ashixue and Zutaiyang meridian acupoints are usually chosen for acupuncture.^24,⇓-26 Taking low back pain as an example, Shenshu (BL23), Dachangshu (BL25), Yaoyan (EX-B6) and Weizhong (BL40) are usually selected as the main acupoints for acupuncture treatment combining the rules of Traditional Chinese Medicine point selection and clinical practice.²⁷ In addition, there are often other meridian-related symptoms that accompany diseases. Moreover, corresponding acupoints can be used to alleviate the symptoms. The Mingmen (GV4) and Houxi (SI3) acupoints are usually selected for symptoms related to Du Meridian;²⁸ Kunlun (BL60) is often chosen for symptoms related to Bladder Meridian;²⁹ and Yaoyangguan (GV3), Geshu (BL17), Zhishi (BL52), and Taixi (KI3) are regarded as supplementary acupoints for different types of lumbago.³⁰

After the acupuncturist marks the acupoints, the OptiTrack motion capture system records the 3D world coordinates of the acupoints in real time for robot control.

2.1.1. The acupoint location based on the OptiTrack motion capture system

The OptiTrack is a real-time optical motion capture system. The motion capture system has the advantages of high precision, low latency, and real-time performance and can meet the requirements of acupoint positioning for acupuncture robots. The system we used consisted of seven OptiTrack infrared motion capture cameras, OptiHub connectors and calibration angle rulers, calibration sticks, markers, and the accompanying Motive Control software.

In this work, the OptiTrack motion capture system includes two steps: (a) system construction and (b) data acquisition and transmission. To construct the motion capture system, first, the motion capture cameras should be fixed around the field to ensure that the camera view covers the entire site and that each camera is connected to the PC with Motive Control software through OptiHub. Camera calibration is subsequently carried out to establish a three-dimensional world coordinate system. Four markers are used to determine the XOY plane. A T-shaped calibration stick with several Markers fixed on it is used for further calibration. By continuously swinging the stick in the scene, each camera can capture at least 8000 pieces of data to iteratively optimize the parameters based on the original initial values, ensuring the accuracy of the acupoint coordinates captured by the motion capture system.

Data acquisition and transmission can be performed when the system is built and calibrated. The markers will be placed near the acupoints. Each motion capture camera emits infrared light into the scene, and the special coating will reflect the infrared light on the marker. With the use of reflected infrared light, a computer can access the location of acupoints in real-time through a network cable. The introduction of the OptiTrack motion capture system allows the robot system to perform automatic acupoint locations for subjects of different body sizes and genders. The subsequent acupuncture operations are operated based on the results of the acupoint location, which indicates that the designed robot system is subject-independent.

2.1.2. The coordinate transformation system

Due to the special materials of the markers, they can be placed only near the acupoint and not on it. Thus, the position deviation between the acupoints and the markers on the y-axis should be first considered. After the deviation correction, the coordinates of acupoints in the motion capture system can be denoted as ${[X_m,Y_m,Z_m]}^{\mathrm{T}}$ . Since the motion capture system and robot control system use two different coordinate systems, coordinate transformation from the motion capture system to the robot is needed to ensure that the robot can use the coordinate information captured by the motion capture system.

In this subsection, we first calculate the transformation matrix between the two coordinate systems. We know that the transformation between these two coordinate systems is a rigid body transformation; that is, there is no deformation between them. Additionally, they have the same scale. Therefore, the transformation between two coordinate systems consists only of translation and rotation. The coordinate system transformation relationship from the motion capture system to the robot system can be described as ${[X_r,Y_r,Z_r]}^{\mathrm{T}}=R{[X_m,Y_m,Z_m]}^{\mathrm{T}}+T$ , where X, Y, and Z represent the 3D coordinates. The subscripts r or m indicate the robot coordinate system and motion capture coordinate system, respectively, while R and T are the rotation and translation matrices, respectively.

For simplicity we write the coordinates of the motion capture system and the robot as A and B, respectively, and we can obtain $B=RA+T$ . To solve for R and T, we must know the coordinates of at least three points that are not collinear in the two coordinate systems. We minimize the least squares error of the system of equations, as shown in Eq. (1).

$$error={\sum }_{i=1}^N{‖{RA}^i+T-B^i‖}^2$$ (1)

Our work can be described in the following steps:

(a) Normalization and calculation of centroids

All the coordinate values are normalized for subsequent processing. To obtain the coordinate system rotation relationship, we calculated the centroids of the two sets of coordinate points as $cenp={\sum }_{i=1}^N{p}^i/N$ , where $cenp$ represents the center point and $p$ represents the set of points. When the centroids of two sets of points coincide, they have only a rotational relationship between them, which is visualized in Figure 1. Therefore, we need to consider only the relative coordinate values of the coordinate points about the centroids.

Figure 1. The experimental setup of acupuncture robot for treating low back pain.

A: the experimental setup of acupuncture robot system; B: the selected acupoints; C: the acupoints in the lumbar region; D: Weizhong (BL40) acupoint; : optitrack infrared motion capture camera; : motion capture Markers; : the control system (A computer); : acupuncture robot; : a dummy with marked acupoints; : the selected acupoints.

(b) Calculation of the rotation matrix $R$ based on SVD

We use singular value decomposition (SVD) to solve the least squares problem to obtain the rotation matrix. To apply SVD to calculate the rotation matrix, we specify that $E={(A-{cenp}_A)(B-{cenp}_B)}^{\mathrm{T}}$ , where $A-{cenp}_A$ and $B-{cenp}_B$ are equivalently two sets of points with the same centroids. Thus, performing SVD on $E$ , we obtain the unitary matrices $U$ and $V$ as $[U, \_ , V]=SVD\left(E\right)$ , where the unitary matrix $U$ and unitary matrix $V$ consist of eigenvectors $E^{\mathrm{T}}E$ and $E{E}^{\mathrm{T}}$ , respectively. Therefore, we obtain the rotation matrix $R=V{U}^{\mathrm{T}}$ .

(c) Determination of the translation matrix T

It is easy to know that the same rotation and translation relationships exist between the centroids of the two sets of coordinate points, while we already know the rotation matrix between the two coordinate points. The translation matrix $T$ can be obtained by directly bringing $R$ and the centroid into $B=RA+T$ , which is calculated as $T={cenp}_B-R{cenp}_A$ . As a result of the above work, we obtained the conversion matrix from the motion capture system coordinate system to the robot coordinate system. Applying the transformation of ${[X_r,Y_r,Z_r]}^{\mathrm{T}}=R{[X_m,Y_m,Z_m]}^{\mathrm{T}}+T$ to the coordinates of all marker points collected by the motion capture system yields the coordinates needed for robot control.

2.2. The acupuncture manipulation system based on robot control

In this work, a flexible collaborative robot, ROKAE xMate ER3 Pro, and an intelligent gripper, BY-E140, are assembled to perform complicated acupuncture manipulations. Based on the integrated acupuncture robot system, the robot operating system is applied to perform communication between the components and the motion control of the ROKAE robot is designed to perform the acupuncture manipulations.

2.2.1. The main hardware devices of the acupuncture robot

(a) The flexible collaborative robot --- ROKAE xMate ER3 Pro

Each joint of the ROKAE xMate ER3 Pro robot is equipped with torque sensors, which guarantees high precision control and flexible obstacle avoidance. The robotic arm has 7 degrees of freedom, which meets the flexibility requirements of acupuncture. Rotation around the x-axis and y-axis enables different insertion angles of the needle. The rotation around the z-axis is used for the twirling-rotating manipulations of acupuncture. The last DOF can be used to adjust the posture of the robot while the position of the end-effector remains unchanged. The expansive workspace provided by the robotic arm can achieve acupuncture treatment for patients with lumbago. It is capable of targeting the same end position in different poses. The configuration of sensors and multiple degrees of freedom enables the robot flexibility, making acupuncture robots viable and safe.

(b) The intelligent gripper --- BY-E140

Given that the robotic arm has no end-effector, an intelligent gripper, BY-E140 is added to act as the hand of the acupuncturist to perform acupuncture operations. The gripper supports the XMLRPC interface. The interface provides the calibration command and the movement command of the gripper, which can grab or release the millimeter-thin needle.

2.2.2. Trajectory control algorithm for the robot system

The trajectory control of the designed robot system is implemented by controlling the 7-DOF of the robotic arm. The dynamic equation of the system is as follows:

$$M\left(q\right)\ddot{q}+C\left(q,\dot{q}\right)\dot{q}+G\left(q\right)=\tau$$ (2)

where $q,\dot{q},\ddot{q}\in R^n(n=7)$ represent the joint position, velocity, and acceleration vectors of the system, respectively. $M\left(q\right)\in R^{n\times n}(n=7)$ denotes the symmetric positive definite inertia matrix, $C\left(q,\dot{q}\right)\in R^{n\times n}(n=7)$ is the Gaussian force and centrifugal force matrix, $G\left(q\right)\in R^n(n=7)$ is the gravity vector, and $\tau \in R^n(n=7)$ represents the input vector of the controller.

The tracking error is defined as follows:

$$\left\{\begin{array}{c}e=q-q_d\\ \dot{e}=\dot{q}-{\dot{q}}_d\\ \ddot{e}=\ddot{q}-{\ddot{q}}_d\end{array}\right.$$ (3)

where $q_d\in R^n(n=7)$ represents the expected trajectory for robotic arm joints.

The sliding-mode surface of the controller controlling the trajectory tracking is designed as follows:

$$s=\dot{e}+\alpha e$$ (4)

where $\alpha$ is a positive integer.

The sliding mode reaching law of the sliding mode controller is designed as follows:

$$\dot{s}=-k_{1}s-k_{2}sign\left(s\right)$$ (5)

where $k_1$ , $k_2$ are positive integers. $sign(\bullet )$ is the signal function, whose equation is shown in Eq. (6).

$$sign\left(s\right)=\left\{\begin{array}{c}1 ‖s‖>0\\ 0 ‖s‖=0 \\ -1 ‖s‖<0\end{array}\right.$$ (6)

Taking the derivative of Eq. (4) yields:

$$\dot{s}=\ddot{e}+\alpha \dot{e}=\ddot{q}-{\ddot{q}}_d+\alpha \dot{e}$$ (7)

Substituting Eq. (2) into Eq. (7) yields:

$$\dot{s}=\ddot{q}-{\ddot{q}}_d+\alpha \dot{e}=M^{-1}\left(q\right)\left(\tau -C\left(q,\dot{q}\right)\dot{q}-G\left(q\right)\right)-{\ddot{q}}_d+\alpha \dot{e}$$ (8)

The input of the controller can be obtained by substituting Eq. (8) into Eq. (5):

$$\tau =M\left(q\right)({\ddot{q}}_d-\alpha \dot{e}-k_{1}s-k_{2}sign(s\left)\right)+C\left(q,\dot{q}\right)\dot{q}+G\left(q\right)$$ (9)

2.2.3. The automated implementation of acupuncture manipulations

The coordinates of the acupoints to be stimulated are packaged as a message and sent to the robot control system via the robot operating system. Based on the coordinates of the acupoints, the robot control system plans the acupuncture path according to the principle of "from top to bottom, from far to near". The acupuncture robot performs the acupuncture operation with motion control. Each acupuncture session includes three parts: trajectory control, intelligent gripper control and acupoint stimulation. First, the acupuncture robot reaches the placement of the millimeter-thin needle, which is divided into the motion in the x-y plane and vertical motion along the z-axis. Then, the gripper grabs the needle when it receives a grab signal from the robot control system during operation. Next, the robot targets the acupoint's position and performs lifting-thrusting and twirling-rotating manipulations to stimulate the acupoint. After stimulating the acupoint, the needle will remain in the human body, and the robot will repeat the above steps to stimulate the remaining acupoints.

(a) Trajectory control from the start position to the needle position

The starting point of the robot is assumed to be $\left(x_0,y_0,z_0\right)$ while the coordinate of the target acupoint is $\left(x_1,y_1,z_1\right)$ in the robot coordinate system. The position of the robotics defaults to that of the midpoint in the robot flange. The millimeter-thin needle is fixed in a place whose coordinates are $\left(x_2,y_2,z_2\right)$ . Given that an intelligent gripper is added to the flange to grab the millimeter-thin needle, the end of the robotic arm transforms into the needlepoint. The robot coordinate system should be transformed into the needle coordinate system accordingly. The coordinate system transformation equation is $\left(x_n,y_n,z_n\right)=\left(x_r,y_r,z_r\right)+(\mathrm{0,0},-l)$ , where $\left(x_r,y_r,z_r\right)$ is a point in the robot coordinate system and $\left(x_n,y_n,z_n\right)$ is the coordinate of the same point in the needle coordinate system. $l$ is the length from the midpoint of the flange to the tip of the needle. For the acupoints related to lumbago, 3-inch acupuncture needles are usually used clinically. Based on the transformation equation, the coordinates of the starting point, the target acupoint and the placement of the millimeter-thin needle are transformed to $\left(x_0,y_0,z_0-l\right)$ , $\left(x_1,y_1,z_1-l\right)$ and $\left(x_2,y_2,z_2-l\right)$ , respectively.

In this work, the robot first moves to the upward side of the millimeter-thin needle and then reaches the position of the needle placement along the z-axis. The control process of the two trajectories is described as follows.

(ⅰ) The trajectory control in the x-y plane

The motion velocity should be appropriate for controlling the robot to reach the fixed point with a small error. In this work, a velocity adjustment strategy is adopted to balance the operating speed of the system and the positioning precision. When the distance from the starting point to the target point exceeds 2 cm along both the x-axis and y-axis, the motion velocity is set to $v=v_1=0.0015m/s$ . If the distance along one axis is less than 2 cm, the motion velocity is adjusted to $v=v_2=0.0010m/s$ , which guarantees that the mean distance error from the desired point is less than 2 mm and avoids uncontrollability of the robot due to excessive speed. Assuming that the coordinates of the upward side of the starting point and the upward side of the placement of the millimeter-thin needle are $(x_0,y_0,z_{up})$ and $(x_2,y_2,z_{up})$ , respectively, the direction of the robot motion can be expressed by the unit vector ${dir}_v=(x_2-x_0,y_2-y_0)/\sqrt{{(x_2-x_0)}^2+{(y_2-y_0)}^2}$ . Based on the orthogonal decomposition, the velocity components in the x and y directions can be calculated as $∆x=(x_2-x_0)\bullet v/\sqrt{{(x_2-x_0)}^2+{(y_2-y_0)}^2}$ and $∆y=(y_2-y_0)\bullet v/\sqrt{{(x_2-x_0)}^2+{(y_2-y_0)}^2}$ .

(ⅱ) Trajectory control along the z-direction

After the robot reaches the upward side of the placement point of the millimeter-thin needle, it should move along the z-axis toward the position of the needle and the needle should be grabbed by an intelligent gripper. From the upward side of the placement point of the millimeter-thin needle $(x_2,y_2,z_{up})$ to the placement point of the millimeter-thin needle $\left(x_2,y_2,z_2-l\right)$ , there is displacement only in the z-direction. Then the control function of the robot can be denoted by $∆z=v_z$ , where $v_z$ is the speed at which the robot moves in the z-direction.

Most acupoints related to the lumbago are concentrated in the lumbar region and are usually stimulated by a vertical stabbing technique. The acupuncture depth ranged from 0.3 inches to 1.5 inches. A 1.5-inch to 3-inch millimeter-thin needle is usually chosen for clinical acupuncture operations based on the patient’s body shape. The acupuncture robot in this work simulates real acupuncture treatment. The mechanical gripper and acupuncture needles are sterilized at the beginning, which ensures the entire process is safe.

(b) Needle grabbing by the intelligent gripper

After the robot reaches the placement point of the millimeter-thin needle, the robot control system sends a grab signal to the intelligent gripper via the robot operation system. Then, the gripper executes the action of the grabbing needle, during which the robot maintains its current position.

(c) Control of the trajectory from the needle position to the target acupoint

The robot first moves back to the upward side of the placement of the millimeter-thin needle $(x_2,y_2,z_{up})$ along the z-axis to prevent injury to the patient from the needle tip during movement. Then, the device moves to the upward side of the target acupoint $(x_1,y_1,z_{up})$ and then reaches the position of the acupoint $(x_1,y_1,z_1-l)$ along the z-axis. The trajectory control is similar as that described in part (a).

A dual feedback control strategy is adopted here to ensure the accurate positioning of the acupoints before needle insertion in the robot control system. One is the current location feedback control. When the robot moves in the x-y plane, the current position is updated, and the coordinates are sent to the robot control system to adjust the motion direction. Even if the robot passed the target acupoint, the motion direction is adjusted until it reaches the designated location. The other is visual feedback control. Since the coordinates of the target acupoint change due to the movement of the patient’s body, a real-time update of the acupoint coordinates is introduced during the robot control process. Since the patient is lying on a fixed platform, it is assumed that the patient has slight displacement only in the x and y directions. Before the robot moves according to the control command, it receives the coordinates of the acupoints captured by the motion capture system through the robot operating system. The location of the target acupoint $\left(x_1,y_1,z_1\right)$ is variable, and the motion direction varies accordingly. The real-time strategy adaptively adjusts the motion of the robot to the target acupoint even when the patient moves in the x- and y-directions during acupuncture treatment. This ensures the safety and efficacy of robot acupuncture. The dual feedback control strategy ensures that the robot is always moving in the direction of the target position, which ensures system security.

(d) Control of the lifting-thrusting and twirling-rotating manipulations

When the millimeter-thin needle reaches the designated acupoint, the acupuncturist will use acupuncture techniques to make the patient feel needle sensation, adjust the strength of the needle sensation or promote the diffusion and conduction of needle sensation in a certain direction; this approach is called “Xingzhen” in Traditional Chinese Medicine³¹. The use of Xingzhen is directly related to the therapeutic efficacy of acupuncture operations. “Treating the deficiency syndrome with the tonifying method while treating the excess syndrome with the purgative method” is one of the basic rules in Traditional Chinese Medicine treatment. Different acupuncture manipulations correspond to different treatment methods. There are two commonly used acupuncture manipulations, lifting-thrusting and twirling-rotating.

The lifting-thrusting method is defined as lifting and thrusting down the needle at the acupuncture point. Generally, the finger force needs to be even, and the amplitude and frequency of lifting-thrusting should not be too large. The twisting-rotating method refers to the manipulation of bidirectional rotation at various fre-quencies. Twirling should not occur in a single direction to avoid stalling the needle. Depending on the patient's specific situation, the two common manipulations can be used alone or in conjunction with each other.

The use of “Xingzhen” is important for ensuring the efficacy of acupuncture. Lifting-thrusting manipulation and twirling-rotating manipulation are often used in conjunction with each other to relieve low back pain. In this work, it is assumed that the strength of acupuncture operations can be adjusted by controlling the speed and rotation frequency of the acupuncture robot. Moreover, the acupuncture robot reproduces the compound manipulation of lifting-thrusting and twirling-rotating movements by quantifying the velocity and distance of displacement and the frequency and angle of rotation. For vertical-stabbed acupoints, the robot motion includes displacement along the z-axis and rotation around the z-axis. After the robot inserts the needle into the acupoint, the location of the flange is denoted as $(f_x,f_y,f_z)$ , and the pose matrix of the robot $I_p$ can be expressed as follows:

$$I_p=\left[\begin{array}{cc}{p}_m& {pt}_m\\ 0& 1\end{array}\right]$$ (10)

with:

$$p_m=\left[\begin{array}{ccc}\mathrm{c}\mathrm{o}\mathrm{s}(x,x_i)& \mathrm{c}\mathrm{o}\mathrm{s}(y,x_i)& \mathrm{c}\mathrm{o}\mathrm{s}(z,x_i)\\ \mathrm{c}\mathrm{o}\mathrm{s}(x,y_i)& \mathrm{c}\mathrm{o}\mathrm{s}(y,y_i)& \mathrm{c}\mathrm{o}\mathrm{s}(z,y_i)\\ \mathrm{c}\mathrm{o}\mathrm{s}(x,z_i)& \mathrm{c}\mathrm{o}\mathrm{s}(y,z_i)& \mathrm{c}\mathrm{o}\mathrm{s}(z,z_i)\end{array}\right]$$ (11)

$${pt}_m={\left[f_x f_y f_z\right]}^T$$ (12)

where $p_m$ is the posture matrix and represents the posture of the rigid body coordinate system in the base coor-dinate system. The twirling-rotating transformation matrix $R_c$ is set to control the robot rotation around the z-axis at a certain frequency, which can be denoted as follows:

$$R_c=\left[\begin{array}{cccc}\mathrm{c}\mathrm{o}\mathrm{s}\left(\theta \right)& -\mathrm{s}\mathrm{i}\mathrm{n}\left(\theta \right)& 0& 0\\ \mathrm{s}\mathrm{i}\mathrm{n}\left(\theta \right)& \mathrm{c}\mathrm{o}\mathrm{s}\left(\theta \right)& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]$$ (13)

The transformed robot pose matrix $N_p$ can be calculated via matrix multiplication. In addition, the displacement in the z- direction is controlled by $∆z=v_z$ . The control function of the "down thrust" can be expressed as follows:

$$\left\{\begin{array}{c}{N}_p=I_p\times R_c\\ \theta =-\frac{\pi }{2}(1-cos(\frac{\pi }{18}\times time\left)\right)\\ ∆z=-v\times time\end{array}\right.$$ (14)

The control function of the “upward lift” can be expressed as follows:

$$\left\{\begin{array}{c}{N}_p=I_p\times R_c\\ \theta =\frac{\pi }{2}(1-cos(\frac{\pi }{9}\times time\left)\right)\\ ∆z=2v\times time\end{array}\right.$$ (15)

Eq. (12) and Eq. (13) show that the frequency and velocity of the “upward lift” are twice those of the "down thrust". This is the tonic method of acupuncture, which involves heavy insertion and light lifting, with the main force being the lower insertion.

(e) Needle release by the intelligent gripper

Similar to the needle grabbing, the gripper executes the action of releasing the needle when it receives a release signal from ROS. After releasing the needle, the robot moves back to the upward side of the acupoint $(x_1,y_1,z_{up})$ to continue the acupuncture of the next acupoint or stop motion if the acupuncture ends.

After the needles remain in the patient’s body for a period of time, the needles should be removed for safety reasons. The acupuncture robot removes the needles in the reverse order of needle insertion. Correspondingly, the robot motion control is opposite to that of needle insertion. Robotic needle retrieval reduces the rate of missed needles, which ensures that all needles are removed and reduces the workload of the acupuncturist.

2.3. The De Qi sensation detection system based on speech keyword recognition

De Qi sensation refers to the patient's sensation in response to needling during acupuncture operations.³² De Qi sensation usually manifests as soreness, numbness, distention, and heaviness.³³ According to Traditional Chinese Medicine theory, De Qi is an “energy” phenomenon that eliminates diseases by regulating the flow of “Qi”.³³ Therefore, De Qi is an indispensable requirement for ensuring the efficacy of acupuncture.³⁴

In this work, a De Qi detection system is designed to detect whether acupuncture is effective. After the acupuncture manipulations for one acupoint is performed, the patient's speech is recorded. The reception equipment includes a Yamaha UR12 sound card and a TAKSTAR PC-K500 microphone. A deep learning method combining DCNN and CTC is used to recognize speech in real time. The collected speech is converted into log-mel spectrograms and data augmentation is performed using the SpecAugment method.³⁵ The spectrograms after data augmentation are input into the DCNN, and features are extracted through a series of convolutional layers. A batch normalization layer is added after each convolution layer to increase the training speed and convergence speed of the network and prevent overfitting. In the above DNN, the CTC is adopted as the loss function to achieve end-to-end speech recognition.

The input Chinese speech is output as the corresponding Pinyin sequence through the network. When keywords such as "tong (pain)", "suan (soreness)", "zhang (distension)" and "ma (numbness)" are detected in the Pinyin sequence, it is regarded as De Qi, and the next acupoint manipulations are carried out. Otherwise, the needle is lifted subcutaneously, repositioned and inserted into the current acupoint. The running time of the De Qi system is 0.19 s.

3. RESULTS

In this work, Shenshu (BL23), Dachangshu (BL25), Yaoyan (EX-B6) and Weizhong (BL40) are selected for the use of the acupuncture robot system. In the clinic, bilateral acupoints should be needled to balance Yin and Yang. The bilateral acupoints are symmetrical about the Du meridian. The experimental setup is shown in Figure 1. The robotic arm and the gripper are connected to act as the hand of the acupuncturist. An experimental dummy lies on a bed. The above acupoints are marked with a green pen. There are motion capture markers near the acupoints to locate them. The data on the acupoint acupuncture results in this work are obtained from repeated experiments on the experimental dummy.

3.1. Implementation of the acupuncture robot system

The proposed acupuncture robot system can perform acupuncture operations on patients. Figure 2 shows how the system performs acupuncture on the specified acupoints. Based on the relative position between the patient and the acupuncture robot, the acupuncture order is in accordance with “from top to bottom, from far to near”.

Figure 2. The acupuncture process of the proposed acupuncture robot system acupoints.

As shown in Figure 2, the acupuncture robot first moves to a fixed placement to pick up a millimeter-thin needle. Then the robotic arm lifts the needle to the acupoint without hurting the patient. Based on the acupoint position obtained by the motion capture system, the robotic arm moves just above the acupoint and inserts the needle into the skin. The acupoint and body shape of patients are evaluated to determine the insertion depth. After needle insertion, lifting-thrusting and twirling-rotating manipulations are carried out to stimulate the acupoint and relieve the patient's pain. The two manipulations are used in conjunction to reduce the discomfort of patients during acupuncture. In this work, a sine-like discrete function is introduced to stimulate the acupoint. The process is shown in Figure 2 (f-1) to Figure 2 (f-12). Moreover, the frequency and velocity of the “upward lift” are twice those of the "down thrust", which is a tonic method. The steps mentioned above are repeated to stimulate the remaining acupoints, and the acupuncture results are shown in Figure 2.

Figure 2 shows that the needle insertion position is not the center of the acupoint marker. There is a little tolerance for the needle insertion position in clinics. In other words, the acupuncturist can insert the needle around the acupoint to perform the acupuncture operations, which is still effective for pain relief. However, the insertion position deviation should not be too large.

3.2. Insertion accuracy of the acupuncture robot

According to the flow chart of the acupuncture robot control in Figure 2, the acupuncture robot should move to a fixed placement to pick up the millimeter-thin needle and then move to an acupoint to insert the needle into the skin. The positioning accuracy of the former ensures that the robot arm picks up the needle and keeps it on the central axis of the intelligent gripper, which is significant for the needle to accurately pierce the acupoint. The latter positioning accuracy is directly related to the therapeutic effects of acupuncture. To evaluate the positioning accuracy of the acupuncture robot, repeatability experiments were conducted.

In this work, eight acupoints were selected for surgery via the acupuncture robot system, where $l$ and $r$ represent the acupoints on the left and right sides, respectively. Twenty experiments were conducted to calculate the positioning accuracy of the robot (Table 1).

Table 1.

Results of the repeatability experiments for positioning accuracy of the robot (mm)

The positioning target	Minimum error	Maximum error	Mean error	Standard deviation
Needle-grabbing location	1.73	3.57	1.97	0.20
Yaoyan (EX-B6) (r)	1.25	1.60	1.47	0.08
Shenshu (BL23) (r)	1.47	2.76	1.62	0.27
Dachangshu (BL25) (r)	1.23	1.71	1.52	0.10
Weizhong (BL40) (r)	1.99	2.58	2.31	0.16
Yaoyan (EX-B6) (l)	1.54	1.68	1.62	0.04
Shenshu (BL23) (l)	1.35	1.70	1.50	0.08
Dachangshu (BL25) (l)	1.46	2.56	1.62	0.22
Weizhong (BL40) (l)	2.07	2.71	2.41	0.20

As shown in Table 1, the mean error in positioning the needle-grabbing location is 1.97 mm, and the standard deviation is 0.20 mm. The positioning error of each acupoint range from 1.47-2.41 mm, and the standard deviation ranges from 0.04-0.27 mm. The positioning error of the needle-grabbing location is within the clamping range of the intelligent gripper, which indicates that the acupuncture robot can complete the needle-grabbing process. The positioning errors of acupoints are within the effective range of acupuncture points,³⁶ which meets the effectiveness requirements of acupuncture operations. The standard deviations of the needle grabbing location and acupoint positioning are all less than 0.3 mm, which shows that the proposed acupuncture robot system is stable in terms of positioning. The above results show that the robotic arm can move to the designated position in the specified direction and at the desired speed, ensuring the accuracy of subsequent acupuncture operations.

3.3. The trajectory accuracy of the acupuncture robot

Uniform motion is adopted to ensure the smooth operation of the robot in the control system. In the x-y plane and the z-direction, the trajectory of motion is determined by the movement's starting point and target point. To evaluate the control performance of the acupuncture robot, the deviation between the real path and the planned path is calculated.

3.3.1. The evaluation of the planar linear trajectory accuracy

From the needle placement to the acupoint location or vice versa, the acupuncture robot moves along a straight line in the x-y plane. The closer the path of the robot’s motion is to the planned path, the more accurately the robot can reach the corresponding position and perform the appropriate function. This approach also reduces the probability of the robot losing control to a certain extent. In this work, the mean deviation distance (MDD) and the deviation angle (DA) are proposed to express the planar linear trajectory accuracy. The mean deviation distance refers to the average distance between the real and designed paths. When the time is t, the expected motion position is $x_t^e,y_t^e,z_t^e$ while the actual motion position of the acupuncture robot is $x_t^a,y_t^a,z_t^a$. If the robot reaches the target point after $tim{e}_r$ milliseconds, the mean deviation distance can be expressed as follows.

$$MDD=\frac{\sum _{t=1}^{time}\sqrt{{(x_t^e-x_t^a)}^2+{(y_t^e-y_t^a)}^2}}{{\text{ time }}_r}$$ (16)

The deviation angle is the angle between the actual and planned paths. Based on the coordinate point set $\left\{(x_t^e,y_t^e,z_t^e)|0<t<{time}_r,t\in N\right\}$ of the expected path, a planar straight line can be fitted by the least squares method. The equation of the expected path is denoted by $y=k_{e}x+b_e$ , if the slope of the straight line exists. Similarly, the equation of the actual motion path can be expressed as $y=k_{a}x+b_a$ . The deviation angle can be computed as follows:

$$DA=arctan\frac{k_e-k_a}{\sqrt{1+k_e{k}_a}}$$ (17)

If the slope of one trajectory does not exist, the deviation angle is computed by DA = arctan k, where $k$ is the slope of another trajectory. If the slope of both lines does not exist, the deviation angle is noted as zero. The results of the trajectory accuracy are computed and listed in Table 2.

Table 2.

Results of the trajectory accuracy

The name of the acupuncture point	MDD (mm)	DA ( $°$ )
Yaoyan (EX-B6) (r)	0.01	0.03
Shenshu (BL23) (r)	0.01	0.02
Dachangshu (BL25) (r)	0.01	0.04
Weizhong (BL40) (r)	0.02	0.02
Yaoyan (EX-B6) (l)	0.02	0.03
Shenshu (BL23) (l)	0.01	0.01
Dachangshu (BL25) (l)	0.02	0.04
Weizhong (BL40) (l)	0.02	0.06

Notes: MDD: the mean deviation distance; DA: the deviation angle.

Table 2 shows that the mean deviation distance of the trajectory from the needle-grabbing location to each acupoint is approximately 0.02 mm (only two decimal places are shown in Table 2). The deviation angles of the trajectories are all less than 0.1°. A small mean deviation distance indicates no significant error in the entire path of the robot arm moving from the needle point to the acupuncture point. The small deflection angles indicate that the robot arm's movement trajectories are consistent with the set trajectories. The above results show that the proposed acupuncture robot system can reach the designated acupoint from the needle-grabbing location with a small error.

3.3.2. The evaluation of the Xingzhen quantification

After the needle is inserted into the acupoint, lifting-thrusting and twirling-rotating manipulations will be performed to generate De Qi. It is assumed that the initial insertion depth of the millimeter-thin needle is $d$ mm. In Traditional Chinese Medicine, the stimulation of acupoints follows a gradual progression and generally starts from shallow to deep. In this work, triple stimulation is used to quantify Xingzhen manipulation. + 5 mm, + 8 mm, and + 10 mm are chosen for stimulating the acupoint. According to the operation of lifting-thrusting, the depth of needle tip penetration into the skin is denoted by $c1=\{d+5,d,d+8,d,d+10,d\}$ . In fact, the depth reached by the acupuncture robot is $c2=\{d_1,d_2,d_3,d_4,d_5,d_6\}$ . The stimulation depth is directly related to the efficacy of acupuncture. To evaluate the precision of the Xingzhen quantification, the mean depth error (MDE) is proposed. The mean depth error is defined as the average distance between $c1$ and $c2$ , and can be expressed as follows:

$$MDE=\left({\sum }_{i=1}^6|{c1}_i-{c2}_i|\right)/6$$ (18)

where ${c1}_i$ and ${c2}_i$ are the $i^{th}$ elements in the $c1$ set and $c2$ set, respectively.

The results of the quantization error for each acupoint are as follows. For Yaoyan (EX-B6), Shenshu (BL23), Dachangshu (BL25), and Weizhong (BL40), the Mean Deviation Error (MDE) values were 1.98, 2.06, 2.04, and 2.01 mm, respectively. For Yaoyan (EX-B6), Shenshu (BL23), Dachangshu (BL25), and Weizhong (BL40), the MDE values were 2.02, 1.99, 2.00, and 1.97 mm, respectively. These results indicate that the mean depth error of the acupuncture manipulation of each acupoint is approximately 2 mm. The recommended depth of lifting-thrusting in the acupuncture textbook is 3-5 fen (9.525-15.875 mm).³⁷ An error of 2 mm is within the depth range of needle penetration (6.35 mm), which indicates that the designed acupuncture robot can achieve De Qi through Xingzhen manipulation. In addition, the robot's needle penetration depth error at each acupoint is similar, indicating that the robot's motion in the z-direction has good stability.

3.4. Accuracy of the De Qi sensation detection system

To verify the performance of the De Qi sensation detection system, the detection accuracy of the De Qi detection system on De Qi keywords are tested. In this work, six public Chinese speech datasets, THCHS30,³⁸ ST-CMDS, Primewords Chinese Corpus Set 1, AISHELL-1,³⁹ Aidatatang200 and MAGICDATA Mandarin Chinese Read Speech Corpus,⁴⁰ which contain 1373 h of speech recorded by 3271 people from different regions of China with different dialects, were used. The speech data used in this work are all sampled at 16kHz and cover a variety of scenarios, including article verses, voice chat and intelligent voice control utterances. Some of the speech contains background noise that does not affect speech recognition. Training with noisy data enhances the robustness of the robot system, making it more adaptable to applications in hospitals, clinics, and other environments that contain a variety of noises. The proposed system uses the training sets in the above six datasets for training. The system is implemented under the TensorFlow framework, and its initial learning rate is set to 0.0001.

From the test sets in THCHS30, ST-CMDS, and AISHELL-1, 935 speech samples containing the keywords "tong", "suan", "zhang" and "ma" are extracted as the test set for this system. When the system detects keywords in the test set, the detection is correct. The detection results of the system on the test set are shown in Table 3.

Table 3.

Results of the Deqi sensation detection system

Keyword	Number of samples	Accuracy (%)	Recall (%)	F1 score
"Tong"	355	97.01	93.24	0.96
"Suan"	200	98.07	91.00	0.95
"Zhang"	234	97.97	94.44	0.96
"Ma"	219	97.33	97.26	0.94
All	935	95.19	95.19	0.98

As shown in Table 3, the detection accuracy of "Tong", "Suan", "Zhang", and "Ma" ranges from 97.01% to 98.07%, the recall ranges from 91.00% to 97.26%, and the F1 scores range from 0.94 to 0.98. The experimental results show that result indicators of the system identifying each single keyword are more than 90%. The last row of Table 3 records the overall results of the system's recognition of the four keywords. It can be seen from Table 3 that the overall accuracy of the system in recognizing the keywords of De Qi is 95.19%, the recall is 95.19%, and the F1 score is 0.98. The above results show that the proposed De Qi sensation detection system can achieve high detection accuracy for individuals of different genders and from different regions.

3.5. Stability validation of the proposed acupuncture robot

To verify the stability of the designed acupuncture robot, acupoints’ coordinates localized by the acupuncture robot are compared with the Gold Standard using significant level tests. The Gold Standard is labeled by a professional acupuncturist. The test of significant level in this subsection is the t-test, and the value of significant level is 0.05. The t-test results of 20 repeated experiments are recorded separately, which are shown in Table 4. Each experiment consists of acupuncture on eight acupoints.

Table 4.

Significant level test results between acupoints’ coordinates localized by the robot and the Gold standard

Number of experiments	P value			Number of experiments	P value
	x-coordinates	y-coordinates	z-coordinates		x-coordinates	y-coordinates	z-coordinates
1	0.97	0.88	0.48	11	0.96	0.88	0.46
2	0.97	0.86	0.47	12	0.96	0.88	0.46
3	0.97	0.88	0.47	13	0.96	0.91	0.46
4	0.96	0.85	0.46	14	0.96	0.89	0.46
5	0.97	0.88	0.46	15	0.96	0.88	0.46
6	0.96	0.87	0.46	16	0.96	0.88	0.46
7	0.96	0.87	0.47	17	0.96	0.87	0.46
8	0.96	0.87	0.47	18	0.96	0.88	0.46
9	0.96	0.88	0.46	19	0.96	0.88	0.46
10	0.96	0.88	0.46	20	0.96	0.88	0.46

As shown in Table 4, the P-values between acupoints’ coordinates localized by the robot and the Gold Standard are all greater than 0.05. The results of the significant level tests indicate that acupoints’ locations localized by the designed robot are not significantly different from the Gold Standard labeled by the acupuncturist, which suggests that the robot can maintain stable performance in multiple experiments.

4. DISCUSSION

The main contributions of this work include the design of an acupuncture robot system capable of performing acupuncture manipulations automatically. This system integrates automatic acupoint positioning, acupuncture manipulations, and De Qi detection, covering the entire acupuncture process. An automatic acupoint localization system is proposed based on the OptiTrack motion capture system, taking into account the variations in acupoint positions for patients with different body types. Additionally, an acupuncture manipulation system is developed using robot control, which employs a dual feedback control strategy and a velocity adjustment strategy to enhance accuracy and safety. Lastly, a De Qi sensation detection system based on speech keyword recognition is introduced to enable real-time evaluation and adjustment of the acupuncture procedures.

The proposed acupuncture robot system accurately locates acupoints using the OptiTrack motion capture system with high precision. The acupoint coordinates are subsequently computed through the coordinate transformation system and received by the robot control system via the robot operating system. Using this information, the robotic arm and intelligent gripper perform needle insertion and Xingzhen manipulations, stimulating the acupoints to relieve the patients’ symptoms. In addition, the De Qi sensation detection system based on speech keyword recognition is used for real-time evaluation and adjustment of acupuncture.

Through repeated experiments, the acupuncture robot achieves a positioning error within 3.3 mm, which is within the allowable range of the robot and ensures effective needle extraction and acupoint insertion. During needle insertion, the robot arm follows the prescribed trajectory with a mean deviation distance of only 0.02 mm and a deviation angle of less than 0.15°. The experimental results show that the robot arm can reach designated acupoints according to the prescribed route. To verify the effectiveness of acupuncture, the lifting thrusting operation in the process of Xingzhen is performed. The results show that the mean acupuncture depth error of the designed acupuncture robot is approximately 2 mm, which is within the recommended depth range for Xingzhen operation. The performance of the De Qi sensation detection system is evaluated by testing the accuracy of the system's ability to detect De Qi keywords. The experimental results show that the average detection accuracy of De Qi keywords is 94.52%, which meets the requirements of acupuncture effect testing when faced with different dialects.

The experimental results show that the acupuncture robotic system designed in this work can meet the positional accuracy and maneuverability requirements needed for acupuncture, due to the combination of the OptiTrack motion capture system and the acupuncture manipulation system. The dual feedback control strategy and velocity adjustment strategy incorporated in the acupuncture manipulation system ensured the accuracy and safety of the system. The design of the lifting, inserting, twisting and rotating operations make the robot more anthropomorphic. In addition, the proposed De Qi sensation detection system further ensures the effectiveness of the robotic system by evaluating and adjusting the acupuncture in real time. These findings indicate that the proposed acupuncture robot system automatically and quantitatively performs acupuncture operations, which is favorable for quantitative research on acupuncture.

Funding Statement

Supported by Modernization of Traditional Chinese Medicine Project of National Key R & D Program of China: The construction of the theoretical system of Traditional Chinese Medicine non-pharmacological therapy based on body surface stimulation (2023YFC3502704); Sichuan Provincial Science and Technology Program Project: Research and Development of Chinese Medicine Intelligent Tongue Diagnosis Equipment for Digestive System Chinese Medicine Advantageous Diseases (2023YFS0327); Research and Development of Chinese Medicine Intelligent Detection System for Intestinal Functions (2024YFFK0044); Research and Application of Chinese Medicine Diagnosis and Treatment Program for Herpes Zoster Treated by Shu Pai Fire Acupuncture (2024YFFK0089); Major Research and Development Project of The China Academy of Chinese Medical Sciences Innovation: Construction and application of the theoretical research mode of Traditional Chinese Medicine diagnosis and treatment of modern diseases (CI2021A00104)