Collaborative Control Method and Experimental Research on Robot-Assisted Craniomaxillofacial Osteotomy Based on the Force Feedback and Optical Navigation

Objective:

Surgical robot has advantages in high accuracy and stability. But during the robot-assisted bone surgery, the lack of force information from surgical area and incapability of intervention from surgeons become the obstacle. The aim of the study is to introduce a collaborative control method based on the force feedback and optical navigation, which may optimally combine the excellent performance of surgical robot with clinical experiences of surgeons.

Materials and Methods:

The CMF ROBOT system was integrated with the force feedback system to ensure the collaborative control. Force-velocity control algorithm based on force feedback was designed for this control method. In the preliminary experimental test, under the collaborative control mode based on force feedback and optical navigation, the craniomaxillofacial surgical robot entered the osteotomy line area according to the preoperative surgical plan, namely, right maxillary Le Fort I osteotomy, left maxillary Le Fort I osteotomy, and genioplasty.

Results:

The force sensor was able to collect and record the resistance data of the cutting process of the robot-assisted craniomaxillofacial osteotomy assisted in real time. The statistical results showed that the repeatability of collaborative control mode was acceptable in bilateral maxillary Le Fort I osteotomies (right, P=0.124>0.05 and left, P=0.183>0.05) and unfavorable in genioplasty (P=0.048<0.05).

Conclusion:

The feasibility of robot-assisted craniomaxillofacial osteotomy under the collaborative control method based on the force feedback and optical navigation was proved in some extent. The outcome of this research may improve the flexibility and safety of surgical robot to meet the demand of craniomaxillofacial osteotomy.

The craniomaxillofacial bones are relatively small in size and irregular in shape. They are wrapped by soft tissues, which contain many blood vessels and nerves. The deep region of the bones is adjacent to the brain and important blood vessels and nerves, such as internal carotid artery, plexus pterygoideus, facial nerve, trigeminal nerve, etc. In contrast, patients who need the medical treatment with craniomaxillofacial osteotomy have high expectations of functional reconstruction and aesthetic restoration. Therefore, craniomaxillofacial osteotomy should be performed accurately through minimally invasive or intraoral approaches, it is necessary for surgeons to have fine skills and rich clinical experiences during the operation.

Aimed at improving the accuracy and safety of bone surgeries, digital surgical technologies are widely used in clinical practice, including 3-dimensional (3D) virtual surgical planning,1,2 computer-aided design/computer-aided manufacturing,3,4 image-guided navigation,5 augmented reality technology,6 3D printing,7 etc. As a hot issue in the research of digital surgery, surgical robot has advantages in high accuracy and stability.8–11 Nowadays, it has been widely used in urology,12 thoracoscopic surgery,8 cardiac surgery,13 orthopaedics,9,11,14 neurosurgery,15 etc. Recently, surgical robot technology was applied to craniomaxillofacial osteotomy by some scholars.10,16,17 This research was to explore the collaborative control method based on force feedback and optical navigation, and experimental test of robot-assisted craniomaxillofacial osteotomy in such method was carried out, which preliminarily proved the feasibility of this technology.

MATERIALS AND METHODS

The Component Parts of Surgical Robot System

To meet the needs of clinical practice and experimental tests, a robot system CMF ROBOT was developed for craniomaxillofacial surgery, which consists of a C8L robotic arm (EPSON Robots, Long Beach, CA) with 6 degrees of freedom, a self-developed end-effector, and a Polaris Vicra optical tracker (Northern Digital Inc, Waterloo, Canada).8 On this basis, the force feedback system, including a force sensor and a force feedback device, was integrated to ensure the collaborative control (Fig. 1). The L2000 force sensor (Xiyuan electronic technology LTD, Yangzhou, China) was assembled on the self-developed end-effector, collecting force information about the surgical area (Fig. 2). The Omega 6 force feedback device (Force Dimension, Nyon, Switzerland) accessed the force information of the surgical area and transmitted it to the surgeon, providing the force feedback in 6 directions (Fig. 3). The robot software system adopted the self-developed CMF Robot Plan 1.0, which had the functions of preoperative virtual surgical design, robotic surgical path planning and intraoperative navigation. The surgical instrument was a TCM3000 reciprocating saw (NOUVAG, Goldach, Switzerland).

FIGURE 1.

The CMF ROBOT with force feedback and optical navigation.

FIGURE 2.

The self-developed end-effector. The green arrow showed the force sensor, red arrow showed the electric motor, yellow arrow showed the mounting position of dynamic reference frame (DRF), and white arrow showed the holding position of reciprocating saw.

FIGURE 3.

The Omega 6 force feedback device.

Design of Force-Velocity Control Algorithm Based on Force Feedback

After the surgical instrument on the end-effector of the surgical robot enterd the predefined working area (the distance information can be obtained by visual servo), the surgeon controlled the end-effector through the force feedback device and hoped to make sure that the reciprocating saw moved when the force was input, and stopped when the force was lost; the greater the force was added, the faster the saw moved. Therefore, the simplest velocity control rule that could be adopted theoretically was as follows:

$$\dot{x}=k\cdot \Delta f\cdot st:\Vert \dot{x}\Vert \le \Vert {\dot{x}}_0\Vert ;x\in D_x$$ (1)

In this formula, $\Delta f$ was the difference between the force input by surgeon $f_s$ and the environmental reaction force $f_r$ (Fig. 4). As shown in Formula 2, $\alpha$ and $\beta$ were, respectively, the adjustment ratio coefficients of $f_s$ and $f_r$ (the resistance detected by the force sensor). In addition, considering the safety of the surgery, this research laid certain constraints on the velocity $\dot{x}$ and the position $x$ of the end-effector during the surgery. Set artificially, ${\dot{x}}_0$ as the maximum speed of the end-effector. The operative space during the surgery was $S_a$ . Provided that the Cartesian space reachable by the robotic arm was $U$ , the danger space referred to the complementary space ${\overline{D}}_x$ of $D_x$ in the total space $U$ .

FIGURE 4.

Force simulation diagram and sawing force analysis at the end-effector.

$$\Delta f=\alpha \cdot f_s-\beta \cdot f_r$$ (2)

In fact, the force feedback device could not directly measure and give the manual input force data. However, it could provide real-time data such as the position $P_h$ , speed $v_h$ and feedback force $f_r$ of the joystick, which were used to simulate the manual input force $f_s$ . The preliminary definition of $f_s$ was: $f_s=f_r+m\cdot {\dot{v}}_h$ . In fact, this simulation method could meet the predetermined requirements.

During the process of osteotomy, the feedback force $f_r$ generated by the force feedback device was also the active force in the collaborative mode. To ensure that the surgical instrument was completely controlled by the surgeon under input force $f_s$ , and not going to move in the opposite direction due to the mutation of reaction force $f_r$ , a function $Sgn(f)$ should be defined to prevent the opposite movement in the collaborative control mode.

$$Sgn(f)=\{\begin{array}{cc}0& f\le 0\\ 1\dots & f>0\end{array}$$

At the same time, the speed $\dot{x}$ of the end-effector in the surgical process controlled by the surgeon and the virtual constraint around the operative space were implemented by adding reaction force ${f\prime }_r$ through a program and feeding back to the surgeon through the force feedback device. This force was defined as the virtual (feedback) force, and its 1-norm modulus was defined as:

$$\Vert f_r^{\text{'}}\Vert =\{\begin{array}{c}0x\in D_x;\Vert \dot{x}\Vert \le \Vert {\dot{x}}_0\Vert \\ (w_1\varphi (x,V_D)+w_2\psi (\dot{x},{\dot{x}}_0,f_r))\cdot F{}_{max}\end{array}$$ (3)

In the above formula, $V_D$ was the interface of $D_x$ and ${\overline{D}}_x$ , $w_1$ and $w_2$ were the weight coefficient (c $w_1+w_2=1$ ). $F_{max}$ was the maximum virtual feedback force artificially set. Function $\varphi ,\psi$ were used to represent the degree of the position and speed of the end-effector entering the dangerous space or falling into the dangerous operating state. From the perspective of fuzzy control theory, they were the corresponding Degree of Membership Function. Formula 3 demonstrated system control variables, which showed the fuzzy control of the virtual feedback force. As the length of the article is limited, the details of fuzzy control algorithm is not going to be repeated. In addition, the ratio adjustment coefficient of ${f\prime }_r$ was $\gamma$ .

In short, the PD control law of the force feedback control loop given by Formula 1 was improved by adding the dynamic quantity $M$ and $C$ as the adjustment coefficient of the acceleration and the velocity, as follow:

$$M\cdot \ddot{x}+C\cdot \dot{x}=k_v\cdot Sgn(\alpha f_s-\beta f_r-\gamma f_r^{ʹ})\cdot (\alpha f_s-\beta f_r-\gamma f_r^{ʹ})$$ (4)

Implementation of Collaborative Mode Based on Force Feedback and Optical Navigation

The surgical robot system was developed for the craniomaxillofacial bony surgery. Firstly, 3 operation modes could be selected through the human-computer interface: R mode, H, mode and H-R mode, which, respectively, represented automatic mode, manual mode, and human-robot collaborative mode. The functions of H and R under H-R mode were mainly as follows: R (Robot) mode took advantage of the high precision and stability of the robot and performed the surgery with the assistance of the optical navigation system; H (Human) mode meant that the surgeon controlled the moving speed and progress of the end-effector based on the force feedback of the force feedback device, mainly by adjusting the cutting frequency of the reciprocating saw or implementing the emergency braking of the robot. In H-R (human-robot) mode, randomness and inaccuracy of control caused by hand tremor was going to be filtered out through control algorithms such as virtual constraints. In this way, the subjective judgment of the surgeon and the high accuracy and stability of the robotic arm could be optimally combined owing to the collaborative control of the surgical robot system. Figure 5 showed the communicative workflow of the different operating modes.

FIGURE 5.

The communicative workflow of the different operating modes.

According to the needs of craniomaxillofacial osteotomy, the surgery was mainly divided into 4 steps. The surgeon (H mode) moved the reciprocating saw on the robotic arm from any point A to the point B near the skull through the 6-dimensional joystick of the force feedback device. Then the reciprocating saw was interpolated linearly (R mode) to the initial point C of the osteotomy line, and the osteotomy was going to be started in the human-robot collaborative mode (H-R mode). After the reciprocating saw reached the end point D, it was automatically going to interpolate (R mode) to the end point E of any suitable posture far away from the patient, which was convenient to move out of the surgical robotic arm. The CD section of the osteotomy line and the relatively safe sections BC and DE must be planned by the robot software preoperatively. The position of the points A and B could arbitrarily be specified by the control of the force feedback manipulator. The trajectory of the end-effector during craniomaxillofacial osteotomy was roughly as shown in Figure 6.

FIGURE 6.

The trajectory of the end-effector during craniomaxillofacial osteotomy. A indicates any point far away from patient; B, the point near the skull which closed to the beginning of osteotomy line; C, the initial point of osteotomy line; D, the end point of osteotomy line; E, the final point of trajectory in any suitable posture far away from patient.

Experimental Test and Analysis

Preoperative surgical simulation and trajectory planning: the corresponding osteotomies were going to be completed on 3 skull models in the software system, namely, right maxillary Le Fort I osteotomy, left maxillary Le Fort I osteotomy, and genioplasty (Fig. 7A). The above skull models were reconstructed 3D ones by importing the DICOM data of clinically acquired skull computed tomography (CT) into CMF Robot Plan 1.0. They were stored in Stereolithography (STL) format. The STL file of the virtual 3D skull model was imported into the 3D printer Objet260 Connex3 (Stratasys Ltd, MN), and the 3D printing resin material MED620 (Stratasys Ltd, MN) was applied to manufacture the experimental skull model (Fig. 7B). At the same time, the 3D skull model was used as reference data for robot-assisted experimental osteotomies in the software system CMF Robot Plan 1.0.

FIGURE 7.

Skull models. (A) virtual model. (B) 3D-printed model. The red line showed the right maxillary Le Fort I osteotomy, green line showed the left maxillary Le Fort I osteotomy, and yellow line showed the genioplasty.

Osteotomy procedure: under the collaborative control mode based on force feedback and optical navigation, the craniomaxillofacial surgical robot entered the osteotomy line area according to the preoperative surgical plan. The execution speed set by the CMF ROBOT system was 0.5 mm/s. It was going to be repeated for 3 times. The vibration frequency of the reciprocating saw was controlled at 25,000 cpm. During the cutting process, the force sensor was used to collect the tangential resistance data of the surgical area, which could be recorded by the software system. Meanwhile, the surgeon was able to feel the force changes of the surgical area by holding the force feedback joystick.

Statistical analysis: SPSS 19.0 software package (IBM, Chicago, IL) and Origin 2019 software (OriginLab, Northampton, MA) were used for data statistics of the measurement results. The force feedback function of the craniomaxillofacial surgery robot was preliminarily tested, and the feasibility of the collaborative control method based on force feedback and optical navigation was evaluated.

RESULT

The force sensor was able to collect and record the resistance data of the cutting process of the robot-assisted craniomaxillofacial osteotomy assisted in real time. As shown in the Figure 8, the pink wave line was the tangential resistance.

FIGURE 8.

The screenshot of force data collection. The pink wave line was the tangential resistance.

Through the joystick of the force feedback device, the surgeon was able to perceive the force changes during the process of cutting in the surgical area, obtaining a sense of on-site operation, who was also able to control the joystick to adjust the movement of the surgical robot through the software system or by switching buttons according to his own experience. In clinical practice, the vibration frequency of the reciprocating saw could also be adjusted to control the cutting speed, and therefore control the surgical robot.

All the tangential resistance data were shown in Supplemental Table 1, Supplemental Digital Content 1, http://links.lww.com/SCS/E349. According to the tangential resistance data, the force data box diagram obtained by Origin 2019 software could be seen in the Figure 9.

FIGURE 9.

The tangential force data box diagram of the osteotomies.

To investigate the differences of 3 groups in tangential resistance data of the craniomaxillofacial osteotomy (the 3 groups represent the measurement records of 3 skull models, the same below), the homogeneity of variance test was first performed. The result showed F(2,3361)=445.96, P=0.000<0.001 in right maxillary Le Fort I osteotomy, F(2,3762)=213.02, P=0.000<0.001 in left maxillary Le Fort I osteotomy, and F(2,6758)=132.44, P=0.000<0.001 in genioplasty, indicating that the variance of the data was uneven and did not meet the applicable conditions of variance analysis. Therefore, the Kruskal-Wallis H test was performed on the tangential resistance data of osteotomies, as shown in Supplemental Table 2, Supplemental Digital Content 2, http://links.lww.com/SCS/E350.

The statistical result showed that in the right maxillary Le Fort I osteotomy, H=4.18, P=0.124>0.05. It demonstrated that there was no statistically significant difference of the three groups in the tangential resistance data of the right maxillary Le Fort I osteotomy. In the left maxillary Le Fort I osteotomy, H=3.40, P=0.183>0.05. It meant that there was no statistically significant difference in the tangential resistance data of the left maxillary Le Fort I osteotomy. In genioplasty, H=6.06, P=0.048<0.05. It could be considered that there were differences of the three groups in the tangential resistance data of genioplasty, and the differences were statistically significant. After pair comparison, the difference in tangential resistance of genioplasty between Group 1 and Group 2 was significant (P=0.057 after adjustment); the difference between Group 2 and Group 3 was not statistically significant (P=0.206 after adjustment); there was no statistically significant difference between Group 3 and Group 1 (P=1.000 after adjustment).

DISCUSSION

In recent years, surgical robot is applied in the field of craniomaxillofacial surgery, which can be used for tongue base and pharyngeal tumor resection,18 radical neck dissection,19 microsurgery,20 but they belong to soft tissue surgery. Few experimental operations about craniomaxillofacial bone surgery such as robot-assisted dental implantation21,22 or mandibular reconstruction23,24 were reported, no analysis of force data was carried out. Therefore, this article introduced the collaborative control method and experimental research on robot-assisted craniomaxillofacial osteotomy based on the force feedback and optical navigation.

The biomechanics of craniomaxillofacial bones is complicated.25,26 The anatomy of the craniomaxillofacial bones is complex for its irregular shape and the uneven distribution and great variability of the cortical bone and the cancellous bone. Moreover, human bones vary for body parts, sex, and ages. There are also bone abnormalities caused by bone lesions such as benign and malignant tumors of the jaw, cyst of the jaw and fibrous dysplasia, etc. These factors make the stress distribution of the surgical area more complex.

When the craniomaxillofacial surgical robot performs bone surgical tasks such as cutting, sawing, drilling, grinding, etc., it can access real-time information of the force changes of the surgical area, which is helpful to protect the important blood vessels and nerves around it. Research team27–29 of Heidelberg University in Germany reported that when the surgical robot assisted to the experiment on animals of craniomaxillofacial osteotomy, it was able to integrate the force sensor so as to restrain the reciprocating saw at the end-effector and improve the security, but it was not able to provide force information for the surgeon simultaneously and truthfully.

In our previous work, the surgical robot could complete an osteotomy according to the preoperative virtual surgical planning with good accuracy and feasibility, the osteotomy error was 1.12±0.20 mm.10 Hereby, with the assistance of the collaborative control method based on force feedback and optical navigation of craniomaxillofacial surgical robot, the surgeon is able to simultaneously perceive the force changes of the surgical area through force feedback system and obtain the sense of on-site operation in experiments. Also, the surgeon is able to effectively control the proceeding of the robot’s performance in accordance with his clinical experience. The force feedback system works simultaneously and effectively, which preliminarily verifies the feasibility of the technical route of robot-assisted craniomaxillofacial surgery under the collaborative control method based on force feedback and optical navigation.

However, it is necessary to establish an objective evaluation scale for the simulation of on-site operation based on the force feedback system, invite surgeons with different experience to evaluate the scale and give some suggestion through experimental operations, which is going to further improve the effect of the collaborative control mode based on force feedback and optical navigation. In this way, surgeons are able to control the movement of surgical robots more sensitively.

Sawing Force Analysis at the End-effector of Surgical Robot

When surgical robots perform the sawing surgical task, the blades of the reciprocating saw at the end-effector are mainly subjected to 2 main forces. The first one is the cutting force, also known as the sawing force, which has the same force direction as the vibration of the reciprocating saw. The power of the force is mainly related to the number of saw blade tooth, vibration frequency, cutting speed, and quality of the cutting material.30–33 The second one is the feed resistance of the reciprocating saw in the direction of movement, which is opposite to the direction of the surgical robot. The power of resistance is related to factors such as the force generated by the feed speed and the contact force between the blade and the surgical area. The forces of the reciprocating saw are shown in the Figure 4.

In the actual sawing process, the cutting force and the feed resistance can be understood as the normal resistance whose direction is along the normal direction of the movement track (z-axis) and a tangential resistance along the tangential direction of the movement path (x-axis). Mainly affected by external factors, the normal resistance is uncontrollable. Besides, like the drilling and grinding process,34,35 the force changed with the temperature during the cutting process. Continuous water irrigation on the cutting surface was applied in this experimental test. Because the external irrigation had a major effect on reducing the temperature and minimalizing the thermal effect.35,36 Therefore, the tangential resistance becomes the main reference to the force of the sawing process and the basis of reproducing force sense.

Analysis of Experimental Conditions

During the sawing surgical task, the forces on the surgical robot were relatively complex and there were many factors. This experiment conducted a statistical analysis of the test results in experimental conditions to reduce the errors caused by different experimental conditions of different groups.

To ensure the consistency of the cutting objects, this experiment was conducted by using the same original data of the skull model, the same 3D printer and the 3D printing resin material with the same brand.

All the osteotomy tasks were completed by the craniomaxillofacial surgical robot system CMF ROBOT under the collaborative control mode based on force feedback and optical navigation. The preoperative surgical simulation and trajectory planning were unified, with the operating speed set at 0.5 mm/s to ensure the full amputation of the skull model and the consistency of experimental methods.

The power system and the saw blade were the same. Their vibration frequencies were set at 25,000 cpm to keep the experimental conditions consistent.

Result Analysis of Tangential Resistance

The tangential resistance data of bilateral maxillary Le Fort I osteotomy were similar, whereas the data of genioplasty were significantly higher, which had similarities to the anatomy and biomechanics of craniomaxillofacial bones.

On the basis of the statistical results, the osteotomy speed of surgical robot was set at 0.5 mm/s, and there was no statistically significant difference in these 3 tangential resistance data obtained from the right maxillary Le Fort I osteotomy and the left maxillary Le Fort I osteotomy, which indicated that the force feedback system was effective in obtaining force data from the surgical area repeatedly.

However, 3 groups of tangential resistance data obtained by genioplasty showed no statistically significant difference between Group 1 and Group 3 or between Group 2 and Group 3, whereas the difference between Group 1 and Group 2 was statistically significant. The possible reason was that the thickness of the chin is larger than that of the maxilla, and the bone condition was more complex than that of the bilateral maxilla. Thus, the resin density of the skull model was different from that of the maxilla, which resulted in greater adhesion force due to the heat generated by sawing. This uncontrollable force might be an important interfering factor in this experiment.

In addition, there were many findings from the experiment. (1), the bone mechanical properties of the skull model were hard to simulate. The surgical trajectory and the operating speed of each group were the same, but the data of tangential resistance obtained each time were different, which indicated the biomechanical characteristics of the skull model and craniomaxillofacial bone, such as hardness, stiffness, etc., were different.37,38 The simulation material was not able to accurately distinguish the compact bone and cancellous bone. Therefore, it is necessary to carry out animal experiments or cadaver experiments for verification in further studies. (2), the number of samples was few. However, the number of animal or cadaver simulation surgery samples can reasonably be estimated on the basis of this preliminary experimental results. (3), the tangential resistance data obtained in this experiment was smaller than that of the cadaveric craniomaxillofacial bones.30 In the following studies, more force data should be obtained through experiments on animals or cadavers, and then the force feedback device should be adjusted based on these data. Besides, surgeons with different clinical experience should also be invited to evaluate the force simulation of different craniomaxillofacial osteotomies. The parameters of the force feedback device, such as its speed and accelerating speed, should be adjusted so as to improve the simulation of the on-site operation sense and give full play to the force feedback system in robot-assisted surgery. (4), the operating speed was set at 0.5 mm/s to test the repeatability of the force feedback system. As the speed was slower than that in clinical practice, more studies on tangential resistance should be carried out at different speeds. The relationship between force and speed may lay a theoretical foundation to improve the intelligence of surgical robots, and therefore the robot is able to actively take charge of the surgery by perceiving force changes from the surgical area.

Finally, the craniomaxillofacial surgical robot was only able to perform osteotomy, such as simple translation and rotation at present. And the fast predetermined speed made it easy to deform surgical instruments, thus generating abnormal resistance. Therefore, it is necessary to do further research to make the surgical robot imitate the up-and-down shaking of the human hand during the process of osteotomy, which is expected to overcome the osteotomy jam caused by excessive feed resistance and insufficient cutting force.

CONCLUSION

The feasibility of the collaborative control method based on force feedback and optical navigation of robot-assisted craniomaxillofacial osteotomy was preliminarily proved by experimental operation. In this method, the craniomaxillofacial surgical robot can complete the specified osteotomy task according to the preoperative planning, achieving accurate surgical outcome and saving the surgeon’s energy. The navigation system provides visual feedback to the surgeons. The force feedback system provides force feedback to surgeons, they can obtain the force information from the surgical area through the force feedback joystick. They can interact the robot-assist osteotomy depending on their clinical experiences, improving the robot’s safety.

In the not so near future, with the development of telecommunication technology, under the collaborative control method based on force feedback and optical navigation, surgeons, and their teams might control surgical robot to carry out craniomaxillofacial osteotomy for long-distance patients through network. The flexibility and intelligence of surgical robot can be improved by deep learning to master the force characteristics of surgeon in different bone surgery.