A digital twin-driven human-robot collaborative assembly approach in the wake of COVID-19

Abstract

In the wake of COVID-19, the production demand of medical equipment is increasing rapidly. This type of products is mainly assembled by hand or fixed program with complex and flexible structure. However, the low efficiency and adaptability in current assembly mode are unable to meet the assembly requirements. So in this paper, a new framework of human-robot collaborative (HRC) assembly based on digital twin (DT) is proposed. The data management system of proposed framework integrates all kinds of data from digital twin spaces. In order to obtain the HRC strategy and action sequence in dynamic environment, the double deep deterministic policy gradient (D-DDPG) is applied as optimization model in DT. During assembly, the performance model is adopted to evaluate the quality of resilience assembly. The proposed framework is finally validated by an alternator assembly case, which proves that DT-based HRC assembly has a significant effect on improving assembly efficiency and safety.

1. Introduction

Due to the influence of COVID-19, the demand for medical equipment is increasing greatly. Meanwhile, the disruption of supply chain caused by transportation restriction further aggravates the manufacturing pressure. 3D printing uses CAD model from computer to generate corresponding physical model, which can effectively alleviate the shortage of products and parts. During epidemic, the wide application of 3D printing will accelerate the design work of digital model and digital manufacturing. For complex medical devices, the assembly parts of product are numerous with various assembly constraints. The traditional assembly mode for this kind of products is mainly manual assembly. Because of the low efficiency and poor accuracy consistency, manual assembly can not meet the demand of assembly market. Recently, robots have been widely used in different industrial fields, such as welding, assembly, spraying and processing [1,2]. The automatic production line with multiple robots can reduce the risk of epidemic transmission and improve the assembly efficiency. Due to the limitation of robot’s ability, the complex and flexible operations in assembly tasks still need to be done by human. So the HRC assembly in a shared environment is an appropriate way to realize safe and efficient assembly under the epidemic [3]. Meanwhile, the 3D printing integration [4] in HRC assembly system can make the components completely mapped from their digital models, which promotes the formation of industrial automation and intelligent factory.

The HRC assembly contains many modules, such as modeling of assembly tasks, capability evaluation between human and robot, generation of task sequence, allocation of assembly tasks, planning of action trajectory and so on. For enhancing the safety of HRC, the intelligent algorithms are used to plan the robot’s motion path and selectively avoid human worker in assembly. Besides, co-robots will provide assistance by recognizing the operation intension of human, making HRC more harmonious. Human-robot interaction (HRI) is an important part in HRC assembly which is multimodal and real-time. But the current HRC assembly system lacks enough adaptability to update strategy quickly when assembly environment changes. At the same time, the perception ability towards environmental data is weak and the assembly system lacks strong cognitive ability.

Digital twin (DT) is proposed by Grieves with three basic parts: physical products in real space, virtual products in virtual space and connections of data and information. It maps all the elements of physical space into virtual space with digital model. The functions of monitoring, prediction and optimization are compatible for HRC assembly. In DT-based HRC assembly, the human-robot interface (HRI) is adopted for presenting the assembly process and providing visual control. The environmental data is real-time collected by physical sensors and the assembly strategy is optimized synchronously. The prediction module is used to predict the assembly task and the state of assembly system. According to the collected data, the optimization module will provide an optimal solution for each assembly state. In order to enhance the adaptability of DT-based HRC assembly system, this paper takes reinforcement learning (RL) as the optimization model of DT. The RL model in assembly system can learn the optimal HRC strategy and action sequence for assembly tasks. Because RL model accumulates knowledge by the interaction between assembly system and environment, it can generate assembly strategy quickly whether assembly task changes. The proposed DT-based HRC assembly system can efficiently carry out the assembly task with the safety of human and robot ensured.

The contributions of this paper are summarized below:

• •Propose a HRC assembly system based on DT to deal with the rapidly increasing demand for medical equipment. Then the physical assembly work is guided through real-time data collection and policy update.
• •The assembly environment is segmented after analyzing the assembly elements. Then the static and dynamic information of assembly environment is recorded in XML files with digital form.
• •The information of assembly task is extracted from XML files and input into data management system to form the action sequence which is related to HRC assembly. The D-DDPG is proposed to realize the selection of cooperative strategy and planning of action path.
• •From the perspective of assembly process and product quality, the performance evaluation model is used to assess the comprehensive performance of DT-based HRC assembly.

The remainder of this paper is organized as follows: related work is briefly presented in Section (II). The DT model for HRC assembly is introduced in Section (III). The environment of HRC assembly is stated in Section (IV). Section (V) describes the specific operations of HRC assembly. Section (VI) presents an HRC assembly case for evaluating the proposed system. Finally, conclusions and future works are discussed in Section (VII).

2. Related work

With the application of industrial robots, automatic assembly line composed of robots has been widely used. The robots can effectively deal with monotonous and heavy tasks with their high precision and repeatability. In industry, the robot is controlled by teaching which means that users need to compile fixed program in advance. At the same time, the assembly process of flexible and complex products is accompanied by certain deformation and uncertain collision risk, making robot unable to complete overall assembly task alone. Due to the emergence of co-robots, more research is focused on the process of human-robot cooperation. This method combining the flexibility of human and stability of robot is suitable for the current assembly requirements. However, the cooperative environment is dynamic with multi-source heterogeneous data, the control strategy of robot needs to be adjusted accordingly. For achieving intelligent perception and decision-making, DT maps physical assembly space to virtual environment which realizes the monitoring, prediction and optimization function. Next, the review work will be presented from robotic assembly and HRC assembly.

2.1. Assembly using robot

Robotic assembly refers to the assembly process which is completely executed by one robot or a group of robots. In that, energy consumption and production efficiency are two important factors to be considered. Table 1 shows the research on optimizing energy consumption and efficiency of robotic assembly. The restarted annealing algorithm adopted by Li [5] can deal with the complexity of model and obtain the well-spread Pareto-optimal set with unique structure. Janardhanan [6] proposed an improved multi-objective co-operative co-evolutionary (MOCC) algorithm to enhance local search capability and speed, making energy consumption less and production efficiency higher. After that, the Migrating Birds Optimization (MBO) algorithm [7] was compared with common heuristic algorithms and presented the better search performance. Zhou [8] divided the energy consumption of assembly into four parts and considered the conversion time based on sequence. Then the enhanced decomposition-based multi-objective (MOEA/D) algorithm was used in organizing robotic assembly line and effectively achieve the trade-off between production and energy consumption.

Table 1.

Optimization of robotic assembly on energy and efficiency.

Method	Author	Remarks
A new mixed-integer programming model	Li [5] (2016)	Minimize the cycle time and energy consumption.
An improved MOCC algorithm	Janardhanan [6] (2017)	Minimize carbon footprints and maximize assembly efficiency.
A recently developed MBO algorithm	Janardhanan [7] (2019)	Minimize the cycle time of robotic assembly.
An adaptive MOEA/D method	Zhou [8] (2020)	Realize the trade-off between productivity target and energy consumption target.

The safety, accuracy and success rate reflect the performance of robotic assembly line. Meanwhile, these indicators are usually improved by some robotic control strategies. As shown in Table 2 , Jayaweera [9] adopted the non-contact sensor to calculate the deformation of parts and output the correct assembly position. For flexible assembly task, Chen [10] has proposed a robust control strategy to avoid damaging the assembly parts or system. In order to further improve the accuracy of assembly position, a monitoring algorithm of assembly process was designed by Lee [11]. This method was validated to identify the assembly status and improve the assembly safety, accuracy and success.

Table 2.

Optimization of robotic assembly on safety, accuracy and success.

Method	Author	Remarks
Laser vision system and ‘Best-fit’ algorithm	Jayaweera [9] (2007)	Provide assembly position and improve assembly accuracy.
A robust impedance control method	Chen [10] (2013)	Provide more robust and safer performance in controlling robot.
An assembly process monitoring algorithm	Lee [11] (2019)	Identify assembly state and improve success rate of assembly.

The action and assembly sequence of robot is related to the assembly efficiency. In Table 3 , the planning methods of action sequence and assembly sequence can be divided into three categories: Programming algorithm represented by random tree, Heuristic method and Reinforcement learning. Guo [12] put forward the RRT-BwC (Bi-direction with constrained) algorithm to overcome the obstacles of high-dimension space planning. To improve the planning efficiency, Ying [13] proposed a Cyber-Physical Assembly System (CPAS)-based metaheuristic to realize rapid planning of assembly sequence. Comparing to these traditional methods, reinforcement learning has gradually become the hot-spot method for dynamic assembly planning. Watanabe [14] adopted Q-learning algorithm for searching efficient assembly with integrating past learning knowledge. Song [15] made the visual and force perception as observation of deep Q learning. After continuous interaction with environment, the assembly strategy containing action sequence could be obtained.

Table 3.

Optimization of robotic assembly on action and assembly sequence.

Method	Author	Remarks
RRT-BwC algorithm	Guo [12] (2020)	Provide effective operation sequence for robot operation.
A CPAS-based metaheuristic	Ying [13] (2021)	Generate more efficient assembly sequence in shorter computing time.
Q learning algorithm	Watanabe [14] (2020)	Search efficient assembly sequence and assign tasks
Deep Q learning algorithm	Song [15] (2021)	Observe assembly status and learn assembly strategy.

2.2. HRC assembly

For complex and flexible assembly task, the robot lacks the ability to deal with it independently. The effective method to improve the flexibility of assembly system is to combine the flexibility of human with the efficiency of robot. Human-robot cooperation can reach the adaptive level based on smooth interaction and intention recognition. Liu [16] modeled the assembly task of product as human motion sequence. Then the robot could be controlled to assist human worker by predicting human motion. Due to the strong autonomy and interoperability of Cyber-Physical Production System (CPPS), it is rapidly applied in manufacturing process. Yao [17] introduced the function blocks (FBs) in CPPS and improved the efficiency of task planning, monitoring and control in HRC assembly. Darvish [18] proposed a flexible HRC assembly architecture which integrates sensing, representation, planning and control. The online reasoning was carried out to complete HRC assembly after recognizing human actions. For exploring the HRC in shared workspace, Wang [19] pointed out the direction of research according to the current state of multimodal human-robot interaction.

DT was first proposed by Grieves [20] which was a real-time simulation and optimization process with multi-disciplinary, multi-physical quantity, multi-scale and multi-probability. Based on the three-dimensional model, Tao [21] put forward the five-dimensional model which contains Physical entity, Virtual equipment, Services, DT data and Connection. With the comprehensive functional feature, DT has been applied and researched on many occasions. In Fig. 1 , Sierla [22] obtained the DT from digital product description, then the assembly planning and resource allocation could be automatically achieved. Sun [23] studied the assembly-commissioning process of high precision products (HPPs) with DT. Based on the total factor model of DT assembly, the assemblability prediction and assembly optimization were realized. Guo [24] put forward a digital twin-enabled Graduation Intelligent Manufacturing System (DT-GiMS) to solve the assembly work in fixed island. This framework integrated the physical layer, digital layer and service layer, which could provide visual information for the manager. In assembly, assembly process reflects assembly sequence and operation method. Yi [25] not only proposed a DT model for intelligent planning of assembly process, but also provided an application framework for DT-based assembly with three layers.

Fig. 1.

HRC assembly and digital twin.

In HRC, the safety of human is difficult to be ensured with the uncertainty of robot operation. So the description of variable space and human body is particularly important. DT can provide users with visual control function by collecting real-time data in physical space and restoring it in virtual space. Droder [26] has researched the function of machine learning on controlling the behavior of robot in DT and found that robot can automatically avoid obstacles with the machine learning methods. Oyekan [27] has built the DT workshop to analyze the human reaction towards the predictable and unpredictable motion of robot. The reaction of human to robot’s behavior in real scene was related to that in virtual scene, which indicated that DT could provide information for safe HRC. Driven by DT, the interaction between human and robot will become smooth and frequent. Ma [28] used the five-dimensional DT model and designed human-robot interface to realize convenient human-machine interaction (HMI). The DT HMI provided rich information of high assurance virtual model, product life cycle data and services, making HRC more efficient. Wang [29] applied the visual question answering (VQA) technology in the DT for efficient HMC. Compared with the traditional method based on static image, the proposed VQA method could understand the video information which is more suitable for dynamic HRC scene.

Due to the functions of monitoring, prediction and optimization in DT, it can provide an effective cooperation strategy for HRC assembly. Malik [30] used the digital part which was continuously mirroring the physical part to simulate the assembly plan. The proposed DT framework could carry out on-line or off-line experiment, avoiding any economic loss and personal injury in actual production. Bilberg [31] has established a corresponding DT for flexible assembly unit in which the use of simulation model is extended to real-time control, task allocation, task sequencing and program development. This framework could avoid any economic loss and personal injury with improving the efficiency of HRC. Kousi [32] used the digital modeling technology in production system, making the system reconfiguration realized through shared environment and process awareness. The experimental results showed that the application of DT could be adaptive to different scenes and improve the flexibility of HRC assembly system. Baskaran [33] has built the DT model of human and robot with the Simens Tecnomatix suite to explore the relationship between ergonomics and assembly tasks. During HRC assembly, the robot in virtual space provided physical help to human and effectively evaluate the assembly results.

Due to the demand change of external environment, the production mode of manufacturers needs to be dynamically adjusted. The production line needs to be configurable and resilient so that it can sensitively switch the production plans according to the change of orders. The current DT-enabled HRC assembly system is an appropriate way to meet the flexible assembly requirement. However, the services of DT usually adopt separate model to achieve the design of HRC strategy and the planning work of robot motion path. The lack of relevance among services affects the planning efficiency of action sequence between human and robot. At the same time, the assembly system lacks the ability to enhance learning which is not suitable for dynamic assembly scene. The application of DT model with learning ability to HRC assembly will provide power support for diversified and rapid manufacturing industry.

3. The resilience assembly framework

3.1. Resilience framework

The HRC assembly workshop contains a set of parallel assembly working stations which have all elements of HRC assembly environment. In that, a single workstation will handle the assembly task of one product. During assembly, the parallel workstations can share assembly tasks when one workstation stop assembly by machine fault or assembly error. With the dynamic adjustment of assembly resource, the performance of assembly system can be steadily improved.

In Fig. 2 , a DT framework for HRC assembly is proposed with four parts: physical assembly space, virtual assembly space, data management system and DT data. The virtual assembly space of DT-based HRC assembly system is mapped from physical assembly space. The data in physical space is collected by multiple sensors and transmitted to data management center in real time. Through analysis of DT data, the geometric model, assembly behavior and assembly performance will be updated accordingly. Before assembly, the list of assembly tasks will be generated based on the assembly matrix. The feasible assembly sequence is obtained according to the matching relationship among parts. To achieve the balance of human-robot load, the appropriate evaluation index towards task properties, the capability of human and robot is proposed in data management system. The cooperation strategy of assembly task is determined by calculating the comprehensive score. The obtained assembly strategy will first drive the virtual model for simulation, then decide whether to use in real assembly. Different cooperation strategies have different action sequences, which can be represented in the form of trajectory points. Therefore, the movement of robot will be controlled by the command of “MoveL”、“MoveP”in TCP/IP protocol. Based on the 3D bounding box of human body, the static motion range of human is marked, which provides convenience for trajectory optimization. The recognition of operator’s intention is the key of HRC in shared environment. After the action recognition network is integrated into data management center, the assembly behavior of human will be recognized automatically. Then the system gets the next assembly task by reasoning and control the robot to assist human. In picking part, the cooperative behavior between human and robot is as follows: “Human behavior: extend palms straight out—Robot response: deliver the captured part to human”. In assembly, robot assists human fix the part by screws with cooperative behavior as follows: “Human behavior: place the part on base part with constraints—Robot response: assist in screwing”. In Fig. 2, RL is taken as the optimization model in DT and used for trajectory optimization and posture adjustment. At the same time, human workers occupy the dominant position in the proposed framework with the ability of intervening and controlling robot in multimodal.

Fig. 2.

The resilience assembly framework based digital twin.

The assembly behavior of HRC in physical space is fully synchronized with virtual space. It has integrated some features such as geometry, physics, behavior and rules in virtual space. For current assembly task, it can verify the cooperative strategy and action sequence by simulation. After that, the DT data will be transferred to data management system and output the optimal assembly plan, including resource scheduling, assembly sequence, action sequence. The visual HRI shows the whole process of assembly and human can master the key information of HRC.

Compared to manual assembly or pre-programming assembly, there are some advantages in DT-based HRC assembly:

• (1)With physical sensors, the monitoring of HRC assembly is realized by the collection of real-time data.
• (2)The digital equipment improves the storage of real-time data and accelerates the simulation of cooperative strategies.
• (3)The RL model used in assembly provides optimal action sequence and improves the learning ability of assembly system.
• (4)In DT-based HRC assembly, the safety of human and robot is guaranteed by intelligent planning of motion.
• (5)Based on the recognition of human assembly actions, robot can make auxiliary preparation in advance.
• (6)The data interaction or fusion among DT layers improves the interoperability of assembly system.

3.2. Data fusion and interaction based digital twin

3.2.1. Data fusion

In complex assembly, the components of products have different structures, constraints, making large-scale data involved. The position and structure data of parts can be imported into data management system with visual equipment. During assembly, the physical space provides the real assembly data while virtual space generates the simulation data and monitoring data. The accuracy data and control data constitute the main information flow of DT assembly. In HRC assembly, the accuracy reflects the quality of assembly operation and results. The accuracy data of HRC assembly includes dimensional accuracy, relative position accuracy, relative motion accuracy and contact accuracy. For the control data of HRC assembly, it includes action data, sequence data and force data. The conversion of data format which is mainly divided into structured data and unstructured data is the key to realize the interaction of DT layers. For structured data, the assembly system can directly process it and output optimal solution. The structured data will be stored in data management system and used for model interoperability. Except for images and point cloud, the vibration noise caused by machine operation or action stream from human is also unstructured data. This kind of data needs to be translated into structured data so that it can be used for analysis and decision. In Fig. 3 , the main features of image and point cloud are extracted by convolutional neural network (CNN) and stored in data table. Recurrent neural network (RNN) is suitable for processing the vibration noise and predict the stream data with its temporal characteristics. According to the assembly constraints of products, the assembly sequence is mainly divided into two categories: sequential or non-sequential sequence. The latter can be transformed into sequential by data matching and sequence rebuilding. To achieve data flow among DT layers, the TCP/IP protocol is adopted to provide conditions for smooth data transmission.

Fig. 3.

Data fusion between digital twin assembly spaces.

3.2.2. Data interaction

In HRC assembly, the main features of products are the structure, size and assembly constraints of components. These features will be encoded and sent to data management center for generating the assembly sequence. After optimization, the optimal assembly sequence will be executed in physical space. Then the effective assignment of tasks between human and robot is formed according to the results of comprehensive evaluation. In addition to the static data of assembly parts, the dynamic data related to the motion of human and robot will also be sent to data management system. The bounding box of human body presents the range of human motion. The center of 3D bounding box (3D bbox) is used for obtaining the spatial position and moving speed of human. At the same time, the 3D skeleton of human is adopted for recognizing the assembly actions of human. After recognizing the intention of human worker, robot performs the auxiliary operation for specific task. The control command of assembly system is mainly represented as the position of trajectory point, the posture of gripper or control signal to the gripper. During assembly, the running data of robot reflects the health state, such as current, power, vibration, magnetic and so on. With the interaction between physical space and data management system, the performance of HRC assembly is improved in physical space. The data interaction between these two layers is shown in Fig. 4 .

Fig. 4.

Data interaction between physical space and data management system.

In Fig. 5 , the data interaction between virtual space and data management system enables the real assembly. The geometric model of virtual space is established and updated by collecting the physical parameters from physical space, such as size, structure and appearance. The assembly sequence and results of task assignment will work as the running command to drive the collaborative assembly in virtual space. In digital HRC assembly, the ergonomic data and assembly results obtained by monitoring can be analyzed in data management system. At the same time, the state of assembly environment will output from virtual space, such as position and speed of environmental elements. As an important module in virtual space, simulation provides the performance of assembly strategy. After simulation, the assembly data is got by means of statistics, such as safety, efficiency, accuracy and stability. Besides, the function and operation performance of products are measured by satisfaction table. This part of data will finally be transferred to data management system through data interface. Then the assembly strategy is adjusted by further analyzing the assembly effect. The new assembly strategy includes the assembly behavior of human and robot, the trajectory of robot and the posture of gripper. Through format conversion, this strategy will be executed in the form of controllable instructions in virtual space. Due to the high consistency between physical assembly space and virtual assembly space, the assembly strategy which meets the production requirements will be applied to real assembly from simulation.

Fig. 5.

Data interaction between virtual space and data management system.

4. Human-robot collaborative environments

4.1. The segmentation of assembly environment

In actual assembly, the characteristics of assembly environment include diversity, dynamism and complexity. So it is necessary to segment the environment and establish the all-factors DT model. Through the digital representation of assembly environment, the assembly simulation, prediction and optimization will be more convenient.

In HRC assembly, the point cloud of environment is directly obtained by laser scanner. The point cloud is unstructured data with high redundancy, which can not be directly processed in virtual space for model reconstruction. So the point cloud segmentation algorithm is adopted to segment the assembly environment. Vo [34] proposed a novel octree-based region growing algorithm for point cloud segmentation. This algorithm was applied in surface patch segmentation of urban environment. For the complex HRC assembly space, the octree-based point cloud segmentation algorithm is suitable for segmenting this environment. The operation of segmentation is mainly divided into two stages: region growing and refinement. The first step is to set the size of smallest voxel or the maximum number of sampled points in voxel. Then the original point cloud is continuously decomposed according to the principle of octree segmentation. After growing of region, the initial segmentation block is incomplete and many unassigned points exist in the assembly space. So the second step is to refine the original block and assign the unallocated points to the incomplete blocks. The boundary voxels of incomplete segmentation blocks are extracted and the voxels near boundary voxels are found in refinement stage. Then the unassigned point will be classified into the nearest block by sorting the distance of this point and segmentation blocks. Finally, the surfaces of all blocks in assembly environment will be smooth and the environmental factors can be obtained.

4.2. The digital representation of assembly environment

The physical properties and current assembly state are complex and diverse, making it challenging to represent the physical entities in virtual space. So using the digital form to describe the features of products and environment is essential for synchronizing DT spaces. The XML files are applied to save the features of point cloud and update the models in digital environment. In Fig. 6 , the XML files corresponding to static environment mainly contain three parts: physical properties of assembly parts, assembly relationship, assembly process. The current state of assembly parts contains position, occlusion and posture. The XML files record the object IDs of two parts which have the assembly relation. Then the relative distance, relative posture and assembly constraints will be presented to make the virtual and real environment synchronous.

Fig. 6.

The digital representation for assembly environment.

Although assembly environment is constantly changing during assembly. The dynamic elements can also be obtained by proposed point cloud segmentation algorithm in temporal sequence. In Fig. 6, the elements in dynamic environment are represented by segmentation blocks which are composed of voxels. The feature of object is represented by a group of voxels and the moving speed is transformed from the speed of voxels. Besides, the boundary voxels can be used to analyze the risk of human-robot collision. When the trajectory of human and robot intersects, the data management system will output an improved solution to solve it. Human body has some special attributes, including gender, height, weight, and movable range while robot has the specific attributes, such as the opening of gripper, movable range, maximum load and working time. In XML file, the service block marks the working state of behavior subjects. For the state representation, human and robot have different items to be labeled. The position, moving speed, posture and assembly action can reflect the assembly state of human. Except for position, speed and posture, the state of robot contains other dynamic signals: power, current, vibration, magnetism and moment. These dynamic signals contained in XML files can be visualized in HRI and updated with the assembly process.

Due to the complexity and dynamics of assembly environment, the assembly system is affected by many factors: (1) the status of co-robot; (2) the characteristics of human worker; (3) the properties of assembly tasks; (4) the stacking status of parts; (5) the communication quality of network. During assembly, the running signal of robot can represent its health state. When the robot working for a long time appears excessive voltage, power or vibration, it will be stopped to health assessment and troubleshooting. From the perspective of human, the fatigue caused by long working will reduce assembly efficiency and enhance the risk of HRC. Besides, the response of different operators to the predictable and unpredictable motion of robot is different, making the smoothness of HRC operation inconsistent. For the assembly tasks that human and robot are not suitable, the system will refuse to assemble and prompt the user to adjust plan. At the same time, the stacking condition among assembly parts will lead to the additional cost to adjust the posture of gripper. The effectiveness of communication is related to the effectiveness of interaction. Because the robot is driven to assist human by control command, the quality of communication network affects the assembly performance to some extent.

5. Human-robot collaborative operations

5.1. Task classification and re-scheduling

For new task, there are assembly constraints among components in product. As shown in Fig. 7 , the assembly task of battery box consists of several major parts and screws. The stability constraints matrix ($SCM$) describes the reliability of current assembly. When the process of assembling part $p_j$ on part $p_i$ is invalid or unstable, mark $(p_i,p_j)$ in $SCM$ as 0. Otherwise, it is 1. For example, in Fig. 7, the part $f_1$ is not valid to be assembled on part $c_1$. Thus, $SCM(f_1,c_1)$ is 0.

Fig. 7.

The constraints matrix of assembly task.

The assembly constraints matrix ($PCM$) is developed from matrix ($SCM$) and used to describe the assembly relationship among parts. If the part $p_j$ is next to the part $p_i$, the element ($p_i$, $p_j$) in the matrix $PCM$ is 1. Otherwise, it is 0. For example, in Fig. 7, the part $c_1$ and $c_2$ have the assembly relationship, the element $PCM(c_2,c_1)$ is 1.

As shown in Fig. 7, the assembly process of new product can be obtained from matrix ($PCM$) and ($SCM$). Then the optimal assembly sequence will be chosen from all feasible assembly sequences.

After getting the assembly sequence, all tasks need to be assigned to human and robot. The sub-task is evaluated from the object attributes and the capacities of human and robot. The physical properties include weight, volume, length, flexibility, interface tolerance and others. The human capacity is represented by grasp range, grasp strength, flexible ability, perception range, task adaptability and others. For robot capacity, the main indicators are grasp load, grasp range, moving speed, perception range, continuous operation time and others. In above evaluation dimensions, the score range of each evaluation item is [0,1]. The average scores are obtained by summing the values of all evaluation items and averaging it as Eqs. (1), (2), (3). In Eq. (4), the final scores are related to the results of task assignment.

$$S_O=(\sum _{i=0}^l{P}_i^O)/l({0<P}_i^O<1)$$ (1)

$$S_H=(\sum _{i=0}^m{P}_i^H)/m(0<P_i^H<1)$$ (2)

$$S_R=(\sum _{i=0}^n{P}_i^R)/n(0<P_i^R<1)$$ (3)

$$C_1={\omega }_1\times S_O;C_2={\omega }_2\times S_H;{C_3=\omega }_3\times S_R$$ (4)

Where $S_O$ stands for the average score of part attribute, $S_H$ stands for the average score of human capacity and $S_R$ represents the average score of robot capacity. $l,m,n$ are the length of evaluation items from three aspects. $C_1,C_2,C_3$ stand for final scores and ${\omega }_1,{\omega }_2,{\omega }_3$ are scoring weights.

The reference table for selecting action sequence is presented in Fig. 8 . In this table, there is the priority relationship among these three final scores ($C_1,C_2,C_3$). If $C_1$ is in range of [0, 0.5], the assembly task initially belongs to the human-focused task. Otherwise, the assembly task initially belongs to robot-focused task. $C_2$ and $C_3$ are additional conditions which are used to further confirm whether human or robot is competent for the current task. For example, human can better handle the assembly of flexible parts ($C_2\epsilon [0.5,1]$) while robot is not suitable for flexible tasks ($C_3\epsilon [0,0.5]$). For human-focused task, the robot will pick up object and hand it to human for assembly. While the action of human and robot is reversed in robot-focused task.

Fig. 8.

Task classification and re-scheduling for emergence needs.

5.2. Action sequence

The result of task classification depends on the elements of [$C_1,C_2,C_3$]. So RL is adopted to obtain the best weights in dynamic environment. As shown in Fig. 9 , a double deep deterministic policy gradient (D-DDPG) framework is designed to realize joint learning of action sequence and action path. In d-DDPG framework, DDPG1 is mainly responsible for obtaining the best action sequence and DDPG2 is used for getting the optimal robot trajectory. For each DDPG model, the relationship between long-term expected return and current reward is shown in Eq. (5). In DDPG1, the element of $\left[{\omega }_1,{\omega }_2,{\omega }_3\right]$ will adjust dynamically with the change of Q value, so that the action sequence will change with less time consumed in assembly task.

Fig. 9.

The framework of D-DDPG.

D-DDPG is adopted to speed up the training process of networks and increase the reward. In that, two specific tricks are adopted for improving the performance: (1) In a single epoch, the action sequence (represented by groups of start position, target position and action type)) of human and robot which are output from the target network of actor will form as the input of DDPG2. (2) The optimal trajectory obtained from DDPG2 is stored in trajectory memory pool. Then the moving trajectory of robot in DDPG1 will be replaced by the optimal trajectory sampled from memory pool.

The training time of D-DDPG model is reduced by bidirectional acceleration function. At the same time, the proposed model can output better action sequence and motion trajectory of robot in HRC assembly.

$$V\left(s_t\right)=r_t+\gamma r_{t+1}+\dots +{\gamma }^{T-t}{r}_T(1\le t<T)$$ (5)

$$\left[{\omega }_1,{\omega }_2,{\omega }_3\right]=f(Q(s_t,a_t))$$ (6)

In training, the state space, action space and reward function are presented in Table 4 . For DDPG1, the state space consists of six values: working time of human ($t_H$), working time of robot ($t_R$), the complete time of task ($t_{all}$), fluency of collaboration ($F$), fatigue of worker ($f$), safety of worker ($S$). Besides, the action sequence obtained from reference table will form as the element in action space. In Table 4, the reward function of DDPG1 is related to the elements of state space. The corresponding reward will be bigger when the completion time of task is less. [${\alpha }_1、{\alpha }_2、{\alpha }_3、{\alpha }_4、{\alpha }_5、{\alpha }_6$] is a weight tuple which will drive agent train in a right direction.

Table 4.

State space, action space and reward function of D-DDPG.

RL Loop	Observation	Action	Reward
DDPG1	[$t_H,t_R,t_{all},F,f,S$]	$seq[a_{r_1},a_{r_2},a_{h_1},a_{h_2}]$	$r_1={\alpha }_1{t}_H+{\alpha }_2{t}_R+{\alpha }_3{t}_{all}+{\alpha }_{4}F+{\alpha }_{5}f+{\alpha }_{6}S$
DDPG2	[$t_R,L_{path}$]	[${\theta }_1,{\theta }_2,{\theta }_3,{\theta }_4,{\theta }_5,{\theta }_6$]	$r_2={\alpha }_7{t}_R+{\alpha }_8{L}_{path}$

For DDPG2, the observation includes completion time of motion ($t_R$) and length of motion path ($L_{path}$). For action sequence of robot, the motion trajectory is composed of several points. So DDPG2 is mainly used to optimize the trajectory of robot by taking different actions. In exploration stage, the action strategy is the angles of six joints. When taking a certain strategy, the reward will be obtained which is in proportion to $t_R$ and $L_{path}$.

Each element in ${[\alpha }_1\text{、}{\alpha }_2\text{、}{\alpha }_3\text{、}{\alpha }_4\text{、}{\alpha }_5\text{、}{\alpha }_6\text{、}{\alpha }_7\text{、}{\alpha }_8]$ is obtained through experiment.

For a complex assembly task, the whole assembly process is composed of many cooperative strategies, such as human-focused task, robot-focused task and so on. After getting the action sequence of human and robot by D-DDPG, the control sequence of human and robot is shown in Fig. 10 . Collaborative strategy 1 indicates the process of robot transferring parts or assembly tools to human workers. While Collaborative strategy 2 shows the process that human pick up the part and deliver it to robot.

Fig. 10.

The control sequence of HRC.

5.3. Human-in-the-loop collaborative process

Fig. 11 presents the interaction process between human and robot in HRC assembly. In that, the human skeleton in assembly environment can be captured by Kinect camera. If human needs the robot hand over the part to him, he should reach out his hand. At this time, the human skeleton is recognized through action recognition network. Then the intention of human will be estimated by using the Hidden Markov Model (HMM). For example, when human reaches out his hand, the robot can know the next action of human: Obtain the part from robot. The local points of human hand are extracted from human skeleton. So the position difference between key points and TCP point is obtained. The robot can place the part or assembly tool on human hand according to the position difference.

Fig. 11.

Human-in-the-loop collaborative assembly.

When human aligns the part with the base part, the information of human skeleton will be input into data management system through data interface. Then the current action of human can be recognized by action recognition network. The next behavior of human is obtained by HMM and robot will prepare for auxiliary assembly in advance. For example, in Fig. 11, when human aligns parts with assembly relationship, the robot will know the next action of human: fixing the part by screws. Then human first put the screws into holes on assembly and robot will take the screwdriver as the end effector to fix the screws. When the robot closes to the assembly hole, the main constraints are [$x,y,z,\alpha$] and [$x_h,y_h,z_h,{\alpha }_h$] which describe the coordinate of 3D space and dip angle of normal. Through RL, the robot can learn the optimal trajectory to complete the current task.

5.4. The resilience performance evaluation

In Fig. 12 , the performance model of HRC assembly mainly evaluates the performance from two aspects: assembly process and assembly result. The safety of HRC is evaluated by calculating the collision rate and the collaborative efficiency is represented by the number of successful operations in unit time. When the assembly task changes, the adaptability of cooperation is related to the efficiency difference between different assembly tasks. In picking part, the performance index is mainly composed of picking time, picking accuracy and success rate. Besides, the running time of robot is recorded to reflect the picking speed. The accuracy of picking part mainly refers to the deviation between TCP and the center point of part when closing the gripper. For the success rate of picking operation, it is equal to the ratio of successful numbers to total numbers of picking operation. Comparing to the performance evaluation in grasping stage, there are also many evaluation indexes in assembly stage. The complexity of current assembly is denoted by the time consumed in single assembly. Meanwhile, the total deviation is calculated by summing the deviation of all assembly interfaces to measure the accuracy of assembly. Finally, the success rate of assembly can be calculated as picking stage.

Fig. 12.

The performance evaluation model.

In Fig. 12, the performance evaluation of product is taken as the measurement of assembly results. The function of product contains basic function and expanding function. User will feed back a satisfaction score after using this product. So the final evaluation score can be obtained by weighting average of above three items. Except for the above, the operation data of product is also an evaluation index to reflect the reliability of DT-based HRC assembly. The running data of product is multi-source with some representative characteristics, such as the pressure and flow state are the main features in using the ventilator. Through the comprehensive analysis of different signals, the operation performance of product will be output.

6. Case study and discussion

The ventilator and alternator have some common features, such as high complexity and multiple constraints, so that they are mainly assembled by hand or fixed program. With the shortage of medical ventilators, the alternator is used to carry out HRC assembly in this experiment. The DT-based HRC assembly is compared with the traditional assembly from action sequence and assembly performance.

6.1. Experiment environment

The assembly space of HRC is complex which is composed of many elements: human, collaborative robot, sensors, assembly parts, assembly tools and others. The information of these elements will flow into data management system through sensors, so that the virtual space can use this data to update its state. In Fig. 13 , a graphical user interface (GUI) is designed to display the virtual assembly process. Four function modules are set up on this interface to realize the automatic assembly. Then the function of each module is shown as follows:

• (1)Environment building module: the assembly elements can be added or modified in interface to form a new environment.
• (2)Virtual assembly module: the HRC assembly process will be carried out in virtual space with 3D models. The simulation process is used to test the effect of HRC, while the synchronization process is adopted for real-time interaction.
• (3)Control module: human can control the assembly process after confirming the safety of assembly environment.
• (4)Statistics and analysis module: assembly results, such as success rate, assembly quality and abnormal data, will be displayed and analyzed for optimization.

Fig. 13.

The environment of HRC assembly.

6.2. Discussion

In this paper, a group of experiments are set up from three aspects: action sequence, picking part, collaborative assembly. The method used in this experiment is divided into two types: (1) DT-based model; (2) pre-programming model. For the first experiment, the generation of action sequence by DT model or manual designing is researched. In second and third experiment, these two models are used to complete the operation of picking part and assisting human to fix the part. During experiment, the main observation is time, success rate and collision rate. The number of experimental groups is 5 and the final results are the average of all parallel groups.

The alternator used in this experiment consists of 16 different components. According to the assembly constraints matrix, the HRC assembly process includes 15 assembly tasks. After task scoring and initial assignment, the virtual space of assembly system will verify the performance of assignment result. For making the assembly time less and the fluency of HRC higher, the assembly strategy is continuously optimized. In Fig. 14 , the assembly tasks are classified into different categories: Robot-focused task and Human-focused task. After executing the action sequence obtained from DT model, two important results are got as follows:

• (1)With DT assembly method, the total time of HRC assembly reaches 412 s, while the total time of pre-programming assembly is 550 s.
• (2)In pre-programming assembly, human-focused tasks account for 70 % of the whole assembly process, while these tasks only account for 49.76 % in DT assembly.

Fig. 14.

The comparison of HRC assembly between different methods.

In picking part, the different optimization methods of data management system are compared, such as pre-programming method, DDPG model and D-DDPG model. Before training, the number of episodes in DDPG and D-DDPG is set to 1000 and the capacity of memory pool is set to 5000. Meanwhile, the number of steps is 10 and the exploration noise is initialized to 0.1. The assembly part to be picked is determined within the capacity of robot, and the results are the average value of all groups.

Table 5 presents the experimental results of picking with different models. It only costs approximately 14.832 s to pick a specific part with D-DDPG model, while the consumed time is 16.354 s with DDPG model. Compared to the previous models, the pre-programming mode will take 25.645 s for single task which is far from 16 s. The reason for this phenomenon is that the method of pre-programming will make the robot run in a redundant path. From collision aspect, the collision rate of pre-programming assembly has reached 5.208 % which is far from 1.563 % with DDPG model and 0.521 % with D-DDPG model. The reason for this phenomenon is that traditional method can not provide a safe path for robot in dynamic environment. The accuracy of picking is defined by measuring the deviation between TCP point and the center point of part. The picking deviation with pre-programming method is 2.447 mm which is bigger than that with DDPG model (1.875 mm) and D-DDPG model (0.895 mm). For this phenomenon, itis caused by the reason that the robot can not adaptively adjust the grasping pose with fixed program when the position of object changes.

Table 5.

The performance of picking with different methods.

Control method	Picking time (s)	Collision rate	Picking accuracy ($\Delta \mathit{d}$)
Pre-programming	25.645	5.208 %	2.447 mm
DDPG D-DDPG	16.354 14.832	1.563 % 0.521%	1.875 mm 0.895 mm

In the stage of cooperative assembly, the action recognition models based on multi-LSTM layers and the spatio-temporal attention mechanism are compared from accuracy aspect. For specific behavior, the temporal attention mechanism assigns attention weights to different timesteps, which is highest in timestep 8 in Fig. 15 (a). The attention of key points on human skeleton will also be assigned according to the importance of different key points, which is highest in key point 10 in Fig. 15(b). Fig. 15(c) shows the accuracy of different action recognition models. In that, the accuracy of recognition model will increase with the increase of training epochs. Meanwhile, the accuracy of action recognition model is related to the number of LSTM layers. When the number of LSTM layers is 4, the recognition accuracy reaches the maximum value (0.912453). This accuracy is lower than the spatio-temporal attention model with the accuracy of 0.966006. Fig. 15(d) presents the average waiting time of robot in cooperative tasks. For different experimental groups, each group has tested 10 assembly tasks. The sensitivity of robot is defined by counting the waiting time of robot after recognizing human actions. From Fig. 15(d), the average waiting time of robot in transferring part and cooperative assembly is displayed which is approximately within 4 s. The action recognition model locates in data management system of DT. So the waiting time of robot explains that current DT-based HRC assembly can meet the needs of efficient production.

Fig. 15.

The experimental results of HRC assembly.

The test of collaborative assembly refers to the stage of robot assisting human to fix the part after picking and transferring it. In Table 6 , the results of collaborative assembly test from aspects of assembly time, collision rate and assembly accuracy are presented. For assembly time, the collaborative assembly with D-DDPG only costs 12.453 s while it takes 14.564 s to finish the same task with DDPG. Compared to the previous models, 24.845 s is taken to complete the task by the pre-programming mode. The reason is that robot will move in a fixed path without adjusting the trajectory when the position of human changes. For assembly collision, the pre-programming assembly mode will cause a high collision rate (5.128 %) which is far from the collision rate with DDPG model (1.538 %) and D-DDPG model (0.513 %). The reason for this phenomenon is that robot can not adaptively adjust its posture according to the change of human posture. The assembly deviation is related to the assembly accuracy, which is reduced to 0.587 mm with the D-DDPG model. At the same time, the assembly deviation reaches 0.964 mm with DDPG model while it is 1.884 mm by pre-programming mode. This phenomenon is caused by the poor response of robot to assembly tasks with fixed program.

Table 6.

The performance of collaborative assembly with different methods.

Control method	Assembly time (s)	Collision rate	Assembly accuracy ($\Delta \mathit{d}$)
Pre-programming	24.845	5.128 %	1.884 mm
DDPG	14.564	1.538 %	0.964 mm
D-DDPG	12.453	0.513 %	0.587 mm

7. Conclusions and future works

In this paper, a DT-based HRC assembly framework is proposed. The DT model is established from three aspects: total elements, assembly process and assembly performance. Besides, the effect of DT model on HRC assembly is verified by experiments. The main conclusions are as follows:

• (1)The proposed DT-based HRC assembly system can improve the assembly efficiency, safety and accuracy.
• (2)The D-DDPG model in data management system promotes the process of task assignment, which makes the overall assembly efficiency increased by approximately 25.09 %. Meanwhile, the workload of operator is reduced by 20.24 %.
• (3)In HRC assembly, the trajectory of robot which is optimized by d-DDPG model is more suitable for dynamic environment.

In the future, it is worth to research the method of building the high-fidelity model in digital space. Besides, the multi-channel interaction between human and robot also needs to be studied for achieving better HRC.

Declaration of Competing Interest

The authors report no declarations of interest.