Generating Human Arm Kinematics Using Reinforcement Learning to Train Active Muscle Behavior in Automotive Research

Abstract

Computational human body models (HBMs) are important tools for predicting human biomechanical responses under automotive crash environments. In many scenarios, the prediction of the occupant response will be improved by incorporating active muscle control into the HBMs to generate biofidelic kinematics during different vehicle maneuvers. In this study, we have proposed an approach to develop an active muscle controller based on reinforcement learning (RL). The RL muscle activation control (RL-MAC) approach is a shift from using traditional closed-loop feedback controllers, which can mimic accurate active muscle behavior under a limited range of loading conditions for which the controller has been tuned. Conversely, the RL-MAC uses an iterative training approach to generate active muscle forces for desired joint motion and is analogous to how a child develops gross motor skills. In this study, the ability of a deep deterministic policy gradient (DDPG) RL controller to generate accurate human kinematics is demonstrated using a multibody model of the human arm. The arm model was trained to perform goal-directed elbow rotation by activating the responsible muscles and investigated using two recruitment schemes: as independent muscles or as antagonistic muscle groups. Simulations with the trained controller show that the arm can move to the target position in the presence or absence of externally applied loads. The RL-MAC trained under constant external loads was able to maintain the desired elbow joint angle under a simplified automotive impact scenario, implying the robustness of the motor control approach.

Introduction

Motor control mechanisms in humans manage and modify the stiffness of skeletal joints by generating active muscle forces as determined by the central nervous system (CNS). Active muscle forces enable the human body to maintain posture and balance, perform motor tasks, and react to external perturbations. In automotive loading scenarios such as pre-impact bracing in low-severity crashes, active muscle forces may alter the occupant response and injury modes due to the change in body kinematics during the loading phase or by stiffening up of joints. Computational human body models (HBMs) are extensively used in the automotive industry to predict the response of occupants and pedestrians in motor vehicle collisions (MVCs), leading to injury during the vehicle design phase. Incorporating active motor control mechanisms into HBMs will help improve our understanding of the mechanisms and tolerances of injury and will help accelerate the development of injury countermeasures.

Previous computational studies in vehicle safety research have widely used two approaches for modeling muscle control in HBMs. The first approach involves the application of a predetermined activation-time history to muscle groups responsible for carrying out specific motions around the joint. This simplified approach has been utilized in both multibody (MB) models [1,2] as well as finite element (FE) models [3–5] and have demonstrated the effect of muscle activation on the response of HBMs under external loads. Some studies have also derived an optimized activation scheme for muscles to maintain the corresponding joint at a predefined position [6]. Unfortunately, these predetermined activation schemes have limited utility for general use applications of HBMs, and these approaches cannot be used under the various cases of loading where activation patterns or targeted kinematics are not known a priori.

The second control approach, which has become state-of-the-art in active muscle control for HBMs used in automotive applications, is based on closed-loop feedback (PID) mechanisms that drive muscle activation levels by some error signal based on current and target joint positions or muscle lengths. These controllers are designed to output muscle forces or activation levels to restore the HBMs to a desired position. Kistemaker integrated a PID controller into a MB model of the upper arm to reproduce fast elbow motion like in humans [7,8], using the error calculated as the difference between muscle length and a target muscle length (equilibrium point control). The number of muscles was reduced by combining different muscle units into four lumped muscles, with two of the lumped muscle responsible for elbow extension and the remaining two for flexion. The output from the PID controller was muscle stimulation for each group. One limitation of using target muscle length for control is that it is difficult to correlate the muscle length with different values of target joint angles. Östh used a similar approach to model musculoskeletal control in a FE model of the arm [9]. In the FE model, nine different muscle units were modeled, and the control approach grouped the muscles as extensors and flexors. The PID controller was developed using the joint angle error as the input, and the control signal was used to derive the muscle activation level for each group. The PID controllers have also been used on whole body HBMs to predict the occupant response during motor vehicle load cases. Östh et al. used three PID controllers with head, neck, and lumbar spine angles as the error signal [10] for posture control during a frontal crash. Iwamoto et al. used an FE model with a PID controller to control the kinematics of an occupant in a low severity side impact scenario [11]. Twenty-two PID controllers were modeled to control the joint motion with the joint angles as error signals with muscles classified into right or left control groups. Martynenko et al. used PID controllers with 370 active muscles of the neck, torso, and upper extremities [12]. The error signal for each PID was based on target muscle lengths to predict the body kinematics of the occupant under the combination of emergency braking and lane change maneuvers. Inkol et al. used nonlinear torque generators for HBM joints in place of the control of Hill-type muscles [13]. Parameter control optimization were done to simulate athlete performance in golf, cycling, and wheelchair propulsion. Walter et al. [14] developed a hierarchical control architecture with PID controllers to maintain posture and simulate squat movement using full body musculoskeletal model with 36 muscles.

The feedback based PID control mechanisms require precise tuning of controllers to fit the validation data generated by volunteer testing. Although useful under many circumstances, the PID control approach suffers from two major limitations: (1) the controllers are tuned for a limited range of possible external loads and may not output accurate responses beyond the loading scenarios they are tuned for and (2) the feedback controllers require a predefined muscle recruitment strategy on how muscles are organized as agonist or antagonist groups for the preferred joint kinematics. The human musculoskeletal system is redundant in nature; there are more actuators (muscles) than the kinematic degrees-of-freedom (DOF) at the joints. For simple joints, assumptions can be made regarding extensor and flexor groups, however for more complex body regions, it is not feasible to isolate individual muscles responsible for motion along any standard joint direction for a generalized response. Rather, movements are carried out by intricate coordination of different muscles activated at different points of time by the CNS [15–17], which is difficult to replicate using linear feedback gains [18,19].

The present study aims to use reinforcement learning (RL) for motor control in HBMs. Deep RL algorithms are recent advances in the field of machine learning which use an iterative approach to train the controller to generate desired outputs through a system of rewards and penalties [20]. Reinforcement learning is a biologically inspired learning routine that allows for the controller (called agent) to identify the optimal sequence of actions to take from a given state of the control environment to achieve a predefined goal [21]. State refers to the parameters that can be used to define the control environment. Reinforcement learning enables the control model to learn how muscle actuation affects joint kinematics by rewarding good responses and penalizing bad responses. This process is analogous to how children learn to use and coordinate their muscles to eventually interact with the environment around them. Deep RL leverages the performance of neural networks with reinforcement learning algorithms that enable the agent to reach the desired objective [22]. Neural networks (NNs) can be considered as function approximations, which map the state action pair to its corresponding value. In a RL problem, the neural networks are trained to predict the efficacy of each state-action pair and take the best possible action. The efficacy of actions is quantified using a reward function, which awards desirable actions (actions that moves the system closer to its objective) and penalizes the undesirable actions.

In this study, deep deterministic policy gradient (DDPG) is used to model the agent, which belongs to the class of RL algorithms called the actor-critic method [23]. One advantage of DDPG is that it can be used in continuous action spaces [24,25]. DDPG concurrently trains two neural networks to identify and output the best action. The actor network is the policy network that maps the state parameters to actions. The critic network updates the Q-value based on the state parameter and action from previous time-steps. The Q-value of a state-action is the cumulative reward that the agent is expected to receive when it undertakes the given action from the present state. The actor network outputs the action, which maximizes the expected reward.

Deep RL algorithms have demonstrated the ability to show human-level performance while playing video games [22] and learning complex games like alpha-go from scratch [26]. RL for control has also been used in the field of robotics for navigation in complex environments [27–30]. HBMs present a further control challenge due to physiological redundancy and nonlinearity of the musculoskeletal system.

Previously RL control schemes have been used for simulating arm reaching tasks [31–34], for generating motion about shoulder joint [35] and for maintaining the stability of joints under gravity [36,37]. Deep RL algorithms have also been used for synthesizing locomotion using multibody models of humans [38,39] and animals [40], control the kinematics of neck FE model in the sagittal plane [41] under rear impact scenario, and have aided in the design and analysis of limb assistive exoskeletons [42]. Human beings can adapt to changes in external loads [18,19], and the ability of RL agents to replicate such adaptive response have been studied previously [43]. While previous RL musculoskeletal control studies for motion related tasks have been performed in detail [31–33,39,44,45], the ability of RL control mechanisms to extend to dynamic events such as automotive impacts, where the response time is much faster, and the environment is more chaotic, remains to be studied. The biofidelity of muscle activation patterns corresponding to joint stability in such cases and the effect of change in external environment on the muscle synergy also need to be investigated.

The current study makes use of DDPG algorithms to model and integrate active muscle control in HBMs with the aim of evaluating the biofidelity of the controller and verifying its adaptability in automotive environments. For this purpose, a MB model of the human arm with the anatomy of an average (50th percentile) male has been developed to demonstrate the utility and implementation of this active muscle modeling approach using a simplistic anatomical model. The reinforcement learned muscle activation controller (henceforth, referred to as RL-MAC) developed in this study was used to simulate the desired motion at the elbow or maintain the stability of the joint in the presence of external impact loads.

Methodology

The human arm MB model was developed in matlab r2020b using the Simscape multibody toolbox. The multibody model was integrated with the matlab reinforcement learning toolbox to develop the RL-MAC and carry out training and simulation of the control model.

Development of the Arm Multibody Model.

A simplified model of a 50th percentile human arm was developed comprising of scapula, humerus, radius, and ulna (Fig. 1(a)). The bones were modeled as rigid bodies. In this study, only the extension–flexion motion about the elbow was considered, thus the glenohumeral joint was constrained. The radius and ulna were also combined into one rigid body, and mass and inertial properties of the lower arm (below the elbow) were applied. In the present study, the focus was on reproducing the elbow extension–flexion motion, thus a revolute joint was defined between radius–ulna and humerus at the elbow with a joint stiffness of 0.6 N·m/rad [9,12]. Joint damping of 0.4 N·ms/rad was also used to prevent any minor oscillations at the neutral position in the passive condition. Previous studies have measured the damping of the elbow joint between 0.2 N·ms/rad and 1 N·ms/rad [46,47]. Popescu et al. also determined that the damping value was almost constant during the entire time of the elbow rotation [48]. The simulation space of the elbow was restricted at the limit of the extension–flexion angle between 0 deg to 160 deg when measured from full extension of the arm.

Fig. 1.

Multibody model of the human arm: (a) rigid bones with elbow joint and (b)MBmodel with muscles

The muscles in the model were defined with the suitable origin and insertion points [49,50], along the line of action of the muscle forces. Force magnitudes were calculated according to the Hill-type muscle model considering the muscle length and contractile velocity during the simulation.

Hill-type muscle model provides for the calculation of muscle forces in numerical analysis [51,52] (Fig. 2). Hill-type muscle consists of a contractile element (CE) simulating the active forces generated by the muscles (F_CE). The passive element (PE), which is in parallel to the contractile element, computes the forces due to muscle stiffness (F_PE).

Fig. 2.

Hill-type muscle model with contractile element and passive element

The forces generated by the muscles are nonlinear in nature and depend on the muscle length and the muscle velocity. The active muscle forces are stimulated by the CNS, which generates the muscle activation levels (a_t). The activity level or activation varies between 0 (fully passive) and 1 (fully active), and its value is determined by the CNS depending on external loads and the current joint stability.

The Hill-type muscle parameters like normalized force-length (F_l) and force-velocity (F_v) relationships are shown in Fig. 3. The total forces generated by the active part of the muscle are dependent on F_l, F_v, and the activation level (a_t) [52,53]

$$F_{\text{CE}}=\hspace{0.17em}{a}_t\times F_{\max }\times \hspace{0.17em}{F}_l(L)\hspace{0.17em}\times \hspace{0.17em}{F}_V(V)$$ (1)

Fig. 3.

Hill-type muscle parameters: (a) force-length relationship of active (F_l) and passive (F_P) elements and (b) force-velocity (F_V) relationship of the active element

F_max is the maximum force a muscle can generate, which is a characteristic property of the muscle and is dependent on the anatomical cross section area. The active tension–length curve F_l describes the relationship between active muscle forces and normalized length of muscles (L), with the maximum force which occurs at an optimal length (L_opt). The F_V curve is the relationship between contractile velocity and F_CE. When the velocity is positive, i.e., the muscle elongates, the force (F_v) asymptotes at a value near F_max.

For the passive element, the force was calculated using an exponential function of length. The passive force (F_P) only starts acting when the length of the muscle exceeds the optimum length [54]

$$F_{\mathrm{PE}}=\hspace{0.17em}\frac{1}{\exp \left(K_{\mathrm{sh}}\right)-1}\hspace{0.17em}\left\{\exp \left[\frac{K_{\mathrm{sh}}}{L_{\max }}\left(L-1\right)\right]-1\right\}\hspace{0.17em}\hspace{0.17em}\mathrm{for}\hspace{0.17em}\hspace{0.17em}L>1$$ (2)

K_sh is a dimensionless parameter influencing the rise of the passive force with length. The total force generated by the muscle is the sum of the magnitudes of passive force and active force.

The MB model incorporated muscles responsible for both the flexion and extension motion (Fig. 1(b)). The muscles included in the model are the biceps brachii long head and short head, the brachialis, the brachioradialis, the pronator teres, the extensor carpi radialis longus, and the triceps long head, lateral head, and medial head. Some of the muscles which have large cross section and wide bone insertion regions were divided into several strands to distribute the muscle forces. The Hill-type muscle parameters like normalized force-length (F_l) and force-velocity (F_v) relationships were defined as a curve and were identical for each arm muscle [5,55].

The muscles origin and insertion points in the model are approximated from various sources of anatomical data available [49,50]. The optimum muscle length is the length of the muscles at the neutral position when the humerus and radius are at right angles to each other [56]. The muscle properties included in the model are tabulated below (Table 1).

Table 1.

Properties of muscles in MB model

Muscles	Fmax (N) [9]	No. of strands in model	Lopt (mm)
Biceps brachii long head	360	1	336
Biceps brachii short head	248	1	327
Brachialis	568	2	166, 156
Brachioradialis	152	1	283
Pronator teres	320	1	157
Extensor carpi rad longus	176	1	312
Triceps long head	456	1	324
Triceps lateral head	360	1	296
Triceps medial head	360	3	239, 211, 170

Note: Optimum length (L_opt) of the muscles is obtained at the neutral position (90 deg flexion angle).

Muscle Control Framework.

As discussed before, the CNS actuates different muscles in coordination for carrying out any motion about a joint. Muscle forces are highly nonlinear in nature and are affected by the delay between the neural stimulation generated by the CNS and the actuation of the muscle. Humans can also adapt to changes in external environments during their movements, and Smeets et al. argued that human movement patterns cannot be explained using simple feedback mechanisms [18]. This study explores the feasibility of using the RL-MAC mechanism for muscle control under varied external environments. The RL-MAC used in this paper uses deep NN, which can efficiently approximate nonlinear behaviors from known predictors [57]. Trained RL agents have also been found to adapt to changes in environments in various applications [58]. All these factors make RL-MAC potentially suitable for HBMs.

Figure 4 shows the RL-MAC framework for controlling arm motion by incorporating a DDPG agent. The RL-MAC reads the state parameters from the arm multibody model. The controller outputs neural stimulations (u_t), which are converted to muscle activation (a_t) using activation dynamics. The resultant activations are applied to the corresponding Hill's muscles in the MB model, which also obtain the value of muscle length and velocity from the model to simulate the muscle forces (Eq. (1)). The active muscle forces are tensile in nature, i.e., forces developed try to pull the origin and insertion points toward each other. During the motion, different sets of muscles are activated by the RL-MAC to carry out the relevant joint motion.

Fig. 4.

RL-MAC framework for the arm model

The state of the controller is defined by the elbow joint angle, the joint velocity, the error (difference between target angle and current angle), and the muscle activations. The activation (a) buildup in the muscles as a result of the neural stimulation (u) using activation dynamics proposed by Zajac [52]

$$\frac{da}{dt}=\frac{1}{{\tau }_{\mathrm{act}}}[u-(1-\delta )au-\delta a]$$ (3)

The τ_act is the time constant for generating muscle activity from neural stimulation, which represents the time delay between the neural stimulus (u) and the commencement of muscle activity (a). δ is the ratio ${(\tau }_{\mathrm{act}}/{\tau }_{\mathrm{deact}})$, where τ_deact was the depletion time constant which controls the time lag of drop in activation level (a) after reduction of the stimulation (u). The length and velocity were determined during the simulation by measuring the Euclidean distance between the origin and insertion points of the muscles. The parameters used to calculate the muscle forces are tabulated in Table 2. The total muscle forces were calculated as the sum of forces generated by the contractile element and passive element.

Table 2.

Muscle and activation parameters used in MB model

Muscle parameters	Value
τ act	0.02 [59]
τ deact	0.06 [59]
Minimum activation (ao)	0.005
F–L curve	Graph input [5]
F–V curve	Graph input [5]
K sh	6.15 (Flexors) [9] 3 (Extensors)

The overall architecture of the RL-MAC, which was based on the DDPG agent (Fig. 5), was similar to the one proposed by Lillicrap et al. [23]. The actor network consisted of a feedforward neural network with one hidden layer between the input and the final layer. Input to the hidden layer and final layer was activated with a rectified linear (ReLu) function. The output of the final layer was activated using a sigmoid function as the action space varies uniformly between 0 and 1 representing the neural stimulation (u_t). The number of nodes in the final layer was equal to the number of muscles required to be stimulated. The critic network is comprised of three layers with the ReLu transfer function after the hidden layer. The input of the critic network included the state parameters and the actions from the actor network. The state observations were activated with a ReLu function before connecting to the hidden layer. The actions from the actor network are connected to a hidden layer skipping the ReLu activation [23]. The critic layer outputs the Q-value associated with the state-action pair. Since the Q-value is a scalar value, the critic network has one node in the final layer.

Fig. 5.

Architecture of DDPG agent

In this study, the objective of the RL-MAC was to perform goal directed motion of the elbow joint within its range of motion and maintain its stability in the presence of external perturbations. The agent was trained to move the forearm to a target position from any given starting position and stabilize it at the final position. During the training phase, the controller learns to minimize the error between the current angle and the target angle. In the reward function, the controller is penalized proportionally to the magnitude of the error. The reward function also awards the agent if it manages to stabilize the elbow angle within 0.1 rad (5.7 deg) of the target value. Due to the redundant nature of the human musculoskeletal system, there are many combinations of muscle activation patterns that can produce the same desired movement. One method to reduce the ambiguity of the muscle activations scheme that is commonly used in muscle activation biomechanics is to minimize the metabolic costs of active muscle activity [60]. There are two major metabolic cost functions associated with muscles that are most commonly used. The first is the energy cost which seeks to minimize the total forces generated in the muscles or work done by the muscles [61,62]. Second is the muscle fatigue or muscle effort cost function, which considers minimizing the muscle activation over the time of motion [63]. In the control model, we have considered minimizing the activation, but alternate energy costs can also be implemented

$$\text{Reward}\hspace{0.17em}=\hspace{0.17em}-\alpha \hspace{0.17em}\text{Error}\hspace{0.17em}-\beta \hspace{0.17em}\sum a(t)\hspace{0.17em}+\hspace{0.17em}\gamma \hspace{0.17em}(|\text{Error}|\hspace{0.17em}<\hspace{0.17em}0.1\hspace{0.17em}\text{rad})$$ (4)

The reward calculated using Eq. (4) at each time-step of the simulation, while the agent tries to maximize the cumulative reward over the simulation time. α, β, and γ are positive constant terms that calibrate the different components of the reward function. Equal weightage was assigned to the activation of each muscle in the reward function. During the training of the RL-MAC Ornstein–Uhlenbeck (OU) process was used to add noise with a standard deviation of 0.09 for adequate exploration of the action space. OU noise was found to explore the action space better for coordinated actuation of muscles [40]. The RL training was considered to converge when the average cumulative reward over the most recent 250 iterations reached a predetermined value.

Model Evaluation, Training, and Validation.

Before training the control model using RL-MAC, the passive structural behavior of the arm MB model was evaluated. A moment with a magnitude varying between −1 N·m and 1 N·m was applied at the elbow revolute joint with the humerus fixed, and the rotation of the forearm was measured. The resultant stiffness of the joint was calculated from the moment-angle data, and the magnitude was verified with published literature [56,64].

After verifying the structural stiffness of the MB model, the control training was carried out with the system of rewards mentioned in Eq. (4). The model was trained in two different scenarios described in the Training Scenario 1: Bare Arm Motion Control and Training Scenario 2: Arm Motion Control Under External Load sections (also summarized in Table 3), and the ability of the trained model to synthesize motion under a novel environment was also evaluated (testing scenario, summarized in Table 3). In both cases, the humerus and the scapula were fixed, and the forearm was free to rotate about the elbow revolute joint. The elbow angle was measured from the arm's extension limit (Fig. 6(a)).

Table 3.

Training and simulation scenarios

Training/simulation case	Load applied	Model response
Evaluation of passive structural response	Torque at the elbow revolute joint	Moment-angle response of the joint
Training scenario 1: targeted motion of the forearm with RL-MAC integrated MB model	No external loads	Angle-time response of the elbow joint
Training scenario 2: targeted motion of the forearm under external loads	Point mas attached to the radius and gravity	Angle-time response of the elbow joint
Testing scenario: response to novel loads	Simplified crash pulse applied to the humerus proximal end	Angle-time response of the elbow joint

Fig. 6.

(a) Arm parameters for training, (b) flexor muscle group, and (c) extensor muscle group

Training Scenario 1: Bare Arm Motion Control.

In the first scenario, the MB arm model with integrated RL-MAC was trained to perform a fast goal-directed motion of the elbow joint from a given initial position to a target position of the joint. For the purpose of training, two different muscle recruitment strategies were used. In the first recruitment strategy, called group activated muscle recruitment (GAMR), the arm muscles were grouped as extensors and flexors (Figs. 6(b) and 6(c)), and identical activation was applied to muscles belonging to the same group. The flexor group included the biceps brachii long head and short head, the brachialis, the brachioradialis, the pronator teres, and the extensor carpi. The triceps long head, lateral head, and medial head made up the extensor group. For this case, the actor network had two nodes in the output layer prescribing the stimulation for the extensor and flexor groups. The second recruitment strategy was called individual activated muscle recruitment (IAMR), in which each muscle was actuated individually regardless of being an extensor or flexor and having an activation level independent of the activation levels of other muscles. For this activation scheme, the actor network had nine nodes in the final layer, one for neural stimulation for each muscle. During the training process, the initial and target angles were randomly varied in the elbow rotation space so that the trained agent does not overfit to any particular set of input data.

After the activation models were trained to convergence, the arm was repositioned to test the model's response to a series of initial and target angles based on independent human volunteer data from available literature [8]. The validation case used in this study required six male volunteers to perform fast targeted extension or flexion of the arm, and the elbow angle—time history of the motion was recorded. Due to the simple nature of the test setup, a similar motion could be replicated in the MB model. Depending on the rotation case, the arm was stabilized at the initial angle for the first 100–125 ms before the target angle was set to the desired value for the rotation. The angle-time data generated by the trained RL-MAC was compared with the volunteer kinematics data to verify its efficiency.

Training Scenario 2: Arm Motion Control Under External Load.

Training is performed using the targeted arm motion detailed in scenario 1, but in this case, a mass was attached to the radius (Fig. 7(a)), and gravity was introduced in the multibody model. The model was trained using the IAMR scheme, and the reward function is described in Eq. (4). Only changes to the multibody environment were made, and no change was made to the control model for this scenario, i.e., no additional information on the magnitude of the added mass or direction of gravity was provided to the controller. This enabled the RL-MAC to formulate a response pattern, which was independent of the external loads applied. During the training, the added mass was randomly varied between 1 kg and 5 kg (in increments of 1 kg), and the direction of gravity was adjusted every iteration such that the arm was trained to perform the flexion–extension motion both along and against the gravity. The results from the trained model in scenario 2 enabled the assessment of the utility of the RL-MAC framework for training the HBM using the same architecture for varied sets of loads generally associated with to HBMs.

Fig. 7.

(a) Arm model with variable mass and (b) the trained arm model subjected to load case representing simplified automotive impact

Testing Scenario: Response to Novel Environments.

The ability of the RL-MAC framework to produce a robust response within an environment for which it was not explicitly trained was also evaluated in this study. The model trained in the presence of an acceleration field (scenario 2) was subjected to a simplified crash pulse representative of the inertial load experienced by the upper extremity in a typical frontal MVC (Fig. 7(b)) [65]. The radius was pinned with a revolute joint representing the occupant's hand position on the steering wheel. The scapula was free to move in the planar direction. A stiffness of 1000 N/m was applied to the scapula in the vertical direction to effectively reproduce the effect of the lower body weight. The crash pulse was applied to the scapula in the horizontal direction. At the start of the simulation, the arm was set at a neutral position (flexion angle = 90 deg). The period of the pulse was similar to that used by Happee et al. to design a linear controller for two-dimensional arm motion [66]. However, the pulse magnitude applied to the MB scapula (Fig. 7(b)) was measured in the thoracic region in automotive frontal tests [65] and was higher than the constant force field under which the agent was trained. The RL-MAC trained in scenario 2 was evaluated in its ability to stabilize the arm under forward loads (representing frontal collision) and in rearward loads (rear collision). The testing step is important to verify the range of utility of the trained RL-MAC agent.

Results

The response of the passive part of the MB arm model was measured and compared with previously published data. The stiffness of the model with inactivated muscles was found to be 0.955 N·m/rad. The stiffness magnitude measured was close to the values reported by Hayes and Hatze [56], Wiegner and Watts [64], and Howell et al. [67]. The resultant stiffness of the elbow joint with the unactuated muscles is due to the combination of prescribed revolute joint stiffness and passive muscle stiffness. Previous studies have added damping to the muscles to improve the passive and active behavior at the elbow joint [9,68]. In this study, the damping of muscles was not considered; thus, all the nonlinearity at the joint was the result of the joint damping assigned.

For the training cases, the MB model with RL-MAC was simulated for 600 ms in each episode of the training. The training was distributed over 20 CPUs using the matlab parallel computing toolbox. Each episode took 20–30 s to simulate, and the DDPG agent was updated every 5 ms of the training simulation.

Training Scenario 1: Bare Arm Motion Control.

The arm model was trained to reach a randomized target joint angle from any randomized initial angle within the joint range of motion for extension/flexion. The training was carried out with both the IAMR and GAMR schemes. The variation of average reward with each iteration is displayed in Fig. 8. The average reward value considered for convergence was 1500, but it can vary depending on the control objective and the reward function. The IAMR converged faster compared to the GAMR scheme. A possible explanation can be that since the muscles are actuated individually, IAMR provides greater overall control of the arm motion. Also, given the randomized nature of the training algorithms, the reward versus episode response during the training varies slightly for each time the model is trained.

Fig. 8.

Plot showing the variation of average reward for individual activation (IAMR) and group activation (GAMR) with each episode during training. (The reward averaged over 250 episodes).

During the testing and validation phase of this scenario, the angle time histories of the movements produced by the trained RL-MAC were compared with volunteer kinematics angle-time plots and showed excellent agreement with the experimental data (Fig. 9). Both the IAMR and the GAMR could reproduce the motion of the volunteers in both flexion and extension after completion of training. In some cases, the simulated angle was lower or higher than the target angle due to the nature of the reward function, which allowed for the error of ±0.1 rad.

Fig. 9.

Elbow rotation angles by trained RL controllers in response to prescribed target angles and comparison with volunteer data with the individual (IAMR) and group (GAMR) activation scheme: (a) arm in flexion and (b) arm in extension

The muscle activity level for the duration of arm motion was also measured for both the activation strategies (Fig. 10). The IAMR model predicted different activation levels for each muscle during the simulation, and these responses were different than those using the GAMR scheme. In the IAMR scheme, some of the muscles remained inactivated throughout the duration of motion. The activation decreases rapidly for the muscles once the joint has been stabilized at the intended position. The GAMR scheme, which prescribed identical activation to all the muscles of the same group, resulted in long-term, low-magnitude activation for both extensors and flexors trying to balance each other even after the target position was reached.

Fig. 10.

Muscle activation plots for extensor and flexor muscles: (a) arm in flexion: 45–145 deg and (b) arm in extension: 145–45 deg

The muscle activity patterns in the arm simulations are biphasic or triphasic in nature, similar to that expected in fast goal directed movements [69,70]. At the onset of the error signals, the agonist muscles are activated, and after an initial burst, the activation declines. The initial activities of the agonist muscles are directly related to the amount the elbow is required to rotate [71]. Near the completion of the motion, the antagonist muscles are activated, causing deceleration of the limb. In some of the simulations, we also saw a second burst of muscle activity in the agonists [72]. Happee argued that the third phase of muscle activation is essential in some goal directed movements where the antagonist activations have not reduced significantly once the target position is reached [73]. In our simulations, the third activation phase was more prominent in cases where the initial error was low or when a mass was attached to the distal forearm (scenario 2).

Training Scenario 2: Arm Motion Control Under External Loads.

The training of the RL-MAC using the IAMR scheme with the added random loading under scenario 2 converged in 8947 episodes. The trained RL-MAC was tested in its ability to produce extension–flexion motion and adapt to a range of mass attached at the radius (Fig. 11).

Fig. 11.

Rotation time history of the arm in presence of external load: (a) arm in flexion: 45–135 deg and (b) arm in extension: 135–45 deg

The trained model was able to perform the desired motion in the presence of external loads in the extension and flexion of the elbow. The elbow angle remained stable at the end of the simulation for both loading modes. During the flexion motion, the arm was able to carry weight up to 4.8 kg, and activation patterns for each muscle were different for different values of applied loads (Fig. 12). For the 4.8 kg flexion case, the activations of the arm flexor muscles were nearly at maximum, indicating that the 4.8 kg weight was the structural load limit of the MB arm model during flexion. The arm could carry up to 10 kg against gravity in extension, even though the mass was limited to 5 kg in training, showing that the trained RL-MAC than that for which can actuate muscles for loads different it has been trained. For extension, the arm extensor muscles achieved a maximum level of actuation while carrying a 10 kg mass.

Fig. 12.

Activation time history of the agonist muscles in presence of external load (a) arm in flexion: 45–135 deg (solid—1 kg, dashed—4.8 kg) and (b) arm in extension: 135–45 deg (solid—1 kg, dashed—10 kg)

The muscle activation profiles show a similar triphasic pattern. With the increase in mass against gravity, the antagonist activity period was shorter, followed by a noticeable burst of agonist activity in the third phase. The drop-in agonist activity after reaching in target angle during flexion, even with maximum load, was due to a decrease in the moment arm. On the contrary, the agonists in extension motion remain actuated throughout.

Angle-time history of RL-MAC trained in scenario 2 was simulated in the absence of external loads, and the response was compared with the volunteer tests, and scenario 1 trained RL-MAC with the IAMR scheme. The peak velocity of the RL-MAC in scenario 2 during the goal directed motion was lower than that seen in the experiments (Fig. 13). Nevertheless, the elbow stabilized at the target angle around the same time, after 300 ms of the onset of the error signal for both trained RL agents.

Fig. 13.

Comparison of RL-MAC from training scenario 1 (IAMR) and scenario 2 (a) arm in flexion: 45–145 deg and (b) arm in extension: 145–45 deg

Testing Scenario: Response to Novel Loads.

The model trained in scenario 2 was used to control the arm motion under a simplified impact scenario design to replicate a driver holding on to the steering wheel and bracing for a frontal MVC. The response generated with the trained IAMR in scenario 2 under the simplified MVC pulse was compared to the response of a completely passive arm model without the active muscles (Fig. 13). The arm was kept at a neutral position (90 deg) at the start of the simulation followed by application of MVC crash pulse (Fig. 7(b)) after stabilization of the joint. The crash loading was applied horizontally at the scapula and humerus proximal end for 100 ms. During the application of the crash pulse, the objective of the active arm model was to maintain the neutral position. The active model was able to maintain the stability of the joint position of the elbow for both loading conditions. The passive case, however, was unstable for the duration of the application of the load (Fig. 14).

Fig. 14.

Rotation time history comparison of the arm in the presence of external pulse (a) load applied in rear and (b) load applied forward

Discussion

The present study demonstrated the development and implementation of an active muscle control framework for a simplified human arm model based on recent RL algorithms with the intention of simulating untrained impulsive loading responses typically seen in the automotive crash environment. The RL algorithms have been used in other active human body simulations typically associated with the control of body motion [31,34,38,42,74,75], but the applicability of RL-MAC to the automotive environment has not been studied.

In this study, training of the RL-MAC was carried out with two different activation schemes. The RL-MAC was trained to minimize the error between the current and the target angle using minimum possible activation. The activation minimization criterion was included to ensure that the extensor and flexor muscles do not cocontract at the stabilized final position. Previous arm movement control studies have used the contractile element length error during the development of feedback controller [8,12]. Kistemaker et al. developed a muscle controller based on the feedback of CE element length (λ control) to simulate the same volunteer tests [7,8]. Martynenko et al. used a similar approach to model the fast arm movements [12]. In both studies, presimulation with the passive arm was performed to determine the target CE lengths of the muscles. In a more complicated anatomical system with more DOFs, CE length for a corresponding joint position is complex, and a strategy using CE length as the controller feedback to move from one arbitrary position to another may be challenging to determine. In the current study, we used elbow angle error for the RL-MAC representing a proprioceptive signal to get to a randomly defined targeted position from a randomly defined initial position [9]. Happee et al. developed a controller for the stability of the head and neck multibody model with both the head kinematics feedback and muscle length feedback to maintain the head at the neutral position [76]. However, in that study, the initial and targeted positions (and thus muscle lengths) were identical.

The trained agent could produce the required motion in both extension–flexion from any given start angle, which indicates that the controller was trained to generate arm kinematics independent of the initial position of the forearm. Both the activation schemes resulted in trained models that had similar kinematics when tested under unloaded motion scenarios, although evaluation of the activation patterns showed that the individual activation schemes actuate only those muscles which are required for the motion and imparting low activations to the others. In the grouped muscles activation framework, all the muscles belonging to the same group were activated together at the same level, which may result in a higher energy cost of motion. Grouping muscles together is a simplification that may lead to accurate external kinematics but incorrect internal loading on the tissues. The separate muscle scheme is likely more biofidelic. However, further investigation is required to clearly identify the effects of the activation frameworks on the HBM.

The control architecture could also train the arm model when the MB model was modified with added randomized point mass in a constant gravity field. This information was not provided to the RL agent, instead the controller had to formulate a generalized response of the muscles with the same states as in training scenario 1 based on how the model was performing relative to its objective. Apart from the reward function, the RL-MAC framework required negligible user input on the performance of the arm model or how the various muscles should behave as a system to accomplish an objective. Adding mass and gravity direction to the RL-MAC state could improve the learning process and overall kinematics of the arm MB model as both the parameters are important for the control process [17,19]. However, expanding the states to include mass and gravity would result in potentially overfitting the agent to the training case. Instead, RL-MAC devised a muscle control strategy to respond to the sudden change in mass from kinematics based-feedback. This kind of muscle activity modification is not possible using linear closed-loop controllers as multiple muscles need to be actuated simultaneously [18,19]. It was also found that linear models for neuromuscular control overfit the training datasets [66]. In the RL-MAC training process, both the initial kinematic parameters and environmental variables can be modified in each iteration, which can reduce overfitting. The generalized RL-MAC (scenario 2) in the absence of an external force field was found to undershoot the response of the volunteer dataset, but the overall kinematics was similar during the time frame of evaluation (Fig. 13). The initial disparity in response is because the RL-MAC in scenario 2 has not been provided with the complete information of the environment and has to rely on the kinematics feedback from the arm MB model to decipher the constant force acting on the forearm in each iteration.

The two training scenarios have demonstrated the ability of the RL framework to actuate independent muscles within a system to achieve a response. This feature will be critical for body regions more complex than the arm, where it is difficult to identify and associate different muscles working in a system to achieve motion in many different DOF. The RL-MAC efficiency was found to depend primarily on how the reward function was defined. Crowder et al. reported that an inaccurate balance between coefficients of the reward function might not be able to train the musculoskeletal system even with an increase in training time [33]. Hence, to improve the control framework, it is desirable to explore how the different components of the reward function affect the controller response and potentially inform the reward function development with various sources of new or existing volunteer data.

Reinforcement learning algorithms have the potential to train active HBMs to control their motion using scenarios where there is the potential for a lot of human volunteer data (e.g., lifting weights or exercising) [43], while providing good generalization and response in untrained cases where there is not an abundance of data (e.g., muscle response during impact). This is analogous to how humans learn to control their body motion from everyday activities but still, they need to react in some fashion to scenarios they have never encountered (like in an automotive crash). In this study, it was shown that the RL-MAC was able to maintain arm stability under conditions comparable to a driver bracing on the steering wheel before an automotive frontal crash pulse. This is notable because the muscle control model with arm kinematics as input was only trained to control its motion by using a simple weightlifting scenario (training scenario-2), and the nature of the loads in an impact scenario is substantially different with higher magnitude forces acting on the elbow joint for a shorter time span. Although this case is a simplified representation of a human's response to a crash, this study demonstrates the potential for a RL controller to respond appropriately under a different set of loads for which it has not been explicitly trained. This potentially makes for a more generalizable muscle control scheme than the traditional PID approaches and eliminates the need for retraining or calibrating the model under multiple load cases, which can be computationally expensive. With RL-MACs, training can be accomplished on a simplified, fast-running model, and the trained controller can be transferred to a more complex and detailed HBM capable of simulating impact and injury. The ability of trained RL controller to generate response under novel scenarios is also useful for the development of injury countermeasures and assistive devices [42] and for training and deployment of biomimetic devices [44,77] for which the nature and models of loads acting on the devices in real-world scenarios can be different from the training environment.

The purpose of this study was to implement and demonstrate the utility of the RL-MAC for training active muscle responses in a human body model for automotive scenarios, and to accomplish this, simplifications were made to the human body model. A multibody model of the human arm was developed using Hill-type line muscles, and for this study, the series element in the Hill-type muscle have been assumed to be rigid as the focus was the development of the controller. Some previous studies have considered a series element in Hill's muscle model to represent the stiffness of tendons [78]. However, in a study by Bayer et al., it was shown that the series element of the Hill-type model contributes around 7.6% of the total muscle forces [59]. In this study, the passive structural properties of the model were verified with the stiffness data available in the literature, demonstrating that the assumption of rigid tendon here did not affect the overall response of the arm MB model, and it certainly did not affect the objective of the study.

The MB model of the arm was simplified with a revolute joint at the elbow, the pronation, supination, and motion about the elbow were not considered. Only the flexion–extension motion of the arm was modeled mainly to compare the current RL-MAC with volunteer data [8] and similar arm models with PID controllers [8,9,12]. The potential of the deep RL controllers will be better understood in body regions like the head and neck, which have multiple nonlinear joints and complex coordination of muscles which makes the control system more complicated [76,79,80]. The development of a control model for such body regions with input kinematic parameters and reward functions will be assessed in future studies. In this study, we have penalized muscle activation to minimize fatigue (Eq. (4)). The effect of implementing other cost functions such as muscle work [62] or joint energy expenditure [81] also needs to be studied. Further, it also needs to be investigated whether a trained RL-MAC can adapt to changes in anthropometries.

Conclusions

This study implemented a methodology for integrating a robust muscle control mechanism of the elbow joint. In the current study, RL-MAC could produce goal directed arm movement, synthesize the same motion in the presence of a constant force field, and the trained controller could react to high magnitude impulse loads which provide evidence of its potential for controlling HBMs in commonly encountered chaotic scenarios. Such a control mechanism is important for motor vehicle impact scenarios which can be injurious, and thus collecting volunteer data under these situations is not possible.

The current control methodology can be extended to study the response of other more complex body regions with numerous muscles in an automotive environment and can be incorporated at the whole-body level. Active HBMs will be important tools for the development of improved injury countermeasures and protective gear in the future. Furthermore, the RL-MAC framework can also be used in biomechanics applications such as gait and occupational health research.